Students in the Wikipedia Education Program in the United States and Canada improved article quality on the English Wikipedia by an average of 88 percent during Spring 2012, according to new research conducted by Luis Campos, an external data analyst. In the Wikipedia Education Program, professors assign their students to improve course-related articles, with support from Wikipedia Ambassadors who help students learn the basics of Wikipedia editing.

Experienced Wikipedia editors evaluated a random sample of articles students worked on as part of the Wikipedia Education Program in the United States and Canada in Spring 2012. The metric evaluators used, with assessment areas for comprehensiveness, sourcing, neutrality, readability, formatting, and illustrations, on a 26-point scale, is based on the Wikipedia 1.0 metric used across English Wikipedia. Evaluators provided two ratings, one for the article quality immediately prior to the first edit the student made, and one after the class had wrapped up their work; reviewers also used the same metric to evaluate articles that students created from scratch. A total of 124 articles formed the sample. Altogether (counting both new and pre-existing articles), articles improved on average 6.5 points, from 7.4 to 13.98 points on the 26-point scale. The graph below shows the quality distribution of articles before students worked on them (in blue), and the quality distributions of articles after students worked on them (in red).

Article quality improvement of sample of Wikipedia articles edited by students participating in the Spring 2012 Wikipedia Education Program in the United States and Canada. This graph shows overall improvement (both existing and new articles).

The 124-article sample included 82 existing articles and 42 new articles created by students. Existing articles improved 2.94 points on average, from 11.26 to 14.2, with the most improved article improving by 10.25 points. An example of such an article that a student improved is the article on vocabulary development. You can see the versions prior to students’ first edits and the status it was after the class finished. The graph below shows the distribution of pre-existing articles before (blue) and after (red) student work.

Article quality improvement of sample of Wikipedia articles edited by students participating in the Spring 2012 Wikipedia Education Program in the United States and Canada. This graph shows improvement of existing articles only.

New articles had an average score of 13.55. You can see a sample of what a student contributed to a new article by reading Temptation, a Václav Havel play. The graph below shows the distribution of quality of new articles students created through the Wikipedia Education Program.

The Spring 2012 numbers show improvement over the 2010–11 quality of students contributions from the Public Policy Initiative pilot of the U.S. program, where articles improved an average of 5.8 points. We’re encouraged to see improvement in Wikipedia’s article quality through the Wikipedia Education Program.

LiAnna Davis, Wikipedia Education Program Communications Manager

Predicting admin elections by editor status and similarity; flagged revision debates in multiple languages; Wikipedia literature reviewed

With contributions by: Tbayer, DarTar, Jodi.a.schneider, Njullien and Piotrus

How editors evaluate each other: effects of status and similarity

A team of social computing researchers based at Stanford and Cornell University studied how users evaluate each other in social media.^[1] Their paper, presented at the 5th ACM Web Search and Data Mining Conference (WSDM ’12), focuses on three main case studies: Wikipedia, StackOverflow and Epinions. User-to-user evaluations, the authors note, are jointly influenced by the properties of the evaluator and the target; as a result, differences in properties between the target and the evaluator should be expected to affect the evaluation. The study looks specifically at how differences in topic expertise and status affect peer evaluations. The Wikipedia case focuses on requests for adminship (RfAs), the most prominent example of peer evaluation in Wikipedia and a topic that has attracted considerable attention in the literature (Signpost research coverage: September 2011, October 2011, January 2012). Similarity is measured based on article co-authorship, and status as a function of an editor’s number of contributions. Previous research by the same authors showed that the probability an evaluator will evaluate a target user positively drops dramatically when the status of the two users is very similar, and there is general evidence that homophily and similarity in editing activity have a strong influence on peer evaluation in RfAs. The study identifies two effects that jointly account for this singular finding:

• “Elite” or high-status users are more likely to participate in evaluations about other users who are active in their areas of interest or expertise.
• Low-status users tend to be judged differently than those with moderate or high status

In a direct application of these results, dubbed ballot-blind prediction, the authors show how the outcome of an RfA can be accurately predicted by a model that simply considers the first few participants in a discussion and their attributes, without looking at their actual evaluations of the target.

Sociological analysis of debates about flagged revisions in the English, German and French Wikipedias

At the center of debates on “Coercion or empowerment”: Icons signifying accepted (left) and not yet accepted (right) revisions under a flagged revisions scheme

In an article to appear in Ethics and Information Technology, Paul B. de Laat analysed debates occurring in the English, German and French Wikipedias about the evolution of the rules governing new edits.^[2] As noted by the analysis of the English Wikipedia’s rules, by Butler et al., 2008,^[3] these rules are numerous and have increased in number and complexity; they range from the more formal and explicit (intellectual property rights) to the more informal.

(more…)

Gender gap and conflict aversion; collaboration on breaking news; effects of leadership on participation; legacy of Public Policy Initiative

With contributions by: Tbayer, Piotrus, Jodi.a.schneider, Hfordsa and DarTar

Wikipedia research at CSCW 2012

The annual 15th ACM conference on computer-supported cooperative work (CSCW 2012) featured two sessions about Wikipedia Studies. The first one was titled “Scaling our Everest” (in amusing contrast to an earlier metaphor for the role of Wikipedia in that field of research: “the fruit fly of social software”), and covered four papers. A second session likewise comprised four papers and notes. Below are some of the highlights from these two sessions.

Gender gap connected to conflict aversion and lower confidence among women

The Gender Gap hub on Meta.

Since January 2011, Wikipedia’s “Gender gap” has received much attention from Wikimedians, researchers and the media – triggered by a New York Times article that cited the estimate that only 12.64% of Wikipedia contributors are female. That figure came from the 2010 UNU-MERIT study, which was based on the first global, general survey of Wikipedia users, conducted in 2008 with 176,192 respondents using a methodology that had raised some questions (e.g. about sample bias and selection bias), but other studies found similarly low ratios. A new paper titled “Conflict, Confidence, or Criticism: An Empirical Examination of the Gender Gap in Wikipedia”^[1] has now delved further into the data of the UNU-MERIT study, examining the responses to questions such as “Why don’t you contribute to Wikipedia?” and “Why did you stop contributing to Wikipedia?”, finding strong support for the following three hypotheses:

(more…)

The mobile web is growing at a phenomenal pace. According to research, it will outpace the desktop internet web in 2014, when approximately 1.7 billion users will access the net on their mobile phone, many of them from the Global South, compared to 1.65 billion desktop web users. As part of our mission to provide free knowledge to everyone, we are committed to enhancing our mobile platform, and have made several improvements to the reading user experience. But most importantly, we recently launched a partnership with Orange to provide Wikipedia at no data cost to mobile readers in Africa and the Middle East.

To understand our current Wikipedia mobile users across different geographies and prioritize product features, we conducted a survey of Wikipedia mobile readers. You can read more about its methodology on Meta wiki.

Looking at the data from the survey, there is a strong case to be made for making Wikipedia accessible without data charges on mobile devices.  Over half of Wikipedia mobile readers (52 percent) said that having Wikipedia free for their mobile data plans would increase their Wikipedia usage. Moreover, 28 percent indicated that it would increase their likelihood to buy from that mobile provider.  Another 16 percent said that they would be willing to switch their mobile providers to have free Wikipedia access.

Q. If certain mobile phone service providers provided Wikipedia for free on their data plans, how might that affect your actions? Base: 6700 (Those currently pay for a data plan)

Looking globally, we found that Wikipedia readers in the Global South, specifically in Brazil, Latin America and MENA, indicated that they would use Wikipedia more often if no data costs were accrued, and even suggested this as a key motivating factor for switching to or considering alternative service providers.

Q: If certain mobile phone service providers provided Wikipedia for free on their data plans, how might that affect your actions? Base: 6700 (Those currently pay for a data plan)

We found high interest in Wikipedia access without data charges despite a majority of readers (54 percent) stating that their mobile data plan is not a significant monthly expense for their household.  But it should be noted that the data is based on current mobile readers, and doesn’t survey those who don’t have current mobile Wikipedia access, some of whom might not have access to the mobile web due to high cost.  Only 14 percent of respondents stated that their data plan was either a significant expense with their household actively managing usage, or too expensive leading to issues of affordability. In addition, about 32 percent stated that it was a significant expense, but that they were not concerned about it.

Q: Which of the following statements best describes how expensive your data plan is relative to other expenses that you have? Base: 6700 (Those currently pay for a data plan)

If you are interested in more data from the mobile survey, please check out the toplines or read our summary report or read key findings.

Mani Pande, Head of Global Development Research

Ayush Khanna, Data Analyst, Global Development

Language analyses examine power structure and political slant; Wikipedia compared to commercial databases

With contributions by: Tbayer and Piotrus

Admins influence the language of non-admins

An Arxiv preprint titled “Echoes of power: Language effects and power differences in social interaction”^[1] looks at the language used by Wikipedia editors. The authors look at how conversational language can be used to understand power relationships. The research analyzes how much one adapts their language to the language of others involved in a discussion (the process of language coordination). The findings indicate that the more such adoption occurs, the more deferential one is. The authors find that editors on Wikipedia tend to coordinate (language-wise) more with the administrators than with non-administrators. Furthermore, the study suggests that one’s ability to coordinate language has an impact on one’s chances to become an administrator: the admin-candidates who do more language coordination have a higher chance of becoming an administrator than those who don’t change their language. Once a person is elected an administrator, they tend to coordinate less.

A blog post on the website of Technology Review summarized the results using the headline “Algorithm Measures Human Pecking Order” and highlighted the fact that one of the authors is Jon Kleinberg, known as inventor of the HITS algorithm (also known as “hubs and authorities”).

Can Wikipedia replace commercial biography databases?

California State University, East Bay: Could it rely on biographical information from Wikipedia and the web alone?

An article^[2] by a librarian and professor at California State University offers a comparison of “biographical content for literary authors writing in English” between Wikipedia, “the web” (i.e. top Google search results) and two commercial databases: the Biography Reference Bank (BRB, now part of EBSCO Industries) and Contemporary Authors Online, motivated by the decision of the author’s institution to cancel its subscription to the latter database (CAO) during a budget crisis in 2008-2009, which among other reasons had been accompanied by “a comment that this information is ‘on the web’”.

The paper starts out with a literature review on the reliability of Wikipedia and then describes how the author compiled a list of 500 authors (mostly from the US and UK) by “examining curricula and textbooks from English literature courses across the USA” and soliciting additional suggestions from peers. These names were then searched on BRB, CAO (as part of the Literature Resource Center), Wikipedia and Google.

(more…)

In a previous blog post, we discussed our readers’ perception of article quality. In addition, we asked our readers to compare Wikipedia as a whole to other prominent websites – Facebook, Twitter, New York Times, Google, YouTube, Yahoo and CNN. Of course, there are several key differences between them, but we wanted to understand how Wikipedia stacks up against other high-traffic websites.

Readers from all 16 countries in our sample compared Wikipedia’s interface and ease of navigation to other Internet properties. If we look at the sample as whole, Wikipedia (8.09 on 10) was rated a close second to Google (8.44) on these measures. What makes this even more interesting is Wikipedia’s relationship with the search engine, which we mentioned in an earlier blog post. Although ratings varied across countries quite significantly, in most cases there was little deviation in ratings relative to other websites, with some exceptions.

Interface/look and feel

When asked about the Wikipedia interface, readers scored Wikipedia 7.92 out of 10 on average, just behind Google (8.3). About 46 percent of our readers scored the interface 9+ out of 10, compared to 54 percent for Google. We did not find significant deviations across countries or languages, with one exception: Readers in Egypt (and by extension, Arabic speakers) rated Wikipedia lower than YouTube, Facebook and Yahoo. A desire for better right-to-left support is one plausible explanation for the result.

D8a. How appealing do you find the interface or look of the following sites?

Ease of Navigation

Readers scored Wikipedia 8.27 on this metric, slightly lower than Google (8.59). 53 percent of our readers rated the ease of navigation 9+ out of 10, compared to 63 percent for Google. As above, Arabic/Egyptian readers rated Wikipedia below YouTube, Facebook, and Yahoo.

D8b. How easy do you find it to navigate the following sites?

Mani Pande, Head of Global Development Research

Ayush Khanna, Data Analyst, Global Development

We recently conducted an online survey of Wikipedia readers, limited to 250 participants each in 16 countries. This is the seventh in a series of blog posts summarizing our findings. If you are interested, you can find out more about the methodology of the survey here.

With contributions by: Tbayer, DarTar and Jodi.a.schneider

Mental health information on Wikipedia more accurate than Britannica and Kaplan & Sadock psychiatry textbook

Wikipedia articles on schizophrenia and other mental health topics were assessed for accuracy, richness of references and readability.

In an article for Psychological Medicine,^[1] ten researchers from the University of Melbourne conclude that “the quality of information on depression and schizophrenia on Wikipedia is generally as good as, or better than, that provided by centrally controlled websites, Encyclopaedia Britannica and a psychiatry textbook.”

The study focused on ten mental health topics (e.g. “antidepressants and suicide in young people” or “side-effects of antipsychotics”), five each in the areas of depression and schizophrenia. “Using the topic terms (or synonyms) as key words for the searches or through manual browsing, content relating to these topics was extracted from [Wikipedia and 13 other websites selected for prominent Google results for depression and schizophrenia] and from the most recent edition of Kaplan & Sadock’s Comprehensive Textbook of Psychiatry … and the online version of Encyclopaedia Britannica” by two reviewers. For both depression and schizophrenia, three psychologists with clinical and research expertise in that area evaluated these extracts on accuracy, up-to-dateness, breadth of coverage, referencing and readability, on a scale from 1 to 5 (“e.g. Accuracy: 1 = many errors of fact or unsubstantiated opinions, 3=some errors of fact or unsubstantiated opinions, 5 = all information factually accurate”). As in an earlier study of the quality of health information on Wikipedia (Signpost coverage: “Wikipedia’s cancer coverage is reliable and thorough, but not very readable“), readability was also measured using a Flesch–Kincaid readability test, which is calculated from word and sentence lengths.

For both depression and schizophrenia, Wikipedia scored highest in the accuracy, up-to-dateness, and references categories – surpassing all other resources, including WebMD, NIMH, the Mayo Clinic and Britannica online. In breadth of coverage, it was behind Kaplan & Saddock and others for both areas. And “of the online resources, Wikipedia was rated the least readable [by the human reviewers], although some of its topics received an average rating.” Likewise, the Wikipedia content had relatively high Flesch–Kincaid Grade Level indices (around 16 for schizophrenia and 15 for depression – indicating that a tertiary level of education is necessary to understand the content), similar to that of Britannica but higher than most other resources examined.

The authors note that their “findings largely parallel those of other recent studies of the quality of health information on Wikipedia” (citing eight such studies published between 2007 and 2010):

“Despite variability in the methodologies and conclusions of these studies, the overall implication is that Wikipedia articles on health topics typically contain relatively few factual errors, although they may lack breadth of coverage. … Given the number of patients, would-be patients and concerned others using the internet to search for information on health issues, it seems that Wikipedia is an appropriate recommendation as an information source.“

Psychologists gauge impact of Wikipedia’s Rorschach test coverage

(more…)

On Thursday, December 8th, the Wikimedia Foundation will launch its second semi-annual survey (2011) of Wikipedia editors.  In order to capture editor trends, we are using the same methodology as the April 2011 Editor Survey – editors logged in to Wikipedia will receive a notification, as every editor is eligible to participate. To ensure that all editors have an equal probability of participating in the survey, all logged-in users will see the invitation only once. We’ll do a soft launch on Thursday (all Wikipedias, excluding English) and switch it on for the English Wikipedia next week, to accommodate the Harvard/Sciences Po survey that is launching soon on the English Wikipedia. We urge all Wikipedia editors to give us their feedback and participate in the survey. For more information, you can read the FAQ we’ve posted detailing the survey.

The survey is currently available in various languages in addition to English, including: Chinese (traditional, Hong Kong), Chinese (simplified), Arabic, Catalan, German, Spanish, Japanese, Portuguese, Polish, French, Hebrew, Hungarian, Italian, Russian and Serbo-Croatian. The Foundation will conduct the survey in languages for which translations are available, and for the remainder of Wikipedia language projects the survey will be available in English.  The survey will take about 15 minutes to complete.  Since we are interested in trending the data, about 90% of the questions are the same as in the April 2011 survey. We have added a few new questions based on findings from Wikipedia Summer of Research project and other research work that has been conducted at the Foundation.

The current survey covers the following topics:

• Demographics
• Brief section on editors’ technology usage
• Editing activities and contributions
• Editor interactions
• Opinions of editors about chapters, the Foundation and participation in board elections.

We’re looking forward to participation from editors all around the world while the survey is active. Please spread the word, and we would like to thank you in advance for taking the time to contribute your views!

Mani Pande, Head of Global Development Research

In our recently concluded Annual Plan, we identified increasing the number of active contributors as one of our strategic priorities. As of September 2011, there are 79,890 active Wikipedia contributors (active is defined as those making five or more edits in a month), while we want to increase active editors to approximately 95,000 on all Wikimedia projects in June 2012.

a. Only 6% of our readers have ever made an edit to Wikipedia

b. Most readers are happy to just read, many cite lack of expertise

c. Avid Wikipedia readers, readers with heavy online activity, Twitter users, men, younger readers and online contributors are strong candidates for editors (more…)

Vol: 1 • Issue: 5 • November 2011 [archives]

Quantifying quality collaboration patterns, systemic bias, POV pushing, the impact of news events, and editors’ reputation

With contributions by: Tbayer, Hfordsa, DarTar and Romanesco

Collaboration pattern analysis: Editor experience more important than “many eyes”

One of the motifs indicating article quality: One editor (top) having worked on several related articles (bottom)

A paper titled “Characterizing Wikipedia Pages Using Edit Network Motif Profiles”^[1] by three researchers from University College Dublin indicates that the quality of a Wikipedia article can be predicted from characteristics of its “edit network” – a graph derived from the collaboration of Wikipedians in that area. Network motifs are small graphs which occur particularly frequently as sub-graphs of networks of a certain kind, and can be regarded as its building blocks in some sense. (The concept is popular in bioinformatics, where it is applied to gene regulatory networks.) In this paper, the authors use graphs with at most five nodes consisting of users and articles, which are connected by an edge if the user has edited the article – giving 17 possible “Wikipedia network motifs”. (Anonymous users are disregarded.) For a Wikipedia article, the researchers form an “ego network” consisting of that article, articles which link to it (and have been edited by at least one of the users who edited the core article), and the users who edited them. For a sample of around 2000 articles from the History and United States categories, the frequencies of the 17 “Wikipedia network motifs” in those article’s “ego networks” were calculated.

Using machine learning techniques, the researchers are able to discern with some certainty articles of basic quality (defined as having been assessed as Start class by Wikipedians) from those of good quality (defined as Featured or B class), solely based on this set of motif frequencies in the article’s edit network. Looking at the impact of each of the 17 types separately, they found that “all network motifs have some potential to discriminate between good and basic Wikipedia articles” in the sample, but that among the four best predicting motifs, three are “stars with editors at their centre”:

“This is interesting because it shows that many eyes is not really the defining characteristic of quality; instead experience is important – the editors should have worked on many other articles.”

(more…)