Download the complete Volume 1 (PDF)

The success of Wikipedia continues to attract an enormous amount of attention from researchers who are trying to understand what made this one of the most remarkable collaboration projects in history, and unearth valuable insights that may help to improve it. The monthly Wikimedia Research Newsletter launched in mid-2011 – shortly after the announcement of the Wikimedia Research Index – with the aim of covering recent academic research about Wikipedia and other Wikimedia projects. Published jointly by the Wikimedia Research Committee and the Signpost (the English Wikipedia’s community-edited newspaper), it has established itself as a comprehensive outlet enabling both researchers and Wikipedians to stay on top of current research, aiming to facilitate exchange between these two communities.

The inaugural Signpost edition

The six issues published in the first volume (July-December 2011) featured 87 unique references (93 citations) and attracted altogether more than 17,000 pageviews in 2011, not counting the WMF blog edition. The complete Volume 1 is now available as a downloadable 45-page PDF, and a print version can be ordered from Pediapress. The full list of publications reviewed or covered in the Newsletter in 2011 can be browsed online or downloaded (as a BibTeX, RIS, PDF file or in other formats), ready to be imported into reference managers or other bodies of wiki research literature. Open access papers in this collection have been marked with a special open_access tag in the reference list and with an OA icon in the body of each issue.

We are also happy to announce the launch of @WikiResearch: a news feed on Twitter and Identi.ca, covering new preprints, papers or research-related blog posts, before they are reviewed more fully in the Newsletter.

Follow @WikiResearch for fresh Wikimedia research news

What’s more, the Newsletter is now also available in the form of an HTML email newsletter (in addition to the announcements of each new issue on the Wikiresearch-l mailing list, which only contain the table of contents). Sign up here to receive a copy of each new issue in your inbox as soon as it comes out.

The Newsletter is a collaborative effort and would not exist without those who have contributed reviews and summaries so far: Boghog, DarTar, Drdee, Hfordsa, Jodi.a.schneider, Junkie.dolphin, Lilaroja, Mietchen, Phoebe, Piotrus, Romanesco, Steven Walling, Tbayer. We are also grateful for the help of several Signpost collaborators in copyediting and preparing the final publication every month.

Finally, thanks to everyone for reading the Wikimedia Research Newsletter, and please
consider contributing by pointing us to new research we should cover, or by volunteering to review new publications.

The editors of the Wikimedia Research Newsletter:

Dario Taraborelli, Senior Research Analyst, Strategy
Tilman Bayer, Movement Communications

Gender gap and conflict aversion; collaboration on breaking news; effects of leadership on participation; legacy of Public Policy Initiative

With contributions by: Tbayer, Piotrus, Jodi.a.schneider, Hfordsa and DarTar

Wikipedia research at CSCW 2012

The annual 15th ACM conference on computer-supported cooperative work (CSCW 2012) featured two sessions about Wikipedia Studies. The first one was titled “Scaling our Everest” (in amusing contrast to an earlier metaphor for the role of Wikipedia in that field of research: “the fruit fly of social software”), and covered four papers. A second session likewise comprised four papers and notes. Below are some of the highlights from these two sessions.

Gender gap connected to conflict aversion and lower confidence among women

The Gender Gap hub on Meta.

Since January 2011, Wikipedia’s “Gender gap” has received much attention from Wikimedians, researchers and the media – triggered by a New York Times article that cited the estimate that only 12.64% of Wikipedia contributors are female. That figure came from the 2010 UNU-MERIT study, which was based on the first global, general survey of Wikipedia users, conducted in 2008 with 176,192 respondents using a methodology that had raised some questions (e.g. about sample bias and selection bias), but other studies found similarly low ratios. A new paper titled “Conflict, Confidence, or Criticism: An Empirical Examination of the Gender Gap in Wikipedia”^[1] has now delved further into the data of the UNU-MERIT study, examining the responses to questions such as “Why don’t you contribute to Wikipedia?” and “Why did you stop contributing to Wikipedia?”, finding strong support for the following three hypotheses:

(more…)

Wikimedia Foundation sites are the fifth most popular web property. So when we conducted our Readers Survey, we expected to hear from our readers that they valued Wikipedia as a reliable source of online information. But we were still pleasantly surprised to see the overwhelming positive feedback from readers mixed with a deep appreciation of Wikipedia and its community for the existence of the free and easily available online encyclopedia that offers a wide breadth of content. As expressed by some of the more than 2000 readers who took the opportunity to leave a free-form comment at the end of the survey: “love the site” “blessing” “thank you for existing and “you can find everything in Wikipedia.” Praise for Wikipedia was a common thread among many readers. “It is great.” “It is a living organism – constantly evolving.”

Word cloud of open ended responses from the Wikipedia Readers Survey

The real credit for building Wikipedia goes to the thousands of editors who volunteer hours every month editing Wikipedia, and as one reader from Egypt put it: “thank you to all who have contributed in this great work.” Readers who had never edited, but were aware that volunteers wrote Wikipedia articles acknowledged that they were embarrassed that they had never edited Wikipedia.

Although readers from all age groups read Wikipedia, with the average reader being 36 years old, Wikipedia remains an important tool for school students. “Great for school work,” said one. Acknowledging the role of Wikipedia as the first stop for information for school work, a reader in India said: “I wish I had Internet and Wikipedia when I was in school in the 70s.  My grades would have sky rocketed.” Parents also acknowledged using Wikipedia for helping children with their school work.

Some readers took the opportunity to point out features that they believe can be improved on Wikipedia, like search, the ability to improve the reading experience through customizable fonts and more audio and visual content to facilitate understanding of complex topics. We heard similar needs from users when we conducted user experience research in India, Brazil and the US. We have improved search functionality on the mobile site, and other features to improve the reading experience are on the product roadmap.

To our surprise (this was the first time participants thanked us for providing them the opportunity to participate in a survey), many readers told us that they loved participating in the study since it helped them expand their knowledge about Wikipedia (e.g. learning about features that they had not been aware of, or about WMF’s non-profit status). “From today, I will certainly be using Wikipedia more, excellent survey, it made me understand things I have been missing,” said a reader from the UK.

 
Mani Pande, Head of Global Development Research

Ayush Khanna, Data Analyst, Global Development

We recently conducted an online survey of Wikipedia readers, limited to 250 participants each in 16 countries. This is the last in a series of blog posts summarizing our findings. If you are interested, you can find out more about the methodology of the survey here.

Mobile Wikipedia readers in Brazil

As part of our commitment to provide free knowledge to everyone, the foundation has been redesigning our mobile platform (m.wikipedia.org and mobile.wikipedia.org) to enhance the reading experience and allow editing.  As a first step towards the redesign of the mobile gateway to better meet the needs of our users in the Global South, we conducted user experience research in India and Brazil among current and future users of Wikipedia mobile last summer.  We also carried out user experience research in the US to have a comparison with a mobile market which is more mature in terms of smartphone and 3G penetration, and has a more widespread adoption of tablets.

Our research in India and Brazil brought forth the following three opportunities with the greatest perceived impact for the mobile platform:

1. Improving our search:  Our research revealed that there was a need to provide search suggestions, autocomplete, autocorrect and other tools that ease typing and search burdens on mobile devices; support search in all language Wikipedias as well as allowing users to chose and switch between languages; incorporate transliteration tools for languages with fonts and characters that have poor mobile support; support and even enhance users’ existing habits to use Google to reach Wikipedia articles; and enable users to search within a Wikipedia page. We are happy to report that drawing from the research our mobile team has already implemented some of these opportunities like full page search, autocomplete  and inter-wiki links into our mobile beta site.
2. Optimizing our reading experience for mobile devices and generalized use.  Through our research, especially in India, we found that we were not redirecting a large breadth of devices in use to our mobile site. The mobile team quickly fixed this issue with the adoption of the open source library tera-WURFL for detecting mobile devices.  After speaking with respondents in India and Brazil, we found that there was a desire among users to modify or set one-time preferences for the display of images, the font size, and any element that affects page loading time and size. Similarly, there is an opportunity for allowing  preferences for language and navigation; the ability to watch or bookmark articles; or save content offline; offer content in more digestible pieces, or with quicker access (i.e. preview or easy access to the first paragraph, or a new “mobile summary”); search offline, i.e., while in transit or without a data plan; and generally follow expectations set by mobile web interactions and standards.  Some of these recommendations have been incorporated into our mobile product strategy.  Through this research we felt it was crucial to offer both an official iOS and Android app (which was officially released in January) that offers at minimum a simple and easy search and reading experience.
3. Using the mobile platform to both increase user engagement and awareness of features on Wikipedia as well as providing new opportunities for participation. The mobile site and potential apps provide many new pathways for both engagement, participation, and contribution.  At present, the mobile site can be used to build awareness around existing features on the site that current users are blind to (i.e. watchlists, accounts, editing, inter-language links, history); to provide features that make opening a Wikipedia account worth having, something that the majority of our participants do not currently see any reason to have; increase visibility of local language Wikipedias, especially in India since many English readers were not aware of the existence of Indic Wikipedias; prompt users to download an official app when possible; and interface with other web content on mobile devices (Google, news, entertainment, and sports content, for example “Wikitap”).  The contributions that showed the highest potential for adoption were adding photographs, “flagging” or “marking” something that needs to be edited, removing or marking vandalism, adding links, adding location or geodata, and potentially making small typing or formatting edits.
4. Mobile Editing. And finally, the mobile site can support the editing practice of existing editors by first offering those features in a mobile friendly format which are currently in high use on the site.  Those with the highest demand and potential are the “recent changes” page, which is consumed like an update feed or email; accessing watch lists; making reverts, especially with respect to vandalism; logging in and accessing account and user pages; and serving discussion pages and article histories.

If you are interested in reading about our research in India and Brazil in detail, we have compiled the insights in a report which is available in PDF and wiki format. You can also watch video highlights from the interviews and check out some photographs from the field work in India and Brazil.

Mani Pande, Head of Global Development Research

We recently concluded our annual fundraiser, and it was a great success. With over one million individuals pitching in from almost every country, our donors are a diverse and interesting group in themselves. While we don’t directly collect demographic or other information from our donors, we used our readers survey as a means of understanding who our donors are, and what motivates them to donate to Wikipedia. Please note that these results are from a sample of our reader population, not drawn from actual donor data.

a. Only about half of our readers realize that Wikipedia is a non-profit, editors much more likely to donate
b. An appeal from Jimmy is a popular reason for donating, but people in different regions are motivated differently
c. About one-fourth of our respondents said they would donate; readers from US, Egypt and India most likely to do so
d. Readers cite affordability as the biggest reason for not donating
(more…)

Language analyses examine power structure and political slant; Wikipedia compared to commercial databases

With contributions by: Tbayer and Piotrus

Admins influence the language of non-admins

An Arxiv preprint titled “Echoes of power: Language effects and power differences in social interaction”^[1] looks at the language used by Wikipedia editors. The authors look at how conversational language can be used to understand power relationships. The research analyzes how much one adapts their language to the language of others involved in a discussion (the process of language coordination). The findings indicate that the more such adoption occurs, the more deferential one is. The authors find that editors on Wikipedia tend to coordinate (language-wise) more with the administrators than with non-administrators. Furthermore, the study suggests that one’s ability to coordinate language has an impact on one’s chances to become an administrator: the admin-candidates who do more language coordination have a higher chance of becoming an administrator than those who don’t change their language. Once a person is elected an administrator, they tend to coordinate less.

A blog post on the website of Technology Review summarized the results using the headline “Algorithm Measures Human Pecking Order” and highlighted the fact that one of the authors is Jon Kleinberg, known as inventor of the HITS algorithm (also known as “hubs and authorities”).

Can Wikipedia replace commercial biography databases?

California State University, East Bay: Could it rely on biographical information from Wikipedia and the web alone?

An article^[2] by a librarian and professor at California State University offers a comparison of “biographical content for literary authors writing in English” between Wikipedia, “the web” (i.e. top Google search results) and two commercial databases: the Biography Reference Bank (BRB, now part of EBSCO Industries) and Contemporary Authors Online, motivated by the decision of the author’s institution to cancel its subscription to the latter database (CAO) during a budget crisis in 2008-2009, which among other reasons had been accompanied by “a comment that this information is ‘on the web’”.

The paper starts out with a literature review on the reliability of Wikipedia and then describes how the author compiled a list of 500 authors (mostly from the US and UK) by “examining curricula and textbooks from English literature courses across the USA” and soliciting additional suggestions from peers. These names were then searched on BRB, CAO (as part of the Literature Resource Center), Wikipedia and Google.

(more…)

In a previous blog post, we discussed our readers’ perception of article quality. In addition, we asked our readers to compare Wikipedia as a whole to other prominent websites – Facebook, Twitter, New York Times, Google, YouTube, Yahoo and CNN. Of course, there are several key differences between them, but we wanted to understand how Wikipedia stacks up against other high-traffic websites.

Readers from all 16 countries in our sample compared Wikipedia’s interface and ease of navigation to other Internet properties. If we look at the sample as whole, Wikipedia (8.09 on 10) was rated a close second to Google (8.44) on these measures. What makes this even more interesting is Wikipedia’s relationship with the search engine, which we mentioned in an earlier blog post. Although ratings varied across countries quite significantly, in most cases there was little deviation in ratings relative to other websites, with some exceptions.

Interface/look and feel

When asked about the Wikipedia interface, readers scored Wikipedia 7.92 out of 10 on average, just behind Google (8.3). About 46 percent of our readers scored the interface 9+ out of 10, compared to 54 percent for Google. We did not find significant deviations across countries or languages, with one exception: Readers in Egypt (and by extension, Arabic speakers) rated Wikipedia lower than YouTube, Facebook and Yahoo. A desire for better right-to-left support is one plausible explanation for the result.

D8a. How appealing do you find the interface or look of the following sites?

Ease of Navigation

Readers scored Wikipedia 8.27 on this metric, slightly lower than Google (8.59). 53 percent of our readers rated the ease of navigation 9+ out of 10, compared to 63 percent for Google. As above, Arabic/Egyptian readers rated Wikipedia below YouTube, Facebook, and Yahoo.

D8b. How easy do you find it to navigate the following sites?

Mani Pande, Head of Global Development Research

Ayush Khanna, Data Analyst, Global Development

We recently conducted an online survey of Wikipedia readers, limited to 250 participants each in 16 countries. This is the seventh in a series of blog posts summarizing our findings. If you are interested, you can find out more about the methodology of the survey here.

With contributions by: Tbayer, DarTar and Jodi.a.schneider

Mental health information on Wikipedia more accurate than Britannica and Kaplan & Sadock psychiatry textbook

Wikipedia articles on schizophrenia and other mental health topics were assessed for accuracy, richness of references and readability.

In an article for Psychological Medicine,^[1] ten researchers from the University of Melbourne conclude that “the quality of information on depression and schizophrenia on Wikipedia is generally as good as, or better than, that provided by centrally controlled websites, Encyclopaedia Britannica and a psychiatry textbook.”

The study focused on ten mental health topics (e.g. “antidepressants and suicide in young people” or “side-effects of antipsychotics”), five each in the areas of depression and schizophrenia. “Using the topic terms (or synonyms) as key words for the searches or through manual browsing, content relating to these topics was extracted from [Wikipedia and 13 other websites selected for prominent Google results for depression and schizophrenia] and from the most recent edition of Kaplan & Sadock’s Comprehensive Textbook of Psychiatry … and the online version of Encyclopaedia Britannica” by two reviewers. For both depression and schizophrenia, three psychologists with clinical and research expertise in that area evaluated these extracts on accuracy, up-to-dateness, breadth of coverage, referencing and readability, on a scale from 1 to 5 (“e.g. Accuracy: 1 = many errors of fact or unsubstantiated opinions, 3=some errors of fact or unsubstantiated opinions, 5 = all information factually accurate”). As in an earlier study of the quality of health information on Wikipedia (Signpost coverage: “Wikipedia’s cancer coverage is reliable and thorough, but not very readable“), readability was also measured using a Flesch–Kincaid readability test, which is calculated from word and sentence lengths.

For both depression and schizophrenia, Wikipedia scored highest in the accuracy, up-to-dateness, and references categories – surpassing all other resources, including WebMD, NIMH, the Mayo Clinic and Britannica online. In breadth of coverage, it was behind Kaplan & Saddock and others for both areas. And “of the online resources, Wikipedia was rated the least readable [by the human reviewers], although some of its topics received an average rating.” Likewise, the Wikipedia content had relatively high Flesch–Kincaid Grade Level indices (around 16 for schizophrenia and 15 for depression – indicating that a tertiary level of education is necessary to understand the content), similar to that of Britannica but higher than most other resources examined.

The authors note that their “findings largely parallel those of other recent studies of the quality of health information on Wikipedia” (citing eight such studies published between 2007 and 2010):

“Despite variability in the methodologies and conclusions of these studies, the overall implication is that Wikipedia articles on health topics typically contain relatively few factual errors, although they may lack breadth of coverage. … Given the number of patients, would-be patients and concerned others using the internet to search for information on health issues, it seems that Wikipedia is an appropriate recommendation as an information source.“

Psychologists gauge impact of Wikipedia’s Rorschach test coverage

(more…)

A team of researchers at the Berkman Center for Internet & Society at Harvard University and Sciences Po Paris, led by Professor Yann Algan and Wikimania 2011 keynote speaker Professor Yochai Benkler, invites English Wikipedia contributors to participate in an interactive online experiment on decision making. The goal of this study is to better understand the dynamics of interactions and behavior in online social spaces. The project has been reviewed by the Wikimedia Research Committee and extensively discussed on the English Wikipedia Administrators’ noticeboard. The Wikimedia Foundation is happy to support this project which we believe will help advance research on our community.

Starting today, logged-in eligible editors will receive an invitation to participate in this study via a CentralNotice banner. To reduce banner overload, the invitation banner has been coded so as to be displayed only to a sample of English Wikipedia contributors meeting eligibility conditions for this study. If you are among the editors selected for this study, please consider participating. If you disable the banner, it will not be displayed to you anymore.

The survey takes about 25 minutes to complete. It combines a series of interactive experiments on decision making with questions about attitudes and practices. Based on their decisions and those of other participants in the study, participants will earn money that they can then choose to donate to the International Committee of the Red Cross or the Wikimedia Foundation if they so wish.

The data collected by this study is subject to European privacy protection protocols and will be used for research purposes only. All research outputs and data, while preserving full anonymity of participants, will be made available under an open access license. The research team will present its findings at a Wikimania conference.

Dario Taraborelli, Senior Research Analyst, Strategy

On Thursday, December 8th, the Wikimedia Foundation will launch its second semi-annual survey (2011) of Wikipedia editors.  In order to capture editor trends, we are using the same methodology as the April 2011 Editor Survey – editors logged in to Wikipedia will receive a notification, as every editor is eligible to participate. To ensure that all editors have an equal probability of participating in the survey, all logged-in users will see the invitation only once. We’ll do a soft launch on Thursday (all Wikipedias, excluding English) and switch it on for the English Wikipedia next week, to accommodate the Harvard/Sciences Po survey that is launching soon on the English Wikipedia. We urge all Wikipedia editors to give us their feedback and participate in the survey. For more information, you can read the FAQ we’ve posted detailing the survey.

The survey is currently available in various languages in addition to English, including: Chinese (traditional, Hong Kong), Chinese (simplified), Arabic, Catalan, German, Spanish, Japanese, Portuguese, Polish, French, Hebrew, Hungarian, Italian, Russian and Serbo-Croatian. The Foundation will conduct the survey in languages for which translations are available, and for the remainder of Wikipedia language projects the survey will be available in English.  The survey will take about 15 minutes to complete.  Since we are interested in trending the data, about 90% of the questions are the same as in the April 2011 survey. We have added a few new questions based on findings from Wikipedia Summer of Research project and other research work that has been conducted at the Foundation.

The current survey covers the following topics:

• Demographics
• Brief section on editors’ technology usage
• Editing activities and contributions
• Editor interactions
• Opinions of editors about chapters, the Foundation and participation in board elections.

We’re looking forward to participation from editors all around the world while the survey is active. Please spread the word, and we would like to thank you in advance for taking the time to contribute your views!

Mani Pande, Head of Global Development Research