Newbie_quality.by_semester.rows.good_faith

The proportion of quality newcomers over time. (2006-2011)

As part of the 2011 Wikimedia Summer of Research, we uncovered a possible correlation between the decline in new active editors that began in 2007 and the rise of warnings issued to new users by bots and automated tools, which started in 2006.

For those of us studying editor trends, the following question has continued to puzzle us: did the change in communications to new users lead to the decline, or can the rise in warnings be explained by a decrease in quality contributions from new users? Perhaps, as some Wikipedians have argued, the new users of today are being reverted and warned more aggressively than those who entered the project in 2001-2006 because their edits are qualitatively worse (e.g., more self-promotional or spammy, less serious and encyclopedic) than those of previous generations of editors.

While the complexity involved in determining what constitutes a “good” contributor to Wikipedia may never allow us to definitively answer this question, our research argues against the theory that today’s newbies just plain suck.

Newbie_rejected.by_semester.rows.good-faith

The proportion of rejection for quality newcomers over time.

To test the hypothesis that new contributors who entered the project in recent years have been more harmful and less interested in positively contributing to the encyclopedia, we randomly sampled the first edits of newcomers to the English Wikipedia from the earliest days of the project to the present. With the help of some experienced Wikipedians, we hand-categorized the edits of 2,100 new users according to a four point quality scale – blatant vandal (obscene language, obvious vandalism), bad faith (jokes and nonsense), good faith poor-quality edit (bad formatting, unreferenced, but trying to add value), and golden (good faith good edits that should not be reverted).

What we found was encouraging: the quality of new editors has not substantially changed since 2006. Moreover, both in the early days of Wikipedia and now, the majority of new editors are not out to obviously harm the encyclopedia (~80 percent), and many of them are leaving valuable contributions to the project in their first editing session (~40 percent). However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen. What this means is that while just as many productive contributors enter the project today as in 2006, they are entering an environment that is increasingly challenging, critical, and/or hostile to their work. These latter findings have also been confirmed through previous research.

Newbie_survival.by_semester.rows.good_faith

Survival rate of newcomers over time.

This study has many important implications for community and Wikimedia Foundation efforts to engage and retain new editors. To begin, it reasserts the centrality of one fundamental policy on the project, “Assume good faith.” This research strongly supports efforts in the community and at the Foundation to do a better job of integrating new editors into Wikipedia and its sister projects, not simply for the sake of gaining new editors, but for the quality of these new editors’ contributions overall.

At the Foundation level, this includes major software changes like the creation of a visual editor to lower the technical barrier to entry, as well as more experimental pilot projects like template A/B testing, an attempt to make the template messages received by new users more personalized and clear, and the Teahouse, which gives new users a friendly, low-pressure space to seek help from experienced Wikipedians. With better software and an inviting and supportive atmosphere, the encyclopedia can continue to grow both in quality of material and quantity of dedicated contributors.

  • Find out more about this study at Research:Newcomer quality
  • This work is part of a journal article in submission to a special issue of American Behavioral Scientist on Wiki Research
  • A special thanks to R. Stuart Geiger from UC Berkeley, as well as Maryana Pinchuk, Steven Walling, and Oliver Keyes from the Wikimedia Foundation, for their assistance with this study.

Aaron Halfaker,
Wikimedia Foundation Research Analyst and University of Minnesota PhD candidate