The proportion of quality newcomers over time. (2006-2011)

As part of the 2011 Wikimedia Summer of Research, we uncovered a possible correlation between the decline in new active editors that began in 2007 and the rise of warnings issued to new users by bots and automated tools, which started in 2006.

For those of us studying editor trends, the following question has continued to puzzle us: did the change in communications to new users lead to the decline, or can the rise in warnings be explained by a decrease in quality contributions from new users? Perhaps, as some Wikipedians have argued, the new users of today are being reverted and warned more aggressively than those who entered the project in 2001-2006 because their edits are qualitatively worse (e.g., more self-promotional or spammy, less serious and encyclopedic) than those of previous generations of editors.

While the complexity involved in determining what constitutes a “good” contributor to Wikipedia may never allow us to definitively answer this question, our research argues against the theory that today’s newbies just plain suck.

The proportion of rejection for quality newcomers over time.

To test the hypothesis that new contributors who entered the project in recent years have been more harmful and less interested in positively contributing to the encyclopedia, we randomly sampled the first edits of newcomers to the English Wikipedia from the earliest days of the project to the present. With the help of some experienced Wikipedians, we hand-categorized the edits of 2,100 new users according to a four point quality scale – blatant vandal (obscene language, obvious vandalism), bad faith (jokes and nonsense), good faith poor-quality edit (bad formatting, unreferenced, but trying to add value), and golden (good faith good edits that should not be reverted).

What we found was encouraging: the quality of new editors has not substantially changed since 2006. Moreover, both in the early days of Wikipedia and now, the majority of new editors are not out to obviously harm the encyclopedia (~80 percent), and many of them are leaving valuable contributions to the project in their first editing session (~40 percent). However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen. What this means is that while just as many productive contributors enter the project today as in 2006, they are entering an environment that is increasingly challenging, critical, and/or hostile to their work. These latter findings have also been confirmed through previous research.

Survival rate of newcomers over time.

This study has many important implications for community and Wikimedia Foundation efforts to engage and retain new editors. To begin, it reasserts the centrality of one fundamental policy on the project, “Assume good faith.” This research strongly supports efforts in the community and at the Foundation to do a better job of integrating new editors into Wikipedia and its sister projects, not simply for the sake of gaining new editors, but for the quality of these new editors’ contributions overall.

At the Foundation level, this includes major software changes like the creation of a visual editor to lower the technical barrier to entry, as well as more experimental pilot projects like template A/B testing, an attempt to make the template messages received by new users more personalized and clear, and the Teahouse, which gives new users a friendly, low-pressure space to seek help from experienced Wikipedians. With better software and an inviting and supportive atmosphere, the encyclopedia can continue to grow both in quality of material and quantity of dedicated contributors.

• Find out more about this study at Research:Newcomer quality
• This work is part of a journal article in submission to a special issue of American Behavioral Scientist on Wiki Research
• A special thanks to R. Stuart Geiger from UC Berkeley, as well as Maryana Pinchuk, Steven Walling, and Oliver Keyes from the Wikimedia Foundation, for their assistance with this study.

Aaron Halfaker,
Wikimedia Foundation Research Analyst and University of Minnesota PhD candidate

Does a constant stream of new editors really make Wikipedia better? Increasing participation is one of the top five priorities in our strategic plan. But when we talk about retention of newly registered editors, some readers and experienced editors rightfully wonder exactly how many edits by newbies actually improve the free encyclopedia.

In the Community Department, we’re facilitating the WikiGuides pilot program on the English Wikipedia to reach out to new contributors and mentor them. To do that successfully, we must quickly identify which new editors are actually doing good work.

So one of our working questions is: How many contributions by new editors are made in good faith and are worth retaining or improving?

We took a randomly selected batch of 155 new registered users on the English Wikipedia who made at least one edit in mid-April of this year. We looked at their first edit and ranked it on a 1-5 scale, with 1 being pure vandalism and 5 being an edit that is excellent, meaning it adds a significant chunk of verified, encyclopedic content and would be indistinguishable from a very experienced editor. Here’s what that composition looks like:

So you can see that even with a very high standard for quality — we only handed out a single “5” edit — most new editors made contributions worth retaining in some way, even if they weren’t perfect. More than half of these first edits needed no reworking to be acceptable based on current Wikipedia policy. Another 19% made good faith edits but needed additional help to meet standards defined in policy or guideline.

In order to investigate whether this has changed over time, we took a similar cohort from the same period in April 2004 and made the same qualitative assessment.

The key thing to note in comparing the two samples is that the percent of acceptable edits made by newbies did not dramatically decrease from 2004 to 2011. That’s despite the fact that the bar for quality has been raised over time, and that there are arguably fewer obvious contributions to make now that Wikipedia has grown by millions of articles.

Another relevant fact to consider is that while both cohorts are of 155 new editors, it took several days for that many new editors to join Wikipedia in 2004. In 2011, our sample is a tiny slice of the new editors arriving every month. For example: on Monday of this week more than 1,800 editors joined English Wikipedia and made at least one edit. On the equivalent day in 2004 there were only about 60.

Our sample strongly suggests that thousands of new editors still join Wikipedia every month with valuable contributions to make. Ensuring that we welcome these newcomers and show them the ropes is a top priority for ensuring Wikipedia’s continued success in our second decade.

(This is the first in what will be a new series of blog posts coming out of the Community Department at the Wikimedia Foundation. Starting now and continuing through the summer, we will be sharing the questions, experiments, and fresh data that currently drive our work. While you’ll get an inside look at what we’re doing, our numbers and analysis are still evolving and should be taken with a grain of salt.)

Steven Walling
Wikimedia Foundation Fellow, on behalf of the Community Dept. – especially Philippe Beaudette, James Alexander, and Maryana Pinchuk.