Kids these days: the quality of new Wikipedia editors over time

The proportion of quality newcomers over time. (2006-2011)

As part of the 2011 Wikimedia Summer of Research, we uncovered a possible correlation between the decline in new active editors that began in 2007 and the rise of warnings issued to new users by bots and automated tools, which started in 2006.

For those of us studying editor trends, the following question has continued to puzzle us: did the change in communications to new users lead to the decline, or can the rise in warnings be explained by a decrease in quality contributions from new users? Perhaps, as some Wikipedians have argued, the new users of today are being reverted and warned more aggressively than those who entered the project in 2001-2006 because their edits are qualitatively worse (e.g., more self-promotional or spammy, less serious and encyclopedic) than those of previous generations of editors.

While the complexity involved in determining what constitutes a “good” contributor to Wikipedia may never allow us to definitively answer this question, our research argues against the theory that today’s newbies just plain suck.

The proportion of rejection for quality newcomers over time.

To test the hypothesis that new contributors who entered the project in recent years have been more harmful and less interested in positively contributing to the encyclopedia, we randomly sampled the first edits of newcomers to the English Wikipedia from the earliest days of the project to the present. With the help of some experienced Wikipedians, we hand-categorized the edits of 2,100 new users according to a four point quality scale – blatant vandal (obscene language, obvious vandalism), bad faith (jokes and nonsense), good faith poor-quality edit (bad formatting, unreferenced, but trying to add value), and golden (good faith good edits that should not be reverted).

What we found was encouraging: the quality of new editors has not substantially changed since 2006. Moreover, both in the early days of Wikipedia and now, the majority of new editors are not out to obviously harm the encyclopedia (~80 percent), and many of them are leaving valuable contributions to the project in their first editing session (~40 percent). However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen. What this means is that while just as many productive contributors enter the project today as in 2006, they are entering an environment that is increasingly challenging, critical, and/or hostile to their work. These latter findings have also been confirmed through previous research.

Survival rate of newcomers over time.

This study has many important implications for community and Wikimedia Foundation efforts to engage and retain new editors. To begin, it reasserts the centrality of one fundamental policy on the project, “Assume good faith.” This research strongly supports efforts in the community and at the Foundation to do a better job of integrating new editors into Wikipedia and its sister projects, not simply for the sake of gaining new editors, but for the quality of these new editors’ contributions overall.

At the Foundation level, this includes major software changes like the creation of a visual editor to lower the technical barrier to entry, as well as more experimental pilot projects like template A/B testing, an attempt to make the template messages received by new users more personalized and clear, and the Teahouse, which gives new users a friendly, low-pressure space to seek help from experienced Wikipedians. With better software and an inviting and supportive atmosphere, the encyclopedia can continue to grow both in quality of material and quantity of dedicated contributors.

  • Find out more about this study at Research:Newcomer quality
  • This work is part of a journal article in submission to a special issue of American Behavioral Scientist on Wiki Research
  • A special thanks to R. Stuart Geiger from UC Berkeley, as well as Maryana Pinchuk, Steven Walling, and Oliver Keyes from the Wikimedia Foundation, for their assistance with this study.

Aaron Halfaker,
Wikimedia Foundation Research Analyst and University of Minnesota PhD candidate

15 Show

15 Comments on Kids these days: the quality of new Wikipedia editors over time

Maryana Pinchuk 2 years

Dave, thanks for your comment. For an informative glimpse into why we still need more contributors, I urge you to type “Wikipedia:Backlog” into your Wikipedia search bar sometime. You’ll find a sobering view of our supposedly “complete” English Wikipedia. And on the rest of the 280+ projects, the cleanup categories are even more dramatic. Bottom line, there may be a bit of truth to the “low-hanging fruit” theory, but it’s by no means the whole story.

Dave 2 years

This entire post assumes there’s a problem with Wikipedia. There isn’t. If anything, there’s a problem with the public perception of Wikipedia (or it’s perceived “brand promise”, in marketing speak).

Wikipedia is commonly understood to be the website that anyone can edit. There are two big reasons that this is not true anymore:
– The vast majority of the low-hanging fruit, in terms of simple edits that beginners can do, has been picked (as commenter Danard said “The common man has no more information to add to Wikipedia…”)
– The scale of Wikimedia has necessitated a lot of rules and guidelines to keep things organized

So there’s the clash. New editors think they can walk up to any article and “fix” something, but the reality is that they’re more likely to be breaking both the rules and the content.

Again, there isn’t a problem with Wikipedia. It’s hands-down the best resource in its category that the world has ever seen, and it’s improving and growing every day. The “problem” is that the unavoidable fallout from a high-quality, mature system is that it takes more work to get started.

The quality of Wikipedia has outpaced the abilities of the average person. That means you need a change in branding strategy, not a change to the very successful system of editors and editing.

John Aspinall 2 years

I was active in editing for about a year, I’d say. By the end of the year, I’d had enough of the wiki-lawyering, the triumph of process over content, and the death by a thousand reverts to call it quits. Mark Atwood, above, has it exactly right. I see an inward-focused culture, more concerned with their own selves than with their (putative and advertised) goals.

Michael Restivo 2 years

This is very interesting. I hope more research is conducted that looks into new editor retention. Looking forward to reading the ABS article once published; should be a great issue overall.

Mark Atwood 2 years

Established editors who take a proprietarian approach to “their” articles in the face of edits by real world domain experts, editors who speak wiki jargon, the very concept of “deletionism”, these are what is strangling the life out of Wikipedia. Reading the comments above where established editors with stung egos is illustrative.

David L 2 years

Last edit I made (adding a musician to a name’s disambiguation list) was knocked back by a bot. I made a vague effort to follow the recommendations to have the edit accepted, but in the end I gave up.

WTF666 2 years

x

Wikipedia newbies whine all the time. It usually comes down to their laziness of not wanting to spend the time to add citation references and follow wikipedia rules.

x

naugtur 2 years

Let me translate what I have written in response to changes announcement in an article I contributed: http://pl.wikipedia.org/wiki/Dyskusja_wikipedysty:Naugtur

Here’s why I gave up contributing. Articles I submited were written in cooperation with a Prof. who specializes in computional linguistics and became a prof doing that, but it didn’t stop an experienced wikipedia editor from “fixing” the article. I also remember seeing an article contributed by a 60yr old MIT professor removed, because some twenty-something experienced editor… Because he thought it was a case of self-advertising.

So why would I want to contribute scientific content if anyone can “fix” it to use words he likes if he has over N edits?

FYI, I don’t think that’s the attitude newcomers from academic background should face.
I still love the Wikipedia project, but I’m not willing to fight my way through people who know better.

PS. I probably won’t track the comments here, but I’m open for discussion and I left my address for anyone wanting to reach me for a follow-up of the above thoughts.

Daniel Swiecki 2 years

If I stepped into my university library and six people shouted at me at the top of their lungs that I was not qualified to walk here, that my foot was not in fact attached to my body, and that I should get the hell out, I would never go to the library again.

That also happens to be my summary of my experience submitting a single edit to wikipedia.

Scott 2 years

I remember reading an article somewhere about a notable scientist who had published many papers and won many awards, and was generally regarded as one of the best living scientists in his field not having a wikipedia page. Someone tried to create a wikipedia page about him, and the “notability police” came around and rejected it. It seems wikipedia needs to consider expanding what it considers notable while cracking down on having so many web pages devoted to fictional characters and events. (There are over 9,000 pages containing the phrase “is a fictional character”.)

It’s honestly getting difficult to add new content.

Oliver Keyes 2 years

Totally agreed on all three of those points. But I think it has become fairly clear, as a 4th element, that we have started rejecting “good faith” edits far more over the last few years, even though the rejection of “golden” edits remains the same. That, to me, is the most interesting thing about the study (although obviously I have a COI).

Danard 2 years

The common man has no more information to add to Wikipedia and, fortunately, barriers have been lifted against those that add unreliable material. It’s that simple.

Prasanth 2 years

I am not a wiki editor but am a passionate follower and as I understand these trends I think one of the reason for this downfall is that wiki has become increasingly popular to students and so called non-nerds or non-geeks or non-web proactive guys. Thus so to say even common man is trying to contribute to wiki but then the standards will obviously come down. Also, increasingly, articles on various not-so-famous topics (like those that may not have been covered by web journals or news reports) are being taken up by new bees and its not their mistake that the quality is bad but the inherent lack of references on these new topics may also be one of the reasons for this decline in articles.

John Broughton 2 years

The statement that “However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen” is absolutely untrue. That would have been obvious if someone had actual calculated correlations. For example, the rejection rate for good-faith editors was essentially flat between 2006 and 2010, but the survival rate fell from around .20 to about .05. Moreover, even if there was correlation, that’s not the same as causation.

The Foundation seems absolutely committed to the theory that a more welcoming environment will reverse the steadily decreasing number of new editors and the worsening of editor retention. There is – as far as I know – no hard data to support that theory. For example, what is the survival rate for the thousands of students who first edited Wikipedia in a college class?

Moreover, there are alternative theories to explain editor loss (and no need to pick a single one to explain 100% of the loss). To list just three:

*The editing interface is horrible, compared to newer alternatives (Facebook, Twitter);

*As the more widely read articles get better, on average, there is less incentive for a reader to jump in and become an editor, and more difficulty making the value-added edits that give new editors a sense of accomplishment;

*New editors whose edits are not rejected still come to realize, fairly quickly, that Wikipedia’s rules (not to mention technical arcana) are so extensive that they would take a lifetime to learn.

Joshua Issac 2 years

Maybe there would be better retention rates if guidelines like ‘Do not bite the newcomers’ were better enforced.

Leave a Reply

Your email address will not be published. Required fields are marked *