Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

How much do new editors actually improve Wikipedia?

Does a constant stream of new editors really make Wikipedia better? Increasing participation is one of the top five priorities in our strategic plan. But when we talk about retention of newly registered editors, some readers and experienced editors rightfully wonder exactly how many edits by newbies actually improve the free encyclopedia.

In the Community Department, we’re facilitating the WikiGuides pilot program on the English Wikipedia to reach out to new contributors and mentor them. To do that successfully, we must quickly identify which new editors are actually doing good work.

So one of our working questions is: How many contributions by new editors are made in good faith and are worth retaining or improving?

We took a randomly selected batch of 155 new registered users on the English Wikipedia who made at least one edit in mid-April of this year. We looked at their first edit and ranked it on a 1-5 scale, with 1 being pure vandalism and 5 being an edit that is excellent, meaning it adds a significant chunk of verified, encyclopedic content and would be indistinguishable from a very experienced editor. Here’s what that composition looks like:

So you can see that even with a very high standard for quality — we only handed out a single “5” edit — most new editors made contributions worth retaining in some way, even if they weren’t perfect. More than half of these first edits needed no reworking to be acceptable based on current Wikipedia policy. Another 19% made good faith edits but needed additional help to meet standards defined in policy or guideline.

In order to investigate whether this has changed over time, we took a similar cohort from the same period in April 2004 and made the same qualitative assessment.

The key thing to note in comparing the two samples is that the percent of acceptable edits made by newbies did not dramatically decrease from 2004 to 2011. That’s despite the fact that the bar for quality has been raised over time, and that there are arguably fewer obvious contributions to make now that Wikipedia has grown by millions of articles.

Another relevant fact to consider is that while both cohorts are of 155 new editors, it took several days for that many new editors to join Wikipedia in 2004. In 2011, our sample is a tiny slice of the new editors arriving every month. For example: on Monday of this week more than 1,800 editors joined English Wikipedia and made at least one edit. On the equivalent day in 2004 there were only about 60.

Our sample strongly suggests that thousands of new editors still join Wikipedia every month with valuable contributions to make. Ensuring that we welcome these newcomers and show them the ropes is a top priority for ensuring Wikipedia’s continued success in our second decade.

(This is the first in what will be a new series of blog posts coming out of the Community Department at the Wikimedia Foundation. Starting now and continuing through the summer, we will be sharing the questions, experiments, and fresh data that currently drive our work. While you’ll get an inside look at what we’re doing, our numbers and analysis are still evolving and should be taken with a grain of salt.)

Steven Walling
Wikimedia Foundation Fellow, on behalf of the Community Dept. – especially Philippe Beaudette, James Alexander, and Maryana Pinchuk.

20 Responses to “How much do new editors actually improve Wikipedia?”

  1. ecemaml@es.wikipedia says:

    As usual, a piece of news that does not refer to Wikipedia but actually to the English Wikipedia. Would it be possible for the Foundation not to behave as the English Wikipedia chapter?

  2. Jane says:

    Interesting charts! You forgot to mention however, that unregistered users cannot create pages today, but they could create pages back in 2004, so technically your sample should include the new pages from unregistered users in 2004 as well. This also impacts the sample because I think users who are registering are less likely to be vandals. A better sample would be 1000 random edits in-a-row, comparing 2004 to today. I bet that would show some interesting results percentage-wise regarding content-based edits vs talk-page or techie edits (template changes etc).

    • Steven says:

      Jane,

      That would be interesting just to look at edits, but since our project was done in order to support identifying and mentoring new editors with accounts, it would not serve our purposes well. We kept in mind the change in the ability of anonymous IP editors to create pages, which is why I didn’t focus in the post on the change in the percent of vandalism. I agree that it’s probably why there is such a difference in the number of vandal accounts that registered.

  3. Steven says:

    Hi Felipe,

    Just to answer your second suggestion: no, we’re not really looking at editor retention comparatively over the years, since we’re already pretty clear about those from the Editor Trends Study. I do think that Philippe and James are tracking retention of editors selected for mentorship by WikiGuides though. The same goes for matching “unacceptable” edits and reverts by the community; we really just wanted to look at quality of newbie edits, so we stuck to that for last week’s experiment.

    Thanks for your comments, they are all super insightful!

  4. Hi there!

    Nice findings. I also agree with the previous comments in that more info about the quality classification criteria would be great to put these results in context.

    Suggestions:

    1. It would be nice to increase a bit the size of the sample (although there is a trade-off with number of edits that can be assessed through manual inspection).

    2. Do you plan to measure % of retention of editors according to outcome of the first edit? This might spot some light on the potential influence of first experience with the project to keep on contributing.

    Quick question: is there a match between non-acceptable edits and reverts? That is, did you evaluated the edits independently or is this just what wikipedia editors decided about this first-time contributions?

    Thanks for sharing!!

  5. Hello,
    I think it would help to justify your sampling choices if you based them on some clear sampling methodology. Then you could state with certainty that your results are valid, with some margin of error.

    With regard to the study itself – I think this data becomes meaningful only when compared with the quality of edits of editors who are no longer new. Maybe they are also vandals?

    Finally, I agree with comments above that a simple 1-5 does not suffice: how do you compare a significant edit to a small – but factually correct – correction, and that to a spelling edit? Also, have you used multiple coders to evaluate the edits?

  6. Very interesting results. I think both graphs are interesting in that the vast majority of edits are acceptable, whether in 2004 or 2011. And that the graph didn’t move significantly away from that center.

    I redrew the data as a bar chart. It’s easier to see the differences and similarities between the two data sets.

  7. elya says:

    Thank you for this work. You answered a bit of it in the last post, however it would be nice to know what were the measures of quality for each of the classifications? I know that opinions about what is „acceptable“ may vary a lot between Wikipedians. And it would be very interesting to compare different language versions. Thanks + regards, elya

  8. Hi!

    Thanks for the work and for sharing this.

    Something which also confirm this study and might be good to reflect on is a quiet significant increase of vandalism. Then, it is also curious to me the profile of excellent new contributors. I wonder, who might they be?. Just to though out some ideas: People that don’t need to go though the learning curve to get into how Wikipedia works? Previously anonymous editors? Experienced editors who create a new account? People coming from other wiki experiences? “Perfectionist” or wiki-shy people who do not do an editing until they are sure it is going to be “good”? I would be curious to know if there are more women among them.

    Finally, is there a special reason why you did your sample with 2004?

    Thanks again. Cheers! Mayo

  9. Paul says:

    How are minor edits like spelling corrections classified? Are they “acceptable” or are they “low quality”? Most of my edits are of that nature.

    • Steven says:

      Great question Paul. Any edit that followed Wikipedia content policy and style was considered acceptable at least. So even if you just fixed spelling or grammar, that was classified as acceptable as far as our study goes.

      Wikipedia has a long tradition of encouraging “WikiGnomes” which mostly do these kinds of edits, so it’s a widely recognized kind of contribution in the community, even if we don’t always talk about it.

  10. Mary says:

    Another graph would be interesting to complete the two above : Quality of 155 randomly selected recent edits (2011) from new account creators in 2004. And : number of edits from old account creators vs recent new account creators.
    My guess is that an account activity on wikipedia decreses as time goes by and then wikipedia in my opinion needs new accounts creators to stay up to date.

  11. Elihu Harris says:

    I just want to know how accurate is your information? And can I use wikipedia as a reputable footnote source, or will I be laughed out of the room.?

  12. Miguel says:

    I agree with you that evaluating the sample plays a major role as the next step in the research process. However, the pattern that you speak of doesn’t guarantee persistence. Firstly because a sample of 155 editors in a given day taken from 1000′s of that month is far too small. Surely that cannot seem right to you? and secondly because they’re implying that the remaining months in 2011 should yield the same results as April. Again,arriving at quick conclusions from 1 given month out of 12. In fact, I am willing to bet that the results of April could differ from May or July or November and so forth

    • Philippe Beaudette says:

      If we were going to stop there, I’d agree with you. But I’ve just requested samples from 06 and 07 as well, so we can see if there is correlation. :)

  13. Steven says:

    It’s not the size of the sample, it’s how you select it and evaluate it.

    These are two randomly selected groups from the same date range, years apart, and yet the trends match. That clearly suggests a pattern that will persist no matter how large the sample size.

  14. Miguel says:

    The sample is way too small (both nu
    mber of editors and month)

  15. That doesn’t happend in id.wikipedia.org, see: http://stats.wikimedia.org/EN/ChartsWikipediaID.htm. Duh.

    • Steven says:

      When you say “that” I assume you mean thousands of new editors arriving? Yes, if you look at each language individually there are different growth statistics of course. But when we say “thousands of new editors still join Wikipedia every month”, we mean Wikipedia overall. See: http://stats.wikimedia.org/EN/TablesWikipediansNew.htm

      We only did the quality sampling in English this week, but it would be great to check it in other languages. At the size of id.wikipedia, we could do the same study for every new Wikipedian in a month or two, which is actually kind of cool I think. :)

Leave a Reply