Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Study Suggests Majority of New English Wikipedians Edit in Good Faith

Last month we presented findings from an experimental qualitative analysis of the first edits by new contributors to English Wikipedia in 2004 and 2011. Assessing these edits, we discovered that the majority of initial contributions in both of those years were of acceptable quality according to Wikipedia standards.

But we did not extend that study over all years, and we didn’t look at the complete contribution history of each new crop of editors to see if they edited in good faith overall. In the past week we’ve done exactly that.

Our randomized sample is of about 150-200 new editors to the English Wikipedia each year, sampled from February. As you can see, a clear majority of new editors in the sample participated in good faith. The good faith participation reached its lowest proportion in the sample at a little over 66%.

Note that “good faith” does not refer to whether edits were very high quality, only that editors did not act in clear, intentional disregard for the pillars of Wikipedia. Another practical note: identifying when an account is a duplicate made by a previous editor (i.e. a sockpuppet) is quite difficult, so for our analysis only editors blocked specifically for abusing multiple accounts were counted as such.

We also calculated a projected amount of editors who would fall into each of these categories, based on the comprehensive number of new editors each February from stats.wikimedia.org. The chart below represents these projected total numbers.

If our samples are correctly representative of the whole new editor population, there were about four times as many new good-faith editors who began contributing in February 2011 as in 2005.  While there has been an increase in the amount of vandals and spam since 2006, the proportion of such bad faith participation has not continued to grow exponentially.

Steven Walling and Maryana Pinchuk,
Wikimedia Foundation Fellows

(This is one in a series of posts about the ongoing research within the Community Department at the Wikimedia Foundation. Methodologies and results are very much experimental, and as such should be looked at with a skeptical eye. You can take a closer look at our data and other experiments on Meta.)

5 Responses to “Study Suggests Majority of New English Wikipedians Edit in Good Faith”

  1. DFB says:

    Did you count all new users, or just new registered users? Because around 2005 vandalism was around 5%: http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Vandalism_studies#Conclusions_from_study_1

    • Steven says:

      DFB,

      Only new registered users. It would be interesting to do an updated study to try and compare quality of anonymous and registered editors to see what percent of total vandalism comes from either group. I suspect the proportions would be about the same as in 05, but that’s just my guess.

  2. tedder says:

    I’m really suspicious of the categorization of edits. Poor edits can easily look like good-faith edits. I also strongly suspect observation/attribution/confirmation bias.

    If this is taken further, the list of sampled edits should be published, with how they were categorized. It’d be easy to audit the edits for confirmation bias.

    • Steven says:

      Ted,

      You said it’s hard to tell a “poor” edit from a good faith one, but for this sample we did not make an assessment of whether edits were high quality or not. I’d be happy to give you more specific definitions with examples, but we don’t really want to publish the complete list of sampled editors alongside each categorization (Our total numbers are already on Meta).

      I strongly feel the positive side of our method is that a qualitative classification by humans with experience reading diffs and understanding Wikipedia guidelines is more accurate than machine-driven methods of understanding the intentions of editors or quality of edits. (We worked to set up clear shared definitions from the get-go, and did extensive double checking of our work.) The disadvantage is that doing that totally in public is not really possible ethically-speaking, because it’s human beings making judgement calls about editors who have not agreed to being publicly examined as research subjects.

  3. Bence says:

    It would be interesting to similarly study the behaviour of anonymous editors. I expect that creating an account is inherently a good-faith step, whereas those who do not create accounts could be more representative of the overall population. Seeing changes in the behaviour of anons would perhaps lead to some interesting insight.