Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

The Rise of Warnings to New Editors on English Wikipedia

(The following is the second installment in the Community Department’s series on preparations for our summer research into Wikipedia. As with the last post, keep in mind that these numbers and findings are preliminary – at this stage we are only testing out different methods and practices. Over the next few months, we will work on scaling each of these experiments and continue to probe more intensely into their methods and results.)

In preparing for this summer’s research projects, we have been experimenting with qualifying and classifying comments left on user talk pages, a space where editors can discuss and receive feedback about their contribution to the encyclopedia. In the process, we began to see a troubling trend that appeared at all stages of our sampling.

We prepared a random sample of several hundred edits made to user talk pages of new registered users on English Wikipedia from 2004 to 2011. These edits were made by other contributors within 30 days of a new person’s first edit. We then tagged each edit with several labels and noted the tone of each message. We excluded talk pages of vandals or users who were banned for spam or abuse of multiple accounts – in other words, we tried to include only talk pages of new editors who were clearly trying to participate in good faith.

When we began to add up the totals per year, the results were striking. Again, we want to strongly caution that the purpose of this exercise was to experiment with methodologies for sampling of Wikipedia data – not an end in and of itself.

Nevertheless, when we analyzed this sample, we discovered a distinct trend: a marked decrease in praise for contributions (anything from a simple “great job on that article!” to a barnstar), and a simultaneous increase in warnings and criticism delivered via templates (e.g., third- and fourth-level vandalism warnings, copyright violation warnings with aggressive images like stop signs or red X marks, and threats of block or bans) since 2006. Note that we excluded warning templates from this group if they were extremely gentle (e.g. “Welcome to Wikipedia. Your test has been reverted, please use the Sandbox…” etc.)

As you can see from this graph, in 2011 almost 40% of all initial edits to new user talk pages in our sample were negative templates, while praise was virtually nonexistent. Even accounting for anomalies within individual years of our samples, the trend suggested is clear.

In our last blog post, we pointed out that while vandalism has been on the rise in the past five or so years of the project, it appeared that good-faith editors still represent over half of all new registered contributors – a trend that we found in this sample as well. So while the increasing use of negative template warnings may be a reaction to increased vandalism on English Wikipedia, it is possible that large numbers of good-faith editors are getting caught in the crossfire.

Over the next few weeks, we will be looking at ways to fine-tune our system by getting more involvement from the Wikipedian community. If you would like to find out more about our classification system or join the experiment, please sign up on our project page on Meta. Documentation of our process and data is included there as well.

Maryana Pinchuk, Research Fellow, on behalf of the Community Department

15 Responses to “The Rise of Warnings to New Editors on English Wikipedia”

  1. aldreds says:

    This posting is right on the money. A few years ago I became interested in contributing to WikiPedia and full of enthusiasm I began by adding an image of a camper trailer I designed and built along with a sentence of text.
    Initially my experience was good–someone came in and adjusted the image size and then another found a more relevant area to place the content I authored.
    Unfortunately, after that initial good experience the editorial zealots took over and issued a show cause notice why my image should not be deleted as they claimed that it was a breach of copyright. I of course explained that the camper was my own design and I built it myself and I then provided the appropriate creative commons clearance. These were ignored and the image and the text I provided were eventually deleted.
    It’s a shame that the same zeal in not applied to racist vandalism I repaired in aboriginal Australians in WikiPedia around the same time.

    Anyway my experience was a disincentive to contribute to WikiPedia again and I have not done so since that time.

  2. Maryana says:

    Lode,

    Thanks for the feedback! We hear stories like this all the time, and now we’re finally starting to collect the numbers to back them up.

    An update about where to discuss this issue: here is a good place to start.

  3. Lode Claassen says:

    This is very interesting data! I can confirm the “anecdotal evidence” as noted in the comments. I do a few edits per year, and get notified of older edits / articles I’ve contributed to. All reactions / notifications are negative.

    Also I see, slightly related, a change in extreme measures when something doesn’t meet the standards. I see it more and more that (old/long) articles get plain deleted because they don’t meet the standards. Instead of working together to let them meet the standards. This is of course not the same as comments on user pages, but to me it is the same tone of voice.

    As DFB said (http://blog.wikimedia.org/blog/2011/05/02/neweditorwarnings/#comment-7378), where is the discussion?

  4. Maryana says:

    Dana,

    We did see speedy deletion templates cropping up more and more after 2007, and those definitely tend to be very negative (big red icons, bolded text, etc.), but we didn’t keep track of the exact numbers. We’ll be sure to flag this question for further study.

  5. GreenReaper says:

    That may be the case, but who has time to praise them? I have to spend all my time trying to fend off deletionists from articles in “my” WikiProject! ;-)

    Perhaps it was done in the past by recent changes reviewers, but from what I can see those people have been replaced by bots and Twinkle cyborgs – in part because almost a quarter of new creators’ edits are vandalism, and serving on the front lines is a clear route to the mop. Now the decision is not “What should the response be to this editor?” but “Is this edit vandalism? No? Go to the next edit.”

    It might be interesting to see whether those people being warned did in fact “deserve” it. The first talk page edit probably does not relate to the user’s first edit, and even those whose initial contribution was at an average level may contribute lower-quality edits later. For example, they may upload an image without a fair-use template, or remove a section without an edit summary.

  6. Dana says:

    Do you have any data on what percentage of the negative messages were due to the new editor creating an article that was deleted? (“An article that you created has been tagged for speedy deletion”, etc.) That might have relevance for the current discussion on restricting article creation to autoconfirmed editors.

  7. karl says:

    For someone who is contributing at an irregular pace but still a few times through years and having started a few pages. My experience varies depending on the topic, but it is happening more often that the first edits of a page trigger reproaches when it is still in the birth moment of the page.

    Some editors seem to want to more complete first page, which is cutting the good faith and will and even desires to continue. Maybe an intermediate solution could be found. The page having a kind of draft stage for a little while and instead of « you do bad » a message « here you could improve by… »

  8. GreenReaper says:

    I can think of a few possibilities:

    * new users today are worse than those in the past – perhaps because those most likely to thrive at Wikipedia might have been attracted early, because they are contributing for less altruistic reasons, or because the page rank means everyone and their 8-year-old ends up on the site
    * new users are less likely to have a positive impact because the quality of existing articles is higher than in the past
    * new users are just as bad as in the past, they’re just getting caught earlier (before they get any better) and are being warned earlier and more frequently due to increased automation

    Good faith might shield you from a ban, but it won’t prevent criticism. A negative tone is preferred because the goal is to protect what is there already – not to encourage or tutor new editors. Warning signs are believed to be effective in deterring further editing.

    Wikipedia is the “big leagues” nowadays. Sure, anybody can edit, but they have to do it right. If you want to learn how to edit, you have to do it on your own wiki.

    • Steven says:

      GreenReaper,

      When it comes to the question of how good the first edits by these newbies are, we did some research that suggests there are just as many people making quality edits that are worth being praised instead of being warned. (Even accounting for the higher editorial standards for the encyclopedia.)

  9. Maryana says:

    DFB,

    While I’d caution once more that our sampling at this point is far from scientific, anecdotal evidence does suggest that this graph reflects a major problem on English Wikipedia – an increasingly hostile environment for new users. Discussion of how to fix it is on-going.

    Some work has been done by the Wikimedia Foundation Board to get the discussion going (see their latest resolution on openness in the community). In English Wikipedia, this has led to the creation of the WikiGuides project to protect and nurture good new editors, and in the Russian Wikipedia, there is a similar pilot Incubator project that mentors new users by letting them create articles and receive positive feedback in a specially designated namespace.

    We’re hoping that these are the first of many innovative community-led projects to help retain new editors, and the more data we accrue over this summer, the easier it will be for community members to find practical solutions to the problem.

  10. DFB says:

    The trend clearly shows a problem. Where is the discussion about how to fix it?

  11. Interesting work. It seems it’s going to be a really interesting Summer for Wikimedia projects.

    Two comments:

    1. I’m curious about the approach followed to label emotional traits of comments in this sample (that is, to classify them as positive or negative feedback). As you suggest above, some of the apparently negative comments may be related to assessment about policies or guidelines that new editors should follow (again, suggesting the need to create a clearer path for new editors to discover and learn them before editing).

    2. I suspect that, as a result of development efforts to be conducted next Summer, you will produce many lines of valuable (open source) code. Please, stop for a moment to think about the best way to organize access and documentation to that code. Otherwise, we will make the current situation even worse (code and tools scattered around in many different places, so it’s really difficult to keep track of all of them).

    • Steven says:

      Hi Felipe,

      Thanks for your very thoughtful comments! Responding to your points…

      1. Does it make it clearer to point out we classified type and tone separately? So if there were feedback comments of any kind, we noted that regardless of tone. We then noted afterwards the tone of every edit in the sample. So our dataset was structured in a way where we can see type or tone individually, as a whole or associated with a certain kind of talk page edit. We specifically did that because you’re correct, negativity of tone definitely doesn’t mean commentary was not focused on constructive topics necessary for acculturating new editors. The line between what is a helpful lesson in policy (for example) and what is badgering that might drive someone away is very fine and subjective.

      2. Absolutely correct, thank you. For the students coming in the summer, their more comprehensive code will be checked in to the public svn repository. The Meta page(s) about the “WMF Summer of Research” will have complete descriptions of the methodology and code available. For this data set, the only actual code involved was a single SQL query run against the toolserver to grab the initial raw sample. All the analysis of type and tone was done by hand.

  12. Steven says:

    Hi Mike,

    We actually did mean percent. I know the numerals on the y-axis are slightly confusing because we didn’t label them as percentages, but our charting program failed to let us label 40% as that instead of .40. Sorry for any confusion!

    To answer your other question: we wanted to save the deeper analysis of the rest for another time to make this point, but if you’d like to see the raw data let me know.

    A few other trends I can describe though: the majority of all interactions in all years were either warnings, tips, or corrections of some kind about procedure or policy. Those could be either positive or negative in tone (most new editors need constructive guidance), so those plus the steady number of welcome messages/templates throughout the years made for a fairly even mix of positive and negative messages throughout the rest of the sample.

  13. Mike Peel says:

    I think you meant “Fraction” on the y-axis rather than “Percent”. But even then: what about the remaining >60% of comments? Were they ‘gentle’ ones?

Leave a Reply