(The following is the second installment in the Community Department’s series on preparations for our summer research into Wikipedia. As with the last post, keep in mind that these numbers and findings are preliminary – at this stage we are only testing out different methods and practices. Over the next few months, we will work on scaling each of these experiments and continue to probe more intensely into their methods and results.)
In preparing for this summer’s research projects, we have been experimenting with qualifying and classifying comments left on user talk pages, a space where editors can discuss and receive feedback about their contribution to the encyclopedia. In the process, we began to see a troubling trend that appeared at all stages of our sampling.
We prepared a random sample of several hundred edits made to user talk pages of new registered users on English Wikipedia from 2004 to 2011. These edits were made by other contributors within 30 days of a new person’s first edit. We then tagged each edit with several labels and noted the tone of each message. We excluded talk pages of vandals or users who were banned for spam or abuse of multiple accounts – in other words, we tried to include only talk pages of new editors who were clearly trying to participate in good faith.
When we began to add up the totals per year, the results were striking. Again, we want to strongly caution that the purpose of this exercise was to experiment with methodologies for sampling of Wikipedia data – not an end in and of itself.
Nevertheless, when we analyzed this sample, we discovered a distinct trend: a marked decrease in praise for contributions (anything from a simple “great job on that article!” to a barnstar), and a simultaneous increase in warnings and criticism delivered via templates (e.g., third- and fourth-level vandalism warnings, copyright violation warnings with aggressive images like stop signs or red X marks, and threats of block or bans) since 2006. Note that we excluded warning templates from this group if they were extremely gentle (e.g. “Welcome to Wikipedia. Your test has been reverted, please use the Sandbox…” etc.)
As you can see from this graph, in 2011 almost 40% of all initial edits to new user talk pages in our sample were negative templates, while praise was virtually nonexistent. Even accounting for anomalies within individual years of our samples, the trend suggested is clear.
In our last blog post, we pointed out that while vandalism has been on the rise in the past five or so years of the project, it appeared that good-faith editors still represent over half of all new registered contributors – a trend that we found in this sample as well. So while the increasing use of negative template warnings may be a reaction to increased vandalism on English Wikipedia, it is possible that large numbers of good-faith editors are getting caught in the crossfire.
Over the next few weeks, we will be looking at ways to fine-tune our system by getting more involvement from the Wikipedian community. If you would like to find out more about our classification system or join the experiment, please sign up on our project page on Meta. Documentation of our process and data is included there as well.
Maryana Pinchuk, Research Fellow, on behalf of the Community Department