Students in the Wikipedia Education Program in the United States and Canada improved article quality on the English Wikipedia by an average of 88 percent during Spring 2012, according to new research conducted by Luis Campos, an external data analyst. In the Wikipedia Education Program, professors assign their students to improve course-related articles, with support from Wikipedia Ambassadors who help students learn the basics of Wikipedia editing.

Experienced Wikipedia editors evaluated a random sample of articles students worked on as part of the Wikipedia Education Program in the United States and Canada in Spring 2012. The metric evaluators used, with assessment areas for comprehensiveness, sourcing, neutrality, readability, formatting, and illustrations, on a 26-point scale, is based on the Wikipedia 1.0 metric used across English Wikipedia. Evaluators provided two ratings, one for the article quality immediately prior to the first edit the student made, and one after the class had wrapped up their work; reviewers also used the same metric to evaluate articles that students created from scratch. A total of 124 articles formed the sample. Altogether (counting both new and pre-existing articles), articles improved on average 6.5 points, from 7.4 to 13.98 points on the 26-point scale. The graph below shows the quality distribution of articles before students worked on them (in blue), and the quality distributions of articles after students worked on them (in red).

Article quality improvement of sample of Wikipedia articles edited by students participating in the Spring 2012 Wikipedia Education Program in the United States and Canada. This graph shows overall improvement (both existing and new articles).

The 124-article sample included 82 existing articles and 42 new articles created by students. Existing articles improved 2.94 points on average, from 11.26 to 14.2, with the most improved article improving by 10.25 points. An example of such an article that a student improved is the article on vocabulary development. You can see the versions prior to students’ first edits and the status it was after the class finished. The graph below shows the distribution of pre-existing articles before (blue) and after (red) student work.

Article quality improvement of sample of Wikipedia articles edited by students participating in the Spring 2012 Wikipedia Education Program in the United States and Canada. This graph shows improvement of existing articles only.

New articles had an average score of 13.55. You can see a sample of what a student contributed to a new article by reading Temptation, a Václav Havel play. The graph below shows the distribution of quality of new articles students created through the Wikipedia Education Program.

The Spring 2012 numbers show improvement over the 2010–11 quality of students contributions from the Public Policy Initiative pilot of the U.S. program, where articles improved an average of 5.8 points. We’re encouraged to see improvement in Wikipedia’s article quality through the Wikipedia Education Program.

LiAnna Davis, Wikipedia Education Program Communications Manager

Does a constant stream of new editors really make Wikipedia better? Increasing participation is one of the top five priorities in our strategic plan. But when we talk about retention of newly registered editors, some readers and experienced editors rightfully wonder exactly how many edits by newbies actually improve the free encyclopedia.

In the Community Department, we’re facilitating the WikiGuides pilot program on the English Wikipedia to reach out to new contributors and mentor them. To do that successfully, we must quickly identify which new editors are actually doing good work.

So one of our working questions is: How many contributions by new editors are made in good faith and are worth retaining or improving?

We took a randomly selected batch of 155 new registered users on the English Wikipedia who made at least one edit in mid-April of this year. We looked at their first edit and ranked it on a 1-5 scale, with 1 being pure vandalism and 5 being an edit that is excellent, meaning it adds a significant chunk of verified, encyclopedic content and would be indistinguishable from a very experienced editor. Here’s what that composition looks like:

So you can see that even with a very high standard for quality — we only handed out a single “5” edit — most new editors made contributions worth retaining in some way, even if they weren’t perfect. More than half of these first edits needed no reworking to be acceptable based on current Wikipedia policy. Another 19% made good faith edits but needed additional help to meet standards defined in policy or guideline.

In order to investigate whether this has changed over time, we took a similar cohort from the same period in April 2004 and made the same qualitative assessment.

The key thing to note in comparing the two samples is that the percent of acceptable edits made by newbies did not dramatically decrease from 2004 to 2011. That’s despite the fact that the bar for quality has been raised over time, and that there are arguably fewer obvious contributions to make now that Wikipedia has grown by millions of articles.

Another relevant fact to consider is that while both cohorts are of 155 new editors, it took several days for that many new editors to join Wikipedia in 2004. In 2011, our sample is a tiny slice of the new editors arriving every month. For example: on Monday of this week more than 1,800 editors joined English Wikipedia and made at least one edit. On the equivalent day in 2004 there were only about 60.

Our sample strongly suggests that thousands of new editors still join Wikipedia every month with valuable contributions to make. Ensuring that we welcome these newcomers and show them the ropes is a top priority for ensuring Wikipedia’s continued success in our second decade.

(This is the first in what will be a new series of blog posts coming out of the Community Department at the Wikimedia Foundation. Starting now and continuing through the summer, we will be sharing the questions, experiments, and fresh data that currently drive our work. While you’ll get an inside look at what we’re doing, our numbers and analysis are still evolving and should be taken with a grain of salt.)

Steven Walling
Wikimedia Foundation Fellow, on behalf of the Community Dept. – especially Philippe Beaudette, James Alexander, and Maryana Pinchuk.