Introducing Wikipedia Editor Satisfaction Index
The Wikimedia Foundation is working on new products and global initiatives to increase participation in our projects, specifically Wikipedia. To help inform the development of this work we’ve been researching the trends and patterns of Wikipedia editors, most recently through the Wikipedia Summer of Research initiative and also with data from the 2011 Wikipedia Editors Survey.
While studying editor participation trends, we have hypothesized that acrimony and disagreement in the editing community could be a leading cause of a decrease in project participation. To test this hypothesis as a segment of our analysis of responses to the Editor Survey (report here), we defined the Wikipedia Editor Satisfaction Index (WESI). The WESI is a metric gauging the overall satisfaction of the editing community and interactions/assessments of fellow editors. We used responses to two questions on the survey: how they described their fellow editors (picking from a set of adjectives), and whether they believed community feedback had helped them personally. These responses were weighted, and then normalized to a 0-10 rating.
The results were encouraging. About 47 percent of editors surveyed scored 10/10. In all, about 77 percent of those surveyed scored 7.5 or higher, indicating that the majority of our editing community is very satisfied with their experience on Wikipedia and has a healthy assessment of fellow editors. This is great news – as Wikipedia continues to focus on improving the editing experience, while also making efforts to foster new participation (especially in the Global South), the community’s support is vital.
Distribution of WESI scores across all surveyed Wikipedia editors
In order to understand what factors determine an Editor’s satisfaction with Wikipedia, we performed a multilinear regression1 on the WESI metric. Some interesting findings:
- Help is appreciated: Having others from the community add content or correct grammatical mistakes greatly increases the likelihood of an editor reporting a positive experience.
- Peer recognition matters a lot more than any other kind of recognition: Editors highly value the respect and recognition of their peers. Editors who received barnstars or any other form of reward from their peers were much more likely to report a higher score. Interestingly, events like having an article featured or promoted to the front page did not have a very significant effect on editor satisfaction.
- Explanations for reverts are key: When an edit is reverted, not explaining why has a strong negative impact on editor satisfaction. Similarly, an explanation actually has a strong positive influence on editor satisfaction.
A comparision of WESI scores reveals that women are, on average, less satisfied than men, though not by much – about 5 percent. Although transsexuals/transgenders (marked below as Others) together account for only 0.5 percent of our sample, it’s important to note that their satisfaction scores are significantly lower.
WESI score comparision by Gender
The Editor Survey Report highlights some more findings, but the emerging theme is simple: be nice to each other, and help out where you can!
As we work towards establishing the WESI metric as a standard for understanding the community’s experiences on Wikipedia, we’ll continue to share more findings (and implications) of the Wikipedia Editor Satisfaction Index.
Mani Pande, Head of Global Development Research
Ayush Khanna, Global Development Intern
(This is the eleventh in series of blog posts where we previously shared insights from the April 2011 Editors Survey.)
Removing the default edit summary text for “undo” and similar actions would help a great deal. Unexplained reverts are poison to collaboration, and most editors wouldn’t leave the edit summary blank even though they won’t modify default undo summary text. Use tags from the filter rules instead of wasting the edit summary.
How did you select the members of your sample? I hope you chose editors of a variety of tenure lengths. Otherwise, your results will be skewed toward the positive side (and it will probably be skewed toward the positive even if you did choose a variety of tenure lengths); those who are dissatisfied with Wikipedia tend to become inactive or leave, so they might not receive an invitation to your survey, and their experience may be overlooked. Did you contact any editors who have been inactive or left Wikipedia?
You might also investigate the unyielding attachment to the status quo. I’ve left Wikipedia twice; both resignations were a result of attitude from other editors, but the second was also influenced by Wikipedians’ stringent refusals to change. I suggested positive changes — these were supported by some other editors — but my suggestions were ultimately overridden simply because of Wikipedia’s contrary precedent.
I understand that Wikipedia has sort of an alternative policy on governance and decision making, but I think this is something that should be reconsidered. I have strong professional and educational credentials in the area where I suggested change, but a group of editors who have zero relevant credentials overthrew my proposition.
And, on a side note, I really hate the “ignore all rules” rule. That is the epitome of a mixed message, and mixed messages inspire chaos in any sort of governing body.
That’s a little bit of a vent session mixed with constructive criticism, but I appreciate that you’re trying to improve Wikipedia.
Dear J,
according to the report, the methodology is described thus:
“The survey was conducted online on all Wikipedia language sites in the last week of April. Using the help of the community, the survey was translated into 21 languages beyond English, including: Chinese (traditional, Hong Kong), Chinese (simplified), Serbian, Russian, Portuguese, Polish, Dutch, Macedonian, Italian, Hungarian, Croatian, Hebrew, French, Finnish, Spanish, German, Danish, Welsh, Catalan, Bulgarian and Arabic. The survey was conducted in native languages for which translations were available, and for the remainder of Wikipedia language projects the survey was available in English.
The survey was limited to only registered Wikipedia users, and each user saw a link to the survey only once. This ensured that all users (editors) had an equal probability of participating in the survey, and the survey was not biased towards those editors who edit more frequently. We had to set a cookie within the Central Notice banner for the survey to guarantee that it would only show up once per user. The foundation used an open source survey tool, Lime Survey, to field the survey, and the survey was hosted by the foundation. Survey participants also had the option of saving the survey and taking it later.
The survey ran for seven days on the website, with a total of 31,699 views of the banner invitation. By the end of the week, we had collected a total of 8,193 responses to the survey. After data cleaning, which involved removing instances of vandalism, deleting unfinished surveys, etc. we had a total of 5,073 responses.”
Hope this answers your query in your first paragraph.