We’re happy to announce that the Wikimedia Foundation has started testing a new version of the Article Feedback Tool, to engage readers to help improve Wikipedia — and to become editors over time. We’re very excited about this new development, and look forward to getting more people to contribute to Wikipedia as a result.

Earlier this year, a first version of the Article Feedback Tool (“Rate this Page”) was rolled out to all articles on the English Wikipedia.  The idea behind this feature was two-fold: to provide a measurement of article quality from readers and to provide a potential on-ramp for these readers so that some may become editors.  We found through our analysis that while direct quality assessment is a very tricky matter (a rating of the Justin Bieber page says as much about the rater’s opinion of Bieber as it does about the quality of the article), the use of ratings as a form of low-barrier participation showed promise.  We also received plenty of feedback from the community around how we might improve this feature.

In October, we began development of the next generation of the tool (AFTv5).  Instead of focusing on explicit quality ratings, we shifted the direction of the tool towards finding new ways for readers to help build the encyclopedia.  So rather than primarily asking them to rate the quality of the article, we are asking readers for their input on how to improve the article. We are still testing different lightweight quality metrics, as well.

We are approaching this development in several phases.  The first phase, which went live today, is a test deployment of three new versions of the tool on approximately 10,000 randomly selected articles on the English Wikipedia and on a small number of manually selected articles. For examples, see Android, Wikipedia, and Global Warming.

Here is one of the three versions that are being tested:

This new version of the tool asks the reader whether they found what they were looking for, and if not, prompts them to explain what is missing.  The intent of this version is to provide editors with some idea of feedback on what readers are actually hoping to see when they read a Wikipedia article.  This information may then be used by the editing community when deciding how to improve the page.  The other two versions also ask for reader comments, but with different questions: the second version lets you make a suggestion, give praise, report a problem or ask a question; the third version lets you review the article. These new forms were developed by OmniTI, a web development firm, and were based on designs created by the Wikimedia Foundation in collaboration with the Wikipedia community. To learn more, visit the AFTv5 project page.

We are inviting members of the editing community to evaluate the quality of the comments coming in from each of these three versions of the feedback form.  The goal is to determine which of these versions is most effective at providing high quality feedback that can help improve articles.  Aaron Halfaker, a Wikipedia researcher from the University of Minnesota and a WMF contractor, has developed an evaluation tool that will enable Wikipedia editors to systematically evaluate the quality of the feedback provided. Assuming that these new versions provide constructive feedback, the next step would be to expose these comments in Wikipedia.  To that end, a “Feedback Page” is now under development with community input, and will provide a space where editors can view article feedback, moderate the comment stream, and promote the best contributions to the article talk page.

Oliver Keyes, a member of the English Wikipedia community, is under contract with the Wikimedia Foundation as a Community Liaison to involve editors in this project.  In this role, Oliver is moderating discussions, collecting feedback about the tool, and working with the development team to incorporate this feedback.  Many of the ideas that are in the current test versions came from discussions with these editors.  We will continue to work with the community very closely in the next stages of product design and development. If you’re part of the editing community and want to get involved, please email Oliver (okeyes at wikimedia dot org). Our immediate need is to help evaluate the comment streams generated by each option.  Very soon, we will also need editors to help us design the Feedback Page, which will be used to review and potentially act on the feedback comments.

We hope this new feature can help engage a broader community of readers to provide constructive feedback on articles, share what they know and contribute regularly on Wikipedia.

Howie Fung, Senior Product Manager

Fabrice Florin, Product Consultant

The development of a Visual Editor is one of the Foundation’s top priorities for the upcoming year, as laid out by the 2011-2012 Annual Plan.  There is plenty of evidence that wiki-markup is a substantial barrier that prevents many people from contributing to Wikipedia and our other projects.  Formal user tests, direct feedback from new editors, and anecdotal evidence collected over the past several years have made the need for a visual editor clear.

Developing a web-based visual editor is an extremely complex task.  It is perhaps the most challenging technical project ever undertaken in the history of MediaWiki development.  Here are some of the characteristics that make this project unique:

• We have to support editing in both the new way (via the Visual Editor) and the traditional way (via wiki markup).  This is important since it’s what our communities have used for more than 10 years: We can’t completely change the way they do their work overnight.  We need to, however, simultaneously support potential editors who are not comfortable with wiki markup.  So any editing system will need to be able to go back and forth between the Visual Editor and wiki markup with minimal, if any, disruption to the end user.  We will have to perform back-and-forth transformations without breaking things.  Anyone who has used an editor that has both “visual” and “html” modes should have a feeling for the challenges, but it’s even harder with wiki markup, because:
• Wiki markup is enormously expressive, complex and complicated, and there’s a huge amount of content which uses every facet of this markup language. Wikipedia articles employ a rich set of layout features, including images, tables, citations, mathematical formulas, “infoboxes” and other dynamically loaded templates which preserve a consistent look and feel for certain information, and many other elements that enable a compelling and educational reader experience (see the article on Calculus as an example).  Supporting compatibility with the full breadth of these features is an enormous technical challenge.

Over the past several months, the engineering team at WMF has made a lot of progress in developing this visual editor.  Today, we’d like to share the first prototype of a basic editing surface which supports the translation of what’s on the screen into wiki markup.  The demo, which can’t yet save or edit documents, supports both basic formatting (e.g., bold, italics, section heading) as well as many of the required features that people take for granted (e.g., cut/paste and undo/redo). However, it’s still very fragile, and you may easily end up with an unusable document. In the best case scenario, you can use it to generate valid wiki markup that you can copy and paste into an edit box on any MediaWiki wiki.

This version of Visual Editor should support most of the modern browsers but was tested mostly on Firefox, Chrome and IE9. We do support IE8 as well, but not IE7 (yet). The editor isn’t internationalized yet, but will be with the next release.

Try the visual editor sandbox

You can view the demo and see the wikitext translation by visiting the visual editor sandbox on mediawiki.org and playing around with any of the articles available for pre-loading.

Manipulation of an example document, showing the link editor.

Using the debugging tools in the top right, you can switch to side-by-side view of different content representations, including wikitext (icon with square brackets), which are dynamically updated as the text on the left changes.

We would love to get your input on our progress.  Please leave us comments by clicking on the “Leave Feedback” in the upper right hand corner of the demo, which will place your feedback on this page.  Thoughts on which tasks this interface makes easier or harder compared to your current workflow would be particularly helpful.  We’re very excited to share this progress and look forward to your feedback.

Where do we go from here? From here on, we will iteratively release features, bug-fixes, and updates.  We’ll continue to make this tool useful for more real-world use cases, and tick off additional features: creating pages, saving them, editing existing pages or sections, adding/removing images, editing data in templates, editing tables. . .the list goes on.

Our goal is to enable real-world editing of a subset of content soon, but it’ll still be some time until we can serve all the needs of even a small wiki community, let alone Wikipedia’s. Currently we’re targeting June 2012 for first production use at scale, either on a smaller wiki or a section of a larger one. It’s the biggest and most important change to our user experience we’ve ever undertaken, and we look forward to your help in making it happen.

– The Visual Editor Team, Wikimedia Foundation
Trevor Parscal, Inez Korczyński, Neil Kandalgaonkar, Roan Kattouw, Brion Vibber, Gabriel Wicke

Recently the tech team at WMF has deployed a couple experimental tools for gathering feedback on the experiences of new editors. With MoodBar, new editors can quickly and easily provide feedback on their editing experience by entering a 140 character comment. All these comments are posted as a feed on the Feedback Dashboard. Since then, over 8,500 of pieces of feedback have been created by thousands of users. You can watch stats roll in real-time on this report.

The Feedback Dashboard shows data provided by new editors through the MoodBar feature.

Today we are introducing new functionality that will enable experienced editors to easily respond to this feedback. Experienced editors who want to help new editors through their initial few edits may now respond in-line without leaving the dashboard. The new editor (who left the initial comment about being happy, sad, or confused) will then receive the reply on their talk page. This feature will make it easier for experienced users to lend a helping hand to new users, guiding them through their initial experiences editing Wikipedia.

It is now possible for more experienced users to respond to MoodBar messages directly from the Feedback Dashboard.

Steven Walling and Maryana Pinchuk have also started a Response Team of experienced editors willing to help out. So far, over 30 editors have joined. If you’re an editor, please consider helping out by signing up on the Response Team page.

Steven and Maryana will also be holding an “office hours” this Sunday, December 4th, for anyone interested in learning more about how to respond to new editor feedback and discussing the feature. If you’re interested, please attend!

Howie Fung and Brandon Harris
Wikimedia Foundation Tech Team

There are still three weeks left in the Wikipedia Participation Challenge (see prior blog post)!  So far, the competition has exceeded our expectations.  As of this morning, 78 teams (167 total individuals) from across the world have participated in the competition, with a total of 735 entries submitted. Half of these teams have beat the benchmark we set at the beginning of the competition, which is a testament to the quality of the teams and their submissions.   We can’t wait to see what great algorithms the participants are developing.

There’s still time to jump in before the competition closes on September 20, 2011, so if you haven’t done so, download the data and start crunching.  And those who want to cheer from the sidelines may follow the competition on Kaggle’s leaderboard.

Howie Fung
Senior Product Manager, Wikimedia Foundation

Diederik van Liere
Research Consultant, Wikimedia Foundation

Since May, the Article Feedback Tool has been available on 100,000 English Wikipedia articles (see blog post). We have now kicked off full deployment to the English Wikipedia at a rate of about 370,000 articles per day and will continue at this rate until deployment is complete.

Ratings interface for the Article Feedback Tool

We wanted to take a moment to briefly recap what we’ve learned so far, what lies ahead, and how we can work with the community improve this feature.   Features like Article Feedback can always be improved, so we will continue to experiment, measure, and iterate based on user and community feedback, testing, and analysis of how the feature is being used.

Rating data from the tool is available for your analysis — please dig in and let us know what you find. Toolserver developers can also access the rating data (minus personal information) in real-time to develop new dashboards and views of the data.

What We’ve Learned So Far

Readers like to provide feedback. The survey we’re currently running shows that over 90% of users find the ratings useful.  Many of these raters see the tool as a way to participate in article development — when asked why they rated and article, over half reported wanting to “positively affect the development of the page.”

Users of the feedback tool also left some enthusiastic comments (as well as some critical ones) about the tool. For example:

The option to rate a page should be available on every page, all the time, once per page per user per day.

As a high school librarian, I want my students to assess the sources of information they use.  This feature forces them to consider the reliability of Wiki articles.  Glad you have it.

Ratings seem like an interesting idea, I feel like the metrics used to determine the overall value of the page are viable, and I’ll be interested to see how the feature fares when it’s rolled out and has some miles under its belt.

The vast majority of raters were previously only readers of Wikipedia.  Of the registered users that rated an article, 66% had no prior editing activity.  For these registered users, rating an article represents their first participatory activity on Wikipedia.  These initial results show that we are starting to engage these users beyond just passive reading, and they seem to like it.

The feature brings in editors. One of the main Strategic Goals for the upcoming year is to increase the number of active editors contributing to WMF projects.  The initial data from the Article Feedback tool suggests that reader feedback could become a meaningful point of entry for future editors.

Once users have successfully submitted a rating, a randomly selected subset of them are shown an invitation to edit the page. Of the users that were invited to edit, 17% attempted to edit the page.  15% of those ended up successfully completing an edit.  These results strongly suggest that a feedback tool could successfully convert passive readers into active contributors of Wikipedia.  A rich text editor could make this path to editing even more promising.

While these initial results are certainly encouraging, we need to assess whether these editors are, in fact, improving Wikipedia.  We need to measure their level of activity, the quality of their contributions, their longevity, and other characteristics.

Ratings are a useful measure of some dimensions of quality.  In its current form, the Article Feedback Tool appears to provide useful feedback on some dimensions of quality, while the usefulness of the feedback on other dimensions of quality is still an open research question. Completeness and Trustworthy (formerly “Well-Sourced”) appear to be dimensions where readers can provide a reasonable level of assessment.  Research shows that ratings along these dimensions are correlated with the length and amount of citations, respectively.  We need to determine whether the ratings in “Objective” and “Well-Written” meaningfully predict quality in those categories. We released public dumps of AFT data and would love to hear about new approaches of measuring how well ratings reflect article quality.

We received feedback from community members on how to improve the feature. We’ve received a fair amount of feedback from the community on the usefulness of AFT, mainly through IRC Office Hours and on the AFT discussion page.  There have been many suggestions on how to make the feedback tool more valuable for the community.  For example, the idea of having a “Suggestions for Improvement”-type comment box has been raised several times.  Such a box would enable readers to provide concrete feedback directly to the editing community on how to improve an article.  We plan to develop some kind of commenting system in the near future.

Illustration of a potential "Suggest Improvements" feature

AFT could help surface problematic articles in real time, as well as articles that may qualify for increased visibility. We’ve started experimenting with a dashboard for surfacing both highly rated and lowly rated articles.   Ultimately, the dashboard could help identify articles that need attention (e.g., articles that have been recently vandalized) as well as articles that might be considered for increased visibility (e.g., candidates for Featured Articles).  We will continue to experiment with algorithms that help surface trends in articles that may be useful for the editing community.

Next Steps

Over the coming weeks, we will continue to roll out the Article Feedback Tool on the English Wikipedia.  Once this rollout is complete, we will start planning the next version of the tool.  For those interested in following the discussion, we will be documenting progress on the Article Feedback Project Page.  We would love to get your feedback (pun intended!) on how the feature is being used, what’s working, and what might be changed.  We also encourage folks to dig into the data.  Once the feature is fully deployed, there will be mountains of data to sift through and analyze, which will be a boon to researchers and developers alike.

We’d especially like to encourage members of the community to get involved in the further development of the feature.  If you’re interested in getting involved (e.g., design input, data analysis/interpretation, bug-squashing, etc.), please drop a note on the project talk page.

Howie Fung, Senior Product Manager

Dario Taraborelli, Senior Research Analyst

Erik Moeller, VP Engineering and Product Development

We are pleased to announce the launch of the Wikipedia Participation Challenge, a data modeling competition to develop an algorithm that predicts future editing activity on Wikipedia. The competition is hosted by Kaggle, a platform for data modeling and prediction competitions.  The Participation Challenge is open to community members and anyone else who is interested in analyzing Wikipedia data.  This is the first of two data competitions the Wikimedia Foundation will sponsor this year.

The goal of this competition is to gain a better understanding of the factors that encourage or discourage people from editing Wikipedia. Increasing the number of active editors is one of our strategic priorities. Both the Wikipedia communities and the Wikimedia Foundation stand to benefit from models that quantify the factors that determine whether a Wikipedia editor is likely to continue contributing. The competition asks contestants to develop a model to predict the number of edits a given editor will make in six month’s time.

The data used in this competition comes from the publicly available English Wikipedia XML data dump.  An anonymous donor has generously contributed $10,000 as prize money. There will be a Grand Prize for the best prediction, as well as special prizes awarded for the use of open source software. The Grand Prize winner will also be given the opportunity to present their prediction model at the 2011 IEEE International Conference on Data Mining.  The competition starts today and will continue until September 20, 2011.

Head over to our competition portal, download the data, and start crunching the data! And don’t forget to follow us on Twitter: #wikichallenge and @dvanliere.

Howie Fung
Senior Product Manager, Wikimedia Foundation

Diederik van Liere
Research Consultant, Wikimedia Foundation

We all like to feel valued. According to the 2011 survey of Wikipedia editors (see top-line data), among 17 variables, “being looked down on by more experienced editors” is the most likely to cause people to say they will edit less frequently (69% agreement), while “having others compliment you on your edits/articles” is the most likely to cause people to say they will edit more frequently (78% agreement).

On the other hand, editing Wikipedia has tended to become harder over time, and the likelihood that new users will receive correction/criticism has increased. This is reflected by various efforts to code and analyze the experience of new users, such as the recent Newbie teaching strategy research sprint undertaken within the scope of our Summer of Research.

Warnings, teaching and criticism tend to dominate the experience of new users

This chart shows the relative increase and decrease in warning messages, praise/thank you messages, criticism and teaching messages over time.

The drive for quality and reliability has led to the development of sophisticated automation mechanisms that aid in socializing new users to Wikipedia’s norms, policies and conventions. The act of expressing appreciation for other users, by contrast, is a largely manual effort. Whether it’s welcoming new users, inviting users to participate in specific topics or discussions, recognizing effort using barnstars and trophies, or just sending a whimsical note, expressing appreciation is not an activity that is facilitated by the software — in spite of its known importance for people’s likelihood to want to edit.

WikiLove is a simple experiment in appreciation. It makes it easy and fun to send barnstars or whimsical messages of appreciation to other users. The tool was first built by Wikimedia Foundation developer and Wikimedian Ryan Kaldari as a small gadget, and the new editor engagement team at the Wikimedia Foundation has developed it into a full feature over the last few weeks.

WikiLove is invoked from a user page by clicking the “Heart” icon.

How you can help

Currently, the Wikilove extension is only deployed on our prototype site.  While it’s a simple feature, we would still appreciate your help in testing and evaluating it.  To do so, create a test account on prototype (please remember to check the “remember me for 30-days” box — sorry, there is a known bug on prototype that requires the box to be checked).  Once your test account created and you are logged in, visit any userpage or user talk page.  You will notice a red heart icon to the left of the search box.  Click on the icon, send WikiLove, and let us know what you think by providing feedback on the WikiLove talk page.

What’s next

Our plan is to enable WikiLove on the English Wikipedia on June 29 (we may push the date depending on any issues surfaced). Users will be able to disable the WikiLove feature by going to My Preferences → Editing → Labs Features and unchecking the box that is labeled “Enable showing appreciation for other users with the WikiLove tab (experimental)”.

The tool has built-in tables that will help us measure how frequently WikiLove is used, and allow us to understand whether its usage actually affects (new) editor activity. Irrespective of that, we’re also interested in exploring tools specifically built for welcoming new users and inviting them to edit articles related to their expressed interests.

We’re not presupposing that we know how appreciation can best be expressed in the many different languages and cultures that make up the Wikimedia community. The selection of barnstars and other forms of appreciation that we’re starting with is just that — a selection. Wiki administrators will be able to modify the user interface by following the instructions for customization — so that whether you want to award gifts of chocolate or stroopwafels or baklava, you can do so!

Howie Fung
Senior Product Manager, Wikimedia Foundation

Erik Moeller
Deputy Director, Wikimedia Foundation

On March 14, we launched v2.0 of the Article Feedback Tool.  Version 2.0 is represents a continuation of the work we started last September.  To quickly recap, the tool was originally launched as a part of the Public Policy Initiative.  In November, the feature was added to about 50-60 articles on the English Wikipedia, in addition to the Public Policy articles.  The purpose of adding the tool to these additional pages was to provide us with additional data to help understand the quality of the ratings themselves, namely do these ratings represent a reasonable measurement of article quality?

Since then, we’ve been evaluating the tool using both qualitative and quantitative research.  We conducted user research on the Article Feedback tool both to see how users actually used the tool and to better understand the motivations behind rating an article.   Readers liked the interactivity of the feature, ease of use, and the ability to easily provide feedback on an article.  On the other hand, some of the labels (e.g., “neutral”) were difficult to understand.   A detailed summary of the user research has been posted here.

We also did some quantitative research on the ratings data.  Though the ratings do appear to show some correlation with changes in the content of the article, there is ample room for improvement (see discussion of GFAJ-1).  It also appears as though articles of different lengths show different ratings distributions.  For example, there appears to be a correlation between Well-Sourced and Completeness and length for articles under 50kb, but for articles over 50kb in length, the correlation becomes far weaker (see Factors Affecting Ratings).

Based in part on the results from the first version, v2.0 of this feature was designed with two main goals in mind.

• First, we wanted to see if we could improve the correlation between ratings and change in article quality by segmenting ratings based on the rater’s knowledge of a topic.  We introduced a question which asks the user whether she is “highly knowledgeable” about the topic.  The answers to this question will enable us to compare ratings from users that self-identify as highly knowledgeable versus ones that don’t.
• Second, we wanted to see if rating an article could lead to further participation — does rating an article provide an easy way to contribute, leading to additional participation like editing?  We wanted to test this hypothesis in light of the recent participation data.  We don’t know whether this will actually be the case, but we wanted to get some data.  In v2.0, there is a mechanism that shows a user a message (e.g., “Did you know you can edit this article?”) after they submit a rating.  We will measure how well these messages perform.  (These messages are dismissible by clicking a “Maybe later” link).

We also made some UI changes based on the feedback from the user study.  For example, “Neutral” was changed to “Objective” (as were some other labels) and the submit button has been made more visually obvious.  There are a number of other improvements which may be found on the design page.

Finally, in an effort to get a wider variety of articles to research, we increased the number of articles with the tool.  We knew from our early analysis that articles in different length bands received different rating distributions, so we created length buckets (e.g., 25-50kb) and selected a random set of articles within each length bucket.  User: Kaldari wrote a bot which takes the list of articles and places the tool on the articles in the list [10].  As of March 24, there are approximately 3000 articles that the tool is currently active on.  We may expand this list if we can do so without impacting performance of the site.

We’ll be publishing analysis on v2.0 in the coming weeks.  In the meantime, please let us know what you think on the workgroup page.  Or better yet, join the workgroup to help develop this feature!