We are currently planning to roll out a new version of the FlaggedRevs extension to all wikis on Tuesday, November 23 starting roughly 3:15pm PST (23:15 UTC). This is used for Pending Changes on en.wikipedia.org and Flagged Revisions on many other wikis (such as de.wikipedia.org and pl.wikipedia.org). The update sports a new reject button, some diff page load optimizations to help complicated diff pages load faster by displaying the diff prior to displaying the old revision, and many under-the-hood code improvements.

We have several test environments in place with FlaggedRevs/Pending Changes configured:

• 1.16wmf4 core + trunk FlaggedRevs extension (this is the closest to what we plan for our production environment):
  • en.wikipedia.org: http://prototype.wikimedia.org/flaggedrevs/
  • de.wikipedia.org: http://prototype.wikimedia.org/flaggedrevsde/
• trunk core + trunk FlaggedRevs extension (this contains several changes that won’t be deployed for a while, in addition to the updated FlaggedRevs extension):
  • de.wikipedia.org: http://prototype.wikimedia.org/de.wikipedia.org/
  • pl.wikipedia.org: http://prototype.wikimedia.org/pl.wikipedia.org/

Please let us know if you have any problems.

Update 2010-11-23: this work is now complete.

As mentioned in this post in January, the English Wikipedia will be trying out the Flagged Revisions extension, using a configuration we’re calling Pending Changes.

This new configuration requires new features, which in turn required substantial code changes to Flagged Revisions. For technical reasons, we can’t release that code just to the English Wikipedia, so we will upgrade all copies of Flagged Revisions in use on Wikimedia Foundation projects. Happily, that will result in a number of minor improvements for all Flagged Revisions users.

We’ll be rolling this code out on Monday, June 14th, around 23:00 UTC. We believe this won’t cause any trouble, but if bugs do crop up, we’d like to hear about them. Our team will be standing by, prepared to fix any small issues immediately, or roll back if there are any big problems. The best way to contact us will be via the wikimedia-tech IRC channel or our issues page. (If you happen to read this blog and are a participant in a wiki that uses Flagged Revisions, we’d appreciate you checking your local Village Pump around then, so as to minimize language issues.)

For more information on the rollout of Pending Changes for the English Wikipedia, keep an eye on the main Wikimedia Foundation blog for an upcoming post, and check out the Pending Changes help page and the details of the trial.

There has been a lot of interest lately in Flagged Revisions, a quality control mechanism for MediaWiki. In particular, people want to know when and how that’s getting used on the English language Wikipedia.

I’m William Pietri, a San Francisco software consultant who recently came on part time to do project management for this. In addition to my long experience building web software for on-line communities, I’m also a Wikipedian. Although I haven’t done much more than small corrections lately, I started editing in 2004, and became an admin in 2007.

My job on this has two main parts. The first is to make sure that everybody working on this gets everything they need to make progress. The second is to communicate progress to the wider world. In the spirit of open communication, this is an update in question-and-answer form. Mostly from real questions people have actually asked me, but I’m going to sneak in a couple that I expect somebody will ask shortly.

What is Flagged Revisions?

You can find more detail here, but Flagged Revisions is basically a way to insert a quality review step between someone editing an article and that article version being published for the general public to see. It has been in use on the German Wikipedia since May 2008, and implemented in other languages and projects since then (see this page on Meta for a full list). Typically, in those use cases, every single article is treated in this way, and every change by a new user has to be reviewed.  There are a number of ways Flagged Revs can be used, and the proposed implementation for English Wikipedia (described below) is quite different.

Fundamentally, the objective of this technology is to reduce the exposure of readers both to subtle and not-so-subtle malicious changes in articles (whether it’s the insertion of blatant nonsense, or claiming the death of a celebrity), and to reduce the workload of people patrolling these changes by reducing duplication of effort.

What about the English language Wikipedia?

The use of Flagged Revisions on the English Wikipedia has been under discussion for a long time. Ultimately, the English Wikipedia volunteer community developed a proposal titled “Flagged protection and patrolled revisions“, which garnered strong support. It is fundamentally different from the way the technology has been used so far. Instead of requiring every change by a new or untrusted user to be reviewed, the mechanism would be activated on a per-page basis only, as an alternative to existing tools to restrict editing.

Notably, thousands of articles in the English Wikipedia, typically pages with a very high risk of malicious editing (e.g. major political figures), are currently “semi-protected”, meaning that new or unregistered users cannot make any changes at all. This new tool would make it possible to open up these pages for editing, provided that potentially problematic changes receive positive review. As a result of the more open approach, more high-risk pages could be made subject to this level of community moderation.

Initially, the English Wikipedia volunteer community wants to trial the system for two months. In addition to this alternative to page protection, the proposal calls for implementation of a new feature called “patrolled revisions”, which doesn’t impact what readers see, but is designed to make it easier for change patrollers to organize their work.

How is the Wikimedia Foundation responding to this proposal?

The Wikimedia Foundation (WMF), together with Wikimedia Germany, has driven and funded the development of the FlaggedRevisions technology since 2007 to the point where it has been able to scale to more than a year of production use in our second-largest project, the German language Wikipedia. WMF has carefully reviewed the English Wikipedia proposal, and allocated resources to assess its impact and support implementation along the principles outlined in the proposal.

The technology as proposed markedly differs from the way it’s been used before, so it’s a substantial development and design effort to get it right. For example, the notion of a per-page setting necessitates an entire set of user interface changes that allow changing the setting of a page, and that make it clear to a reader what the state of a particular page is. See this mock-up by Howie Fung as an example of what revised per-page controls could look like. WMF will post further mock-ups for feedback and prototypes for testing as they are built.

Who is currently working on the project?

• Aaron Schulz, a contract developer with Wikimedia, is the lead developer of FlaggedRevisions.
• Howie Fung, a contract product manager who also works with Wikimedia’s Usability Initiative, is supporting usability and product review of the software.
• I, William Pietri, support the project management of the English Wikipedia roll-out as described above.
• Erik Zachte, Wikimedia’s Data Analyst, will develop metrics specifically assessing the impact of the English Wikipedia rollout.

When will it be done?

This question has been asked a lot lately. Because this isn’t the flipping of a switch but a software development project, answering it requires me to let you in on a secret about software development projects. There are basically four ways to deal with dates, but only three of them are sane:

1. It’s done when it’s done. Nobody mentions dates. The developers code until they’re finished. Then you release, get feedback, and code some more.
2. Measure progress and project dates. You lay out all the work, estimate relative size, and then measure how much you get done over time. That data is used to figure out release dates.
3. Pick a date and release whatever you finish. If you’re building, say, annual tax return software, it’s better to ship on time and drop features than it is to finish late with everything.
4. Make up dates to please people. This is very popular, and has the advantage of making people happy at first, but it rarely works out well.

Until recently, we were using the first approach. That’s how most Mediawiki development (and most other open source development) works, and it has many advantages. But because a lot of people are eager for this project to launch, we’re shifting to the second approach.

The developer, Aaron Schulz, has estimated all of the items on the work list and already started in on them. The holidays complicate things some, but I expect we’ll have enough data to make a first guess at the estimated release date by the middle of January.

Wait, there’s only one developer on this? Is the Wikimedia Foundation taking this seriously?

Yes, absolutely. Aaron has been working on the Flagged Revisions extension for years, and nobody knows it better. We talked about adding developers, but unfortunately adding more people now wouldn’t help. I haven’t dug into the history much, but it looks like the real slowdown lately wasn’t in the coding; it was in turning the many-voiced community response into a clear set of things to do.

Having spent time with all the people involved, it’s clear to me that the Foundation takes this project very seriously. It’s one of small number of high-priority projects, which include things like keeping the site running, organizing the annual fundraiser, and Wikimedia’s usability initiative.

How can I keep track?

There are a few ways. First, we’ll mention big updates (and the eventual release) here on this blog. Second, keep an eye on the labs site. That will be updated regularly with the latest code and configs. You can judge for yourself how we’re doing, and make sure we do it right. And third, I’ve put the work queue into a public web-based tool called Pivotal Tracker. It’s one of the few software project management tools made for the measure-and-project approach we’re using; if you’re the sort of person who likes way too much detail, you can find real-time updates there.

How can I get involved?

Go to the labs site, play with the current implementation, give feedback, post your own user interface design suggestions, report bugs, and so forth.  Further community discussion in the English Wikipedia about the proposed roll-out is happening on the page about flagged protection and patrolled revisions. I’m always glad to hear feedback, either on my talk page or via email. And of course, you can comment on this very blog post.

William Pietri
Contractor, Wikimedia Foundation

One of the wonderful characteristics of Wikimedia’s wikis, including Wikipedia, is that every change ever made to a page is recorded, back to the very first version (compare, for example, the first version of the article about chess with the most recent version of the same article). This characteristic also makes it possible to assign quality assessments to specific versions, thereby giving our readers greater transparency about the perceived current or past quality of an article.

A very powerful software feature called Flagged Revisions makes it possible to systematize such quality assessments.  It’s been in production use in many of our wikis for more than a year now, including the second-largest Wikipedia, the German language edition. Fundamentally it’s a very flexible feature, and different project communities (the German Wikipedia, the English Wikibooks, etc.) can come up with configurations that suit their needs. By means of our public issue tracker, they can then request from the Wikimedia Foundation that such configurations be turned on.

Even though we’ve made no official announcements about this, you may have seen media reports that Flagged Revisions will soon be enabled in the English Wikipedia. Indeed, there is a specific proposal that was developed by the English Wikipedia community, entitled Flagged protection and patrolled revisions. It’s a very thoughtful proposal that attempts to balance the desire for higher quality, and more systematic assessment thereof, with the immediacy of Wikipedia as it exists today, and was supported by a large majority of interested Wikipedia editors. The idea behind this proposal is to allow regular contributors to systematize a first, basic assessment of all edits by new contributors. However, this assessment will be purely for informational purposes to the reader: a reader will see whether or not the version of an article they look at has been patrolled, and if not, whether a prior patrolled version is available.

Only in a small percentage of cases, we would require changes to be patrolled before becoming the default view for readers. The proposal is to do so initially in the case of articles at high risk of vandalism, including high risk biographies of living people, where false information can do the most serious harm to an individual.

A popular media narrative of this proposal (in the cases where it has been reported roughly correctly to begin with) is that it represents a “clamping down” on Wikipedia’s open editing process. That is nonsense. It is presently the case that many high-risk articles are completely uneditable by new contributors, which is referred to as page protection. For example, as a completely new user, you are not able to alter the article about Barack Obama. These kinds of protections of high-risk articles have been common for many years now. If the proposed model works as intended, it will actually allow us to open up many articles for editing which are currently protected from being edited. Edits will have to be patrolled, which is clearly a step up from edits not being possible at all.

It is true that some implementations of Flagged Revisions are more conservative than that. Any edit in the German Wikipedia by a new or unregistered user has to be patrolled before becoming visible to readers. This is definitely not the case in the proposed English Wikipedia configuration. We believe in letting our communities experiment with different approaches in an attempt to find the right balance.

A test wiki for the English Wikipedia configuration has just been set up in the Wikimedia Labs, and we’ll be importing articles from Wikipedia soon and make a broad call for testing. It’s important for us to get this right – we want to make sure that we don’t make Wikipedia harder to use, for our readers or our editors, in the process of deploying this functionality. That said, we hope to be able to deploy Flagged Revisions in production use on the English Wikipedia within 2-3 months.

From Wikimania in lovely Buenos Aires,
Erik Moeller
Deputy Director, Wikimedia Foundation

[UPDATE 8/26] This post originally said that all biographies of living people would be “flagged protected”. This is not correct. The current proposal is for for articles that are currently under normal mechanisms of protection (where new and unregistered users cannot edit) to be eligible for the new protection model, which allows for more open editing. I apologize for the confusion; thanks to Sage Ross for the quick correction.

Wikipedia was founded on radically open collaboration. Pick any article you know something about, and the “edit this page” link at the top allows you to make an instant change.

By editing a Wikipedia article, you get instant access to the “guts” of the page. Whether you’re just changing some text, adding a reference, or inserting an image: Wikipedia is open to new contributions at any time.

Instead of moderating edits when they are made, the wiki model has always been to systematically review changes as they come in:

• by storing every version of every article ever created;
• by allowing anyone to restore prior versions;
• by providing numerous tools for experienced editors to review and patrol changes.

This gives writers the instant gratification to see their changes published, while – hopefully – leading to high quality articles over time as more and more people review and improve a page.

In addition to the constant mutual peer review, there are countless Wikipedia processes used to identify articles of the highest quality, articles with various problems, or articles that should be deleted. (The Wikipedia Signpost, a community newsletter, has just published an interesting history of the featured article candidacy process.)

New processes and technologies for quality assurance are developed and tested all the time. But few are as long-awaited and potentially game changing as FlaggedRevs.

The FlaggedRevs Extension

The German Wikipedia is currently trialing a new extension (what’s an extension?) to our software, called “FlaggedRevs“. The extension, which has been under development for more than a year, is a very powerful set of tools for reviewing, labeling and selecting changes made in a wiki. We believe that FlaggedRevs represents a milestone in the development of wiki technology. To our knowledge, there is no other tool available today that provides comparable functionality.

So what, exactly, does it do?

In a nutshell, FlaggedRevs (short for “flagged revisions”) can be used to give a defined group of authors the ability to attach quality labels (flags) to individual versions (revisions) of articles. It can also be used to determine which version of an article should be shown to a reader visiting the wiki: the most recent one, or the highest quality version available?

These two features are not necessarily linked. In the most basic use scenario imaginable, FlaggedRevs can simply be used to patrol a wiki for malicious changes (“vandalism“). When a change has been found not to be malicious, a trusted user can label it as such. This has two key advantages compared to the current patrolling model:

• It reduces duplicate effort in basic change patrolling, allowing users to focus on un-reviewed changes and thereby directing their attention more effectively.
• It ensures higher coverage of changes. In particular, when malicious changes are followed by good faith edits, malicious changes are sometimes overlooked. In the FlaggedRevs model, reviewers can systematically examine every change.

In addition, both human and non-human readers can select “known good” versions of Wikipedia articles which do not include malicious changes. Whether you’re a teacher printing Wikipedia articles for the classroom, a student using them for research, or a publisher creating a DVD copy, you can pick the articles which have been checked for basic vandalism by trusted editors, instead of simply choosing the most recent version.

As a user of the German Wikipedia, you will notice that some articles have the following icon in the top right corner:

This icon indicates that the version you are looking at hasn’t been checked for vandalism yet. (If an older version that has been checked is available, this is indicated below the icon.)

The End of Immediacy?

While this configuration is simple enough, it should be noted that until about a couple of weeks ago, the German Wikipedia was using a different setup in which any change by a user without the permission to review changes for vandalism (which includes all unregistered users and relatively new ones) had to be reviewed before becoming the default version shown to readers. In other words, if you were not in the group with permission to review edits, your own changes did not become the “live version” until someone else looked at them.

This was a controversial change, as some users felt it significantly reduced the incentive for new contributors to start editing Wikipedia. So far, there has been limited analysis of the data collected during this experiment, which lasted from May until July 2008, and we hope to analyze the effects in greater detail over the coming weeks. (Some real-time statistics are available, thanks to André Karwath.)

Should changes to Wikipedia by new and unregistered users be reviewed before becoming the default shown to readers? There might be a middle ground solution: On most articles, changes would continue to be applied immediately, under the assumption that the benefit of radically open collaboration is greater than the risk. But, on a subset of pages, changes by unregistered and new users would have to be reviewed before becoming visible. This subset could consist of articles which are frequently the target of vandalism, such as the biography of the US President, but it could also include those pages which have reached a very high standard of quality as determined by the Wikipedia community. In other words, when the drawbacks of radical openness outweigh the risks, editing would be throttled.

This would, in fact, represent an opening up of Wikipedia rather than a closing down, as many of the affected pages are currently “semi-protected”, meaning that they cannot be edited at all by new and unregistered users due to the perceived risks of malicious edits. Being able to make changes that do not immediately become visible is surely preferable to not being able to make changes at all.

What’s next?

The Wikimedia Foundation has authorized all Wikimedia project communities to conduct experiments with FlaggedRevs through a process of self-organization. The process by which a Wikimedia community (e.g. the French Wikipedia, the Russian Wikibooks, etc.) can request the FlaggedRevs extension to be enabled is open and transparent. As the process unfolds, we will try to support the communities by collecting data about the use of the extension. Depending on our findings, we may eventually make a simple configuration of FlaggedRevs the default for all wikis.

There are other potential future uses of FlaggedRevs:

• Use for identification of article versions which meet standards of accuracy and quality as determined by experts. Potentially, FlaggedRevs could interface with external expert communities (such as universities or expert-driven encyclopedia projects like the Encyclopedia of Life) to identify versions of Wikipedia articles which meet scholarly standards of quality.
• Use for identification of article versions which meet internally defined standards of quality beyond the simple check for vandalism. The original German Wikipedia proposal for FlaggedRevs includes a more in-depth community quality review stage, which is still being discussed. A simple way to tie into community review mechanisms would be to use FlaggedRevs to “tag” versions which have passed through processes like “Featured Article Candidates“.
• Use to collect basic reader feedback on articles. Asking our readers whether information in Wikipedia articles is useful to them, and whether it meets their quality standards, could be a good way to track reader satisfaction over time. The lead developer of the FlaggedRevs extension, Aaron Schulz, is currently implementing such reader feedback tools.

The development of this technology represents the commitment of the international Wikimedia community to achieving the highest possible standards of quality in all our projects. In particular, the German Wikipedia community and the German chapter have been leaders and pioneers in this process. Philipp Birken from the German chapter gave a compelling presentation at the recent Wikimania on this very topic.

We welcome your feedback in making this technology more useful. An English demo version is set up in the Wikimedia Labs.

Erik Möller, Deputy Director