We’ve received many complaints about strange behavior on various wikis we host starting last night. These problems were directly related to an attempted deployment.

A bit of background about the 1.17 release:

• In Oct 2010 we committed to more frequent releases in response to community requests.
• Simultaneously, we committed to cutting through the backlog of code review requests from the community. As of this writing, the Code Review Team we formed has reduced the backlog of over 1400 un-reviewed core revisions down to zero in the 1.17 branch, as well as dispatching roughly 4000 other revisions in extensions (figuring out which ones we needed to review, and reviewing the important revisions there, too).
• 1.17 was an omnibus collection of fixes, including a large number of patches which had been waiting for review for a long time. The Foundation’s big contribution to the release was the ResourceLoader, a piece of MediaWiki infrastructure that allows for on-demand loading of JavaScript. Many other incremental improvements were made in how MediaWiki parses and caches pages and page fragments.

As is our usual practice, we review all code before trying to deploy it This practice has generally been good enough in the past that we have been able to quickly address anything we don’t catch in review within the first few minutes of deployment. The 1.17 release process has been longer than we would have liked, which has meant more code to review, and more likelihood for accumulating a critical mass of problems that would cause us to abort a deployment.

Our preparation for deployment uncovered a few issues, including a schema change, an update to the latest version of the diff utility and various other small issues which were discovered during the initial deployment to test.wikipedia.org. Pushing to test.wikipedia.org turns out to have been hugely useful, and in future we will take it as a lesson learned that any large deployment must successfully deploy to test.wikipedia.org at least 24 hours prior to general deployment.

When we finally deployed last night, our Apaches started complaining pretty much immediately. We rolled back to the previous version, worked on debugging and thought we had a suitable fix. We attempted deployment again but found the same issue very quickly. What we discovered was that our cache miss rate went from roughly 22% with the old version of the software (1.16) to about 45% with 1.17. The higher miss rate increased the load on our Apaches to the point where they couldn’t keep up, at which point they start behaving unpredictably. This can cause cascading failures (for example, caching bad data served by overloaded Apaches), and can result in strange layout problems and other issues that many people witnessed today.

By the way, whenever we do a large deployment, a number of WMF staff and community developers meet online to work through any issues that might arise. We schedule deployments late at night in the US to take advantage of lulls in request traffic, so everybody is working late. By the second failure, these people had been awake for many hours and we started to be concerned about their ability to work efficiently on little sleep, so I vetoed further attempts at deployment today.

We are currently combing the logs for further clues about how to mitigate risks of a similar outcome when we next attempt to deploy 1.17, which most likely won’t happen until later this week (at the earliest). We’re are also closely investigating the check-ins related to parsing and caching, and evaluating our profiling data. We plan to regroup tomorrow, decide how confident we are in the fixes we are able to implement in the past 24 hours, and make a decision as to when we should target to deploy.

Today at 20:00 UTC we saw a traffic surge on our load balancing and caching infrastructure, resulting in intermittent outages in Wikipedia service worldwide. This was due to a complex interaction of factors, including issues in our Amsterdam caching center and the Fundraiser launch, which has generated much more than expected interest today. We switched all traffic to Tampa, which experienced service problems due to high traffic and the additional load. Currently service is fully recovered worldwide, and we are continuing to closely monitor all systems.

Danese Cooper
CTO, Wikimedia Foundation

Here are results from Hack-A-Ton DC:

-Number of WMF Staff attending: 16

-Number of non-staff attending: 11

-Last-minute cancellations: 2

-Number of local developers attending: 8

-Number of “Fix-me’s” that were closed as resolved: 44 (roughly 30% of the queue)

-Number of Ops RT issues that were closed as resolved: 5

-Most Fixes (prize winners): 1st: Roan Kattouw, 2nd: Siebrand Mazeland, 3rd: Chad Horohoe. Honorable mention to Bryan Tong Minh who came in 4th.

-Furthest Traveled: 1st: Naren Datha, 2nd Tim Starling, 3rd Niklas Laxström

-Special Thank You (for setting up the MeetUp): Katie Filbert:aude

-Community and Engineering Discussions included:
a) Demo of WikiBhasha by Naren Datha of Microsoft,
b) Presentation of Test Suite methods by Rob Lanphier and Trevor Parscal, Priyanka Dhanda and Markus Glaser
c) Discussion of how to resolve Code Review / Deployment backlog by pretty much everybody
d) Demo and discussion of Sentence Level Editing by Jan Pavle Posma
e) Discussion of Pending Changes roadmap by Rob Lanphier, Alolita Sharma and team
f) Discussion of Resource Loader by Trevor Parscal and Roan Kattouw
g) Discussion of the future of WMF Release Engineering by Tim Starling, Rob Lanphier and team
All discussions were followed by prototyping or sprints on that topic

-Number of people attending the WikiMeetUp Saturday night: 52

Danese Cooper, Chief Technology Officer

Picture of most of the hackers sporting their new tee-shirts:

Nearly everybody...outside for a change.

So far we have 30 people gathered for a weekend of MediaWiki hacking (and we have room for more, so…come on down! http://www.mediawiki.org/wiki/Hack-A-Ton_DC ). The good folks at the Embassy Suites Tyson’s Corner have been very helpful (and we’ve not taken down the wifi yet).

See below for some snaps of the action. We’ll take a Group Picture outside in our Hack-a-Thon shirts for later posting.

Danese Cooper, Chief Technology Officer

Concentration

Naren Datha presents on WikiBhasha

The Hacking Room (partial view)

Folks over at Microsoft Research have been thinking about ways to improve content translation between instances of Wikipedia.  For example, today the largest collection of articles is at English Wikipedia (more than 3,000,000).  Compare that number with the collection at Hindi Wikipedia (which as of July 31 of this year had 55716).  One proven way to increase the articles in Hindi is machine translation, but such translations still need human review and often subtle editing to make them elegantly readable.

Enter WikiBhasha, formerly known as WikiBABEL, which launches today as both a MediaWiki extension project and a bookmarklet.  WikiBhasha takes content from a targeted Wikipedia page and displays a machine translation to a second language side-by-side.  Users can edit, add to or delete the translated content, preview their work and then submit it to the second language Wikipedia.

What’s especially interesting to me about this project is the fact that its author, researcher A. Kumaran, has tirelessly persuaded Microsoft to allow him to open source the client.  The code has been checked into the MediaWiki code tree under the Apache License 2.0, which means that the powerful side-by-side editing tools developed by Mr. Kumaran can potentially be used in other MediaWiki projects.  I’m very pleased to see Microsoft take this step, and I hope you will join me in welcoming WikiBhasha.

Danese Cooper, Chief Technical Officer

Around 18:00 UTC today, all Wikimedia projects experienced an unplanned outage caused by a cascade of events originating with the Image Scalers and eventually spreading through our web servers and load balancers due to an apparent bug in PyBal code. Situation was remedied by restarting key servers and rebalancing the load between subsystems. Full services availability was restored at 19:30 UTC.

Please join me in congratulating Mark Bergsma on his promotion last week to Operations Engineering Programs Manager.

Mark has been a volunteer since 2004, and a paid Network Engineer on our team since August 2006. He’s been helping us with our extreme scaling issues (by debugging and tuning our Squid setup, creating our Netherlands caching center, and generally developing our network strategy) since the very beginning. For some time now Mark has been unofficially in charge of managing the entire Ops Team’s deliverables including designing and implementing our new Primary Data Center in Ashburn, Virginia, and the other Ops activities mentioned here.

Mark has expressed an interest in gaining some experience with people management skills as a logical next step in his career, and to that end we will gradually add direct reports under Mark over the next year, starting with the Data Center Ops crew. He will continue to report to me until we hire a Director of Technical Operations.

I know you will do all you can to support Mark in his new role.

Danese Cooper
CTO, Wikimedia Foundation

Once again in 2010, Wikimedia Foundation is participating in Google Summer of Code.  I’m happy to announce that we’ve selected six students to participate this summer:

• Extension management platform
  Student: Jeroen De Dauw
  Mentor: Brion Vibber
  Goal: Creating an awesome extension management platform for MediaWiki, facilitating the installation, updating, removal and configuration of extensions. (student application)
• Improve metadata support
  Student: Brian Wolff
  Mentor: Chad Horohoe
  Goal: Improve metadata support for uploaded media in mediawiki by displaying embedded IPTC and XMP metadata (student application)
• General RDF export/import in Semantic MediaWiki
  Student: Samuel Lampa
  Mentor: Denny Vrandecic
  Goal: Extend the import/export functionality of Semantic MediaWiki (SMW) to allow also full, general RDF import. (student application)
• Javascript overhaul of Semantic MediaWiki
  Student: Sanyam Goyal
  Mentor: Yaron Koren
  Goal: Improve and extend the Javascript for Semantic MediaWiki and some of its spinoff extensions, most notably Semantic Forms – this would include transferring over much of the Javascript to use the jQuery library, which is now becoming a MediaWiki standard. (student application)
• Wikisource Legal Tool
  Student: Stephen LaPorte
  Mentor: Ariel Glenn
  Goal: Creating a tool to format judicial decisions, legal scholarship, and statutes for Wikisource. (student application)
• Reasonably efficient interwiki template transclusion
  Student: Peter Potrowl
  Mentor: Roan Kattouw
  Goal: The aim is to allow MediaWiki users to insert (transclude) templates from a wiki to another on WikiMedia Foundation (WMF) wikis (Wikipedia, Wikimedia Commons, etc.). (student application)

We had an exceptional set of really great proposals this year, and an engaged mentor group helping with the selection process. It was both wonderful to have so many choices, and really sad that we couldn’t pick them all, but in the end, we had to narrow the list down. Our six slots represent 100% growth from previous years’ Summer of Code engagements and that’s a pretty exciting stretch.

To the students that weren’t selected: do know that we were inspired by the quality level of all of the proposals, and we had to turn down some really exceptional proposals. Please don’t be discouraged, and do consider us next year!

To the students selected: congratulations! Welcome aboard! We really look forward to working with you to make sure you are successful and have a great time in the process.

To everyone volunteering as a mentor who helped with the selection process: thank you for your effort and dedication! There was a lot to sort through, but I think we can all feel great that we have a group of very capable students on the case this year thanks to your work.

More detailed info available here.

Towards the Fun!