December 2010 WMF Engineering Update

Welcome to the December monthly report from WMF Engineering! As always, we’re reporting on what we’ve been working on and what’s coming up. In November, our more visible work involved launching the Fundraiser and the Upload Wizard on commons. Behind the scenes, we worked on the next iteration of Article Feedback, continued to improve our infrastructure (e.g. monitoring, media storage, backups, analytics infrastructure, credit card handling) and continued to chip away at our code review backlog. We continue to hire at a rapid pace, looking to fill many different roles. More below….

Operations

Virginia Data Center – Setting up a world-class primary data center for Wikimedia Foundation websites

• Status: Data center has been selected and we’re signing a lease. We are also procuring equipment, to start the initial buildout in January.
• Program manager: Mark Bergsma

Media Storage – Re-working our media storage architecture to accommodate expected increase in media uploads.

• Status: We just hired Russ Nelson to investigate media storage solutions to replace the current NFS-based (and non-scalable, non-redundant) image server.

• Program manager: Mark Bergsma

Monitoring – Enhancing both Operations and public monitoring to a) notice potential outages sooner, b) increase transparency to the community, c) support progress tracking required in the 5-year plan.

• Status: We now have text message notification of downtime on critical services. We’re now turning our attention to implementing Watchmouse, which will help us spot trends and provide a public status page. We plan to report separately on our site reliability on a periodic basis.
• Program manager: Mark Bergsma

Virtualization cluster – more easily deploy temporary machines for testing and experimentation. This cluster is intended for use not just by WMF staff, but will be available to volunteers working on important projects as capacity allows.

• Status: We’re in the process of getting hardware for this cluster, and we hope to have the system available by mid-January.
• Program manager: Mark Bergsma

Backups – Improve backup coverage of Wikimedia-hosted data

• Status: We have a new backup system implemented for internal data using Amanda. We’re turning our attention to offsite backups in Amsterdam of all critical data. The concurrent buildout of the new primary data center will provide additional backup.
• Program manager: Mark Bergsma

Content Quality Tools

Article Feedback – Feature to collaboratively assess article quality and incorporate reader ratings on Wikipedia.

• Status: In development. The initial version (phase 1) has been in production since the end of September on English Wikipedia. Phase 2, focusing on usability improvements, performance improvements and wider deployment, is slated for release in January 2011.
• Program manager: Alolita Sharma

Pending Changes – Pending Changes is a new review feature deployed to English Wikipedia, which allows changes made by anonymous and new users to be reviewed before they appear as the primary version of an article.

• Status: We deployed an update for Pending Changes on November 23, with improvements to diff page viewing speed and small user interface improvements. We’re currently awaiting input from the English Wikipedia community about the future of this feature for en.wikipedia.org, while implementing minor bugfixes and continued incremental improvements.
• Program manager: Rob Lanphier

Threaded Discussions

Liquid Threads – LiquidThreads is an extension that brings threaded discussions capabilities to Wikimedia projects and MediaWiki.

• Status: Development on hold till developer availability in January.
• Program manager: Alolita Sharma

Multimedia Tools

Upload wizard – The upload wizard is an extension for MediaWiki that provides an easier way of uploading files to Wikimedia Commons, the media library associated with Wikipedia.

• Status: This feature has been in development for the past year, funded by the Ford Foundation. We have deployed this on Nov 30th on Wikimedia Commons.
• Program manager: Alolita Sharma

Media Projects – improved media handling and key infrastructure support tools. Many media components are developed in an open source development partnership with Kaltura, including Metavid, MwEmbed, and the Video Editor.

• Status: In development. Look for more next month.
• Program manager: Alolita Sharma

MediaWiki Infrastructure

Resource loader – The resource loader aims to improve the load times for JavaScript and CSS in MediaWiki. This will enable faster loading of the Vector skin, media extensions, and anything else that makes extensive use of Javascript and CSS.

• Status: Integration and testing in progress. We hope to have this ready for deployment in January, so that it can be part of a MediaWiki release sometime after that.
• Program manager: Alolita Sharma

General Engineering

Analytics Revamp – Incorporate a web analytics solution that can help the Wikimedia movement understand how Wikimedia sites are used. This project ncludes reworking our primary system (udp2log) as well as augmenting it with other systems (like Open Web Analytics)

• Status: We are feature complete on the implementation of Open Web Analytics, and we’re awaiting provisioning of production hardware. We have an update of udp2log in development that supports multicast, which will allow us to scale out to multiple logging servers.
• Program managers: Rob Lanphier & Tomasz Finc

Test framework deployment – Build an automated test environment for MediaWiki using CruiseControl, Selenium, and PHPUnit.

• Status: We’ve engaged Calcey Solutions to build out new Selenium tests that integrate into our testing framework.
• Program manager: Rob Lanphier

Code review – improving the way we provide code reviews for MediaWiki

• Status: Our team of code reviewers continues to make headway on the backlog of outstanding checkins in “new” status. We peaked at 1400 unreviewed checkins back in September, and we’re now under 800, putting us on track to be through the backlog sometime in February or March (assuming the current pace). We are now working to accelerate this further. See the statistics.
• Program manager: Rob Lanphier

Technical Documentation – Improve our technical documentation by making small, incremental improvements to the docs and docs process.

• Status: – Zak Greant is currently pairing up with many of the core committers in documentation sessions.
• Program Managers: Rob Lanphier / Zak Greant

wmsync – Replace our current deployment tools (e.g. “scap”) with more robust software.

• Status: Tim Starling and Mark Bergsma are in the very early stages of collaborating on the design of this, with some prototype software written in Python.
• Program Managers: Rob Lanphier

Fundraising

2010 Fundraiser – The engineering tasks necessary to run a successful fundraiser, with sub-projects involving fraud prevention, CentralNotice, and the analytics upgrade.

• Status: The annual fundraising campaign launched November 12, with the development team providing support, upkeep, and analytics for the fundraising infrastructure. We have developed functionality for the Foundation and chapters to quickly create and test landing pages. Statistics for all fundraising years is available.
• Program manager: Tomasz Finc

Credit card server upgrade – Upgrading our current payments infrastructure to support 1-click donations.

• Status: Done. We expanded our existing payments infrastructure from one layer to multiple caching layers so that we can allow fundraising banners to link directly to the payment form (we previously required two steps so as to minimize SSL load). We made use of Squid, nginx and haproxy to make this work. We did a lot of research on the requirements for running a large amount of SSL/TLS traffic on the cluster which should be useful for other applications of SSL.
• Program manager: Tomasz Finc

Mobile

Mobile site rewrite – Porting our existing gateway for easier support, development and participation

• Stats: Still in community discussions and pending our hire of a mobile engineer (see Hiring below)
• Program manager: Tomasz Finc

Offline

Offline – better support for offline reading of Wikimedia projects

• Status: We’re currently in the process of putting together project plans for openZim support for collections extension, as well as improvement to offline Wikipedia reader Kiwix
• Program manager: Tomasz Finc

Internal support

We have a number of small projects of note:

• Working with our HR department on evaluating HR management software evaluation (Project Manager: Alolita Sharma)
• Evaluating online stores (Project Manager: Tomasz Finc)

Hiring

We have a lot of hiring coming up before the end of the year. Job descriptions are already posted for the following:

• Performance Engineer
• Software Developer (Features)
• Software Developer (Mobile)
• Data Analyst
• Bugmeister

Look for an update on the Director of Technical Operations soon.
In addition, we hope to post the following positions over the next few months:

• Senior QA Engineer
• Release Engineer
• Technical Writer
• DBA/Storage Engineer (contractor)
• Network Engineer (contractor)
• Volunteer Development Coordinator

(Note: all of these positions may change as Foundation requirements evolve.)

Misc

Data summit – We had planned to have a data summit in October, but that meeting has been postponed to early 2011. The goal is to have a working meeting to explore new data concepts in MediaWiki such as enhancing structured data and analytics.