September WMF Engineering Update

The Wikimedia Foundation Engineering staff has grown quite a bit over the past year, which has made it a lot harder for everyone to keep track of what we’re all working on. In an effort to make things a little clearer, we plan to report monthly on all of our active efforts, and maintain information pages on all of our active projects. Note that this isn’t (yet) a complete list of everything that the Wikimedia Foundation engineering team is up to, but we plan to make this increasingly comprehensive and more organized as we get better at putting together these reports. Here is a full list of projects.

You’ll see that each of these areas has a program manager assigned to the area. That’s the person who is responsible for coordinating the activity in that area, and someone from whom you can expect to get more detailed updates.  More below the fold…

Operations

Virginia Data Center – Setting up a world-class primary data center for Wikimedia Foundation properties.

  • Status: We’re in the final selection phase for which facility will house our new primary data center in the Ashburn, Virginia area.
  • Program manager: Danese

Media Storage – Re-vamping our media storage architecture to accomodate expected increase in media uploads.

  • Status: We currently use Solaris/ZFS as the file system for media storage. Due to the rollout of our media-related projects (see “Multimedia tools” below) which have the potential to increase the load on our media storage infrastructure, we’re currently evaluating whether we are going to stay on ZFS, as well as what sort of infrastructure we need to implement in concert with whatever file system implementation we choose. This project will try to look at the whole strategy to design / implement a solution that will scale sufficiently for the next couple of years of projected growth.
  • Program manager: Danese

Monitoring – Enhancing both ops and public monitoring to a) notice potential outages sooner, b) increase transparency to the community, c) support progress tracking required in the 5-year plan.

  • Status: We use Nagios for systems/load monitoring, but we haven’t taken the time to tune its alert throwing to be really useful to us. We need to increase tooling to better monitor performance metrics such as page load time in target markets (such as India).
  • Program manager: Danese

Content Quality Tools

Article assessment – Working on feature to collaboratively assess article quality and incorporate reader ratings on Wikipedia

  • Status: We’re in the beginning phase of this project, figuring out requirements and generally determining the scope of our near-term and long-term efforts in this area. We are currently working on a pilot rating system which will be available as part of the Public Policy pilot program in late September.
  • Program manager: Alolita

Pending changes enwiki trial Pending Changes is a new review feature recently deployed to en.wikipedia.org, which allows changes made by anonymous and new users be reviewed before they appear as the primary version of an article.

  • Status: The official trial period has ended, with a straw poll now underway. Nimish Gautam and Devin Finzer put together some helpful statistics that we hope helps everyone how the feature performs on a per-article basis. Howie has provided some additional analysis to help interpret the numbers. Chad Horohoe has done some work on diagnosing and fixing some lingering performance issues with the feature as his schedule allows. Aaron Schulz is helping out when and where he can as he tackles other obligations, generally advising the rest of us on many aspects of the system.
  • Program manager: RobLa

Threaded discussions

Liquid Threads – LiquidThreads is an extension that brings threaded discussions capabilities to Wikimedia projects and MediaWiki.

  • Status: Most of the back-end work is now complete. We are currently focusing on user experience improvements necessary for much wider deployment. Some of our latest design work can be found here: http://www.mediawiki.org/wiki/LiquidThreads/Redesign
  • Program manager: Alolita

Multimedia tools

Upload wizard – The upload wizard is an extension for MediaWiki providing an easier way of uploading files to Wikimedia Commons, the media library associated with Wikipedia.

Add media wizard – The Add-media wizard is a gadget to facilitate the insertion of media files into wiki pages. Its development is supported by Kaltura.

  • Status: This tool was originally released as a gadget on a test server. It’s currently being adapted to run as an officially supported extension and be better integrated into MediaWiki. This effort will be assisted by the deployment of the Resource Loader (see below).
  • Program manager: Alolita

MediaWiki Infrastructure

Resource loader – The resource loader aims to improve the load times for JavaScript and CSS components on any wiki page.

  • Status: Trevor and Roan are busy implementing this feature, with hopeful completion sometime in the next month or so.
  • Program manager: Alolita

Central Notice – CentralNotice is a banner system used for global messaging across Wikimedia projects.

  • Status: We’re revamping the CentralNotice extension to make it easier to add, manage, and test new banners and campaigns. We’re also looking into including new functionality like geo-location and tightly coupling in analytics to improve our decision making. We’re not only looking to make this tool more usable for fundraising but also simple and broad enough to benefit the Wikimedia community as a whole.
    Ryan Kaldari recently finished up our first phase of making input simpler. The interface has gotten a huge face lift and is quickly approaching the discussed mockups at http://meta.wikimedia.org/wiki/CentralNotice_upgrades . We’ve tried to focus on not overwhelming our users and have chosen to collapse, hide and/or remove certain components so that banner input is simpler.
    In our second phase, we’ll be looking for volunteers to help test the new geo-location functionality. We’re also working on a better testing infrastructure for our CentralNotice banners.
  • Program manager: Tomasz

Analytics Revamp – Incorporate an analytics solution that can grow and answer the questions that the Wikimedia movement has.

  • Status: We are evaluating several possible analytics frameworks such as Open Web Analytics as a supplement or even replacement for our homegrown system(s), based on therecommendations from the Strategy task force. We plan to make a decision soon about the system we will use for (at least) this year’s fundraiser, but with an eye toward deploying a more generally useful system.
  • Program managers: RobLa & Tomasz

Software Quality Infrastructure

Selenium deployment – Building an automated browser testing environment for MediaWiki.

  • Status: Markus Glaser wrote the original set of tests using the Selenium web application testing framework. Ryan Lane has set up a cluster of machines dedicated to Selenium testing. Priyanka Dhanda and Ryan have been working to refine the requirements and generally get us to the point where we can build out a large suite of automated tests for MediaWiki, and Mark Hershberger is starting to figure out how to drive automated runs of Selenium and PHPUnit tests using CruiseControl. A small group has started to meet regularly to plan a more coordinated push in this area. Ping any one of us on IRC or the mailing list if you’re interested in chipping in!
  • Program manager: RobLa

Fundraising

Fraud Prevention – This project will focus on integrating new fraud prevention schemes within our credit card donation pipeline.

  • Status: We’ve wrapped up our pilot phase of this project, developing in-house solutions along with adopting industry standard practices to safeguard our donors, payment processors, and local systems. We’ve now incorporated many improvements to our credit processing pipeline.
  • Program manager: Tomasz

CiviCRM Upgrade – Upgrading from our heavily customized CiviCRMv2 install to a mostly stock CiviCRMv3 install

  • Status: Upgrade complete!
  • Program manager: Tomasz

Misc

Google Summer of Code – Several projects from students funded by Google.

  • Projects:
    • Extension management platform (Student: Jeroen De Dauw, Mentor: Brion Vibber)
    • Improve metadata support (Student: Brian Wolff, Mentor: Chad Horohoe)
    • General RDF export/import in Semantic MediaWiki (Student: Samuel Lampa, Mentor: Denny Vrandecic)
    • Javascript overhaul of Semantic MediaWiki (Student: Sanyam Goyal, Mentor: Yaron Koren)
    • Wikisource Legal Tool (Student: Stephen LaPorte, Mentor: Ariel Glenn)
    • Reasonably efficient interwiki template transclusion (Student: Peter Potrowl, Mentor: Roan Kattouw)
  • Status: This has turned out to be a very successful year for us. Though not all projects were finished completely as specified, all were completed to a sufficient degree that we felt very comfortable passing all of the students. While there’s no guarantee that everything here will get beyond the proof-of-concept stage (though at least a couple already are), there’s a lot of promising work to look forward to.
  • Program manager: RobLa

Process improvement – Increase transparency and generally organize Wikimedia Foundation’s engineering efforts more efficiently

  • Status: We’re currently figuring out the general practices that don’t involve new tools (such as this blog post, and the wiki pages), as well as figuring out what tools will help us work best together with each other and the larger community. We’re also working to figure out what ways our existing tools (such as Bugzilla) can be configured to make it clear the order that we plan to tackle tasks clear and obvious to anyone who wants to find out.
  • Program manager: RobLa

If you read this far, thanks for sticking with us! We hope you found this useful. Please let us know what we can do to make this more useful for you.

7 Show

7 Comments on September WMF Engineering Update

Lewis Cawte 4 years

Great post, keep them, and the changes to Wikimedia sites coming!

robla 4 years

Jarry1250, one way to stay on the leading edge: we’ll be working on the draft blog post publicly. The October draft is here: http://www.mediawiki.org/wiki/WMF_Engineering_Overview_October_2010

I may also pick a time to have a public work session which is sort of like an office hour, but with an expectation that people are there to help build out the update. Details TBD.

Guillaume Paumier 4 years

Jarry1250 :
Would it be too much to ask to have less but more often? (Or more and more often :P )

We’re planning to keep monthly general engineering updates on the tech blog, but you should see posts about specific projects more often.

Jarry1250 4 years

Hurray, says the guy responsible for news reporting on such topics for the Signpost. Would it be too much to ask to have less but more often? (Or more and more often :P ) A step in the right direction, nevertheless.

Stu West 4 years

Yes this is a great update. Thanks especially for the links to the current pages for the different projects. I’m looking forward to reading each month!

Dror Snir 4 years

Indeed, a great update! Thank you so very much. Lots of things to follow-up on (and pray for a successful implementation of).

Derk-Jan Hartman 4 years

Wow, very nice update. This posting was very welcome.

Leave a Reply

Your email address will not be published. Required fields are marked *