Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Posts by Rob Lanphier

Request for proposals: MediaWiki release management

MediaWiki, the software that powers most of the Wikimedia movement, is an amazing piece of technology. It brings the power of a wiki-world to millions of people. Not only those who are amongst the 500 million who visit a Wikimedia movement site each month, but also those who participate on one of the countless other wikis it powers.

MediaWiki is being used in all kinds of environments, from internal and private corporate wikis to other Free Culture wikis. That is, of course, the great benefit behind Free and Open Source Software; the software can be modified and used in new situations the original authors didn’t necessarily expect.

Because the Wikimedia Foundation wants the MediaWiki project to be as healthy as possible, and also address the needs of as many different constituencies as possible, the Foundation invests a lot of time and effort into ensuring the entire MediaWiki community feels empowered, not just those that happen to have an @wikimedia.org email address. You can see this effort most notably from the Engineering Community Team and the efforts especially around volunteer coordination and outreach.

To encourage further outside investment in MediaWiki, we are opening a Request for Proposals (RFP) (PDF) for the release management of MediaWiki. The long-term goal of this effort is to jump-start these activities as community-supported functions, thus encouraging widespread leadership in the future of MediaWiki.

The process for this RFP is a community-involved one. There is a three-week period for organizations to prepare and submit their proposals, after which the community can comment on and ask questions of the proposers. The Wikimedia Foundation will take all of this feedback into account when making the final decision for who will lead the release management of MediaWiki for the next year.

With this, the future of MediaWiki looks bright, and we’re excited to see where this will lead us!

Greg Grossmeier, Release Manager
Rob Lanphier, Director of Platform Engineering
Wikimedia Foundation

Breaking through walls of text: How we will create a richer Wikimedia experience

Wikimedia consists of many projects, Wikipedia most notable among them. However, the name “Wikimedia” suggests a world beyond text. Indeed, Wikimedia Commons, our repository of freely-licensed media files, already contains more than 16 million images, sound files, and videos.

Well, mostly images. Right now, there are fewer than 30,000 video files, and fewer than 170,000 audio files. And while Wikipedia articles are often richly illustrated, they still share the old-school feel of a print-based experience. Projects like Snow Fall by the New York Times show what an immersive reader experience can look like, with video elements prominently featured and blended into the core of the content. In contrast, Wikipedia articles rarely have videos, and if they do, those videos are usually very short and included at the bottom of the article.

Of course, well-written text forms the foundation of most high quality educational content.  Text is versatile, adaptable, accessible, efficient, and relatively easy to collaborate on.  It will form the core of the Wikimedia experience for a long time to come. Still, we can greatly improve the educational value of our sites by empowering everyone to share media, collaborate on improving that media, and using that media well throughout our sites.

In the last three years, Wikimedia has seen some very significant multimedia developments:

  • The Wikimedia movement has launched successful photo contests and competitions, notably the “Wiki Loves Monuments” competition, which was recognized as the world’s largest photo competition by the Guinness Book of Records. In the 2012 competition, more than 350,000 photos were taken by volunteers. It was organized by Wikimedia chapters and volunteers in 33 countries (see jury report).
  • Wikimedia chapters and volunteers have also formed partnerships into the cultural sector (e.g. museums, galleries, archives), resulting in hundreds of thousands of photographs, reproductions of paintings, and other media being made available on Wikimedia Commons.
  • Wikimedia Foundation has developed a number of enhancements and features focused on multimedia:
    • the Upload Wizard, an easy-to-use tool for uploading media files that’s been used to upload more than 2.2 million files to Wikimedia Commons;
    • upload features for the mobile web that make it easy to enrich any article requiring a photograph using a smartphone;
    • a new HTML5 video player with support for the open WebM video format and encoding of videos in multiple resolutions;
    • dedicated upload apps for iOS and Android are in development;
    • a feature to import photographs from Flickr (started as a Google Summer of Code project)
    • an experimental feature to upload files up to 500MB in size.

In combination, these efforts have already borne fruit. The number of contributors to Wikimedia Commons has increased significantly in the last 3 years.  In January 2010, only 13219 users had contributed at least one upload.  That number increased to 20161 users by January 2013.

At the same time, we haven’t invested enough. With the exception of the work of our mobile team, much of the above work has been done by one or two developers at a time, often in between other priorities or by engineers working as volunteers. There has never been a well-resourced team fully dedicated to multimedia engineering work at the Wikimedia Foundation. This is about to change.

The Wikimedia Foundation is hiring at least three engineers and additional product/design support to fully focus on improving the user experience for contributing, curating and reviewing multimedia. Right now, you can apply for the following positions:

Here are some of the key challenges for the new team:

  • further improvements to the upload experience. Contributing an image or video to an article while you’re editing should not require leaving the “edit mode” — it should be integrated with the editing process.
  • solidifying experimental features such as large file uploads;
  • improving transcoding features for video files to reduce the learning curve for video uploaders;
  • improving media search and discovery;
  • improving display of images, videos and sound files in Wikipedia articles, including a standard lightbox viewer for media embedded in an article and related media from Wikimedia Commons (building on some of the excellent submissions in our October 2011 Coding Challenge).

As we continue to provide new means for uploading media, we need to ensure that the Wikimedia community is empowered to curate and categorize the images. Curation includes removal of content that is out of scope or incorrectly licensed. To more effectively patrol content, the development of curation tools similar to the Page Curation feature developed for Wikipedia may become necessary.

Beyond Wikimedia’s category system, we will likely want to explore implementation of lightweight tagging systems, possibly in partnership with the Wikidata team.

As if this weren’t enough, the long term frontiers for multimedia include web-based editing of images, video and sounds, improvement for subtitle editing, browser-based audio recording features, and more.

In short, breaking through walls of text and creating a richer media experience for all our projects will keep the Wikimedia Foundation and the Wikimedia movement busy for many years to come. Please help us expand our library of freely-licensed educational media, and help us ensure it gets used effectively on the world’s fifth-most popular website.  Apply today.

Rob Lanphier, Director of Platform Engineering
Erik Möller, Deputy Director; Vice President of Engineering and Product Development

The MediaWiki Core group

This is the last in my series of introductory posts about Wikimedia Platform Engineering, focusing on the MediaWiki Core group.  This group is responsible for our sites’ stability, security, performance and architectural cleanliness.  This ends up translating into a lot of code review, along with infrastructure projects like disk-backed object cache, heterogeneous deployment, continuous integration, and performance-related work.  While it’s not a prerequisite, everyone on this team started off as a volunteer developer.  The whole engineering organization has some level of responsibility for our code review process, but this group has more of a primary responsibility for it than most groups.  We have an open position in this group.

(more…)

Technical Liaison; Developer Relations (TL;DR)

Golden Gate Bridge seen from the Presidio in San Francisco 42

This is yet another post about what everyone does in Platform Engineering, this time focusing on the Technical Liaison; Developer Relations (TL;DR) group.

The TL;DR group is responsible for development community relations, ensuring a healthy relationship between the Wikimedia Foundation and our volunteer development community. This team is responsible for removing obstacles to effective volunteer participation, communicating about what we’re up to now, and patrolling for new opportunities for volunteers to get involved and new volunteers to involve.

While everyone in Engineering is responsible for those things to some extent, this team helps fortify our commitment to this. And, just like they prodded me into the last two posts, they’ve prodded me to make this post. Once again, the reason for posting this is twofold: 1) because it’s generally good that everyone knows what it is that the WMF invests in, and 2) because we’re hiring, and we (still) want to get the word out. (more…)

Data analytics at Wikimedia Foundation

This post is a follow-on to my previous post “What is Platform Engineering?” .  In this post, I’ll describe the history of our analytics work, talk about how we derive and distribute our statistics, and ask you to join us in building our platform.  Summary:  we’re hiring, and we want to tell you what a great opportunity this is.

Our Data Analytics team is responsible for building out our logging and data mining infrastructure, and for making Wikimedia-related statistics useful to other parts of the Foundation and the movement.  Up until fairly recently, Erik Zachte has been the main analytics person for Wikimedia (with support from many generalists here), working first as a volunteer building stats.wikimedia.org, then on behalf of Wikimedia Foundation starting in 2008.  It started off as a large number of detailed page view and editor statistics about all Wikimedia wikis, large and small, and has since been augmented to include various summary formats and visualizations.  As the movement has grown, it has played an increasingly important role in helping guide our investments.

MediaWiki 1.18 deployment today to all Wikimedia sites

As reported two weeks ago, we’re planning to deploy MediaWiki 1.18 to all wikis, starting today (Tuesday, October 4, 23:00-03:00 UTC). We have been running MediaWiki 1.18 on several wikis already, representing about 2% of our total traffic. A big thank you to the early adopter wikis; it was very helpful getting some real world testing prior to deploying to the other 98%.

Our deployment process works like this. During our four hour deployment window, we’ll be deploying to several wikis sequentially. Our tentative plan for today is deploying first to fr.wikipedia.org, then pl.wikipedia.org, then en.wikipedia.org, and then probably a few more sequentially before deploying to the rest in bulk.  At the end of our window, we will be stopping deployment, even if we’re not done (scheduling a followup window if needed).

To report issues in real time (especially during the deployment window), IRC is the best venue; please join us in #wikimedia-tech on Freenode (web access). For those of you that are comfortable with Bugzilla and other development tools, we would love your help with confirming issues and getting appropriate issues filed in our bug tracker. If you don’t feel comfortable using Bugzilla, you can leave a message on the talk page of our announcement on meta. Our developers can keep track of issues much better when you use Bugzilla, so filing it there makes it more likely your problem will be noticed and eventually addressed.

Thanks for your patience!

Rob Lanphier
Director of Platform Engineering

Update: October 5 05:14 UTC – we didn’t get as far as we wanted, but we deployed to fr.wikipedia.org, pl.wikipedia.org, en.wikipedia.org, and commons.wikimedia.org.  We’re planning to have one more deployment window in a little less than 18 hours (October 5, 23:00-October 6, 03:00 UTC) to deploy to the remaining wikis.

MediaWiki 1.18 is coming

[Update 2011-09-24: The initial test deployment and stage 1 have gone well, with only minor glitches that we've mostly cleaned up.  Stage 2 and 3 are currently on schedule.  We've decided to add incubator.wikimedia.org to the list of wikis we'll be deploying to, which is reflected below.]

MediaWiki 1.18 will soon be deployed to all Wikimedia sites, including Wikipedia. As you may know, MediaWiki is the wiki software developed by the Wikimedia community, and 1.18 is the upcoming version of the software that has been in development since December.

Thanks to the completion of the heterogeneous deployment project, we are now able to run different versions of MediaWiki concurrently on Wikimedia sites. This means that we don’t have to upgrade all sites at the same time any more, which should limit the problems we encounter.

The deployment is scheduled to happen in several stages, starting next week:

Wikis in Stage 1 and 2 may experience more issues, so we plan to focus our attention to those wikis during these periods, and be particularly responsive. If you’d like to help make sure we catch problems before we roll out to your wiki, please help us test, by trying out the test wiki starting Tuesday, and report the issues you find.

(more…)

What is “Platform Engineering”?

If you’ve been following this blog or other Wikimedia Foundation updates closely over the past year, you may have seen several references to the “Platform Engineering” group (nee “General Engineering”), which is the group I’ve been managing for the past year. I’d like to explain who we are, and what we’re doing. We always strive for transparency as a group, but one ulterior motive for this particular narrative is that we’re hiring (more on that in a bit), and we hope this helps people understand what we’re looking for.

(more…)

Welcome, Sumana Harihareswara, Volunteer Development Coordinator

I’m thrilled to welcome Sumana Harihareswara to WMF Engineering!  Sumana will be filling the role of Volunteer Development Coordinator.  We interviewed many great candidates for the role, but decided the role would be best served by WMF continuing to work with Sumana on a contract basis.  She’ll be working from her home in New York.Sumana started in a part-time capacity back in March coordinating our participation in Google Summer of Code, as well as helping plan WMF’s participation in the Berlin Developer meeting happening later this month.  Starting after the Berlin Developer meeting, she’ll be dedicating her working time to Foundation issues.

In addition to the specific initiative above, she’ll be recruiting and encouraging volunteers more generally.  In the near term, she’ll be evangelizing movement priorities within the development community, and working toward matching interested volunteers and organizations to important movement work.  She’ll be working with Bugmeister Mark Hershberger on bug triage and finding volunteers to test and fix MediaWiki.  She’ll also gather some baseline metrics about our volunteer and corporate communities to measure our progress against.  And she’ll be coordinating WMF development work in other open source communities as appropriate.  Her Open Source Bridge talk last year (“ The Second Step: HOWTO encourage open source work at for-profits”) is particularly relevant to this last task.

Sumana is currently an active contributor in the GNOME community, as a writer and editor for GNOME Journal, and recently led the marketing effort for GNOME 3.0.  She is also a blogger at GeekFeminism, and a longtime participant in open source communities.  She has worked at the GNOME Foundation, QuestionCopyright.org, Collabora, Fog Creek Software, and Salon.com, and contributed to the AltLaw, Empathy, Miro, and Zeitgeist open source projects.  She’s written a weekly newspaper column and has performed (and taught) stand-up comedy.

Sumana intends on communicating with the MediaWiki and Wikimedia communities in many ways: via IRC and mailing lists, conference calls, and frequent visits to WMF headquarters from New York City and to relevant conferences, both MediaWiki-related and not.  For example, she’ll be speaking again this year at Open Source Bridge, giving a talk titled “Learn Tech Management in 45 Minutes”.

If you’re interested in learning more, or dropping a comment on her talk page, Sumana’s user page on MediaWiki.org has much more information.

Welcome, Sumana!

–Rob Lanphier, Engineering Programs Manager for General Engineering

MediaWiki 1.16.4 security release

MediaWiki 1.16.4 is a second security release this week.  Shortly after previous release (1.16.3), Masato Kinugawa discovered that one of the XSS problems that the 1.16.3 release was designed to address hadn’t been fully addressed, and reported bug 28507.  As a consequence, Internet Explorer 6 users visiting a site running 1.16.3 will still be vulnerable to an XSS attack.  After more thorough testing (thanks Roan Kattouw!), we’re releasing 1.16.4.

Full details are in Tim Starling’s 1.16.4 release announcement.  Sorry for the inconvenience of a second release, and thank you everyone involved in getting this fixed!