What is Git?

Git is a distributed version control system originally developed by Linus Torvalds and others to manage the Linux kernel. In the past couple of years, it has taken off as a very robust and well-supported code repository. “Distributed” means that there is no central copy of the repository. With Subversion, Wikimedia’s servers host the repository and users commit their changes to it. In contrast, with Git, once you’ve cloned the repository, you have a fully functioning copy of the source code, with all the branches and tagged releases at your disposal.

Why switch?

Three major reasons:

To encourage participation: Since Git is distributed, it allows people to contribute with a much lower barrier to entry. Anyone will be able to clone the repository and make their own changes to keep track of them. And if you’ve got an account in our code review tool (Gerrit), you’ll be able to push changes for the wider community to review.

To fix our technical process: Subversion has technical flaws that make life difficult for developers. Notably, the implementation of branching is not very easy to use, and makes it hard to use “feature branches”. Our community is very distributed, with many parallel efforts and needs to integrate many different feature efforts, so we’d like to use feature branches more. Git branches are very easy to work with and merge between, which should make things easier for our development community.  (Several other large projects, such as Drupal and PostgreSQL, have made the same switch for similar reasons, and we’ve done our best to learn from their experiences.)

Some quotes from our community:

“I love git just because it allows me to commit locally (and offline).” – Guillaume Paumier

“[Y]ou can create commits locally and push them to the server later (great for working without wifi), you can tell it ‘save my work so I can go do something else now’ in one command, and it’ll allow us to review changes before they go into “trunk” (master)…. without human intervention in merging things into trunk. Gerrit automates this process.” – Roan Kattouw

And finally, to get improvements to users faster: with better branching and a more granular code review workflow that suits our needs better, plus our ongoing improvements to our automated testing infrastructure, we won’t have to wait months before deploying already-written features and bugfixes to Wikimedia sites.

We had years of discussion before we finally decided to switch, but now we can look forward to more flexibility and power in our engineering processes.

What are we doing?

We’ve now done almost all the back-end work of preparing our repository for the move and are in the final steps of preparation (details). We’ve also written explanations of the new workflow, the migration schedule, issues yet to be addressed, and other related topics. Right now, we’re asking people to stop creating any new extensions in Subversion right now, and to watch the wikitech-l mailing list for more updates.

What are the next steps?

Over the next two and a half weeks, the Git repository that contains MediaWiki core and extensions will be brought in step with Subversion, and at first it will be read-only (no one will be able to push changes). This will allow developers to start cloning it to their local machines and getting used to things.

For MediaWiki core and for extensions that the Wikimedia Foundation deploys on its wikis, the switchover is pencilled in for the weekend of March 3rd. We’ll do core first, and then extensions after, but hopefully all in the same weekend. After the successful migration, the Subversion repository (for the directories that have moved to Git, such as /trunk/phase3/) will be made read-only.

See the full schedule.

I develop for a Wikimedia project. Do I have to switch to Git?

Only two projects are affected immediately: the core of MediaWiki and the extensions that get deployed on Wikimedia Foundation projects.

So, if you work on an extension that the Wikimedia Foundation does not use, or on a non-MediaWiki project hosted at svn.wikimedia.org, you have more time to decide. Talk it over with your community and decide whether you would like to move to Git immediately, move to Git sometime over the next several months, or move to another hosting provider sometime before mid-2013. We would like to gradually migrate all projects currently on Wikimedia’s Subversion repository so that we can make all of svn.wikimedia.org read-only by the middle of 2013, and thus only have to support one source control infrastructure.

More details.

Will training and documentation be available? When?

Yes, we will provide training and documentation to help you use the new workflow. Check our Git page and its links now, and watch that space! There will be more documentation as well as some interactive training sessions before the big switchover in early March.

If you have any questions, please ask in #mediawiki on Freenode or on wikitech-l.  Thank you!

Chad Horohoe
Git migration lead
Platform Engineering department
Wikimedia Foundation

Many people worked on Wikimedia mobile and the SmartPhone software, others worked on MediaWiki and its extensions. Bugs got fixed and functionality got extended.

One of the surprises was two people working on the localisation for the Mongolian language. The inclusion of a web font that will support the Dzonka language is another.

Dzongkha is the official language of Bhutan and according to Ethnologue, the script used is either Tibetan script, Uchen style or the Tibetan script, Umed style. These scripts and styles are also used for the Tibetan language, it is not only Dzongkha that stands to benefit.

One of the highlights of the work on the SmartPhone app is support for scripts that are written from right to left, this is now “beta” functionality. The result of more people looking at the code was that several bugs received the attention needed to make them go away. Scrolling was one area that got attention; this results in a smoother user experience.

New input methods have been created for Punjabi transliteration and for an Gujarati input method to be included in Narayam. The continued collaboration with RedHat engineers ensures that our work benefits both MediaWiki and RedHat/Fedora. We do realise that there is still a lot to do and it is not only documentation. Additional work was done on the “visual on-screen keyboard” that was started at the previous hackathon in Pune, it still needs more testing and design work.

Thanks,
Gerard Meijssen
Internationalization / Localization outreach consultant

Wikimedia sites will gradually be upgraded to version 1.19 of MediaWiki over the second half of February 2012.

This article is available in other languages on mediawiki.org.

Wikimedia engineers are putting the final touches to the latest version of MediaWiki, the software that powers Wikipedia and its sister sites. This version, labeled “1.19wmf1″, will be deployed to Wikimedia sites in stages, starting next week.

We’ve recently set up a Beta cluster, replicating a selection of Wikimedia wikis, where Wikimedians have tested the new version and checked that it worked reasonably well with their local wiki’s specific customizations.

Things are looking good, and the current plan is to run the deployment in five stages between February 15th and March 1st, 2012. The schedule may change based on unexpected issues, so you should refer to the MediaWiki 1.19 roadmap for an up-to-date schedule of when your wiki will be affected.

Many new features and bug fixes brought by MediaWiki 1.19 are back-end, behind-the-scenes changes, for example infrastructure work to support our ongoing move to Swift as our media storage platform.

There are also more visible improvements, like better diff readability for colorblind people, and better support of the user’s gender and language in the interface. A list of all changes is available in the draft release notes.

Check JavaScript and Gadgets compatibility with ResourceLoader

A particular area of improvement in MediaWiki 1.19 relates to JavaScript. While most of the legacy site scripts, user scripts and gadgets will continue to work, it’s also possible that the new version be less forgiving of assumptions and errors in code. For example, faster load times may uncover errors in scripts that don’t explicitly declare the modules they’re using.

Furthermore, a new version of ResourceLoader will be deployed later this year, that will bring improvements specific to gadgets, but will require gadgets to be made compatible with ResourceLoader.

Gadgets maintainers are therefore strongly advised to start upgrading their scripts now, to avoid major disruption later. The migration guide to ResourceLoader is the main document for Gadget developers; a list of JavaScript deprecations and default modules are available as well.

You can also join the 2011 Resource Walker, an attempt to go through all Wikimedia wikis and update outdated JavaScript. An IRC workshop is planned to facilitate the process; more information will be posted later on this blog.

Moving towards transparent upgrades

As we move towards more frequent software upgrades, we expect them to be less and less painful — and ideally, at some point they’ll go so smoothly that users won’t even notice them, except for the new features that will appear. We’re not completely there yet, but we’ve made progress in the past year or so, and we’re committed to continue our efforts, both for the benefits of developers and users.

In the meantime, please bear with us if, despite our efforts, you encounter issues due to the upgrade; we’ll try and fix them as soon as we can. It’s not too late to visit the Beta cluster and report issues there or in our bug tracker. The more people test beforehand, the smoother the deployment should go.

Guillaume Paumier
Technical communications manager

Mobile Wikipedia readers in Brazil

As part of our commitment to provide free knowledge to everyone, the foundation has been redesigning our mobile platform (m.wikipedia.org and mobile.wikipedia.org) to enhance the reading experience and allow editing.  As a first step towards the redesign of the mobile gateway to better meet the needs of our users in the Global South, we conducted user experience research in India and Brazil among current and future users of Wikipedia mobile last summer.  We also carried out user experience research in the US to have a comparison with a mobile market which is more mature in terms of smartphone and 3G penetration, and has a more widespread adoption of tablets.

Our research in India and Brazil brought forth the following three opportunities with the greatest perceived impact for the mobile platform:

1. Improving our search:  Our research revealed that there was a need to provide search suggestions, autocomplete, autocorrect and other tools that ease typing and search burdens on mobile devices; support search in all language Wikipedias as well as allowing users to chose and switch between languages; incorporate transliteration tools for languages with fonts and characters that have poor mobile support; support and even enhance users’ existing habits to use Google to reach Wikipedia articles; and enable users to search within a Wikipedia page. We are happy to report that drawing from the research our mobile team has already implemented some of these opportunities like full page search, autocomplete  and inter-wiki links into our mobile beta site.
2. Optimizing our reading experience for mobile devices and generalized use.  Through our research, especially in India, we found that we were not redirecting a large breadth of devices in use to our mobile site. The mobile team quickly fixed this issue with the adoption of the open source library tera-WURFL for detecting mobile devices.  After speaking with respondents in India and Brazil, we found that there was a desire among users to modify or set one-time preferences for the display of images, the font size, and any element that affects page loading time and size. Similarly, there is an opportunity for allowing  preferences for language and navigation; the ability to watch or bookmark articles; or save content offline; offer content in more digestible pieces, or with quicker access (i.e. preview or easy access to the first paragraph, or a new “mobile summary”); search offline, i.e., while in transit or without a data plan; and generally follow expectations set by mobile web interactions and standards.  Some of these recommendations have been incorporated into our mobile product strategy.  Through this research we felt it was crucial to offer both an official iOS and Android app (which was officially released in January) that offers at minimum a simple and easy search and reading experience.
3. Using the mobile platform to both increase user engagement and awareness of features on Wikipedia as well as providing new opportunities for participation. The mobile site and potential apps provide many new pathways for both engagement, participation, and contribution.  At present, the mobile site can be used to build awareness around existing features on the site that current users are blind to (i.e. watchlists, accounts, editing, inter-language links, history); to provide features that make opening a Wikipedia account worth having, something that the majority of our participants do not currently see any reason to have; increase visibility of local language Wikipedias, especially in India since many English readers were not aware of the existence of Indic Wikipedias; prompt users to download an official app when possible; and interface with other web content on mobile devices (Google, news, entertainment, and sports content, for example “Wikitap”).  The contributions that showed the highest potential for adoption were adding photographs, “flagging” or “marking” something that needs to be edited, removing or marking vandalism, adding links, adding location or geodata, and potentially making small typing or formatting edits.
4. Mobile Editing. And finally, the mobile site can support the editing practice of existing editors by first offering those features in a mobile friendly format which are currently in high use on the site.  Those with the highest demand and potential are the “recent changes” page, which is consumed like an update feed or email; accessing watch lists; making reverts, especially with respect to vandalism; logging in and accessing account and user pages; and serving discussion pages and article histories.

If you are interested in reading about our research in India and Brazil in detail, we have compiled the insights in a report which is available in PDF and wiki format. You can also watch video highlights from the interviews and check out some photographs from the field work in India and Brazil.

Mani Pande, Head of Global Development Research

We’ve been able to manage the load for quite a while by using two servers with lots of local storage — (10 and 30TB), but we’re pushing against that limit and we would like a more fault-tolerant option. So, for the last few months, we have been working on replacing the infrastructure that holds all that data.

Our goal is to have a storage system that will allow us to scale more easily, and accept large collections of media from projects like Wiki Loves Monuments, and the U.S. National Archives’ donation of their collection of photographs by Ansel Adams.

After evaluating a number of options, we chose to pursue OpenStack Swift. Swift is a distributed object storage system with automatic replication, so that if one host has problems the requseted file is retrieved from another server with no interruption of service. Aside from meeting our needs around performance, reliability, and scalability, it is a good fit considering we are also using OpenStack products for Wikimedia Labs.

We have just completed the first milestone along the road to replacing our existing storage systems with Swift: all image thumbnails (scaled images such as a 320px version of a picture) are now stored on Swift. Our current production Swift cluster is made up of 4 back-end storage nodes with 22TB each and 2 front-end proxy nodes that handle user web requests. This new architecture provides us the scalability and reliability we need going forward.

Over the next few months we will build a second Swift cluster in our Virginia data center, then work on migrating all of the original media over to Swift as well. For more detail on the implementation and plan for Swift, you can read up on the documentation on Wikitech, ask questions in the comments below, or come and visit us in #wikimedia-tech on Freenode in IRC.

Ben Hartshorne
Operations
Wikimedia Foundation

• Tech support for the black-out protesting SOPA & PIPA;
• The successful San Francisco hackathon;
• The release of the official Wikipedia Android app;
• A new beta cluster for Wikimedians to test upcoming software before it’s deployed to production.

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Events

Recent events

• English Wikipedia anti-SOPA blackout — The engineering team supported this online protest by developing and deploying the blackout code and design, including the CongressLookup extension for helping people find and contact their representative, pulling from Sunlight Foundation APIs and other sources. The Operations team disabled editing during the 24 hour time period as agreed upon by the community, and helped keep other systems up and running, including the temporarily overloaded Wikimedia blog.

• San Francisco hackathon (21–22 January 2012, San Francisco, California, USA) — More than 90 participants learned and hacked during this outreach-focused developers week-end. Sumana Harihareswara and several other WMF staffers worked with the coworking venue pariSoma to organize the event. The teams of participants demonstrated more than a dozen projects. The demos, a speech by Erik Möller, and tutorials in mobile, Gadgets, and the web API were recorded and are available on Commons, along with photos of the event.

• October 2011 Coding Challenge — The winners of the coding challenge were announced. They include an Android app for uploading to Wikimedia Commons, a user script for surfacing pages with a lot of recent editing activity, and a user script for displaying relevant images in an article as a lightbox slideshow.

• Linux.conf.au — Trevor Parscal and Roan Kattouw visited Ballarat, Australia and presented Low-hanging Fruit vs. Micro-optimization, Creative Techniques for Loading Web Pages Faster, a talk about ResourceLoader. A recording is available.

Upcoming events

• Pune hackathon (10–12 February 2012, Pune, India) — Preparation began and registration continued for an outreach-focused developers week-end to take place in Pune, India, and led by Alolita Sharma. Approximately 70 participants are expected to learn and contribute at this event, focusing on the gadgets framework, mobile Wikimedia access, and language support (i18n/L10n).

• GLAMcamp DC (10–12 February 2012, Washington, D.C., USA) — The Foundation’s Ryan Kaldari and Asaf Bartov plan to attend the technical track of this GLAM conference. Engineers will work on mass upload and analytics functionality, which cultural institutions find useful in partnering with Wikimedia.

Personnel

Job openings

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

• Developers and engineers:
  • Senior Software Engineer Front-end
  • Interaction Designer
  • Systems Engineer (Data Analytics)
  • Software Developer (Back-end, Data Analytics)
  • Software Developer (Rich Text Editing, Features)
  • Software Developer (Front-end)
  • QA Lead
  • Software Developer (Mobile)
  • Software Security Engineer
  • Operations Engineer (Labs)

• Management & Product:
  • Director of Features Engineering

• Requests for proposals:
  • Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.
  • Mobile QA — Help us set up testing and automation processes for all Wikimedia Mobile projects.

Short news

• Andrew Otto joined the Platform engineering team as Software Developer for Analytics (announcement).
• Fabrice Florin joined the Product Development team as Product Manager for New Editor Engagement (announcement).
• Andre Engels joined the Mobile team as data analyst contractor (announcement).
• Howie Fung, previously Senior Product Manager, was promoted to Director of Product Development, a newly formed group within the Engineering department (announcement).
• Chris McMahon joined the Platform engineering team as Quality Assurance Lead Engineer (announcement).
• Neil Kandalgaonkar left the Wikimedia Foundation in January (announcement).

Operations

Site infrastructure

• Data Centers [?] — Work continued on building up the EQIAD datacenter in Virginia. We added a new bastion host, a ganglia server, new dataset servers and upgraded the database servers, with a new chained replication topology and heartbeat-based replication monitoring. We upgraded mailman and migrated it to a new server in EQIAD from ESAMS. We have also successfully tested the new thumbnail system and the text squid implementation that we’ll start rolling out fully in February. As for our Tampa datacenter, we have been upgrading the database servers as well as adding new ones for redundancy and capacity needs. We have at least 4 databases per cluster now (except s5, where the fourth one is being added). At the same time, we have retired 40 old servers, freeing up 2 racks of space and the much-needed IP addresses. The retired servers will be available for donation soon.

• Media Storage [?] — Performance testing of Swift continued in January. We confirmed that performance degrades above roughly 10 million objects in a bucket, so we adjusted both the Swift middleware and MediaWiki support to allow sharded containers. Our current plan is to shard the Commons and English Wikipedia containers. The test hardware for object storage arrived, and we validated that it works as expected. We have thus ordered what will be the production hardware, and expect it to arrive and go into service towards the end of February. We have started populating the current cluster in Tampa with thumbnails in preparation for putting it into production service.

• HTTPS — HTTPS work is still on hold in favor of other projects. We did have some activity thanks to volunteer Abe Music who fixed a portion of our UDP logging module for Nginx. There is one more fix still needed before HTTPS page views are properly tracked in our statistics. The following outstanding issues have been fixed: nagios (replaced self-signed cert), upload (served wrong cert via IPv6), integration (wrong cert / certificate chain). There are remaining issues related to: jobs, status and mail. The wiki table of HTTPS-less domains has been updated, and details can be found in the page’s edit history.

Testing environment

• Wikimedia Labs [?] — To keep up with project growth, the virtualization controller (virt1) has been converted into a compute node. Doing so let virt1 and virt4 join the instance storage, doubling the filesystem storage available. The additional compute node also allows Labs to grow by up to another 30 instances. An old ruby gateway server (mobile2) was converted into the virtualization controller (virt0). A number of projects were added or moved to Labs, including incubator, ganglia, deployment-prep, globaleducation, a number of mobile projects, and a bunch more. Labs was very useful during the SF hackathon. A number of projects were created, implemented, and demoed using Labs during the hackathon. We also had a project created, implemented, tested, and deployed to production during the hackathon. We are still waiting for the gluster storage to arrive for volume storage; it should arrive early February. There are now 46 projects, 80 instances and 103 users.

Backups and data archives

• Data Dumps [?] — A problem with the rsync to our mirror site was located and fixed. Another organization agreed to mirror the dumps as well, and we are waiting for their server to come online. Back issues of dumps from 2002 through 2006 were made available, for folks interested in historical data. New hardware has arrived in our Virginia datacenter, and we’ll be copying all dumps over there as soon as it’s ready. We’re thinking about how to provide image dumps in some fashion, even if we don’t keep local copies of the dumps or they are not run on a regular basis. We also cleaned up the dumps documentation and drafted this year’s development plans. Finally, we have a contractor, Christian Aistleitner, who will be working on a test suite for dump generation.

Other news

• Some users complained of slowness and pages not rendered on Wikipedia in Occitan. Domas Mituzas, one of our volunteers, helped fix the issue temporarily. After further investigation, he found that the root cause was badly constructed templates, which were hoarding up server memory RAM. Fixes are planned to prevent this kind of problem from surfacing again.
• Some US-based users of Wikipedia experienced slow page rendering time for 10–15 minutes on January 20, 2012; this was due to the bits.wikimedia.org servers in EQIAD being overloaded. The issue was investigated and quickly resolved.

Features Engineering

Editing tools

• Visual editor [?] — January was a bit slower on the visual editor front, as parts of the team took some well-deserved vacation after the successful prototype launch in December. During the SF Hackathon, a lot of issues were fleshed out. Plans for the second phase of the editor project were formulated. Inez Korczynski investigated a possible use of contenteditable to help with input methods and text selections on mobile devices. Gabriel Wicke extended the parser with the ability to fetch and expand templates in a parallel and asynchronous fashion. The parser now supports most parts of the English Wikipedia Main Page.
• Internationalization and localization tools [?] — January 2012 was slow on the code production side because of the code slush preparing for the branching of MediaWiki 1.19. The Localization team invested a lot of time in writing user documentation for translation tools, input methods and web fonts. Thanks go in particular to the volunteers that assisted in writing and proofreading the documentation. Much of the user documentation can be translated using the Translate extension. Amir worked to solve the old bug in the EasyTimeline extension, that prevented it from working with Indic and right-to-left scripts. Another focus was to improve test coverage for WebFonts and Narayam. In February we expect to deliver further improvements for the translation process, notably on Meta, with workflow improvements, the introduction of a translation memory and (subject to delay) notifying translators of newly available translations from inside MediaWiki by e-mail.

Participation and editor engagement

• Article feedback [?] — Fabrice Florin led development on the next round of Article Feedback Tool v5 features, including a new feedback page, to be released in early February by OmniTI, our development partner. Aaron Halfaker, Oliver Keyes and Dario Taraborelli continued to collect valuable data from the community about the usefulness of comments coming in from each of the three forms launched in December. A survey to get comments from readers about the effectiveness and attractiveness of each design was also launched, and the team has been compiling the various sets of data to produce a report on the pros and cons of each forms in early February. The target date for an expanded feedback page is Feb. 15 for pre-deployment testing on en-labs, then wider deployment on Feb. 22.
• Feedback Dashboard [?] — We implemented a leaderboard of recent top responders on the feedback dashboard. New editor feedback is now added to a dedicated log. When feedback is marked as helpful, that fact is displayed on the feedback dashboard itself. Other than a few other smaller changes, we’re now moving the project into maintenance mode to focus on article creation workflow and New Page Triage.
• Article Creation Workflow [?] — Benny Situ, Ryan Kaldari, Brandon Harris, Alolita Sharma, Oliver Keyes, Howie Fung, and Ian Baker met to discuss sprint planning. They mapped out various user flows leading to article creation, agreed on a proposed landing system and defined changes that are going to be required.

Multimedia Tools

• TimedMediaHandler [?] — The beginnings of a TimedMediaHandler test setup were put into place in Wikimedia Labs, including video transcoding infrastructure, at http://commons.wikimedia.beta.wmflabs.org. Work on this test setup will continue in February, with the goal to begin executing the test plan in preparation for deployment.

MediaWiki infrastructure

• ResourceLoader [?] — Roan Kattouw and Timo Tijhof fixed bugs that would have affected the upcoming 1.19 deployment, and implemented an experimental asynchronous loading feature that will make JavaScript load faster.

Feature support

• Wikipedia Education Program [?] — Jeroen De Dauw implemented many features, like institution & course management, and instructor & student workflows. He also implemented logging and revision histories.

Mobile

• Android Wikipedia App [?] — The Mobile team released the first version of the Wikipedia Android application into the Android Market. In just over three weeks, we’ve had over 900,000 downloads, became the #1 search result for “Wikipedia”, became the #1 trending app, and received a consistent 4/5 stars in the Android Market. We released two minor updates to fix bugs, and are processing user feedback to guide our next version.

• WikipediaZero — We launched our first demo version of WikipediaZero for carrier testing. While there is still much work to be done in order to integrate with as many carriers as we’d like to see, we’re already starting to make progress on how to simplify our implementations.

• Wikipedia over SMS/USSD — Patrick Reilly, along with the PraeKelt Foundation, worked on a demo instance of a SMS/USSD gateway to access Wikipedia. We’re hoping to have a complete demo in time for Mobile World Congress next month.

• GPS Storage/Retrieval — An early prototype of the GPS storage retrieval API went live this month. We still have a large to-do list in order to roll it out in production, but it’s showing great early stage progress.

• FeaturedFeeds — Max Semenik, with the help of Arthur Richards, deployed the first version of FeaturedFeeds to production. Wikimedia communities can now make use of these RSS feeds to better surface their content to other applications. A list of existing feeds is available on the Toolserver.

Special projects

Fundraising support

• 2011 Wikimedia fundraiser [?] — After meeting the budget goals, the 2011 Fundraiser was wrapped up in early January, after which the team participated in a two-day retrospective. Work began on the Fundraiser 2011 cleanup, including the addition of recurring donation and auditing support for GlobalCollect. A two-day inception was held for preliminary planning of the 2012 Fundraiser.

Offline

• Kiwix UX initiative — We have finished the second round of the UX initiative. Focus is now on porting Kiwix to Android/ARM, and we expect to release a first beta version for the end of February. Another goal is to reduce the show-stopper bug list for final release of Kiwix Desktop v0.9.

Platform Engineering

MediaWiki Core

• MediaWiki 1.19 [?] — A new Beta cluster, replicating the production environment, was set up to allow Wikimedians to test upcoming software (including MediaWiki 1.19) on Wikimedia Labs before deployment. A preliminary schedule was drafted, according to which deployment of MediaWiki 1.19 will start on February 13th and complete on March 1st.
• Continuous integration [?] — The team has rearranged Jenkins jobs to make them easier to manage in the long run, and to add capacity. TestSwarm is pending testing of the new Special:JavaScriptTest page.
• Git conversion [?] — The MediaWiki phase3 repository has been successfully converted with branches. Another upcoming test repository will include release tags, and extension projects will be set up shortly.
• SwiftMedia [?] — Aaron Schulz added container sharding to FileBackend, and continued to make performance improvements and fixes. The Swift back-end now passes all unit tests; the code is being reviewed and cleaned up. Some of Aaron’s code for purging thumbnails will be backported to MediaWiki 1.18. Ben Hartshorne has prepared interim hardware for production deployment, which will begin the week of February 6.
• Lua scripting [?] — A team of Wikimedia engineers agreed on Lua as the language to implement as a production-ready replacement for MediaWiki markup-based templates. Tim Starling will lead this effort after the 1.19 deployment and Git migration.

Wikimedia analytics

• Report Card [?] — The new key/value storage approach has been approved by Rob Lanphier. Andrew Otto and Diederik van Liere have started working on a data pipeline framework. All reportcard related code can be found in git.

Technical Liaison; Developer Relations

• Bug management [?] — In January, Mark Hershberger worked with developers to prepare for the (planned) 1.19 deployment in February. He worked with volunteers to launch the beta cluster and held a triage to review 1.19 deployment blockers. The beta cluster has already begun to show some promise with the bugs it has helped reveal.
• Summer of Code 2011 [?] — Merges of student code from GSoC 2011 were substantially on hold in January as developer attention focused on the deployment of MediaWiki 1.19.
• Engineering project documentation [?] — Besides the usual ongoing maintenance of project pages, and putting together this report, Guillaume Paumier also wrote a how-to guide about how to create, use and update project and status pages.
• Volunteer coordination and outreach [?] — In preparation for the San Francisco hackathon, Guillaume Paumier rewrote How to become a MediaWiki hacker along the lines suggested by Yuvaraj Pandian, and cleaned up the documentation about gadgets. Sumana Harihareswara focused on improving the API documentation, and wrote and edited tutorial references for building the Wikipedia Android application, MediaWiki’s web API, and Gadgets. Nine developers got commit access, including seven volunteers. Sumana continued to follow up on contacts and recruit new contributors to the Wikimedia tech community (especially for commit and patch review), and mentor new contributors. Sumana also prepared for the February Pune hackathon and the May hackathon organized by Wikimedia Germany, introduced a friendly space policy for WMF technical events, and recruited participants for upcoming events.
• MediaWiki architecture document [?] — This project is now completed; Guillaume Paumier did some final polishing of the text in collaboration with the book editors. We expect the Architecture of Open Source Applications book to come out in March 2012.
• Wikimedia blog maintenance [?] — The new WMBlog plugin (which brings functionality specific to the Wikimedia blog independently of the theme) was deployed in January, as well as tweaks to the theme. Due to the SOPA/PIPA blackout-related traffic, the Operations team moved the blog to a newer, more powerful server and added caching layers (Varnish & Memcached), and Guillaume Paumier fixed the blog’s theme to paginate comments. Guillaume also fixed issues with attachment pages, and created a custom “meta” widget that adds a link to the Wikimedia blog guidelines to the WordPress standard meta widget.

Future

The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.

This article was written collaboratively by Wikimedia engineers and managers. See revision history and associated status pages. A wiki version is also available.

Participants at the San Francisco hackathon in January 2012

In January, 92 participants gathered in San Francisco to learn about Wikimedia technology and to build things in our first Bay Area hackathon.

After a kickoff speech by Foundation VP of Engineering Erik Möller (video), we led tutorials on the MediaWiki web API, customizing wikis with JavaScript user scripts and Gadgets, and building the Wikipedia Android app.  (We recorded each training; click those links for how-to guides and videos.)  We asked the participants to self-organize into teams and work on projects.  After their demonstration showcase, judges awarded a few prizes to the best demos.

First prize: SMSpedia, an SMS and text-to-speech application in which the user texts a page title to a phone number, then is called up by the service and read the Wikipedia entry aloud. Team members Judy Tuan, Neil Kandalgaonkar, and Romy Ilano will each get travel and accommodation to a Wikimedia event of their choice, such as Wikimania. The judges (Leslie Carr, Roan Kattouw, and Brion Vibber) appreciated the design thinking that went into this project, which gives owners of non-smart phones (“featurephones”) a more intuitive and usable way to access Wikipedia content, without needing broadband. If you watch the showcase video, you’ll hear the burst of spontaneous applause from the crowd when that synthesized speech started talking on Judy’s phone. Roan raved, “It was an awesome idea, it was completed, and … it did something in a new and different way that would promote access.” Brion added that he liked the team’s “impactful demo.”

Second prize (Wikipedia shoulder bags) went to a widget to visualize article quality. It is a cool tool for something whipped up over a weekend! Congratulations to the team of Ben Plowman, Mahmoud Hashemi, Sarah Nahm, and Stephen LaPorte.

A Foundation staffer and new participants collaborate at the hackathon

Honorable mentions:

• Adding ProofreadPage to Wikimedia Labs to benefit Wikisource (Thibaut Horel and Derk-Jan Hartman)
• Articles near the article I’m looking at (Jeroen De Dauw)
• Android search integration (Patrick Reilly and Tomasz Finc)
• Mobile tablet themes (Brion Vibber)
• Home screen widgets (Elizabeth Mezias and Kenneth Ng)
• iOS port (Brion Vibber, Suyash Joshi, and Derk-Jan Hartman)
• Messaging from MediaWiki via RSS or Twitter (Jesse Zbikowski)
• Visualizing the effects of the January 18th blackout (Nimish Gautam, Ryan Faulkner, Andrew Clarkson, and Karen Rustad)

Many hackathon participants were completely new to Wikimedia technologies.  Our experiences working with and ramping up new contributors led to useful text and video documentation that we’re already reusing — for example, Ryan Lane’s talk explaining Wikimedia Labs.

And in between mentoring newbies on Gadgets, mobile, and the API, developers in the room made other strides.  WMF’s Leslie Carr and volunteer Ryan Anderson improved our Puppet infrastructure.  WMF’s Roan Kattouw and volunteer Rusty Burchfield fixed issues in our upcoming migration from Subversion to Git, specifically smoothing our future code review workflow with git-review.  And the operations team worked with new volunteer Abe Music to fix issues in our UDP log and SSL servers.

Mixed in with the hacking and collaboration, attendees enjoyed free food, a few social mixers, and the traditional free t-shirt.  Thanks to Rachel Farrand, Dana Isokawa, and many other Wikimedia Foundation colleagues and fellow Wikimedia community members for organizing and peopling the event with me, and thanks to pariSoma for providing the venue!

Our next developers’ event is a hackathon in Pune, India February 10-12 concentrating on internationalization, localization, and mobile work, and the Foundation’s Ryan Lane will be speaking in Palo Alto on February 27th on “How does Wikipedia Work?”  To find out about other upcoming Wikimedia technical events, check the meetings wiki page, and follow @MediaWikiMeet on Identi.ca or Twitter.

Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation

Group photo from San Francisco hackathon January 2012, at pariSoma

The "Open Advice" book is available for free download, or purchase as print from lulu.com.

The Open Advice book, a collection of essays, stories and lessons learned by members of the Free Software community, is out!

The book was just announced at FOSDEM, the Free and Open Source Software Developers’ European Meeting, in Brussels over the week-end.

About 50 authors from many different projects of the free software community were brought together by Lydia Pintscher, the book’s editor, who started the project in early 2011.

A year and 380 pages later, the book is now available, and tries to provide an answer to the question: What’s the key thing you would have liked to know when you started contributing?

Authors answer that question for many topics, ranging from “Writing patches” to “Documentation for Novices”, to business models, conferences, translation, design, and more.

I contributed “Learn from your users”, a chapter on user experience and usability testing. You’ll also recognize other names from the Wikimedia community, like Evan Prodromou, Markus Krötzsch and Felipe Ortega.

You can learn more about the book and the authors on the book’s website.

All the content of the book is released under the same license as Wikipedia, the Creative Commons Attribution Share-Alike license.

Check it out! You can download the book for free as a PDF file, order a print from lulu.com if you prefer paper books, or fork the text on GitHub.

I hope you’ll like the book, and it’ll prove useful, whether you’re new to the world of software, or you’re a seasoned contributor already.

Guillaume Paumier
Technical Communications Manager

Because of the way we are planning, it is apparent how much code review actually gets done. This sprint we added a review of the ArticleFeedback extension for its internationalization and localization aspects. This is a logical development considering that, with 280+ languages, we are not developing for one language. Our objective for this job is: “As a user I can use the functionality of the ArticleFeedbackv5 so that nothing looks odd in my language from an internationalization and localization perspective”. Reviews like this have been performed informally in the past by translatewiki.net staff. This review, however, will be done during Wikimedia hours and reported through Wikimedia channels.

One old open bug is about EasyTimeline.  It started its life in 2005 and it is finally getting the attention it deserves. The bug explains the lack of support for languages like Arabic, Hebrew and Farsi that are written from right to left. The software has Ploticus as a dependency and for a long time the waiting was for a version of this software that does support RtL languages. We are not waiting any longer and you can read in our story 230 about the complexities involved.

You could say that implementing a translation memory for page translation is a bit more adventurous; it is however debatable if that functionality is new; a translation memory has for a long time been functional at translatewiki.net. It is also very much a feature that makes people more productive. Our team has always had the goal of making life easy and productive for our editors and translators.

The “grammar” functionality for JavaScript is part and parcel of the i18n tooling for our developers. It was not ready before the “slush” and it does make our lives difficult not having it available in the code. When you are building tests for “gender” and “plural”, it is so obvious to create them for “grammar” as well. In this sprint, “grammar” will be included in the code for all these good reasons.

This is the first time that there is a story for outreach. We are reaching out to all the Wikipedia language communities to have their own language support team. It will make a difference when all our language communities have been asked to provide their expertise to us. We already have found that many people show an interest and issues do get raised as a result.

Thanks,
Gerard Meijssen
Internationalization / Localization outreach consultant

• Mobile Wikipedia: Uploading images and other media via your smartphone
• Slideshows: Showcase Wikipedia’s beautiful multimedia
• Realtime: Surface changes to Wikipedia’s content more dynamically

While we received lots of interesting submissions in all three categories, ultimately we had to pick three winners. The grand prize winners in each category get to attend a 2012 Wikimedia event of their choice at our expense. Two runners-up in each category are receiving a certificate of excellence acknowledging the high quality of their submission.

The submissions were reviewed by three judges: Tomasz Finc, Brion Vibber, and Timo Tijhof. Many thanks to all three for offering their time. Without further ado, the winners:

The "Upload to Wikimedia Commons" app is a straightforward, fully functional uploader for Android smartphones

Mobile

The winning submission is “Upload to Wikimedia Commons” by Michiel Minderhoud (User:Michiel1972). It’s a fully functional Android app which makes it easy to share any picture taken on a smartphone by using the “Share via” menu.

What impressed the judges most about this implementation is that it just works, including the fact that it’s already available in the Android market for easy installation. The app source code is available on GitHub for anyone who’d like to participate in its continued development.

We’d like to also recognize two excellent runners-up:

• Thomas Pellissier Tanon (User:Tpt) created the Mobile Upload Wizard building on the code of the Wikipedia Android app. While not a complete experience yet, the choices of architecture and workflow, and the quality of the completed pieces of implementation, merit its recognition as an excellent submission.
• Linas Valiukas created WikiShoot, an iOS app “to take photos of unphotographed locations and upload them to Wikimedia Commons”. Like the previous submission, it doesn’t provide a full upload experience yet, but has a nice implementation of selecting locations and photos, as well as a great workflow document.

 

Michael Müller's "mostEdited" lists not only articles, but specific sub-sections which are being frequently modified.

Realtime

The winning submission is “mostEdited” by Michael Müller (User:Schnark), which “shows you a list of those pages that were edited many times in the last hour (or any other period). These pages seem to be interesting to other users just now, so they are probably interesting to you, too, at least some of them.” The judges recognized Michael’s work as a great foundation for “condensing ‘interesting activity’ out of a raw Recent Changes feed.” The code is available as a user-script on-wiki; Michael also maintains a sizable libraryof user scripts on his German Wikipedia userpage.

• John Du Hart’s LiveEdit extension gives a visual indication (by highlighting the edit tab, and showing a balloon tip when mousing over it) that another user has recently attempted to edit the page you’re on. This can help to avoid edit conflicts, and beautifully fits the theme of “Making Wikipedia More Alive”.
• Omar Ziranhua’s project builds on the wikistream codebase and makes innovative use of node.js and the IRC feeds of recent edits to notify the user about changes to the page they’re on. A bit complex to set up and use, but an interesting use of the underlying open source technologies.

Slideshow

The winning submission is the slideshow script by Marian Beermann (User:Phiarc). It’s a very elegant implementation which, when installed, adds a “Slideshow” tab which appears only on articles with images. The tab triggers a scaled up lightbox view of images in the article, as well as any images included in a linked-to Wikimedia Commons category. The images are captioned as in the article. Other features include a mobile viewing mode, user preferences accessed via a “slideshow settings” dialog, and keyboard navigation.

There were two really great runners-up in this category, too:

• Magnus Manske, who is known for creating the very first PHP implementation of Wikipedia’s software, as well as for many other inventions, created wikiPic, a true labor of love. It adds a menu when mousing over an image which makes its various features accessible. Its slideshow feature tries to identify files, using a variety of characteristics, which are similar to the current one, and displays up to 50 of them in a lightbox mode, and optionally in a full-screen mode. A really beautiful implementation and worth trying out.
• Aleksejs Popoffka submitted a really lean implementation of a slideshow viewer. It also adds a tab, and is a lot less feature-rich than Magnus’ or Marian’s implementation, but quickly and simply pulls up large, lightboxed versions of images in an article.

Once activated by clicking the "Slideshow" tab on an article with images, the user script allows the viewer to comfortably see larger versions of those images.

More coding challenges?

This was the first-ever Wikimedia coding challenge. We’re really impressed with the effort and love that participants have put into these and other submissions. We’ve made many connections through this process with developers we’d otherwise never have interacted with, and have received concrete and useful additions to the corpus of software which enriches the user experience on Wikimedia sites. So overall, this has been a resounding success.

But there are always things we can improve. In future iterations of this, we’d love to aim for:

• More teamwork. The model we tried was designed for a one-submitter-per-submission approach, but we think we could have created more sustainable momentum by inviting folks to work together on specific approaches that are identified ahead of time through a “project pitches” phase.
• More public feedback. While we’ve sent feedback to the winning submitters, some parts of the process (like viewing submission details) were private by default, only visible to judges. This was to ensure that judges can express honest and explicit views without accidentally offending submitters, and to protect the privacy of submitters, especially if they don’t end up in the final lot. But all things considered, a very public review process would probably have lots of advantages that outweigh those considerations.
• Better tips to get started. The newly created gadget kitchen was heavily utilized by many people new to user script development, and we’ll need (and are developing) more comprehensive training materials and tutorials for all parts of the MediaWiki ecosystem.
• Faster turnaround. In the end, it took us too long to review all the submissions, which is partially owed to the bottleneck-y review process and could be mitigated by a more public approach.

Did you (try) to participate and have some feedback? Let us know in the comments!

Big thanks to everyone who made the challenge possible, including Greg DeKoeningsberg, Sumana Harihareswara, Jeroen de Dauw, Sam Reed, Tomasz Finc, Brion Vibber, Timo Tijhof, Dana Isokawa, Heather Walls, Brandon Harris, Roan Kattouw, Trevor Parscal, and Alolita Sharma.

Finally, to everyone who participated: Thank you. We hope you’ll continue your involvement in Wikimedia and MediaWiki.

– Erik Moeller
VP of Engineering and Product Development, Wikimedia Foundation