Earlier this month, about thirty MediaWiki developers and interested technologists gathered in New Orleans to learn and to work on Wikimedia’s technical infrastructure.  We made broad progress on the infrastructure of innovation at Wikimedia (notes).  Specifically:

• We are now much closer to officially opening the doors to Wikimedia Labs and giving far more people the ability to contribute to MediaWiki without having to set up and maintain their own development environments at home.  Wikimedia Labs will provide hosted, virtualized test and development sandboxes for new and experienced programmers and systems administrators.  Many developers got beta Labs accounts, we tested at a larger scale, and we fixed several bugs.
• Developers agreed to create a file backend abstraction layer to enable large-scale MediaWiki installations to use one of several storage systems to contain big collections of big media files.  (Wikimedia plans on using Swift, which is open source.) Microsoft’s Ben Lobaugh and SAIC’s DJ Bauch collaborated towards improving MediaWiki’s performance on Microsoft technologies as well.  Developers made architectural decisions, refactored some existing code, and improved documentation and tests for the SwiftMedia extension to MediaWiki.
• 

  Chad Horohoe teaching developers unit testing

  We now have a continuous integration server up and running.  This will continuously run tests checking on the latest new features and bugfixes that developers write, resulting in fewer bugs and faster development. Developers will need to write tests to reap the benefits, so Chad Horohoe taught a test-writing workshop.

• Max Semenik finished and demonstrated the first version of his API Query Sandbox.  This allows software developers anywhere to experiment with ways to automatically get data from Wikipedia or other sites that run MediaWiki, thus enabling wider and deeper reuse of Wikimedia content.
• Operations folks continued the Puppetization of our infrastructure: they completely reworked Varnish management in Puppet, and worked on Puppet configurations for SwiftMedia testing. This configuration management work will ensure that ops can move faster and more confidently in building and maintaining Wikimedia infrastructure. And Canonical’s Mark Mims and Kapil Thangavelu worked on improving methods for Wikimedia developers “to spin up stacks of services within the labs environment” using Juju (more details).
• 

  Brion Vibber leading developers into the "glorious Git future"

  Since the engineering department is planning a switch from Subversion to Git in the next few months, Brion taught nearly everyone there how Git works (slides, audio), and how we’ll be using Git in the future. This change in our source code repository and workflow will, we hope, enable more speed and flexibility in development, both for WMF developers and community contributors.
• We prioritized and addressed several open requests for the operations team and defect reports about the latest version of MediaWiki, 1.18, which had just been deployed across WMF sites.
• Roan found and fixed an issue that was spouting symbolic link errors into our Apache logs, so now it’ll be easier for us to see more dangerous errors in those logs.
• Google Summer of Code students Salvatore Ingala and Kevin Brown made progress on integrating their summers’ work into MediaWiki as used and deployed by others; Salvatore and WMF developer Roan Kattouw have a plan for getting his user scripts improvements reviewed and deployed, so they can benefit Wikimedia readers and editors.
• A volunteer came in on Friday night knowing nothing about developing for MediaWiki, and by the end of the weekend had a working development environment on her laptop and had some ideas about how to contribute.
• We had substantive conversations about the summer internship program and about third-party collaboration that will affect how we work in the future.

We also ate dinner together, walked Bourbon Street, and generally got to know colleagues we’d never met before.  I expect these relationships will bear fruit for years to come.

Thanks to Ryan Lane and Dana Isokawa for organizing the event with me, and thanks to Launch Pad New Orleans for providing the venue!

Our next developers’ event is a hackathon in Mumbai November 18-20 concentrating on internationalization, localization, and mobile work.  To find out about other upcoming Wikimedia technical events, check the meetings wiki page, and follow @MediaWikiMeet on Identi.ca or Twitter.

Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation

On May 24th and 25th, the Wikimedia Foundation hosted a CiviCRM coding sprint in our San Francisco office. CiviCRM is the premier open source constituent relationship manager; WMF uses it to store donor and contribution information. Our CiviCRM database contains more than a million contact records and a million contribution records.

The sprint was a terrific success. The eight participants squashed many CiviCRM bugs — and the Foundation directly benefited, as they improved CiviCRM contact/contribution search performance by 15 to 25 times! Formerly, it could take more than two minutes for someone to search among the contribution records. The developers’ tweaks, hacks and patches whittled that down to about 4-6 seconds per search. This will save innumerable hours for WMF administrators and fundraisers.

The Foundation’s Arthur Richards, a fundraising engineer, enthused: “Any software tool, open source or not, comes with headaches; the beauty of tools like CiviCRM is that we can solve our own problems. Thanks to having some great hackers in one place, we managed to mitigate one of our biggest CiviCRM pain points in a matter of hours.”

You can read more details about the sprint on Donald Lobo’s CiviCRM blog.

Richards was especially excited to “highlight how awesome it is working with other open source projects and using other open source tools. We get to scratch each other’s backs, which helps support a sustainable, healthy ecosystem of software/communities. Also, using open source tools like CiviCRM – while not without their (often big) pain points – is great because we can fix the software ourselves. While the tools are free to use, with a little bit of elbow grease and some resources, they can be molded and fixed to meet our needs much easier (and likely much cheaper) than relying on proprietary tools. Plus, the CiviCRM community has been instrumental in helping us troubleshoot, solve problems and add new features to meet our usage requirements.”

The CiviCRM community is planning to run another code sprint in the fall in Northern California; please contact them if you’d like to participate or even host it. In the meantime, Wikimedia and thousands of other nonprofits will enjoy the CiviCRM improvements developed in May.

-Sumana Harihareswara
Volunteer Development Coordinator, Wikimedia Foundation

While GLAMCamp NYC finished on Sunday (Signpost coverage), the work initiated there will continue throughout the GLAM community.  Representatives from cultural institutions and Wikimedia chapters, as well as individuals, are working on several projects.  The projects concerning web badges for free culture allies, a metadata standard for use in the mass uploader/data ingestion tool, and the web analytics proposal are in particular seeking contributors and project managers; please comment at the coordination page to signal your interest.

Also available: the collaborative notes from Friday, Saturday, and Sunday, and specifically for discussion of the Ambassadors program, the Point Of Entry project, the data ingestion tool, and the metrics/analytics proposal.

Thanks to the organizers and participants for a productive and illuminating weekend.

-Sumana Harihareswara
Volunteer Development Coordinator, Wikimedia Foundation

Today, several Wikimedians and representatives from galleries, libraries, archives and museums (GLAM institutions) met in New York City to kick off GLAMCampNYC.  New York City’s public Science, Industry, and Business Library is hosting the event.

Liam Wyatt, the Wikimedia Foundation’s Cultural Partnerships Fellow (aka GLAM fellow), introduced two keynoters: Meg Bellinger, discussing open access at Yale, and Maarten Zeinstra, presenting the Europeana public domain calculator.  The conference continues through Sunday.  Participants are discussing and building the GLAM outreach wiki, writing documentation, sharing best practices, and building tools.

Developers at GLAMCamp are developing a data-munging tool, based on pywikipediabot, to aid in mass uploads (more details).  According to Wyatt, the most common requests from GLAM institutions are (1) mass upload of audiovisual media and (2) metrics, “easily exportable statistics based on analytics on a GLAM’s relationship with Wikimedia.”  The data-munging or data ingestion tool will aid in the import of metadata from large sets of files, thus speeding the difficult part of mass uploads.  Attendees will be hacking on it in sprints this weekend, starting 3pm-4:30pm UTC time tomorrow, Saturday the 21st. Join them in person (11am local time), or in #glamwiki on Freenode.

See notes from today’s general talks and discussion and from the discussion of the GLAM Ambassadors program, or follow #glamwiki and #glamcamp on Twitter and Identi.ca.

-Sumana Harihareswara
Volunteer Development Coordinator, Wikimedia Foundation

Most MediaWiki developers who attended the Berlin hackathon this weekend have left the German capital and returned home, after three days of collaborative coding, group discussions, short presentations, and bug fixing.

A lot of work was already accomplished on Friday and Saturday, including presentations on test frameworks, coding of new features, discussions on wikitext parsers, and a usability testing session.

Things were a bit slower on Sunday, but lack of sleep didn’t stop developers from coding and smashing bugs. Brandon Harris gave a short talk about identity, editor retention and social features. Domas Mituzas talked about how to improve performance; Tim Starling followed by discussing adding HipHop support for MediaWiki, and its planned deployment to Wikimedia sites.

Mark Bergsma also gave an overview of the situation of the Wikimedia infrastructure regarding IPv6 (and our participation in IPv6 Day) and Mathias Schindler discussed WebP support. All the live notes taken yesterday are available.

The rest of the day was used to continue to code, discuss and smash bugs. Some groups explored the city before returning home. The day ended with participants hacking and socializing at the C-base.

If you couldn’t attend, the videos of all the talks are available for you to watch (or re-watch). Many pictures of the event are already on Wikimedia Commons, and more will follow. Presentation slides will be added to the hackathon page as they come in.

We hope the live video streaming, real-time note taking, and IRCing / tweeting was useful for remote attendees; please tell us what we did right and what needs improving. We’d love to get feedback on what worked for you, and what didn’t.

We’d like to thank everyone who was involved in making this event awesome, and particularly the participants, who came from all over the world to work together to improve our technical platform.

Many thanks to the team from Wikimedia Deutschland as well, who masterminded the whole event: Nicole Ebber, Daniel Kinzler, Cornelius Kibelka, and the rest of their team.

Participants agreed they were looking forward to more hackathons, in Berlin and elsewhere. We’ll see you there!

–
Guillaume Paumier

Photo from Wikimedia Commons by Tobias Schumann, under CC-by-sa 3.0 Germany.

With tired eyes, and fueled by ridiculously large amounts of coffee, Wikimedia developers and engineers are now starting their third and last day of collaborative coding at the Berlin “hackathon”.

The event, organized by Wikimedia Deutschland, has been going on since Friday. About a hundred participants are enjoying our third day at coworking / hackspace Betahaus.

Yesterday, more coding happened, and even more bugs were smashed: about 65 since we started on Friday. There remains plenty to work on during this hackathon, though, if you’d like to help.

Saturday afternoon was also devoted to the discussions about the possible evolutions of the MediaWiki parser (see notes), a step towards a visual editor for Wikipedia and other MediaWiki-powered sites. (“Visual editor” seems to have reached consensus as a more social class-neutral replacement for “rich text editor”.)

Yesterday, the hackathon also hosted a usability testing session on the Kiwix offline app, led by Ryan Kaldari. The ops team is continuing its ongoing work on HTTPS & IPv6, and Victor Vasiliev partially implemented a long-awaited feature for Wikimedia wikis: a global watchlist.

The day ended with a party (with free beer and food) organized by our friends from Wikia.

You can take a look at all the live notes taken yesterday. People are also taking photos, and more will follow.

Some talks that were originally scheduled for Saturday are happening today, including Brandon Harris’ short presentation on “identity”, Mark Bergsma’s on IPv6, and the discussions on performance and HipHop, with Domas Mituzas and Tim Starling.

You can participate remotely in real time by watching the live video stream (all talks are recorded), and participating in our live note-taking in Etherpad.

You can also join us on IRC in #mwhack11 or #mediawiki on Freenode, and follow our activity using the #mwhack11 hashtag on Twitter and Identi.ca.

This year’s motto is “talk less, code more”. Happy coding!

–
Guillaume Paumier

MediaWiki developers and Wikimedia engineers are starting their second day of coding, discussing and bug-smashing today in Berlin, Germany. This “hackathon”, organized by Wikimedia Deutschland, started yesterday, and will last until tomorrow Sunday.

After a short introduction yesterday, participants quickly moved on to group discussions, short presentations and coding. The event is run as an unconference, and this format has proven to be quite effective so far.

Lightning talks yesterday included presentations about the new datacenter (by Mark Bergsma), Kiwix and offline (Emmanuel Engelhart), PhotoCommons (Hay Kranen), OpenStreetMap integration (Tim Adler), WikiLove (Ryan Kaldari), PHPunit (Ashar Voultoiz), the new mobile gateway (Patrick Reilly), community-oriented testing (Ryan Lane), Narayam (Purodha Blissenbach) and distributed JavaScript testing (Timo Tijhof).

Several bugs were also fixed yesterday, but there remains quite a bit to smash during this hackathon.

A lot of group discussions (e.g. about HipHop, and the MediaWiki release plan) and actual coding happened during the afternoon and evening. You can take a look at all the notes taken yesterday in real time.

Today’s talks include discussions on “Identity” (Brandon Harris), performance, including plans to use HipHop for PHP (Domas Mituzas and Tim Starling), as well as many discussions and short talks about wikitext parsers.

To participate remotely in real time: You can still watch the live video stream (all talks are recorded), and participate in our live note-taking in etherpad.

You can also join us on IRC in #mwhack11 or #mediawiki on Freenode, and follow our activity using the #mwhack11 hashtag on Twitter and Identi.ca.

Another way to participate is by testing some of the tools people are developing. For example, Purodha Blissenbach is looking for testers for Narayam (a keyboard mapping for Indic languages), and Hay Kranen would like people to test the PhotoCommons WordPress plugin. Please contact them if you want to get involved.

This year’s motto is “talk less, code more”. Happy coding!

–
Guillaume Paumier

MediaWiki developers and Wikimedia engineers have flown from all over the world to meet up in Berlin.

For the third time, Wikimedia Deutschland is organizing a “hackathon”, a coding event where developers work together to improve the MediaWiki software and the technological platform for Wikipedia.

The event started today at the betahaus, a coworking and social space in Berlin’s Kreuzberg neighborhood, close to Moritzplatz. It will last until Sunday; a rough schedule is available.

Two other groups of Wikimedians are also meeting this week-end at the betahaus: Wiki loves Monuments enthusiasts, and the Language Committee.

Work at the hackathon is notably focused on improvements in MediaWiki’s text editor, development tools, improvements in the parser, mobile apps, and bug fixing. We’re also having a few lightning talks.

These developer days are included in the program of the Berlin Web Week, a series of events happening in Berlin in May and bringing together Internet and software communities and industry players.

To participate remotely: join us on Twitter and Identi.ca, where we’re using the #mwhack11 hashtag. We’re posting links there to our public notes taken in real time.

You can also watch the live video stream (all talks are recorded), join us on IRC in #mwhack11 or #mediawiki on Freenode, and check out the event page on facebook.

This year’s motto is “talk less, code more”. Happy coding!

–
Guillaume Paumier

This is a reminder to anyone who might have been considering coming to the Hack-A-Ton in DC. If you haven’t been following page on MediaWiki.org, we’ve recently updated it with additional information, including the venue.

This year, we will be meeting at the Embassy Suites in Tyson’s Corner, just outside of the city itself. By the end of this week we will have a breakdown of what to expect and when.

We’ve already got a fair number of people coming so far, and I’d love to see more there. So if you’ve been thinking about coming but haven’t said anything or know of someone who would want to come, please get in touch with either myself or Danese for more information.

Registration for the Developers’ Workshop in Berlin on April 14.-16 is now open: please use the registration form. Registration is required and will be open until March 21., but there are only 50 places available. So, sign up soon!

Wikimedia Germany invites all MediaWiki developers, Toolserver users, Gadget hackers, and other people interested in the technical side of Wikimedia projects to come to the Workshop. We have a very nice venue and a cool option for accommodation, details to be announced soon.

For updates and more information, watch meta:Wikimedia_Conference_2010/Developers’ Workshop. You can also get updates via twitter or identi.ca.  If you have questions, please contact us at conference@wikimedia.de.