Today we launch our search for the next Executive Director of the Wikimedia Foundation.

About six weeks ago, the Wikimedia Foundation’s Executive Director Sue Gardner told us she will be stepping down from her role. Happily, she is staying on until we find her successor, and we are now launching that search.

It will be a challenge to find someone who is able to fill Sue’s shoes, but I am glad to say that the Board of Trustees, Sue and the senior staff of the Wikimedia Foundation are aligned in our quest for a successor who will build on Sue’s considerable accomplishments, and steer the Wikimedia Foundation toward even greater success in the future.

The Wikimedia Foundation is the internationally-active San Francisco-based non-profit organization that operates Wikipedia, the free encyclopedia. It supports a global community of tens of thousands of volunteers in collecting, developing, and making the sum of all the world’s knowledge freely available. Over half a billion people use Wikipedia and its sister projects every month. We are the fifth most popular website in the world, and the only donor-supported site in the top 100. We’re widely recognized as the most influential and important organization in the free knowledge movement.

Our Executive Director reports to the Board of Trustees and acts in partnership with the global volunteer community, providing the leadership and setting the strategy for the Wikimedia Foundation, while managing its day-to-day operations and activities. The Executive Director is responsible for modernizing the user experience and nurturing, growing and diversifying the community of people who write our projects. He or she also ensures our grantmaking supports innovation across the Wikimedia movement and enables contributor growth in underrepresented demographics and geographies.

Our Executive Director needs to understand and advance the Wikimedia movement’s core values. They need to have proven management skills in technology and product development in order to effectively lead a high-traffic website, and have experience designing and implementing planning processes with a high built-in assumption of fast and iterative change. He or she will need to have exceptional communication skills, and possess both a drive to achieve transformative results and a deep respect for collaborative processes. The Executive Director’s ability to effect change in partnership with Wikimedia’s community will be decisive not just to their success, but to Wikimedia’s lasting impact.

It’s impossible to know where our next Executive Director will come from: there is no career path that makes running the Wikimedia Foundation somebody’s obvious next step. The right person might or might not currently work at a big web site. They might or might not be in the non-profit sector. They could have a background in education, or product development, or media, or community development, or something entirely different. They may live in the United States, or outside it. In this search, we want to cast a wide net for candidates, so that we can find the person with the rare mix of skills, experiences and values needed for this important role.

If you’re reading this post you know how much the work of the Wikimedia Foundation matters. I’m asking you for your help in spreading the news of this unique opportunity. Please share this post widely in your networks.

For more information, to suggest potential candidates or to put yourself forward, please write to info@moppenheim.com.

Some details on the recruitment:

• We have retained the search firm m/Oppenheim Associates to assist in finding and screening candidates. We’ve worked successfully with m/Oppenheim in the past to fill senior roles at the Foundation. They know us well, and we trust they’ll do a great job with this hire.
• The full position description is available on the Wikimedia Foundation site,  hosted at jobs.wikimedia.org.
• The hiring process will unfold over the next three to six months; we hope to have a new Executive Director in place by October. That said, we’re going to take the time we need to find the best possible candidate. We are glad to restate that our current Executive Director, Sue Gardner, will stay with us throughout the recruitment process until we have a new Executive Director in place.
• Following initial screening of the candidates a short-list of applicants will be interviewed by Board members and members of the senior staff, and we will encourage them to get involved with the Wikimedia community (if they aren’t already) to learn more about our movement. (We would also encourage anyone interested in the role to take a look at our guiding principles, or to pick up one of the books documenting and describing the Wikimedia movement.)
• We’ve set up some pages on meta wiki, the central collaboration wiki, where Wikimedia community members can find more information and also get involved in a public discussion about the role and the recruitment process.

Thanks in advance for helping spread the word about this rare and important opportunity.

Kat Walsh
Chair, Board of Trustees, Wikimedia Foundation

This past Wednesday marked a milestone in the evolution of Wikimedia’s Database infrastructure: the completion of the migration of the English and German Wikipedias, as well as Wikidata, to MariaDB 5.5.

For the last several years, we’ve been operating the Facebook fork of MySQL 5.1 with most of our production environment running a build of r3753. We’ve been pleased with its performance; Facebook’s MySQL team contains some of the finest database engineers in the industry and they’ve done much to advance the open source MySQL ecosystem.

That said, MariaDB’s optimizer enhancements, the feature set of Percona’s XtraDB (many overlap with the Facebook patch, but I particularly like add-ons such as the ability to save the buffer pool LRU list, avoiding costly warmups on new servers), and of Oracle’s MySQL 5.5 provide compelling reasons to consider upgrading. Equally important, as supporters of the free culture movement, the Wikimedia Foundation strongly prefers free software projects; that includes a preference for projects without bifurcated code bases between differently licensed free and enterprise editions. We welcome and support the MariaDB Foundation as a not-for-profit steward of the free and open MySQL related database community.

Preparing For Change

Major version upgrades of a production database are not to be made lightly. In fact, as late as 2011, some Wikipedia languages were still running a heavily patched version of MySQL 4.0 — the migration to 5.1 required both schema changes, and direct modifications of data dumps to alter the padding of binary-typed columns. MySQL 5.5 contains a variety of incompatibilities with prior versions, thanks in part to better compliance with SQL standards. Changes to the query optimizer between versions may also change the execution plan for common queries, sometimes for the better but historically, sometimes not. SQL behavior changes may result in replication breakage or data consistency issues, while performance regressions, whether from query plan or other changes, can cause site outages. This calls for a lot of testing.

Compatibility testing was accomplished by running MariaDB replicas outside of production, watching for replication errors, replaying production read queries and validating results. After identifying and fixing a couple of MediaWiki issues that surfaced as replication errors (along the lines of trying to set unsigned integer types to negative values which previously caused a wrap-around instead of an error) we replayed production read queries using pt-upgrade from Percona Toolkit. Pt-upgrade replays a query log against two servers, and compares the responses for variances or errors. Scripts originally developed for our recent datacenter migration to simultaneously warmup many standby databases from current production read traffic helped with rough load testing and benchmarking. Along the way, a pair of bugs in MariaDB 5.5.28 and 5.5.29 were identified, one of which was a rare but potentially severe performance regression related to a new query optimizer feature. The MariaDB team was very responsive and quick to offer solutions, complete with test cases.

Performance Testing In Production

As a read-heavy site, Wikipedia aggressively uses edge caching. Approximately 90% of pageviews are served entirely from the edge while at the application layer, we utilize both memcached and redis in addition to MySQL. Despite that, the MySQL databases serving English Wikipedia alone reach a daily peak of ~50k queries/second. Most are read queries served by load-balanced slaves, depending on consistency requirements. 80% of the English Wikipedia query load (up to 40k qps) are typically handled by just two database servers at any given time. Our most common query type (40% of all) has a median execution time of ~0.2ms and a 95th percentile time of ~50ms. To successfully use MariaDB in production, we need it to keep up with the level of performance obtained from Facebook’s MySQL fork, and to behave consistently as traffic patterns change.

Ishmael views of pt-query-digest data collected via tcpdump for the most common Wikipedia read queries (pdf). The first page of a query shows data from db1042, running 5.1fb-r3753, the second from db1043 over the same time period, running MariaDB 5.5.30.

Once confident that application compatibility issues were solved and comfortable with performance obtained under benchmark conditions, it was time to test in production. One of the production read slaves from the English Wikipedia shard was taken out of rotation, upgraded to MariaDB 5.5.30, and then returned for warmup. The load balancer weight was then gradually increased until it and a server still running MySQL 5.1-facebook-r3753 were equally weighted and receiving most of the query load.

Also from the Percona Toolkit, we use pt-query-digest across all database servers to collect query performance data which is then stored in a centralized database. Query data is collected from two sources per server and stored in separate buckets — from the slow query which only captures queries exceeding 450ms, and from periodic brief sampling of all queries obtained by tcpdump. Ishmael provides a convenient way to visualize and inspect query digest data over time. Using it, along with direct analysis of the raw data, allowed us to validate that every query continued to perform within acceptable bounds.

For our most common query type, 95th percentile times over an 8-hour period dropped from 56ms to 43ms and the average from 15.4ms to 12.7ms. 50th percentile times remained a bit better with the 5.1-facebook build over the sample period, 0.185ms vs. 0.194ms. Many query types were 4-15% faster with MariaDB 5.5.30 under production load, a few were 5% slower, and nothing appeared aberrant beyond those bounds.

From there, we upgraded the remaining slaves one by one, before finally rotating in a newer upgraded class of servers to act as masters. The switch was seamless and performance continues to look good. We’ll be completing the migration of shards covering the rest of our projects over the next month. Beyond that, we’re looking forward to the future release of MariaDB 10 (global transaction IDs!), and are continually assessing ways to improve our data storage infrastructure. If you’re interested in helping, the Wikimedia Foundation is hiring!

Asher Feldman, Site Architect

Wikimedia consists of many projects, Wikipedia most notable among them. However, the name “Wikimedia” suggests a world beyond text. Indeed, Wikimedia Commons, our repository of freely-licensed media files, already contains more than 16 million images, sound files, and videos.

Well, mostly images. Right now, there are fewer than 30,000 video files, and fewer than 170,000 audio files. And while Wikipedia articles are often richly illustrated, they still share the old-school feel of a print-based experience. Projects like Snow Fall by the New York Times show what an immersive reader experience can look like, with video elements prominently featured and blended into the core of the content. In contrast, Wikipedia articles rarely have videos, and if they do, those videos are usually very short and included at the bottom of the article.

Of course, well-written text forms the foundation of most high quality educational content.  Text is versatile, adaptable, accessible, efficient, and relatively easy to collaborate on.  It will form the core of the Wikimedia experience for a long time to come. Still, we can greatly improve the educational value of our sites by empowering everyone to share media, collaborate on improving that media, and using that media well throughout our sites.

In the last three years, Wikimedia has seen some very significant multimedia developments:

• The Wikimedia movement has launched successful photo contests and competitions, notably the “Wiki Loves Monuments” competition, which was recognized as the world’s largest photo competition by the Guinness Book of Records. In the 2012 competition, more than 350,000 photos were taken by volunteers. It was organized by Wikimedia chapters and volunteers in 33 countries (see jury report).
• Wikimedia chapters and volunteers have also formed partnerships into the cultural sector (e.g. museums, galleries, archives), resulting in hundreds of thousands of photographs, reproductions of paintings, and other media being made available on Wikimedia Commons.
• Wikimedia Foundation has developed a number of enhancements and features focused on multimedia:
  • the Upload Wizard, an easy-to-use tool for uploading media files that’s been used to upload more than 2.2 million files to Wikimedia Commons;
  • upload features for the mobile web that make it easy to enrich any article requiring a photograph using a smartphone;
  • a new HTML5 video player with support for the open WebM video format and encoding of videos in multiple resolutions;
  • dedicated upload apps for iOS and Android are in development;
  • a feature to import photographs from Flickr (started as a Google Summer of Code project)
  • an experimental feature to upload files up to 500MB in size.

In combination, these efforts have already borne fruit. The number of contributors to Wikimedia Commons has increased significantly in the last 3 years.  In January 2010, only 13219 users had contributed at least one upload.  That number increased to 20161 users by January 2013.

At the same time, we haven’t invested enough. With the exception of the work of our mobile team, much of the above work has been done by one or two developers at a time, often in between other priorities or by engineers working as volunteers. There has never been a well-resourced team fully dedicated to multimedia engineering work at the Wikimedia Foundation. This is about to change.

The Wikimedia Foundation is hiring at least three engineers and additional product/design support to fully focus on improving the user experience for contributing, curating and reviewing multimedia. Right now, you can apply for the following positions:

• Software Engineer – Multimedia Systems
• Software Engineer – Multimedia User Interfaces

Here are some of the key challenges for the new team:

• further improvements to the upload experience. Contributing an image or video to an article while you’re editing should not require leaving the “edit mode” — it should be integrated with the editing process.
• solidifying experimental features such as large file uploads;
• improving transcoding features for video files to reduce the learning curve for video uploaders;
• improving media search and discovery;
• improving display of images, videos and sound files in Wikipedia articles, including a standard lightbox viewer for media embedded in an article and related media from Wikimedia Commons (building on some of the excellent submissions in our October 2011 Coding Challenge).

As we continue to provide new means for uploading media, we need to ensure that the Wikimedia community is empowered to curate and categorize the images. Curation includes removal of content that is out of scope or incorrectly licensed. To more effectively patrol content, the development of curation tools similar to the Page Curation feature developed for Wikipedia may become necessary.

Beyond Wikimedia’s category system, we will likely want to explore implementation of lightweight tagging systems, possibly in partnership with the Wikidata team.

As if this weren’t enough, the long term frontiers for multimedia include web-based editing of images, video and sounds, improvement for subtitle editing, browser-based audio recording features, and more.

In short, breaking through walls of text and creating a richer media experience for all our projects will keep the Wikimedia Foundation and the Wikimedia movement busy for many years to come. Please help us expand our library of freely-licensed educational media, and help us ensure it gets used effectively on the world’s fifth-most popular website.  Apply today.

Rob Lanphier, Director of Platform Engineering
Erik Möller, Deputy Director; Vice President of Engineering and Product Development

Wikitext, as a Wikipedia editor has to type it in (above), and the resulting rendered HTML that a reader sees in her browser (below)

When the first wiki saw the light of the world in 1995, it simplified HTML syntax in a revolutionary way, and its inventor Ward Cunningham chose its name after the Hawaiian word for “fast.” When Wikipedia launched in 2001, its rapid success was thanks to the easy collaboration using a wiki. Back then, the simplicity of wiki markup made it possible to start writing Wikipedia with Netscape 4.7 when WYSIWYG editing was technically impossible. A relatively simple PHP script converted the Wikitext to HTML. Since then, Wikitext has always provided both the edit interface and the storage format of MediaWiki, the software underlying Wikipedia.

About 12 years later, Wikipedia contains 25 million encyclopedia articles written in Wikitext, but the world around it has changed a bit. Wikitext makes it very difficult to implement visual editing, which is now supported in browsers for HTML documents, and expected by web users from many other sites they are familiar with. It has also become a speed issue: With a lot of new features, the conversion from Wikitext to HTML can be very slow. For large Wikipedia pages, it can take up to 40 seconds to render a new version after the edit has been saved.

The Wikimedia Foundation’s Parsoid project is working on these issues by complementing existing Wikitext with an equivalent HTML5 version of the content. In the short term, this HTML representation lets us use HTML technology for visual editing. In the longer term, using HTML as the storage format can eliminate conversion overhead when rendering pages, and can also enable more efficient updates after an edit that only affect part of the page. This might all sound pretty straightforward. So why has this not been done before?

Lossless conversion between Wikitext and HTML is really difficult

For the Wikitext and HTML5 representations to be considered equivalent, it should be possible to convert between Wikitext and HTML5 representations without introducing any semantic differences. It turns out that the ad-hoc structure of Wikitext makes such a lossless conversion to HTML and back extremely difficult.

In Wikitext, italic text is enclosed in double apostrophes (”…”), and bold text in triple apostrophes (”’…”’), but here these notations clash. The interpretation of a sequence of three or more apostrophes depends on other apostrophe-sequences seen on that line.
Center: Wikitext source. Below: As interpreted and rendered by MediaWiki. Above: Alternative interpretation.

• Context-sensitive parsing: The only complete specification of Wikitext’s syntax and semantics is the MediaWiki PHP-based runtime implementation itself, which is still heavily based on regular expression driven text transformation. The multi-pass structure of this transformation combined with complex heuristics for constructs like italic and bold formatting make it impossible to use standard parser techniques based on context-free grammars to parse Wikitext.
• Text-based templating: MediaWiki’s PHP runtime supports an elaborate text-based preprocessor and template system. This works very similar to a macro processor in C or C++, and creates very similar issues. As an example, there is no guarantee that the expansion of a template will parse to a self-contained DOM structure. In fact, there are many templates that only produce a table start tag (<table>), a table row (<tr>...</tr>) or a table end tag (</table>). They can even only produce the first half of an HTML tag or Wikitext element (e.g. ...</tabl), which is practically impossible to represent in HTML. Despite all this, content generated by an expanded template (or multiple templates) needs to be clearly identified in the HTML DOM.

(more…)

In the 10+ years since its founding, Wikipedia has become an indispensable source of quality information for Internet users everywhere. Here at the Wikimedia Foundation, we’re very proud to support such a project. Yet, despite being a household name, there remain some issues with our user experience that are deeply troubling.

This is especially true for the smaller contingent of people who are the regular contributors to the encyclopedia. Wikipedia’s user interface has failed to keep pace with the the encyclopedia’s growth and the lack of a modernized editor experience has contributed to both a decline in the recruitment and retention of editors (a trend that started around 2007).

The Editor Engagement Experiments team tries to reverse this trend by defining, measuring and fixing these important editing workflows, and improving the experience of Wikipedia volunteers who create content. In this post, we’ll show you one of these editing workflows and invite developers to try their hand at implementing a solution.

An example problem

Imagine you want to create an article for English Wikipedia. You begin by searching for the article on Wikipedia and find that there isn’t one on the topic yet.  This is the screen you get.  Can you figure out how to create the article?

The answer is to click on the red link — that’s intuitive, right?

Even if you figure this out, you’re going to have problems. If you don’t have an account (like most readers), you’ll encounter another hurdle: the site will simply tell you that you don’t have permission to create the page. The solution is to create an account, but it doesn’t say that on the page.

Let’s say you register for an account (or log in if you have one) and then get back to the task at hand. Great. But not so much if you’re new to Wikipedia, because all we do is dump a blank text box on you and hope you know what you’re doing. There’s no warning that articles not meeting Wikipedia quality standards will be swiftly deleted. You could start by getting your feet wet by trying out one of the several workflows that are safer for starting a page, but none of these alternatives is presented as an option.

Thousands of people are subjected to this experience every month and all they’re trying to do add to the world’s collective knowledge. If all of this makes you a bit angry, keep reading.

(more…)

This post is available in 2 languages:

العربية  • English

"Welcome to Wikipedia" (Wikipedia brochure front cover in Arabic)

In English

The Wikimedia Foundation is excited to share the first draft of our Arabic Language Initiative strategy, and to invite applicants for the role of an Arabic Language Initiative Director (consultant) based in the region to lead the initiative. We are seeking a creative leader who is passionate about the mission of freely sharing the sum of all knowledge in Arabic. Great candidates will have an understanding of the challenges facing the Arabic Wikipedia and Arabic web content in general, and will have regional experience that enables him/her to navigate comfortably with communities from Marrakech to Muscat.

The Arabic Language Initiative was initiated in October 2011 as a catalyst strategy program alongside similar programs in Brazil and India. Our ultimate goal is developing a vibrant Arabic Wikipedia community that will build and sustain a rich encyclopedia that meets the aspirations of over 350 million Arabic speakers. Currently, there are 645 active Arabic Wikipedia contributors (as of April 2012) and we seek to expand this number to 1,000 within a year.

Over the past eight months, Wikimedia and the Arabic Wikimedia community have focused on empowering community participation in decision making, seeking out opportunities for partnerships in Arabic language countries, and learning from past and ongoing activities. Our approach for expanding the contribution base depends on online and offline programs. In April, we began work to design an editors contribution pilot program to develop new approaches to attracting and supporting new editors on Arabic Wikipedia. We have spent time in six countries (Algeria, Egypt, Jordan, Lebanon, Morocco and Tunisia) to encourage the existing Wikipedia editors to build local communities and to form relationships with groups and organizations who share our mission and are interested in developing programs that help advance the Initiative.

We have already established partnerships with the Qatar Foundation’s Qatar Computing Research Institute (QCRI) which provides partial financial support and seeks to develop supporting programs. We recently entered a partnership with Taghreedat, the popular Twitter-based initiative. They are planning to conduct Wikipedia workshops in a number of cities and are exploring other initiatives. We are working with groups in Morocco, Jordan, Egypt, Tunisia and Lebanon to explore partnership opportunities and are on the look out for additional partners.

In July, we will conclude the first semester of the Education Pilot program in Cairo. To date, the seven participating classes at Cairo University and An-Shams University have successively contributed to editing 267 articles, adding more than 1.1 million bytes to the Arabic Wikipedia. We will be holding a workshop in Cairo at the beginning of July to review the results and plan the next steps for the program and we expect to expand the pilot in the coming months.

We will finalize the strategic plan after more community discussion and as we engage the Initiative Director. We welcome new ideas from the community and are seeking to support initiatives at the local level and/or new pilots on the wikis.

We believe in the strong opportunities of a growing and developing Arabic Wikipedia, and we need all hands in sustaining the growing momentum around the Arabic activities. Please keep an eye on our progress, share your ideas and actively reach out to us to become part of the mission that shall dramatically change the status of Arabic content on the web.

Moushira Elamrawy

Egypt-based Arabic Projects consultant

Community Fellow, Sarah Stierch

I’m pleased to announce Sarah Stierch has been awarded a Wikimedia Community Fellowship for 2012.  Sarah’s fellowship is intended to support her commitment to encouraging women’s participation in Wikimedia projects.

As a volunteer, Sarah moderates Wikimedia’s gender gap mailing list, has done outreach to hundreds of editors in order to conduct a survey of women in Wikimedia, and curates a scoop.it collection of media related to women and Wikimedia.  She also serves on the advisory board for the Ada Initiative, a non-profit organization that supports women in open-culture communities like Wikipedia.  Sarah has been an editor on English Wikipedia since 2004, and has been active in GLAM-Wiki projects since 2009.  An art historian by training, Sarah was a 2011 Wikipedian-in-Residence at the Archives of American Art in Washington D.C., organizes edit-a-thons on art-related topics, and is in the process of finishing her master’s degree in museum studies at George Washington University.

Her experience working with female editors in the community and enthusiasm for outreach makes Sarah a great candidate for what we hope will be the first of several fellowships focused on the gender gap.  Sarah’s initial project will be a new-editor support pilot where she’ll build a team of volunteers to actively reach out to promising new editors (particularly women) to offer help, mentorship and peer support, encouraging them to continue editing and become more integrated into the Wikipedia community.

Congratulations, Sarah, the Wikimedia Foundation looks forward to partnering with you!

And, as a reminder, we’re still looking for more fellows to join Sarah in 2012.  The deadline to apply for this round is January 15th, please contact fellows at wikimedia dot org with any questions.

Siko Bouterse, Head of Community Fellowships

This is the last in my series of introductory posts about Wikimedia Platform Engineering, focusing on the MediaWiki Core group.  This group is responsible for our sites’ stability, security, performance and architectural cleanliness.  This ends up translating into a lot of code review, along with infrastructure projects like disk-backed object cache, heterogeneous deployment, continuous integration, and performance-related work.  While it’s not a prerequisite, everyone on this team started off as a volunteer developer.  The whole engineering organization has some level of responsibility for our code review process, but this group has more of a primary responsibility for it than most groups.  We have an open position in this group.

(more…)

Wikimedia Community Fellows are spearheading community projects, undertaking research, and piloting new models for engagement to help scale and increase sustainability of volunteer work in the Wikimedia movement.

For example…

Liam Wyatt’s Cultural Partnerships Fellowship has built enthusiasm, awareness, and working models for cultural institutions around the world to partner with Wikipedians in producing open-access, freely-reusable content for the public. Liam piloted the Wikipedian in Residence program at the British Museum in 2010, and so far there have been 10 other Wikipedian in Residencies across the globe. The GLAMCamp gatherings that Liam introduced are providing opportunities for volunteers to come together in person to strategize, document, develop tools, share best practices and forward the GLAM movement. Liam’s 1 year fellowship wraps up this month, and he will be missed, but GLAM projects meanwhile continue to grow!

Jon Harald Søby‘s Translation Fellowship is modeling new ways to engage with volunteer translators, and as a result this year’s fundraiser has almost 1400 active translators with 500 translations completed in 112 languages to date. (For comparison, thats already over 1000 more translators and 30 more languages than last year, and Jon’s not done yet!).

We’re recruiting

Want to be like Jon and Liam? The Wikimedia Foundation is now seeking fellowship applicants and project ideas for Spring 2012 Fellowships! Submissions are encouraged to focus on the theme of improving editor retention and increasing participation in Wikimedia Projects. The deadline to apply is January 15^th, 2012.

If you’d like to work with the Wikimedia Foundation on projects to boost participation and retention, or know someone who should be recommended for a fellowship, or if you’ve got ideas for a fellowship project WMF could support, we’d like to hear from you! Please visit the Fellowships Program page for more information.

Siko Bouterse, Head of Community Fellowships