The Wikimedia Language Engineering team, along with Red Hat, organised the Fall edition of the Open Source Language Summit in Pune, India on November 18 and 19, 2013.

Members from the Language Engineering, Mobile, VisualEditor, and Design teams of the Wikimedia Foundation joined participants from Red Hat, Google, Adobe, Microsoft Research, Indic language projects, Open Source Projects (Fedora, Debian) and Wikipedians from various Indian languages. Google Summer of Code interns for Wikimedia Language projects were also present. The 2-day event was organised as work-sessions, focussed on fonts, input tools, content translation and language support on desktop, web and mobile platforms.

Participants at the Open Source Language Summit, Pune India

The Fontbook project, started during the Language Summit earlier this year, was marked to be extended to 8 more Indian languages. The project aims to create a technical specification for Indic fonts based upon the Open Type v 1.6 specifications. Pravin Satpute and Sneha Kore of Red Hat presented their work for the next version of the Lohit font-family based upon the same specification, using Harfbuzz-ng. It is expected that this effort will complement the expected accomplishment of the Fontbook project.

The other font sessions included a walkthrough of the Autonym font created by Santhosh Thottingal, a Q&A session by Behdad Esfahbod about the state of Indic font rendering through Harfbuzz-ng, and a session to package webfonts for Debian and Fedora for native support. Learn more about the font sessions.

Improving the input tools for multilingual input on the VisualEditor was extensively discussed. David Chan walked through the event logger system built for capturing IME input events, which is being used as an automated IME testing framework available at http://tinyurl.com/imelog to build a library of similar events across IMEs, OSs and languages.

Santhosh Thottingal stepped through several tough use cases of handling multilingual input, to support the VisualEditor’s inherent need to provide non-native support for handling language content blocks within the contentEditable surface. Wikipedians from various Indic languages also provided their inputs. On-screen keyboards, mobile input methods like LiteratIM and predictive typing methods like ibus-typing-booster (available for Fedora) were also discussed. Read more about the input method sessions.

The Language Coverage Matrix Dashboard that displays language support status for all languages in Wikimedia projects was showcased. The Fedora Internationalization team, who currently provides resources for fewer languages than the Wikimedia projects, will identify the gap using the LCMD data and assess the resources that can be leveraged for enhancing the support on Desktops. Dr. Kalika Bali from Microsoft Research Labs presented on leveraging content translation platforms for Indian languages and highlighted that for Indic languages MT could be improved significantly by using web-scale content like Wikipedia.

Learn more about the sessions, accomplishments and next steps for these projects from the Event Report.

Runa Bhattacharjee, Outreach and QA coordinator, Language Engineering, Wikimedia Foundation

The Beta Features preferences page.

We’re pleased to announce Beta Features, a way you can try out new features on Wikipedia and other Wikimedia sites before they are released for everyone.

Beta Features lets developers roll out new software in an environment where lots of users can use these features, then give feedback to help make them better.

You can think of it as a digital laboratory – where community members can preview upcoming changes and help designers and engineers make improvements based on their suggestions. (more…)

The VisualEditor beta release is being gradually rolled out to all Wikipedia editors in all languages. This is one the most exciting developments in the history of Wikipedia, because it will make editing the site accessible to the general public, rather than just to the people who have the patience to learn Wikipedia’s arcane markup language.

To make this accessibility really complete, however, the VisualEditor’s user interface needs to be completely translated to all the languages in which there is a Wikipedia. Its interface includes over a hundred new strings, and if they aren’t translated, they will appear in a foreign language on that Wikipedia (i.e. English text on Polish Wikipedia).

Take a look at the translation statistics for the VisualEditor. As you can see, the translation to a lot of important languages is far from complete or entirely absent: Arabic, Portuguese, Hindi, Swahili, Hungarian, Bulgarian, Tagalog, Urdu, Lithuanian, and many others. If you know a language in that list and the translation to it is not at 100 percent, please click the language name and complete the translation. (You’ll have to create an account at translatewiki.net, if you don’t have one already.)

The article “Vilnius” in the Lithuanian Wikipedia, being edited in the VisualEditor. Note that most of the buttons are written in Lithuanian, but the buttons on the toolbar are in English: “Edit source”, “Page settings”, “Cancel”, “Save page”, “Paragraph”. These buttons weren’t translated yet, so they are unusable for people who don’t know English.

Even if the translation to your language is currently complete, please check your language’s page every few days—the VisualEditor beta is in very active development, the messages to translate are updated literally every day, and you want your language to be at 100 percent all the time.

This is also an opportunity to thank the hundreds of translatewiki.net contributors, who work quietly, but persistently, and make MediaWiki and its extensions into one of the most thoroughly localized pieces of software ever.

If you haven’t joined the translatewiki.net community yet, you are very welcome!

Amir E. Aharoni
Software Engineer, Language Engineering team, Wikimedia Foundation

Google Summer of Code (GSoC) 2013 is well underway on the coding phase. Four projects this year are related to various aspects of MediaWiki and Wikimedia internationalization initiatives. On completion of these 4 projects, we expect to present:

• an Android app for MediaWiki translation,
• browser extensions for the input methods in jQuery.ime (Project Milkshake),
• Right-to-Left language support in the VisualEditor, and
• a dashboard for Wikimedia language projects

These projects are being mentored by members of the Wikimedia Language Engineering team along with members from the WMF Mobile and VisualEditor teams. In this post, we touch base with each of the projects about the challenges that they have faced so far and on their accomplishments.

MediaWiki VisualEditor internationalization and right-to-left languages support

A screenshot of a draft version of the VisualEditor language inspector.

Moriel Schottlender is working on adding better support for non-English languages to the VisualEditor. Her project consists of two main parts. The first is triaging and fixing bugs in handling right-to-left text reported by volunteer editors who test the currently deployed version of the editor in languages such as Arabic and Hebrew. She has already fixed several bugs such as moving the cursor in the correct direction using the arrow keys and adapting the design of VisualEditor’s dialog boxes to right-to-left layout. The other part of the project is developing a “language inspector”–a tool for setting the language of a piece of text in an article. This is needed very frequently in Wikipedias in all languages to set properties such as font, size or direction of a foreign name or quotation. Nowadays it is done using a multitude of templates and HTML tags, and Moriel’s project will make it easy and unified.

jQuery.IME extensions for Firefox and Chrome

Part of Project Milkshake, jQuery.ime is an input method library. Making a good start, Praveen Singh has already implemented working jQuery.ime extensions for both Chrome and Firefox. Rather than loading all input methods at once, input method scripts are loaded only when the user selects a particular input method. The jQuery.ime and jQuery.uls upstream projects were added as git submodules in the extensions. This will provide a way to synchronize the extensions with the upstream projects in the future by simply updating the respective submodule. Universal Language Selector (ULS) has been successfully integrated in the extensions, thus providing the users with an easy way to choose among different languages. The extension remembers a user’s most recently selected languages and their corresponding input methods, and offers an easy way to choose among those languages.

Language Coverage Matrix Dashboard

The Language Coverage Matrix dashboard was a project conceived during the last Open Source Language Summit held in Pune, India earlier this year. The project began as a shared spreadsheet and over the next few months, the data was filtered to uniquely identify the internationalization support status for each language and its variants. The dashboard will provide an interface to search this data and present visual data representations. The test instance set up by Harsh Kothari on wmflabs, showcases the search features. The spreadsheet data has been ported to a MySQL database. Plans for the coming few weeks include additional query implementation, code refactoring and start of the visualization representations.

Phone app for MediaWiki translation

The Translate extension is used for translating MediaWiki content. This project will bring the convenience of this widely used extension as an Android app. The iPhone app created by the student Or Sagi, is already available for download and testing. The similarly designed Android app will provide features for translation and proofreading. Due to conflicts with examination schedules, major work on this project will effectively start from late July.

Getting more updates

More details about the progress of each of these projects can be found on the project home pages. The students and mentors also meet up every week for demos. Please let them know if you’d like to be part of any of these sessions.

Runa Bhattacharjee
Outreach and QA coordinator, Language Engineering, Wikimedia Foundation

Today, the Wikimedia Foundation enabled VisualEditor–the visual interface to edit wiki pages–for all logged-in users on the English Wikipedia. Now, we need your help to ensure that editors are informed about how VisualEditor works, and that they can find help if they need it.

Among the tools brought by VisualEditor, the Reference dialog provides an interface for editing references that is more convenient than editing <ref> tags in the middle of the page’s source wikitext.

There are various reasons that lead existing and prospective contributors not to edit; among them, the complexity of wiki markup is a major issue. One of VisualEditor’s goals is to empower knowledgeable and good-faith users to edit and become valuable members of the community, even if they’re not wiki markup experts. We also hope that, with time, experienced editors will find VisualEditor useful for some of their editing tasks.

VisualEditor has been available for several months now on the English Wikipedia, as an opt-in alpha test. A week ago, we also started an A/B testing campaign where VisualEditor was automatically enabled for half of the newly-registered contributors, while the other half continued to use the classic wikitext source editor.

Now, all registered users can benefit from VisualEditor without needing to explicitly turn it on in their preferences. Of course, the classic wikitext source editor remains available to edit both pages and page sections. While we hope that experienced editors will gradually transition to VisualEditor, this new interface doesn’t yet support the broad range of functionality that raw wikitext allows. We also appreciate that some editors may prefer to use wikitext; we are happy to stress that there are currently no plans to remove the ability to edit the wikitext source.

VisualEditor brings major changes to how you and your fellow contributors edit Wikipedia. In order to minimize the disruption, we’ve prepared a lot of documentation to guide users through the process. This includes a Portal page, a list of frequently asked questions and an extensive user guide illustrated with many screenshots.

The template dialog allows you to change template parameters in a way that is more robust than with wikitext, provided that the right metadata have been added to the template.

Early adopters of VisualEditor have provided us with a mountain of feedback, and we’re immensely grateful for your help. This has allowed us to work with developers to fix many of the bugs and improvements that you reported. Thanks to you, VisualEditor has been much improved, and it is in good shape for wider adoption.

But there’s also a caveat: not all browsers will be supported by VisualEditor immediately. Supported browsers include the current and recent versions of Chrome, Safari, and Firefox. Note that Internet Explorer is not a supported browser at this time.

How you can help

You can help with the following activities:

• Continue to provide feedback as you notice issues with VisualEditor.
• Update Wikipedia’s Help pages — Over the years, volunteers have created many help pages to guide new editors as they figured out the wiki markup language. We now need help to update them so that they reflect the new editing interface.
• Add TemplateData — Templates are one of the most difficult wikitext elements to edit. VisualEditor provides a nice interface in the form of a dialog where you can add values to a template’s parameters. In order for this to work automatically, however, metadata need to be added to the template’s documentation subpage. Usually, the information is already there in the form of a list of possible parameters, and it just needs to be formatted using the TemplateData syntax.
• Translate and localize the documentation — If you edit Wikipedia in languages other than English, VisualEditor will soon be enabled on your wiki as well, and we need your help to make sure your fellow contributors can benefit from the documentation and help pages we’ve prepared.

The next few weeks are going to be busy, as we iron out issues and work together to adapt to the new interface, updating help pages and templates, and welcoming new editors. We hope that you’ll join us in our efforts to make this transition as painless as possible.

Philippe Beaudette, Director, Community Advocacy
James Forrester, Product Manager, VisualEditor and Parsoid
Wikimedia Foundation

 
After several years of development and testing, VisualEditor, the new visual interface to edit Wikipedia pages, will soon be available in “beta” form for all users. This lets Wikipedia editors create and modify articles visually, using a new system where the articles they edit will look the same as they show for reading, and their changes show up as they enter them — like writing a document in a word processor.

VisualEditor removes the need to learn complex wiki markup, and so simplifies editing for both new and experienced editors. We hope that this will open up editing to more people, and along with other efforts will encourage more editors to start and continue to contribute.

We plan to enable it for all logged-in users of the English Wikipedia in early July, later that month extending it to logged-out users, and then the other Wikipedias. Ahead of rolling out VisualEditor in July, we will be carrying out a test of VisualEditor for some randomly-selected new accounts on the English Wikipedia beginning on 17 June. During this testing period, we will be monitoring the impact on users, listening to feedback, and solving problems.

The “alpha” prototype was previously available only to users with a registered account who opted in to test out VisualEditor. First made available on the English Wikipedia in December 2012, it was extended to 16 more language editions in April, and will be made available on all remaining Wikipedias later this week. A lot of valuable feedback has been provided by the early testers of this alpha, and we would like to thank them for their help.

Visual HTML editors are now common on the Web, but building one for Wikipedia (and its sister sites) has been a challenge in itself, due to our specialized requirements and the need to integrate with our existing software, MediaWiki. Behind the scenes, VisualEditor heavily relies on Parsoid, a new complex software component for MediaWiki that translates between wiki markup and annotated HTML+RDFa.

We need your help!

What you can do to help: over the past few months, we have asked you to try out the alpha version of the VisualEditor, and many of you did. Since then, it has changed significantly, and so we’re asking that you try it again. It’s very important that we fix as many critical issues as possible prior to the deploying for everyone in a few weeks’ time — of course, we’d love to fix them all, but that may not be possible. So please, enable the VisualEditor (it’s in your preferences, under the editing tab — check the box labeled “Enable VisualEditor”) and submit any bugs that you find. Your early testing means that we can ensure a better VisualEditor and a smoother deployment for everyone.

Philippe Beaudette, Director, Community Advocacy
James Forrester, Product Manager, VisualEditor and Parsoid
(more…)

This post is available in 6 languages:

English • Deutsch  • Français • Español • Svenska • 日本語 • На русском языке

English

The VisualEditor is now on 15 language Wikipedias

Editors on fifteen Wikipedias – Arabic, Chinese, Dutch, English, French, German, Hebrew, Hindi, Italian, Japanese, Korean, Polish, Russian, Spanish and Swedish – can now get an idea of what the VisualEditor looks like in the “real world”, so they can give us feedback about how well it integrates with their current editing processes. We also want to get their thoughts on what aspects of development we should be prioritizing in the coming months.

The editor is still at an early stage and is missing significant functions, which we will address in the coming months. Because of this, we are mostly looking for feedback from experienced editors; the alpha VisualEditor is insufficient to really give new volunteers a proper experience of editing. We don’t want to promise an easier editing experience to new editors before it is ready.

As we develop improvements, we will push them live every two weeks to the wikis, allowing you to give us feedback as we go, and tell us what you want us to work on next.

How can I try it out?
The VisualEditor is now available to all logged-in accounts as a new preference, switched off by default, on the fifteen Wikipedias listed above. If you go to your “Preferences” screen and click into the “Editing” section, it will have an option labelled “Enable VisualEditor.”

Once enabled, for each article you can edit, you will get a second editor tab labelled “VisualEditor” next to the “Edit” tab. If you click this, after a little pause you will enter the VisualEditor. From here, you can play around, edit and save real articles and get an idea of what it will be like when complete.

At this early stage in our development, we recommend that after saving any edits, you check whether they broke anything. All edits made with the VisualEditor will show up in articles’ history tabs with a “VisualEditor” tag next to them, so you can track what is happening.

How can I help?
It’s vital that our software is available in the native language of as many of our volunteers as possible. If you speak one of these languages – or any of the other 280 languages that we support, like Welsh, Punjabi, Urdu or Scots Gaelic - please consider looking at the translations and helping us improve them!

We would love your feedback on what we have done so far — whether it’s a problem you discovered, an aspect that you find confusing, the areas you think we should work on next, or anything else, please do let us know.

James Forrester, Product Manager, VisualEditor and Parsoid

(more…)

Wikitext, as a Wikipedia editor has to type it in (above), and the resulting rendered HTML that a reader sees in her browser (below)

When the first wiki saw the light of the world in 1995, it simplified HTML syntax in a revolutionary way, and its inventor Ward Cunningham chose its name after the Hawaiian word for “fast.” When Wikipedia launched in 2001, its rapid success was thanks to the easy collaboration using a wiki. Back then, the simplicity of wiki markup made it possible to start writing Wikipedia with Netscape 4.7 when WYSIWYG editing was technically impossible. A relatively simple PHP script converted the Wikitext to HTML. Since then, Wikitext has always provided both the edit interface and the storage format of MediaWiki, the software underlying Wikipedia.

About 12 years later, Wikipedia contains 25 million encyclopedia articles written in Wikitext, but the world around it has changed a bit. Wikitext makes it very difficult to implement visual editing, which is now supported in browsers for HTML documents, and expected by web users from many other sites they are familiar with. It has also become a speed issue: With a lot of new features, the conversion from Wikitext to HTML can be very slow. For large Wikipedia pages, it can take up to 40 seconds to render a new version after the edit has been saved.

The Wikimedia Foundation’s Parsoid project is working on these issues by complementing existing Wikitext with an equivalent HTML5 version of the content. In the short term, this HTML representation lets us use HTML technology for visual editing. In the longer term, using HTML as the storage format can eliminate conversion overhead when rendering pages, and can also enable more efficient updates after an edit that only affect part of the page. This might all sound pretty straightforward. So why has this not been done before?

Lossless conversion between Wikitext and HTML is really difficult

For the Wikitext and HTML5 representations to be considered equivalent, it should be possible to convert between Wikitext and HTML5 representations without introducing any semantic differences. It turns out that the ad-hoc structure of Wikitext makes such a lossless conversion to HTML and back extremely difficult.

In Wikitext, italic text is enclosed in double apostrophes (”…”), and bold text in triple apostrophes (”’…”’), but here these notations clash. The interpretation of a sequence of three or more apostrophes depends on other apostrophe-sequences seen on that line.
Center: Wikitext source. Below: As interpreted and rendered by MediaWiki. Above: Alternative interpretation.

• Context-sensitive parsing: The only complete specification of Wikitext’s syntax and semantics is the MediaWiki PHP-based runtime implementation itself, which is still heavily based on regular expression driven text transformation. The multi-pass structure of this transformation combined with complex heuristics for constructs like italic and bold formatting make it impossible to use standard parser techniques based on context-free grammars to parse Wikitext.
• Text-based templating: MediaWiki’s PHP runtime supports an elaborate text-based preprocessor and template system. This works very similar to a macro processor in C or C++, and creates very similar issues. As an example, there is no guarantee that the expansion of a template will parse to a self-contained DOM structure. In fact, there are many templates that only produce a table start tag (<table>), a table row (<tr>...</tr>) or a table end tag (</table>). They can even only produce the first half of an HTML tag or Wikitext element (e.g. ...</tabl), which is practically impossible to represent in HTML. Despite all this, content generated by an expanded template (or multiple templates) needs to be clearly identified in the HTML DOM.

(more…)

We need your help to test VisualEditor and uncover bugs before we enable it on more wikis.

One of the most important and challenging software development projects at the Wikimedia Foundation right now is VisualEditor: a rich-text editor for Wikipedia that does not require users to learn MediaWiki’s markup syntax. Today, we need your help to make it more robust and reliable.

The alpha version of VisualEditor enabled on the English Wikipedia in December was focused on basic functionality. We’re now moving toward supporting more complex editing operations, notably involving non-Latin characters and character sets.

In order for all language editions of Wikipedia and its sister projects to benefit from VisualEditor, we need to test it extensively, and we need your help to break it (and fix it) before we enable it everywhere.

Non-Latin characters (like math symbols: ⟂) and scripts (like Chinese: 嘗試, and Hebrew: סה) can be more difficult to support than the set of Latin characters we use for example in English.

Starting today (Monday, January 28th, 2013) and continuing all week long, we need your help to test how VisualEditor functions when working with non-Latin characters. We’re relatively confident that VisualEditor can reliably load a wiki page and save that page without losing any information. What is less clear is whether it behaves properly when manipulating non-Latin text, special characters, and other less common aspects of the greater set of Unicode characters.

If you care at all about VisualEditor, internationalization and localization, accessibility, or you simply enjoy hunting down bugs in software, join us this week to identify those issues! You’ll help to improve VisualEditor before it’s enabled more widely.

Our test plan should tell you everything you need to know to get started. We’re also available on IRC for real-time collaboration; all the details are in our coordination page.

The Wikimedia Foundation’s software development model is iterative: we release software early, get feedback, improve it, get more feedback, etc. We’ve set up a dedicated group for this kind of testing that you may want to join. At this time, thoughtful feedback about how VisualEditor manages non-Latin characters is crucial to the next steps of our new editor. We hope to take these steps with you.

Chris McMahon, QA Lead

Yesterday we launched an alpha, opt-in version of the VisualEditor to the English Wikipedia. This will let editors create and modify real articles visually, using a new system where the articles they edit will look the same as when you read them, and their changes show up as they type enter them — like writing a document in a word processor.

Why launch now?

We want our community of existing editors to get an idea of what the VisualEditor will look like in the “real world” and start to give us feedback about how well it integrates with how they edit right now. We also want to get their thoughts on what aspects should be priorities in the coming months.

The editor is at an early stage and is still missing significant functions, which we will address in the coming months. Because of this, we are mostly looking for feedback from experienced editors at this point, because the alpha VisualEditor is insufficient to really give them a proper experience of editing. We don’t want to promise an easier editing experience to new editors before it is ready.

As we develop improvements, they will be pushed every two weeks to the wikis, allowing you to give us feedback as we go, and tell us what you want us to work on next.

How can I try it out?

The VisualEditor is now available to all logged-in accounts on the English Wikipedia as a new preference, switched off by default. If you go to your “Preferences” screen and click into the “Editing” section, it will have an option labelled “Enable VisualEditor.”

Once enabled, for each article you can edit, you will get a second editor tab labelled “VisualEditor” next to the “Edit” tab. If you click this, after a little pause you will enter the VisualEditor. From here, you can play around, edit and save real articles and get an idea of what it will be like when complete.

At this early stage in our development, we recommend that after saving any edits, you check whether they broke anything. All edits made with the VisualEditor will show up in articles’ history tabs with a “VisualEditor” tag next to them, so you can track what is happening.

We would love your feedback on what we have done so far — whether it’s a problem you discovered, an aspect that you find confusing, what area you think we should work on next, or anything else, please do let us know.

James Forrester, Product Manager, VisualEditor and Parsoid