Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

How is the usability beta doing?

Thank you for your feedback and comments about the usability beta through the survey and via the usability project wiki. The usability beta initially started off with the new skin called Vector and the toolbar (release nickname: Acai) in early July and it was enhanced with the navigable table of contents within the editing box and dialogues for links, tables, and search & replace (release nickname: Babaco) at the end of September.

The Babaco features have to be activated through the user preferences. The most recent enhancement was to move the tool for watching and unwatching pages into the top navigation (star icon). This feature was implemented based on a high volume of related user feedback. Navigation tabs now collapse in case tabs start overlapping when the screen resolution is reduced, or where tab widths are wider because of language specific characters.

The usability beta was visited and tested by close to 380,000 users by the end of November 2009. The beta has been drawing roughly 100,000 users every month, and close to 300,000 users have kept the beta enabled as of December 1, 2009. As I briefly summarized in my blog post in September, the beta program was adopted relatively well by the beta users of English Wikipedia (83% retention rate), and in other English language projects such as the English Wikinews (95% retention rate). Spanish and Portuguese Wikipedia beta users have the second highest retention rate at 81%. German, Russian, Chinese, French and Italian Wikipedia beta users are retained in the range between 70% and 79%. Retention rate for Polish and Japanese was relatively low, with 65% and 60% respectively.

The list of the beta retention rates for the top ten most visited Wikipedia languages is summarized in this table and the list of all projects in all languages can be found here. The opt-out survey and feedback forms were available to beta users and the survey responses provided rich quantitative and qualitative data to understand how the beta is received and understand the reasons for what people like about the beta and leaving the beta.

One of the questions we sought to answer via the opt-out surveys was the impact of different browser types. The usability tech team has been pounding on Internet Explorer related problems and we thought the browser type will have significant impact on the beta adoption pattern. According to our own browser statistics, about 56% of users access Wikipedia through a version of Microsoft Internet Explorer, and about 30% of users use a version of Mozilla Firefox. The browser distribution pattern for the beta users for the ten mostly visited Wikipedia language family indicates stronger favor towards Firefox over Internet Explorer except for Japanese and Chinese Wikipedia whose browser distribution is dominated by Internet Explorer instead of Firefox. If we slice the beta retention rate by browser distribution, we see the the repeated low retention rate for Internet Explorer users. However the variance of retention rates by browser types is not significant enough to conclude that browser distribution has a substantial impact on the beta adoption.

The qualitative survey responses brought up language-specific issues. Japanese Wikipedia beta users found the font used in the new interface is too small. Chinese beta users also expressed the difficulty adjusting to the smaller font in addition to the perceived slowness of the new interface. Switching between Traditional and Simplified Chinese was not well supported. We hope issues with overlapping tabs expressed by German users have been addressed by the recently released collapsible tabs. Finally, we are aware that we often break popular user gadgets, and we appreciate that developer communities gradually embrace the usability updates so that the gadgets are compatible with the usability beta. The survey comments are available in ten languages so far, and we will continue making the surveys available for more languages.

The usability beta continues to evolve and the usability team is actively working on the next release, Citron. This release will have the new features such as collapsed templates, form interface for editing templates, and side-by-side preview. Citron is currently scheduled to be available in January. Beta feedback will be incorporated in this release to address language specific issues which were surfaced from the survey.

As the general acceptance of the usability beta is supported by a majority of beta users, we would like to start discussing the timing of making the new editing interface and the toolbar the default interface after resolving any remaining language specific issues.

This survey data was analyzed and put together by the team effort. I would like to thank Howie Fung, Product Consultant, for organizing and analyzing the survey data, Nimish Gautam, for integrating machine translation into the qualitative survey, and Roan Kattouw for tracking and normalizing the data.

Naoko Komura, Program Manager, Usability Initiative

10 Responses to “How is the usability beta doing?”

  1. Naoko says:

    @ Gregory: “Leave Beta” link is visible for users who wish not to continue using the beta. Thus users do not need to go to user preferences to turn off Vector.

    @ Amgine: We are evaluating both from quantitative and qualitative data. Parul posted the findings from the second usability study with which we evaluated the current beta. Study participants found the new interface easy to use. You can read the findings and view videos from the study through her blog posted on November 18, 2009.

    @ Get A Trip: I can’t agree with you more that Wikipedia and Wikimedia projects are generating invaluable assets for generations to come. Hopefully the usability improvements will help you to contribute your knowledge easier and invite more people to join and collaborate.

  2. Amgine says:

    Similar to Gregory Maxwell, I’d prefer an unbiased reporting of data. This isn’t a marketing campaign; it’s an effort to improve a user interface rather than convince people it’s a good idea to continue the usability project.

  3. Get A Trip says:

    “Usability Beta” – great! After donating to this Wikipedia project we can’t help but feel we are supporting something of the highest educational nature for generations to come — look forward to more involvement…

  4. Gregory Maxwell says:

    I think this is miscounting retention. In my experience readers don’t bother to figure out how to turn vector off, they simply stop logging in or create a new account.

    Could someone measure the actual vector usage based on the number of HTTP requests to the appropriate CSS, or alternatively by counting logged in pageviews?

  5. Kozuch says:

    @Naoko: Thanks for unlocking tha main page!

  6. Gifts says:

    This survey data was analyzed and put together by the team effort.

  7. Naoko says:

    @ Waldir and Marlita, thank you for your support and encouragement.

    @ Kozuch You have been contributing to the usability wiki from the early phase of the project. Thank you. The reason we locked the main page was to prevent it from severer vandalism which we experienced a few times earlier this year, and maintain consistent messaging as we use the main page for public announcement. At early time of the project, we didn’t have enough people to watch the usability wiki. But you are right, it is time to open up the main page as there are folks like you who are watching out our wiki.

  8. Kozuch says:

    Thumbs up for the usability output, but your wiki just SUCKS, I want to redesign Main Page but it is locked. This is not a way wiki should work. :(

  9. Marlita says:

    You are doing a wonderful job of balancing complex needs and processes. Congratulations

  10. Waldir says:

    Congrats to all the Usability team, and everyone else involved — you are doing an awesome job! I can’t wait for the next release, the new features sound great! :D