The Wikipedia Usability Initiative partnered with Bolt Peters and Davis Research to evaluate the changes we’ve implemented so far and inform our work moving forward.  If you don’t know what changes we are talking about, check out our Beta (including a new skin, new toolbar, improved search, and more) by following these instructions.

Overall, the study confirmed that we are on the right track with our beta features – showing us room for improvements, maybe a bug or two along the way, and work yet to be done!  You can view the full report (and soon the full videos) on our project wiki, but we thought we’d share with you some highlights:

Success

“It was easy, and I wouldn’t have thought it would be that easy.”

“Before there were a lot of tools, and I liked that they were all spread out in front of you, but this actually makes a lot of sense. I had to muddle my way through the older system, but this one seemed fine.”

“Websites don’t have common sense, but programmers do.”

The majority of our 8 interview subjects found and used our features and tools without instruction and with success.  Special victories go to our more spacious and grouped tabbed navigation, improved search and new searchbox location, and built-in toolbar.  In using these features, users were not only less intimidated, but also showed a greater ease of use and increased performance.  All of the 8 users successfully found the “edit” tab with a minimum of hunting; no one resorted to Google to get to the Wikipedia article they were seeking; two of our users even expressed pleasure and delight in the process!  Perhaps small victories, but a major change from our first study if you remember!

Needs Improvement

“Uh-oh, I think I may have made the wrong kind of link before. I’ll go to the preview window to see if this is a link. It would have been nice to just edit it in the preview.”

“This is different, it’s got these hot-links [the table of contents]. That’s nice.”

“Links are so easy to screw up. I’m not sure if we’ve correctly typed the link markup. Ah, there are these buttons…”

Some of our tools are definitely still rough around the edges – their flaws and failures were seen in technicolor when observing people using them.  Our link dialog caused the most confusion.  6 of our 8 users initially made some errors in using it, and some received a false positive assurance when they had not actually accomplished the link behavior they were attempting.  Oops!  Our features need to err on the side of a user’s expectation rather than giving users access to the technical structure or wiki syntax, which they did not in this case.  For example, to create a new link in our prototype, users were asked to specify whether they wanted to create an “external link” (to a website) or “internal” link (to a different article) – a differentiation that exists in wiki code, but not in the eyes of a novice user.  Additionally, our toolbar buttons need to behave consistently and be grouped accordingly.  Having dialogs for links and tables, and not having one for a reference was not acceptable and led to some quite confused and persistent button pushing by our subjects.

Speaking of buttons, our “Bold” and “Italics” toolbar buttons use the roman character “a” – the result of our struggled effort to be accessible to an international community while attempting to take advantage of software standards.  In our effort to generalize, we became too general – even those users who correctly guessed the purpose of these buttons had to hover over or use them to confirm their assumptions.  We’re going all in – look out for our efforts to make our toolbar icons language specific soon!

As if we didn’t already know it – adding media or “embedding a file” was the least understood toolbar action of our study.  Most users avoided it, but when they did the sample text that it inserted provided no additional insight.

Moving Forward

“I’m completely intimidated by that [template].”

“I’m not sure what that is. I’m going to save it and then see, because this preview is too confusing.”

Our study illustrated how large an effect a small change can have and brought to our attention tweaks and enhancements that need to be made to our current features.  It also showed us that we are just a slice of what is a very, very large pie.  We had many deja vu moments seeing users flounder around previewing and saving, many times adopting strange techniques and multiple windows to add a simple sentence.  The terms “code,” “computer lingo,” “html” often came up and highlighted the separation users feel from their content while editing.  The expectation for editing a wiki to be similar to editing a blog or word processing document was still prevalent.  And though our Table of Contents and built-in cheat sheet put out some small fires, when navigating an lengthy article or searching for help, we again heard “there sure is a lot of stuff to read” and “this is where I’d give up.”

As we’ve mentioned before, we cannot tackle the full scope of issues that our study participants surface.  But I think I can speak for our team when I say we all felt a certain amount of satisfaction in the results of those problems we did address and it has only made us more eager to attack new problems and iterate on solutions we’ve proposed.  As always, we look forward to your comments, insights, and feedback!  We also appreciate your contributions during our fundraiser – it’s in part community support like this that makes the Foundation’s work possible.

Parul Vora + the Wikimedia Usability Initiative

Earlier today, English Wikinews adopted the usability beta as a default interface. The usability team is thrilled that en.wikinews community has reached the consensus to be the first adopter of the usability beta as default. We will continue enhancing the interface to simplify and make it easy to navigate and edit.  Our sincere appreciation goes to the entire en.wikinews community for embracing our work. It is a great day for the usability team. We feel blessed.

Naoko Komura on behalf of the entire usability team
Program Manager, Usability Initiative

Since the launch of the beta invitation to the first set of usability improvements on August 6th, about 173,000 people tried out the beta and about 134,000 people continue to use the beta as of September 12th.

Beta retention rate is interpreted roughly 77%. These numbers are aggregation of all Wikimedia projects in all available languages. If we look at the retention rate by project or by language, the number varies significantly. For example, the beta retention rate of English Wikipedia is 82% and Spanish Wikipedia is 80%, while the beta struggles to retain beta trial users of the language communities such as Japanese and Korean at the retention rate of 59% and 54% respectively.

We are reviewing the survey feedback and trying to isolate specific issues of languages whose retention rate is below average. If you are curious about how the beta opt-in and opt-out look like at daily or weekly basis, you can visit the preference statistics page. Here is the example link to English Wikpedia. Just change the language prefix or project name to get to the project of your preference.

Naoko Komura
Program Manager
Wikipedia Usability Initiative

Hopefully by now you’ve had a chance to try the new skin ‘Vector‘ that the Usability Team has implemented as part of our first release, also known as Acai!  If you haven’t yet, we invite you to check it out by going to Appearance > Skin in your preferences and selecting ‘Vector’.  As you play with the new skin and enhanced toolbar, the Usability Team is looking forward to our next release – codename Babaco – the next alphabetical tropical fruit in what we expect to be a delicious series.

We hope to continue adding features to enhance the editing process – including but not limited to further toolbar enhancements; dialogs for the creation of links, tables, and references; and tools to aid in navigation of article content during editing.  Our research, design, and development depends on feedback from users, community members, and interested parties like you.  We invite you to take a look at our work and if you have any opinions, praise, concerns, or criticism, please let us know here.  We look forward to hearing from you!

Parul Vora, Wikimedia Usability Initiative

Another usability improvement we are working on is the action-grouped toolbar, which hides the overwhelming number of tool icons which are not being used by novice users, but they are available for power users in the expanded mode. Lots of clutters will be removed from the editing interface.

You can see some of design mock-ups from our project page. Some of these proposed design concepts will be staged at the prototype environments this month and for any future improvements. (The action-based tool bar will not be seen on prototypes till June 8.)

If all goes well, these new features will be available from a user preference configuration early July. In the subsequent release in August, additional navigation aids are in the plan. We will be sharing the product feature at our project page in coming weeks. We look forward to your feedback.

Naoko Komura
The Wikipedia Usability Team

http://wikimediafoundation.org/wiki/Job_openings/Software_Developer_(project)

Naoko Komura
Wikipedia Usability Initiative