UX + Usability Study Take Two!

 
The Wikipedia Usability Initiative partnered with Bolt Peters and Davis Research to evaluate the changes we’ve implemented so far and inform our work moving forward.  If you don’t know what changes we are talking about, check out our Beta (including a new skin, new toolbar, improved search, and more) by following these instructions.
Overall, the study confirmed that we are on the right track with our beta features – showing us room for improvements, maybe a bug or two along the way, and work yet to be done!  You can view the full report (and soon the full videos) on our project wiki, but we thought we’d share with you some highlights:
Success
“It was easy, and I wouldn’t have thought it would be that easy.”
“Before there were a lot of tools, and I liked that they were all spread out in front of you, but this actually makes a lot of sense. I had to muddle my way through the older system, but this one seemed fine.”
“Websites don’t have common sense, but programmers do.”
The majority of our 8 interview subjects found and used our features and tools without instruction and with success.  Special victories go to our more spacious and grouped tabbed navigation, improved search and new searchbox location, and built-in toolbar.  In using these features, users were not only less intimidated, but also showed a greater ease of use and increased performance.  All of the 8 users successfully found the “edit” tab with a minimum of hunting; no one resorted to Google to get to the Wikipedia article they were seeking; two of our users even expressed pleasure and delight in the process!  Perhaps small victories, but a major change from our first study if you remember!
Needs Improvement
“Uh-oh, I think I may have made the wrong kind of link before. I’ll go to the preview window to see if this is a link. It would have been nice to just edit it in the preview.”
“This is different, it’s got these hot-links [the table of contents]. That’s nice.”
“Links are so easy to screw up. I’m not sure if we’ve correctly typed the link markup. Ah, there are these buttons…”
Some of our tools are definitely still rough around the edges – their flaws and failures were seen in technicolor when observing people using them.  Our link dialog caused the most confusion.  6 of our 8 users initially made some errors in using it, and some received a false positive assurance when they had not actually accomplished the link behavior they were attempting.  Oops!  Our features need to err on the side of a user’s expectation rather than giving users access to the technical structure or wiki syntax, which they did not in this case.  For example, to create a new link in our prototype, users were asked to specify whether they wanted to create an “external link” (to a website) or “internal” link (to a different article) – a differentiation that exists in wiki code, but not in the eyes of a novice user.  Additionally, our toolbar buttons need to behave consistently and be grouped accordingly.  Having dialogs for links and tables, and not having one for a reference was not acceptable and led to some quite confused and persistent button pushing by our subjects.
Speaking of buttons, our “Bold” and “Italics” toolbar buttons use the roman character “a” – the result of our struggled effort to be accessible to an international community while attempting to take advantage of software standards.  In our effort to generalize, we became too general – even those users who correctly guessed the purpose of these buttons had to hover over or use them to confirm their assumptions.  We’re going all in – look out for our efforts to make our toolbar icons language specific soon!
As if we didn’t already know it – adding media or “embedding a file” was the least understood toolbar action of our study.  Most users avoided it, but when they did the sample text that it inserted provided no additional insight.
Moving Forward
“I’m completely intimidated by that [template].”
“I’m not sure what that is. I’m going to save it and then see, because this preview is too confusing.”
Our study illustrated how large an effect a small change can have and brought to our attention tweaks and enhancements that need to be made to our current features.  It also showed us that we are just a slice of what is a very, very large pie.  We had many deja vu moments seeing users flounder around previewing and saving, many times adopting strange techniques and multiple windows to add a simple sentence.  The terms “code,” “computer lingo,” “html” often came up and highlighted the separation users feel from their content while editing.  The expectation for editing a wiki to be similar to editing a blog or word processing document was still prevalent.  And though our Table of Contents and built-in cheat sheet put out some small fires, when navigating an lengthy article or searching for help, we again heard “there sure is a lot of stuff to read” and “this is where I’d give up.”
As we’ve mentioned before, we cannot tackle the full scope of issues that our study participants surface.  But I think I can speak for our team when I say we all felt a certain amount of satisfaction in the results of those problems we did address and it has only made us more eager to attack new problems and iterate on solutions we’ve proposed.  As always, we look forward to your comments, insights, and feedback!  We also appreciate your contributions during our fundraiser – it’s in part community support like this that makes the Foundation’s work possible.
Parul Vora, Wikimedia Usability Initiative

Can you help us translate this article?

In order for this article to reach as many people as possible we would like your help. Can you translate this article to get the message out?

Start translation