Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

How do you establish a QA & Testing practice for an open community?

To keep up with the growth of Wikipedia and its community, one goal of the engineering team at the Wikimedia Foundation for this year is to establish a Quality Assurance (QA) practice for software development, including MediaWiki itself, extensions, and also projects like Article Feedback and Editor Engagement. But how do you establish a QA & Testing practice for a development process that involves so many contributors, with code coming in from so many sources and projects?

In software development, QA is often conflated with software testing, but testing is only a small part of QA in general. The goal of modern software testing is not only to discover defects, but also to investigate software in order to provide valuable information about that software from every point of view, from the user experience to how the software is designed. On the other hand, Quality Assurance is process work, examining the process by which the software is created, from design to code to test to release and beyond.

Dozens of (volunteer and paid) developers contribute code to Mediawiki every month, in areas as varied as MediaWiki’s core, MediaWiki extensions, and localization. Thousands of power users on Wikimedia’s wikis can also contribute code directly on the sites, in the form of JavaScript “gadgets”. With so many entry points for fast-paced development, starting a QA/testing practice is challenging. Our strategy is to focus on two areas: test automation; and building a testing community. We’re hiring people to coordinate these two areas.

As QA Lead, I have created an example of what I believe to be the best available test automation “stack”, to pave the way and start the process of what I intend to be a reference implementation, an industry standard for high-quality browser test automation. We’re now hiring a QA Engineer whose primary responsibility will be to create and maintain browser-level test automation. In the course of creating those automated tests, we will be improving our use of the source code repository recently migrated from Subversion to git, we will be improving the beta labs test environment, and we will be expanding the use of our Continuous Integration in Jenkins.

But test automation isn’t everything, and we also have an opportunity to apply the Wikimedia community’s expertise in online volunteer collaboration to software quality. We’ve already started to explore this path with success: in May, we collaborated with Weekend Testing to validate the new frequent release schedule of MediaWiki to Wikimedia sites. Weekend Testing is an established global group of professional software testers who gather online every month for a different testing project, and testing Mediawiki versions on Wikipedia was a complex effort, executed well. In June, we collaborated with OpenHatch.org to test a near-final version of the new Article Feedback system that will be released to all of Wikipedia in the coming weeks. OpenHatch is an organization
dedicated to matching interested participants in open source software to projects that need participants. This was the first testing project for OpenHatch, and it went well; Article Feedback is much improved because of it.

We are now hiring a Volunteer QA Coordinator, who will be working to create a culture of quality testing and investigation of software related to Wikimedia, both within the Wikimedia community itself, and in collaboration with the greater software testing culture. And we are already planning future activities with both Weekend Testing and OpenHatch.

My first few months as QA Lead at the Wikimedia Foundation have been devoted to creating an environment where the QA Engineer and the Volunteer QA Coordinator will thrive. I am really looking forward to collaborating with the talented people we will hire for these roles. My own role will be shifting as these new practices start to take hold. I will be looking to the future, to bring in innovative and creative approaches to software QA and testing of the highest possible quality.

Chris McMahon
QA Lead Engineer

Comments are closed.