Over the next couple of years, the Wikimedia Foundation will be building capacity among program leaders around evaluation and program design. A better understanding of how to increase impact through better planning, execution and evaluation of programs and activities will help us to move a step closer to achieving our mission of offering a free, high quality encyclopedia to our readers around the world.

With this in mind, we are pleased to announce the first Program Evaluation and Design Workshop, on 22-23 June 2013 in Budapest, Hungary.

We have only 20 slots available for this workshop and the application deadline ends on May 17th. This two-day event will be followed by a pre-conference workshop at Wikimania 2013. Ideally, applicants would commit to attending both events.

Our long-term goals for the workshop are:

• Participants will gain a basic shared understanding of program evaluation
• Participants will work collaboratively to map and prioritize measurable outcomes, beginning with a focus on the most common programs and activities
• Participants will gain increased fluency in common language of evaluation (i.e. goals versus objectives, inputs and outputs versus outcomes and impact)
• Participants will learn and practice how to extract and report data using the UserMetrics API
• Participants will commit to working as a community of evaluation leaders who will implement evaluation strategies in their programs and activities and report back at the pre-conference workshop at Wikimania 2013
• …and participants will have a lot of fun and enjoy networking with other program leaders!

We will publish a detailed agenda for the event in Budapest soon on Meta-Wiki.

During the workshop in Budapest, we will only have a limited amount of time. Therefore, we will be focusing on the some of the more common programs and activities:

• Wikipedia editing workshops where participants learn how to or actively edit (i.e. edit-a-thon, wikiparty, hands-on Wikipedia workshop)
• Content donations through partnerships with galleries, libraries, archives and museums (GLAMs) and related organizations
• Wiki Takes/Expeditions where volunteers participate in day-long or weekend events to photograph site specific content
• Wiki Loves Monuments, which takes place in September
• Education program and classroom editing where volunteers support educators who have students editing Wikipedia in the classroom
• Writing competitions, which generally take place online in the form of contests, the WikiCup  and other challenges – often engaging experienced editors to improve content.

Contributors who play an active role in planning and executing programs and activities as described above in the Wikimedia community are highly encouraged to apply. Your experience and knowledge will make this workshop a success!

Hotels, flights and other transportation costs will be the responsibility of your chapter; the Wikimedia Foundation will provide the venue, handouts, breakfasts, light lunches, and a dinner for all participants on Saturday. If you’re not affiliated with a chapter and cannot afford to attend the event, please email me after you apply – we have a small amount of money set aside for those cases.

Remember, applications are open until May 17. You can apply via this Google Form.

Thanks for your interest, and I look forward to a great group of participants!

Sarah Stierch, Program Evaluation and Design Community Coordinator, Wikimedia Foundation

The English Wikipedia and its other language and sister projects are some of the most popular websites in the world, created, edited and sustained by volunteers. So you might not be surprised to learn that volunteers handle all the emails and inquiries about the sites, what is called Wikimedia’s Open-source Ticket Request System (OTRS), powered by the Volunteer Response Team. We’re very proud of the work the teams handle. This past July, Wikipedia was ranked first in the American Consumer Satisfaction Index for the third straight year.

For the first time we have compiled a statistical report of general activity of emails sent to our OTRS system for 2012. Emails sent to OTRS are assigned “queues” based upon the nature of the request, the language of the request, and the Wikimedia project it might represent. This report reflects queues that are not related to the Wikimedia Foundation, Wikimedia Chapters, GLAM, or any other project that may share space on OTRS. It also does not reflect spam and irrelevant email processing.

Emails are processed as “successful,” meaning that the response was satisfactory to the email; “unsuccessful,” meaning that there was no resolution to the email; or “no response needed,” meaning that the email did not merit a reply as part of processing. In total the information queues received 39,729 emails that were relevant to subject, and 34,799 of those were closed as successful, with 4,357 closed unsuccessful and 573 as no response needed.

This report provides an interesting look at only a fraction of what processing emails to Wikimedia involves. Before being answered, the mail has to be in the right place, in the right language, and relevant to Wikimedia projects. Volunteers come from all background on Wikimedia projects and like with editing, volunteers are free to choose their assignments.

While we take pride in our accomplishments so far, we look forward future statistical reports to help us learn and develop our customer service further. For more information or to get involved, please see this link: https://meta.wikimedia.org/wiki/OTRS

Keegan Peterzell, Volunteer Response Team

Improving the quality of articles has long been one of the primary aims of contributors to Wikipedia, and is one of the Wikimedia movement’s 2010-15 strategic priorities, but measuring it objectively has remained a challenge. In 2005, Nature famously reported that Wikipedia articles on scientific topics contained just four errors per article on average, compared to three errors per article in the online edition of Encyclopaedia Britannica. Britannica objected to the report, but Nature stood by it, and the report remains widely cited today.

Since that time, however, there have been relatively few independent analyses of Wikipedia article quality, despite the enormous growth of the project. Wikipedia today counts more than 23 million articles across languages (more than 4 million articles in the English Wikipedia alone) compared to 3.7 million total articles in 2005; today it ranks 6th by overall traffic according to Alexa, while it ranked 37th in 2005.

With increase in size and reach, how has quality evolved? How does Wikipedia compare today to other online encyclopedias, quality-wise? And what are good methods to measure the quality of encyclopedic articles?

The Wikimedia Foundation is announcing the release of a pilot study conducted by Epic, an e-learning consultancy, in partnership with Oxford University – “Assessing the Accuracy and Quality of Wikipedia Entries Compared to Popular Online Alternative Encyclopaedias: A Preliminary Comparative Study Across Disciplines in English, Spanish and Arabic.”

The study compared a sample of English Wikipedia articles to equivalent articles in Encyclopaedia Britannica, Spanish Wikipedia to Enciclonet, and Arabic Wikipedia to Mawsoah and Arab Encyclopaedia. 22 articles in the sample were blind-assessed by 2 to 3 native speaking academic experts each, both quantitatively and qualitatively.

The small size of the sample does not allow us to generalize the results to Wikipedia as a whole. However, as a pilot primarily focused on methodology, the study offers new insights into the design of a protocol for expert assessment of encyclopedic contents. For our editor community and for the Foundation, which commissioned the study in 2011, it also offers evidence to inform the design of quality assessment mechanisms and quality metrics that may be used on Wikipedia itself.

The results suggest that Wikipedia articles in this sample scored higher altogether in each of the three languages, and fared particularly well in categories of accuracy and references. As the report notes, the English Wikipedia fared well in this sample against Encyclopaedia Britannica in terms of accuracy, references and overall judgement, with little differences between the two on style and overall quality score. Similar results were found when comparing Wikipedia articles in Spanish to Enciclonet. In Arabic, Mawsoah and Arab Encyclopaedia articles scored higher on style than Wikipedia, but no significant differences were found on accuracy, references, overall judgment and overall quality score. None of the encyclopedias considered in this study were rated highly by the academics in terms of suitability for citation in academic publications.

We hope that the results of this study will encourage further independent research on the quality of Wikipedia articles. To this end, Epic and Oxford University are releasing the full version of the report of this study under a Creative Commons Attribution-Share Alike license. They have announced the report here and have released an anonymized dataset under a Creative Commons Zero dedication. The team welcomes comments and feedback on the talk page of the project.

We are very encouraged by the results for this small sample of Wikipedia articles in three languages. While pointing the way forward for further research, these results affirm the quality of the collaborative work of our editor community.

Dario Taraborelli, Senior Research Analyst

 

Siete años tras “Nature”, estudio piloto compara favorablemente a Wikipedia frente a otras enciclopedias en tres diferentes lenguas.

Hace tiempo que mejorar la calidad de los artículos es uno de los principales objetivos de los editores de Wikipedia. Es además una de las prioridades estratégicas del movimiento Wikimedia para 2010-2015, pero la capacidad de medir objetivamente este aspecto continúa siendo un desafío. En 2005, una famosa publicación de la revista “Nature” encontró que Wikipedia contenía un promedio de sólo cuatro errores por artículo sobre temas científicos contra los tres por artículo de la edición en línea de la Enciclopedia Británica. Enciclopedia Británica cuestionó el trabajo pero Nature lo reivindicó y continúa siendo citado con frecuencia hasta el día de hoy.

Desde entonces, sin embargo, hubo relativamente pocos análisis independientes de la calidad de los artículos de Wikipedia, esto a pesar del enorme crecimiento del proyecto. Wikipedia cuenta hoy con más de 23 millones de artículos en todos los idiomas (más de cuatro millones sólo en inglés) frente a los 3,7 millones de artículos en total que tenía en 2005. Hoy es el sexto sitio con mayor tráfico general según las estadísticas de Alexa, cuando en 2005 ocupaba el puesto 37. ¿Cómo evolucionó la calidad con este incremento de alcance y tamaño? ¿Cómo se compara hoy la calidad de los artículos de Wikipedia con otras enciclopedias en línea? ¿Qué métodos son apropiados para medir la calidad de un artículo enciclopédico?

La Fundación Wikimedia anuncia el lanzamiento de un estudio piloto realizado por Epic, una consultora de enseñanza en línea, en colaboración con la Universidad de Oxford: “Assessing the Accuracy and Quality of Wikipedia Entries Compared to Popular Online Alternative Encyclopaedias: A Preliminary Comparative Study Across Disciplines in English, Spanish and Arabic” (“Evaluación de la exactitud y calidad de las entradas de Wikipedia en comparación con otras conocidas enciclopedias alternativas en línea: un estudio preliminar comparativo interdisciplinario en inglés, español y árabe”).

El estudio compara una muestra de artículos de Wikipedia en inglés con sus equivalentes en la Enciclopedia Británica, Wikipedia en español con Enciclonet, y Wikipedia en árabe con Mawsoah y la Enciclopedia Árabe. 22 artículos de cada una de estas obras fueron presentados a dos o tres expertos académicos hablantes nativos de estas lenguas, quienes las evaluaron en términos cuantitativos y cualitativos.

Lo pequeño de la muestra nos impide generalizar los resultados a toda Wikipedia. Sin embargo, desde lo metodológico, el estudio ofrece nuevas líneas para el diseño de un protocolo que permita la revisión por expertos de contenido enciclopédico. También brinda a nuestra comunidad de editores y a la Fundación, que encargó el estudio en 2011, información para respaldar el diseño de mecanismos de control y medición de calidad que pueden ser usados en la propia Wikipedia.

Los resultados sugieren que los artículos de Wikipedia muestreados tienen en general un puntaje superior a sus contrapartes en los tres idiomas evaluados, con un desempeño especialmente bueno en cuanto a exactitud y provisión de referencias. Según destaca el informe Wikipedia en inglés se compara positivamente frente a la Enciclopedia Británica en términos de exactitud, referencia y juicio general, con una pequeña diferencia de puntaje entre ambas en estilo y calidad general. Los resultados de la comparación entre Wikipedia en español y Enciclonet fueron similares. En árabe, los artículos de Mawsoah y la Enciclopedia Árabe superaron a Wikipedia en cuanto a estilo, pero no se encontraron diferencias significativas en exactitud, referencias, juicio ni calidad general. Los expertos no consideraron que ninguna de las enciclopedias evaluadas fuera superior a las demás en cuanto a la posibilidad de cita en publicaciones académicas.

Esperamos que los resultados del estudio incentiven posteriores investigaciones independientes sobre la calidad de los artículos de Wikipedia. Para contribuir a ese fin Epic y la Universidad de Oxford publican la versión completa del informe con licencia Creative Commons Atribución-CompartirIgual. Con licencia Creative Commons Zero se publica también una colección de información anónima generada por el estudio. El equipo de trabajo espera comentarios y retroalimentación en la página de discusión del proyecto.

Estamos muy motivados por los resultados de esta pequeña muestra de artículos de Wikipedia en tres idiomas. Aún cuando abren un camino para la investigación futura, estos resultados confirman la calidad del trabajo colaborativo de nuestra comunidad de editores.

Dario Taraborelli, analista de investigación senior

 

بعد سبع سنوات من دراسة مجلة نيتشر، دراسة جديدة تقارن محتويات ويكيبيديا بموسوعات أخرى بثلاث لغات

إن تطوير جودة المحتويات هو أحد الأهداف الرئيسية للمساهمين في ويكيبيديا، وأحد أهداف الخطة الخمسية الاستراتيجية لحركة ويكيبميديا بين الأعوام ٢٠١٠-٢٠١٥، إلا أن قياس تلك الأهداف بشكل موضوعي كان ولازال أحد التحديات القائمة. في عام ٢٠٠٥ قامت مجلت نيتشر بنشر مقالة عرضت بأن مقالات ويكيبيديا احتوت ٤ أخطاء بالمعدل في مقابل ٣ أخطاء في مقالات موسوعة بريتانيكا على الإنترنت. اعترضت بريتانيكا على التقرير إلا أن مجلة نيتشر أصرت عليه، ولا زال التقرير واسع الانتشار حتى اليوم.

منذ ذلك الحين ظهر فقط بعض الدراسات التحليلية عن جودة محتويات ويكيبيديا، على الرغم من التوسع الكبير للمشروع. يبلغ تعداد مقالات ويكيبيديا اليوم ما يزيد على ٢٣ مليون مقالة عبر اللغات المتعددة (أكثر من ٤ مليون مقالة منها باللغة الإنكليزية وحدها) بالمقارنة مع ٣.٧ مقالة بالمجموع في عام ٢٠٠٥، تحتل ويكيبيديا اليوم وفقا لترتيب موقع ألكسا المركز السادس من حيث عدد الزيارات، بينما كان ترتيبها ٣٧ في عام ٢٠٠٥.

ومع الزيادة في الحجم والانتشار، فيكون التساؤل المطروح عن تغير جودة المحتويات؟ كيف من الممكن مقارنة ويكيبيديا اليوم بالموسوعات الأخرى المنتشرة على الإنترنت من حيث الجودة؟ وما هي الطرق المثلى لقياس جودة المقالات الموسوعية؟

وهنا تعلن مؤسسة ويكيميديا عن إطلاق دراسة أولية تم القيام بها من قبل مجموعة إيبك الاستشارية بالمشاركة مع جامعة أوكسفورد تحت عنوان “تقييم دقة وجودة مقالات ويكيبيديا بالمقارنة مع موسوعات الإنترنت المنتشرة الأخرى : دراسة مقارنة أولية باللغات الإنكليزية والإسبانية والعربية”

قامت الدراسة بمقارنة نماذج من ويكيبيديا الإنكليزية مع مقالات مقابلة من موسوعة بريتانيكا، وويكيبيديا الإسبانية بمقالات مقابلة من موسوعة إينسيسلونيت، وويكيبيديا العربية مع الموسوعة العربية العالمية، والموسوعة العربية، حيث تم تقييم عينة من ٢٢ مقالة من قبل ٢ – ٣ متحدثين أصليين باللغات الثلاث من المجتمع الأكاديمي وذلك من حيث الكم والجودة.

إن حجم العينة الصغير لا يسمح بتعميم النتائج على ويكيبيديا عموما. إلا أن الدراسة الأولية ركزت بشكل رئيسي على الطريقة، كما أن الدراسة طرحت تصميم جديد لتقييم الخبراء للمحتويات الموسوعية. كما أن الدراسة تقدم لمجتمع ويكيبيديا ولمؤسسة ويكيميديا التي مولت الدراسة في عام ٢٠١١ دليلا يساعد على تصميم آليات لتقييم الجودة ووضع معايير لها لاستخدامها على ويكيبيديا نفسها.

تلخص الدراسة إلى أن مقالات ويكيبيديا سجلت علامات أعلى بشكل عام في كل من اللغات الثلاث، وتميزت بشكل خاص في فئتي الدقة والمراجع المستخدمة. وكما يشير التقيري إلى ويكيبيديا الإنكليزية حققت علامات جيدة مقابل موسوعة بريتانيكا من ناحية الدقة واستخدام المراجع والتقييم العام مع فروقات صغيرة بين من حيث التنسيق وعلامة الجودة الكلية. كما أن نتائج مماثلة تم الوصول إليها عند مقارنة ويكيبيديا الإسبانية مع موسوعة إينسيسلونيت. وفي اللغة العربية فقد حققت الموسوعة العربية العالمية والموسوعة العربية علامات أعلى من ويكيبيديا من حيث التنسيق، لكن لم يكن هناك أي فروقات من حيث الدقة، استخدام المراجع، أو التقييم الكلي للجودة. ولم تحصل أي من الموسوعات في هذه الدراسة على علامة عالية من حيث قابليتها للاستخدام كمرجع في الأبحاث الأكاديمية.

نحن نأمل بأن نتائج هذه الدراسة ستشجع أبحاث مستقلة أخرى حول مواضيع تقييم جودة مقالات ويكيبيديا. إن إيبك وجامعة أوكسفورد تنشران النسخة الكاملة من التقرير تحت رخصة المشاع الإبداعي. كما تم نشر نسخة ببيانات مجهولة الأسماء تم توليدها من قبل هذه القائمة تحت رخصة المشاع الإبداعي صفر. إن فريق الدراسة يرحب بالملاحظات والتقييم على صفحة نقاش المشروع.

لقد شجعتنا هذه النتائج عن مقالات ويكيبيديا بلغات ثلاث بشكل كبير. وبالإضافة إلى أنها تسهل الطريق إلى أبحاث مستقبلية أخرى، فإن هذه النتائج تؤكد على جودة العمل المشترك لمحرري مجتمع ويكيبيديا.

Dario Taraborelli, Senior Research Analyst

Since December 2011, the Wikimedia Foundation has been testing a new version of the Article Feedback Tool, a feature first introduced on the English Wikipedia in 2010. The goal of version 5 (AFTv5) is to engage Wikipedia readers to become more active contributors, by inviting them to provide feedback on articles they read, and encouraging them to become editors over time. 

Early tests of AFTv5 helped us answer the question of what design of the tool produces a desirable balance between volume and usefulness of the feedback collected. In this post we report results from two additional experiments designed to answer the following questions:

1. Does a prominent invitation to use the tool affect the usefulness of submitted feedback?
2. How does an invitation to leave feedback affect the conversion of readers into editors?

Our findings suggest that a prominent invitation to post feedback converts a significant number of readers into editors. These new editors appear less productive than other first-time Wikipedians; but their feedback appears just as useful, as below. These findings suggest that article feedback can increase the number of new editors on Wikipedia and can also help existing editors improve the encyclopedia based on reader feedback.

Prominence of Feedback Invitation

(more…)

Wikipedia editors are hardworking volunteers who have created the most extraordinary knowledge resource in history. Many contributors have made tens of thousands of edits. Some have made hundreds of thousands. But sometimes, even the most seasoned editor could do with a bit of motivation.

The WikiCup is one such source of motivation, a friendly editing competition developed by the volunteer community with the goal to improve content and make editing more fun.

The Cup runs annually from January to October, with five rounds of elimination. Points are awarded each round for contributing different types of content, including Featured Articles (FAs), Good Articles (GAs), Did You Know’s (DYKs) and Featured Lists, among other article quality categories. Administration of the event is handled by judges, who also resolve disputes and review talk page discussions. This year’s judges are Josh Milburn (User:J Milburn) and Eddie Erhart (User:The ed17), both of whom started judging in 2009.

While a lot of WikiCup contestants are already prolific editors, Erhart believes the competition is still very effective in driving content creation. “[I enjoy] seeing these editors go out and add boatloads of content to the encyclopedia,” he said. “Many would have been doing this anyway, but the Cup provides an incentive for them to do more. I think the idea of a competition is a strong motivating factor to go out and improve content.”

In this year’s WikiCup, Stefano Magliocco (User:Grapple X) led the first two rounds mostly thanks to his GAs about The X-Files and Millenium television series. He says he initially entered the Cup as a means to keep himself motivated. “I had a lot of stuff planned, but I usually find myself losing focus on things over time,” said Magliocco. “I’ve been doing a lot of work for The X-Files WikiProject, and I had planned out a lot of long-term projects. The Cup seemed like a good means of lighting a fire under my arse to get these done.”

Since Magliocco has entered the Cup, he has noticed an increase in not only the quantity, but also the quality of his contributions. “I’ve generally had a strong burst of activity at the start of each round, where my normal work rate doubles or triples,” he said. “From there, it’s really just a case of the Cup motivating me to edit better, rather than more.”

Magliocco feels that encouraging the creation of GAs is one of the main ways the WikiCup accomplishes its stated mission of improving content on Wikipedia. “I think the level of investment versus reward given for the promotion of GAs has definitely helped the project as a whole,” said Magliocco, who likes to keep track of the ratio of GAs to total articles. “At the minute, about 1 in 275 articles are of GA status, whereas it was 1 in 280 at the start of the Cup.”

Last year’s WikiCup winner, Andrew Hink (User:Hurricanehink), agreed that increasing the number of GAs is an important step in improving the quality of Wikipedia. “I think [having 500,000 GAs] is very much in the realm of possibility in the next 10 years,” he said. “As long as it’s all well-cited, all well-written, that’s a good goal to have, and it’s very doable.”

(more…)

Wikipedia Education Program participants from the United States added more than three times as much quality content as regular new users, a quantitative analysis shows.

In the Wikipedia Education Program, professors assign their students to edit Wikipedia articles as a grade for class, assisted by volunteer Wikipedia Ambassadors. In fall 2011, 55 courses participated in the program in the United States, with students editing articles on the English Wikipedia. On average, these students added 1855 bytes of content that stayed on Wikipedia, compared to only 491 for a randomly chosen sample of new users who joined English Wikipedia in September 2011. These numbers establish that students who participate in the Wikipedia Education Program contribute significantly more quality content that stays on Wikipedia than other new users.

Examining the distribution of content that survived on Wikipedia for both of these groups, we found that almost half of the Wikipedia Education Program participants added 1,000 or more bytes that stayed on Wikipedia in the first six months. In contrast, more than half of the random sample of new editors added no content that stayed on Wikipedia in the first six months. The targeted recruitment of students, combined with the support provided by the Ambassador Program and instructors, results in a much larger percentage of new editors who contribute quality content to Wikipedia.

To understand the collective impact of the Wikipedia Education Program in fall 2011, we compared the amount of content students added to Wikipedia to the content added by the random sample of new editors. The numbers show that the 920 student editors who participated in the program in fall 2011 added the same amount of content as 2250 typical new editors (editors are defined as users who made at least one edit to an article). In terms of new content, students have twice the impact as typical new editors.

An important consideration for any outreach project is editor retention. Data showed that students who are introduced to editing Wikipedia through the U.S. Education Program are just as likely to continue editing as any other newcomer.

The Wikipedia Education Program has now grown to Egypt, Brazil and other regions beyond North America. With an increased global presence, measuring and understanding the contributions of new student editors (and how they differ from other new users that join Wikipedia) has gained importance. Establishing a common metric for measuring the impact of the Wikipedia Education Program on various Wikipedias is another key motivation for a quantitative study.

There’s a lot more work to be done on measuring the program’s impact. So, stay tuned for more information about these metrics.

Methodology for this research can be found at: http://meta.wikimedia.org/wiki/Research:Wikipedia_Education_Program_evaluation#Methods

Ayush Khanna, Data Analyst, Global Development

(with input from Mani Pande, Head of Global Development Research)

 

The Wikimedia Foundation, in collaboration with editors of the English Wikipedia, is developing a tool to enable readers to contribute productively to building the encyclopedia. To that end, we started development of a new version of the Article Feedback Tool (known as AFTv5) in October 2011. The original version of the tool, which allows readers to rate articles based on a star system, launched in 2010. The new version invites readers to write comments that might help editors improve Wikipedia articles. We hope that this tool will contribute to the Wikimedia movement’s strategic goals of increasing participation and improving quality.

Testing new feedback forms

On December 22, 2011, we started testing three different designs for the AFTv5 feedback forms:

• Option 1: Did you find what you were looking for? (shown above)
• Option 2: Make a suggestion, give praise, report a problem or ask a question
• Option 3: Rate this article

The purpose of this first experiment was to measure the type, usefulness and volume of feedback posted with these feedback forms. For example, does asking a reader to describe what they were looking for (option 1) provide more actionable feedback than asking them to make a suggestion (option 2)?

We enabled AFTv5 on a small, randomly selected set (0.6%) of articles on the English Wikipedia, as well as a second set of high-traffic or semi-protected articles. A feedback form, randomly selected from the above three options, was placed at the bottom of each page. The feedback form was also accessible via a link docked on the bottom right corner of the page.  The resulting comments were then analyzed along a number of dimensions.

(more…)

(For the Portuguese version, please see the Wikimedia Brazil site.)

It was during Wikimania 2011, in a small restaurant in Haifa, when the news was announced: the largest popular computer manufacturer in Brazil, Grupo Positivo, is interested in installing an offline Portuguese Wikipedia version in their products. All of us from Wikimedia Brazil who were present got excited because of the tremendous potential of such a distribution in spreading the free encyclopedia and its mission around Brazil. In other words, this meant the Portuguese Wikipedia for approximately 13% of the national market of personal computers and with a greater penetration in the lower-income strata.

Despite the good news, a race against time began. It was necessary to prepare the offline version of the Portuguese Wikipedia, with 5000 articles of good quality, within a very short time: March 2012. The challenge was huge and to overcome it we needed to step on the gas.

The list of 5000 articles which were critical to include in the offline version was created in only three months, with the great assistance of Wikimedia Brazil volunteers. But the volunteers found that the quality of these articles still was not high enough: they were in desperate need of improvement before being taken offline. It was then we had the idea of hosting our own “Grand Prix” – like the famous auto race. No cars and no laps, but with articles to be improved and many awards for the “pilots” who accept this challenge. Thus began the “I GP Wikimedia Brazil,” where each improved article is a completed lap.

The take-off will begin in January 2012, and it is very easy to attend! Just subscribe to one of the existing teams or join a new team. The registration will last until January 7. At the moment of publishing this blog, we have 51 subscribers divided into 15 teams, but the goal is to have at least 100 participants. After all, this is a Grand Prix where everyone wins!

Prizes will be distributed as teams improve the quality of the articles included in the list. There are buttons, stickers, notebooks and t-shirts with the brand of Wikipedia, as well as trophies and medals on the userpages of the participants. The rules of the award will be released soon after the formation of the teams, but we know that the biggest prize is the offline version of Wikipedia in Portuguese!

Imagine a world in which every single human being can freely share in the sum of all knowledge. That’s our commitment. Imagine, now, a Brazil where thousands of people – some of them even without access to Internet – will share a little sum of this knowledge. This is what we will do. Join a team and participate of this Grand Prix too!

(Written by the Wikimedia Brasil Community)

We’re happy to announce that the Wikimedia Foundation has started testing a new version of the Article Feedback Tool, to engage readers to help improve Wikipedia — and to become editors over time. We’re very excited about this new development, and look forward to getting more people to contribute to Wikipedia as a result.

Earlier this year, a first version of the Article Feedback Tool (“Rate this Page”) was rolled out to all articles on the English Wikipedia.  The idea behind this feature was two-fold: to provide a measurement of article quality from readers and to provide a potential on-ramp for these readers so that some may become editors.  We found through our analysis that while direct quality assessment is a very tricky matter (a rating of the Justin Bieber page says as much about the rater’s opinion of Bieber as it does about the quality of the article), the use of ratings as a form of low-barrier participation showed promise.  We also received plenty of feedback from the community around how we might improve this feature.

In October, we began development of the next generation of the tool (AFTv5).  Instead of focusing on explicit quality ratings, we shifted the direction of the tool towards finding new ways for readers to help build the encyclopedia.  So rather than primarily asking them to rate the quality of the article, we are asking readers for their input on how to improve the article. We are still testing different lightweight quality metrics, as well.

We are approaching this development in several phases.  The first phase, which went live today, is a test deployment of three new versions of the tool on approximately 10,000 randomly selected articles on the English Wikipedia and on a small number of manually selected articles. For examples, see Android, Wikipedia, and Global Warming.

Here is one of the three versions that are being tested:

This new version of the tool asks the reader whether they found what they were looking for, and if not, prompts them to explain what is missing.  The intent of this version is to provide editors with some idea of feedback on what readers are actually hoping to see when they read a Wikipedia article.  This information may then be used by the editing community when deciding how to improve the page.  The other two versions also ask for reader comments, but with different questions: the second version lets you make a suggestion, give praise, report a problem or ask a question; the third version lets you review the article. These new forms were developed by OmniTI, a web development firm, and were based on designs created by the Wikimedia Foundation in collaboration with the Wikipedia community. To learn more, visit the AFTv5 project page.

We are inviting members of the editing community to evaluate the quality of the comments coming in from each of these three versions of the feedback form.  The goal is to determine which of these versions is most effective at providing high quality feedback that can help improve articles.  Aaron Halfaker, a Wikipedia researcher from the University of Minnesota and a WMF contractor, has developed an evaluation tool that will enable Wikipedia editors to systematically evaluate the quality of the feedback provided. Assuming that these new versions provide constructive feedback, the next step would be to expose these comments in Wikipedia.  To that end, a “Feedback Page” is now under development with community input, and will provide a space where editors can view article feedback, moderate the comment stream, and promote the best contributions to the article talk page.

Oliver Keyes, a member of the English Wikipedia community, is under contract with the Wikimedia Foundation as a Community Liaison to involve editors in this project.  In this role, Oliver is moderating discussions, collecting feedback about the tool, and working with the development team to incorporate this feedback.  Many of the ideas that are in the current test versions came from discussions with these editors.  We will continue to work with the community very closely in the next stages of product design and development. If you’re part of the editing community and want to get involved, please email Oliver (okeyes at wikimedia dot org). Our immediate need is to help evaluate the comment streams generated by each option.  Very soon, we will also need editors to help us design the Feedback Page, which will be used to review and potentially act on the feedback comments.

We hope this new feature can help engage a broader community of readers to provide constructive feedback on articles, share what they know and contribute regularly on Wikipedia.

Howie Fung, Senior Product Manager

Fabrice Florin, Product Consultant

Much of Wikipedians’ efforts is devoted to ensuring the quality of the encyclopedia they are producing collaboratively – the community is constantly working to improve it. The effectiveness of this work has been recognized many times, perhaps most notably in a study published in 2005 by the scientific journal Nature which compared entries in the English Wikipedia with those in the online edition of Encyclopaedia Britannica. Nature reported four errors per Wikipedia entry and three per Encyclopaedia Britannica entry, a result that is still widely cited today even though Wikipedia is now more than twice as old, having matured in many ways.

The Wikimedia Foundation has commissioned a new small-scale study to examine the quality and accuracy of Wikipedia articles. This study, currently being undertaken by Epic, a UK-based e-learning company, and Oxford University, employs greater rigor than the Nature study, involves academics and scholars, and will examine more than just English language entries, and subjects other than solely science. Our hope is that the study’s findings will inspire and inform more extensive, independently funded research related to the quality of information found in Wikipedia and other free knowledge projects.

This project will explore methods to define a baseline for the quality of Wikipedia entries and to help the community identify shortcomings, as well as strategies to address them. Wikipedia has several advantages over commercially available online encyclopedias – it is freely accessible to hundreds of millions of users worldwide, it is available in over 270 languages, and it is updated at remarkable speed, relying on the ability of a vast number of non-paid contributors rather than the academic credentials of a few paid experts. However, errors do exist and concerns have been raised that articles may be colored by contributors’ personal opinions or misunderstandings. A comparative analysis of the quality of Wikipedia’s articles and other popular alternatives is crucial to identifying avenues for improvement.

Dario Taraborelli, Senior Research Analyst, Strategy

Tilman Bayer, Movement Communications