Vol: 3 • Issue: 4 • April 2013 [contribute] [archives]

Sentiment monitoring; Wikipedians and academics favor the same papers; UNESCO and systemic bias; How ideas flow on Wikiversity

With contributions by: Piotr Konieczny, Oren Bochman, Taha Yasseri, Jonathan T. Morgan and Tilman Bayer

Too good to be true? Detecting COI, Attacks and Neutrality using Sentiment Analysis

Traditional methods for detecting sentiment are less objective

Finn Årup Nielsen, Michael Etter and Lars Kai Hansen presented a technical report^[1] on an online service which they created to conduct real-time monitoring of Wikipedia articles of companies. It performs sentiment analysis of edits, filtered by companies and editors. Sentiment analysis is a new applied linguistics technology which is being used in a number of tasks ranging from author profiling to detecting fake reviews on online retailers. The form of visualization provided by this tool can easily detect deviation from linguistic neutrality. However, as the authors point out, this analysis only gives a robust picture when used statistically and is more prone to mistakes when operating within a limited scope.

The service monitors recent changes using an IRC stream and detects company-related articles from a small hand-built list. It then retrieves the current version using the MediaWiki API and performs sentiment analysis using the AFINN sentiment-annotated word list. The project was developed by integrating a number of open source components such as NLTK and CouchDB. Unfortunately, the source code has not been made available and the service can only run queries on the shortlisted companies which will limit the impact of this report on future Wikipedia research. However, it seems to have potential as a tool for detecting COI edits that tend to tip neutrality by adding excess praise or attacks which tip the content in the other direction. We hope the researchers will open-source this tool like their prior work on the AFINN data-set, or at least provide some UI to query articles not included in the original research.

“A Comparative Study of Academic impact and Wikipedia Ranking”

A paper^[2] with this title investigates the relation between the scientific reputation of scientific items (authors, papers, and keywords) and the impact of the same items on Wikipedia articles. The sample of scientific items is made of the entries in the ACM digital library including more than 100 k papers, 150 k authors and 35 k keywords. However, only a tiny subset of these could be found in English Wikipedia pages (the authors considered all Wikipedia pages in the English edition which contain at least two mentions of any of the scientific items in the sample). The academic reputation is calculated based on three criteria: frequency of appearance, number of citations each item receives from the others, and PageRank calculated on the citation network. The Wikipedia ranking is based on three popularity measures of all the pages that have mentioned the item: number of mentions, sum over PageRank of all the mentioning pages, and sum over in-degrees of all the mentioning pages in Wikipedia’s hyperlink network.

These 3 times 3 choices give 9 combinations of academic ranking and Wikipedia ranking for 3 types of scientific entities (authors, papers, keywords). All these 27 pairs are shown to be correlated according to Spearman’s Rank Correlation, indicating that in general Wikipedia mentions are non-randomly driven by scientific reputation. However, most of the combinations are less significant. Surprisingly, the most relevant Wikipedia ranking criterion turns out to be the pure total number of mentions, compared to the more sophisticated ones, i.e., PageRank and in-degree measures.

In a separate part, authors define two sets of scientific items, those which are mentioned in Wikipedia, and those which are not mentioned at all (the latter is larger in size by a factor of 2 for keywords, 100 for authors, and 300 for papers). They show that for all 3 types, the set of items which are mentioned in Wikipedia have a better academic rank on average.

1970s UNESCO debate applied to Wikipedia’s systemic bias in the case of Cambodia

According to the author, the Angkor period dominates Cambodian historiography as well as tourist attention in the country, corresponding to an unevenness in the quality of Wikipedia articles

An article^[3] in the Journal of the American Society for Information Science and Technology rated the quality of Wikipedia articles on the history of Cambodia (defined as those linked in the corresponding navbox, using four measures: 1) the article’s ratio of the number of citations per the number of words, 2) the number of editors who have commented on its talk page, 3) the quality of the cited sources, rated in five categories (“traditional reference” like print encyclopedias, “news reports” including both newspapers and news websites such as CNN, “academic periodicals”, “books”, and “miscellany” like reports by governments or NGOs, or personal websites) and 4) “the number of unique authors cited”, assuming that articles which are based on a larger variety of perspectives are of higher quality. The findings are summarized as follows:

The early history of Cambodia is represented by an extremely weak article, but there is an improvement in the articles dealing with the early kingdoms of Cambodia. The improvement ends abruptly with articles on the ‘dark age’ of Cambodia, the French Protectorate, the Japanese occupation, and early postindependence periods being of a much lower quality. Afterward, the quality picks up again with especially good articles on the American intervention in Cambodia, the Cambodian-Vietnamese War, and the People’s Republic of Kampuchea. However, the quality does not last; as we near contemporary times, the articles take another turn for the worse.

From this, the author concludes that “the Wikipedia community is unconsciously mimicking the general historiography of the country”, in particular a glorification of Angkor and other early kingdoms at the cost of later periods, and observes a “continuing dominance of the traditional historiographical narrative of Cambodian history in Wikipedia.” The subsequent section of the paper tries to put these results into the context of the historical debates in the late 1970s and early 1980s about the New World Information and Communication Order (NWICO), a suggested remedy for problems with the under-representation of the developing world in the media, put forth by a UNESCO commission in the MacBride report (1980):

Wikipedia provides access—it is free to use by anyone with an Internet connection, and print versions can also be distributed. But the whole thrust of the NWICO argument is that content matters and those who create content matter perhaps even more, with the commission stressing that countries needed to ‘achieve self-reliance in communication capacities and policies’ … Contrary to popular belief, in the new ‘information age’ content is, once again, the preserve of the few, not the many, and a geographically concentrated few at that.

The author’s argument is somewhat weakened by asserting erroneously that “there exists no Cambodian-language Wikipedia”, but generally aligns with other quantitative research that has found a geographic unevenness of coverage in Wikipedia. The author is an information studies professor at Singapore’s Nanyang Technological University and previously published a related paper in the same journal examining the Wikipedia article History of the Philippines, reviewed in the August issue: “The limits of amateur NPOV history“.

Reasons why wikilinks are added and removed

Julia Preusse, Jerome Kunegis, Matthias Thimm, Thomas Gottron and Steffen Staab investigate^[4] mechanisms of changes in a wiki that are of structural nature, i.e., which are a direct result of the wiki’s linking structure. They consider if the addition and removal of internal links between pages can be predicted using just information about the network connecting these articles. The study’s innovation lies in considering the removal of links, which account for a high proportion of removals and reverts. The authors performed an empirical study on Wikipedia, stating that traditional indicators of structural change used in the link analysis literature can be classified into four classes, which indicate growth, decay, stability and instability of links. These methods were then employed to identify the underlying reasons for individual additions and removals of knowledge links.

The network created by links between articles in Wikipedia is characterized by preferential attachment. Prior work on social networks has identified a phenomenon called “liability of newness”, in which new connections are more likely to be broken than older ones. To provide a better predictive model of link evolution the team considered five hypotheses:

1. Preferential attachment: The number of adjacent nodes is a good indicator for link addition.
2. Embedding : The embeddedness of a link is suitable to predict the appearance of links and the non-disappearance of existing links.
3. Reciprocity: The presence of a link makes the addition of a link in the opposite direction more likely and the removal of a reciprocal link less likely.
4. Liability of Newness: Old age of an edge or a node is a good indicator for link persistence.
5. Instability The less stable two nodes are, the less stable the link connecting them is, or would be if it does not exist.

To test these hypotheses, they created networks based on the history of the mainspace articles till 2011 of the top five Wikipedias after the English one. For example, in the French Wikipedia, 41.7 million links were added and 17.3 million removed during that time. The data was used to create a link creation predictor and a link removal predictor. These were then evaluated using the area under the receiver operating characteristic curve.

The results were that Preferential attachment and Embedding are good indicators of growth. Liability of Newness did not turn out to be a good indicator of link removal, but more of article instability. Reciprocity is also an indicator of growth, but is not as significant since most links in a wiki are not reciprocated.

Generation Z judges [[Generation Z]], questioning role of amphetamines

An article^[5] in the Journal of Information Science, titled “Understanding trust formation in digital information sources: The case of Wikipedia”, explores the criteria used by students to evaluate the credibility of Wikipedia articles. It contains an overview of various earlier studies about credibility judgments of Wikipedia articles (some of them reviewed previously in this space, example: “Quality of featured articles doesn’t always impress readers“).

The authors asked “20 second-year undergraduate students and 30 Master’s students” in information studies to first spend 20 minutes reading “a copy of a two-page Wikipedia article on Generation Z, a topic with which students were expected to have some familiarity”, and answer an open-ended question explaining how they would judge its trustworthiness. In a subsequent part, the respondents were asked to rank a list of factors for trustworthiness in case of “either (a) the topic of an assignment, or (b) a minor medical condition from which they were suffering”. One of the first findings was a “low pre-disposition to use [Wikipedia], possibly suggesting a propensity to distrust, grounded on debates and comments on the trustworthiness of Wikipedia” – possibly to the fact that the example article contained an example of vandalism, a fact highlighted by several respondents (e.g. “started off as a valid entry … due to citations strengthening this … however came to the last paragraph and the whole document was marred by the insert of ‘writing articles on Wikipedia while on amphetamines’ [as purported hobby of Generation Z members]… just feels that you can’t trust anything now”).

Among the given trustworthiness factors, the following were ranked most highly:

authorship, currency, references, expert recommendation and triangulation/verification, with usefulness just below this threshold. In other words, participants valued having articles that were written by experts on the subject, that were up to date, and that they perceived to be useful (content factors). … Interestingly these factors all seemed more or less equally important for both contexts, with the exception of references, which for predictable reasons were seen as having greater importance in the context of assignments.

Visualizing the “flow of ideas” on Wikiversity

In a conference paper titled “Analyzing the flow of ideas and profiles of contributors in an open learning community”^[6] (see also audience notes from the presentation), the authors construct a graph from the set of revisions of a set of Wikiversity pages, with two kind of edges: 1) “Update edges”, linking a page’s revision to the directly subsequent revision. These are understood as representing “knowledge flow over the course of the collaborative process on a single wiki page”. 2) “Hyperlink edges” between two revisions of different pages with a wikilink between them – but pointing in the opposite direction, because the idea is that they indicate knowledge flowing from the linked page to the linking page. By requiring the source node of a hyperlink edge “as the latest revision of the hyperlinked page at the moment of creation of the target revision”, both kinds of links point forward in time, resulting in a two-relational directed acyclic graph (DAG), which is “depicting the knowledge flow over time.” After filtering out “redundant” hyperlink edges and attaching authorship information to each node (page revision).

The authors apply this procedure to a set of Wikiversity articles in the area of medicine, starting with v:Gynecological History Taking. The results are interpreted as follows:

the beginning, short after the category medicine was founded, the authors in this category built up the basic structure of the knowledge domain. The main relations and idea flows between the learning materials were established early in the development of the domain. After that the authors have been focusing on elaborating the articles without introducing new important hyperlinks. The overall picture of the learning process in this domain suggests a divergent evolution of ideas after an initial period of mutual fertilization between different topics. This conforms to the idea of groups of learners that followed different interests in the medicine domain with little inter-group collaboration on the creation of new shared learning resource.

The method is subsequently applied to profile the activities of various users.

The authors have integrated these algorithms, including visualization tools, into a “network analytics workbench … used in the ongoing EU project SISOB which aims to measure the influence of science on society based on the analysis of (social) networks of researchers and created artifacts.”

In brief

“Wikipedia Vs. Encyclopedia Britannica: A Longitudinal Analysis”

The authors review^[7] how Wikipedia and Britannica coverage of topics related to several major corporations has changed in the past 6 years. They find unsurprisingly that Wikipedia coverage is usually much more detailed than that of Britannica; more interestingly, they note that one of the key differences is that Wikipedia focuses more on issues such as corporate social responsibilities and legal and ethical issues, whereas Britannica will focus more on traditional aspects such as financial results. They note that both encyclopedias, while striving towards some form of neutrality, contain non-neutral (“positively and negatively framed”) content, although it is more common to find it in Wikipedia. They also note that this content seemed to peak around 2008–2010, and attribute it to the negative views of major corporations common among the general public around that time, whose view was more likely to be represented on Wikipedia than on Britannica, also correlating this with the economic recession. The authors note that increasingly, knowledge available to the general public comes from social media collaboration projects such as Wikipedia, and are doubtful whether more traditional models like that of Britannica have a future. See also earlier coverage of a related paper by the same authors: “Are articles about companies too negative?” and of another where one of them (controversially) argued against the “bright line” rule on conflict of interest editing: “Wikipedia in the eyes of PR professionals“.

“Wikipedia uses in learning design: A literature review”

An article with this title^[8] presents a relatively useful literature review of publications about the “teaching with Wikipedia” approach. The authors analyzed several scholarly databases (not explaining, however, why the selected ones were chosen and others were not), finding 30 works on related themes, and selecting 24 of those. They provide a number of useful breakdowns (2/3 of the works deal with higher education, 1/3 with secondary, none with primary) and analyze expected learning outcomes (the most popular being learning research methodology), knowledge fields that the papers represented (mostly fields of social science), and an overview of student tasks. While containing few revelations, the paper is a solid example of a literature review of an emerging field, and contains a valuable observation that more research is needed on how Wikipedia is used by elementary school students.

Wikipedia assignment has positive impact on students’ “research persistence”

A paper by two Californian librarians, titled “From Audience to Authorship to Authority: Using Wikipedia to Strengthen Research and Critical Thinking Skills”^[9] and presented at the recent conference of the Association of College and Research Libraries (ACRL), describes the results of two case studies “in which Wikipedia was used as the platform for assignments” for students, one of which “had, overall, a positive impact on research persistence” of the students.

Co-authorship patterns around Pope Francis, and Boston bombing views

In his blog,^[10] Brian Keegan (known to readers of this research report for his previous research of Wikipedia’s coverage of breaking news events) provides a refreshing preview of his upcoming research with visualization of Co-authorship patterns around the Pope Francis article. The Social network analysis produced by factoring edits from 607 editors who worked on the new pope’s article and then adds all other articles they collaborated on since. The results which look like an abstract art masterpiece show a number of complex pattern in the data. However we will have to wait until the publication of the paper for these to be explained. Keegan does provide a number of teasers but you will have to visit his blog to read these. In another blog post,^[11] Keegan examines pageviews of various articles related to the Boston marathon bombings.

Mining content removed from articles on breaking news events.

A short paper^[12] accepted to the 2013 WWW conference describes a new tool designed for mining the information removed from Wikipedia articles during breaking news events. The Wikipedia Event Reporter identifies “bursts” of editing activity in an article, then uses machine learning techniques to identify sentences from the revision history of the article that were added during these bursts but which are not contained within the current version, and finally displays this information to the user—all in real time! The designers of the tool state that the Event Reporter will be useful for “a journalist or a student studying about history [who wants] a comprehensive view of an event, and not only the socially accepted final interpretation”. While Event Reporter looks to be both useful and intriguing, this reviewer challenges the assumptions behind the authors’ intended scenario of use. On Wikipedia, information about breaking news events is often removed because it is factually incorrect, not for the “sake of brevity”, out of considerations of political correctness, or other (possibly nefarious) social motives. The authors do not address the issue of determining factual accuracy in their paper—hopefully their intended audience (journalists!) will keep that issue in mind if they decide to re-publish the mined information. The reviewer would also like to have seen a performance evaluation of their Vector Machine Classifier, which relies on hand-labelled training data, included in the paper. Nonetheless, this seems to be a fascinating and very powerful piece of software. One cool future direction for the Event Reporter team might be to mine the content of the article talk page during and directly after these bursts as well, and employ the same classification technique to provide the end user with a better sense of why certain content was revised or removed.

Spam on the rise as reason for user blocks

User:Ironholds examined^[13] the English Wikipedia’s block log from 2006 to 2012 for the stated blocking reasons, and found “spam” being used more and more frequently.

10k birth places and 40k almae matres from Wikipedia biographies, human-vetted

Google has published^[14] “a human-judged dataset of two relations about public figures on Wikipedia: nearly 10,000 examples of “place of birth”, and over 40,000 examples of “attended or graduated from an institution”. Each of these was judged by at least 5 raters, and can be used to train or evaluate relation extraction systems.”

How Wikipedia’s Google matrix differs for politicians and artists

Continuing the authors’ research on the Google matrix of Wikipedia articles and links between them (earlier coverage: ‘Wikipedia communities’ as eigenvectors of its Google matrix“), an ArXiv preprint studies the “Time evolution of Wikipedia network ranking”^[15], finding among other things that “PageRank selection is dominated by politicians while 2DRank, which combines PageRank and CheiRank, gives more accent on personalities of arts”.

A Wikipedia search algorithm that emphasizes serendipity

A two-page paper to be presented at the upcoming WWW 2013 [2] conference explores algorithms for “Searching for Interestingness in Wikipedia and Yahoo! Answers” ^[16], or “Serendipitous search” – defined as “when a user with no a priori or totally unrelated intentions interacts with a system and acquires useful information”. The authors modify some standard information retrieval metrics by including sentiment analysis and a measure of a page’s quality – in the case of Wikipedia, “the number of dispute messages inserted by editors to require revisions”, which may be seen as questionable. The resulting two algorithms for ranking search results on both sites are tested for some popular search terms (drawn from the Google Zeitgeist lists), by asking test subjects to rank the results “for relevance, interestingness to the query, and interestingness regardless of the query”. In the end, the authors suggest that they be combined into a hybrid system.

Usability study recommends 18-point font for Wikipedia

An “experi­ment with 28 participants with dyslexia [comparing] reading speed, comprehension, and subjective readability” found “that font size has a significant effect on the readability and the understandability of the text, while line spacing does not”. On that basis, the four researchers from Barcelona “recommend using 18-point font size when designing web text for readers with dyslexia.^[17]

OpenSym, Wikisym, ClosedSym?

This year the WikiSym conference will be co-located with OpenSym. This marks a step forward from a conference focused mostly on CSCW application of wiki technology to a broader investigation of OpenCulture. This year the conferences will be collocated with Wikimania 2013 in Hong Kong. The WikiSym conference is funded in part by a grant by the WMF. However, as reported last month (Wikimedia funding for Wikisym ’13 despite open access concerns), there has been debate about the tension between the requirements of WMF on supporting open access research and the fact that the conference papers will be published by the ACM – a closed access provider. In two subsequent blog posts, the organizers explain why they have not been able^[18] to find an open publisher with a reputation comparable to the ACM for the 2013 proceedings, but formulate requirements for a suitable publisher for next year ^[19].

Wikimedia France research award winner announced

The French Wikimedia chapter has announced the winner of its research award: “Can history be open source? Wikipedia and the Future of the Past”,^[20] an influential 2006 essay by the historian Roy Rosenzweig, which received the most votes among the jury-selected five finalists. The prize money of € 2,500 will go to the Roy Rosenzweig Center for History and New Media, founded by the paper’s late author.

Provenance graphs

A conference paper by two computer scientists from the University of Newcastle^[21] presents code to convert metadata from Wikipedia revision history and user contribution pages (e.g. the author of a particular revision, or articles edited by an editor) into provenance data in the W3C‘s PROV-DM data model. The graph of revisions and editors is visualized. Code and examples are provided on Github.

References

Vol: 3 • Issue: 3 • March 2013 [contribute] [archives]

“Ignore all rules” in deletions; anonymity and groupthink; how readers react when shown talk pages

With contributions by: Amir E. Aharoni, Piotr Konieczny, Taha Yasseri, Oren Bochman, Heather Ford, Tilman Bayer, Giovanni Luca Ciampaglia, Daniel Mietchen.

Wikipedia’s “Ignore all rules” policy (IAR) is a double edged sword in deletion arguments

A beetle larva ignoring the rules while negotiating deletion with the frog.^{[mediasource 1]}

A paper presented at last month’s CSCW Conference, titled “Keeping eyes on the prize: officially sanctioned rule breaking in mass collaboration systems”^[1] observes that “Mass collaboration systems are often characterized as unstructured organizations lacking rule and order”, yet Wikipedia has a well developed body of policies to support it as an organization. Rule breaking in bureaucracies is a slippery slope quickly leading to potentially dangerous exceptions, so Wikipedia has a mechanism called “Ignore all rules” (WP:IAR) for officially sanctioned rule breaking. The researchers have considered IAR’s impact within the scope of deletion requests. The results show that the IAR policy has meaningful influences on deliberation outcomes, which rather than wreaking havoc, provides a positive, functional governance mechanism.

This paper is another welcome addition to the growing literature on AfD, examining the effectiveness of rule breaking using WP:IAR within these discussions. It starts with an in depth extermination of rule breaking within collaborative environments. Then these six hypotheses are postulated:

1. Invocation of WP:IAR in support of vote correlates with increased likelihood of the decision that the vote will be on the winning side.
2. This effect is expected to increase with the number of policies cited in the deletion proposal (since they may be contradicting each other).
3. Invoking IAR to override the deletion proposal’s policy citation tends to reduce the proposal’s likelihood of success.
4. When IAR is used together with another policy domain (e.g. Content/Conduct/Legal) as the proposal’s rationale, it will negate the proposal’s success.
5. Increased dissonance between policies arising in the discussion will increase the chance that the IAR argument will be successful.
6. IAR will increase in effectiveness as the policies invoked increase in complexity.

To test these, the researchers scoured AfD discussions starting from April 2006 to October 2008, collecting those where WP:IAR had been invoked. These were then supplemented by randomly drawing a control group from non IAR AfD discussions from the same date. The resulting dataset contained 555 AfD discussions. These were coded by Outcome, for Keep/Delete and IAR usage in Keep/Delete vote, Policy Match and Category Match. Each hypothesis and the control were fitted to a linear regression model. The results were as follows:

H1 was supported only in cases where IAR is used in keep vote, but showed insignificant impact as a delete argument. H2, H3 & H4 look for conditions in which IAR’s impact on the ultimate decision would be strengthened. H2 was supported only marginally; H3 was not supported; H4 was not supported and actually indicated that in the case where a keep voter has invoked IAR with another policy this will only increase the chance of a delete outcome! H5 and H6 consider if IAR fares better when pitted against increasingly contradictory or complicated policies and both of these are supported. Overall, the authors conclude that IAR plays a significant role in Wikipedia’s policies, and recommend its use to other communities. They point out that IAR is also an indicator of where policy is weak in addressing the community’s needs.

Activity of content translators on Wikipedia examined

The Japanese phrase “itterashai”, uttered by a budgerigar.^{[mediasource 2]}

This image of the flatworm Pseudorhabdosynochus morrhua has descriptions in currently 29 languages, i.e. 10% of Wikipedia languages.

Another CSCW paper titled “Could someone please translate this?”: activity analysis of Wikipedia article translation by non-experts”^[2] analyzes the work of a volunteer translator of Wikipedia articles. It goes into great detail: it breaks down the big translation task into many sub-activities, such as looking up complicated words in the source language, choosing the right translation, using editing software, etc. It presents all the activities according to the Activity theory methodology. Though there are other papers that deal with translation of Wikipedia content, it is the first paper to examine the actual volunteer translator’s activity.

Interestingly, this paper notes the importance of the Simple English Wikipedia several times, as a tool that may help people translate the content, with the assumption that the language of the main English Wikipedia may frequently be complex and challenging (this assumption is based on another paper, which compared the English and Simple English Wikipedias). It relies on the Simple English Wikipedia a bit too much, though; for example, it cites its main page as a source for some statistics, which would better be obtained directly from stats.wikimedia.org, Wikimedia’s main statistics site.

It has some shortcomings, which should be addressed in future works on the subject:

• It lists several possible definitions of “Wikipedia translation”: Translation of articles, with which it deals, and also translation of talk pages, translation of WikiProject pages, etc. It also mentions several software tools that are related to Wikipedia translation and multilinguality, such as WikiBhasha and Omnipedia. However, it notably omits any mention of MediaWiki’s Translate extension, which is used on the translatewiki.net website for translating of the user interface of MediaWiki and its many extensions, making MediaWiki one of the most thoroughly localized software packages ever, and also for various documents on Wikimedia sites, such as Meta, MediaWiki.org and Wikidata. Though they are certainly not identical, the latter workflow of translating documents is especially similar to the workflow of Wikipedia article translation. (Disclaimer: The reviewer is one of the developers of the Translate extension.)
• It provided pre-selected articles to translate to the subjects of the experiment. This may have been unavoidable in a first small controlled experiment, but it misses an important activity of volunteer translators in Wikipedia: selection of the article to translate. This is done in several ways, among which are:
  • Selection by translators themselves, based on their interests or other factors.
  • Projects such as [Translation of the week]].
  • Requests from other users who speak the target language.
  • Requests from users who speak other languages. Notable examples of these are the Wikipedia articles about Kurów, a town in Poland, and True Jesus Church, a Christian denomination, which are at least partially translated to nearly all languages in which a Wikipedia is available.
• It only deals with translation from English to other languages, but not with translation from other languages into English and other languages. For many reasons, English is not the only source for translation and this must be noted.
• The paper notes that one of the criteria for choosing the articles for the experiment was that the article content is representative of the general Wikipedia article language complexity”. It is not clear, however, how this was measured.
• New users of Wikipedia were chosen and not experienced editors. Testing with new users is valuable, but it would be useful to repeat the experiment with veteran Wikipedians.
• It claims that it found that paraphrasing machine translation can be a more desirable strategy for translating conceptual articles than biographical articles in Wikipedia, even though the English language might be more complex. This may be true, but it is unclear how such a bold statement could be made from such a small sample of source content.

Despite these shortcomings, this paper is valuable for several reasons:

• Opening the topic of close examination of Wikipedia’s translators work is important in itself.
• Its bibliography has many useful pointers to other articles about Wikipedia’s multilinguality and volunteer translation.
• Its high level of detail in analyzing the translators activity is commendable, and with some improvements, this methodology could be useful for people who design translation tools.
• Its particular comments about the special challenges of translation to Chinese should be very useful for optimizing future translation tools for this language. Of course, the experiment should be repeated with other languages, too.

Finally, the article promises further research and suggestions about building tools for translator support, which would be very interesting to read.

Comparison of collaborative editing in OpenStreetMap and Wikipedia

A preprint titled “Has OpenStreetMap a role in Digital Earth Applications?”^[3] studies OpenStreetMap, the wiki-based collaboratively editable map, as a predominant example of Volunteered Geographical Information projects. The paper addresses two main research questions: 1) How successful is the OSM project in providing spatial data and to which extent can it be compared to Wikipedia in this sense, 2) what are the main characteristics of OSM stemming from its crowd-sourced nature? The paper gives a very comprehensive overview of the work-flow of OSM, reviews the main characteristics of its collaborative mapping process very well, and tries to compare these characteristics with those of Wikipedia: In contrast with Wikipedia, the administrative structure of OSM is unknown and not very well defined within the community of its editors; however both platforms show the same Zipfian characteristics among their editors; a few editors are responsible for large numbers of contributions and many editors have only a few contributions. Although the criteria are quite different on the two platforms, the paper finds that the relative population of OSM Featured Objects is evidently larger than the ones of Wikipedia (Featured Articles). In the conclusion, the authors express that they “believe that OSM will continue its growth for the foreseeable future”. However, the route to this conclusion is not very well described in the manuscript.

Wikipedia’s coverage of breaking news stories is still a fertile field of research

The Chelyabinsk meteor on February 15 did not just leave its traces in the Ural region, but Wikipedia entries on the event had been started in 29 languages by the end of that day. Today, there are 44.

In MJ no more: Using Concurrent Wikipedia Edit Spikes with Social Network Plausibility Checks for Breaking News Detection^[4] by Thomas Steiner, Seth van Hooland and Ed Summers, the controversial (per WP:Recentism and WP:RS) field of breaking news articles is investigated. Motivated by the overloading of Wikipedia during the breaking of the news of Michael Jackson’s death, researcher Thomas Steiner created an open source exploratory tool called The Wikipedia Live Monitor. This tool allowed his team to examine clusters of related activity based on edit spikes in a 5 minute window within multiple streams fed by Wikipedia’s recent changes; Twitter Feeds; Google+ and Facebook. The main research question posed is: are edit spikes in Wikipedia, clustered with related social network activity, useful indicators for identifying breaking news events, and with what delay? By considering action along multiple streams, they are able to cross-check the plausibility of information being disseminated by many less reliable sources.

Their approach is based on prior work by S. Petrović, M. Osborne, and V. Lavrenko in Streaming First Story Detection with Application to Twitter, who used the document vector space model from classic information retrieval to cluster twitter feeds. But in this case the researchers are clustering multiple streams which can potentially hold far more information when a story breaks and can therefore detect these very quickly. While they could locate breaking news, they may need more work to optimize the timing parameters of the algorithm. Further research is planned into automating the classification of edits, which could reform future use of non-reliable sources.

A WikiSym 2012 paper titled Staying in the Loop: Structure and Dynamics of Wikipedia’s Breaking News Collaborations^[5] looked at the trajectory of article construction which captures the collaboration structure embedded in the creation of breaking news stories. They have shown that these stories, fueled by mass media and social networks, tend to create a social melting pot surrounding the editing of these events. A social network analysis of the relations between editors of breaking news stories located editors in diverse social roles, such as Creators, early contributors, the highly centralized activity coordinators (admins) and the marginal vandals and their tireless opponents, the spam fighting bots and recent changes patrollers. Another result is that most articles – those which are not breaking news stories – lack the dense creation trajectories found in breaking news stories.

Exposing talk page discussions leads to drop in perceived article quality

As once observed by Ward Cunningham, one important feature by which Wikipedians improved his invention, the wiki, was to introduce “a talk page or a discussion page behind every page, so you don’t actually have to see the discussion and it makes a much more finished product”. Yet surfacing this deliberation could engender trust in the process if the deliberation process appears fair, well-reasoned, and thorough. Alternatively, it could encourage doubts about content quality, especially if the process appears messy or biased. In a CSCW ’13 paper titled “Your process is showing: controversy management and perceived quality in wikipedia”,^[6] the researchers report on an experiment in which they found that exposing discussions generally led to a drop in the perceived quality of the related article, especially if the discussion revealed conflict.

Motivated by how university students learn to assess reliability of controversial articles such as Supreme Court decisions or about individuals like Pope Pius XII and Yasser Arafat, the researchers considered how beneficial it would be to reveal the process of articles creation. In wikis the discussions used to produce the articles are hidden from view using talk pages and other coordination spaces. It was believed that when deliberations appear fair, well-reasoned, and thorough it should engender trust in the reader and that a process which appears biased or chaotic should diminish the confidence in the article’s quality. The paper outlines the issues involved in assessing the credibility of online information sources. The paper first considers prior work on article quality but reframes the issues based on an idea presented in the recent best seller Thinking, Fast and Slow by economics Nobel laureate Daniel Kahneman. The research questions posed are:

• RQ1: What is the effect of exposing discussions about article content on perceived article quality?
• RQ2: Do different kinds of conflict resolution have different effects on perceptions of content quality?
• RQ3: What do participants believe about how viewing the discussion may have changed their perceptions?

These questions are then interpreted using Kahneman’s System 1 (slower deliberative thinking) and System 2 (faster associative thinking). The questions were investigated in an experiment run on Amazon’s mechanical Turk — a crowdsourcing platform allowing micropayments. Beginning with 3500 controversial articles, the researchers selected featured articles, and discarded newsworthy items leaving only 50 articles. Elite Turkers were then shown ten brief vignettes illustrating talk page discussion about a selected controversy, meant to display one of ten forms of editor coordination or conflict activities. They then had to answer a questionnaire, and complete two reading comprehension tasks. The researchers noticed that exposing Wikipedia readers to such discussions with any type of conflict generally led to a drop in the perceived quality of the related article. They point out that the magnitude of the reader’s negative perception depends on the type of editors’ interaction. Finally they note that while participants may have suffered a confidence crisis with respect to specific articles, at the same time they gained respect for Wikipedia in general. A final conclusion is that while the experiment, especially the comprehension task, was designed to engage readers in System 1 thinking, watching the discussions may well have triggered a System 2 critical response.

Briefly

100 million hours spent editing Wikipedia

Edit counts are often used as a measure of the amount of activity on a wiki, but as the work that goes into one edit can vary between mere seconds and many hours or even days, they don’t translate easily into work time. Still, in 2008, Clay Shirky and IBM researcher Martin Wattenberg estimated as a “back-of-the-envelope calculation” that “about 100 million hours of thought” had gone into Wikipedia (a number later featured prominently in Shirky’s book Cognitive Surplus). A CSCW 2013 paper titled “Using Edit Sessions to Measure Participation in Wikipedia”^[7] calculates work time as the length of edit sessions, defined as “a sequence of edits made by an editor where the difference between the time at which any two sequential edits are saved is less than one hour”. They estimate that a total of 102,673,683 labor hours were spent editing Wikipedia (in all languages) until April 2012 (which was compared to 168 lifetimes of work) and 61,706,883 hours on the English Wikipedia. The paper also contains a list of the 20 editors who (by this measure) spent the most time editing the English Wikipedia in March 2012.

Wiktionary and sign language

In “Between Wictionary [sic] and a Thesaurus : Some Dilemmata of a Sign Language Dictionary”,^[8] apparently an abstract of a paper to be presented at a conference, the author presents the challenges to writing a dictionary of sign language in the world of modern lexicography. In the author’s opinion, Wiktionary in general, and the Czech Wiktionary in particular, is an important example of one of the latest innovations in lexicography: It is based on contributions by volunteers who are not necessarily professional to achieve a work of a volume that would be extremely expensive to produce in traditional professional lexicography, although it sacrifices some of the advantages of the latter, such as a carefully selected glossary and rigorous standardization. The author sees future in using a wiki technology for creating dictionaries for sign languages that will be better than the current dictionaries at least in some characteristics, and makes some suggestions on how to implement it well. Notably, the author discusses displaying the signs as illustrations and videos and does not mention SignWriting – a system of standardized characters for representing signs, which was already used for several dictionaries and websites; it is not encoded in Unicode yet, but experimental support for SignWriting is available for MediaWiki as an extension. A minor nitpick is the misspelling of the name “Wiktionary” – the author writes it with a ‘c’ rather than a ‘k’.

Wikipedia compared to Q&A website in Korea

In South Korea, Wikipedia lags behind several other services in popularity, such as Naver‘s KnowledgeiN knowledge market Q&A service. A new paper^[9] compares the English and Korean Wikipedias to the KnowledgeiN service, and analyzes some of the factors involved in how users perceive quality in wikis and Q&A services. About 200 users of each of the three websites participated in the survey. The authors found that perceived quality helps to determine how useful the users are going to see a given site. Previous research suggesting that community expertise, size and diversity all contribute to quality is confirmed, and those factors are recognized and valued by the general public. As might be expected, the authors find that users of Q&A sites value expertise of contributors more than users of wikis. In turn, wikis rely on the size of their community to achieve quality. Predictably, the authors conclude that the smaller Wikipedias such as the Korean one suffer from small community size, and recommend that to improve the quality and popularity of such Wikipedias, more editors should be recruited. The study notes a number of limitations that affected it; notably it did not take into account any possible cultural differences, and it does not provide any discussion of why Wikipedia’s popularity in Korea is lacking compared to many other websites, such as KnowledgeiN.

The development of the ureteric bud in C57BL/6 mice in the absence (left) or presence (right) of the protein Cer1.^{[mediasource 3]}

Wikipedia articles on nephrology reliable, but hard to read

An article^[10] by four Toronto-based medical authors concludes that “Wikipedia is a comprehensive and fairly reliable medical resource for nephrology patients that is written at a college reading level”. Comprehensiveness was measured by coverage of ICD-10 items pertaining to this area of medicine. The reliability of articles was also measured in a purely quantitative way, based on “(i) mean number of references per article, and (ii) mean percentage of ‘‘substantiated’’ references—which we defined as references corresponding to works published in peer-reviewed journals or from texts with an associated International Standard Book Number (ISBN)”. Readability was measured using three standard formulae including the Flesch-Kincaid grade level.

Comparing English and Arabic Wikipedia POV differences

Khalid, Schutze and Kantner compare point of view (POV) differences between English and Arabic articles about “international personalities”.^[11] They use Amazon Mechanical Turk to annotate sentences as positive, negative and neutral and build a statistical classifier to predict the POV score of a document. The authors find that Arabic articles are generally more positive than their English counterparts and conclude that there are at least two possible reasons for a POV difference: either because of a generally lower or higher level of absolute POV in a language, or because of a genuinely different evaluation of a personality in different Wikipedias. The article also contains rich detail about the challenges of evaluating POV differences using both human and automatic classifiers.

The overrepresentation of cricket on English Wikipedia

This article^[12] for the 2013 edition of The International Journal of the History of Sport analyses 115 English wikipedia articles about Australian sportspeople and finds that a disproportionately large number are cricketer biographies. They find that, instead of reflecting the most popular sports in Australian society (of which cricket is one of the least popular), Wikipedia reflects the interests of a small special interest group. In this case, two Wikipedians are behind the creation and maintenance of almost all the content of the high-quality cricket articles. The authors note that cricket is also generally better represented in literary sources where the sport takes on a nostalgic narrative embodying traditional Australia. They conclude with the question of whether the extensive literature on cricket is reflected in Wikipedia articles and, if so, whether the existence of the same factors that have led to the creation of high-quality articles on Australian cricket – are relevant or whether there are other dynamics at play.

Grumpiness due to a “serious typographical error”

A serious typographical error may have led a 2008 personality study to wrongly claim Wikipedians are close-minded. A blog post for Psychology Today^[13] re-examined a widely quoted 2008 survey among 69 Israeli Wikipedians^[14] that (as summarized by the New Scientist at the time) had concluded that “Wikipedians are grumpy and close-minded” (Signpost coverage). The author found that the paper “contains serious errors and even contradicts itself … [C]ontrary to what was reported, Wikipedia members of both sexes actually had higher mean scores on openness to experience compared to non-members, not lower ones. Perhaps the authors’ were confused by the presence of a serious typographical error that appears in the Results section of their article”.

Wikipedians do not tend to conform more to groupthink when in a less anonymous situation

In a survey,^[15] 106 editors on the English Wikipedia were asked (with approval of the Wikimedia Foundation Research Committee) how they would act in three real-world scenarios (not involving Wikipedia – e.g. “a group of tenants dealing with a noisy/problematic neighbor”), each “carefully designed so that the individual would have a high incentive to resolve the problem, but would also incur some sort of penalty for voicing a dissenting opinion”, and assuming varying levels of anonymity (e.g. complete anonymity, pseudonymity, or use of real names). The paper’s main hypothesis, “that with higher levels of anonymity the likelihood of not conforming increases as well”, found only weak support, which the author calls “a promising result for online communities and the future of online communication. Given that non-conformity in this study meant ensuring a contribution of alternatives to the group, this is a positive outcome for preventing groupthink.”

Estimate for economic benefit of Wikipedia: $50 million by 2006 already

In a recent article, The Economist examined the question “How to quantify the gains that the internet has brought to consumers”, citing a 2009 paper by two economists that attempted to calculate the monetary value of consumer surplus generated by broadband Internet, focusing on how much value Internet is providing for free (that otherwise people would be prepared to pay). While this paper did not mention Wikipedia, The Economist cited one of the authors (Shane Greenstein, known to readers of this research report for his work on political POV language in Wikipedia articles, reviewed in the January 2012 and February 2012 issues), who “thinks Wikipedia accounted for up to $50m of that surplus” as of 2006 – in other words, Wikipedia provides a good that otherwise people would be willing to buy, spending $50m on it that instead they get to spend on something else. The Economist commented that “such numbers probably understate things” as the paper’s methodology assumed that “internet access meant the same thing in 2006 as it did in 1999.”

Journalists cover many topics.^{[mediasource 4]}
So does Wikipedia.

91% of German journalists use Wikipedia

A survey^[16] conducted by a PR agency among “over 2,600 journalists from France, the UK, America and Germany” asked them about various aspects of their work including Wikipedia usage, finding among other results “91% of the German national media journalists admitting to using Wikipedia to research stories.”

Inserting weblinks on Wikipedia to drive traffic

A case study published in D-Lib Magazine (“the magazine of digital library research”), titled Using Wikipedia to Enhance the Visibility of Digitized Archival Assets^[17] reported on “the use of Wikipedia by the Ball State University Libraries as an opportunity to raise the visibility of digitized historic sheet music assets … by adding links to specific items in this collection to relevant, existing Wikipedia articles”. In a blog post, Europeana also reported on exposure to its content received via Wikipedia, in a somewhat different approach – by providing the content on Wikipedia itself.^[18]

Case study on “Accommodating the Wikipedia Project in Higher Education”

A 94 page master’s thesis^[19] finds that the University of Windsor, treated as a case study, is torn between two groups: one encouraging the use of new digital tools like Wikipedia, and the other, conservative, opposed to it. There is a general lack of understanding of Wikipedia (a finding similar to a study reviewed in last month’s issue: “UK university lecturers still skeptical and uninformed about Wikipedia“). Many participants (instructors, scholars) use Wikipedia and recognize it has been improving and becoming more convenient, but are mostly unwilling to contribute to it; one participant noted that doing so would be a career “academic suicide”. Nonetheless the study also suggest that there is significant sympathy for Wikipedia, and many interviewees indicated that they would like to contribute, but are stymied by “lack of time, lack of academic credit, and overall lack of resources to do work not directly related to their professional responsibilities.” Wikipedia outreach to academia is seen as noble, but likely not to progress quickly due to those issues.

Wikipedia student club participation

A dissertation titled “Investigation of Disassembling Polymers and Molecular Dynamics Simulations in Molecular Gelation, and Implementation of a Class-Project Centered on Editing Wikipedia”^[20] contains some observations on the first ever Wikipedia student club in the US: “the students who enter the Wikipedia community through the student club have a different editing contribution pattern than the general population and the students who enter through a class project. These editors still remain active after a year from creating the account. Although they start at a lower editing efficiency, they peak later in the year and have a more gradual decline in active editing activity.” (p.170)

Monthly edits still on the rise

Erik Zachte, a data analyst for the Wikimedia Foundation, observed in a blog post^[21] that “the overall volume of manual edits by registered users on all Wikimedia wikis combined is still increasing, slowly but steadily” (somewhat different from the number of active editors, which has been slightly decreasing or stagnating over the last few years), generating some discussion on the possible reasons.

How many Wikipedia edits come from locals?

On the “Zero Geography” blog, researcher Mark Graham continued his series about geostatistical aspects of Wikipedia, presenting “A map of edits to articles about Egypt”,^[22] providing an overview article on some earlier results that appeared in a Rwandan magazine^[23] and asking “What percentage of edits to English-language Wikipedia articles are from local people?”.^[24]

New overview page of Wikimedia data for researchers

A new page on Meta-Wiki gives an overview for researchers of various sources of open data published by the Wikimedia Foundation about Wikipedia and its sister projects (Wikipedia dumps, stats, live feeds, etc.)

Wikimedia funding for Wikisym ’13 despite open access concerns

A request for financial support from the Wikimedia Foundation for the 2013 WikiSym/OpenSym conference – as in previous years – was approved this month, but not without serious concerns among the Foundation’s volunteer-based Grant Advisory Committee about the organizer’s choice of the ACM Digital Library as the publication venue of the conference proceedings, which makes them available for download cost-free but not under a free license. The issue had been brought up as early as in 2010, when the contribution of one conference speaker was not included in the proceedings because he had insisted on republishing it under a CC-BY-SA license.

Research newsletter started on French Wikipedia

“Nouvelles du Wikilab” is a new community-written research newsletter on the French Wikipedia, summarizing and sometimes enriching this monthly research report in French. It offers subscription (for delivery to one’s user talk page on the French Wikipedia). There are also ideas for an on-wiki French language research review journal named Wikilogie, to publish original research about Wikipedia which could be useful for the community, and to facilitate dialogue with researchers.

Inferring relationships from editing behavior on Wikipedia

A paper presented at the 8th Cyber Security and Information Intelligence Research Workshop (January 8 – 10, 2013, link to event) reports on the application of Transfer Entropy, a promising information-theoretic tool originally devised by neuroscientists to study causal connections among biological neurons, to infer a network of “social relationships” among editors of Wikipedia, using only information about their editing behavior.^[25] As Wikipedia lacks explicit information about social ties among editors, the authors needed to define a “ground truth” network using interaction on User Talk pages. The method attains a high level of precision but a very low level of recall. The contribution won the best paper award at the workshop in which it was presented.

Google Research releases the WikiLinks Corpus: 40M mentions to Wikipedia pages collected from 10M web pages

Researchers at Google recently released a Natural Language Processing dataset of 40M terms occurring in 10M pages, obtained by crawling the Web and looking for links that point to Wikipedia articles. According to the blog post about the release, the dataset is the largest set of disambiguated mentions to date, nearly 100 times bigger than the second largest database publicly available. A technical report covers in detail the collection, generation, and curation of the dataset.^[26]

References

Image sources

Vol: 3 • Issue: 2 • February 2013 [contribute] [archives]

Wikipedia not so novel after all, except to UK university lecturers; EPOV instead of NPOV

With contributions by: Piotr Konieczny, Taha Yasseri, Heather Ford, Sage Ross, Daniel Mietchen and Tilman Bayer.

Wikipedia in historic context: “Stigmergic accumulation” is not new

Page with the entry Encyclopédie from Diderot and D’Alembert’s Encyclopédie. The work was the result of the collaboration of more than 100 contributors.

“Wikipedia and Encyclopedic Production”^[1] by Jeff Loveland (a historian of encyclopedias) and Joseph Reagle situates Wikipedia within the context of encyclopedic production historically, arguing that the features that many claim to be unique about Wikipedia actually have roots in encyclopedias of the past. Loveland and Reagle criticize characterizations of Wikipedia that they believe to be ahistorical and exaggerated, laying special blame with authors who compare Wikipedia’s anonymous production to Encyclopedia Britannica’s production by named experts, and thus ignore the rich tradition of encyclopedic production through the centuries. The authors then set about characterizing the history of encyclopedic production as composed of three overlapping forms: compulsive collection, stigmergic accumulation, and corporate production.

‘Compulsive collection’ refers to the work of compiling encyclopedias that has traditionally been done by a few dedicated, tireless, detail-oriented individuals. Loveland and Reagle point out that, although Wikipedians share this compulsive behavior with past encyclopedists, the crucial distinction lies in the fact that the vast majority were motivated by money (even if this motive existed alongside more idealistic motivations) whereas Wikipedia editors are unpaid.

Loveland and Reagle use the term ‘stigmergic accumulation’ to refer to the process of production by accretion onto a previous text. Even those responsible for a singly authored encyclopedia were relying on predecessors, the authors argue, ‘building on their work and using the cumulative character of texts and knowledge as a ladder of sorts’. Examples of existing texts included the use of a previous edition of an encyclopedia that ran into multiple editions, and the practice of borrowing between different encyclopedias that was sometimes illegal but more often viewed as ‘piratical’ i.e. morally wrong.

The category of ‘corporate production’ is used by Loveland and Reagle to describe the process of encyclopedic editing by a group – groups that topped a thousand contributors in the 20th century. Editors of early encyclopedias like Diderot and D’Alembert’s Encyclopédie in the 1700s faced the challenge of trying to coordinate the contributions of about 140 contributors in a similar way to Wikipedia having to confront issues of consistency that result in debates about how important a subject must be to merit an article. In contrast to other encyclopedias, write Loveland and Reagle, Wikipedia settles these debates through community decision-making and in the open. The authors also note that previous encyclopedias didn’t always recruit on the basis of expertise and that some recognized that it would be cheaper and sometimes more accurate to have non-experts summarizing the works of experts.

The authors conclude by writing that although Wikipedia is in some respects unique in terms of its size, its reliance on volunteers and nonprofit nature, the continuities between Wikipedia and past encyclopedias are ‘numerous and significant’. Their hope is that this paper ‘will help scholars avoid ahistorical claims about Wikipedia, identify historical material germane to the social scientist’s concerns (such as the motivations of encyclopedia-producers), and show that contemporary questions about Wikipedia (such as what exactly should be counted as a contribution) have a lifespan exceeding the past decade.’

The collaboration of the two authors began after Reagle had learned (via a summary in this research report) of a review where Loveland had criticized Reagle’s 2010 book Good Faith Collaboration: The Culture of Wikipedia for having “one major weakness, namely in historical contextualization”. The resulting paper has received media attention starting with an article in The Atlantic, see the February 4 issue of The Signpost.

UK university lecturers still skeptical and uninformed about Wikipedia

Work has been ongoing for a while along the lines of recommendation 3: a brochure by the Wikipedia Education Program explains how to assess Wikipedia article quality. A number of similar materials exist on how to use Wikipedia in educational contexts, some also in other languages.

A study titled “Exploring the Cautionary Attitude Toward Wikipedia in Higher Education: Implications for Higher Education Institutions”^[2] analyzes the attitudes of five British university lecturers towards Wikipedia, through qualitative analysis. The methodology consisted of 90-minute interviews with the lecturers who declared their familiarity with Web 2.0 educational tools, and analysis of university documentation, primarily in the form of “unofficial policy” regarding the use and evaluation of Wikipedia by students, as no official policy on Wikipedia existed. The author finds that Wikipedia is still treated with suspicion by the educators interviewed, due to 1) a lack of understanding of Wikipedia, 2) a negative attitude toward collaborative knowledge produced outside academia, and 3) the perceived detrimental effects of the use of Web 2.0 applications not included in the university suite. Some factors of particular concern included 1) “the difficulty of knowing if an article is correct,” 2) doubts regarding the quality of information produced by anonymous contributors, 3) doubts about whether the Wikipedia crowdsourcing model of knowledge production can really outperform the experts (reviewer note: but of course…), 4) concerns that Wikipedia makes research “too easy” for students and 5) misunderstanding of Wikipedia’s non-profit nature, with two interviewees suspicious of Wikipedia, expressing concerns such as “this is a commercial business; it wants to make money” and “they are obviously doing it from a business perspective”, and another concerned about “politics and motivation behind [Wikipedia's dominant position on the Internet]“. All the interviewed teachers hoped that the library staff would be able to provide guidance to students regarding evaluating when to use Wikipedia; however correspondence with the library staff showed that their guidelines are mostly inapplicable to Wikipedia, and they do not address Wikipedia during their literacy teaching sessions. The research also found that all five lecturers use Wikipedia in personal life, and four, in professional research, with two of them commenting that they feel a bit hypocritical using the same tool they warn the students about. None of the interviewed lecturers contributed to Wikipedia, and only one was aware of any outreach from Wikipedia to academics. As the author notes, Wikipedia is still alien to the academic culture, and while the attitudes are shifting, there is still much misunderstanding about Wikipedia’s reliability, quality and non-profit mission. The author concludes that Wikipedia should address those concerns through the following recommendations: 1) “Increase understanding of Wikipedia, its policies and processes.” 2) “Increase understanding of the nature of open and free collaboratively produced knowledge.” and 3) “Make available Wikipedia guidance and evaluation criteria to students and teaching staff.”

Saint Petersburg has more sisters than any other city in the world

Connections between twin cities. Brighter colours indicate shorter distance.

A research team from Barcelona Media Lab has put a preprint in arXiv recently,^[3] in which the sistership relation between cities is analyzed. Although the paper has little to say about Wikipedia in itself, the corresponding dataset is extracted from the English Wikipedia, which, according to the paper, hosts “the most extensive but certainly not complete collection of this kind of relationships”. Although this statement could be argued, and no evidence is presented in the article to support that claim, it is striking that mass collaboration of Wikipedians can provide such an interesting package of information on a global scale.

After a description of the data extraction process, the paper presents a set of standard Complex Network analysis, e.g., degree distribution, clustering, average separation of nodes on the network, etc. While there is no surprise among the results of the network analysis, the interesting conclusion is about the effects of the geographical distance of sister cities: “the geographical distance has only a negligible influence when a city selects a sister city.”

“Distributed Wiki” proposal to replace NPOV with “every point of view (EPOV)”

In a paper titled “Towards Content Neutrality in Wiki Systems”^[4] (an extended version of a conference keynote), a German computer scientist criticizes Wikipedia’s “Neutral Point of View” policy as based on “an objectivist point of view” that is prominent in parts of the natural sciences, but is challenged in the fields of quantum physics, psychology and the social sciences, according to the author. He offers differences between language versions of the Wikipedia articles on Osama Bin Laden and the Mossad as further arguments against the NPOV concept. He admits that “two examples, based on machine translation and subjective classification by an author who wants to prove his point do not show anything”, but claims that even more systematic studies “would suffer the same objection”, proving “that the object under consideration, i.e., a neutral point of view, logically may be considered an ill-defined concept”. However, it is still used on Wikipedia, and even ingrained in its architecture where the “linear version history evokes the illusion that there is one ‘currently best’ version of an article”. To explain why so many Wikipedians do not accept his own logical conclusion, the author offers the psychological diagnosis of cognitive dissonance: “Accepting the illusion of NPOV, one does not have to live with never ending edit-wars on the ultimately right article and one does not have to suffer dissonant feelings in every article”.

Heading back towards his own area of academic expertise, the author then outlines “a variant-augmented Wiki system” to modifying the linear versioning of usual wikis. He bases it on a proposed concept of “content neutrality” requiring “knowledge management platforms” “to store and provide content without any evaluation of its merit”, modeled after network neutrality. He argues that “a collection of every-point-of-view, contradictory, possibly emotionally charged articles may provide a better approximation to reality than a synthetic and illusionary neutral point-of-view” and hopes that such a “content-neutral, EPOV knowledge base [has] a real chance of becoming a truly helpful instrument for science.” Various possible problems with such an architecture are outlined, together with some suggested solutions, but the discussion remains brief and without detail (“The formal design and security analysis of low-level protocols are left to a later paper”). To protect against malicious site operators, the concept is extended to “distributed Wiki hubs”. The paper mentions the existence of “a preliminary implementation as a MediaWiki plug-in prototype” and ongoing work on a use case “which can be described as a Wiki / Blog merge”.

A section on related work cites a few earlier papers discussing distributed wikis or Wikipedia forks (out of many more, see e.g. this reviewer’s “Timeline of distributed Wikipedia proposals“).

Briefly

• Mildly negative feedback makes newbies work harder: A paper^[5] to be presented at the upcoming Conference on Human Factors in Computing Systems (CHI’13) reports on a “field experiment on Wikipedia to test the effects of different feedback types (positive feedback, negative feedback, directive feedback, and social feedback) on members’ contribution.” The team from Carnegie Mellon University left user talk page messages for 703 English Wikipedia editors who had recently created a new article. These messages (example) were varied to test the effect of the different feedback types on the user’s subsequent activity. To the researchers’ surprise, none of these had a significant effect on experienced users. But for new editors, “positive feedback and social messages increase people’s general motivation to work”. Negative feedback and directive feedback still increased newbies’ edits on the corresponding articles. The researchers note those negative messages (example: “I noticed there are some holes that may need filling: the references in the article do not follow Wikipedia guidelines”) “were intentionally designed to be milder than negative feedback messages actually sent between Wikipedia editors” which other research has found to decrease participation. In a 2011 paper, three of the authors had studied similar phenomena in a non-participatory analysis (review: “How different kinds of leadership messages increase or decrease participation“).
• Students editing at PhD level in APS Wikipedia initiative: Another conference paper for CHI’13^[6] reports on a project where 640 undergraduate and graduate students edited Wikipedia articles on scientific topics in 36 university courses. This classroom editing project was part of the US Association for Psychological Science‘s Wikipedia Initiative. The authors found that the “students substantially improved the scientific content of over 800 articles, at a level of quality indistinguishable from content written by PhD experts” (measured in a content persistence metric).
• Students unimpressed by professors’ disapproval of Wikipedia: A paper on the factors affecting Wikipedia use,^[7] based on a survey of 184 undergraduate students in Singapore, has a few suggestive findings:

  Female students were more likely to use Wikipedia.

  Believing that authority figures (such as professors) disapprove of Wikipedia use did not affect students’ likelihood of using Wikipedia.

  Peer influence—whether or not students think their friends use Wikipedia—was significantly correlated with Wikipedia use.

  Whether these findings hold true for larger and more diverse groups of students is an open question.

• Voting open for most important paper in Wikipedia research: As reported earlier, the French Wikimedia chapter is providing an award for the most influential paper published between 2003 and 2011. Out of more than 30 submissions, a jury of researchers has now selected five finalists, and until March 11 all Wikimedians are invited to vote to select the winning paper among them. Its authors will receive a grant of 2500 Euros.
• Teahouse compared to Help Desk: A workshop paper titled “How to ask questions the n00b way: designing social Q&A for new users^[8], presented at the CSCW 2013 “Workshop on Social Media Question Asking” covers the Teahouse, the support space for new editors launched on the English Wikipedia in 2012. It summarizes results from a longer paper presented at the same conference, which has been covered in the December issue of this research report. These include a survey among participants indicating that the Teahouse was generally well-received, and a comparison of edit rates showing, somewhat unsurprisingly, that newbies who followed an invitation to join the Teahouse tended to make more edits than those who ignored the invitation. (No attempt was made to measure the effect of invitations directly by comparing with non-invited newbies, because of the possible bias caused by Teahouse hosts avoiding inviting editors whose first edits do not seem productive.) Questions in the Teahouse generated more responses than those on the Help Desk, the English Wikipedia’s longer-running help forum which is less focused on social elements and new editors.
• Predicting admin elections based on social network analysis: A paper^[9] modeled all admin elections on the Polish Wikipedia (since 2005) based on “multidimensional behavioral social networks derived from the Wikipedia edit history” of candidates and voters, finding that “we can classify the votes in the RfA procedures using this model with an accuracy level that should be sufficient to recommend candidates.” (See also our review of an earlier paper by the same authors: “What it takes to become an admin: Insights from the Polish Wikipedia“)
• “Analysing the Entire Wikipedia History with Database Supported Haskell“: Four researchers from Germany describe^[10] a technique that allowed them to scale a quantitative analysis from a Wikisym 2010 paper (that had examined the collaboration on a smaller sample of 4,733 articles and 4,679 users) to the entire revision history of the German Wikipedia.
• Traffic analysis report and research ethics: The Signpost’s special report titled “Examining the popularity of Wikipedia articles: catalysts, trends, and applications”^[11] gave an overview of several research topics regarding pageviews on the English Wikipedia, including a list “the most viewed pages on Wikipedia in a one hour period” since 2010 that generated media attention, e.g. in the Atlantic (“If You Want Your Wikipedia Page to Get a TON of Traffic, Die While Performing at the Super Bowl Half-Time Show“). The report also discusses possible causes of such page view spikes, and applications of research on pageviews, such as focusing efforts to improve article quality and assessing the impact of vandalism. The latter idea drew from earlier studies of one of the authors, including an experiment that had generated controversy in 2010 and caused the English Wikipedia’s ArbCom to block the author temporarily, as reported in the Signpost at the time: “Large scale vandalism revealed to be ‘study’ by university researcher“. A recent paper about research ethics that the author coauthored with his doctoral advisor and two others^[12] criticized the University of Pennsylvania‘s institutional review board for its hesitant approval of the 2010 experiment, justified the lack of advance notification of Wikipedia community (because it would have biased the results of the experiment) and also talks about the “extremely mixed” response from reviewers of the resulting paper (reviewed in the September 2011 issue of this research report: “Link spam research with controversial genesis but useful results“).
• “Faces of Wikipedia” dataset: Two researchers from the École Polytechnique de Montréal have compiled^[13] a database with facial images of over 50,000 Wikipedia article subjects, used to test facial recognition algorithms.
• Detecting news events from Wikipedia edits: A paper to be presented at the annual European Conference on Information Retrieval^[14] describes the detection of “real-world events such as political conflicts, natural catastrophes, and new scientific findings” from Wikipedia edits. Apart from bursts (peaks) in the editing activity of an article, another indicator used is the appearance of a current or recent date in the diff of an edit.
• Evidence for damage caused by personal attacks and wikilawyering: A preprint titled “Stay on the Wikipedia Task: When task-related disagreements slip into personal and procedural conflicts” ^[15] reported on an analysis of 96 Wikipedia articles and the corresponding talk pages which found “that when group members’ disagreements – originally task-related – escalate into personal attacks or hinge on procedure, these disagreements impede group performance.”
• Corpus of 200+ research papers on Wikipedia from 2012: With this monthly research update having completed its second volume recently, we have released a bibliographical dataset listing all of the more than 200 academic publications that were covered in 2012.

References

Twelve years after its launch, Wikipedia continues to attract a large amount of attention from scholarly research trying to understand what made this one of the most remarkable collaborative efforts in history and what makes it work. Researchers have called Wikipedia “our Everest” (because of its complexity and cultural importance) or “the Drosophila (fruit fly) of social software” (because the project’s transparency and freely available data make it accessible and popular as a research subject).

Download the complete Volume 2 (PDF)

In 2011, we launched a monthly Wikimedia Research Newsletter with the aim of covering recent academic research about Wikipedia and other Wikimedia projects. Published jointly by the Wikimedia Research Committee and the Signpost (the English Wikipedia’s community-edited newspaper), it has established itself as a comprehensive outlet enabling both researchers and Wikipedians to stay on top of current research, aiming to facilitate exchange between these two communities.

Today we are announcing the release in the public domain of a curated corpus containing the bibliographic references of all 225 publications reviewed or covered in the second volume of the newsletter, forming a historical record of Wikipedia research in the year 2012. This corpus can be browsed online or downloaded, ready to be imported into reference managers or other literature collections. Papers in this dataset have been marked as either open access or closed access .

Last year, we published a similar dataset for volume 1 (2011). Together, these releases complement other efforts to catalogue the research literature on Wikipedia, in particular the WikiLit project which focuses on publications until June 2011, prior to the launch of the newsletter.

Follow @WikiResearch for fresh Wikimedia research news

A year ago we launched the @WikiResearch news feed on Twitter and Identi.ca, covering new preprints, papers or research-related blog posts, before they are reviewed more fully in the Newsletter. As of February 2013, it has gained 745 followers and continues to be actively updated.

We also started offering the newsletter in form of an HTML email newsletter (in addition to the announcements of each new issue on the Wikiresearch-l mailing list, which only contain the table of contents). This experiment proved successful, too, with almost 100 subscribers to date (adding to the thousands of pageviews each issue receives when published as part of the Signpost, on Meta-wiki and on this blog). You can sign up to receive a copy of each new issue in your inbox as soon as it comes out.

The Newsletter is a collaborative effort and would not exist without the following 22 people who contributed reviews and summaries in 2012:

Aaron Shaw, Amir E. Aharoni, Angelika Adam, Bence Damokos, Benjamin Mako Hill, Daniel Mietchen, Dario Taraborelli, Diederik van Liere, Evan Rosen, Heather Ford, Jodi Schneider, Giovanni Luca Ciampaglia, Lambiam, Nicolas Jullien, Oren Bochman, Phoebe Ayers, Piotr Konieczny, Adam Hyland, Sage Ross, Steven Walling, Taha Yasseri, Tilman Bayer

More than half of our contributors are researchers themselves, who have published about Wikipedia in peer-reviewed publications. We are also grateful for the help of several Signpost collaborators in copyediting and preparing the final publication every month.

Finally, thanks to everyone for reading the Wikimedia Research Newsletter, and please
consider contributing by pointing us to new research we should cover, or by volunteering to review new publications.

The editors of the Wikimedia Research Newsletter:

Tilman Bayer, Senior Operations Analyst
Dario Taraborelli, Senior Research Analyst

Vol: 3 • Issue: 1 • January 2013 [archives]

Lessons from the research literature on open collaboration; clicks on featured articles; credibility heuristics

With contributions by: Taha Yasseri, Piotrus, Aaron Shaw, Tbayer and Lui8E

Lessons from the wiki research literature in “American Behavioral Scientist” special issue

A special issue of the American Behavioral Scientist is devoted to “open collaboration”.

• Consistent patterns found in Wikipedia and other open collaborations: In the introductory piece^[1], researchers Andrea Forte and Cliff Lampe give an overview of this field, defined as the study of “distributed, collaborative efforts made possible because of changes in information and communication technology that facilitate cooperative activities” – with open source projects and Wikipedia among the most prominent examples. They point out that “[b]y now, thousands of scholars have written about open collaboration systems, many hundreds of thousands of people have participated in them, and millions of people use products of open collaboration every day.” Among their “lessons from the literature”, they name three “consistent patterns” found by researchers of open collaborations:
  • “Participation Is Unequal” (meaning that some participants contribute vastly more than others: “In Wikipedia, for example, it has long been shown that a few editors provide the bulk of contributions to the site.”)
  • “There Are Special Requirements for Socializing New Users”
  • “Users Are Massively Heterogeneous in Both How and Why They Participate”
• “Ignore All Rules” as “tension release mechanism”: The abstract of paper titled “Rules and Roles vs. Consensus: Self-Governed Deliberative Mass Collaboration Bureaucracies” ^[2] explains “Wikipedia’s unusual policy, ignore all rules (IAR)” as a “tension release mechanism” that is “reconciling the tension between individual agency and collective goals” by “[supporting] individual agency when positions taken by participants might conflict with those reflected in established rules. Hypotheses are tested with Wikipedia data regarding individual agency, bureaucratic processes, and IAR invocation during the content exclusion process. Findings indicate that in Wikipedia each utterance matters in deliberations, rules matter in deliberations, and IAR citation magnifies individual influence but also reinforces bureaucracy.”
• Collaboration on articles about breaking news matures more quickly: “Hot Off the Wiki: Structures and Dynamics of Wikipedia’s Coverage of Breaking News Events”^[3] analyzes “Wikipedia articles about over 3,000 breaking news events, [investigating] the structure of interactions between editors and articles”, finding that “breaking articles emerge into well-connected collaborations more rapidly than nonbreaking articles, suggesting early contributors play a crucial role in supporting these high-tempo collaborations.” (see also our earlier review of a similarly-themed paper by the same team: “High-tempo contributions: Who edits breaking news articles?“)

A fourth paper in this special issue, titled “The Rise and Decline of an Open Collaboration System: How Wikipedia’s Reaction to Popularity Is Causing Its Decline”, found considerable media attention this month, starting with an article in USA Today. It was already reviewed in the September issue of the research report.

Mathematical model for attention to the promoted Wikipedia articles

While the size and growth rate, editorial workflow, and topical coverage of Wikipedia have been vastly studied, there is little work done on the understanding of public attention to the Wikipedia articles. In a working paper by a team from the Barcelona Media Foundation and the University of Twente, placed on arXiv just before Christmas,^[4] the number of clicks on the featured articles promoted to the Wikipedia Main page is analysed and modeled.

A total of 684 featured articles are considered and the page view statistics of them is rescaled by the average circadian view rate extracted from a larger set of 871 395 articles in a period of 844 days. The 4-day lifetime of the promoted articles on the Main page is characterised by four phases. A very rapid growth in the number of article clicks just after the article appears on the Main page, followed by a rather homogeneous period of the first day of the promotion. As the article is replaced by a new featured article, and placed in the “recently featured” part of the Main page, the rate of clicks drops dramatically, and finally, the fourth flat phase is experienced during the remaining 3 days at this location.

In the next step, the authors introduce a rather intuitive model based on a few parameters to fully describe the 4 days cycle in a mathematical framework. The model is tuned based on the data of a set of 100 featured articles to predict the number of page hits for the rest of the sample, given the number of the clicks after the first hour of promotion for each article. The model is relatively accurate in predicting the number of clicks, and this accuracy could be even improved by feeding the model with the number of clicks at the end of the first day instead of the first hour after promotion. While the paper is very clear in describing the methodology, it fails to discuss and provide a deeper understanding of the social mechanisms of popularity and public attention, as it is mentioned repeatedly by the authors.

The featured article icon and other heuristics for students to judge article credibility

A paper in Information Processing and Management titled “College students’ credibility judgments and heuristics concerning Wikipedia”^[5] used the theory of bounded rationality and a heuristic-systematic model to analyze American college students’ credibility judgments and heuristics concerning Wikipedia. Not surprisingly, authors observe that students used a heuristic (a mental shortcut, such as An article with a long list of references is more credible than with of a short one) in assessing the credibility of Wikipedia. Students (regardless of their knowledge) were much more likely to focus on the number of references than on their quality, and the same article would be seen as more credible depending on how many references it had. The authors conclude that educators need to teach students how to judge the quality of Wikipedia articles that goes beyond checking whether the article has references (and how many). The authors recommend that Wikipedia makes its own assessments (such as the Featured Article star, currently visible only as a small bronze star icon on the top right-hand corner of the article’s page) much more prominent. (This reviewer strongly agrees with the conclusion, but unfortunately the last community discussion appears to have achieved little.)

More interestingly, the authors also find that people with more knowledge found Wikipedia more credible, suggesting that people with low knowledge may be more uneasy with Wikipedia. The authors suggest that the reliability of Wikipedia would be increased if more professional associations implemented programs such as Association for Psychological Science Wikipedia Initiative. In addition to getting the experts more involved in Wikipedia content creation, the authors suggest that a good idea may be for “professional associations themselves [to] provide their own endorsement for the quality of articles in their fields.”

The authors also note that peer endorsement is an important factor in credibility, and that the Wikipedia:Article Feedback Tool is a step in the right direction, as it provided another credibility assessment for the readers. They note, however, that compared to similar tools implemented on other sites (such as Amazon), “Wikipedia readers need to click on ‘‘View Page Rating,’’ which requires one more step to find out that information. The average reader may not be inclined to do so. It would be useful to display ratings without clicking”.

Briefly

• “Free as in sexist?”: In a paper in this month’s First Monday,^[6] Joseph Reagle talks about the gender gap in free culture and free and open source software communities. Wikipedia is one of the case studies discussed, but Reagle makes valid observations that it is not so much an exception but a rule in this wider context.
• Further criticism of “most influential people” infographic: On Ethnography Matters, a blog run by four ethnographers, one of the authors, Heather Ford, discussed ^[7] a Wired infographic on “History’s most influential people, ranked by Wikipedia reach”. Like the reviewer in our December issue, the author criticizes the infographic and the accompanying article for lacking any serious description of methodology. She notes that given those shortcomings, the claims made by the article are rather dubious, and the cited research might have well been misquoted. The author further notes that any research that attempts to draw conclusions about “national culture” from analyzing different language Wikipedias runs into a major issue, which is that languages don’t always map easily onto national cultures (consider: what is the national culture of Portuguese or English?). She further illustrates this by discussing how often African-language Wikipedias are edited primarily by individuals living outside the country most often associated with a given language.
• “Sustainability of Open Collaborative Communities:” In an article^[8] published in the Technology Innovation management Review (based on a similar work presented at HICCS 2013 and reviewed in the October 2012 edition of WRN), Kevin Crowston, Nicolas Jullien, and Felipe Ortega present a preliminary comparison of the recruitment efficiency of 36 of the largest Wikipedias. The concept of recruitment efficiency refers to the ability of these Wikis to recruit editors from the total population of readers and potential contributors. The authors estimate this quantity using aggregated data on the number of Internet users and tertiary (college) educated speakers of each of the 36 languages that correspond to the Wikipedias included in their analysis. They find suggestive patterns in the results of this comparison, including: (1) Wikipedias of moderate and smaller size exhibit great variations in terms of their recruitment efficiency; and (2) larger Wikipedias appear less efficient in recruiting new members, suggesting a pattern of decreasing returns to scale. The authors conclude that these findings warrant further investigation and analysis, but that they provide preliminary support for the idea that larger Wikipedias face distinct conditions of community sustainability compared to smaller ones. See the extended summary from the October 2012 Wikimedia Research Newsletter for further details.
• Language comparison algorithm finds new ghosts in England and Scotland: A paper by four Japanese researchers titled “Extracting lack of information on Wikipedia by comparing multilingual articles”^[9] proposes “a method for extracting information that exists in one language version, but which does not exist in another language version.” Their method uses various steps, starting from a users’ search query in their native language Wikipedia, which is automatically translated (using a dictionary) to other “non-native” Wikipedias, and involves use of the link structure between articles, the section structure within one article, and finally the cosine similarity between the nouns of different articles – a low similarity score indicating that information from one article is missing from the other. A small-scale test brought some successes, e.g. the detection of examples in Black dog from England and Scotland that were not present in the corresponding article on the Japanese Wikipedia, but also showed problems with the proposed algorithm. The four authors previously published a related paper titled “Extracting Difference Information from Multilingual Wikipedia”, covered in the April edition of this research report.
• Sentiment analysis of articles about politicians: Researcher Finn Årup Nielsen, who works on a project funded to do “Wikipedia sentiment analysis for companies“, blogged about applying sentiment analysis to articles about politicians on the Danish Wikipedia.^[10]
• “Clustering Wikipedia infoboxes to discover their types”: A paper^[11] presented at the CIKM’12 conference describes a method to use infoboxes to detect the entity type of an article (e.g. “movie” for Avatar). The authors explain that Wikipedia’s existing category system is not sufficient for this: “Because Wikipedia category names are folksonomic, i.e., they are created by a group of people without the control of a central authority, they are also an unreliable source for inferring the conceptual entity type.” As example, the authors cite the article about (the 1981 film) Chariots of Fire, and argue that based on the “categories, a system like Yago would assign to the infobox concepts like film, winner, olympics, culture, university, and sport. However, only film corresponds to the entity described in the infobox.” On the other hand, the naming of infoboxes (as templates) is not consistent enough either: “For example, the entity type Film is associated with template names Infobox Film, Infobox Movie, Television Film Infobox, TV film, James Bond film, Chinese film, infobox Korean film, etc.” The algorithm described in the paper measures the similarity of different infoboxes based on their set of attributes: “For example, the attribute cluster discovered for the entity type Movie includes the attributes {Directed by, Produced by, Written by, Starring}.” The authors report that their clustering algorithm, “WIClust”, performed successfully on a sample of “48,000 infoboxes spanning 862 infobox templates”, and that in some cases it corrects shortcomings of DBpedia, e.g. by discovering “that the templates Infobox Movie, Bond film, Japanese film, Chinese film, and Korean film belong to the same group as Infobox Film.”
• How Indic language Wikipedias fared in 2012: In a blog post, Indian Wikimedian Shiju Alex compared the article numbers, user activity levels and pageviews of Wikipedias in Indic languages between December 2011 and December 2012.^[12]
• Students detect vandalism: Three student projects in a course on machine learning at Stanford University concerned the automatic detection of vandalism edits on Wikipedia.^[13]
• “Algorithmic governance” in the German Wikipedia: Leonhard Dobusch, an assistant professor for organization theory at FU Berlin, blogged about an ongoing research project on the sighted revisions on the German Wikipedia as a case of “algorithmic governance”.^[14]
• Map visualization of links between geotagged articles: The “Collaborative Cybernetics” blog published maps visualizing the links between Wikipedia articles containing geocoordinates^[15] and the geographic distribution of certain topics^[16] (example: skiing-related terms).
• Map of sister cities extracted from Wikipedia: Four researchers from the Barcelona Media Foundation (three of whom also co-authored the paper on featured article pageviews reviewed above) published a preprint^[17] where they “extracted the network of sister cites[sic] as reported on the English Wikipedia, as far as we know the most extensive but certainly not complete collection of this kind of relationships”, and analyze the resulting social network, including a map visualization of worldwide twin city pairings.
• New pageview files available: On his personal blog^[18], Wikimedia Foundation data analyst Erik Zachte announced the release of “Monthly page requests, new archives and reports”.
• Upcoming book on “Global Wikipedia”: A call for chapters has been issued for an upcoming book titled “Global Wikipedia: International and cross-cultural issues in online collaboration”.
• Wikipedia as part of one’s personal memory: In a draft paper titled “Extended Cognition in Science Communication”,^[19] to appear in Public Understanding of Science, David Ludwig (a scholar of philosophy at Columbia University and admin on the German Wikipedia) argues “that we should treat Wikipedia as part of your memory“.
• Is Wikipedia built on “good faith collaboration” or “destructive editing”? : In his 2010 MIT Press book about Wikipedia (now available online under a free license), Joseph Reagle posited that Wikipedia is based on a culture of “Good Faith Collaboration”. In his 2012 thesis at the University of Cambridge, titled “Destructive Editing and Habitus in the Imaginative Construction of Wikipedia”,^[20] User:Thedarkfourth argues against this, highlighting the importance of conflicts instead. This month, Reagle responded to the criticism on his blog,^[21] asserting that it was based on “a new scholasticism. In this view, a work’s contribution consists exclusively of interpreting an interesting phenomenon in the light of dead philosophers”. Reagle argues that this view holds that scholars should look at what came before prior to explaining a new phenomenon; they should first refer to libraries and bibliographies before drawing a new hypothesis onto the whiteboard. He defends his position by arguing that his book is not guilty of ahistoricism, as Wallis seems to imply.

Notes

Vol: 2 • Issue: 12 • December 2012 [archives]

Wikipedia and Sandy Hook; SOPA blackout reexamined

With contributions by: Daniel Mietchen, Piotrus, Junkie.dolphin, Taha Yasseri, Benjamin Mako Hill, Aaron Shaw, Tbayer, DarTar and Ragesoss

How Wikipedia deals with a mass shooting

Northeastern University researcher Brian Keegan analyzed the gathering of hundreds of Wikipedians to cover the Sandy Hook Elementary School shooting in the immediate aftermath of the tragedy. The findings are reported in a detailed blog post that was later republished by the Nieman Journalism Lab.^[1] Keegan observes that the Sandy Hook shooting article reached a length of 50Kb within 24 hours of its creation, making it the fastest growing article by length in the first day among recent articles covering mass shootings on the English-language Wikipedia. The analysis compares the Sandy Hook page with six similar articles from a list of 43 articles on shooting sprees in the US since 2007. Among the analyses described in the study, of particular interest is the dynamics of dedicated vs occasional contributors as the article reaches maturity: while in the first few hours contributions are evenly distributed with a majority of single-edit editors, after hour 3 or 4 a number of dedicated editors show up and “begin to take a vested interest in the article, which is manifest in the rapid centralization of the article”. A plot of inter-edit time also shows the sustained frequency of revisions that these articles display days after their creation, with Sandy Hook averaging at about 1 edit/minute around 24 hours since its first revision. The notebook and social network data produced by the author for the analysis are available on his website. The Nieman Journalism Lab previously covered the role that Wikipedia is playing as a platform for collaborative journalism, and why its format outperforms Wikinews with an interview of Andrew Lih published in 2010.^[2] The early revision history of the Sandy Hook shooting article was also covered in a blog post by Oxford Internet Institute fellow Taha Yasseri, however with a focus on the coverage in different Wikipedia language editions.^[3]

Network positions and contributions to online public goods: the case of the Chinese Wikipedia

A graph with nodes color-coded by betweenness centrality (from red=0 to blue=max).

In a forthcoming paper in the Journal of Management Information Systems (presented earlier at HICSS ’12^[4]), Xiaoquan (Michael) Zhang and Chong (Alex) Wang use a natural experiment to demonstrate that changes to the position of individuals within the editor network of a wiki modify their editing behavior. The data for this study came from the Chinese Wikipedia. In October 2005, the Chinese government suddenly blocked access to the Chinese Wikipedia from mainland China, creating an unanticipated decline in the editor population. As a result, the remaining editors found themselves in a new network structure and, the authors claim, any changes in editor behavior that ensued are likely effects of this discontinuous “shock” to the network. The paper defines each editor as a node (vertex) in the network and a tie (edge) between two editors is created whenever the editors edit the same page in the wiki. They then examine how changes to three aspects of individual editors’ relative connectedness (centrality) to other editors within the network altered their subsequent patterns of contribution.

The main finding is that changes in the three kinds of editors’ connectedness within the network result in differential changes to their editing behavior. First, an increase in the number of direct connections between one editor and the rest of the network (degree centrality) resulted in fewer edits by that editor, and more work on articles they created. Second, an increase in the overall proximity of an editor to the other members of the network (closeness centrality) resulted in fewer edits and less work on articles they created. Third, an increase in the extent to which an editor connected otherwise isolated groups in the network (betweenness centrality) resulted in more edits and more work by that editor on articles they created. Overall, these results imply that alterations to the network structure of a wiki can change both the quantity and quality of editor contributions. The researchers argue that their findings confirm the predictions of both network game theory and role theory; and that future research should try to analyze the character of the network ties created within platforms for large-scale online collaboration, to better understand how changes to network structure may alter collaborative practices and public goods creation.

Quality of pharmaceutical articles in the Spanish Wikipedia

Ibuprofen, one of the World Health Organisation‘s “essential drugs”, a topic covered in detail by the Spanish-language Wikipedia.

In an online early version of an upcoming article in Atención Primaria,^[5] researchers at the Miguel Hernández University of Elche and the University of Alicante have benchmarked articles on pharmaceutical drugs in the Spanish Wikipedia against information available in a pharmaceutical database, Vademécum.^[6] A subset of the Vademécum corpus of 3,595 drugs was created using simple random sampling without replacement, consisting of 386 drugs. Of these, 171 (44%) had entries on the Spanish Wikipedia, which were then scrutinized along several dimensions in May 2012. Usage of the drug was correctly indicated in 155 (91%) of these articles, dosage in 26 (15%), and side-effects in 64 (37%), with only 15 articles (9%) scoring well in all of these dimensions. The researchers conclude that, while Wikipedia has a high potential to help with the dissemination of pharmaceutical knowledge, the Spanish-language edition does not currently live up to this potential. As a possible solution, they suggest the pharmaceutical community more actively participate in editing Wikipedia. The list of the drugs involved has not been made public, since a similar study is currently underway whose results may be distorted by targeted intervention. The authors have signalled to this research report their intention to make the list available after this second study is complete.

Wikipedia editing patterns are consistent with a non-finite state model of computation

A paper posted to ArXiv^[7] by SFI‘s Omidyar fellow Simon DeDeo presents evidence for non-finite state computation in a human social system using data from Wikipedia edit histories. Finite state-systems are the basis for the study of formal languages in computer science and linguistics, and many real-world complex phenomena in biology and the social sciences are also studied empirically by assuming the existence of underlying finite-state processes, for the analysis of which powerful probabilistic methods have been devised. However, the question of whether the description of a system truly entails a finite or a non-finite, unbounded number of states, is an open one. This is significant from a functionalist point of view: can we classify a system by its computational properties, and can these properties help us better understand how the system works regardless of its material details?

The paper’s contribution lies in its proof of a probabilistic generalization of the pumping lemma, a device used in theoretical computer science as a necessary condition for a language to be described by only a finite number of states. The lemma is applied to the edit histories of a number of the most frequently edited articles in the English Wikipedia, after being properly transformed into coarse-grain sequences of “cooperative” or “non-cooperative (reversion) edits (reverts being identified by means of their SHA1 field). A Bayesian argument is applied to show that the lemma cannot hold for a majority of sequences, thus showing that Wikipedia’s collaborative editing system as a whole cannot be described by any aggregation of finite-state systems. The author discusses the implications of this finding for a more grounded study of Wikipedia’s editing model, and for the identification of detailed computational models of other social and biological systems.

Wikipedia as our collective memory

A protester on Tahrir Square during the 2011 Egyptian revolution.

Michela Ferron, a member of the SoNet (Social Networking) research group at the Bruno Kessler Foundation in Trento, Italy submitted her PhD thesis^[8] in December 2012. She examined the idea of viewing Wikipedia as a venue for collective memory and the language indicators of the dynamic process of memory formation in response to “traumatic” events. Parts of the thesis have already been published in journals and conference proceedings, such as WikiSym 2011 and 2012 (cf. presentation slides).

A full chapter is dedicated to the background on the concept of collective memory and its appearance in the digital world. The thesis continues with an analysis of “anniversary edits”, showing a significant increase in editorial activities on articles related to traumatic events during the anniversary period compared to a large random sample of “other” articles. More detailed linguistic indicators are introduced in the next chapter. It is statistically shown that the terms related to affective processes, negative emotions, and cognitive and social processes occur more often in articles on traumatic events; “Specifically, the relative number of words expressing anxiety (e.g., “worried”), anger (e.g., “hate”) and sadness (e.g., “cry”) was significantly higher in articles about traumatic events”.

In the next step, Ferron tried to distinguish between human-made and natural disasters. It has been observed that “human-made traumatic events were characterized by language referring to anger and anxiety, while the collective representation of natural disasters expressed more sadness”. Finally, a detailed case study of the talk pages of articles on the 7 July 2005 London bombings and the 2011 Egyptian revolution was carried out, and language indicators, especially those related to emotions, were investigated in a dynamic framework and compared for both examples.

SOPA blackout decision analyzed

A First Monday article^[9] reviews several aspects of the Wikipedia participation in the 18 January 2012 protests against SOPA and PIPA legislation in the US. The paper focuses on the question of legitimacy, looking at how the Wikipedia community arrived at the decision to participate in those protests.

The English Wikipedia landing page, symbolically its only page during the blackout on January 18, 2012

The paper provides an interesting discussion of legitimacy in Wikipedia’s governance, and discusses the legitimacy of the decision to participate in the protests. The author notes that the initiative was given a major boost by Jimmy Wales’ charismatic authority, as Wales posted a straw poll about the issue on his talk page on December 10, 2011, as while the issue was discussed by the community beforehand (for example, in mid-November at the Village Pump), those discussions attracted much less attention. It is hard to say whether the protest would have happened without Jimbo’s push for more discussion, as it veers towards “what if” territory; as things happened, it is true that Jimbo’s actions began a landslide that led to the protests. However, this reviewer is more puzzled at the claim made in the introduction to the article that the discussion involved a “massive involvement of the Wikimedia Foundation staff”. While several WMF staffers were active in the discussions in their official capacity, and while the WMF did issue some official statements about the ongoing discussion, the paper certainly does not provide any evidence to justify the word “massive”.

The paper subsequently notes that the WMF focused on providing information and gently steering the discussion, without any coercion; this hardly justifies the claim of “massive involvement”. At the very least, a clear explanation is necessary of precisely how many WMF staffers participated in the discussion before such a grandiose adjective as “massive” is used. It is true that the WMF staffers helped push the discussion forward, but this reviewer believes that the paper does not sufficiently justify the stress it puts on their participation, and thus may overestimate their influence.

The third part of the paper discusses how the arguments about legitimacy or the lack of it framed the subsequent discourse of the voters. The author notes that after initial period of discussing SOPA itself, the discussion of whether it was legitimate or not for Wikipedia to become involved in the protest took over, with a major justification for it emerging in the form of an argument that it was legitimate for Wikipedia to protest against SOPA as SOPA threatened Wikipedia itself. While this is an interesting claim, unfortunately, other than citing one single comment, no other qualitative or quantitative data are provided; nor is the methodology discussed. We are not told how many individuals voted, how many commented on legitimacy or illegitimacy, how many felt that Wikipedia is threatened; we do not know how the author classified comments supporting any of the viewpoints, or the shifts in the discussion … this list could unfortunately go on. In one specific example drawn from the conclusion, the author writes that “The main factor that shaped the multi-phased process was the will to have the community accept the final decision as legitimate, and avoid backlash. This factor especially influenced those who are suspected of relying on traditional means of legitimacy such as charisma or professionalism.” At the same time, we are provided with no number, no percentage, and certainly no correlation to back up this claim. Without a clear methodology or distinct data it is hard to verify the author’s claims and conclusions.

The introduction also notes that “the mass effort of planning an effective political action was not something “anyone [could] edit”” and “the debate preceding the blackout did not follow Wikipedia’s open and anarchic decision-making system”; unfortunately this reviewer finds no justification for those rather strong claims anywhere else in the article.

Overall, this is an interesting paper about legitimacy in Wikipedia, but it seems to overreach when it tries to draw conclusions from the data that is simply not presented to the reader. It suffers from a failure to explain the research’s methodology, making verification of the claims made very hard. Due to the lack of hard data, most conclusions are unfortunately rendered dubious, and the paper has a tendency to make strong claims that are not backed up by data or even developed later on.

Bots and collective intelligence explored in dissertation

Rats (blue trace) interacting with a rat-sized robot (red) controlled by a human who in turn perceives the rat’s movements through those of a human-sized avatar in a virtual reality environment.^[10] The video was uploaded to Wikimedia Commons by the Open Access Media Importer Bot.

In his Communication and Society PhD dissertation,^[11] Randall M. Livingstone of the University of Oregon explores the relationship between the social and technical structures of Wikipedia, with a particular focus on bots and bot operators. After a fairly broad literature review (which summarizes the basic approaches to Wikipedia studies from new media theory, social network analysis, science and technology studies, and political economy), Livingstone gives a concise history of the technical development of Wikipedia, from UseModWiki to MediaWiki, and from a single server to hundreds.

The most interesting chapters for Wikipedians will be V – Wikipedia as a Sociotechnical System – and VI – Wikipedia as Collective Intelligence. Chapter 5 looks at the ways the editing community and the evolution of software (both MediaWiki and the semi-automated tools and bots that interact with editors and articles) “construct” each other. Based on 45 interviews with bot operators and WMF staff, this chapter gives an interesting and varied picture of how Wikipedia works as a sociotechnical system. It will in part be a familiar account to the more tech-minded Wikipedians, but offers an accessible overview of bots and their place in the ecosystem to editors who normally steer clear of bots and software development. Chapter 6 looks at theories of intelligence and the concept of collective intelligence, arguing that Wikipedia exhibits (at least to some extent) the key traits of stigmergy, distributed cognition, and emergence.

Briefly

• “History’s most influential people” according to Wikipedia: While more in the realm of popular science, Wired UK, among others, published^[12] an infographic attributed to César Hidalgo, head of the MIT Media Lab‘s Macro Connections group, visualizing “History’s most influential people”. Unfortunately, beyond noting that rankings “are based on parameters such as the number of language editions in which that person has a page, and the number of people known to speak those languages” the small article does not provide any methodology, nor does it provide much discussion. Until a more extensive description is released, the current graph, while pretty, is little more than a trivia piece.
• Teachers say 75% of teens use Wikipedia (or online encyclopedias) for research assignments: In a Pew Research survey among more than 2000 US middle and high school teachers^[13] 75% said that their teenage students use “Wikipedia or other online encyclopedia” in research assignments, making online encyclopedias the second most popular source for students behind search engines such as Google. This number was lower (68%) “among teachers of the lowest income students (those living below the poverty line)” and higher (80%) for those teaching “mostly upper and upper middle income” students, and it also varied by subject (between 69% for teachers of English and 82% for science teachers). The survey report cautions that the sample “skews towards ‘cutting edge’ educators who teach some of the most academically successful students in the country”.
  

  

  The Google matrix of Wikipedia entries, from an earlier paper by the same authors of this study.^[14]

• “Wikipedia communities” as eigenvectors of its Google matrix: An ArXiv preprint^[14] studies the “Spectral properties of Google matrix of Wikipedia and other networks”. This Google matrix consists of entries for each pair of pages (for the English Wikipedia, including non-mainspace pages like portals), roughly speaking modelling the behavior of a surfer who goes from one page to any of those that it links to, with equal probability (or, with probability , jumps to a random page; the damping parameter is set to around 0.85 in the Google search engine). The PageRank appears as the eigenvector of this matrix for the eigenvalue . The paper studies the spectrum (eigenvalues) and eigenvectors apart from this special case, interpreting them as certain topic areas: “the eigenvectors of the Google matrix of Wikipedia clearly identify certain communities which are relatively weakly connected with the Wikipedia core when the modulus of corresponding eigenvalue is close to unity. For moderate values of we still have well defined communities which however have stronger links with some popular articles (e.g. countries) that leads to a more rapid decay of such eigenmodes.”
• Serial singularities: developing of a network organization by organizing events: In a paper published in the Schmalenbach Business Review,^[15] Leonhard Dobusch and Gordon Müller-Seitz from the Freie Universität Berlin suggest that research on organized events has tended to treat those events as isolated and singular events. Using interviews and other data on Wikimania, chapter meetings, and local meet-ups over several years, the authors challenge this idea and show how many different events on different scales and scopes – each with a distinct character – can interact and reinforce each other to help drive the nature of a large distributed organization like Wikimedia.
• The web mirrors value in the real world: comparing a firm’s valuation with its web network position: In a MIT Sloan Working Paper,^[16] Qiaoyun Yun and Peter Gloor create a measure of US and Chinese firms “social network” position by looking at how those firms are linked to from a variety of web sources – prominently Wikipedia. They find a positive correlation between betweenness centrality of a firm in a social network constructed from links online and its innovation capability and financial performance. They find that Wikipedia only predicts a firm’s performance in the US.
• Teahouse analyzed: Jonathan Morgan, Sarah Stierch, Siko Bouterse and Heather Walls, from the Wikimedia Foundation Teahouse team, report on the impact of the initiative on 1,098 new Wikipedia contributors who joined the Teahouse between February and October 2012, in a paper to be presented at CSCW ’13.^[17] The study reports that participants in the project “make more edits overall, and edit longer”, “make more edits, to more articles” and “participate more in discussion spaces” compared to non-visitors. This paper is part of a research track entirely dedicated to Wikipedia Supported Collaborative Work, featuring three other studies.
  

  

  Slides from the recently published Article Feedback research report.^[18]

• Article feedback: The Wikimedia Foundation published an update about the Article feedback tool on the English Wikipedia, providing statistics about the usage of the feature, and about the moderation activities for the feedback provided.^[18]
• New review of Good Faith Collaboration: The reviewer locates^[19] Joseph Reagle’s 2010 book about Wikipedia (free online version) as following in a wider context of research on Wikipedia: “The reliability of the encyclopaedia’s content.. and quantitative analysis of large-scale public datasets formed the predominant approach in early empirical research on Wikipedia … This was followed by a more social approach and the adopting of qualitative methods. In this switch to social norms and away from an ethnographic approach, Reagle’s book is a main reference, particularly in terms of its cultural and historical specificity.” Overall, the review finds that “The book is well documented, with an elaborative but accessible writing style, which is at times provocative. It results in a form of rich composition of eight pieces (chapters) of Wikipedia ‘puzzle’, even if some readers might miss a more explicit continuum linking the lines together. Finally, the book is a primary reference point for researchers aiming to study Wikipedia, especially for those unfamiliar with it.”
• Measuring the impact of Wikipedia for GLAM institutions: Ed Baker, software developer at the Natural History Museum in London, has started a series of blog posts on “the impact and use of Wikipedia by organisations”.^[20] In the first post, he looked at how the scope of pages linking to the NHM’s website fits with the overall scope of the institution when pages are ranked either by number of page views or by number of links to the NHM. The latter approach could help identify opportunities for a collaboration between GLAM institutions and the Wikimedia communities.

Notes

Vol: 2 • Issue: 11 • November 2012 [archives]

Movie success predictions, readability, credentials and authority, geographical comparisons

With contributions by: Piotrus, Benjamin Mako Hill, Tbayer, DarTar, Adler.fa, Hfordsa, Drdee

Early prediction of movie box-office revenues with Wikipedia data

An open-access preprint^[1] has announced the results from a study attempting to predict early box-office revenues from Wikipedia traffic and activity data. The authors – a team of computational social scientists from Budapest University of Technology and Economics, Aalto University and the Central European University – submit that behavioral patterns on Wikipedia can be used for accurate forecasting, matching and in some cases outperforming the use of social media data for predictive modeling. The results, based on a corpus of 312 English Wikipedia articles on movies released in 2010, indicate that the joint editing activity and traffic measures on Wikipedia are strong predictors of box-office revenue for highly successful movies.

The authors contrast their early prediction approach with more popular real-time prediction/monitoring methods, and suggest that movie popularity can be accurately predicted well in advance, up to a month before the release. The study received broad press coverage and was featured in The Guardian, the MIT Technology Review and the Hollywood Reporter among others. The authors observe that their approach, being “free of any language based analysis, e.g., sentiment analysis, could be easily generalized to non-English speaking movie markets or even other kinds of products”. The dataset used for this study, including the financial and Wikipedia activity data is available among the supplementary materials of the paper.

Readability of the English Wikipedia, Simple Wikipedia, and Britannica compared

The automated readability index, one of the readability metrics used in the study^[2]

A study^[2] by researchers at Kyoto University presents a detailed assessment of the readability of the English Wikipedia against Encyclopedia Britannica and the Simple English Wikipedia using a series of readability metrics and finds that Wikipedia “seems to lag behind the other encyclopedias in terms of readability and comprehensibility of its content”. The paper, presented at CIKM’12, uses a variety of metrics spanning syntactical readability indices (such as Flesch reading ease, the automated readability index and the Coleman–Liau index) as well as metrics based on word popularity (including the Dale–Chall readability formula and word frequency indices derived from Google News or the American National Corpus).

The authors prepared a corpus of matching articles for the purpose of comparison between the English and Simple English Wikipedia. It should be noted that the authors didn’t perform a random selection of articles, but selected a sample based on the existence of a corresponding article in Simple Wikipedia. The findings of the first analysis indicate that Simple Wikipedia consistently outperforms the English Wikipedia on all readability metrics. Wikipedia also appears to contain on average more proper nouns than Britannica – which, the authors speculate, may be due to specific editorial policies. The second section of the paper measures readability for 500 articles for each one of eight topic categories selected from DBpedia (biology, chemistry, computing, economics, history, literature, mathematics, and philosophy).

The comparison indicates that articles in the computing category are the most readable by syntactical and familiarity measures. Biology and chemistry, on the other hand, seem to include the most difficult articles. The final section reviews the readability of Britannica articles, in particular comparing the readability of articles in the “introductory” class with that of Simple Wikipedia articles and the readability of “encyclopedia” class articles with that of Wikipedia articles. The findings indicate that Britannica outperforms Wikipedia in readability overall, while introductory articles outperform Simple Wikipedia articles. It should be noted that the comparisons were not performed on matched pairs and that the authors do not specify what criteria were used to sample articles from Britannica.

A paper whose preprint was previously covered in this research report, and now published as a full research article in PLOS One,^[3] found that the Simple English Wikipedia has a higher degree of complexity than the corpus of Charles Dickens‘ books when measured via the Gunning fog index, but is less complex than the British National Corpus, “which is a reasonable approximation to what we would want to think of as ‘English in general’”. See also the September issue of this research report for a summary of a third readability study which had applied the standard Flesch Reading Ease test to the English and Simple English Wikipedias.

Wikipedia favors established views and scientifically backed knowledge

An article appearing in Information, Communication & Society^[4] studies the discussion pages of English and German September 11 attacks articles, contributing to the ongoing debates on collaborative knowledge creation in the wiki Web 2.0 context, participation of experts and amateurs on Wikipedia, and, indirectly, reliability of Wikipedia. The article’s research question, coming from the sociology of knowledge and social constructivism perspectives, asks to what degree Wikipedia’s “anyone can edit” policy democratizes the production of knowledge, removing it from traditional hierarchies “between experts and lay participants”. The term democratization here is used in the context of such theoretical concepts as wisdom of crowds, participatory culture, produsage and (more critically) the notions of cult of the amateur or digital Maoism. All of these refer to the fact that Wikipedia’s editors are more often amateurs (“lay participants”) than professionally recognized experts.

Using the grounded theory approach, the study focuses not on editors, but on their arguments. It finds that due to community-upheld Wikipedia policies such as Wikipedia:Reliable sources, dissenting opinions (“traditionally marginalized types of knowledge”) such as various conspiracy theories are still marginalized or straight-out excluded; according to the author, this “did not lead to a ‘democratization’ of knowledge production, but rather re-enacted established hierarchies”. The finding should be taken in a certain context; as the author notes, the article was written by amateurs (“lay participants”), who however decided to reproduce traditional knowledge hierarchies, relegating various conspiracy theories and similar points not backed up to reliable sources to obscurity on Wikipedia. The author also concludes that Wikipedia, like other encyclopedias, is prone to a “scientism bias”, i.e. treating scientifically backed knowledge as “better” than knowledge coming from alternative outlets. This despite the “anyone can edit” motto of Wikipedia, the author finds support for the argument that Wikipedia puts more stress on article quality than democratic participation, or in the words of the author: “Although laypeople apparently play a significant part in the text production, this does not mean that they favor lay knowledge. On the contrary, it is clearly elite knowledge of well-established authorities which is finally included in the article, whereas alternative interpretations are harshly excluded or at least marginalized.”

Side-note: This reviewer found the author’s use of a Firefox add-on Wired-Maker for content analysis rather ingenious, and applauds the mentioning of such a practical methodological tip in their paper.

Trust, authority and credentials on Wikipedia: The case of the Essjay controversy

At the Academy of Management conference in Boston, Dariusz Jemielniak presented a paper on Trust, Control, and Formalization in Open-Collaboration Communities: A Qualitative Study of Wikipedia ^[5]. It is built around a detailed description and interpretation of the Essjay controversy on the English Wikipedia in 2007 about the use of inaccurate credentials by active Wikipedian and administrator Essjay. The paper is framed in terms of the literature from organization theory on trust and control. Jemielniak argues that organization theory suggests that organizations must either be able or willing to trust participants or must rely on control systems which essentially obviate the need for trust. Using ethnographic data from Wikipedia, Jemielniak suggests that Wikipedia — and, perhaps, a series of similar computer-mediated “open-collaboration communities” — instead rely on a series of procedures and “legalistic remedies” which provide a previously untheorized alternative to traditional control systems used in organizations.

The working paper is the first in what Jemielniak suggests will be a series of papers based on a long-term participatory ethnographic study: over the past five years, Jemielniak has edited Wikipedia almost daily and is a steward on Wikimedia projects (as well as the chair of the Wikimedia movement’s newly established Funds Dissemination Committee, and recently announced the committee’s recommendations on funding requests by various Wikimedia organizations totaling US$10.4M). Jemielniak uses his own experience as well as detailed on-wiki records from conversations surrounding the Essjay affair to walk through the controversy and its implications in depth. He discusses how Wikipedians construct authority and initially reacted with indifference to the revelation that Essjay had used fake credentials, how this changed when new information about Essjay’s use of his credentials came to light, how a series of proposals to prevent or respond to such issues in the future were raised, and how the community essentially decided to keep the status quo.

The paper paints a detailed, nuanced, and deeply informed portrait of Wikipedians’ responses to the controversy and the ways in which trust and its relationships to authority and credentials are navigated in the project. The author suggests that the creation of rules and legalistic procedures allowed Wikipedians to walk the line between rejecting descriptions of authority per se while minimizing the effects of inaccurate descriptions of authority by suggesting that editors on Wikipedia should rely much more heavily on users’ experience and on the degree to which particular contributions conform to Wikipedia’s content guidelines.

A working paper by the same author, presented at the annual meeting of the Society for Applied Anthropology^[6] gives an overview of Wikipedia’s culture by reviewing the role of its norms, guidelines and policies.

Network of users communicating on Wikipedia article talk pages (Neff et al., p.22).^[7] Edges connecting two Democrats are colored blue, edges connecting two Republicans in red, and edges representing inter-party dialogue are shown in green.

Briefly

• Being Wikipedian is more important than the political affiliation: In a recent preprint^[7], titled “Jointly they edit: examining the impact of community identification on political interaction in Wikipedia”, researchers have studied the political identity of Wikipedia editors and investigated the effect of the social identity on their editorial activity patterns. The paper starts with a long and comprehensive review of the concept of social identity and generalizes it to online social identity. Based on an analysis of a sample of 1390 editors with known political affiliation – either US Democrat or Republican – they conclude that although the social identity of editors is strongly reflected in their editorial interests – that is, the topics on which they are more active – but that being “Wikipedian” dominates political affiliation when it comes to user pages. In contrast with other social media e.g., the blogosphere, where cross-party interactions are very much underrepresented, it appears that Wikipedian dialogues between editors from opposing parties are relatively profound and notable. On the day before the US presidential election, the paper’s results were highlighted on the Wikimedia blog under the headline “In divisive times, Wikipedia brings political opponents together“.
• Eye-tracking study: Readers look at TOC first, then infobox: A conference paper titled “Looking for genre: the use of structural features during search tasks with Wikipedia”^[8] described the results of an eye tracking study, where readers looking for information in a Wikipedia article tended to look first at the table of contents, then at the article’s infobox. Also, readers frequently “skim and scroll” long articles.
• Edit categories in featured and non-featured articles: This article focuses on some differences between featured and non-featured article. Unsurprisingly, the authors main finding is that the featured articles are more stable after promotion; the interesting contribution of the authors lies more in their detailed methodology, and categorization of various types of edits.^[9]
• How the TV schedule influences Wikipedia pageviews: In Germany, several recent consumer studies have found evidence for a rise of what has been called “second screen“: The parallel use of TV and the Internet. To find a partial answer to the question whether this use is unrelated (e.g. checking emails while the TV is running in the background) or integrates both media, a blogger turned to pageview numbers for the German Wikipedia^[10]. From a still unsystematic analysis, he draws two conclusions: “First, the use of Wikipedia is markedly influenced by the TV schedule. On Saturday evenings in particular, but sometimes also during weekdays, the most viewed Wikipedia entries contain many articles related to the currently showing TV program. Secondly, these articles are primarily viewed while the corresponding show is running on TV.” The author also announced a Perl script to convert the raw pageview data provided by the Wikimedia Foundation into a MySQL database, demonstrated in a live list of the 50 most viewed articles of the German Wikipedia.
• A truthfulness verification system based on Wikipedia: Yang Liu’s master’s thesis (paywalled)^[11]discusses the development of WT-verifier, a “a truthfulness verification system based on Wikipedia” that uses information on Wikipedia, rather than general web searches to perform fact checking. Liu finds that Wikipedia “has high reliability of page contents, due to strict rules for page editing and a strong self-fixing mechanism” and adapts T-verifier, an existing system based on Yahoo! searches, applying it to information on Wikipedia. Liu develops what he calls a “truthfulness aware snippet generation algorithm” and finds that the new approach “significantly increases the precision and recall compared to the original T-verifier approach.”
• Characterizing Wikipedia traffic: A paper presented at the 7th International Conference on Internet and Web Applications and Services^[12] gives a breakdown of Wikipedia traffic for 2009 to the 10 largest wikis with a particular focus on content-type. The authors give a high-level overview of Wikipedia traffic but they do not take the opportunity to dive deeper into the data. The current analysis from the paper can also be found on the Wikimedia Report Card and Wikimedia Statistics page. Suggestions for future research include the following: an in-depth analysis of the temporal dynamics of editing behavior. For example, do we see higher editor activity during holidays? An in-depth analysis of the multi-media files/Wikimedia Commons project. Are there differences between wiki projects regarding the use of Commons image files?
• One-year article ratings dump released: The Wikimedia Foundation announced the release of the complete, anonymous data dump of 11M article ratings collected over 1 year (July 2011 – July 2012) from the English Wikipedia via the article feedback tool (v4).^[13] The dataset is released under a CC0 license.
• Measuring countries’ visibility on Wikipedia: On his “Zero Geography” blog, researcher Mark Graham began a series of posts^[14][15] comparing the “geography of views” for different countries on Wikipedia: “we constructed a list of every single article about a place (towns, monuments, historical events, rivers, buildings etc.) in the top 42 Wikipedia language versions, and then queried the number of views that each of those articles received over a two-year period (2009-2011).” (This is part of ongoing research into geographical aspects of the information on Wikipedia by Graham’s team at the Oxford Internet Institute, and will be featured in an upcoming paper.) Content about US locations received the most views across languages, followed by the UK and Germany. Graham observed that the top 10 list by pageviews shows a lot of similarity to the top 10 lists of countries by number of articles, and by number of edits originating from that country, but noted that “the UK [being] Europe’s most visible country … is quite interesting because it isn’t the country in Europe that uses Wikipedia the most (Germany does)”, conjecturing that this might have to do with language differences.
• Ratio of African Wikipedia readers rising, but still low: Erik Zachte, data analyst at the Wikimedia Foundation, blogged an update about “Wikipedia page reads, breakdown by region”^[16], observing among other things that “Africa still has a long way to go to gain equal access to internet: with about 15% of the worlds population, 1.4 % of Wikipedia page views is low, but still one and a half as much as 3 years ago.”

References

Vol: 2 • Issue: 10 • October 2012 [archives]

WP governance informal; community as social network; efficiency of recruitment and content production; Rorschach news

With contributions by: Piotrus, Adler.fa, Bdamokos, Ragesoss, Tbayer, and Phoebe

Wikipedia governance found to be mostly informal

A paper in the Journal of the American Society for Information Science and Technology, coming from the social control perspective and employing the repertory grid technique, has contributed interesting observations about the governance of Wikipedia.^[1] The paper begins with a helpful if cursory overview of governance theories, moving towards the governance of open source communities and Wikipedia. That cursory treatment is not foolproof, though: for example, the authors mention “bazaar style governance”, but attribute it incorrectly—rather than the 2006 work they cite, the coining of this term dates to Eric S. Raymond‘s 1999 The Cathedral and the Bazaar. The authors have interviewed a number of Wikipedians and identified a number of formal and informal governance mechanisms. Only one formal mechanism was found important—the policies—while seven informal mechanisms were deemed important: collaboration among users, discussions on article talk pages, facilitation by experienced users, individuals acting as guardians of the articles, inviting individuals to participate, large numbers of editors, and participation by highly reputable users. Notably, the interviewed editors did not view elements such as administrator involvement, mediation or voting as important.

The paper concludes that “in the everyday practice of content creation, the informal mechanisms appear to be significantly more important than the formal mechanisms”, and note that this likely means that the formal mechanisms are used much more sparingly than informal ones, most likely only in the small percentage of cases where the informal mechanisms fail to provide an agreeable solution for all the parties. It was stressed that not all editors are equal, and certain editors (and groups) have much more power than others, a fact that is quickly recognized by all editors. The authors note the importance of transparent interactions in spaces like talk pages, and note that “the reported use of interaction channels outside the Wikipedia platform (e.g., e-mail) is a cause for concern, as these channels limit involvement and reduce transparency.” Citing Ostrom’s governance principles, they note that “ensuring participation and transparency is crucial for maintaining the stability of self-governing communities.”

Social network analysis of Wikipedia community

This paper looks at the relationships between Wikipedians from the social network analysis perspective (nodes are defined as authors, and links as indicators of collaboration on the same article), treating Wikipedia as an online social network (similar to Facebook).^[2] The authors note that while Wikipedia is not primarily a social network site, it has enough social networking qualities to justify being seen as such. They find that Wikipedia can be seen as a very good source of information about online relationships between actors, due to the transparent and public nature of its data. The authors present a brief overview of previous work with a similar approach. Rather unsurprisingly, the authors find that in the very early days of Wikipedia, editors were much more likely to know one another and collaborate on articles than in the later years. They find that the number of editors is highly correlated to the editors’ familiarity with one another, and is more relevant than the number of articles, as they find that from 2007, when the number of editors roughly stabilized, so did their levels of connectedness through collaboration.

The paper shows that with very few exceptions (low activity, specialized editors) all Wikipedia editors are connected to one another, and there are no isolated groups (or topic areas). The authors also find that the Wikipedia collaborations can be analyzed using the small-world network approach (suggesting that the distance between editors, defined as the average path length, with links being articles contributed to, is very small). The article focuses primarily on the mathematical side of social network analysis, and unfortunately offers little commentary or analysis of the findings. The validity of the results can also be questioned, as the authors treat bots and semi-automated accounts as “regular authors”; considering that the majority of Wikipedia articles have been edited by bots or editors using scripts, the finding that editor A can be connected to editor B through the fact that they both edited different pages which in turn were edited by the same bot or script-equipped editor is hardly surprising.

Wikipedia’s article on the Rorschach inkblot test found to have a limited effect on the test’s results

One of the inkblot images from the original Rorschach test, illustrating the Wikipedia article about it (here shown without the list of popular responses that according to Schultz and Brabender appears to influence test results)

Earlier this month, the Journal of Personality Assessment published a paper titled “More Challenges Since Wikipedia: The Effects of Exposure to Internet Information About the Rorschach on Selected Comprehensive System Variables”.^[3] Summarizing past events (well-known to Wikipedians) from the point of view of psychologists adhering to the Rorschach test as a diagnostic tool, they write: “The availability of Rorschach information online has become of even greater concern in the last few years, since James Heilman, an emergency-room physician from Canada, posted images of all 10 Rorschach inkblots on the popular online encyclopedia, Wikipedia (Cohen, 2009; Wikipedia, 2004[sic]). This Wikipedia article also describes “common responses” to each blot, which frequently correspond to percepts that would be scored Popular under the current coding rules of Exner’s (2003) Comprehensive System (CS).” They remark that “Although many psychologists decried the publishing of the Rorschach inkblots on Wikipedia, before this study, no published studies had examined whether viewing the inkblots and other Rorschach information posted on Wikipedia would impact examinees’ scores.” (As reported last year in this newsletter – see “Psychologists gauge impact of Wikipedia’s Rorschach test coverage” – one of the authors had coauthored a study that had investigated the rise in prominence of information about the test on the Internet due to Wikipedia, but not tested its impact on the test itself.)

Before reporting their own results, the authors cite an unpublished dissertation,^[4] which had compared test subjects’ Rorschach results before and after reading the article. Its tentative results suggested a “significant increase in shading responses [which] then likely affected the corresponding increase in [one variable], but otherwise indicated “that the majority of CS variables do not appear to be affected by exposure to information in the Wikipedia article.”

The authors’ own study involved 50 participants, half of whom had to read an excerpt of the Rorschach test article (while the control group read one of the Philadelphia Phillies article) before trying to “fake good” on the test, impersonating a character which would have a huge incentive to achieve certain results in the test (“Jack is a 35-year-old father of two wonderful children …The judge ordered that Jack have a psychological evaluation done to determine whether or not he should be given custody of his kids.”)

Among the test features defined in the “CS” system, only “Populars” was found to differ significantly “between the control and experimental groups [...] likely due to the fact that the Rorschach [Wikipedia article excerpt] provided pictures of each of the inkblots, along with “common responses,” which, in many cases, corresponded to those responses that are actually coded as Popular according to the CS. However, the Wikipedia information on its own did not appear to directly impact other variables associated with perceptual accuracy.”

Commenting on the paper, Heilman told this research report:

That reading about the Rorschach before testing affects scores in a group of “normal” individuals is not really surprising. This analysis, however, does not show that the availability of information regarding psychological tests affects clinical important outcomes.

Efficiency of Wikipedia in editor recruitment and content production

A paper titled “Is Wikipedia Inefficient? Modelling Effort and Participation in Wikipedia”^[5] will be presented at next year’s HICSS ’13 conference. The main research concern of the authors is whether the saturation observed in the growth of Wikipedia is due to the maturity of the project or is rather caused by editorial obstacles and inefficient collaboration processes. To address this question, they try to investigate the efficiency of collaboration in 39 language editions of Wikipedia. Two different processes are studied. 1) editor recruitment; the ability of Wikipedia projects to attract editors from the pool of potential editors and 2) the article creation process. For each of these two processes corresponding input and output parameters are chosen and by applying a set of Data Envelopment Analysis the relative efficiency of language projects is calculated. For the editor recruitment process the input parameter is the size of the population speaking the language, having access to Internet and being at a tertiary-level of education and the output is the number of Wikipedia editors contributing to the Wikipedia edition of that language. It is shown that the efficiency of some language editions, e.g. Estonian, Hungarian, Norwegian, and Finnish, are much higher than some other language editions, e.g., Malaysian, Arabic, and Chinese. A decreasing return to scale is reported for all of the studied projects; however, the effect is more pronounced for larger ones. In other words, larger projects can be considered as inefficient in attracting new editors. For the production process, the number of Wikipedia editors is considered this time as the input and 3 outputs: number of edits, number of articles, and number of Featured articles. Here, the results generally suggest that for the larger projects the returns to scale are systematically decreasing, showing the difficulties of maintaining the efficiency of the workflow as the project grows. Some projects, such as the Malaysian and Persian Wikipedias, are not as successful in editor recruitment but are still efficient in creating articles given the capacity of their human resources. As for the quality of articles, it is shown that in larger projects like French and German, the focus is more on increasing the quality of the existing articles, whereas in intermediate-size projects, e.g., Russian and Italian, the main effort is still on increasing the number of articles.

The paper notes a positive correlation between efficiency in the number of edits and the efficiency in number of articles and featured articles. Among the limitations of the study, the authors name the time period of the analysed data, being limited to one month, and the possible flaws in the demographic data used to estimate the input of the editor recruitment process. Excluding contributions from unregistered users due to technical reasons could also have induced biases in the results. Since the article starts by raising the question of efficiency of Wikipedia in general, it ends up by comparing different language editions to each other and presenting the results in only relative terms. The English Wikipedia, which could be a benchmark for such comparisons, is entirely excluded from the study. More importantly, applying the data envelopment analysis, which is originally introduced for evaluating activities of not-for-profit entities participating in public programs, on Wikipedia activity data is not well justified.

Student use of Wikipedia

How students find and evaluate information is a perpetual concern for librarians, who act as educators and guides to finding the best resources for student information needs as well as collection curators. Since the arrival of Wikipedia, librarians have grappled with how the site fits in with and compares to a more traditionally published and reviewed collection, and how best to help students understand and use Wikipedia. This study is an up-to-date addition to the body of literature on this subject.^[6] Colón-Aguirre and Fleming-May use a coded qualitative interview approach to understanding undergraduate opinions about Wikipedia, compared to their use of and attitude towards traditional library resources.

The authors conducted interviews with 21 undergraduate students in one college in a large public university in the United States. Based on student responses about their research habits, the authors divided their respondents into three categories: avid library users, occasional library users, and library avoiders. While all categories of students used Wikipedia, there were differences in purpose; avid library users used Wikipedia to gather background information before turning to library-supplied resources like books and journals, while library avoiders relied more on Wikipedia and were lost if they could not find the information they needed on the site or via Google searches. Most of the students interviewed reported getting to Wikipedia via Google or other search engines, and the authors do not report any deep awareness by the students of how the site works or how to evaluate articles; awareness of ability to contribute was not mentioned. Student use of the library versus Wikipedia was also influenced by their perceptions of library resources being difficult to use (both in-person stacks and subscription online resources), particularly compared to the ease of using Wikipedia and online searching; students were also swayed in whether they used the library by their assignment requirements and faculty advice, including professors who advised against using Wikipedia as being “not credible” and required using library resources specifically.

The authors conclude that librarians need to work more with teaching faculty to craft research assignments, and that hands-on instruction in the use of the library does aid student comfort with research. This short article will be most of interest to practicing librarians and undergraduate instructors, who will doubtless see reflections of their own students in the student interviews. Wikipedians who are involved in academic classroom education and outreach will also find this study interesting, if for no other reason than to reinforce the importance of helping students become more knowledgeable about the ways that Wikipedia works with and differs from traditional academic publications.

In brief

• “Conflict positively influences group performance”: Investigating the question “Does conflict matter in the success of mass collaboration?”, a paper in the Chinese Journal of Library and Information Science^[7] investigates conflict on Wikipedia, analyzing it from the social network analysis perspective (nodes are defined as individuals, and links, as indicators of conflict), and differentiating between positive and negative types of conflict. Their goal is to increase understanding of the conflict mechanism in the mass collaboration setting. The authors find that “that participation positively influences task complexity, conflict, and group performance; task complexity positively influences group performance but negatively influences conflict; and conflict positively influences group performance”.
• Generating a lexical network from Wiktionary: ^[8] The researcher has created an open source tool – available at http://dbnary.forge.imag.fr/ – that extracts a lexical network (including definitions, translations, synonyms, antonyms, etc.) from Wiktionary data in RDF format, that can be used in existing semantic tools. The author notes that because Wiktionary – unlike traditional dictionaries – treats homonyms (words that share the same spelling and pronunciation but have a different meaning) on single pages with multiple etymology sections, it has not been possible to properly attribute the senses and lexical relations to the proper etymologies (i.e. lexemes).
• How are article edits and page views related? We still don’t know.: ^[9] This paper attempts to explore the relationship between the “production” and “consumption” of Wikipedia content: the edits that build articles, and the page views from readers. For broad topic areas on English Wikipedia (such as articles in Category:Dance and its subcategories), the pattern of edits mirrors the overall trend of editing activity—rising exponentially until peaking around 2007, with a linear decline in edit rate since then. Page views for these topic areas, by contrast, show an approximately linear rise page views since late 2007 (which is the earliest period for which we have article traffic statistics). According to the authors, this pattern “conforms to a two-phase evolution framework: one of production followed by consumption”, although they do not attempt to establish a causal link between the article content maturation and readership. Unfortunately, the lack of earlier data on article traffic makes it hard to learn much from the relationship between edit rate and article traffic, without taking a more fine-grained approach to identify articles or topic areas whose early phases of rising and peaking edit rates are also covered by page view data.
• More WikiSym reports: Two more reports from August’s annual WikiSym conference were published this month, by the recipient of a travel grant from the UK Wikimedia chapter,^[10] and by a Natural Language Processing (NLP) researcher^[11] who dubbed the conference “WikipediaSym” because “the conference submissions were mostly inclined towards the information analysis and social aspects of using wikis, in particular Wikipedia, and there were very few submissions on the actual applications of wikis (or wiki-like systems) and the open collaboration context”. (See also the overview report in the last issue of the research report)
• German centrality: A discussion paper examined “Centrality and Content Creation in Networks [in] The Case of German Wikipedia”. ^[12]
• Systemic bias: Slides of a presentation by a librarian at the University of Massachusetts Amherst (and active Wikipedian) concern “Systemic Bias in Wikipedia: What It Looks Like, and How to Deal with It”.^[13]
• Few users who edit Middle East/North Africa articles are from the region: A brief conference paper titled “The vocal minority: Local self-representation and coediting on Wikipedia in the Middle East and North Africa”^[14] (presented in a slightly different form at a Workshop at 2012 ACM Web Science Conference in June) analyzed the talk pages of English Wikipedia users who had edited articles geotagged in that region (MENA) “to assess the self-declared locational affiliations of the authors (i.e. where they live, work or were born)” and found that “there exists few authors claiming to be from the MENA region, except for Israel, Iran and to a much lesser extent Egypt.”
• Article Feedback tool as means of “peripheral participation”: A paper to be presented at CSCW ’13^[15] describes the main findings from the early tests of the Article Feedback v5 on the English Wikipeida, from the lens of legitimate peripheral participation theory. The study reviews the costs and benefits of expanding reader contributions to Wikipedia, using both quantitative and qualitative methods. The results, according to the authors (members of the Wikimedia Foundation team working on the tool), indicate that peripheral contributors add value to the encyclopedia as long as the cost of identifying low quality contributions remains low.
• Dynamics of read and edit rates on Wikipedia: The ECCS’12 Conference on Complex Systems saw the presentation of a paper titled “From Time Series to Co-Evolving Functional Networks: Dynamics of the Complex System ‘Wikipedia’”^[16], reporting on research about the “access-rate time series and edit-interval time series” of articles on the English Wikipedia, and about ” three organizational and dynamical networks …: (i) the network of direct links between Wikipedia articles, (ii) the usage network as determined from cross-correlations between access-rate time series of many pairs of articles, and (iii) the edit network as determined from co-incident edit events. The major goal is to find correlations between components of these three networks that characterize the dynamics of information spread in the complex system”.
• Wikipedia articles compared to open source software projects: A paper titled “Similarities, challenges and opportunities of Wikipedia content and open source projects”^[17] argues that “the evolution of Wikipedia pages and the OSS projects share some commonalities in terms of their evolutionary patterns; in particular, it was found that a predefined, cubic model could be used to explain several of the similarities in ‘abandoned’ or ‘completed’ projects and Wikipedia pages.”

References

Vol: 2 • Issue: 9 • September 2012 [archives]

“Rise and decline” of Wikipedia participation, new literature overviews, a look back at WikiSym 2012

With contributions by: Piotrus, Phoebe, DarTar, Benjamin Mako Hill, Ragesoss and Tbayer

“The rise and decline” of the English Wikipedia

A paper to appear in a special issue of American Behavioral Scientist (summarized in the research index) sheds new light on the English Wikipedia’s declining editor growth and retention trends. The paper describes how “several changes that the Wikipedia community made to manage quality and consistency in the face of a massive growth in participation have lead to a more restrictive environment for newcomers”.^[1] The number of active Wikipedia editors has been declining since 2007 and research examining data up to September 2009^[2] has shown that the root of the problem has been the declining retention of new editors. The authors show this decline is mainly due to a decline among desirable, good-faith newcomers, and point to three factors contributing to the increasingly “restrictive environment” they face.

Rate of desirable newcomers. The proportion of newcomers falling into the two desirable quality classes is plotted over time.

Rejection of desirable newcomers. The proportion of reverted desirable newcomers is plotted over time.

Survival of desirable newcomers. The proportion of surviving desirable newcomers is plotted over time.

First, Wikipedia is increasingly likely to reject desirable newcomers’ contributions, be it in the form of reverts or deletions. Second, it is increasingly likely to greet them with impersonal messages; the authors cite a study that shows that by mid 2008 over half of new users received their first message in a depersonalized format, usually as a warning from a bot, or an editor using a semi-automated tool^[3]. They show a correlation between the growing use of various depersonalized tools for dealing with newcomers, and the dropping retention of newcomers. The authors speculate that unwanted but good faithed contributions were likely handled differently in the early years of the project – unwanted changes were fixed and non-notable articles were merged. Startlingly, the authors find that a significant number of first time editors will make an inquiry about their reverted edit on the talk page of the article they were reverted on only to be ignored by the Wikipedians who reverted them. Specifically editors who use vandal-fighting tools like Huggle or Twinkle are increasingly less likely to follow the Wikipedia:Bold, revert, discuss cycle and respond to discussions about their reverts.

As a third factor, the authors note that the majority of Wikipedia rules were created before 2007 and have not changed much since, and thus new editors face the environment where they have little influence on the rules that govern their behavior, and more importantly, how others should be behave toward them. The authors note that this violates Ostrom’s 3rd principle for stable local common pool resource management, by effectively excluding a group that is very vulnerable to certain rules from being able to effectively influence them.

The authors recognize that automated tools and extensive rules are needed to deal with vandalism and manage a complex project, but they caution that the currently evolved customs and procedures are not sustainable for the long term. They suggest Wikipedia editors could copy the strategy of distributed, automated tools that have proven so effective at dealing with vandalism (e.g. Huggle & User:ClueBot NG) to build tools that aid in identifying and supporting desirable newcomers (a task in which Wikipedia increasingly fails^[4]). Further, they recommend that the newcomers are given a voice, if indirectly via mentors, when it comes to how rules are created and applied.

Overall, the authors present a series of very compelling arguments, and the only complaint this reviewer has is that (even though three of the four were among the Wikimedia Foundation’s visiting researchers for the Summer of Research 2011) they do not discuss the fact that the Foundation and the wider community has recognized similar issues, and has engaged in debates, studies, pilot programs and such aimed to remedy the issue (see for example the WMF Editor Trends Study).

Literature reviews of Wikipedia’s inputs, processes, and outputs

Nicolas Jullien’s “What we know about Wikipedia. A review of the literature analyzing the project(s)”^[5] is an attempt at a “comprehensive” literature review of academic research on Wikipedia. Jullien works to distinguish his literature review from previous attempts like those of Okoli and collaborators (cf. earlier coverage: “A systematic review of the Wikipedia literature“) and of Park which tend to split the literature into three main themes: (1) motivations of editors to contribute and relationship between motivation and contribution quality, (2) editorial processes and organization and its relationship to quality and (3) the quality and reliability of production.

Jullien builds on this basic framework by Carillo and Okoli, but distinguishes his from their work in several ways. First, Jullien holds that previous work has focused too little on the outputs, which his analysis emphasizes more. Second and crucially, Jullien’s review is not limited to material published in journals and, as a result, is more representative of fields like computer science, HCI, and CSCW, which publish many of their most influential articles in conference proceedings. Jullien does not consider articles on how Wikipedia is used, questions of tools and their improvement, and studies that only use Wikipedia as a database (e.g., to test an algorithm). Other than this, the study is not limited to any particular field. It covers articles published in English, French and Spanish before December 2011, mostly based on searches in WebofScience and Scopus (sharing the search query used in the latter). The review is structured around inputs, processes, and outputs.

In terms of inputs, Jullien considers broad cultural factors in the broader environment and questions of why people choose to participate or join Wikipedia. In terms of process, he considers questions about the activities and roles of contributors, the social (e.g., network) structure of both the projects and the individuals who participants, the role of teams and organization of people within them, the processes around editing, creation, deletion, and promotion of articles with a particular focus on conflict, and questions of management and leadership. In terms of outputs, the paper divides publications into studies of process, Wikipedia user experience, the external evaluation of Wikipedia articles, and questions of Wikipedia coverage.

A second recent preprint by Taha Yasseri and János Kertész ^[6] likewise gives an overview of vast areas of recent research about Wikipedia. Subtitled “Sociophysical studies of Wikipedia” and citing 114 references, it compares some of the authors’ own results on e.g. editing patterns (covered in several past issues of this research report, e.g.: “Dynamics of edit wars“) with existing literature. The review focuses on quantitative data-driven analyses of Wikipedia production, reproduces and reports a series of previous analyses, and extends some of the earlier findings.

After a detailed description of how Wikipedia works, the authors walk through a series of types of quantitative analyses of patterns of editing to Wikipedia. They use “blocking” of edits to characterize good and “bad” editors and describe different editing patterns between these groups. The authors show that editors, in general, tend to edit in a “bursty” pattern with long periods of breaks and that editing tends to follow daily and weekly patterns that vary by culture. They also walk through several approaches for classifying edits by type, and discuss the characterization of linguistic features with an emphasis on readability.

Much of their article is focused on the issue of conflicts and edit warring. The authors pay particular attention both to the identification of conflicts and of controversial articles and topics, and to characterizing the nature of edit warring itself. The paper ends with the description of an agent-based model of edit warring and conflict.

WikiSym 2012: overview report

The International Symposium on Wikis and Open Collaboration -– “WikiSym 2012″ – was held August 27–29 in Linz, Austria. The three-day conference featured research papers, posters and demonstrations, and open space discussion sessions. About 80 researchers and wiki experts from around the world attended.

WikiSym is an academic conference, now in its eighth year, that seeks to highlight research on wikis and open collaboration systems. This year’s WikiSym had a strong focus on Wikipedia research, with studies that ranged from analyzing breaking news articles on Wikipedia to looking at the behavior of Wikipedia editors and how long they stay active. In all, 17 papers focused on Wikipedia or MediaWiki, and the two keynotes also focused on Wikipedia research.

The first keynote session was given by Jimmy Wales, who discussed challenges for Wikipedia and potential research questions that matter to the Wikimedia community [2][3]; Wales focused particularly on questions around diversity of the editing body, how to grow small language communities, and how to retain editors. The closing keynote was given by Brent Hecht, a researcher from Northwestern University, who spoke on techniques for making multilingual comparisons of content across Wikipedia versions, which in turn allows researchers to identify the potential cultural biases of various Wikipedia editions. Hecht found, for instance, that (looking at interwiki links across 25 languages) the majority of Wikipedia article topics only appear in 1 language; that the overlap between major language editions is relatively small; and that the depth of geographical representation varies widely by language, which a bias towards representing the country or place where that edition’s language is prominent. Hecht also compared articles on the same topic across Wikipedias to see the degree of similarity between them. Hecht described his work as “hyperlingual”, developing techniques to gain a broader perspective on Wikipedia by looking across language editions. His content comparison tool can be seen at the Omnipedia site, and the WikAPIdia API software he developed can be downloaded here. (See also earlier coverage about Omnipedia: “Navigating conceptual maps of Wikipedia language editions“)

In addition to the presented papers, some of which are profiled below, WikiSym has a strong tradition of hosting open space sessions in parallel with the main presentations, so that attendees can discuss topics of interest. This year’s open space topics included helping new wiki users; non-text content in wikis (including videos, images, annotations, slideshows and slidecasting); the future of WikiSym; Wikipedia bots; surveying Wikipedia editors; and realtime wiki synchronization and multilingual synchronization feedback. The conference closed with a panel session entitled “What Aren’t We Measuring?”, where panelists discussed and debated various methods for quantifying wiki-work (by studying editors, edits, and other metrics).

This year’s WikiSym was hosted at the Ars Electronica Center in Linz, a “museum of the future” that hosts the Ars Electronica festival every year. The colorful, dramatic Ars Electronica building is in the heart of Linz, so outside of sessions conference attendees enjoyed exploring and socializing in the city center. The conference dinner was held at the Pöstlingberg Schlössl, which is accessed by one of the steepest mountain trams in the world.

WikiSym 2012 papers and poster and demonstration abstracts may be downloaded from the conference website. Next year’s WikiSym is planned for Hong Kong, just before Wikimania 2013. Updates on the schedule and important dates can be found on the WikiSym blog.

On the “Ethnography Matters” blog, participant Heather Ford looked back at the conference,^[7] stating that “WikiSym is dominated by big data quantitative analyses of English Wikipedia”, asking “where does ethnography belong?” and counting 82% of the Wikipedia-related papers as examining the English Wikipedia and only 18% about other language Wikipedias. A panel at WikiSym 2011 had called to broaden research to other languages (see last year’s coverage: “Wiki research beyond the English Wikipedia at WikiSym“).

WikiSym 2012 papers

The conference papers and posters included, (apart from several ones that have been covered in earlier issues of this report):

• {{Citation needed}}: The dynamics of referencing in Wikipedia^[8]: This paper contributes to the debates on Wikipedia’s reliability. The authors find that density of references is correlated with the article length (the longer the article, the more references it will have per given amount of text). They also find that references attract more references (suggesting a form of a snowball mechanism at work) and that the majority of references is added in short periods of time by editors who are more experienced, and who are also adding substantial content. The authors thus conclude that referencing is primarily done by a small number of experienced editors, who prefer to work on longer articles, and who drastically raise the article’s quality, by both adding more content, and by adding more references.
• Etiquette in Wikipedia: Weening [sic] New Editors into Productive Ones^[9]: The authors of this paper experimented with alternative warning messages, introducing a set of shorter and more personalized warnings into those delivered by Huggle in the period of November 8 0 December 9 2011. Unfortunately, the authors are rather unclear on how exactly the Huggle tool was influenced, and whether the community was consulted on that. While in fact the community and Huggle developers have been aware of, discussed and approved of this experiment – here or here – the paper’s omission to clarify that this was the case can lead to some confusion with regard to research ethics, since a casual reader may assume the researchers have hijacked Huggle without consulting the community. The wording changes were in good faith (making the messages more personalized, friendly and short), and the authors conclude that the new messages they tested proved more conductive to positively influenced new editors who received Level 1 Warnings.
• WikiTrust algorithm applied to MediaWiki programmers: A paper titled “Towards Content-driven Reputation for Collaborative Code Repositories”^[10] reports on an experimental application of the well-known WikiTrust algorithm to the collaboration of programmers on a code repository, namely MediaWiki‘s own SVN codebase (from 2011, before it was switched to Git). In that model, contributors lose reputation when their contributions are reverted or deleted. According to the abstract, “Analysis is particularly attentive to reputation loss events and attempts to establish ground truth using commit comments and bug tracking. A proof-of-concept evaluation suggests the technique is promising (about two-thirds of reputation loss is justified) with false positives identifying areas for future refinement.” An example of such false positives is “The “not now” trap: Frequently a change is reverted with a ‘not now’ justification, e.g., needing to hold for more testing. When that testing is done the changes are likely to be re-committed in much the same form, punishing the benign reverting editor.”

• “Deletion Discussions in Wikipedia: Decision Factors and Outcomes”^[11] found among other things that “69.5% of discussions and 91% of comments are well-represented by just four factors: Notability, Sources, Maintenance and Bias. The best way to avoid deletion is for readers to understand these criteria.” One of the authors also co-presented a demo showing mock-ups of possible “alternative interfaces for deletion discussions in Wikipedia”^[12], which would highlight the prevalence of each type of argument (e.g. notability, sourcing…) in a deletion discussion more clearly.

• “Classifying Wikipedia articles using network motif counts and ratios”^[13]: Similar to an earlier paper by the same authors (earlier coverage: “Collaboration pattern analysis: Editor experience more important than ‘many eyes’“), this paper examined the collaboration network of Wikipedia articles and editors using Network motifs – small graphs which occur particularly frequently as sub-graphs of networks of a certain kind, and can be regarded as its building blocks in some sense. This was then related to the quality ratings of articles: “Pages with good quality scores [e.g. featured articles] have characteristic motif profiles, but pages with good user ratings [from the [[mw:Article feedback|Article Feedback tool] don’t. This suggests that a good quality score is evidence that a collaborative curation process has been pursued. However, not all pages with high quality scores get good user ratings and some pages with low quality scores are trusted by users. Perhaps the Wikipedia quality scale is a low error scale rather than a quality scale?”

• “‘Writing up rather than writing down’: becoming Wikipedia Literate”^[14] applied “the work of literacy practitioner and theorist Richard Darville” to communication among Wikipedians, e.g. new users and experienced users who deleted some of their contributions. “Using a series of examples drawn from interviews with new editors and qualitative studies of controversies in Wikipedia, we identify and outline several different literacy asymmetries.”

• “How long do wikipedia editors keep active?”^[15] found that on the English Wikipedia, “although the survival function of occasional editors roughly follows a lognormal distribution, the survival function of customary editors can be better described by a Weibull distribution (with the median lifetime of about 53 days). Furthermore, for customary editors, there are two critical phases (0–2 weeks and 8–20 weeks) when the hazard rate of becoming inactive increases”.

“First Monday” on rhetoric, readability and teaching

First Monday, the veteran open access journal about Internet topics, featured three Wikipedia-themed papers in its September issue:

• AfD rhetoric examined: “The pentad of cruft: A taxonomy of rhetoric used by Wikipedia editors based on the dramatism of Kenneth Burke”^[16] is an essay “describing a method for classifying arguments made by Wikipedia editors based on the theory of ‘dramatism,’ developed by the literary theorist Kenneth Burke, and demonstrating how this method can be applied to a small sample of arguments drawn from Wikipedia’s ‘Article for Deletion’ (AfD) process.”

• “Readability of Wikipedia”^[17] applied the standard Flesch Reading Ease test to the English and Simple English Wikipedias (at http://www.readabilityofwikipedia.com/ , the authors also offer the possibility to view scores directly). The effort, described as “extensive research” in an university press release found that “overall readability is poor, with 75 percent of all articles scoring below the desired readability score. The ‘Simple English’ Wikipedia scores better, but its readability is still insufficient for its target audience.” See also the detailed earlier Signpost coverage: “Readability of Simple English and English Wikipedias called into question“, and the summary of an earlier paper which applied a more diverse set of readability measures to both Wikipedias: “Simple English Wikipedia is only partially simpler/controversy reduces complexity“

• “Wikis and Wikipedia as a teaching tool: Five years later”^[18] by longtime Wikipedian (and contributor to this research newletter) Piotr Konieczny first gives an overview over the now widespread use of Wikipedia in the classroom and its advantages, and in a second part offers detailed practical advice drawing from the author’s own “five years of experience in teaching with wikis and Wikipedia and holding workshops on the subject”.

Briefly

• Recent changes visualization designed to assist admins: A paper titled “Feeling the Pulse of a Wiki: Visualization of Recent Changes in Wikipedia”^[19] will be presented at the upcoming conference “VINCI 2012 : The International Symposium on Visual Information Communication and Interaction”. It describes a prototype software (apparently not publicly available yet) that is designed “to aid a wiki administrator to perceive current activity in a wiki”, starting out from the idea to map editors and articles in two dimensions: time and activity level. Hosted on the Toolserver, the software directly accesses a wiki’s Recent Changes table, containing edits from the last 30 days. Using their tool, the authors visually discerned “six common editing patterns” on the English Wikipedia. E.g. “New article, many editors, many edits: this is the new popular article pattern which almost invariably reflects a current event”. The authors also compare their tool to the previous “few and limited efforts” to visualize recent changes: WikipediaVision, Wikipulse and Wikistream.
• Unearthing the “actual” revision history of a Wikipedia article: A paper^[20] by two researchers from Waseda University observes that “Unlike what is very common in software development, Wikipedia does not maintain an explicit revision control system that manages the detailed change through revisions. The chronologically-organized edit history fails to reveal the meaningful scenarios in the actual evolution process of Wiki articles, including reverts, merges, vandalism and edit wars”. To extract this “actual” revision graph, where two neighboring nodes correspond to a revision and an earlier one which it was derived from, a similarity measure is needed. The article cites a 2007 paper and other research which had already proposed to understand a page’s revision history as a directed tree and used similarity measures such as tf-idf. The present paper uses a similarity measure based on the frequency of n-grams (sequences of n words) and goes further in regarding the revision history as a directed acyclic graph. This allows for version merges, although the actual algorithm presented still focuses on the case of trees.
• Who deletes Wikipedia – or reverts it: Wibidata, a big data analytics startup based in San Francisco, posted a follow-up^[21] to their “Who deletes Wikipedia” analysis (previous coverage), taking into account the effect of reverts, which several Wikipedians had pointed out in response to their earlier blog post.
• Geospatial characteristics of Wikipedia articles: The authors of this paper attempt to identify what makes Wikipedia articles with geographical coordinates different from others (besides their obvious relation to geographical locations).^[22] They rather unsurprisingly find that more developed articles are more likely to have geo-coordinates, and consequently they find that there seems to be a correlation between article quality and having geo-coordinates links. They also find that articles with geo-coordinates are more likely to be linked to, a likely function of them being of above-average quality.
• Wikipedia’s affordances: This paper, framing itself as part of the ecological psychology field, contribute to the discourse about affordances (property of an object that allows one to take a certain action).^[23] The authors submit that this concept can be developed to further our understanding of how individuals perceive their socio-technical environment. The authors refine the term “technology affordances”, which they define as “functional and relational properties of the user-technology system”. Then use Wikipedia as their case study attempting to demonstrate its value, listing six affordances of Wikipedia (or in other words, they note that editors of Wikipedia can take the following six actions): contribution, control, management, collaboration, self-presentation, broadcasting.
• Hematologists unsure whether “to engage with Wikipedia more constructively”: A letter^[24] to the medical journal BMJ asks “Should clinicians edit Wikipedia to engage a wider world web?” The authors, a student and a senior lecturer in the field of haematology, “simulated 30 opportunistic internet searches for information on haemophilia in the top three search engines using term permutations: haemophilia or hemophilia (with or without A or B); carrier; information; child; treatment. Wikipedia was the most commonly found top 10 site in all search engines.” In an apparent attempt to gauge the authoritativeness of Wikipedia content, “Analysis of editorial authorship of the Haemophilia Wiki [sic] for four weeks found 39 edits by 25 editors, only nine of whom had a profile, and none of whom were experts in haemophilia.” Possibly unaware of Wikipedia’s “no original research” policy, the authors ask “Given the evolving debate about open access to data, should publishers and authors be mandated to place reviews and key studies [...] in a public domain like Wikipedia?” (naming the example of a recent prominent paper in the field, which the Wikipedia article cites only in form of a New York Times news article about it). The letter concludes “as a professional group, we are not sure whether we wish to engage with Wikipedia more constructively”. One-day access to the letter, which is around half a page long, can be purchased at £20/$30/€32 plus VAT, which may not be a very competitive price given the availability of more thorough evaluations of Wikipedia’s quality elsewhere in the academic literature.
• Tracking and verifying sources on Wikipedia: Ethnographer Heather Ford published the final report from her study on how editors track and verify sources on Wikipedia.^[25] The report presents an in-depth qualitative analysis of editor discussions around verifiability of information in the early editing phase of the 2011–2012 Egyptian revolution article and reviews how Wikipedia policies around primary vs secondary sources, notability and neutrality were used to make decisions about what sources to cite.
• A recommender system for infoboxes: A team of computer science researchers at the University of Texas at Arlington developed a classification method to predict infobox template types from articles lacking them, using three types of features: words in articles, categories, and named entities (or words with corresponding Wikipedia entries). The study suggests that articles with infoboxes and articles without infoboxes exhibit a substantially different distributions of the above features. The classifier was tested on data from a 2008 dump of the English Wikipedia.^[26]
• Styles of information search on Wikipedia: A poster presented at the 2nd European Workshop on Human-Computer Interaction and Information Retrieval presents the results of an eye-tracking study looking at patterns of information search in Wikipedia articles. The study looks at task-specific differences in the context of factual information lookup, learning and casual reading activity.^[27]
• Post-edit feedback experiment: The Wikimedia Foundation’s “Editor Engagement Experiments” team reported^[28] on an experiment with a simple user interface change – adding messages that confirm that an edit has been saved – and its effect on the contributions of new editors.
• Pilot study about Wikipedia’s quality compared to other encyclopedias: The results of a pilot study commissioned by the Wikimedia Foundation, titled “Assessing the Accuracy and Quality of Wikipedia Entries Compared to Popular Online Alternative Encyclopaedias: A Preliminary Comparative Study Across Disciplines in English, Spanish and Arabic”^[29] have been announced.
• Wikipedia, the first step toward communism: Sylvain Firer-Blaess and Christian Fuchs, in their “info-communist manifesto”, argue that Wikipedia is an example of the communist mode of production and participatory democracy—”the brightest info-communist star on the Internet’s class struggle firmament”. They suggest that Wikipedia’s future will be a choice between co-option into the broader capitalist economy (through the exploitation of the commercial possibilities of Wikipedia’s free licensing) or, alongside similar “info-communist” projects, displacing more and more capitalist production of informational goods.^[30]
• Quality flaw detection competition: Maintenance templates on the English Wikipedia (e.g. “citation needed”) have attracted the attention of several researchers recently, as easy to parse indicators of quality problems (example). An “Overview of the 1st International Competition on Quality Flaw Prediction in Wikipedia”^[31] summarizes its outcome as follows: “three quality flaw classifiers have been developed, which employ a total of 105 features to quantify the ten most important quality flaws in the English Wikipedia. Two classifiers achieve promising performance for particular flaws. An important ‘by-product’ of the competition is the first corpus of flawed Wikipedia articles, the PAN Wikipedia quality flaw corpus 2012 (PAN-WQF-12)”, which consists of “1 592 226 English Wikipedia articles, of which 208 228 have been tagged to contain one of ten important quality flaws”. One of the two “winners”, the “FlawFinder” algorithm, has been described in a paper covered last month. The competition took place on occasion of the CLEF 2012 conference, as did the first Wikipedia Vandalism Detection competition two years ago (Signpost coverage).

References

Vol: 2 • Issue: 8 • August 2012 [archives]

New influence graph visualizations; NPOV and history; ‘low-hanging fruit’

With contributions by: Piotrus, Ragesoss, Evan, DarTar, Tbayer and OrenBochman

Wikipedia-based graphs visualize influences between thinkers, writers and musicians

A visualization of musical genres related to psychedelic music, based on DBPedia data.

In a blog post titled “Graphing the history of philosophy”,^[1] Simon Raper of the company MindShare UK describes how he constructed an influence graph of all philosophers using the “Influenced by” and “Influenced” fields of Template:Infobox philosopher (example: Plato). This information was retrieved using DBpedia with a simple SPARQL query. After some cleanup, the result, consisting of triplets in the form <Philosopher A, Philosopher B, Weight> was processed using the open source graph visualization package Gephi to create an impressive overview of the philosophers within their respective spheres of influence.

Brendan Griffen extended the idea to “everyone on Wikipedia. Well, everyone with an infobox containing ‘influences’ and/or ‘influenced by’”, arriving at a huge, far more dense “Graph Of Ideas” including not only philosophers, but also novelists, fantasy and science fiction writers, and comedians.^[2] In another blog post,^[3] Griffen added transitive links as well – so that each person is considered to be influenced both directly and indirectly. The most connected people in the graph were ancient Greek thinkers, with Thales, Pythagoras and Zeno of Elea occupying the top three spots. Griffen remarks that this vindicates a statement in Bertrand Russell‘s History of Western Philosophy (1945): “Western Philosophy begins With Thales”.

Also inspired by Raper’s posting, Tony Hirst posted a number of visualizations of the Wikipedia link and category structure (likewise using DBpedia and Gephi, queried via the Semantic Web Import plugin) to visualize related entries and influence graphs in the English Wikipedia. The blog posts (all of which include detailed step-by-step tutorials) examine the related graph of philosophers,^[4] and also visualize an influence graph of programming languages^[5] and one of musical genres related to psychedelic music.^[6] All these visualizations and blog posts by Hirst are released under a Creative Commons Attribution license.

Hirst also mentioned a related tool called “WikiMaps”, the subject of a recent article in the International Journal of Organisational Design and Engineering.^[7] As described in a press release, the tool provides a “map of what is “important” on Wikipedia and the connections between different entries. The tool, which is currently in the “alpha” phase of development, displays classic musicians, bands, people born in the 1980s, and selected celebrities, including Lady Gaga, Barack Obama, and Justin Bieber. A slider control, or play button, lets you move through time to see how a particular topic or group has evolved over the last 3 or 4 years.” A demo version is available online.

See also the recent coverage of a similar visualization, based on wikilinks instead of infoboxes: “The history of art mapped using Wikipedia“

Information retrieval scientists turn their attention to Wikipedia’s page view logs

Found to be connected to the “#euro2012″ hashtag by analyzing Wikipedia pageviews: Euro 2012 football championship

The Time-aware Information Access workshop at this year’s SIGIR (Special Interest Group on Information Retrieval) conference brought a wave of attention to Wikipedia’s public page-view logs. Detailing the number of page views per hour for every Wikipedia project, these files figure prominently in a variety of open-source intelligence applications presented at the workshop.

A group of researchers from ISLA, University of Amsterdam created an API providing access to this data and performing simple analysis tasks.^[8] Though the site appears to be down at the time of writing, the API supports the retrieving a particular article’s page-view time series as well as searching for other wikipedia articles based on the similarity of their time series. In addition to machine-readable JSON results, the API will supply simple plots in png format. While the idea of providing page specific time series is not new, support for finding other pages with similar viewing patterns highlights a fascinating new use for Wikipedia page views.

Two other papers are combining Wikipedia page-view information with external time-series data sets. On the intuition that Wikipedia page views should have a strong correlation with real-world events, researchers from the University of Glasgow and Microsoft built a system to detect which hashtags frequently queried on Bing Social Search were event-related.^[9] For example, the hashtag #thingsthatannoyme doesn’t clearly correspond to an event, whereas a hashtag like “#euro2012″ is about the UEFA European Football Championship. After tokenizing the hashtags into a list of words, the researchers queried Wikipedia for those terms and correlated the time series of hashtag search popularity with the page-view time series for the articles which are returned. This correlation score can be used to indicate which hashtags are likely to be about events, a useful feature for web searches and any other temporally aware zeitgeist application.

In a similar vein, researchers from the University of Edinburgh and University of Glasgow used the Wikipedia page-view stream to tackle the problem known as first-story detection (FSD), which aims to automatically pick out the first publication relating to a new topic of interest.^[10] While traditional techniques primarily focus on newswire or Twitter, the authors used a combination of Twitter and Wikipedia page views to construct an improved FSD system. To improve on state-of-the-art Twitter-only FSD systems, the authors aimed to filter out false positives by checking that the Twitter-based first stories corresponded to a Wikipedia page that was also experiencing heightened traffic during the same period.

Using a simple outlier detection method, the authors created a set of Wikipedia pages with unexpectedly high page views for each hour. Each Twitter-based first story (tweet) was then matched against the corresponding collection of Wikipedia outliers, employing an undisclosed metric of textual similarity that uses only the Wikipedia page titles. If the tweet failed to match any spiking Wikipedia page, it was down-weighted as a first story candidate. The authors showed that this combined approach improves FSD precision in comparison to a twitter-only baseline for all but the most popular twitter-based stories. Though this research makes advances on the difficult task of first-story detection, perhaps the most immediately useful finding is that Wikipedia page views appear to lag behind twitter activity by roughly two hours. In general, we can expect to see an increasing amount of joint models over various open-source intelligence streams as we learn exactly what each stream is useful for and the relationships between the streams.

See also the Signpost coverage of a small study of the highest hourly page views on the English Wikipedia during January-July 2010, and their likely causes: “Page view spikes“

The limits of amateur NPOV history

In “The inclusivity of Wikipedia and the drawing of expert boundaries: An examination of talk pages and reference lists”^[11], information studies professor Brendan Luyt of Nanyang Technological University looks at History of the Philippines, a B-class article that had featured article status from October 2006 until it was delisted at the conclusion of its featured article review in January 2011.

Luyt argues that talk-page discussions, the types of sources cited, and the organization of the article itself, all point to a very traditional view of what constitutes history: in short, great man history concerned mainly with political and military events, and the actions of elites. This style of history does not capture the breadth of approaches used by professional historians, so does not live up to the ideal of NPOV in which all significant viewpoints published in reliable sources are represented fairly and proportionately. In practice, Luyt shows, editors (lacking sufficient knowledge of the relevant professional historical literature) end up using arguments over bias and NPOV to construct a limited and conservative historical narrative—for this article at the least, although a similar pattern could be found for many broad historical topics.

The sources cited are primarily what Luyt calls “textbookese” summaries, easily available online, which focus on bare facts without the historical debates that surround them. Between the valid sources and experts recognized by Wikipedia editors and the good-faith use of the NPOV principle to limit other viewpoints, Luyt concludes that—rather than being more inclusive of diverse views and sources than the typical “expert” community—Wikipedia in practice recognizes a considerably narrower set of viewpoints.

Three new papers about Wikipedia class assignments

An article titled “Assigning Students to edit Wikipedia: Four Case Studies”^[12] presents the experiences of four professors who participated in the Wikipedia Education Program, in a total of six courses total (two of four instructors taught two classes each). The lessons from the assignments included: 1) the importance of strict deadlines, even for graduate classes; 2) having a dedicated class for acquiring skills in editing and for understanding Wikipedia policies, or spreading this over segments of several classes; 3) the benefits of having students interact with the campus ambassadors and the wider Wikipedia community.

Overall, the instructors saw that compared with their engagement in traditional assignments, students were more highly motivated, produced work of higher quality, and learned more skills (primarily, related to using Wikipedia, such as being able to better judge its reliability). Wikipedia itself benefited from several dozen created or improved articles, a number of which were featured as DYKs. The paper presents a useful addition to the emerging literature on teaching with Wikipedia, as one of the first serious and detailed discussions of specific cases of this new educational approach.

“Integrating Wikipedia Projects into IT Courses: Does Wikipedia Improve Learning Outcomes?”^[13] is another paper that discusses the experiences of instructors and students involved in the recent Wikipedia:Global Education Program. Like most existing research in this area, the paper is roughly positive in its description of this new educational approach, stressing the importance of deadlines, small introductory assignments familiarizing students with Wikipedia early in the course, and the importance of close interactions with the community. A poorly justified (or explained) deletion or removal of content can be quite a stressful experience to students (and the newbie editors are unlikely to realize that an explanation may be left in an edit summary or page-deletion log). A valuable suggestion in the paper was that instructors (professors) make edits themselves, so they would be able to discuss editing Wikipedia with students with first-hand experience instead of directing students to ambassadors and how-to manuals; and to dedicate some class time to discussing Wikipedia, the assignment, and collective editing.

A four-page letter^[14] in the Journal of Biological Rhythms by a team of 48 authors reported on a a similar undergraduate class project in early 2011, where 46 students edited 15 Wikipedia articles in the field of chronobiology, aiming at good article status. After their first edits, they were systematically given feedback by one “Wikipedia editor and 6 experts in chronobiology” before continuing their edits (in the paper’s acknowledgements the authors also thank “innumerable Wikipedia editors who critiqued student edits”). Because of the high visibility of the results – most of the articles were ranked top in Google results – students found the experience rewarding. Topics were selected collaboratively by the class, and because students came up with a relatively small number of suggestions, one concern was that the project might, if repeated, run out of article topics in the given subject area.

A literature review presented at July’s Worldcomp’12 conference in Las Vegas about “Wikipedia: How Instructors Can Use This Technology As A Tool In The Classroom”^[15] also recommended to have students actively edit Wikipedia (as well as practicing to read it critically), and concluded that “it is time to embrace Wikipedia as an important information provider and one of the innovative learning tools in the educators’ toolbox.”

Substantive and non-substantive contributors show different motivation and expertise

“Investigating the determinants of contribution value in Wikipedia”^[16] reports the results of a survey of Wikipedians who were asked their opinion about the “contribution value” of their edits (measured by agreement to statements such as “your contribution to Wikipedia is useful to others”), which was then related to various characteristics.

The researchers used Google to obtain a list of 1976 Wikipedia users’ email addresses (using keywords such as “gmail.com” or “hotmail.com”). They sent invitation emails that provided the URL to the online questionnaire. In six weeks, 234 editors completed all the questions. Of these, 205 – Nine females and 196 males – supplied a valid user name and were considered in the rest of the analysis (anonymous editors were removed).

A content analysis was performed of 50 randomly selected edits by each respondent (or all, if the user had fewer than 50 edits), classifying them as “substantive” changes (e.g. “add links, images, or delete inaccurate content”) and “non-substantive changes” (e.g. “reorganizing existing content [or] correcting grammatical mistakes and formatting texts to improve the presentation”), corresponding to “two [proposed] new contributor types in Wikipedia to discriminate their editing patterns.”

An attempt was made to relate this to the “contribution value” the respondents assigned to their own edits, and to their responses in two other areas:

• “interests” (measured by respondent ratings of a variety of different motivations to contribute to Wikipedia on how well each applied to themselves, e.g. “Enhancing your learning abilities, skills and expertise”); and
• “resources” (meaning expertise based on education, profession and hobbies, measured by respondent ratings of their expertise in a variety of fields within eachc, e.g. “Hospitality and tourism”).

The “breadth” of interests and resources was defined as the number of ratings above a certain threshold in each, and the “depth” as the highest rating assigned in each.

In an “important consideration for practitioners”, the authors wrote that:

“[T]o produce valuable contributions, users with high depth of interests and resources should be encouraged to concentrate their efforts on substantive changes. Meanwhile, for users with high breadth of interests and resources, wiki practitioners should advise them to pay more attention to nonsubstantive changes. The findings imply that practitioners can try to identify two distinct types of users. To achieve this objective, they may develop certain algorithms in wikis to automatically detect the frequencies of substantive/non-substantive changes of users. … For example, notification messages about wiki articles that need substantive changes can be sent to users who have high levels of depth of interests and resources. Similarly, well-prepared messages about articles that need non-substantive changes can be delivered to users who have high levels of breadth of interests and resources.”

Is there systemic bias in Wikipedia’s coverage of the Tiananmen protests?

Remembrance of the 20th anniversary of the June 4 events in Hong Kong (replica of the “goddess of democracy” statue)

Remembrance in the West (replica of the same statue at the University of British Columbia, Canada)

Wikipedia: Remembering in the digital age^[17] is a masters dissertation by Simin Michelle Chen, examining collective memories as represented on the English Wikipedia; she looked at how significant events are portrayed (remembered) on the project, focusing on the Tiananmen Square Protests of 1989. She compared how this event was framed by the articles by New York Times and Xinhua News Agency, and in Wikipedia, where she focused on the content analysis of w:Talk:Tiananmen Square protests of 1989 and its archives.

Chen found that the way Wikipedia frames the event is much closer to that of The New York Times than the sources preferred by the Chinese government, which, she notes, were “not given an equal voice” (p. 152). This English Wikipedia article, she says, is of major importance to China, but is not easily influenced by Chinese people, due to language barriers, and discrimination against Chinese sources that are perceived by the English Wikipedia as unreliable – that is, more subject to censorship and other forms of government manipulation than Western sources. She notes that this leads to on-wiki conflicts between contributors with different points of views (she refers to them as “memories” through her work), and usually the contributors who support that Chinese government POV are “silenced” (p. 152). This leads her to conclude that different memories (POVs) are weighted differently on Wikipedia. While this finding is not revolutionary, her case study up to this point is a valuable contribution to the discussion of Wikipedia biases.

While Chen makes interesting points about the existence of different national biases, which impact editors very frames of reference, and different treatment of various sources, her subsequent critique of Wikipedia’s NPOV policy is likely to raise some eyebrows (pp. 48–50). She argues that NPOV is flawed because “it is based on the assumption that facts are irrefutable” (p. 154), but that those facts are based on different memories and cultural viewpoints, and thus should be treated equally, instead of some (Western) being given preference. Subsequently, she concludes that Wikipedia contributes to “the broader structures of dominance and Western hegemony in the production of knowledge” (p. 161).

While she acknowledges that official Chinese sources may be biased and censored, she does not discuss this in much detail, and instead seems to argue that the biases affecting those sources are comparable to the those affecting Western sources. In other words, she is saying that while some claim Chinese sources are biased, other claim that Western sources are biased, and because the English Wikipedia is dominated by the Western editors, their bias triumphs – whereas ideally, all sources should be acknowledged, to reduce the bias. The suggestion is that Wikipedia should reject NPOV and accept sources currently deemed as unreliable. Her argument about the English Wikipedia having a Western bias is not controversial, was discussed by the community before (although Chen does not seem to be aware of it, and does not use the term “systemic bias” in her thesis) and reducing this bias (by improving our coverage of non-Western topics) is even a goal of the Wikimedia Foundation. However, while she does not say so directly, it appears to this reviewer that her argument is: “if there are no reliable non-Western sources, we should use the unreliable ones, as this is the only way to reduce the Western bias affecting non-Western topics”. Her ending comment that Wikipedia fails to leave to its potential and to deliver “postmodern approach to truth” brings to mind the community discussions about verifiability not truth (the existence of this debates she briefly acknowledges on p. 48).

Overall, Chen’s discussion of biases affecting Wikipedia in general, and of Tiananmen Square Protests in particular, is useful. The thesis however suffers from two major flaws. First, the discussion of Wikipedia’s policies such as reliable sources and verifiability (not truth …) seems too short, considering that their critique forms a major part of her conclusions. Second, the argumentation and accompanying value-judgements that Wikipedia should stop discriminating against certain memories (POVs) is not convincing, lacking a proper explanation of the reasons why the Wikipedia community made those decisions favoring verifiability and reliable sources over inclusion of all viewpoints. Chen argues that Wikipedia sacrifices freedom and discriminates against some memories (contributors), which she seems to see as more of a problem that if Wikipedia was to accept unreliable sources and unverifiable claims.

“Low-hanging fruit hypothesis” explains Wikipedia’s slowed growth?

A student paper titled “Wikipedia: nowhere to grow”^[18] from a Stanford class about “Mining Massive Data Sets” argues for the “low-hanging fruit hypothesis” as one factor explaining the well-known observation that “since 2007, the growth of English Wikipedia has slowed, with fewer new editors joining, and fewer new articles created”. The hypothesis is described as follows: “the larger [Wikipedia] becomes, and the more knowledge it contains, the more difficult it becomes for editors to make novel, lasting contributions. That is, all of the easy articles have already been created, leaving only more difficult topics to write about”. The authors break this hypothesis into three smaller ones that are easier to test – that (1) there has been a slowing in edits across many languages with diverse characteristics; (2) older articles are more popular to edit; and (3) older articles are more popular to read. They find a support for all three of the smaller hypotheses, which they argue supports their main low-hanging fruit hypothesis.

While the overall study seems well-designed, the extrapolation from the three subhypotheses to the parent hypothesis seems problematic. The authors do not provide a proper operationalization of terms such as “novel”, “lasting”, and “easy/difficult”, making it difficult to enter into a discourse without risking miscommunication. There may be at least four main issues in the work:

• (a minor but annoying issue): hypothesis II is incorrectly and confusingly worded in the section dedicated to it: “Older articles (those created earlier) will be more popular to read than more newly created articles”; however, their study of hypothesis II is based on the number of edits to the article, not the number of page views (those are analyzed in the subsequent hypothesis III);
• regarding the claim “all of the easy articles have already been created, leaving only more difficult topics to write about”, it is true that the majority of vital/core articles are developed beyond stub, and their subsequent expansion is more difficult (it takes more and more effort to move the article up through assessment classes). However, while the older articles are more popular, they are not necessarily easier to edit, as w:Wikipedia:The Core Contest illustrates. While almost everyone may be able to quickly define (stub) Albert Einstein, it is questionable whether 1) developing this article is easier than developing an article on a less well-known subject, where fewer sources mean the editors need to do less research, and 2) while mostly everyone knows who Einstein was, everyone also has knowledge of at least some less popular subjects. As Wikipedia:Missing articles illustrate, there are still many articles in need of creation, and for a fan/expert, it may be easier to create an article on an esoteric subject than to edit the article on Einstein.
• The claim that “[it is more difficult] for editors to make novel, lasting contributions” is difficult to analyze due to the lack of operationalization of those terms by the authors, but 1) regarding novel, if it means new, see the Missing articles argument above – there is still plenty to write about; and 2) regarding lasting – the authors do not cite any sources suggesting the deletionism in English Wikipedia may be on the rise.

Overall, the paper presents four hypotheses, three of which seem to be well supported by data, and contribute to our understanding of Wikipedia, but their main claim seems rather controversial and poorly supported by their data and argumentation.

See also the coverage of a related paper in a precursor of this research report last year: “IEEE magazine summarizes research on sustainability and low-hanging fruit“

Briefly

• Barnstars at ASA annual conference: Two Wikipedia papers were presented at the 2012 annual meeting of the American Sociological Association last week, both focusing on “barnstar” awards on Wikipedia.
  Michael Restivo and Arnout van de Rijt presented their research on the effect of barnstars, titled “Experimental Study of Informal Rewards in Peer Production”, which had found that assigning “editing awards or ‘barnstars’ to a subset of the 1% most productive Wikipedia contributors … increases productivity by 60% and makes contributors six times more likely to receive additional barnstars from other community members”, as stated in the abstract. See the review in the April issue of this report: “Recognition may sustain user participation“.
  Benjamin Mako Hill, Aaron Shaw, and Yochai Benkler presented “Status, Social Signaling, and Collective Action: A Field Study of Awards on Wikipedia”, with a more skeptical look at the effect of barnstars. According to the abstract, “Willer has argued for a sociological mechanism for the provision of public goods through selective incentives. Willer posits a “virtuous circle” in which contributors are rewarded with status by other group members and in response are motivated to contribute more. [... But] there is reason to suspect that not all individuals will be equally susceptible to status-based awards or incentives. At the very least, Willer’s theory fails to take into account individual differences in the desire to signal contributions to a public good. We test whether this omission is justified and whether individuals who do not signal status in the context of collective action behave differently from those who do in the presence of a reputation-based award. [Analyzing barnstars on Wikipedia,] we show that the social signalers see a boost in their editing behavior where non-signalers do not.”
• How high school, college and PhD students evaluate Wikipedia quality: “Trust in online information A comparison among high school students, college students and PhD students with regard to trust in Wikipedia”^[19] is a master thesis that looks at how these three groups judge the trustworthiness of Wikipedia articles, based on the “3S-model” model by the advisors of the thesis (Lucassen and Schraagen (2011), Factual Accuracy and Trust in Information: The Role of Expertise. Journal of the American Society for Information Science & Technology, 62, 1232–1242). Unsurprisingly, the more educated the group is, the more detailed their analysis will be. High school students usually focus on accuracy, completeness, images, length, and writing style. College and PhD students go beyond those five elements, although looking at authority, objectivity, and structure. Interestingly, the differences between college and PhD students were much smaller than those between high school students and the other two groups. Another important finding of the study was that the less educated the group, the less likely they are to be aware of Wikipedia being open source and open to editing by anyone. Further, high school students seem to have much more difficulty in distinguishing between a high and low quality article, and overall, seem much more likely to simply not question the trustworthiness of the sources given.
• Doctors widely use Wikipedia as a reference: A literature review of 50 articles about the use of social media by clinicians^[20] found that “Wikipedia is widely used as a reference tool” among them, despite concerns about its accuracy. The authors remark that “we found multiple projects that sought to emulate Wikipedia’s success in crowd-sourcing useful medical content, while additionally emphasizing editorial credibility by verifying credentials of contributors. These include RadiologyWiki, announced in 2007 and currently dormant, and Medpedia, which launched in 2009 with substantial institutional backing. We did not find articles reporting success metrics for these projects or similar ones.”
• Predicting quality flaws in Wikipedia articles: A notebook paper to presented at the annual PAN workshop at the Conference and Labs of the Evaluation Forum meeting (CLEF ’12) introduces FlawFinder, a toolset to predict quality flaws in Wikipedia articles.^[21] The paper is one of the winning entries in a Competition on Quality Flaw Prediction in Wikipedia. The paper defines 11 types of quality flaws, spanning low-level issues (such as orphaned or unreferenced articles) and high-level quality flaws (such as notability or original research). It uses a corpus of articles tagged with cleanup templates (154,116 articles from a January 2012 dump of the English Wikipedia) as a training set to predict whether articles in a separate, uncategorized set suffer from the same flaws. The model uses a variety of features of the training set based on revision data, lexical properties, structural properties of the article and the reference section, network properties of the link graph. The results suggest, among other things, that the strongest non-lexical features for the advert flaw are links pointing to external resources, while the number of discussions on article’s talk page is the strongest feature to predict original research.
• Quality of text and quality of editors. A poster presented at the 2012 ACM Conference on Hypertext and Social Media (HT 2012) describes a method to measure the quality of Wikipedia articles by combining text survival metrics and the quality of editors editing these articles, where editor quality is calculated recursively as a function of the quality of their contributions. The method claims to be “resistant to vandalism”, however no empirical validation is presented in the poster.^[22]
• WikiSym 2012: WikiSym, the annual conference “dedicated to wiki and open collaboration research and practice” was happening in Linz, Austria as this issue of the research report went to press. Links to online versions of all conference papers have been posted in the program; expect fuller coverage in the September issue.

References