Vol: 2 • Issue: 1 • January 2012 [archives]

Language analyses examine power structure and political slant; Wikipedia compared to commercial databases

With contributions by: Tbayer and Piotrus

Admins influence the language of non-admins

An Arxiv preprint titled “Echoes of power: Language effects and power differences in social interaction”^[1] looks at the language used by Wikipedia editors. The authors look at how conversational language can be used to understand power relationships. The research analyzes how much one adapts their language to the language of others involved in a discussion (the process of language coordination). The findings indicate that the more such adoption occurs, the more deferential one is. The authors find that editors on Wikipedia tend to coordinate (language-wise) more with the administrators than with non-administrators. Furthermore, the study suggests that one’s ability to coordinate language has an impact on one’s chances to become an administrator: the admin-candidates who do more language coordination have a higher chance of becoming an administrator than those who don’t change their language. Once a person is elected an administrator, they tend to coordinate less.

A blog post on the website of Technology Review summarized the results using the headline “Algorithm Measures Human Pecking Order” and highlighted the fact that one of the authors is Jon Kleinberg, known as inventor of the HITS algorithm (also known as “hubs and authorities”).

Can Wikipedia replace commercial biography databases?

California State University, East Bay: Could it rely on biographical information from Wikipedia and the web alone?

An article^[2] by a librarian and professor at California State University offers a comparison of “biographical content for literary authors writing in English” between Wikipedia, “the web” (i.e. top Google search results) and two commercial databases: the Biography Reference Bank (BRB, now part of EBSCO Industries) and Contemporary Authors Online, motivated by the decision of the author’s institution to cancel its subscription to the latter database (CAO) during a budget crisis in 2008-2009, which among other reasons had been accompanied by “a comment that this information is ‘on the web’”.

The paper starts out with a literature review on the reliability of Wikipedia and then describes how the author compiled a list of 500 authors (mostly from the US and UK) by “examining curricula and textbooks from English literature courses across the USA” and soliciting additional suggestions from peers. These names were then searched on BRB, CAO (as part of the Literature Resource Center), Wikipedia and Google.

Regarding breadth of coverage, only six of the 500 names were “absent” on Wikipedia (meaning that they had “no entry of their own or reference in any other entry”), compared to 14 for LRC, and 50 for the Biography Reference Bank.

While the study does not seem to have attempted a systematic comparison of factual accuracy, it observes that Wikipedia “entries are less uniform than those in commercial databases. The biographical information ranges from extensive to perfunctory”.

The author remarks favorably on Wikipedia’s searchability:

“The databases and Wikipedia deal better than the Web with variant names, pseudonyms, and names that apply to multiple people. Cross-referencing is very good. [...] Wikipedia searching is very easy.There were even cases where it was easier to search Wikipedia than the databases. [...] Wikipedia also ‘disambiguates’ names and offers quick descriptions to enable the searcher to find the correct individual.”

A large part of the comparison consists of examining each resource’s production process. Wikipedians may find parallels to their policies on biographies of living people, self-published sources and notability in the description for the Biography Reference Bank:

“Current Biography [the main content source of BRB] articles rely on secondary sources, but Wilson [the then publisher] has occasionally spoken directly with subjects or their proxies. Upon publication, many articles have been sent to subjects for review before being updated for the print annual and the databases. If subjects raise objections, misinformation is corrected, but not matters of public record. Adjustments may be made for privacy, for example omitting the specific names of children.

“To be included in World Authors [another source of BRB], authors must have published more than one critically acclaimed book. [...]“

“For autobiographies, Wilson attempted to contact subjects in Junior Authors and World Authors for a statement, but not subjects in Current Biography. [... An example offered by a Wilson employee:] For some reason, Jennie Tourel, a Russian-American opera singer, often provided false information, but, according to the Wilson biography, “passports and other documents that surfaced soon after her death helped to correct some of these inaccuracies’”.

In the conclusion, the author answers the initial question by recommending that her employer “re-subscribe to a commercial biographical database” if the budget would permit it again, because “Commercial databases provide a foundation with authoritative core content authenticated prior to publication and integrated with the fabric of information in the library’s holdings. They are easy to search and reliable, although they cannot be as current as Wikipedia or the Web because of their authentication processes. Wikipedia become[sic] more impressive as searching proceeded. The focus may be on verifiability rather than authority and there may be challenges in securing contributors, but the current contributors provide citations and often include unique information.” All in all she seems to favor Wikipedia and the two databases over “The web” (Google results) which “may have plenty of dross and be less reliable, harder to search, and focused on commercialism, but there are gold nuggets.” She worries: “What will happen if contributors to Wikipedia and the web have no authoritative databases to use as sources?”

Students predict connections between Wikipedians

Among the student projects in a class on “Computational Analysis of Social Processes” at Rensselaer Polytechnic Institute, three analyzed social networks of Wikipedia editors:

• The write-up for a project titled “Interaction vs. Homophily in Wikipedia Administrator Selection”^[3] provides an analysis of factors related to one’s participation (or lack thereof) in the Request for Adminship discussions. It confirms previous findings that many participants are drawn to the discussions by their personal contacts and experiences with others. The paper tries to analyze the impact of direct past interaction versus homophily (roughly defined as shared interests). The findings suggest that homophily plays a much smaller role compared to past interactions. Overall, it appears that administrators are often elected (or opposed) not by the community at large, but by a group of their closest peers. To quote from the conclusion of the paper: “This raises questions about the robustness of Wikipedia’s administrator selection process which is then comprised of a very small interaction-selected group of editors.”
• Another project write-up titled “Link Prediction Analysis in the Wikipedia Collaboration Graph”^[4] tested various models to predict the strength of the connection between two Wikipedia editors in a “dynamic collaboration graph” that measures, at a given point in time, how often they recently edited the same page, with more recent edits weighing stronger.
• A third student paper titled “Link prediction on a Wikipedia dataset based on triadic closure“^[5] likewise tested various models on a similar graph consisting of Wikipedia users as vertices, regarding the closure of triangles (i.e. if user A is connected with B, and B with C, is A connected with C as well?). Among the conclusions is that such “triadic closure, while still occurring in Wikipedia, is happening at a slower pace now than before–likely due to the influx of less active editors”.

Language analysis finds Wikipedia’s political bias moving from left to right

A study presented earlier this month at the annual meeting of the American Economic Association which is to appear in The American Economic Review^[6] sets out to test whether the English Wikipedia is truly neutral, by measuring bias within a sample of 28,000 entries about US political topics, examined over a decade. The bias is identified through detecting the use of language specific to one side of the American political scene (Democrats or Republicans). To quote from the article: “In brief, we ask whether a given Wikipedia article uses phrases favored more by Republican members or by Democratic members of Congress” (in the text of the 2005 Congressional Record, using a method developed in an earlier paper by Gentzkow and Shapiro who applied it to newspapers). The authors identified, as of January 2011, 70,668 articles related to US politics, about 40% of which had a statistically significant bias. They find that Wikipedia articles are often biased upon creation, and that this bias rarely changes. Early on in Wikipedia’s history, most had a pro-Democratic bias, and while “by the last date, Wikipedia’s articles appear to be centered close to a middle point on average”, this is simply an effect of a larger amount of new pro-Republican articles than due to the existing ones having been rewritten neutrally.

While the authors made efforts to exclude articles not pertinent to US politics (requiring the terms “United States” or “America” to appear at least three times in the article text), the sample also includes the clearly international article Iraq War. And in what Wikipedians may call out as systemic bias, the authors never question their assumption that for an international encyclopedia, a lack of bias would be indicated by the replication of the spectrum of opinions present in the US Congress. As early as 2006, Jimmy Wales objected to such notions with respect to the community of contributors: “If averages mattered, and due to the nature of the wiki software (no voting) they almost certainly don’t, I would say that the Wikipedia community is slightly more liberal than the U.S. population on average, because we are global and the international community of English speakers is slightly more liberal than the U.S. population. … The idea that neutrality can only be achieved if we have some exact demographic matchup to [the] United States of America is preposterous.” Nevertheless, even if one turns the study on its head and reads it as a statement on average American political opinion compared to the rest of the world as reflected in the English Wikipedia, its results remain remarkable.

Briefly

• Calls for papershave appeared this month for
  • WikiSym 2012, the eighth instance of this annual research conference on wikis and open collaboration
  • Wikimania 2012, the eighth annual global conference of Wikimedians
  • Wikipedia Academy: Research and Free Knowledge, a conference organized by Wikimedia Germany
  • “Academic research into Wikipedia: Beyond English Wikipedia and towards comparative perspectives“, an upcoming issue of the e-journal Digithum
• New effort at comprehensive wiki research literature database: Wikipedian emijrp has announced the launch of WikiPapers, a Semantic MediaWiki-based wiki dedicated to the “compilation of resources (conference papers, journal articles, theses, books, datasets and tools) focused on the research of wikis”. The task of creating such a database has been attempted several times before and its difficulties were explored in a well-attended workshop at last year’s WikiSym conference (see the October issue of this newsletter). Researcher Finn Årup Nielsen (who last year published an overview of such literature,^[7] mentioning well over 1000 publications) pointed out the possibility of exchanging content between the new wiki, the existing (likewise Semantic MediaWiki-based) Acawiki and his own Brede Wiki.
• Review of Good Faith Collaboration: Sociological journal The Information Society reviewed^[8] Joseph Reagle’s 2010 book Good Faith Collaboration: The Culture of Wikipedia (which was recently released online under a Creative Commons license), praising it as “an accurate account of this sociocultural and sociotechnological phenomenon that Wikipedia is”. The reviewer calls Wikipedia a “virtual tool and reference jim-dandy [which] is another flashpoint in our path of social anxieties” and holds that “nobody can think of a true rival [of Wikipedia] in this knowledge contest. The sins are there: lightness, temporary reliability, questionable scholarly approaches, sometimes oversimplification, sometimes data excess; however, these are venial sins and easily absolved.” Somewhat cryptically, he observes that “the European Union is trying to adapt this part of Western academia to the global university system (though inevitably Anglo-American inspired)”. He commends the book as an “accessible analysis [which] makes it clear that Wikipedia is not wasted knowledge; it is human thirst for knowledge and we are simply gathering scattered pieces”. Gentle criticism includes that “though [...] inaccuracies are stated, two other important worries — that it is not financially sustainable, and that Wikipedia has lost touch with its founding ideal—are not as openly dealt with”.
• Predicting categories from links: In a paper titled “Using Network Structure to Learn Category Classification in Wikipedia”^[9] (the write-up of a class project for an Autumn 2011 Stanford course titled “Social and Information Network Analysis”), three students describe the construction of a classifier algorithm that tries to predict from an article’s ingoing and outgoing wikilinks whether it is a member of the Category:American actors – “We chose this particular category because it is one of the largest on Wikipedia (almost 25,000 pages)”.
• Wikipedia vs. library catalogue: An article in Library and Information Research titled “Searching where for what: A comparison of use of the library catalogue, Google and Wikipedia”^[10] analyzed search queries from users of Google (using Hitwise data) and Wikipedia, and a state library in Australia, unsurprisingly finding that the library catalogue is used much less frequently than the former two, but positing that the “fact that popular culture queries accounted for [a very] substantial proportion of Google and Wikipedia queries and almost no [library] catalogue queries indicates that, indeed, people do turn to different information resources for different subjects.”
• “Lexical clues” predict article quality: A paper was presented at the 3rd Symposium on Web Society (SWS) last October^[11] which sought to predict article quality based on eight different ratios derived from counting the number of sentences, words, diverse words, nouns, verbs, diverse nouns, diverse verbs and copulas in the article text. They trained a decision tree on a sample of 200 start-class and 200 featured articles (truncating each of the latter to 800 to 1000 words to arrive at a typical start-class article length) and then tested it on a different sample of 100 start-class and 100 featured articles, achieving precision and recall of more than 83% each.

References

Vol: 1 • Issue: 6 • December 2011 [archives]

Psychiatrists: Wikipedia better than Britannica; spell-checking Wikipedia; Wikipedians smart but fun; structured biological data

With contributions by: Tbayer, DarTar and Jodi.a.schneider

Mental health information on Wikipedia more accurate than Britannica and Kaplan & Sadock psychiatry textbook

Wikipedia articles on schizophrenia and other mental health topics were assessed for accuracy, richness of references and readability.

In an article for Psychological Medicine,^[1] ten researchers from the University of Melbourne conclude that “the quality of information on depression and schizophrenia on Wikipedia is generally as good as, or better than, that provided by centrally controlled websites, Encyclopaedia Britannica and a psychiatry textbook.”

The study focused on ten mental health topics (e.g. “antidepressants and suicide in young people” or “side-effects of antipsychotics”), five each in the areas of depression and schizophrenia. “Using the topic terms (or synonyms) as key words for the searches or through manual browsing, content relating to these topics was extracted from [Wikipedia and 13 other websites selected for prominent Google results for depression and schizophrenia] and from the most recent edition of Kaplan & Sadock’s Comprehensive Textbook of Psychiatry … and the online version of Encyclopaedia Britannica” by two reviewers. For both depression and schizophrenia, three psychologists with clinical and research expertise in that area evaluated these extracts on accuracy, up-to-dateness, breadth of coverage, referencing and readability, on a scale from 1 to 5 (“e.g. Accuracy: 1 = many errors of fact or unsubstantiated opinions, 3=some errors of fact or unsubstantiated opinions, 5 = all information factually accurate”). As in an earlier study of the quality of health information on Wikipedia (Signpost coverage: “Wikipedia’s cancer coverage is reliable and thorough, but not very readable“), readability was also measured using a Flesch–Kincaid readability test, which is calculated from word and sentence lengths.

For both depression and schizophrenia, Wikipedia scored highest in the accuracy, up-to-dateness, and references categories – surpassing all other resources, including WebMD, NIMH, the Mayo Clinic and Britannica online. In breadth of coverage, it was behind Kaplan & Saddock and others for both areas. And “of the online resources, Wikipedia was rated the least readable [by the human reviewers], although some of its topics received an average rating.” Likewise, the Wikipedia content had relatively high Flesch–Kincaid Grade Level indices (around 16 for schizophrenia and 15 for depression – indicating that a tertiary level of education is necessary to understand the content), similar to that of Britannica but higher than most other resources examined.

The authors note that their “findings largely parallel those of other recent studies of the quality of health information on Wikipedia” (citing eight such studies published between 2007 and 2010):

“Despite variability in the methodologies and conclusions of these studies, the overall implication is that Wikipedia articles on health topics typically contain relatively few factual errors, although they may lack breadth of coverage. … Given the number of patients, would-be patients and concerned others using the internet to search for information on health issues, it seems that Wikipedia is an appropriate recommendation as an information source.“

Psychologists gauge impact of Wikipedia’s Rorschach test coverage

A black outline of the first of the ten cards in the Rorschach inkblot test (used in a pre-2009 version of the article Rorschach test)

A paper in the Journal of Personality Assessment^[2] tried to assess the impact of the Wikipedia article Rorschach test on psychologists’ use of that test. As summarized by the authors, “In the summer of 2009, an emergency room physician [User:Jmh649 - James Heilman, MD] posted images of all 10 Rorschach inkblots on [...] Wikipedia. The images were accompanied by descriptions of “common responses” to each blot. … a fierce debate ensued between some psychologists who claimed that posting the inkblots is a threat to test security and other individuals, including some psychologists and other mental health professionals, who argued that all information should be freely available, including full details of the Rorschach”. (In fact, the debates on whether to display versions of the inkblots in the article go back to at least 2005, at first accompanied by rather spurious copyright claims – Rorschach died in 1922.) The authors note that the inkblots had already been revealed to the general public in a 1980s book and cite an earlier study^[3] that had found “particularly damaging information” about personality assessment tests on the Internet as early as 2000, “including examples of test stimuli from… the Rorschach” (presumably including this site). Still, “Internet coverage of the Rorschach appeared to grow exponentially during” the 2009 debate about the Wikipedia article, which made it to the front page of the New York Times (Signpost coverage: “Rorschach test dispute reported“).

The first part of the study examined the top 50 Google search results for “Rorschach” (excluding “watchmen” in order to filter out results about a comic book and film) and “inkblot test”, coding them into four levels representing the “threat each site presents to test security and the extent to which the content of the site might aid an individual in dissimulating on the Rorschach”. 44% of the sites were classified as Level 0 (“no threat”), e.g. home page of bands with “Rorschach” in their name, and 15% as Level 1 (“minimal threat”). The 22% Level 2 (“indirect threat”) sites which “tended to discuss test procedures more explicitly” apparently included “several ‘official’ Rorschach Web sites, where one is able to register for Continuing Education Rorschach workshops, [and which] also allow visitors to purchase materials that contain sensitive test information. For example, certain training Web sites allow individuals to purchase training texts and instructional media without requiring a license or other professional credentials”. The authors find it “disturbing” that many sites in this threat category “were authored by psychologists”. 19% of the sites were classified as the highest level, “direct threat”, e.g. many that contained depictions of one or more Rorschach inkblots, or specific information about how responses are interpreted. Together with results about the high percentage of Internet users consulting Wikipedia for health information (36% in the US in 2007 according to Pew research), the authors conclude that “we can no longer presume that examinees have not been exposed to this information prior to an assessment”.

The second part of the study likewise starts out with a Google News search for “Rorschach” and “Wikipedia”, noting that “of the 25 news stories reviewed, 13 included one or more of the Rorschach inkblots, with Card I as the most frequently displayed”, and eventually arriving at five media stories about the controversy which allowed readers’ comments ([1], [2], [3], [4], [5]). The altogether 520 comments on these stories were “coded according to the opinion expressed by the writer regarding each of the following categories: (a) the field of psychology, (b) psychologists, and (c) the Rorschach.” While the vast majority did not state a clear opinion on the first two categories, the authors note that “Of those comments that did express an opinion toward psychologists [ca. 16%] most were overwhelmingly negative.” Many more of the commenters on the Wikipedia/Rorschach news stories expressed an opinion about the test itself: “In total, 182 (35%) of comments were classified as unfavorable toward the Rorschach, whereas only 55 (11%) were coded as favorable toward the Rorschach. The remaining 283 (54%) comments were categorized as neutral or not mentioned.” Among those who identified as mental health professionals, 61% expressed a favorable opinion about the test and 15% a negative one.

Asked for his comment on the paper, Heilman said: “My main criticism of their paper is that they seem to take as axiomatic that exposure to these images hurts test reliability without any real evidence to back it up. Otherwise it is an interesting piece.” (The paper includes a section reviewing literature on “the impact of ‘coaching’ on psychological tests”, however it does not mention results pertaining specifically to the Rorschach test, and mostly concerns subjects who deliberately try to “cheat” on such tests, rather than those who have accidentally been exposed to a test’s material before.)

Spell-checking the English Wikipedia

Proofreading Wikipedia articles: A screenshot from wikiCheck

University of Nebraska-Lincoln MBA candidate Jon Stacey reports on the results of a proof-of-concept tool to measure the rate of misspelled words in the English Wikipedia over time.^[4][5] A text parser (code available for download) was applied to a random sample of 2,400 articles. Instead of considering the latest revision, a random revision from the history of each article was used. The final corpus was obtained by stripping markup and non-ASCII characters as well as article sections such as the references and table of contents. Words were matched against a dictionary obtained by manually combining 12dicts and SCOWL (source) with Wiktionary.

The results show that the percentage of misspellings has been growing steadily, reaching 6.23% for revisions created in 2011. Several weaknesses with the method are discussed, including the lack of Unicode support, the high rate of false positives, and the possibility that the rising rate might be associated with a rise in the complexity of content. The concluding remarks speculate on how semi-automated spell-checking may support editorial work at a large scale. (Wikipedians have used lists of common misspellings for many years, also integrated in semi-automatic editing tools such as AutoWikiBrowser.)

In related news, the developers of an open-source multilingual proofreading application called LanguageTool released a beta application for proofreading Wikipedia articles. wikiCheck proofreads articles from the English and German Wikipedias based on a set of customizable syntax and grammar rules. A bookmarklet is available to access the application from a browser.

Wikipedians are “smart but fun”, and have expertise in topics they edit

Three researchers from Stanford University and Yahoo! Research used a novel method to construct “a data-driven portrait of Wikipedia editors”, as described in a preprint currently undergoing review for publication.^[6] While earlier studies relied on Wikipedians participating in surveys (and identifying themselves as such), the authors mined data from users of the Yahoo! Toolbar for Wikipedia URLs containing an &action=submit parameter, thereby arriving at a sample of 1900 editors of the English Wikipedia.

Their first main finding is that “on broad average, Wikipedia editors seem, on the one hand, more sophisticated than usual Web users, reading more news, doing more Web search, and looking up more things in dictionaries and other reference works; on the other hand, they are also deeply immersed in pop culture, spending much online time on music- and movie-related websites.” However, these “entertainment lovers … form only a highly specialized subgroup that contributes many edits”.

Based on the toolbar data, the paper also tries to answer the question “Do Wikipedia editors know their domain?” and related questions, positively: “across all topical domains Wikipedia editors show significant expertise. … We also show that more substantial edits tend to come from experts”, and that logged-in editors show more expertise than IP editors. A final result is that “About half of the click chains culminating in an edit start with a Web search, with the other half originating on Wikipedia’s main page.”

Wikipedia as a database for structured biological data

The Oxytocin article is part of Gene Wiki, an open-access collection of Wikipedia articles about human genes.

A special issue of Nucleic Acids Research features 11 articles describing how wikis and collaborative technology can be used to enhance biological databases. A commentary by Robert Finn, Paul Gardner and Alex Bateman^[7] discusses in particular how to leverage Wikipedia, its collaborative infrastructure and large editor community to better integrate articles and biological data entries: the authors argue that the project offers an opportunity for crowdsourcing the curation and annotation of biological data, but faces major challenges for expert engagement, i.e. “how to get scientists en masse to edit articles” and “how to allow editors to receive credit for their work on an article”.

Another article in the same issue^[8] presents the Gene Wiki, an open-access and openly editable collection of Wikipedia articles about human genes. The article describes how structured data available on Gene Wiki articles is kept in sync with the data from primary databases via an automated system and how to automatically compute the quality of articles in the project at word or sentence-level using WikiTrust.

Individual and social drivers of participation in Wikipedia

A thesis entitled Individual and social motivations to contribute to Commons-based peer production was submitted by University of Minnesota student Yoshikazu Suzuki for an MA in mass communication. The thesis presents and discusses the results from a small series of interviews as well as a survey exploring individual and social motivations of Wikipedia contributors, drawing on social identity theory, volunteerism and uses and gratifications theory. The survey, run in July 2011 with support from the Wikimedia Research Committee, collected 208 responses from a random sample of 950 among the top English Wikipedia editors. The results, obtained by applying principal components analysis to the responses, reveal eight distinct motivational factors: providing information, the seeking of creative stimulation, concern for others’ well-being, the need to be entertained, the avoidance of negative self-affect, cognitive group membership, career benefits, and social desirability. An analysis of the relative strength of each factor indicates that providing information, the seeking of creative stimulation, and concerns for others’ well-being were the three strongest motivational dimensions. Grouping the eight factors into two macro-categories according to self- and other-focused motivations, the other-focused motivations were found to be significantly stronger than the self-focused motivations. The thesis reviews the implications of these results for the design of incentives for participation and editor retention. The full text of the thesis^[9] and an executive summary are available under open access.

Mining article revision histories for insights into open collaboration

Article revision histories are a rich data source to study patterns of collaboration on Wikipedia.

A paper in this month’s edition of First Monday, ambitiously titled “Understanding collaboration in Wikipedia”^[10], reports on a statistical analysis of a complete dump of the English Wikipedia (225 million article edits) with regard to several quantities, starting with two that were introduced in a 2004 paper by Andrew Lih “as a simple measure for the reputation of [an] article within the Wikipedia”: the total number of edits an article has received (“rigor”) and the number of (logged-in and anonymous) users who have edited the article (“diversity”). The First Monday paper cites a 2007 study from the same journal, which found that featured articles tend to have more edits and contributors (while controlling for a few other variables)^[11] as a justification for using “rigor” and “diversity” as proxies for article quality, but includes other quantities such as the article size change for an edit. The paper cites earlier work on evaluating Wikipedia article quality (e.g. dismissing the well-known 2005 Nature study based on the mistaken assumption that it had “only focused on featured articles”), but does not discuss existing attempts at more sophisticated quantitative quality heuristics.

The First Monday paper highlighted that if consecutive edits by the same user are counted as one, the overall number of article revisions drops by more than 33%, “revealing that one in three revisions in Wikipedia consist of users responding to their own edits or continuing an ongoing edit begun by themselves”. “Article diversity” ranged up to 12,437 contributors per article, with a median of 12 and an average of 32. One of the main conclusions is that “rather than reflecting the contributions and expertise of a large group of people, the typical article in Wikipedia reflects the efforts of a relatively small group of users (median of 12) who make a relatively small number of edits (median of 21).”

Supporting the assumption that most edits do not result in significant changes in content, the study finds that 31% of all revisions cause a size change of fewer than 10 characters, and 51% a change of fewer than 30 characters, with an apparently significant peak at a 4-character difference, presumably related to the insertion or removal of the four brackets (“[[ ... ]]”) that generate a wikilink.

The author notes the slight decrease in the overall number of edits since 2008, but tentatively explains it by the increasingly complete coverage of encyclopedic topics, and doesn’t share the widespread concerns about declining or stagnating editor activity: “participation in Wikipedia seems to remain as healthy as ever as revisions made per article created each year has annually increased since 2001 without exception”.

A different paper^[12] from last year’s “Collaborative Innovation Networks Conference” similarly promises far-reaching insights from “Deconstructing Wikipedia” solely based on revision history statistics without analyzing the actual content changes, using a much smaller sample – 30 featured articles from the English Wikipedia, but also including timestamps. The data did not confirm the hypothesis that “the editor who initiated an article would have a high level of involvement in the article’s creation”: for only five of the 30 articles, the initial author was the most frequent contributor.

A second conclusion is that for all of the articles in the sample, “there is a single Wikipedian whose contributions far exceed all others”, ranging from 8% to 82% of the articles with an average of 39% (but the analysis does not seem to have sought to quantify the extent to which this exceeds the contributions of the second most frequent contributor). The author indicates that this supports Jaron Lanier‘s “oracle illusion” criticism of a supposed presentation of Wikipedia as a product of “the crowd”. Somewhat tautologically, the author observes “that the control of an individual editor seemed to be reduced as more editors joined the process”, and points to the need to analyze “a significantly larger number of articles” to answer the question whether “too many cooks spoil the stew” (apparently unaware of the significant body of earlier literature on this subject, starting with a 2005 paper that presented an answer in its title: “Too many cooks don’t spoil the broth“, and including the 2007 study which the above reviewed First Monday paper relied on).

A third result of the paper, which likewise might not surprise those already familiar with Wikipedia’s editing processes, is that “the creation process is continuous and can go on for a very long time”, with even articles about historic events from the distant past continuing to receive edits.

The author, an assistant professor in management and marketing at Virginia State University, concludes the paper by urging his readers to start “thinking about how the wiki platform, itself, is influencing the creation process”.

Briefly

• The Wikimedia Research Committee launched a public consultation on the future data/research infrastructure for Wikimedia, in an effort to understand how to best serve the research and developer community with open data from our projects. The consultation will remain open through January 2012 and the full set of responses will be shared under a CC0 license.
• Semantic enhancements: In “Enhancing Wikipedia with semantic technologies”^[13], Lee et al. review existing interfaces for semantic search and present their own platform for enhancements. Based on small-scale user tests, they find that one of their three enhancements – range-based queries – are strongly preferred by users, who would find them desirable not only in Wikipedia but on the wider web. A longer summary is available on AcaWiki.
• English and Finnish Wikipedias egalitarian, Japanese hierarchical: A paper titled “Analyzing cultural differences in collaborative innovation networks by analyzing editing behavior in different-language Wikipedias”^[14] (from the 2010 Collaborative Innovation Networks Conference, as was the revision statistics paper reviewed above) applied social network analysis to collaboration on featured articles on the English, German, Japanese, Korean, and Finnish Wikipedias. It “found notable differences in the communication behavior among egalitarian cultures such as the Finnish, and quite hierarchical ones such as the Japanese. While the English language Wikipedia shows a distinctive pattern, most likely because it is by far the largest and frequently exploring new concepts copied by others, it seems to follow more the Finnish egalitarian, than the Japanese hierarchical style”.
• User:Emijrp shared a link on wiki-research-l listing 2,596 scholarly references on wikis, obtained by scraping Google Scholar results (on December 22, 2011), as part of a project to build a comprehensive bibliography about wikis – a challenging task that has seen various earlier attempts and was the subject of a workshop at this year’s WikiSym (see the October and April editions of this research report).
• Should doctors use and edit Wikipedia?: An editorial in the Journal of the Royal Society of Medicine^[15] asks whether doctors should reject the use of Wikipedia. The two page article (one-day access: US$30.00) cites some results about the popularity of Wikipedia among medical students, young physicians and the general public, and for some reason highlights the malicious edits of British journalist Johann Hari as example for the downsides of Wikipedia’s free editability. It contains a review of the literature on the reliability of Wikipedia’s medical information which is less thorough than that of the Psychological Medicine article reviewed above, and comes to a less approving but still somewhat positive conclusion: “Although Wikipedia entries are often poorly structured and difficult to understand, they are comparable in accuracy to some online resources, such as health insurance websites.” In the end, the authors seems to lean towards recommending against ignoring Wikipedia: “One risk of clinicians disengaging from Wikipedia is that only contributors motivated by personal experience (e.g. patient anecdote) or vested interests (e.g. individual clinicians, institutions or companies promoting their own ideas and products) will remain.”

References

Vol: 1 • Issue: 5 • November 2011 [archives]

Quantifying quality collaboration patterns, systemic bias, POV pushing, the impact of news events, and editors’ reputation

With contributions by: Tbayer, Hfordsa, DarTar and Romanesco

Collaboration pattern analysis: Editor experience more important than “many eyes”

One of the motifs indicating article quality: One editor (top) having worked on several related articles (bottom)

A paper titled “Characterizing Wikipedia Pages Using Edit Network Motif Profiles”^[1] by three researchers from University College Dublin indicates that the quality of a Wikipedia article can be predicted from characteristics of its “edit network” – a graph derived from the collaboration of Wikipedians in that area. Network motifs are small graphs which occur particularly frequently as sub-graphs of networks of a certain kind, and can be regarded as its building blocks in some sense. (The concept is popular in bioinformatics, where it is applied to gene regulatory networks.) In this paper, the authors use graphs with at most five nodes consisting of users and articles, which are connected by an edge if the user has edited the article – giving 17 possible “Wikipedia network motifs”. (Anonymous users are disregarded.) For a Wikipedia article, the researchers form an “ego network” consisting of that article, articles which link to it (and have been edited by at least one of the users who edited the core article), and the users who edited them. For a sample of around 2000 articles from the History and United States categories, the frequencies of the 17 “Wikipedia network motifs” in those article’s “ego networks” were calculated.

Using machine learning techniques, the researchers are able to discern with some certainty articles of basic quality (defined as having been assessed as Start class by Wikipedians) from those of good quality (defined as Featured or B class), solely based on this set of motif frequencies in the article’s edit network. Looking at the impact of each of the 17 types separately, they found that “all network motifs have some potential to discriminate between good and basic Wikipedia articles” in the sample, but that among the four best predicting motifs, three are “stars with editors at their centre”:

“This is interesting because it shows that many eyes is not really the defining characteristic of quality; instead experience is important – the editors should have worked on many other articles.”

Another section of the paper constructs spatializations of the sample (i.e. a two-dimensional mapping where articles with similar motif frequency are close to each other). For the history articles sample, this visualization clearly separated B class and Start class articles, but Featured articles are “more spread out”, with two clusters on opposite sides of the diagram. The researchers made the interesting discovery that this seems related to the assessed importance of the articles:

“It transpires that the Featured Articles on the left are inclined to be low or mid importance compared to high or top importance articles on the right. This niche characteristic is emphasized by the fact that these articles are inclined not to have been featured on the Wikipedia main page. We conclude from this that, at least in edit network terms, some low importance Featured Articles look like more ordinary articles. … It seems that articles on niche topics can reach Featured Article status without a huge amount of collaboration.”

Systemic bias quantified for twenty language Wikipedias

A paper titled “Cultural Configuration of Wikipedia: Measuring Autoreferentiality in Different Languages”^[2] by two researchers from the Universitat Politècnica de Catalunya, published in the proceedings of the “Recent Advances in Natural Language Processing” conference and apparently based on the first author’s masters thesis^[3] attempts to test the hypothesis that “contributing for the visibility of the own national or language related content” is among the motivations to participate in Wikipedia. According to the authors, “some informal surveys in Catalan WP association ‘Amical Viquipèdia’ showed how the national topics were a focus of interest for writing and conflict”. They propose the concept of “autoreferentiality” “to describe the interest of a culture on itself, which in WP translates to the interest of editors for their own local content in a WP language edition”, and set out to measure it by various quantitative features, which are first defined on the article level and then tested on a selection of articles that are assumed to be “local content”, using the Java-based WikAPIdia tool. (This set is formed by starting with a few keywords clearly pertaining to the local language, and then including articles which share categories – as examples from their own language, the authors list ““catalunya”, “català”, and also “valencia” or “mallorquí”” as start words, which “would retrieve titles in articles and categories like “escriptors de catalunya” or “dret català”, referring to writers and law”.) Among the tested quantitative features are:

• “Isolation”, based on the number of interwiki links
• “Effort”, based on the size of the article and the number of internal links it contains
• “Prominence”, based on the number of incoming wikilinks, the number of categories where the article is a member, and its PageRank
• “Edition”, a measure of how diverse the authorship of an article is, specified as the smallest number of editors who together contribute 80% of the page’s edits (assuming to be lower for local content because it is edited by “highly motivated users”)

The paper applies the eventual formula to Wikipedias in twenty languages – the English language edition is excluded “due to its size and difficulties in processing in all dimensions”, and the second and third largest Wikipedias (German and French) are missing as well. In the final “autoreferentiality index”, the Icelandic, Japanese and Swahili Wikipedias come out as the most local-focused among these twenty, while, curiously, the Catalan edition which prompted the research question has the lowest autoreferentiality value.

Caused a “heavy editorial event” earlier this year: Elizabeth Taylor

Does “In the news”-like attention have a positive effect on article quality?

A five page paper^[4] by a Ph.D. student in Computer Science at the University of Iowa examines “The Impact of Heavy Editorial Events on Wikipedia Page Quality” – for example the flurry of edits to the article Elizabeth Taylor after the actor’s death in March 2011. To measure quality, the approach of an earlier paper^[5] is used, which assigns article contributors a reputation value depending on how many of their earlier contributions have been deleted, and by whom, and also takes into account whether the article revision in question was reverted later. The resulting formula was applied to “high editorial events” in 100 articles of the English Wikipedia, from the start of Wikipedia in 2001 until the beginning of 2010. As expected, the data supported the hypothesis that “high editorial events would contribute positively to a page’s quality”. The five articles impacted most positively among the studied sample (biased toward the beginning of the alphabet) were art, Allen Ginsberg, anarcho-capitalism, chiropractic and death. The paper also found that a higher increase in the edit rate was associated with a higher quality increase, but does not address the question of whether the relation could be explained by the mere number of edits (i.e. whether the same number of edits over a longer time might have had the same effect).

Detecting POV pushing editors

A working paper posted this month to ArXiv with the title “Pushing Your Point of View: Behavioral Measures of Manipulation in Wikipedia” presents a method to score the neutrality of Wikipedia contributors and to “detect potential POV pushing behavior”.^[6] The authors propose two metrics to quantify an editor’s involvement in controversial topics. The first metric (Controversy score or C-score) measures the amount of attention spent by an individual editor on controversial articles, where controversiality is defined on the basis of several quantitative factors previously established in the literature. The second metric (Clustered Controversy score or CC-score) quantifies the focus of an editor’s attention on controversial articles on the same topic or very similar topics: the purpose of this metric is to tease apart editors involved in genuine controversy resolution (such as administrators who are likely to participate in a broad range of discussions on controversial topics) from “potentially manipulative users” who focus their attention on a narrow set of controversial topics. To assess the validity of the above metrics the authors test their discriminatory power at identifying which editors are blocked and which are regular users who were never blocked. The remainder of the paper examines the breakdown of edits by administrators immediately after a successful Request for Adminship. The results, based on qualitative coding by a single reviewer, suggests that some topical areas in the English Wikipedia (such as politics and media) are more likely to be frequently edited by administrators with a high C-score and CC-score than any other topical categories.

Historian of encyclopedias reviews Good Faith Collaboration

The most recent issue of Annals of Science (a scholarly journal about the history of science and technology, founded in 1936) contains a four-page review^[7] of Joseph Reagle’s book Good Faith Collaboration: The Culture of Wikipedia (published in 2010 and recently released on the Web under a CC-BY-NC-SA license). The reviewer Jeff Loveland, who has written extensively about the early history of encyclopedias, criticizes the book for having “one major weakness, namely in historical contextualization” (he mentions two 18th-century precedents which should have been given more attention, as they, like Wikipedia, intended to include contributions from the public: Vincenzo Coronelli‘s Biblioteca Universale and Zedler‘s Universal-Lexicon) – and rejects Reagle’s claim that “historically, reference works have made few claims about neutrality as a stance of collaboration, or as an end result”: “References to such values as impartiality, unbiasedness and objectivity are frequent in the prefaces of encyclopaedias over the last three hundred years”. On the other hand, the reviewer praises the book for “com[ing] close to offering” a comprehensive introduction to Wikipedia, “touching as it does on nearly all aspects of the encyclopaedia” and he commends the author’s writing style as “informal, energetic and appropriately paced”. The “insightful and worthwhile” ethnography of Wikipedia is highlighted as the second success of the book.

Regarding chapter 3 of the book, which postulates Neutral Point of View and Assume Good Faith as the two principles at “the heart of Wikipedia collaboration”, the review recommends “Anne Goldgar’s study of conduct as a force binding together the early modern Republic of Letters in Impolite Learning (1995) [as] an interesting point of comparison” regarding “the historical connection between knowledge and civility”. Commenting on chapter 7, which examines criticism of Wikipedia, Loveland observes that “the portrayal by critics of a possible Wikipedian collective intelligence as anti-individualistic, or anti-rationalistic seems opportunistic and off-the-mark. Meanwhile, Wikipedia now bears the brunt of a refurbished but centuries-old accusation against encyclopaedias, namely that they trivialize and fragment knowledge.”

Briefly

• Automatically assessing editors’ reputations: Wöhner, Thomas and Köhler present new metrics for automatic reputation assessment of Wikipedia editors. They evaluate seven potential metrics for reputation assessment including editing frequency and contribution to high-quality articles, plus new metrics that they conceived including ‘efficiency’ which they define as ‘the portion of an author’s contribution that is persistent and quantifies the acceptance of the author within the Wikipedia community.’ They evaluate the metrics using a database of the Germany Wikipedia from January, 2008 and tested their metrics against Wikipedia’s internal user classification of blocked users, administrators and anonymous users. They conclude that editing efficiency is most significant for reputation assessment since it was able to distinguish between blocked and regular authors with an accuracy rate of 86%.^[8]
• Students reflect on Wikipedia assignments: Chen and Reber present the results of a pilot study where students from a Norwegian and a German university were asked to reflect on their experience writing a Wikipedia article as a course assignment. They provide the results of two independent judges who analyzed the written reflections of students on ten dimensions including: relevance for society, learning outcome and difficulty, among others. The authors conclude that students were highly motivated by the task and ‘have learned much about the topic that they wrote about’.^[9]
• Too few newbies?: A paper titled “Too Few New Wikipedians? Modelling Effort and Participation in Wikipedia”^[10] evaluates “the efficiency of the Wikipedia projects in different languages in transforming inputs (people using the Internet) into outputs (articles). We find a decreasing return to scale in the biggest projects, but the size or the age of the projects are not the main explanation for the variations in efficiency we see.”
• How student editors use sources: synthesis vs plagiarism: Sormunen and Lehtio report the findings of a pilot study on how Finnish secondary-school students use sources when they are required to contribute to Wikipedia as part of their coursework. They interviewed, observed, and analyzed the work of 11 groups of students, and found that: (1) the students relied almost exclusively on Web sources, (2) a sizable fraction (33%) of their work was copied verbatim (or very lightly edited) from their sources, (3) 30% of sources used were not cited at all. The dataset in this study is extremely small and the sample was not designed to be at all representative. Still, the conclusions are disconcerting, especially considering the recent controversy over student plagiarism in a related Wikipedia writing program (Signpost coverage: “A post-mortem on the Indian Education Program pilot“). The interviews with the students could potentially provide insight into why student plagiarism occurs.^[11]
• Tracking changes in Wikipedia: A student thesis in Computer Science at the University of Dresden^[12] describes the prototype of a software that tracks and categorizes edits on Wikipedia – trying, among other things, to detect articles that are being affected by external events. In a test sample containing articles that had been subject to major news events in recent years, such as Fukushima Daiichi Nuclear Power Plant or Dominique Strauss-Kahn, “about 74% of the events … have been detected and about 68% of these detected events (74%) are recognized correctly.”
• Gendered language on Wikipedia: Several Wikimedians have announced a study titled “Mind the Gap(s)! Writing Styles of Female Editors on Wikipedia”,^[13] applying algorithms that try to classify a text as “male” or “female” (based on the frequency of “male keywords” and “female keywords”) to text contributions by editors who state their gender on their user page (1,119 females and 722 males). Among the conclusions: “While the data is insufficient to reach the conclusion that Wikipedia attracts females who code their language usage as male in all circumstances on-wiki and off-wiki, we have shown that females use a more male style of writing when writing for Wikipedia.”

References

With contributions by: Boghog, Jodi.a.schneider, Drdee, DarTar, Phoebe and Tbayer

Wiki research beyond the English Wikipedia at WikiSym

Panel discussion at WikiSym 2011

WikiSym 2011, the “7th international symposium on wikis and open collaboration”, took place from October 3-5 at the Microsoft Research Campus in Silicon Valley (Mountain View, California). Although the conference’s scope has broadened to include the study of open online collaborations that are not wiki-based, Wikipedia-related research still took up a large part of the schedule. Several of the conference papers have already been reviewed in the September and August issues of this research overview, and the rest of the proceedings have since become available online.

The workshop “WikiLit: Collecting the Wiki and Wikipedia Literature“^[1], led by Phoebe Ayers and Reid Priedhorsky, explored the daunting task of collecting the scholarly literature pertaining to Wikipedia and wikis generally. Research about wikis can be difficult to find, since there are papers published in many fields (from sociology to computer science) and in many formats, from published articles to on-wiki community documents. There have been several attempts over the years to collect the wiki and Wikipedia literature, including on Wikipedia itself, but all such projects have suffered from not keeping up to date with the sheer volume of research that is published every year. While the workshop did not reach consensus on what platform to proceed with to build a sustainable system, there was agreement that this is an important topic for the research and practitioner community, and the group developed a list of requirements that such a system should have. The workshop followed and extended discussions on the wiki-research-l mailing list earlier this year on the topic.

In a panel titled “Apples to Oranges?: Comparing across studies of open collaboration/peer production“,^[2] six US-based scholars reviewed the state of this field of research. Among the takeaways were a call to study failed collaboration projects more often instead of focusing research on successful “anomalies” like Wikipedia, and – especially in the case of Wikipedia – to broaden research to non-English projects.

Another workshop, titled “Lessons from the classroom: successful techniques for teaching wikis using Wikipedia“^[3] was a retrospective on the Wikimedia Foundation’s Public Policy Initiative.

Among the conference papers not mentioned before in this newsletter are:

• “Mentoring in Wikipedia: a clash of cultures” ^[4], a paper which “draw[s] insights from the offline mentoring literature to analyze mentoring practices in Wikipedia and how they influence editor behaviors. Our quantitative analysis of the Adopt-a-user program shows mixed success of the program”.
• “Vandalism Detection in Wikipedia: A High-Performing, Feature–Rich Model and its Reduction“^[5] – arguing that on Wikipedia “human vigilance is not enough to combat vandalism, and tools that detect possible vandalism and poor-quality contributions become a necessity”, the authors present a vandalism classifier constructed using machine learning techniques.

Wikipedia-related posters included

• “A scourge to the pillar of neutrality: a WikiProject fighting systemic bias“^[6] presenting preliminary findings from an ongoing survey and interviews among members of the WikiProject Countering systemic bias.
• Another poster presentation planned to analyze the contributions of the members of this WikiProject to see what kind of systemic bias they might exhibit themselves (“Places on the map and in the cloud: representations of locality and geography in Wikipedia“^[7]).
• “Participation in Wikipedia’s article deletion processes“^[8] found that “the deletion process is heavily frequented by a relatively small number of longstanding users” and that “the vast majority of [speedily] deleted articles are not spam, vandalism, or ‘patent nonsense’, but rather articles which could be considered encyclopedic, but do not fit the project’s standards”.
• “Exploring underproduction in Wikipedia“^[9] examined “two key circumstances in which collective production can fail to respond to social need: when goods fail to attain high quality despite (1) high demand or (2) explicit designation by producers as highly important”.

Quality of drug information in Wikipedia

Lovastatin, the first statin to be marketed.

A study entitled “Accuracy and completeness of drug information in Wikipedia: an assessment”^[10] in this month’s issue of the Journal of the Medical Library Association of five widely prescribed statins found that while these Wikipedia drug articles are generally accurate, they are incomplete and inconsistent. The study’s authors conclude:

The main criticism by the study is that most of the articles lacked sufficient information on adverse effects, contraindications, and drug interactions and this lack of information might harm the consumer. These criticisms echo earlier ones (two similar studies reported in the Signpost: “Pharmacological study criticizes reliability of Wikipedia articles about the top 20 drugs“, ” Wikipedia drug coverage compared to Medscape, found wanting“). However the authors did note the benefit of Wikipedia hypertext links to additional information that most other web sources on drug information lack and in addition noted that all the Wikipedia articles contained references to peer reviewed journals and other reliable sources. Hence overall, the latest study is somewhat more positive than the earlier two.

Predicting editor survival: The winners of the Wikipedia Participation Challenge

Banner of the Wikipedia Participation Challenge

The Wikimedia Foundation announced the winner of the Wikipedia Participation Challenge. The data competition, organized in partnership with Kaggle and the 2011 IEEE International Conference on Data Mining, asked data scientists to use Wikipedia editor data and develop an algorithm to predict the number of future edits, and in particular one that correctly predicts who will stop editing and who will continue to edit (see the call for submissions). The response was overwhelming, with 96 participating teams, comprising in total 193 people who jointly submitted 1029 entries (listed in the competition’s leaderboard).

The brothers Ben and Fridolin Roth (from team prognoZit) developed the winning algorithm. They developed a linear regression model using Python and GNU Octave. The algorithm used 13 features (2 based on reverts and 11 based on past editing behavior) to predict future editing activity. Both the source code and a description of the algorithm are available. Unfortunately, because it relied on patterns in the training dataset that would not be present in the actual one, the model’s ongoing use is severely restricted.

Second place went to Keith Herring. Submitting only 3 entries, he developed a highly accurate model, using random forests, and utilizing a total of 206 features. His model shows that a randomly selected Wikipedia editor who has been active in the past year has approximately an 85 percent probability of becoming inactive (no new edits) in the following 5 months. The most informative features captured both the edit timing and volume of an editor’s activity.

The challenge also announced two Honourable Mentions for participants who only used open source software. The first Honourable Mention went to Dell Zang (team zeditor) who used a machine learning technique called gradient boosting. His model mainly uses recent past editor activity. The second Honourable Mention went to Roopesh Ranjan and Kalpit Desai (team Aardvarks). Using Python and R, they too developed a random forest model. Their model used 113 features, mainly based on the number of reverts and past editor activity (see its full description).

All the documentation and source code has been made available on the main entry page for the WikiChallenge.

What it takes to become an admin: Insights from the Polish Wikipedia

Logo of the Polish Wikipedia

A team of researchers based at the Polish Japanese Institute of Information Technology (PJIIT) published a study presented at SocInfo 2011 looking at Requests for Adminship (RfA) discussions in the Polish Wikipedia.^[11] The paper presents a number of statistics about adminship in the Polish Wikipedia since the RfA procedure was formalized (2005), including the rejection rate of candidates across different rounds, the number of candidates and votes over the years and the distribution of tenure and experience of candidates for adminship. The results indicate that it was far more complicated to obtain admin status in 2010 than it was in previous years, and that tenure required to be a successful RfA candidate has soared dramatically: “the mean number of days since registration to receiving adminship is nearly five times larger than it was five years before”.

The remainder of the paper studies RfA discussions by comparing the social network of participants based on their endorsement (vote-for) or rejection (vote-against) of a given candidate with an implicit social network derived from three different types of relations between contributors (trust, criticism and acquaintance). The goal is to measure to what extent these different kinds of relations can predict voting behavior in the context of RfA discussions. The findings suggest that “trust” and “acquaintance” (measured respectively as the amount of edits by an editor in the vicinity of those by the other editor and as the amount of discussions between two contributors) are significantly higher in votes-for than in votes-against. Conversely, “criticism” (measured as the number of edits made by one author and reverted by another editor) is significantly higher in votes-against than in votes-for.

This study complements research on the influence of social ties on adminship discussions reviewed in the past edition of the research newsletter.

High search engine rankings of Wikipedia articles found to be justified by quality

An article titled “Ranking of Wikipedia articles in search engines revisited: Fair ranking for reasonable quality?”, by two professors for information research from the Hamburg University of Applied Sciences (which appeared earlier this year in the Journal of the American Society for Information Science and Technology and is now available as open access, also in form of a recent arxiv preprint^[12]) addresses “the fiercely discussed question of whether the ranking of Wikipedia articles in search engines is justified by the quality of the articles”. The authors recall an earlier paper coauthored by one of them^[13] that had found Wikipedia to be “by far the most popular” host in search engine results pages (in the US): In “1000 queries, Yahoo showed the most Wikipedia results within the top 10 lists (446), followed by MSN/Live (387), Google (328), and Ask.com (255)”. They then set out to investigate “whether this heavy placement is justified from the user’s perspective”. First, they re-purposed the results of a 2008 paper of the first author,^[14] where students had been asked to judge the relevance of search engine results for 40 queries collected in 2007, restricting them to the search results that consisted of Wikipedia articles – all of them from the German version. They found that “Wikipedia results are judged much better than the average results at the same ranking position” by the jurors, and that

In order to conduct a more thorough investigation (the 2008 assessments having only focused on the criterion of relevance), the present paper sets out to develop a set of quality criteria for the evalulation of Wikipedia articles by human jurors. It first gives an overview of existing literature about the information quality of Wikipedia, and of encyclopedias in general, identifying four main criteria that several pre-2002 works about the quality of reference works agreed on. Interestingly, “accuracy” was not among them, an omission explained by the authors by the difficulty of fact-checking an entire encyclopedia. From this, the authors derive a set of 14 evaluation criteria, incorporating both the general criteria from the literature about reference works and internal Wikipedia criteria such as the status of being a featured/good article, the verifiability of the content and the absence of original research. These were then applied by the jurors (two last year undergraduate students with experience in similar coding tasks) to 43 German Wikipedia articles that had appeared in the 2007 queries, in their state at that time. While “the evaluated Wikipedia articles achieve a good score overall”, there were “noticeable differences in quality among the examples in the sample” (the paper contains interesting discussions of several strengths and weaknesses according to the criteria set, e.g. the conjecture that the low score on “descriptive, inspiring/interesting” writing could be attributed to “the German academic style. A random comparison with the English version of individual articles seems to support this interpretation”).

The authors conclude:

Both the search engine ranking data and the evaluated Wikipedia article revisions are somewhat dated, referring to January 2007 (the authors themselves note that it “could well be that in the meantime search engines reacted to that fact [the potential of improving results by ranking Wikipedia higher] and further boosted Wikipedia results”, and also that regarding the German Wikipedia, the search engine results did not take into account possible effects of the introduction of stable versions in 2008).

Attempts to predict the outcome of AfD discussions from an article’s edit history

A master’s thesis defended by Ashish Kumar Ashok, a student in computing at Kansas State University, describes machine learning methods to determine how the final outcome of an Article for Deletion (AfD) discussion is affected by the editing history of the article.^[15] The thesis considers features such as the structure of the graph of revisions of an article (based on text changed, added or removed), the number of edits of the article, the number of disjoint edits (according to some contiguity definition), as well as properties of the corresponding AfD, such as the number of !votes and the total length of words used by participants in AfD who expressed their preference to keep, merge or delete the article. Different types of classifiers based on the above features are applied to a small sample of 64 AfD discussions from the 1 August 2011 deletion log. The results of the analysis indicate that the performance of the classifiers does not significantly improve by considering any of the above features in addition to the sheer number of !votes, which limits the scope and applicability of the methods explored in this work to predict the outcome of AfD discussions. The author suggests that datasets larger than the sample considered in this study should be obtained in order to assess the validity of these methods.

In brief

• Why did Wikipedia succeed while others failed?: In a presentation on October 11 at the Berkman Center for Internet and Society (“Almost Wikipedia: What Eight Collaborative Encyclopedia Projects Reveal About Mechanisms of Collective Action“, with video), MIT researcher and Wikimedia Foundation advisory board member Benjamin Mako Hill presented preliminary results of his research comparing Wikipedia and seven other Internet encyclopedia projects or proposals that did not take off, based on interviews with the projects’ founders as well as examinations of their archives. The event was summarised for the Nieman Journalism Lab (reprinted in Business Insider), and in the Signpost: “The little online encyclopaedia that could“. Hill later gave a shorter (ca. 12min) talk about the same topic at the “Digital commons” forum (see below): video, slides.
• “Digital Commons” conference: On October 29-30, the “Building Digital Commons” conference took place in Barcelona, organized by Catalan Wikimedians and Wikimedia Research Committee member Mayo Fuster Morell, and supported by the Wikimedia Foundation. The program featured several presentations about Wikipedia research; further online documentation is expected to become available later.
• Placement of categories examined: A paper from two computer science researchers based at the Katholieke Universiteit Leuven examines the order in which categories are placed on a Wikipedia article and reports on connections between a category’s position in this list and “its persistence within the article, age, popularity, size, and descriptiveness”.^[16] The order in which categories are added is not determined by any explicit rule. However, the research found, older, more persistent and more exclusive categories are consistently placed in lower positions. Categories appearing at lower positions also tend to do so across all the articles they contain and they include articles that are more similar to each other in terms of category overlap.
• Visualizing semantic data: A team from the UCSB Department of Computer Science recently presented^[17] WiGiPedia, a tool visualizing rich semantic data about WIkipedia articles, designed to “inform the user of interesting contextual information pertaining to the current article, and to provide a simple way to introduce and/or repair semantic relations between wiki articles”. The tool builds on structured data represented via templates, categories and infoboxes and queried via DBpedia. By supporting collaborative editing of rich semantic data and one-click semantic updates of Wikipedia articles, the tool aims to bridge the gap between Wikipedia and DBpedia. The source code of the tool doesn’t appear to be publicly released.
• Wikipedia literature review: Owen S. Martin posted to arXiv a 28-page Wikipedia literature review towards his Ph.D. in statistics.^[18] About half the paper gives an overview of Wikipedia’s database structure; the remainder reviews about 30 recent papers from the perspective of assessing their quality, trust, semantic extraction, governance, economic implications and epistemological implications.
• Vandalism detection contest: An “Overview of the 2nd International Competition on Wikipedia Vandalism Detection” has been published.^[19]
• Matching Wikipedia articles to Geonames entries: A four-page paper by two researchers from Hokkaido University^[20] explored the problem of “merging Wikipedia’™s Geo-entities and GeoNames” to form a larger geographical database. This is already being done by the YAGO (Yet Another Great Ontology) database, but the paper uses additional data beyond the article name, such as categories and disambiguation pages on Wikipedia, in order to identify further matching pairs missed in YAGO (and in the process found several errors in GeoNames).
• Attempt to examine evolution of key activities in Wikipedia: A paper titled “Governing Complex Social Production in the Internet: The Emergence of a Collective Capability in Wikipedia”^[21] (presented last month at the “Decade in Internet Time” symposium at the Oxford Internet Institute) undertakes “an exploratory theoretical analysis to clarify the structure and mechanisms driving the endogenous change of [Wikipedia]“, using the framework of capability theory to construct six hypotheses such as “the membership in group(s) of contributors that take up governance tasks varies less than in those revolving on content production”. These are then tested empirically by applying a clustering algorithm to monthly snapshots of the English Wikipedia (until 2009) “to identify distinct groupings of contributors at each month”. However, the clustering algorithm leaves out a group of users “that covers all the observed domains of activity” and “despite its relatively small share of overall contributor population … provides the majority of the work”, which leads the authors to dub it “the core editors of Wikipedia”.

References

Vol: 1 • Issue: 3 • September 2011 [archives]

Top female Wikipedians, reverted newbies, link spam, social influence on admin votes, Wikipedians’ weekends, WikiSym previews

With contributions by: Tbayer, Daniel Mietchen, DarTar and Jodi.a.schneider

What the most active female editors contribute

A paper addressing gender imbalance in Wikipedia (“Gender differences in Wikipedia editing”) by Judd Antin and collaborators won the “Best Short Paper” award at WikiSym.^[1] This follows the awarding of “best full paper” to another study on the gender gap^[2] already covered in previous editions of the research newsletter. The study by Antin and collaborators sampled 256,190 users who created a new account on the English Wikipedia between September 2010 and February 2011 and qualitatively coded their contribution by category of wiki work. The results suggest that, whereas in the lower three quartiles by activity level men and women make roughly the same contributions in each category of wiki work, in the top quartile editors behave in a significantly different way. The researchers found that among the top 25% of Wikipedians by activity level:

• only 27% of all revisions are made by women;
• women tend to make larger revisions than men;
• top female editors make significantly larger revisions than men in at least two categories: “adding new content” and “rephrasing existing text”

Effects of reverts on wiki work

Two plots show the changes in an editor’s boldness after being reverted.

Another WikiSym 2011 paper by GroupLens researchers, including Summer of Research fellow Aaron Halfaker (“Don’t bite the newbies: how reverts affect the quantity and quality of Wikipedia work”), reports on the effects of reverts on the quality and quantity of Wikipedia editors, with a specific focus on newbies.^[3] The study uses a number of key metrics to assess the quality of editor contributions (using reverts per revision and Persistent Word Revisions or PWR, to measure the survival across revisions of words added by an editor, other than stop-word) and changes in editor activity (using a controlled activity delta that calculates an editor’s variation of activity across weeks with respect to the week preceding the revert, normalized by the editor’s daily rate of activity). The results point at the same time at the important role of reverts as a learning and quality improvement process but also at their negative effects on new contributors. Below are highlights from this study:

• Compared with their activity prior to a revert, in the first week after a revert reverted editors decrease their activity by 0.1 standard deviations compared with an increase in a control group of non-reverted editors of about 0.3 standard deviations.
• It matters who performs a revert: editors reverted by a registered editor do not recover to the average level of activity for at least one month, whereas editors reverted by anonymous users recover much faster.
• Reverts affect the quality of one’s work: reverted editors are less likely to be reverted in the future (particularly in the week after the revert), whereas the probability of being reverted in the control group keeps growing every week. Reverted editors are also less likely to make important changes to an article after being reverted, compared with the control group. However, the productivity of reverted editors in the following weeks increases more rapidly than non-reverted editors.
• Reverts affect newbies more negatively. Experienced editors are less affected by reverts on their average activity while newbies are significantly less likely to continue editing after a revert than experienced editors.

These results are consistent with the findings by Summer of Research fellows on the effects of community interactions with new Wikipedians.

Further Wikipedia coverage at WikiSym 2011: Social dynamics and global reach

Geographic location of edits for English Wikipedia article 2011 Egyptian revolution

• The two papers on gender gap mentioned above will be presented in a session titled Understanding Wikipedia, along with other original works some of which were already reviewed in the research newsletter, such as a study by researchers from the University of Pennsylvania examining revision deletion in the English Wikipedia (see also the summary posted on AcaWiki).^[4]
• A second research session will be devoted to Wikipedia as a global phenomenon. It will feature two papers focusing on Wikipedia’s coverage of rapidly changing events such as the 2011 Egyptian revolution and the 2011 Tōhoku earthquake and tsunami. The analysis of Wikipedia coverage of the Egyptian revolution, by a team of Italian researchers based in Trento (from the same lab that released the WikiTrip visualization, previously covered in the Signpost), is available as a preprint.^[5]

Link spam research with controversial genesis but useful results

The “Wiki tools and interfaces” session at WikiSym will see the presentation of a paper titled “Autonomous link spam detection in purely collaborative environments“. According to the five authors from the University of Pennsylvania, link spam is currently “an annoying, but non-pervasive issue”, but could become a grave threat to Wikipedia if new spam techniques that were explored by some of them in another paper (see below) become more widespread.

Using the STiki software by one of the authors, which is already widely used as an anti-vandalism tool on the English Wikipedia, the researchers collected mainspace edits adding external links and extracted a corpus of 5,962 link additions classified as either ham or spam, using criteria such as whether the edit had been rolled back (to determine spam), or whether it had been added by a user with rollback rights (to determine ham). From this, the researchers derived numerous features that indicate link spamming behavior, in three areas: On-wiki evidence (including very simple metrics such as the URL’s length – spam links tend to be shorter – or that older and more popular articles are more likely to be targeted), properties of the landing page that the link points to (these were found to be less useful), and classification from third-party sites, including Alexa and Google Safe Browsing. The backlinks data provided by Alexa proved to be most useful for the classifier that the authors went on to construct, and tested in a live implementation in the STiki tool. They conclude that “it is clear this work will benefit the Wikipedia community”.

In another paper, presented earlier this month at CEAS ‘11, five authors from the same university including two of the same researchers examine the possibility of “Link spamming Wikipedia for profit“. They picture spam detection on Wikipedia as a pipelined process, with the MediaWiki spam blacklist as the first stage (currently containing around 17000 regular expressions), recent changes patrollers (often aided by software tools) as the next – often reacting within seconds after an edit, watchlisters as the third (within minutes to days), and finally review by normal readers as the last stage. Based on a spam/ham corpus constructed as in the other paper, this paper contains some further analysis of the characteristics of link spam destinations and spamming accounts, and of the exposure spammed links receive before they are removed (determined by both the link’s lifespan and the popularity of the spammed page). The most sensitive part of the paper then leverages these results to “describe a novel and efficient spam model we estimate can significantly outperform status quo techniques”, e.g. by rapidly adding links to exploit the time lag of Wikipedia’s spam removal process, or targetting popular pages. In a nod to WP:BEANS, the researchers admit that “there is the possibility that we have introduced previously unknown vectors”, but the “Ethical Considerations” section emphasizes that:

“It is in no way this research’s intention to facilitate damage to Wikipedia or any wiki host. The vulnerabilities discussed in this section have been disclosed to Wikipedia’s parent organization, the Wikimedia Foundation (WMF). Further, the WMF was notified regarding the publication schedule of this document and offered technical assistance.”

The authors also point to the implementation of the spam mitigation tool described in the WikiSym article.

However, the paper fails to mention that last year, one of its authors conducted actual, extensive tests of spamming techniques on the English Wikipedia that are very similar to those outlined in the paper. The spam attacks gained the attention of several IT security news websites, and even involved setting up a fake webshop to measure how many Wikipedia readers would have carried out an actual purchase of the penis enlargement pills advertised in the links. The case led to the researcher’s temporary ban as a Wikipedia user, later lifted by the arbitration committee, and informed the research guidelines drafted later that year by the Wikimedia Foundation’s Research Committee. See Signpost coverage: “Large scale vandalism revealed to be ‘study’ by university researcher” (includes a background interview with the researcher).

How social ties influence admin votes

A paper by three researchers from the University of the Philippines Diliman^[6], presented at the International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2011) two months ago, examined statistical relations between the voting behavior in requests for adminships (RfAs) and the on-wiki social contacts of participants. The paper includes a brief review of existing literature (in particular two papers which already studied the relation with existing social networks^[7][8]). Drawing from a January 2008 dump of the English Wikipedia, they analyzed 2,587 elections conducted between 2004 and 2008 (48% of them successful, with 7,231 users voting or running in at least one RfA, and 80% of the final non-neutral votes being supportive), and “1,097,223 instances of communication between 265,155 distinct pairs of users” who had run or voted in an RfA – from user talk page messages, an undirected social graph was generated. Their results concern three areas:

• “Factors that motivate participation”: As a first result, the researchers found that the number of a user’s contacts who already voted in an RfA, and (more strongly) whether the user had been in contact with the candidate, “contribute positively to the probability of a user’s participation in an election. This may be due to the fact that voters are inclined to support candidates with whom they are acquainted with.”
• As “Factors that influence voting” (i.e. the support/oppose decision) the authors considered the numbers of “support” and “oppose” votes that a user’s contacts have already cast when the user votes, and whether the user had been in contact with the candidate before. All yielded regression coefficients with the expected sign (acquaintance with the candidate weighing positively), and the authors conclude that “we can already explain voting behavior by just examining the immediate neighborhood of a voter”, but note that “it is interesting to note that the presence of contacts who have voted negatively weighs more heavily compared with those who voted positively.”
• Finally, the paper examined “Influential voters in the social network”, by calculating various well-known social network metrics for both the “support” and “oppose” camps in each election (“degree, closeness centrality, betweenness centrality, authority, hub, PageRank, clustering coefficient, and eigenvector centrality“, averaged over all voters in each camp, and combined into a weighted difference). Closeness, PageRank, and eigenvector centrality were found to have the largest regression coefficients in predicting the outcome of an RfA, suggesting to the authors “that decisions of influential nodes can affect the outcome of the RfA process. Although it was not studied in this paper, a possible explanation for this result is that influential users may sway other users to vote the same way and this aggregate behavior may have an impact on the result of the election”.

Wikipedians’ weekends in international comparison

A paper titled “Temporal characterization of the requests to Wikipedia” examined how search requests, read accesses and edits on Wikipedia change over time, and relate to those at the entirety of Wikimedia sites (based on squid logs for the whole year of 2009, provided by the Wikimedia Foundation). Among findings are differences between language versions of Wikipedia, such as that the “the number of edits tends to raise in weekends” for the French, Japanese, Dutch and Polish Wikipedia, but not for other languages. Another paper, titled “Circadian patterns of Wikipedia editorial activity: A demographic analysis”^[9], similarly analyzed “34 Wikipedias in different languages [trying] to characterize and find the universalities and differences in temporal activity patterns of editors”, with the underlying data provided by the German Wikimedia chapter from the toolserver. They found that “in contrast to diurnal [daily] pattern, which is universal to a great extent, weekly activity patterns of WPs show remarkable differences. We could, however, identify two main categories, namely ‘weekends’ and ‘working days’ active WPs.”^[10]

In brief

• Gender bias in Wikipedia and Britannica: An article by Joseph Reagle and Lauren Rhue titled “Gender bias in Wikipedia and Britannica” examines gender bias in biographical coverage, comparing the English Wikipedia and the Encyclopedia Britannica.^[11] The study suggests that “Wikipedia provides better coverage and longer articles, and that it typically has more articles on women than Britannica in absolute terms, but we also find that Wikipedia articles on women are more likely to be missing than are articles on men relative to Britannica”. See the accompanying blog post with the full datasets used in this study.
• Wikipedia as a potlatch: Spanish researcher Felipe Ortega compares Wikipedia to the potlatch, a traditional gift-giving ceremony whose participants gain status based on the generosity of their gifting, in this blog post summarizing his new book with Joaquín Rodríguez ( “El Potlatch Digital: Wikipedia y el Triunfo del Procomún y el Conocimiento Compartido” ["The Digital Potlatch: Wikipedia and the Triumph of Commons and Shared Knowledge"], published in Spanish by Ediciones Cátedra.^[12] Drawing from new qualitative research (interviews with editors of the Spanish Wikipedia) as well as existing quantitative research, the book concludes that recognizing the gifts Wikipedians make, through meritocracy and explicit acknowledgement, helps motivate participation.
• 

  

  Bowel obstruction was one of the articles edited by the University of Texas students during the course.

  How medical students edit Wikipedia: A paper published last month by the Kansas Journal of Medicine asked “Are students able and willing to edit Wikipedia to learn components of evidence-based practice?” ^[13] In 2007 and 2008, two groups of senior medical students at the University of Texas Health Science Center at San Antonio participated in an exercise where they were asked “to place succinct summaries of [medical] studies in Wikipedia” (after a four hour introductory course on wikis). In a survey, 91% of them said that the project should be offered again in the next year, and 71% planned to edit Wikipedia again. (The authors caution that this group was self-selected.) The articles were examined two months after their edits, and 46% of the students had their contribution improved in some way, while “the pages edited by 62% of students had additional edits in response to incidental vandalism to the pages, but in no instance was the vandalism done to an edit by a student”.

• Ethnography of wikiculture set free: Joseph Reagle’s 2010 book on the cultural dynamics of Wikipedia, Good Faith Collaboration, is now freely available to read online, having been released under an accommodating Creative Commons licence (CC BY-NC-SA 3.0).^[14]
• Provenance for Wikipedia articles: A (closed access) doctoral dissertation defended at the University of Arizona presents a “domain ontology of provenance for Wikipedia based on the W7 model”, building on the notion of the five Ws. The author applies this ontology to extract provenance for Wikipedia articles and to assess their quality, thereby identifying “several collaboration patterns that are preferable or detrimental for data quality”.^[15]
• Geographies of the World’s Knowledge: as already mentioned in last week’s Signpost, the floatingsheep collective, in collaboration with the Oxford Internet Institute, released a report titled “Geographies of the World’s Knowledge” visualizing the temporal and geographical distribution of Wikipedia articles.^[16] Drawing from roughly 1.5 million articles in a 2010 database download, the report revealed among other findings that more articles had been written about Antarctica (7,800) than any South American or African nation, that the country with the most internet users (China) accounted for barely 1% of articles, that its biographical articles overwhelmingly geolocate to Western Europe and, from the 18th century on, North America, and that vastly more biographies per year were written for the 20th and particularly the 21st century compared to preceding time periods. The report is released under a Creative Commons BY-NC-ND license.
• Wikipedia found to have grown until 2007: A paper by a sociology researcher from the University of York, titled “Measuring the Development of Wikipedia”^[17], explores the development of the number of edits and the number of participants on the English Wikipedia from 2002 to 2007 (curiously asserting that “there is only 6 years data”). As first result, the research “reveals that the number of edits and the total number of participants both increased in Wikipedia from 2002 to 2007″. The paper’s most tangible contribution appears to consist of histograms plotting the number of users with a particular edit count in each of the years 2002 to 2007, which the author finds “are similar with the Pareto distribution in the shape, [and therefore] we assume that the participation situation in Wikipedia is one type of the Pareto Distribution”. A large part of the four page paper (available for $26) is devoted to general explanations of this distribution. It also mentions the need to use a statistical method such as maximum likelihood estimation to confirm the optical impression that the histograms follow the Pareto distribtion, but it remains unclear if the author actually carried this out. Also, despite emphasizing several times the importance of determining the changes in the k parameter over the years (a measure of the inequality associated with the postulated Pareto distribution) – calling it “vital to model the participation situation in Wikipedia” -, the actual values are never given. The abstract promises “an equation to predict future development trend of Wikipedia”, but it remains unclear to this reviewer which equation this refers to.

References