Wikimedia blog

News from inside the Wikimedia Foundation.org

Archive for July, 2009

PMTPA Router Reboot – Scheduled Downtime (Resolved)

Our primary router for the pmtpa cluster had to be rebooted today at 12:00 GMT.  A line card had died and needed replacing, and the

120px-Gnome-face-sick.svg

system required a reboot for it to fully take effect.  Once that finished, CentralNotice was adding a lot of overhead and had to be disabled for our caching cluster to catch up.  Then the overload caused the primary database master for S3 to overload, and we are in the process of switching database masters to another server.

If all went as planned, this would have been a quick 5 minute router reboot and back online.  Unfortunately, things do not always work smoothly, so what would have been 5 minutes has been awhile.  This post will be updated as more details are resolved.

Update: We have switched database masters successfully and all sites and projects should once again be fully functional as of 14:13 GMT.

The abc’s of Usability!

Basket of berries, remixed by Parul V; photos by Dinowx, Siegert, and Yuval Y (CCBYSA)

Hopefully by now you’ve had a chance to try the new skin ‘Vector‘ that the Usability Team has implemented as part of our first release, also known as Acai!  If you haven’t yet, we invite you to check it out by going to Appearance > Skin in your preferences and selecting ‘Vector’.  As you play with the new skin and enhanced toolbar, the Usability Team is looking forward to our next release – codename Babaco – the next alphabetical tropical fruit in what we expect to be a delicious series.

We hope to continue adding features to enhance the editing process – including but not limited to further toolbar enhancements; dialogs for the creation of links, tables, and references; and tools to aid in navigation of article content during editing.  Our research, design, and development depends on feedback from users, community members, and interested parties like you.  We invite you to take a look at our work and if you have any opinions, praise, concerns, or criticism, please let us know here.  We look forward to hearing from you!

Parul Vora, Wikimedia Usability Initiative

SVG for all… with Flash?

For several years, we’ve supported uploading SVG vector images to Wikimedia sites… with the limitation that they would be rendered to static PNG raster images when actually used inline.

This gives our editors great flexibility in editing, customizing, and translating maps and diagrams using cross-platform free tools like Inkscape, but we’re missing out on some of the big potential in SVG — high-quality scaling for zoomed displays and printing, and animation and scripted interactivity.

In large part we can blame Internet Explorer — the most widely used browser has never supported SVG graphics natively, and Adobe isn’t even maintaining their plug-in anymore! With the majority of users cut out, we’ve had little incentive to move forward with new capabilities that would be closed to most visitors.

But that may be changing, thanks to… Flash??

svgweb implements a highly capable SVG renderer in JavaScript and Flash, bringing high-quality, scriptable SVG support to the ~95% of web users who have either Flash or a naitvely SVG-capable browser.

I love to see Flash’s near-ubiquity used for good — implementing support for modern, open web standards on older and less capable browsers.

One of the chief drivers of the project is Google open standards evangelist Brad Neuberg; we had a great talk today along with Trevor on our Usability team and Michael of Metavid/Kaltura/video awesomeness, and we’re all very excited at the possibilities.

We’re going to see if we can whip together some basic integration in time to show at the SVG Open conference in October, starting with a basic zoom-and-pan view for SVG images which can make use of native or emulated SVG support.

Future ideas that have us really excited include:

  • Live previewing of parameterized images at insert time (localized text, highlighted map segments, charts, etc)
  • On-web basic vector image editing? Sometimes you just need to make an adjustment and installing Inkscape is kind of heavyweight.

Pure SVG + Javascript should be able to provide for selecting, moving, adding, and altering objects, which we could then save back to a new version of the file… svgweb’s powerful scripting support should be able to extend this to Internet Explorer users too!

Use of SVG originals inline in article pages is more dependent on file size issues. We have a lot of files that are just plain huge, especially detailed maps, and the SVG version ends up being a lot slower to download and display.

A project which can help with that is Scour, a tool to optimize SVG source by stripping out unneeded verbosity and rearranging style bits to keep size down.

With further work to strip out detail that will never be visible, a filter like this could let us produce output files that are more suitable for on-screen viewing while still scaling up nicely on zoomed displays and printed output.

Scaling Wikipedia Mobile

Some of the things I learned about new projects and scaling issues.

BTW, I think we can settle that Ruby applications don’t have to be slow. Far from it.

[vimeo 5749262 600 450]

Follow me on Twitter @hcatlin or @WikimediaMobile

We’re adding an off site archive for Commons and the XML snapshots

Thanks are due to eBart consulting and User:Milosh for proving a backup server and storage array at their colocation facility in Europe. This server will store archives of our publicly available data of Wikimedia Commons and the XML snapshots.

Everyone knows that this has been long and coming as having an off site location for our data is extremely important for disaster recovery. With this archive in place we’ll have another external archive space for Commons image data to complement the one living at MIT.

Given the 10T’s donated were likely to also store yearly archives of the XML snapshots.

This won’t stop us from continuing to be rigorous about our internal backups for the same data along with keeping all of our users private data within our own data centers. It will simply be another physical space for us to archive our publicly available content.

While this off line mirror will only be used internally we have some other leads about other sponsors who might be able to offer a publicly available mirror. Over the next weeks we’ll be streamlining the off line archiving process and seeding the initial commons upload which currently comes in at just under 4T’s ! Once we make some sense of how best to manage the archiving process we’ll see who else is able to host our data.

Let’s tango!

Reference icon for the enhanced toolbar

The open source movement is not only about software and knowledge base creation. There are active movements in user interface design as well. tango! is one of the neatest projects in design collaborative world, contributing in the creation of open source software such as Open Office and Ubuntu. We, the usability team, also benefit from such open source design projects which allow us to reuse their icons by modifying to meet our needs. For example the icon on the right is the new reference tool icon which can be found in the enhanced toolbar. It is the reuse of Gnome Desktop icons from Wikimedia Commons.

The first set of usability enhancements, new tab layout, enhanced toolbar, and reorganized search page, are now available in MediaWiki projects except for right-to-left language wikis such as Arabic and Hebrew. The support for right-to-left languages should be available in a few weeks, so just hang in there. We welcome you to try out the usability enhancements by going into your preferences and enable ‘Vector’ and the enhanced toolbar from Appearance and Editing menus.

I hope you find the new interface easy to interact. Let us know your feedback in the discussion page of the most recent release page.

Naoko Komura
Program Manager, Usability Initiative

Internal updates: moving files around

We’re now running all Wikimedia sites on our internal deployment branch of MediaWiki, so we’ve got a consistent record of what software we’re running and can more easily test it offline. There were a couple brief glitches during the switchover, but everything seems to be running smoothly now.

Additionally, we’ve started the process of moving thumbnail images from our primary upload file server to another machine to free up space and reduce the repetitive slowness problems we’ve been seeing lately as the ZFS data pool got near full. Even with an updated OS kernel we’ve been seeing it come back. :(

The transition of thumbs should be smooth and fairly non-disruptive in the background.

Update 2009-07-17 00:22 UTC: With the file server getting hyperslugging again, we’ve temporarily re-disabled uploads to speed up the transfers. Should be back online shortly.

Update 03:31 UTC: Uploads re-enabled after disk space rearranged.

Protecting the public domain and sharing our cultural heritage

Last week, the National Portrait Gallery in London, UK sent a threatening letter to a Wikimedia volunteer regarding the upload of public domain paintings to Wikimedia’s media repository, Wikimedia Commons.

The fact that a publicly funded institution sent a threatening letter to a volunteer working to improve a non-profit encyclopedia may strike you as odd. After all, the National Portrait Gallery was founded in 1856, with the stated aim of using portraits “to promote appreciation and understanding of the men and women who have made and are making British history and culture.” [source] It seems obvious that a public benefit organization and a volunteer community promoting free access to education and culture should be allies rather than adversaries.

It seems especially odd if seen in the context of the many successful partnerships between the Wikimedia community and other galleries, libraries, archives and museums. For example, two German photographic archives, the Bundesarchiv and the Deutsche Fotothek, together donated 350,000 copyrighted images under a free content license to Wikimedia Commons, the Wikimedia Foundation’s multimedia repository. These photographic donations were the successful outcome of thoughtful negotiations between Mathias Schindler, a Wikimedia volunteer, and representatives of the archives. (Information about the Bundesarchiv donation ; Information about the Fotothek donation)

Everybody ended up winning. Wikimedia helped the archives by working to identify errors in the descriptions of the donated images, and by linking the subjects of the photographs to accepted metadata standards. Wikipedia has driven new traffic to the archives. And the more than 300 million monthly visitors to Wikipedia have been given free access to amazing photographs of historic value they would otherwise never have seen.

More examples:

  • During the past few months, Wikimedia volunteers have worked with cultural institutions in the United States, the United Kingdom and the Netherlands to take thousands of photographs of paintings and objects for Wikimedia Commons. This project is called “Wikipedia Loves Art.” Again, everybody wins: the museums and galleries gain greater exposure for the images, Wikipedia is better able to serve its audience, and people around the world are able to see cultural treasures they might otherwise never have had access to. (See the English Wikipedia page about the project and the Dutch project portal.)

  • Individual Wikimedia volunteers work with museums and archives to restore digital versions of old images by removing visible marks such as stains and scratches. The work is painstaking and difficult, but the results are terrific: the work is returned to its original glory, with its full informational value restored. Audiences can appreciate it once again. (Restoration work is coordinated through the “Potential restorations” page, and many examples of restoration can be found among Wikimedia’s featured pictures.)

Three Wikimedia volunteers have summarized these opportunities in an open letter: Working with, not against, cultural institutions. On August 6-7, Wikimedia Australia is organizing an event to explore these and other models of partnership with galleries, libraries, archives and museums (GLAM).

Why do Wikimedia volunteers donate their time to painstaking restoration work, the photographing of art, and the negotiation of partnerships with cultural institutions? Because Wikimedia volunteers are dedicated to making information – including images of historic or informational importance – freely available to people around the world. Cultural institutions should not condemn Wikimedia volunteers: they should join forces with them in a shared mission.

We believe there are many wonderful opportunities for Wikimedia to work together with cultural institutions to educate, inform, and enlighten, and to share our cultural heritage. If you would like to get involved in the discussion, we invite you to join the Wikimedia Commons mailing list. Subscribe and introduce yourself – the list is read by many Wikimedia volunteers and by some volunteers associated with Wikimedia chapters as well as some Wikimedia Foundation staff. Alternatively, if there is a chapter in your country, you may want to get in touch with them directly. You can also contact the Wikimedia Foundation. Please feel free to send me your first thoughts at erik(at)wikimedia(dot)org, and I will connect you as appropriate.

The NPG is angry that a Wikimedia volunteer seems to have uploaded to Commons photographs of public domain paintings that are owned by the NPG. Intitially it sent threatening letters to the Wikimedia Foundation, asking us to “destroy all the images”. (Contrary to public claims, these letters did not include an offer for compromise. The NPG is possibly confusing its correspondence with a letter exchange in 2006 with a Wikimedia volunteer, which the user published here.) The NPG’s position seems to be that the user has violated copyright law in posting the images.

Both the NPG and Wikimedia agree that the paintings depicted in these images are in the public domain – many of these portraits are hundreds of years old, all long out of copyright. However, the NPG claims that it holds a copyright to the reproduction of these images (while also controlling access to the physical objects). In other words, the NPG believes that the slavish reproduction of a public domain painting without any added originality conveys a new full copyright to the digital copy, creating the opportunity to monetize this digital copy for many decades. The NPG is therefore effectively asserting full control over these public domain paintings.

The Wikimedia Foundation has no reason to believe that the user in question has violated any applicable law, and we are exploring ways to support the user in the event that NPG follows up on its original threat. We are open to a compromise around the specific images, but our position on the legal status of these images is unlikely to change. Our position is shared by legal scholars and by many in the community of galleries, libraries, archives, and museums. In 2003, Peter Hirtle, 58th president of the Society of American Archivists, wrote:

“The conclusion we must draw is inescapable. Efforts to try to monopolize our holdings and generate revenue by exploiting our physical ownership of public domain works should not succeed. Such efforts make a mockery of the copyright balance between the interests of the copyright creator and the public.” [source]

Some in the international GLAM community have taken the opposite approach, and even gone so far to suggest that GLAM institutions should employ digitial watermarking and other Digital Restrictions Management (DRM) technologies to protect their alleged rights over public domain objects, and to enforce those rights aggressively.

The Wikimedia Foundation sympathizes with cultural institutions’ desire for revenue streams to help them maintain services for their audiences. And yet, if that revenue stream requires an institution to lock up and severely limit access to its educational materials, rather than allowing the materials to be freely available to everyone, that strikes us as counter to those institutions’ educational mission. It is hard to see a plausible argument that excluding public domain content from a free, non-profit encyclopedia serves any public interest whatsoever.

Erik Moeller
Deputy Director, Wikimedia Foundation

Live from the NIH Wikipedia academy

Today Wikimedia Foundation staff and volunteers are in the middle of the first-ever Wikipedia Academy in the United States – at the National Institutes of Health campus in Bethesda, Maryland (see Frank’s post for more info).

You can follow the twitter dialog live.  Volunteer/speaker/photographer Mathias Schindler is also uploading photos, which will make their way to the Wikimedia Commons.

The Academy runs through July 16.  We’ll post more findings and discussions from the event as they come together.

Jay Walsh, Communications

ESAMS Servers not reachable, some EU traffic affected. (Fixed!)

Starting approx. 03:20 GMT, servers in our ESAMS facility began to roll offline one after another.  After some investigation, it appea

rs power is not being supplied to all the servers.  This has resulted in some slow downs for traffic of EU users.

120px-Gnome-face-sick.svg

We have temporarily migrated all traffic to our primary FL datacenter.  Once the servers are

back online in ESAMS, we will be pushing service back to it as well.

Update: The problem has been identified and finally fixed. Traffic has been returned to normal.

The best guess so far is that there was a cooling failure in the datacenter which caused the Sun boxes to shut themselves down.

An update from Leaseweb/Evoswitch is here:  http://noc.leaseweb.com/status.php?i=389