While GLAMCamp NYC finished on Sunday (Signpost coverage), the work initiated there will continue throughout the GLAM community.  Representatives from cultural institutions and Wikimedia chapters, as well as individuals, are working on several projects.  The projects concerning web badges for free culture allies, a metadata standard for use in the mass uploader/data ingestion tool, and the web analytics proposal are in particular seeking contributors and project managers; please comment at the coordination page to signal your interest.

Also available: the collaborative notes from Friday, Saturday, and Sunday, and specifically for discussion of the Ambassadors program, the Point Of Entry project, the data ingestion tool, and the metrics/analytics proposal.

Thanks to the organizers and participants for a productive and illuminating weekend.

-Sumana Harihareswara
Volunteer Development Coordinator, Wikimedia Foundation

Today, several Wikimedians and representatives from galleries, libraries, archives and museums (GLAM institutions) met in New York City to kick off GLAMCampNYC.  New York City’s public Science, Industry, and Business Library is hosting the event.

Liam Wyatt, the Wikimedia Foundation’s Cultural Partnerships Fellow (aka GLAM fellow), introduced two keynoters: Meg Bellinger, discussing open access at Yale, and Maarten Zeinstra, presenting the Europeana public domain calculator.  The conference continues through Sunday.  Participants are discussing and building the GLAM outreach wiki, writing documentation, sharing best practices, and building tools.

Developers at GLAMCamp are developing a data-munging tool, based on pywikipediabot, to aid in mass uploads (more details).  According to Wyatt, the most common requests from GLAM institutions are (1) mass upload of audiovisual media and (2) metrics, “easily exportable statistics based on analytics on a GLAM’s relationship with Wikimedia.”  The data-munging or data ingestion tool will aid in the import of metadata from large sets of files, thus speeding the difficult part of mass uploads.  Attendees will be hacking on it in sprints this weekend, starting 3pm-4:30pm UTC time tomorrow, Saturday the 21st. Join them in person (11am local time), or in #glamwiki on Freenode.

See notes from today’s general talks and discussion and from the discussion of the GLAM Ambassadors program, or follow #glamwiki and #glamcamp on Twitter and Identi.ca.

-Sumana Harihareswara
Volunteer Development Coordinator, Wikimedia Foundation

Uploading and generation of new thumbs will be temporarily disabled on Wikimedia sites while we patch & reboot the server to fix the performance issues we’ve been seeing.

We hope to be done within a couple hours (by 22:00 UTC or so — 3pm PDT), but it could run shorter or longer.

Rough procedure for the curious:

• Take image thumbnailing servers offline
• Disable uploads
• Unmount file server from web servers
• Patch & reboot file server: rebooted – 21:00 UTC
• Remount file server on web servers – 21:09
• Put image thumbnailing servers back online – 21:12
• Re-enable uploads

<- Done 21:18!

With the kernel fix, the file server should now behave better. We’ll then be able to continue our more leisurely migration of thumbnail files to another server, freeing up disk space on the primary box.

Updated 20:20 UTC: Added our hoped-for ETA

Update 20:44 UTC: A side-effect of taking the image server offline broke account creation and some editing which triggers an anti-bot captcha. Have switched to the simple captcha mode which doesn’t use images for now.

Update 20:56 UTC: Just noting that this affects <math> and <timeline> rendering as well. You may see some math rendering errors until we’ve completed; sorry!

Update 21:12 UTC: File server is back online and uploads are re-enabled. So far so good!

We’ve been seeing some general slowdowns in our image and media file serving recently, including some instances in the last couple days where the sites as a whole have been affected to the point of extreme slowness or temporary inaccessibility.

Domas believes this is related to this reported problem with NFS performance when ZFS snapshots are active. We’ve had some luck so far with it improving after dropping older snapshots (possibly along with restarting NFS and temporarily disabling the image scaler servers to give it a little breathing room to reset).

We’ve been planning for some time to redo the way we access our media files internally which can help reduce the impact on the rest of the site when load problems on the file servers occur, but we might also be able to spread out the load among multiple servers to improve things even more.

Updates will come as we get things back on track…

Update 2009-07-15: We’re temporarily shutting off uploads while we apply the ZFS fix patch and reboot the main file server. You may see some missing images or funky error messages for a little bit, but the sites should otherwise continue working normally until the file server is back up.

Update 2: Server is patched and uploads are back online. This should resolve our performance problems while we continue rearranging the upload servers to be more future-proof.