Wikimedia sites to move to primary data center in Ashburn, Virginia

(Update on January 22nd, 2013, 20:00 (UTC): Our Operations team considers the migration to be over. Major disruption is no longer expected.)

Close-up on Wikimedia Foundation Servers

All Wikimedia sites, including Wikipedia, may encounter temporary interruptions on January 22–24, as they transition to servers in a new data center in Ashburn, Virginia (see more photos).

Next week, the Wikimedia Foundation will transition its main technical operations to a new data center in Ashburn, Virginia, USA. This is intended to improve the technical performance and reliability of all Wikimedia sites, including Wikipedia.

Engineering teams have been preparing for the migration to minimize inconvenience to our users, but major service disruption is still expected during the transition. Our sites will be in read-only mode for some time, and may be intermittently inaccessible. Users are advised to be patient during those interruptions, and share information in case of continued outage or loss of functionality.

The current target windows for the migration are January 22nd, 23rd and 24th, 2013, from 17:00 to 01:00 UTC (see other timezones on timeanddate.com).

Wikimedia sites have been hosted in our main data center in Tampa, Florida, since 2004; before that, the couple of servers powering Wikipedia were in San Diego, California. Ashburn is the third and newest primary data center to host Wikimedia sites.

A major reason for choosing Tampa, Florida as the location of the primary data center in 2004 was its proximity to founder Jimmy Wales’ home, at a time when he was much more involved in the technical operations of the site. In 2009, the Wikimedia Foundation’s Technical Operations team started to look for other locations with better network connectivity and more clement weather. Located in the Washington, D.C. metropolitan area, Ashburn offers faster and more reliable connectivity than Tampa, and usually fewer hurricanes.

The Operations team started to plan and prepare for the Virginia data center in Summer 2010. The actual build-out and racking of servers at the colocation facility started in February 2011, and was followed by a long period of hardware, system and software configuration. Traffic started to be served to users from the Ashburn data center in November 2011, in the form of CSS and JavaScript assets (served from “bits.wikimedia.org“).

We reached a major milestone in February 2012, when caching servers were set up to handle read-only requests for Wikipedia and Wikimedia content, which represent most of the traffic to Wikipedia and its sister sites. In April 2012, the Ashburn data center also started to serve media files (from “upload.wikimedia.org“).

Cacheable requests represent about 90 percent of our traffic, leaving 10 percent that requires interaction with our web (Apache) and database (MySQL) servers, which are still being hosted in Tampa. Until now, every edit made to a Wikipedia page has been handled by the servers in Tampa. This dependency on our Tampa data center was responsible for the site outage in August 2012, when a fiber cut severed the connection between our two locations.

Starting next week, the new servers in Ashburn will take on that role as well, and all our sites will be able to function fully without relying on the servers in Florida. The legacy data center in Tampa will continue to be maintained, and will serve as a secondary “hot failover” data center: servers will be in standby mode to take over, should the primary site experience an outage. Server configuration and data will be synchronized between the two locations to ensure a transition as smooth as possible in case of technical difficulties in Ashburn.

Besides just installing newer hardware, setting up the data center in Ashburn has also been an opportunity for architecture overhauls, like incremental improvements of the text storage system, and the move to an entirely new media storage system to keep up with the growth of the content generated and curated by our contributors.

Wikimedia’s technical infrastructure aims to be as open and collaborative as the sites it powers. Most of the configuration of our servers is publicly accessible, and the Wikimedia Labs initiative allows contributors to test and submit improvements to the sites’ configuration files.

The Wikimedia Foundation currently operates a total of about 885 servers, and serves about 20 billion page views a month, on a non-profit budget that relies almost entirely on donations from readers.

Guillaume Paumier
Technical Communications Manager

 

Categories: Operations
Tags: , ,
16 Show

16 Comments on Wikimedia sites to move to primary data center in Ashburn, Virginia

Chris Robinson 8 months

I’m curious how the transition to the Ashburn facility has been? Any connectivity or latency issues yet? Storm season is upon us again, so it should be interesting to see if the new data center holds up as expected.

Chris

Roan Kattouw 2 years

@12: Latency should decrease for some requests, and stay the same for most. Prior to this migration, most traffic to Tampa was already routed via Ashburn, and cached pages (most anonymous page views) were already being served out of Ashburn. Most places have a lower latency to Ashburn than to Tampa.

Regarding your donation question, these servers were paid for (a few years ago, in fact; the Ashburn data center has been around for a while, but until yesterday it wasn’t able to run the site by itself with (almost) no help from Tampa) out of WMF’s own budget, which in turn is overwhelmingly (meaning much nore than 50% but not quite 100%; I don’t know the exact percentage) funded by individual donations.

Nemo 2 years

Lelala, as far as I know power saving has not been considered. It’s kind of assumed to be unnecessary because the number of servers used by the WMF is almost negligible relative to other websites with a similar amount of visits.

Sameboat 2 years

When uploading new version of image, the source file and the description page preview still retain the older version instead of the updated. Guess I have to be bit more patient for few more days?

Hurricanefan24 2 years

How will server latency be affected by the change to the VA server? i.e., will the speed and reliability of the connection be better than that to of the FL server?

(P.S….was this server funded by WMF donations?)

GEY Pierre 2 years

why always in english………..I’m French you see and I speak,read,only a few little your langage………………it’s for me very difficult follow you……….!

Jessica 2 years

I’m actually kinda excited about this- something REALLY awesome is being moved to my hometown.

Lelala 2 years

Hmmm… how much is the impact on energy consumption cost? I could imagine a lower|higher position for cooling/energy costs in the balance sheet?

Optimist on the run 2 years

Are there any legal differences between Virginia and Florida state law that will affect (e.g.) copyright issues on Commons, or child protection, etc.?

Brent W. Moll 2 years

Congratulations on the new Data Center and I hope the transition to it is smooth and very uneventful.

I am curious about your overall system design; what type of servers, Operating Systems, and other technical details which you could share.

Guglhupfner 2 years

When this is all done, wouldn’t it be a good idea to provide up-to-date information on http://meta.wikimedia.org/wiki/Wikimedia_servers for the technically interested?

Adam Hyland 2 years

Jonathan,

One of the Ops folks explained it in a 2011 Wikimania presentation (slides here: http://commons.wikimedia.org/wiki/File:Wikimedia_Operations_Overview.pdf). VA was chosen because they get good peering and traffic arrangements, for the reasons Brambletonian mentioned.

Toggi3 2 years

This is true many places have good connectivity, but not so many have Ashburn’s degree of connectivity. In-fact, that probably doesn’t exist to the degree it does in Ashburn, anywhere else in the United States, in terms of connectivity with the continental US, much more-so the eastern seaboard. Obviously international connectivity is a different matter. Ashburn has other good things going for it though and the datacenter real estate is cheap for what it is. My opinion is not completely unbiased though, given I work in Ashburn. :)

Brambletonian 2 years

Ashburn is the hub of Internet traffic – it was the former home of UUNET, the greatest company ever! Why would they go anywhere else?

Jonathan Stewart 2 years

This sounds like a good upgrade for Wikimedia. I was wondering why you chose Ashburn, VA–there are quite a few places with good connectivity and a lack of hurricanes. Care to elaborate?

Leave a Reply

Your email address will not be published. Required fields are marked *