Global Outage (cooling failure and DNS)

Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries.

However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.

We apologize for the inconvenience this has caused.

Update: Unfortunately, for many, this outage seems to have lasted longer than an hour. It appears that many ISPs’ DNS resolvers do not honor the so-called Negative Cache TTL that we send (1 hour), and instead use a longer value. We have circumvented this problem by renaming the affected DNS record to something else.

Update 21:32 UTC: Our SSL gateway, secure.wikimedia.org, was disabled due to overload issues, but is now back up.

Categories: Operations, Outage, Technology
97 Show

97 Comments on Global Outage (cooling failure and DNS)

Coral Blenkhorn 5 years

Hello all,

Did anyone get an email claiming to be from windows live hotmail asking the user to fill in name etc?

I have tried to find on this site somewhere to ask/report if it was genuine….I don’t think it was, so I left it.

kind regards
Coral

I am back on line thank you in Tasmania Australia

name 5 years

ps now it works again LOL

name 5 years

It is down again or still down ?. april 24 – 2010 the netherlands.

http://www.mywowin.com 5 years

How hot did the datacenter get so that he servers shutdown?
Thanks, Dennis

liju 5 years

please send me uptodate informations through my email.this website is my very interesting for me!!!!!!!

Geodesic 5 years

So, maybe we need to pitch in to help financially support the Wikipedia initiative. I know I will when I get a job.

Dennis 5 years

How hot did the datacenter get so that he servers shutdown?
Thanks, Dennis

wplacer 5 years

Really bad luck is what happened. Good luck and thank you for the information.

pytajnik 5 years

In Poland still no wikipedia :((((

Tetraktys 5 years

wiki – portuguese is now working ! thanks!

Supergirl 5 years

CSS is still down for me, using OpenNIC Project’s DNS servers.

Blackcat 5 years

I fixed the problem this way:

edited windows (yes I use windows, have to use it for office reasons :( ) … windowssystem32driversetchosts and added these two lines:

91.198.174.2 bits.wikimedia.org
91.198.174.3 upload.wikimedia.org

That’s because some DNS around are not refreshed … :-)

The HD Blog 5 years

Wikipedia is up and running

Blackcat 5 years

Rome, Italy here.

The situation here is more or less a complete mess.

1. Page load veeeeeery slowly with no style (I suppose style sheets are located elsewere than xx.wikipedia.org?):
2. Monobook doesn’t operate (this is a lesser problem as the whole site doesn’t seem to operate at all…)
3. Loggin’ on is pratically impossible (waiting for logging on is more or less like watching the grass growing…)
4. Tried to flush my DNS cache but the situation doesn’t go any better;
5. The problem is the same both from corporate PC and home’s (two different ISP)
6. No, I haven’t lost my mental facility as allegedly reported by Wikinews but am very disappointed because I was caught by the downservice in the middle of a heavy edit session… is like going in the restroom and forgetting the toilet paper if you know what I mean :-)

Sergio aka the Blackcat on it.wiki

Nick Vague 5 years

I would like to offer a donation in the form of dedicated hosting and bandwidth, so the Wikimedia Foundation can have a third presence to use for load balancing, redundancy or anything you see fit. You can get in touch with me with your requirements.

Boyan Peychev 5 years

Hello! I can make slave dns zones at cloudns.net servers for wikipedia. You can contact me by e-mail.

Needs of wikipedia can be integrated to cloudns.net

Regards,
Boyan

Enbéká 5 years

Well, I can’t see any images on Wikipedias and Commons. And the username’s place and the navigation’s place is now UNDER the articles. What happened?

Sam 5 years

We can’t stay away from Wikipedia for even a single hour. We missed you a lot. Hats off to the Wikipedia team for resolving the problem quickly.

Steve 5 years

Hi.
FYI, I don’t know if this info will help you, but I was affected by the outage. When I surfed to http://downforeveryoneorjustme.com/ to check on http://en.wikipedia.org, it reported that the site was up and it was “just me”. This lasted for a while, until the site came back up.
Could be a useful data point for you.

aero 5 years

OH! at last. Thank you GOD.

Leave a Reply

Your email address will not be published. Required fields are marked *