Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Global Outage (cooling failure and DNS)

Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries.

However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.

We apologize for the inconvenience this has caused.

Update: Unfortunately, for many, this outage seems to have lasted longer than an hour. It appears that many ISPs’ DNS resolvers do not honor the so-called Negative Cache TTL that we send (1 hour), and instead use a longer value. We have circumvented this problem by renaming the affected DNS record to something else.

Update 21:32 UTC: Our SSL gateway, secure.wikimedia.org, was disabled due to overload issues, but is now back up.

97 Responses to “Global Outage (cooling failure and DNS)”

    1 2 3 5
  1. Geoffrey Hartwell says:

    RSS problem resolved 1850 thanks

  2. Master Astruvis says:

    @Mazen Abdel-Latif
    Use a proxy if it still isn’t working for you, it worked for me on every proxy site I tried.

  3. Geoffrey Hartwell says:

    At 1824, I could not raise the server for the RSS feed.

  4. jayson says:

    Don’t use wiki for papers.

  5. I could load the site via proxy while it was down, but its working ok now.

  6. xeno says:

    The following entries can be added to one’s local “hosts” file:

    208.80.152.2 en.wikipedia.org
    208.80.152.3 upload.wikimedia.org

  7. ab says:

    Bah, now what to do at work?

  8. Josef A says:

    EvoSwitch, I presume? Was it the innovative cold corridor solution that gave or what?

  9. Duh says:

    Duh…use Google cache already…it has the wikipedia pages cached [w/o images of course, but something is better than nothing]!

  10. Gabbianone94 says:

    @1: not only english speaking ones…

  11. Gary says:

    For you needing to do research, just go to: archive.org and copy and paste the wiki url.

  12. Bill Liao says:

    Given Wikipedia is free is it not our right to criticize when something fails.

    The site is sorely missed and we stand that you solve this issue speedily.

    Well done for everything you do to assist humanity.

  13. Ed C says:

    All you slackers waiting until the last minute. If you fail to plan you should plan to fail

  14. IST says:

    Shit happens :-))

    Nice to see you stil alive

  15. E says:

    Damn, I have a paper due in an hour and a half…

  16. Dave says:

    Who’s data center is that? Is it run by Wikimedia?

  17. Luke says:

    How am I to study for my test without wikipedia!??!?

  18. carefulweb says:

    Nice to see you getting back online!

  19. Jimmy says:

    And so we remember Thursday, March 25, 2010 as the day every English speaking student failed their research papers.

Leave a Reply