Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Search restored after leap second bug

At midnight UTC on July 1, Wikimedia’s search cluster stopped working. A “leap second” inserted by the NTP daemon at that time caused Java processes to lock up, including our Lucene search system. The same bug affected many other websites. Our engineers restored service in less than two hours.

Leap seconds are added to our clocks once every few years so that the sun will be directly overhead of the Royal Observatory in Greenwich at precisely 12:00. Some people believe that the desire to keep these two time standards synchronised is anachronistic, and that it would be better to let them drift apart for 600 years and then add a single “leap hour”. I’m sure many computer engineers would breathe a sigh of relief if such a change were implemented.

Tim Starling, Lead Platform Architect

3 Responses to “Search restored after leap second bug”

  1. duplicatebug says:

    Google has a way to avoid this bug, named “leap smear”:
    http://googleblog.blogspot.de/2011/09/time-technology-and-leaping-seconds.html

    But this maybe not help by java.

  2. Apparently a lot of websites experienced this problem. Glad to see you guys got it fixed. I would have been completely lost if it happend to my website.