I am sure that many folks noticed that on the morning of 2009-06-26, techblog.wikimedia.org and blog.wikimedia.org went down. It turns out that some of the parts of our WordPress installations were compromised. I do not want to get in to a direct show and tell of what they did, but hopefully we have hardened the installation to the point that it will not occur again.
This is why the blogs exist on their own server, so when things like this happen we can minimize the impact. The blogs are both up and running now, along with the other services that were affected. All but techblog was back online before Friday was over, techblog lagged behind until today. (As techblog was the point of exploit, we got everything else back up first.) Other affected services were the Open Conference Systems site for Wikimania 2009, as well as our survey software. Both of those were back online ASAP after the incident and the rest followed after.
Of course, it was hard to get this information out to folks when the blogs were down! It goes to show how easily using the blogs to get info out has been, since without it we had to scramble to get the information out of other channels.
Thanks to everyone who assisted in the restoration, and also thanks to everyone for their patience while the system was fixed.