Database errors on most Wikipedias
Posted by Roan Kattouw on August 7, 2010
At 10:57 UTC, the master database server for s3 (the cluster that holds most of our wikis) had a full disk and stopped writing. For this reason it was no longer possible to edit these wikis. The larger wikis live on separate clusters and were not affected.
After switching to another master database, all wikis are back up and editable as of 12:02 UTC. A few edits that were made during the incident may have been lost.
Heh. That’s a new one :).
Hope you get back up soon!
working again now
Don’t we do capacity planning? And by “we” I mean “you”
I think you should migrate the other large wikipedia (spanish wikipedia) from s3…
There are report on sr.wp that pages edited around this time got wrong text on them.
E.g.
http://sr.wikipedia.org/w/index.php?title=%D0%97%D0%B0%D0%B3%D0%B0%D1%92%D0%B5%D1%9A%D0%B5_%D0%B2%D0%B0%D0%B7%D0%B4%D1%83%D1%85%D0%B0&oldid=3540110
This is a text of another article (not what user submitted). Also note that on the history page the byte count is wrong.
r.
I think you also need to consider migrating the English Wikinews off there too. I notice en.ikipedia and en.wikiquote aren’t on there, nor is commons. And what could possibly be up with having meta.wikimedia on s3? I think there’s a priority review needed there.
This is very strange. It looks like this revision somehow got linked to the wrong page. This is what I get when diffing this weird revision against the latest revision of [[Законодавна_власт]]:
http://sr.wikipedia.org/w/index.php?title=%D0%97%D0%B0%D0%BA%D0%BE%D0%BD%D0%BE%D0%B4%D0%B0%D0%B2%D0%BD%D0%B0_%D0%B2%D0%BB%D0%B0%D1%81%D1%82&action=historysubmit&diff=3540110&oldid=2898320
Mind filing a Bugzilla bug about this?
There seems to be the misconception out there that a wiki being on s3 somehow means it has a lower “priority” or that s3 somehow is structurally less stable than the other DB clusters. None of these things are true. If I wrote anything suggesting otherwise, I apologize for not being clearer.
About 97% of all wikis are on s3. Only wikis that are very large (where “large” is strictly in terms of the amount of data in its database and the volume of queries sent to it) are on other clusters (s1, s2, s4-s6) which contain one or a few large wikis.
The way wikis are distributed over these 6 clusters also doesn’t indicate any kind of priority. That would suggest that some wikis are deliberately made more stable than others, which is not true.
Moreover, the s3 cluster is not structurally less stable than the other clusters, it just happened to be the cluster whose master’s disk filled up yesterday. It could’ve happened to any other cluster, although in that case only a few (large) wikis would’ve been affected.
god bless wiki :)
no idea Y it is laking on Storage :(
tie-up of Google and WikiMedia might solve such problem ! :D
I suggest using http://en.wikipedia.org/wiki/Nagios for disk space monitoring – you can even make it send sms alerts BEFORE the disk gets filled, and it works ! :)
We do use Nagios, but it looks like somehow no one noticed the almost-out-of-space alert.
The deletionists are right! Wikipedia *is* running out of space, it’s just not the English WP which is! =O
(I actually can’t believe no one else has said anything to this effect yet)
This type of problem can actually be anticipated since space disk can be monitored all the time. I hope wikipedia would have funds to upgrade its servers and data storage facility.
Thank’s for you’re support.
Thanks for letting us all know.
Hey,
Is wikitravel.org also down? I am unable to access the websites from a long time.
Wikitravel is not hosted by the Wikimedia Foundation. Any server issues on either site are not connected to the other.