Slashdot Mirror
Power Outage Takes Wikimedia Down
Baricom writes "Just a few weeks after a major power outage took out well-known blogging service LiveJournal for several hours, almost all of Wikimedia Foundation's services are offline due to a tripped circuit breaker at a different colo. Among other services, Wikimedia runs the well-known Wikipedia open encyclopedia. Coincidentally, the foundation is in the middle of a fundraising drive to pay for new servers. They have established an off-site backup of the fundraising page here until power returns."
What happened?
At about 14:15 PST some circuit breakers were tripped in the colocation facility where our servers are housed. Although the facility has a well-stocked generator, this took out power to places inside the facility, including the switch that connects us to the network and all our servers.
What's wrong?
After some minutes, the switch and most of our machines had rebooted. Some of our servers required additional work to get up, and a few may still be sitting there dead but can be worked around.
The sticky point is the database servers, where all the important stuff is. Although we use MySQL's transactional InnoDB tables, they can still sometimes be left in an unrecoverable state. Attempting to bring up the master database and one of the slaves immediately after the downtime showed corruption in parts of the database. We're currently running full backups of the raw data on two other database slave servers prior to attempting recovery on them (recovery alters the data).
If these machines also can't be recovered, we may have to restore from backup and replay log files which could take a while.
If they bought actual servers with dual power supplies and got power from multiple PDUs at their data center, they would be much better off. If this is really because of a tripped breaker, then it's pretty inexcusable, since dual power supplies fed from separate circuits would have prevented it... unlike the LJ outage which was from the power being cut to all circuits.
But if they're going to cobble together some whitebox crap servers, and not change the architecture, they'll be right back to an outage next time it happens.
500GB of disk, 5TB of transfer, $5.95/mo
I was just in freenode joking with Jimbo about this. He said he thought was wondering how long it would be before slashdot ran a story about it (2 hours) and asked people to please stop with the consideracy theories. Meanwhile, the devs are working fairly furiously to get it back up (Kate hasn't slept in 27 hours. Jimbo just declared Feb 22 to be Kate-day) (--A wikipedia admin.)
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
So far one of our database servers has completed a successful recovery (we're working through them all). On a gigabit link it takes something between 90 minutes and 4 hours to rsync from one to another. As soon as we have two database servers working, we'll be restoring service in read only mode. Likely to be that 90 minutes to 4 hours from now as worst case.
I'll post followups to this post later, as we're closer to being fully recovered.
No database can guarantee data integrity in the case of a power failure.
Think again. Techniques to do this have been around for years -- it's called stable storage. You just keep redundant copies of data that's changing, and use a neat and simple procedure to ensure that either they both get updated by a transaction, or the original data can be recovered. Certainly the most recent data might be lost, but there's no reason for the database to be corrupted or even in an inconsistent state.
Karma: Segmentation fault (tried to dereference a null post)
I just love stupid trolls that can't even use Google.
Tsearch2 - full text extension for PostgreSQL
DevX: Implementing Full Text Indexing with PostgreSQL - about Tsearch2.
Tsearch2 is included in the postgresql-contrib package of at least Debian and Novell/SuSE. Is that "out of the box" enough for a clueless MySQL user?
Because the power didn't actually go out?
Because domain names equate to a single IP# (even if that number changes) - a single instance of the object. A URI is just a unique ID across the whole Net, for an object class, which can have single instances. A good URI scheme will take different states of that class into account, like different versions of the object. Domain names, as implemented in DNS, can't give us the one (URI) to many (instances) we obviously need to support scalability and distributed objects.
--
make install -not war
Since at least one of our MySQL database servers has so far restarted successfully with all InnoDB data intact, perhaps you'd care to reconsider your assessment that MySQL is incapable of doing what it just did?
For the rest, we'll see as we get to them and, for any that fail, then look to see whether it was the disk controller or the disk drive lying about having the data written to battery backed up RAM or the disk surface.
Wikipedia hasn't suffered a day of downtime yet for this reason and looks to be down for no more than a few hours this time. A previous incident lasting more than a day was a human or three screwing up and having two copies of the server software writing to the same database files without any locking to prevent conflicting updates. The result of that shouldn't surprise anyone.
You can look things up on answers.com.. They mirror wikimedia, as well as other dictionaries/encyclopedias.
It was Xenu! Great God of the Scientoligists who caused the power outage. He/she/it was angry you didn't pay all your hard earned cash to learn the inner secrets to find out about he/she/it. Read all about it on Wikipedia. Oh, wait you can't!
I find it an interesting coincidence the power outage happened so soon after that the Xenu article was featured. I may be paranoid, but the Scientologists have taken paranoia to a new dimension. They are not above dirty tricks. Karl "Turd Blossom" Rove could learn a thing or two.
Yes. I wrote that cached page and it's now a bit out of date. IF, and it's not certain, local fire regulations permit the use of UPS systems in the racks we're going to be installing them. Decided on that after LiveJournal's unfortunate experience. But don't yet have them.
Nobody's turned Google down. There's been no actual proposals to turn down yet.
But then again, I could be wrong.
Fire code. When someone hits the Big Red Button, all electrical power in the server room must be out. Therefore, UPSs can't be located in server racks (or if they are, you need to go to the effort of wiring them into the BRB).
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
do a search for P2P Web Cache
The colocation facility has diesel generators to protect against the outside power going out. Thanks to the miracle of circuit breakers, power circuits inside the facility shut off (including both circuits feeding our dual-power supply machines).
Chu vi parolas Vikipedion?
The main issue is that we are dealing with north of 1GB of data here
Nice post. I'd just like to add that Wikipedia deals with north of 170 GB, not counting images.
When all you have is a hammer, every problem starts to look like a thumb.
A lot of datacentres don't allow UPSes within customer enclosures, as even if the EPO is triggered they keep supplying power which can be dangerous for fire/rescue crews. I'm aware this wasn't an EPO situation AFAIK, but the rules still apply.
You would know this to if you had read the PostgreSQL documentation.
http://www.neutelligent.com/
Well...Slashdot has fully hit:
Temporary fundraising site: "This account has exceeded it's bandwidth quota and has been temporarily disabled."
http://www.postgresql.org/docs/8.0/interactive/wal .html
And going strong!
The link in the article is broken, here's the proper one:
http://wikimedia.org/fundraising/
Beware: In C++, your friends can see your privates!
Every main database server had corrupt database pages. That is, 3 systems with battery backed up write caching controlles and SCSI drives and 2 SATA systems with write caching SATA controllers, one battery backed up the other not, two different SATA disk drive makers.
Involved:
Obvious speculation involves the controllers not telling the drives not to write buffer or the drives not listening. No point in getting into SCSI or SATA or this disc controller or that controller fights when there's this much variation involved.
Posted on the mailing list wikipedia-l 32 minutes ago:
From: Brion Vibber
Reply-To: wikipedia-l@wikimedia.org
To: Wikipedia-l, Wikimedia Foundation Mailing List, Wikimedia developers
Date: Tue, 22 Feb 2005 04:47:56 -0800
Subject: Re: [Wikipedia-l] Wiki Problems?
Brion Vibber wrote:
> There was some sort of power failure at the colocation facility. We're
> in the process of rebooting and recovering machines.
The power failure was due to circuit breakers being tripped within the colocation facility; some of our servers have redundant power supplies but *both* circuits failed, causing all our machines and the network switch to unceremoniously shut down.
Whether a problem in MySQL, with our server configurations, or with the hardware (or some combination thereof), most of our database servers managed to glitch the data on disk when they went down. (Yes, we use InnoDB tables. This ain't good enough, apparently.)
The good news: one server maintained a good copy, which we've been copying to the others to get things back on track. We're now serving all wikis read-only.
The bad news: that copy was a bit over a day behind synchronization (it was stopped to run maintenance jobs), so in addition to slogging around 170gb of data to each DB server we have to apply the last day's update logs before we can restore read/write service.
I don't know when exactly we'll have everything editable again, but it should be within 12 hours.
Easily. See what those saying that MySQL can't do what MySQL does are promoting.:)
LiveJournal found that it had some disk systems which lied about having committed writes. The have a preliminary tool which copies what it's writing to disk to a networked system and then compares the after power off and recovery state to what the disk system said it could do. Are going to make it available to the community as time allows.
I expect we're going to find the same at Wikipedia. Here's a pretty typical error log, this one from the server which was master database server:
050222 5:11:12 InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 303 1283776146
InnoDB: Doing recovery: scanned up to log sequence number 303 1289018880
InnoDB: Doing recovery: scanned up to log sequence number 303 1294261760
InnoDB: Doing recovery: scanned up to log sequence number 303 1299504640
InnoDB: Doing recovery: scanned up to log sequence number 303 1304747520
InnoDB: Doing recovery: scanned up to log sequence number 303 1309990400
InnoDB: Doing recovery: scanned up to log sequence number 303 1315233280
InnoDB: Doing recovery: scanned up to log sequence number 303 1320476160
InnoDB: Doing recovery: scanned up to log sequence number 303 1325719040
InnoDB: Doing recovery: scanned up to log sequence number 303 1330961920
InnoDB: Doing recovery: scanned up to log sequence number 303 1336204800
InnoDB: Doing recovery: scanned up to log sequence number 303 1341447680
InnoDB: Doing recovery: scanned up to log sequence number 303 1346690560
InnoDB: Doing recovery: scanned up to log sequence number 303 1347688389
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 14 row operations to undo
InnoDB: Trx id counter is 1 935480064
050222 5:11:13 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8617985.
InnoDB: You may have to recover from a backup.
050222 5:12:20 InnoDB: Page dump in ascii and hex (16384 bytes):
Observe that the database engine went back to its last checkpoint, noticed the partial transaction and undid it and was rolling ahead in the write-ahead log when it encountered a database page which failed its checksum test. That failed checksum test is why I think it's a problem with the disk system lying about what was written. You can get that when a database page spans two drives in a stripe set and one has committed the update while the other hasn't.
In more typical situations MySQL simply applies the updates and all is well. I've had a server set up to exceed RAM with swap turned off and get killed every ten minutes for hours and recover every time.
Just to be complete:
You mis-read the comment. (and again, know little about wiki's setup) The point is, there aren't any Microsoft Windows boxes in the cluster. And I don't expect the Wikimedia Foundation to approve the cash outlay for buying windows and mssql licenses for the number of system currently serving as database engines. Plus there's the complexity of those boxes not just being db servers, and the fact that none of the admins are anywhere near the actual hardware -- remote management of windows boxes is not something I would recommend (and not something easily done via ssh.)
MySQL is not inferior in all possible characteristics... MSSQL is a windows only product. I cannot run it on Solaris, OS X, AIX, Tru64, linux, etc. It, thus, loses on that characteristic. Wiki is not a windows shop, so stop wasting your time suggesting a windows product. The cluster is running linux. Bring linux software to the table and we'll talk. Wiki uses mysql because it's free and it's fast. My suggestion would be Oracle, but it's most certainly not cheap or free.
Unfortunately for webhosting the demand for MySQL is higher than for the other available DBMS's, since most available open source software and gratis software that requires a database is going to have been developed originally with MySQL. I would much prefer to be using PostgreSQL for the applications I run with a hosting provider, but the apps I use don't function with it, and the hosting provider (NearlyFreeSpeech.net) doesn't offer anything else, anyway.
I figure once the advantages of the other DBMS systems become more apparent (and enough disaster stories happen to highlight the advantages) the apps will begin to offer and improve support for PostgreSQL and others, and then there will be a demand for them and some hosting providers will begin to offer them. I do understand that PostgreSQL consumes a lot more resources than MySQL, though, so it will not be cheap.
You get what you pay for. How much is reliability worth?
Secession is the right of all sentient beings.
Except it's now been a few years since MySQL incorporated InnoDB, so maybe it's time to move on and rejoice that it's now one of the free database servers with ACID support? This one happens to come with standard replication and fulltext search. Also with a range of other engines to choose. PostgreSQL, last I knew, doesn't have built in replication, fulltext search and alternative storage engines but has it's own particular strengths. In the end, every end user gets to benefit from the competition between excellent tools. Good for us all to be happy about that.
Hi!
InnoDB has used WAL since I wrote it in mid-1990s. To PostgreSQL, WAL came later, around 2000.
Regards,Heikki
Innobase Oy
> > Any real database server, which MySQL is most assuredly not, can guarentee data integrity since the last COMMIT.
> Since at least one of our MySQL database servers has so far restarted successfully with all InnoDB data intact, perhaps you'd care to reconsider your assessment that MySQL is incapable of doing what it just did?
There is a huge difference between being capable of doing something and guaranteeing it. There is a difference between "sometimes, maybe, if you're lucky" and "always". I'm surprised that you can't see it. (Who the hell is moderating this thread, anyway?)