Slashdot Mirror

← Back to Stories (view on slashdot.org)

Power Outage Takes Wikimedia Down

Posted by ryuzaki0 on from the problem-with-non-absolute-power dept.

Baricom writes "Just a few weeks after a major power outage took out well-known blogging service LiveJournal for several hours, almost all of Wikimedia Foundation's services are offline due to a tripped circuit breaker at a different colo. Among other services, Wikimedia runs the well-known Wikipedia open encyclopedia. Coincidentally, the foundation is in the middle of a fundraising drive to pay for new servers. They have established an off-site backup of the fundraising page here until power returns."

8 of 577 comments (clear)

  1. Another indictment of MySql by Anonymous Coward · · Score: 5, Insightful

    Although we use MySQL's transactional InnoDB tables, they can still sometimes be left in an unrecoverable state

    Ya know, I just don't understand why so many projects with such high visibility and requirements for reliability use a toy database like MySQL.

    Someone PLEASE tell me why. Because right now the only thing I can think is that people just don't know how to pronounce "Postgres".

    1. Re:Another indictment of MySql by Anonymous Coward · · Score: 5, Insightful

      No database can guarantee data integrity in the case of a power failure

      This is false. SQL Server 2000 (yeah, I know, instant mod-down) has a transaction log and so does Oracle and I'm sure every other half-decent database. ALL committed transactions are preserved and the data is in a consistent state.

      MySQL does not have this and the developers don't seem to care much about it. This is the problem with open-source in general, if someone is just doing it for fun they aren't going to spend any time on the stuff they don't care about personally.

    2. Re:Another indictment of MySql by Tough+Love · · Score: 4, Insightful

      Since at least one of our MySQL database servers has so far restarted successfully with all InnoDB data intact, perhaps you'd care to reconsider your assessment that MySQL is incapable of doing what it just did?

      But one didn't. That's a much more informative data point.

      --
      When all you have is a hammer, every problem starts to look like a thumb.
  2. Re:mysql bad at disaster recovery? by YU+Nicks+NE+Way · · Score: 4, Insightful

    There's a simple way around this: stick to PostgreSQL, MSSQL, Oracle, DB/2, or some other real database. MySQL doesn't make the grade, precisely because things like this can happen.

  3. Re:Coincidence... ;) by Raul654 · · Score: 3, Insightful

    No no, but with the google deal looming, the tin-foil-hatters are paying close attention to wikipedia, and every little thing gets overly-scrutinized.

    --


    To make laws that man cannot, and will not obey, serves to bring all law into contempt.
    --E.C. Stanton
  4. Re:They should ask for more... by man_ls · · Score: 3, Insightful

    IIRC, that's the Fire Code. The breaker needs to be able to unconditionally kill all power inside the facility. Thus -- it kills the power post-UPS.

  5. Re:They should ask for more... by PornMaster · · Score: 3, Insightful

    Sometimes it costs more to do things wrong, in the long term, than to do them right.

    --
    500GB of disk, 5TB of transfer, $5.95/mo
  6. Re:Distributed Wikipedia? by midom · · Score: 3, Insightful
    Well, distributing a wiki is a task a bit more complex than distributing search index (async!) or seti@home (async). You don't care in async data arrays wether the packet you sent to some node is hour or day old. You care about that in wiki, because every user will be pressing 'edit' button, and data should be consistent everywhere. We are working on distribution.
    • Distributed caches - now majority of hits are served by caches, and some of them are offsite. It was a pilot project for a while and now we're trying to design and build scalable infrastructure for that. But still, lots of edits are served uncached.
    • Distributed file systems - are there any? NFS is single-server system, MS has something, PVFS has no redundancy, GoogleFS is closed and not released, Coda, AFS, all of those just don't work. Right now we're trying to develop MogileFS (the perl-based app-level file storage by LiveJournal) store and sure there are other ideas.
    • Distributed database - there are no proper large database multimaster opensource solutions. MySQL with replication and transactional data store is used. In this event it would be great to have second datacenter nearby with additional DB replicas and gigabit interconnection, but that costs money. And app-level bidirectional replication is in plans for both MySQL and PostgreSQL. And SAN deployment is too costly.
    And yes, MediaWiki code has PostgreSQL support, but migrating from one database to another without proper tests, benchmarks and insurance isn't very mature.