MySpace Down Due To Power Surge

BenelliShooter writes "MySpace.Com - Undergoing Maintenance "hey everyone! there's been a power outage in our data center. we're in the process of fixing it right now, so sit tight. -Tom" That about says it... I suppose we'll see if they had proper back-ups. " Hah. The site says it was supposed to be back up as of ... 7:40 PST PM. Which was something like close to nine hours ago.

Comments

Strange happenings at MySpace

[Tet]

So, I see two possiblities here. Either they're lying about the reason for the downtime, or they're uttlerly inept. According to the most recent figures I've seen, MySpace is the most visited site on the Internet for US surfers, and the 6th most visited site on the net worldwide. Are you seriously telling me that they don't have redundant datacentres?

Hell, with a fairly limited budget, I set up two datacentres in an active/active configuration for the last bank I worked at, and that was only handling a 10 million hits a day. It took a while to get the database replication working right, but once we'd done that, it was all fine, and gave protection against total datacentre failure[1]. MySpace is way larger that we were, and they can certainly afford multiple datacentres to prevent an outage such as this. So why didn't they? As I said, the only explanations I can see are ineptitude, or that they're using this as an excuse to mask some other reason for the outage...

[1] Not that a power failure should ever happen in a datacentre anyway. All of the ones I've used have had multiple power feeds from different suppliers, entering on opposite sides of the building, plus redundant UPSes with diesel generators for when the UPS runs out. If you're still having power outages with that sort of infrastructure in place, then something's seriously wrong. And if you don't have that sort of infrastructure in place, then you've chosen the wrong datacentre.

Re:Strange happenings at MySpace

[Anonymous Coward]

When your site pushes 40+ GBit/s of traffic, and has tens of TB of data in those databases, then you can talk shit. Until then, you should probably shut the hell up.

As a professional in this industry, I can tell you from experience that redundancy at this scale is NOT easy, nor is it inexpensive even for someone as big as MySpace or Fox. Add to that the explosive growth that MySpace is constantly experiencing, and this is much harder than it sounds when you're also trying to keep up with existing growth.

I can additionally tell you that I happen to know what data center they're in. The problem was not a simple power failure. The data center's UPS also failed, which took out the HVAC units. 120 degrees in a data center is not good for hardware.

No, it's not a happy situation, but things like this do happen from time to time.

-AC (ironically, the captcha for this post was 'coo1ing'.)

Re:Strange happenings at MySpace

[mail_stripper]

I work in a datacenter in the building where MySpace keeps their servers, near downtown LA.

I can confirm that they/we had a power outage @ saturday after 6pm , and another one sunday. Needless to say there were a shit-ton of engineers here, some of them bringing in their children, wives, pets(redundant?), etc. I was here for nearly 13 hours after the power outage.

The building provides 'UPS' to all of the tenants, and has *massive* diesel generators as well. Only problem - the failover system failed. This is the second time that this building's "UPS/Generator" system failed at a critical time - the last time was Sept 12th last year during the big ol' blackout ( http://www.cnn.com/2005/US/09/12/la.power.outage/ ).

Nonworking generators *suck*.

I suppose establishing a properly redundant infrastructure is made more difficult with such an exponential rise in popularity. But then again, excuses *suck*.

Re:Time I said this

[bsartist]

That's why I first joined - I have several friends whose bands have pages there.

The irony of it is that MySpace is a great way for non-RIAA bands to promote themselves and network with other bands, finding new places to play, organizing shows, etc. Slashbots continually harp about how bands should be doing that kind of thing, bypassing the RIAA in favor of self-promotion - but when the bands actually start having a little success in doing so, the slashbots all line up to rag on them for it.