MySpace Down Due To Power Surge

BenelliShooter writes "MySpace.Com - Undergoing Maintenance "hey everyone! there's been a power outage in our data center. we're in the process of fixing it right now, so sit tight. -Tom" That about says it... I suppose we'll see if they had proper back-ups. " Hah. The site says it was supposed to be back up as of ... 7:40 PST PM. Which was something like close to nine hours ago.

Comments

Strange happenings at MySpace

[Tet]

So, I see two possiblities here. Either they're lying about the reason for the downtime, or they're uttlerly inept. According to the most recent figures I've seen, MySpace is the most visited site on the Internet for US surfers, and the 6th most visited site on the net worldwide. Are you seriously telling me that they don't have redundant datacentres?

Hell, with a fairly limited budget, I set up two datacentres in an active/active configuration for the last bank I worked at, and that was only handling a 10 million hits a day. It took a while to get the database replication working right, but once we'd done that, it was all fine, and gave protection against total datacentre failure[1]. MySpace is way larger that we were, and they can certainly afford multiple datacentres to prevent an outage such as this. So why didn't they? As I said, the only explanations I can see are ineptitude, or that they're using this as an excuse to mask some other reason for the outage...

[1] Not that a power failure should ever happen in a datacentre anyway. All of the ones I've used have had multiple power feeds from different suppliers, entering on opposite sides of the building, plus redundant UPSes with diesel generators for when the UPS runs out. If you're still having power outages with that sort of infrastructure in place, then something's seriously wrong. And if you don't have that sort of infrastructure in place, then you've chosen the wrong datacentre.

Re:Strange happenings at MySpace

[daveschroeder]

Madison, WI, for one thing (and that's not even a big city): Alliant Energy and MG&E. Also, we can get multiple feeds from different plants from the same supplier.

Re:Strange happenings at MySpace

[Anonymous Coward]

When your site pushes 40+ GBit/s of traffic, and has tens of TB of data in those databases, then you can talk shit. Until then, you should probably shut the hell up.

As a professional in this industry, I can tell you from experience that redundancy at this scale is NOT easy, nor is it inexpensive even for someone as big as MySpace or Fox. Add to that the explosive growth that MySpace is constantly experiencing, and this is much harder than it sounds when you're also trying to keep up with existing growth.

I can additionally tell you that I happen to know what data center they're in. The problem was not a simple power failure. The data center's UPS also failed, which took out the HVAC units. 120 degrees in a data center is not good for hardware.

No, it's not a happy situation, but things like this do happen from time to time.

-AC (ironically, the captcha for this post was 'coo1ing'.)

Re:Strange happenings at MySpace

[mail_stripper]

I work in a datacenter in the building where MySpace keeps their servers, near downtown LA.

I can confirm that they/we had a power outage @ saturday after 6pm , and another one sunday. Needless to say there were a shit-ton of engineers here, some of them bringing in their children, wives, pets(redundant?), etc. I was here for nearly 13 hours after the power outage.

The building provides 'UPS' to all of the tenants, and has *massive* diesel generators as well. Only problem - the failover system failed. This is the second time that this building's "UPS/Generator" system failed at a critical time - the last time was Sept 12th last year during the big ol' blackout ( http://www.cnn.com/2005/US/09/12/la.power.outage/ ).

Nonworking generators *suck*.

I suppose establishing a properly redundant infrastructure is made more difficult with such an exponential rise in popularity. But then again, excuses *suck*.

Re:Strange happenings at MySpace

[TheGax]

"...plus redundant UPSes with diesel generators for when the UPS runs out."

More likely, the UPS is the bridge that runs the place between when utility power drops and until the generators can take the load.

The ginormous UPS at my datacenter can run the whole 23,000 sq feet for about 15 minutes. It takes about 2.5 minutes for the gens to get going.

Re:Strange happenings at MySpace

[astanley218]

Obviously, MySpace decided that this was NOT someone's job. I don't understand the tone of your post anyway. The original poster was only pointing out the tradeoff between a little downtime on your social networking site versus the money, time, and effort needed to create a redundant topology. I realize the issue of lost revenue if your site relies heavily on advertising incoming, and is down. But in the scheme of things a 2 days of downtime over the course of a year is going to lose them less money than they would spend maintaining those redundant datacentres. As the original poster was pointing out - banks and MySpace are not the same. While it may be our job to setup redundant hosting for a bank, it is certainly not MySpace's responsibility to spend their money making sure you can post comments to your buddies 24/7.

just my 2c as an admin working in the field...

Re:Strange happenings at MySpace

[ocelotbob]

I work for a company that rents space in the same building as myspace. The big selling point for the building was that they would handle all that for us; they (supposedly anyways, the last year or so has shown otherwise) have massive battery banks and generators for all the downstairs colo suites. This building also has a number of banks which rely on the same facilities for various operations, like check and payment processing. I've got a feeling that there is going to be some serious head rolling when all this goes down.

All I got

[varmittang]

The Function that you are currently trying to use is disabled and will be back shortly.
We are making some minor changes to this section please bear with us until we can get this back online.
Please do NOT email me about this. Just wait it out. 7/24/2006 -Tom

Its been ongoing since Saturday

[falcon5768]

The site went down sat night, came back up then went down again sunday night. The page saying it would be back up is actually from when it went down Saturday.

Other sites

[Council]

I notice that both Questionable Content and Penny Arcade are also down this morning. Someone suggested to me that they were all at the same datacenter. Is this true?

Even though it could just as easily happen to me, it's still satisfying to say "haha, n00bs".

Although I also note that the datacenter holding my server has on-property generators, which I assumed was pretty standard practice.

Re:Not Uncommon

[m2k1]

They're not exactly running ASP code on IIS servers. They run Cold Fusion code in a third-party .NET thingy. As to be read here: http://slashdot.org/comments.pl?cid=15703724&sid=1 90912&tid=95

Re:get a UPS .. Re:Wow ...

[aplusjimages]

They claim their generators failed to kick in. That actually happened at a Web site company I worked for. One day we had a power outage and our back up generators tried to kick in, but for some reason weren't charged enough, so instead they failed and started smoking. So everyones computers went down. Luckily they store all the sites on several servers in multiple places across the US. So the customers weren't affected. We got to go home though. "Hey McFly, those computers don't work unless you have power!"

It wasn't over use by data mining?

[arthurpaliden]

And here I thought it was because everyone, myself included, was testing the MySpace Data Mining tools released on Freshmeat last night.

Re:Wow...

[kemo_by_the_kilo]

how many people pay for myspace? also myspace is colo'd in/by Navisite , downtown LA they are known for faulty power.... every blackout their UPS blows, and then the redundancy fails. this time it was not just a power issue but a heat issue. it seems the rolling blackouts killed the HVAC and the servers at this colo (myspace's and our company's) by overheating them to reboot.

Re:Time I said this

[bsartist]

That's why I first joined - I have several friends whose bands have pages there.

The irony of it is that MySpace is a great way for non-RIAA bands to promote themselves and network with other bands, finding new places to play, organizing shows, etc. Slashbots continually harp about how bands should be doing that kind of thing, bypassing the RIAA in favor of self-promotion - but when the bands actually start having a little success in doing so, the slashbots all line up to rag on them for it.

Re:Huh?

[warith]

You're so right... 10 years ago (well OK, closer to 9), we were whining about Geocities on Slashdot.

Times have changed radically. ;)