Data Center Power Failures Mount

1sockchuck writes "It was a bad week to be a piece of electrical equipment inside a major data center. There have been five major incidents in the past week in which generator or UPS failures have caused data center power outages that left customers offline. Generators were apparently the culprit in a Rackspace outage in Dallas and a fire at Fisher Plaza in Seattle (which disrupted e-commerce Friday), while UPS units were cited in brief outages at Equinix data centers in Sydney and Paris on Thursday and a fire at 151 Front Street in Toronto early Sunday. Google App Engine also had a lengthy outage Thursday, but it was attributed to a data store failure."

Comments

If only you had listened...

[BillyMays]

I'm guessing that the majority of these were caused by leaks or spilled drinks. If only you guys had listened to me and gotten Zorbeez(tm)[SOAKS UP 10x ITS OWN WEIGHT!].

-B. Mays

Re:"bad week to be a piece of electrical equipment

[Statecraftsman]

Yes, that's clearly Twitter territory.

Damn you Michael Bay!

[StaticEngine]

"A blown transformer appears to be the culprit"

I'd heard the new movie was crude, but I didn't realize how crude it actually was!

Outages

[Solokron]

Outages happen more than that. We have been in several data centers, ThePlanet and The Fortress both have had major outages in the last two years which has affected business.

Re:Outages

[JWSmythe]

    I've had equipment and/or worked in many datacenters over the last decade or so. I've worked with even more clients who have had equipment in other datacenters.

    I've only experienced 3 power related outages that I can think of.

    One was a brownout in that area, which cooked the contactors that switched between grid power and their own DC room.

    One was an accident, where a contractor accidentally shorted out a subpanel, and took out about a row of cabinets. I was there for that one. I saw the flash out of the corner of my eye, and by the time I turned my head, he was just flying into the row of cabinets.

    One was a mistake in the colo, where there was a mislabeled circuit, so they cut power to 1/3 of one of our racks.

    There have been even more outages related to connectivity problems. With one major provider who was just terrible (and is now out of business), they had a fault about once a week or less. Every time we called, they said "there was a train derailment that cut a section of fiber in [arbitrary state], which effected their whole network." It was funny at first, but annoying when we started questioning them about why there was no news about all these train derailments. We had to make up our own excuses for the customers, because we couldn't keep telling them the BS story the provider gave. We were smart about it though, and at least had decent excuses, and the whole staff knew which BS story to give for a particular day. The sad part was, we had a T3, and that was huge at the time.

    At my last job, they wanted a full post-mortum done on any fault. If a customer across the country suffered bad latency or packet loss, it was our job to find out why and "fix" it. The management wouldn't accept that there are 3rd party providers who handle some of the transit. So, we'd call our provider demanding it to be fixed (which they couldn't do), and then call the broken provider (who hung up since we weren't their customer), and then got reamed by the boss because we couldn't fix it. Delay tactics worked best after a while. If you're "investigating" a problem long enough, and hold the phone up to your ear enough, the problem will likely be fixed by those who really can. We'd still log a ticket with our provider, because the boss would eventually call the provider referencing the ticket number, and find out there was still nothing that could be done.

    There's pretty much guaranteed to be a fault of some sort between two points on the Internet every day. All anyone can really do is make sure it isn't with your own equipment. That's something I always did before calling to complain about anything. It's embarrassing to hear "did you reboot your router?" and that turns out to really be the problem.

    The only real solution to this is, redundancy. Not just in one facility, but across multiple facilities. If you spread things out enough, sure an isolated problem will effect some people, but not everyone. You want a service to be reliable, redundant machines in each datacenter is the only way to go. When I was running the network (and everything technical) at one job, a datacenter outage wasn't a concern, it was just a minor annoyance. I filed a trouble ticket, and told them to call me when it was fixed. We'd demand reimbursement on the outage time, and made them handle the difference on our 95th percentile bandwidth charges at the end of the month. I wasn't going to take a hit on the bill just because they had an outage in a city, and my other cities had to take the traffic during the outage. When your bill is measured in multiple Gb/s, you have a little more say in how they handle the billing. :)

Be Redundant!

[drewzhrodague]

Anyone seriously oncerned about their web applications, will have redundant sites, and a way to share the load. Few people pay attention to the fact that DNS requires geographically disparate DNS servers *, such that even in the event of a datacenter fire (or nuclear attack), there will still be an answer for your zone. Couple this with a few smaller server farms in separate places, and there won't be any problems. I went to look it up on wikipedia, but didn't find out where it is required for authoritative DNS servers to be in separate geographic regions. Where did I read this, DNS and BIND?

Re:Be Redundant!

[W3bbo]

The DNS RFCs advise that zone nameservers should be in separate subnets. Specifically RFC 2182 recomends that secondary DNS services be spread around geographically.

Re:Be Redundant!

[Firehed]

In the event of a nuclear attack, you probably have more pressing issues to deal with than your server uptime.

Re:Be Redundant!

[JWSmythe]

    Be nice, people don't read the books nor RFC's any more.

    At the biggest operation I ran, I had redundant servers in multiple cities, and DNS servers in each city. If we lost a city, it was never a big deal, other than the others needing to handle the load. With say 3 cities, a one-city outage only accounted for a 16.6% increase in the other two. Each city was set up to handle >100% of the typical peak day traffic, so it was never a big deal. I don't think we ever suffered a two-city simultaneous failure, even though we simulated them by shutting down a city for a few minutes. Testing days were always my favorite. I loved to prove what we could or couldn't do. I peaked out one provider in a city once. We had the capacity as far as the lines went, but they couldn't handle the bandwidth. It was entertaining when they argued, so I dumped the other two cities to the one in question, and they were begging me to stop. "Oh, so there is a fault. Care to fix it?"

    I could quantify anything (and everything) at that place. I could tell you a month or so in advance what the peak bandwidth would be on a given day, and how many of which class of servers we needed to have operating to handle it. I classed servers by CPU and memory, which in turn gave how many users and how much bandwidth each could do. I only wanted our machines to every peak out at 80%, but sometimes it was fun to run them up through 100%. I set the limits a little low, so we could run at say 105% without a failure.

    Such information let us know if we had a server problem, before we knew we did. I'd notice a server was running 10% low, and that really means that it is going to fail. We'd watch for a little while, and it would. :) We'd power it down, and leave it in the datacenter until we had another scheduled site visit.

Re:Be Redundant!

[ls671]

Best solution for big outfits is to have at least this setup:

1) One party being the main contractor. This party doesn't do ANY hosting per say but only manages the fail-over strategy, doing the relevant testing once in a while.

2) A second party being involved in hosting and managing data centers.

3) A third party, completely independent from party 2, a competitor of 2 is preferable, which also does hosting and manages data centers.

It is the same principle when you bring redundant internet connectivity to a building :

1) Have the fiber from one provider come into the building from, say, the north side of the building.

2) Have a competitor, unrelated business wise, that doesn't use the same upstream providers bring his fibers in from the South side of the building.

Putting all your eggs in the same basket by dealing with only one business entity constitute a less robust solution.

Re:"bad week to be a piece of electrical equipment

[Anonymous Coward]

Indeed, 18465. And we shall get off your lawn as well.

No preventative maintenance?

[Neanderthal Ninny]

My wild guess is they are deferring preventative maintenance on these data centers so we are seeing these major outages now. Fire suppression, UPS, transfer switches, generators, distribution panels, transformers, network gear, server, storage devices and other gear will fail if you don't maintain them properly. As loads increase, the equipment will fail earlier and my guess the people have pushed the limit of this equipment beyond they the lifespan of load rating.

Re:No preventative maintenance?

[ls671]

I would not try to convince him. Just write a memo describing the issues without sounding alarmist. It is up to the boss to evaluate the risks and to take the decision. Once you have written your memo, you are basically covered.

Now could be a good time to write this memo, just remember not to sound alarmist, just describe the possible issues although the risk is slim. You could say that you have been inspired by recent events in big data centers ;-))

As per licensing issues, call your Oracle/MS representatives, they offer special deals for fail-over sites. This will be a good point to mention in your memo (cost).

Downside to consolidation

[Anonymous Coward]

Surprise surprise...there's a downside to consolidation. Hey morons, the internet was invented as a means to ensure redundant communications paths given nuclear warfare. The old central switch (physical switching) was seen as too cumbersome and vulnerable. Now that we have wonderfully redundant communications, and have done away with most of the downsides of physically distributed systems, morons are building logically centralized systems.

NEWSFLASH - Redundant communications and physical virtualization do very little for you if you build a logical mainframe.

Truly distributed systems must be physically AND logically DISTRIBUTED with redundant comms paths in order to gain the full benefits of decentralization. (e.g. Distributed isn't distributed if all your authentication is done at one site or all your traffic must pass through .)

Re:Downside to consolidation

[ls671]

>> "morons are building logically centralized systems"

I have worked with such a moron doing architecture on a big government project ;-)) unbelievable...

His argument was that "The government likes centralized systems" ;-))

Re:Downside to consolidation

[BBCWatcher]

No, I think you have it exactly backwards, or at least you're missing an important nuance. It's really, really expensive to duplicate everything across two (or more) data centers. And it's full scope increase in IT costs: most or all cost categories increase. We're talking more than double the costs, in round numbers. Beyond the cost, it's very hard technically to recover hundreds or thousands of servers simultaneously or even near-simultaneously, because you are typically trying to recover not hundreds or thousands of atomistic, independent servers but all the moment-in-time state and functional dependencies among servers. Very, very difficult, which also means hugely expensive and prone to error. Unfortunately, service interruptions are also extremely expensive. What to do?

You could just buy a pair of mainframes, one at site one and the other (configured with reserve capacity, which is lower cost) at site two. (More only if you need the capacity. Then they just operate like a single machine.) That all works really, really well. As in, credit card holders would have no clue that site #1 just burned to the ground -- the credit cards still keep working. That particular form of consolidation makes disaster recovery a relative breeze. DR is just thoroughly baked into the DNA of such equipment, and the very computing model itself supports rapid recovery. (Down to zero interruption effectively and zero data loss, if that's what you need. Or, in DR lingo, RPO and RTO of zero.)

The critical nuance here is if you only consolidate sites, which a lot of businesses have done, you're reducing business resiliency, ceteris paribus. Yes indeed, if you merely forklift your hundreds or thousands of servers into a smaller number of data centers and do basically nothing to consolidate applications, databases, operating system images, etc., onto better DR-protected assets, then disaster recovery will be much tougher and much more expensive. Site-wide disasters will be more disastrous. The game-changer (otherwise known as re-learning time-tested lessons :-)) is if you untangle the mess and do real consolidation onto a much smaller number of robust, well-protected servers with some decent DR investments and realistic rehearsals. That'd be mainframes and mainframe IT discipline, basically, or at least something that resembles mainframes (if such a thing exists).

Re:XO Communications Genesis Hosting

[socsoc]

Sure, I heard that Genesis Hosting suffered an outage from their ISP; XO Communications in Chicago.

Former critical power field engineer here...

[asackett]

... saying that it's time to reconsider cost cutting measures. In 15 years in the field I never saw a well designed and well maintained critical power system drop its load. I saw many poorly designed and/or poorly maintained systems drop loads, even catching fire in the process. One such fire in a poorly designed and poorly maintained system took the entire building with it, data center and all. The fire suppression system in that one was never upgraded to meet the needs of the "repurposed space" which was originally a light industrial/office space.

Even worse...

[Anonymous Coward]

I'm one of the guys that services the security system in Fisher Plaza. The damn sprinklers killed half my panels near the scene. Turns out they use gas suppression methods in the data centers, not so much in the utility closets. And the city of Seattle REQUIRES sprinklers throughout the building, even right over the precious, precious servers. In defense of the staff there however, they do not keep them all charged 24/7. Other then that, I have no more info, as they're pretty locked down.

Re:"bad week to be a piece of electrical equipment

[Anonymous Coward]

Because out of all of the data centers in the world, there were problems at five? Riiiiight. Good reporting, Slashdot.

Can I sign up for broken water main notices here, too, or do I need to go to another website?

100+ million people daily are "serviced" by these 5 data centers.

Company's such as authorize.net where COMPLETELY unavailable for payments to hundred of thousands of webmasters sites (ya know the people who make money)

If you don't think this is serious news then you are still living at home.

Ya that's what I thought.

So the real question is...

[Dirtside]

...what is the normal (historical) rate of data center power failures, and how does the recent spate compare? Five in a week sounds severe, but what's the normal worldwide average? I can imagine that with thousands of data centers around the globe, there's likely a serious failure occurring somewhere in the world once every couple of days.

Been through too many of these.

[Velox_SwiftFox]

"Major" data center or not, the one your company employing you at the time is using is the important one.
In my experiences, data center backups fail about a third the time power is interupted somewhere.

Servers in an Oakland California center were the victim of the loss of one of three power phases, while the monitoring that would have switched over to the diesel generators was looking at the power level of other phases. UPS systems ran out of power. An extra level of redundancy in the form of rack mount UPSes allowed servers to shut down properly despite the data center's loss of routing.

Data center #2 was the victim of a simple power outage and immediate failure of the main data center UPS system. According to a security guard I talked to, "it exploded". The diesel backup never had a chance to start.

Then the doubly-sourced Power Distribution Unit supplying a rack at a third ISP failed in a way that turned off both sources supplying the servers.

Hint: Add an extra level of UPS redundancy and safe shutdown software daemons, at least. Multiple data centers if you need more nines.

Re:"bad week to be a piece of electrical equipment

[afidel]

authorize.net are apparently complete idiots, if they are that large and all their equipment is in one datacenter then that's bordering on insane. Heck, my little company of under 1k employees has two facilities. Anyone who's should be running a site with 100k+ customers knows better.

Power Fail Often

[blantonl]

Frankly, if data centers are going to proclaim their redundancy, they should test by power failing the entire data center once every two weeks at a minimum. A data center that goes down twice in a month would get ahead of any issue pretty fast. Lessons learned from the staff and the management are very valuable.

The marketing messaging:

"We power fail our data center every two weeks to ensure our backups work..."

Sound scary? Just think about the data center that has never been through this process. at that point, the wet paper bag you tried to market your way out of dried rather quickly and you are now faced with the prospect of slapping around inside of a zip-lock.

Rackspace in Dallas

[Thundersnatch]

We're a Rackspace customer in their DFW datacenter. This is the third power-related outage they've had in the last two years at that supposedly world-class facility.

The first wasn't really their fault: truck driver with health condition runs into their transformers. Generators kick in, but chillers don't re-start quickly enough. Temps skyrocket in minutes, emergency shutdowns. Maybe the transformes should have had some $50 concrete pylons surrounding them?

The second outage was the result of a botched generator upgrade.

This latest outage was the result of a botched UPS maintenance.

None of the outages was long enough to trigger our failover policy to our DR site, but our customers definitely noticed.

While their messaging has been very open and honest about the problems, and the SLA credits have been immediate, we pay them nearly $20K per month. Nedless to say, we are shopping, and looking into a "multiple cheap colos" architecture instead of "Tier-1 managed hosting". Nothing beats geographic redundancy.

Re:Rackspace in Dallas

[zonky]

That isn't quite right, re: their 2007 outage.

It wasn't a power issue as such, but the way their chillers reponded to two quick power fluctuations in succession:

This is what they said:

Without notifying us, the utility providers cut power, and at that exact moment we were 15 minutes into cycling up the data centerâ(TM)s chillers. Our back up generators kicked in instantaneously, but the transfer to backup power triggered the chillers to stop cycling and then to begin cycling back up againâ"a process that would take on average 30 minutes. Those additional 30 minutes without chillers meant temperatures would rise to levels that could irreparably damage customersâ(TM) servers and devices. We made the decision to gradually pull servers offline before that would happen. And I know we made the right decision, even if it was a hard one to make.

Sunspots, Anyone?

[Craig Milo Rogers]

All these data centers failed at roughly the same time as the sunspots returned, but that's just a coincidence, right?