Slashdot Mirror
Car Hits Utility Pole, Takes Out EC2 Datacenter
1sockchuck writes "An Amazon cloud computing data center lost power Tuesday when a vehicle struck a nearby utility pole. When utility power was lost, a transfer switch in the data center failed to properly manage the shift to backup power. Amazon said a "small number" of EC2 customers lost service for about an hour, but the downtime followed three power outages last week at data centers supporting EC2 customers. Tuesday's incident is reminiscent of a 2007 outage at a Dallas data center when a truck crash took out a power transformer."
Think of the poor strippers man!
Mod me down, my New Earth Global Warmingist friends!
It seems you didn't RTFM. Only one switch out of many failed, due to it being set up from the factory incorrectly. The rest of the system switched over properly. I would say that is pretty good considering the data center size and number of switches needed for redundancy.
Strips of steel with holes in them? You're kidding, right?
No. It would be 50*15*5 mm steel with a 10mm hole drilled in each end. A bolt goes through each hole into a threaded attachment point.
Now that you mention it I recall that a four inch nail is good for 100A slow blow but thats cylindrical so it conducts nicely. You'd think the rectangular cross section would not conduct quite as well (sharp corners, etc) but maybe it is also tuned for the desired current. A little saw cut half way between the holes would do that.
http://michaelsmith.id.au
All a fuse is is a piece of metal that will melt fairly quickly when a given amount of current is passed through it. Idea being that it heats up and melts before the wires can. So, the bigger the current, the more robust the metal connecting it. A 100A fuse is usually a fairly large strip of steel.
Now I'll admit that just grabbing an approximate size of steel and placing it in as the GP did isn't going to yield a nice precise fuse. It may have been too high a current. However, it'd work for getting things running again and probably provide a modicum of protection in the event of a short.
Just be glad nobody got killed...
Shorting out something in a main power junction could easily have created a fairly nasty fire...
What the hell is the "white phase"? Unless I am missing some newfangled data-center lingo, you are talking about the neutral, which is not a "phase" at all, and could never produce such a fault current when "shorted" to ground since it is already tied to ground at the panel. Am I missing something?
You have three actives (red, white, dark blue here in .AU), a neutral and an earth. The wikipedia page says different countries have different color codes so maybe that is the confusion.
http://michaelsmith.id.au
I have a friend who is an engineer on one of the projects in the North West Shelf (of Western Australia) a few weeks back he asked "how can they build a rig in the gulf of Mexico for one third of our costs". Two days later One blew up an he got his answer.
Calling someone a "hater" only means you can not rationally rebut their argument.
The hots are black, red, and blue (in that order of prevalence) in the US.
Sure, but a thick cloud with high density can also cover up a lot of important things, like roadways and utility poles.
When was the last time anyone heard of a TV Network going dark for an hour?
Hmm, let me think. How about yesterday?
It's not a matter of I.T. guys not taking the proper steps.
It's a matter of price versus "what if". YOU try to convince a pointy haired boss to spend thousands and thousands of extra dollars on something that "may" happen.
It's often hard enough to convince higher ups to just upgrade old infrastructures that are maxed out on resources. Even if you have proof of issues or near failures. The ONLY time they will happily spend money on upgrades and making your infrastructure more robust is after there has been a critical failure and they actually see their bottom line being hurt and even then if you don't get the approval and dollars fast enough, you run the risk of "What are the chances THAT will happen again?"
More often than not, infrastructure is patches built on patches, one I.T. guy coming in trying to "correct" mistakes of his/her predecessor (who they then realize was working with an underwhelming budget), THEN realizing that it's such a mish mash of bubblegum and duct tape, that any serious fixes would require serious downtime with a complete overhaul. Otherwise you run the risk of the whole thing imploding like a blackhole.
How many I.T. guys seriously have the guts to walk up to their boss after being on the job for only a week and say, "I need 50k and you're network will be going up and down for two weeks as I rebuild and fix it all."
I tried it. I, however, had the ammunition that my company went from 3 people to 40 people in 18 months with another 20 predicted in the next 6 months and that the two box servers were maxed out AND that we were renovating a newly purchased building so we could plan everything from cabling, to telephony to security and future planning for 250+ people.
It also didn't hurt that my boss knows that I.T. is an investment when done right and NOT an expense. Even then with everything on my side it still took 3 months of planning, proving, mapping, designing and quoting from vendor after vendor before approval went through.
Good.. Bad.. I'm the guy with the gun.
Usually, TV stations (that get fined for being off the air for not using their spectrum) and hospitals (which, you know, you can die at if the power goes out depending on your circumstances) have an easier time getting money for redundancy because the bad results are more expensive than if LOLcats is down.