Lightning Strikes Amazon's Cloud (Really)

The Register has details on a recent EC2 outage that is being blamed on a lightning strike that zapped a power distribution unit of the data center. The interruption only lasted around 6 hours, but the irony should last much longer. "While Amazon was correcting the problem, it told customers they had the option of launching new server instances to replace those that went down. But customers were also able to wait for their original instances to come back up after power was restored to the hardware in question."

Comments

Irony?

[Anonymous Coward]

Isn't cloud computing supposed to tackle such instances?

Re:Irony?

[evanbd]

Perhaps that's the problem. I think lightning rods are supposed to be more coppery than irony.

God here...

[Deus.1.01]

Is the message clear?

-RMS

Inconcievable!

[binaryspiral]

While everyone is talking up the cloud and how resilient it is... this is just yet another example to never put all your eggs in one basket. If your service is so damn important that it can't go down - have it hosted in two places.

Notice, Amazon.com didn't go down... :)

Re:Inconcievable!

[nine-times]

Well it does seem like it was pretty resilient:

While Amazon was correcting the problem, it told customers they had the option of launching new server instances to replace those that went down.

So basically a set of servers went down, and it took down the particular instances running on those servers. Customers were still able to take the same exact image and start new instances-- it sounds like immediately. Now sure, it'd be nice if they worked out some kind of automatic clustering and failover to take care of this sort of thing for you, but when my server goes down with my dedicated host, I don't have the option to start up a new host immediately with the same exact configuration.

Do any of you know how they survived?

[mr_stinky_britches]

Do any of you know how an instance could survive a power outage? Surely every operation is written out to disk before it's performed..so how did they design it?

Re:Do any of you know how they survived?

[KahabutDieDrake]

You've never actually worked with enterprise class gear have you? It's standard for most of the servers and all of the data storage to have capacitance/battery backups for just such an emergency.

Typically, the raid controller will have enough on board capacity to clear it's write cache before losing power entirely. While the drive array will be connected to a decent UPS that can hold for at least a few minutes. Meanwhile, the server itself will also likely be connected to the same UPS, or a different one.

The real question at hand is, were the UPS between the power distribution node and the server, or were they on the other side of the distribution node, and therefore worthless in a case like this? I've seen both configurations, but the latter is rarer. Not because of this particular case, but because of efficiency concerns.

If there was a failure of design, it was most likely in the building wiring itself. The building was clearly not properly grounded against lightning strikes, as if it was, the surge would never have hit the internal wiring. It might have kicked the building off the grid for a time, but it should never have reached a power distribution node. Although it's likely the outcome would be similar if not identical.

Re:Do any of you know how they survived?

[RsG]

I'm reading between the lines here (it doesn't actually say this in TFA), but it sounds like this was a direct hit. Not an outage, which is a different beast.

A UPS is about as useful in this instance as antibiotics against a virus - it's a solution to a different problem. Surge protectors don't help much either, not unless the strike was a fairly mild and/or remote one. You could switch over to a disconnected UPS system every time there's a thunderstorm on the horizon, but that seems needlessly complicated and expensive.

That being said, the GP referred to an outage, so you've quite correctly answered his question; it's just the wrong question to ask in this instance. And of course I could be misreading (or Amazon could be misrepresenting) the exact nature of the failure - if it were a regular outage, none of the above would apply.

Re:Do any of you know how they survived?

[sirsnork]

RAID Controllers have batteries so they can remember whats in the cache (for about 48hours), not so they can write that data out to disks befoer they power off. When power is returned and thr disks come back up the cache is flushed before any other action, thereby keeping the array in one piece

Re:What irony?

[mail2345]

I think the poster means popular irony, not irony as it actually means. Popular irony is like getting a fly in your white wine. Regular irony is not wearing your tin foil hat on the one day someone actually does beam thoughts into your brain.

Re:What irony?

[ZorbaTHut]

In Soviet Russia, clouds get hit by lightning?

Yeah, it's sorta weak, but that's what they were going for.

Re:What irony?

[Anonymous Coward]

Regular irony is not wearing your tin foil hat on the one day someone actually does beam thoughts into your brain.

Nope. You've still got it wrong... That's still Morissette irony.

Re:Lightning once striked our office building.

[xrayspx]

I'm thinking critically because Amazon, EMC, VMWare, etc bill The Cloud as a mystical place where you throw your shit and then it's universally available 100%. Nothing bad happens in The Cloud.

So what's the deal with having all copies of these VMs in one datacenter? That's not very The Cloud of them. Maybe they should replicate all of EC2 to GFS. Would The Cloud win then?

Customers being given the option of redeploying their VMs or waiting an unspecified period of time until The Cloud is back online isn't The Cloud we were promised.

/cloud

Re:What irony?

[quanticle]

Perhaps they were referring to the irony of Amazon's EC2 being affected by one of the very natural disasters it advertises protection against.

Its rather like an "unsinkable" vessel going down on her maiden voyage.

Re:What irony?

[Anonymous Coward]

The real irony here is that tinfoil hats are actually required in order to beam thoughts into your head...

Re:Lightning once striked my friends house.

[Kotoku]

In the civilized world, we just call those "walk through holes" doors.

Re:Lightning once striked our office building.

[Achromatic1978]

No, they don't. You're either being disingenuous, or idiotic.

"Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from failure scenarios."

"you can protect your applications from failure of a single location"

Re:Lightning once striked our office building.

[sumdumass]

I have to wonder if those who are critical of Amazon here have ever experienced a direct lightning strike? I doubt it.

Just so people know, this can be a real bitch.

I took a direct lightning strike at one site I work with that entered the corner of the building, traveled down the inside wall leaving a scorch mark on two levels and into the basement where all the servers and switches were located. The lightening then traveled through the electrical service main lines to an encased transformer located in the parking lot next door causing it to explode with enough force that is shattered the windows of the bank building next door and a door panel was found on a rood about a block away. It appears that one half of the electrical system was grounded properly through a specific ground rod and the other half was tied into the plumbing that ran inches away from the lightning rod grounds. When they purchased the building, they didn't redo all the electrical on the side of the building that wasn't remodeled and that way of grounding was normal.

We lost 3 of the 5 servers instantly and couldn't keep the other two stable. Both switches were down, 20 of the 44 workstations along with the tape backup machine, copiers, and networked printers were completely dead when we got there. The entire building had a lightning/surge protector with battery backup and natural gas generator on the mains so they weren't too concerned over in house specific protections. Only the systems with UPS on them directly survived with the exceptions of the servers which I'm not sure if they died from the lightning strike or from getting soaked by the fire sprinklers that was set off by the strike. (surprisingly, there was no fire).

It took us two days at almost 20 hours a day among 5 people with a lot of borrowing from other sites, about 20 trips to five or six computer stores in the surrounding counties, and a generator to come back on line and be operational again. We even had a make shift phone system in place while waiting on a new Avaya to come in. We did this all before the electric company got the transformer replaced and service back on. Until we replaced the other machines that were thought not to be effected, we experienced all sorts of weird behavior on the network and I'm still not confident with the cabling even though it passed the testing. Of course I didn't run the certification so it might just be me not trusting others.

If you get a direct strike, you might as well count on replacing everything in a production environment. When I say direct strike, I mean evidence it actually hit the building and not something down the road and traveled to the building. It will be easier and cheaper in the long run. Now, I have as part of the catastrophe plan, a means to replace every computer and component on the network at one time just to be safe. If it wasn't for two other sites having the same tape drives, we would have had to wait a week for a replacement to come in and start the data recovery process. Thank god for off-site tape storage.