VMware Causes Second Outage While Recovering From First

jbrodkin writes "VMware's new Cloud Foundry service was online for just two weeks when it suffered its first outage, caused by a power failure. Things got really interesting the next day, when a VMware employee accidentally caused a second, more serious outage while a VMware team was writing up a plan of action to recover from future power loss incidents. An inadvertent press of a key on a keyboard led to 'a full outage of the network infrastructure [that] took out all load balancers, routers, and firewalls... and resulted in a complete external loss of connectivity to Cloud Foundry.' Clearly, human error is still a major factor in cloud networks."

Comments

Re:'An inadvertent press of a key on a keyboard'

[verbatim]

This pretty much describes my entire career.

Don't let it happen again

[stumblingblock]

They just have to remove that key from the keyboard. You know, the one that massively crashes the entire system. Poor judgement to have that key there.

Re:Slashdot summary non sensationalist

[Icegryphon]

Don't go knocking my typewriter
It's Electric, and has wonderful BNC connector
for network access. IBM, you did good.

Re:UR DOING IT WRONG!

[Jeremi]

I would like more elaboration on what "touched the keyboard" means.

It was an extreme case of static discharge. The engineer is lucky to be alive -- when doing cloud computing, thunderstorms are a huge hazard.