Slashdot Mirror
Amazon EC2 Failure Post-Mortem
CPE1704TKS tips news that Amazon has provided a post-mortem on why EC2 failed. Quoting:
"At 12:47 AM PDT on April 21st, a network change was performed as part of our normal AWS scaling activities in a single Availability Zone in the US East Region. The configuration change was to upgrade the capacity of the primary network. During the change, one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network. For a portion of the EBS cluster in the affected Availability Zone, this meant that they did not have a functioning primary or secondary network because traffic was purposely shifted away from the primary network and the secondary network couldn't handle the traffic level it was receiving."
But can I get an understandable car analogy here?
No one. No one else remembers AOL.
Dear AWS Customer,
Starting at 12:47AM PDT on April 21st, there was a service disruption (for a period of a few hours up to a few days) for Amazon EC2 and Amazon RDS that primarily involved a subset of the Amazon Elastic Block Store (âoeEBSâ) volumes in a single Availability Zone within our US East Region. You can read our detailed summary of the event here:
http://aws.amazon.com/message/65648
Weâ(TM)ve identified that you had an attached EBS volume or a running RDS database instance in the affected Availability Zone at the time of the disruption. Regardless of whether your resources and application were impacted, we are going to provide a 10 day credit (for the
period 4/18-4/27) equal to 100% of your usage of EBS Volumes, EC2 Instances and RDS database instances that were running in the affected Availability Zone. This credit will be automatically applied to your April bill, and you donâ(TM)t need to do anything to receive it.
You can see your service credit by logging into your AWS Account Activity page after you receive your upcoming billing statement.
Last, but certainly not least, we want to apologize. We know how critical the services we provide are to our customersâ(TM) businesses and we will do everything we can to learn from this event and use it to drive improvement across our services.
Sincerely,
The Amazon Web Services Team
This message was produced and distributed by Amazon Web Services, LLC, 410 Terry Avenue
North, Seattle, Washington 98109-5210
Kriston
I commend Amazon for providing us with this information. Yes, bad things happened, and data is gone forever. Amazon knows what happened and why, and I'm sure they will implement controls to prevent this again. I doubt we'll hear as much from Sony, though.
No. That's the point of the redundant elements and backup of the primary network.
The secondary network they routed traffic to was designed for a different purpose, and never meant to receive traffic from the primary network.
What is an EBS? Is it really just a Xen or VMWare disk image? Which data center corresponds with each availability zone? What are they using for storage iSCSI targets on a SAN?
"At 12:30 PM PDT on April 24, we had finished the volumes that we could recover in this way and had recovered all but 1.04% of the affected volumes. At this point, the team began forensics on the remaining volumes which had suffered machine failure and for which we had not been able to take a snapshot. At 3:00 PM PDT, the team began restoring these. Ultimately, 0.07% of the volumes in the affected Availability Zone could not be restored for customers in a consistent state."
There are some people that if they don't know, you can't tell 'em.
Kudos to Amazon for rapidly explaining, in length, what happened.
Unlike some other company... *cough* Sony *cough*
English is not this
Sony hasn't fixed their issue. Kind of hard to have a post mortem while the solution is still ongoing. There has plenty of extrapolation and bullshit in the information vacuum surrounding the attack though. So when things return to normality it would be in their interest to provide a decent technical overview of what happened, the safeguards that were there before, why they failed and what steps have been made since to improve things.
Have you ever worked in a real environment?
There is ALWAYS a difference between test and production. No matter how many test cases and iterations of changes that you go through, there is always a non-zero percent chance that the change in production will behave differently.
This is why most companies require fall-back procedures for any production change in addition to testing.
It sounds like it may have taken them longer than some might be comfortable to reach the point where they did roll back changes...but I'm sure that this change tested as okay in all of their test cases.
Go and read the entire notice, not just the pathetic snippet a bad submitter used. Makes more sense.
Also, this is a storage network, not an access network. Effectively it's like pulling the SAS cable out of the RAID card while the machines are running.
Trying to become famous by taking photos. Visit my homepage please.