Slashdot Mirror

← Back to Stories (view on slashdot.org)

Amazon EC2 Crash Caused Data Loss

Posted by timothy on from the but-that-was-off-site-backup! dept.

Relayman writes "Henry Blodget is reporting that the recent EC2 crash caused permanent data loss. Apparently, the backups that were being made were not sufficient to recover the lost data. Although a small percentage of the total data was lost, any data loss can be bad to a Website operator."

8 of 112 comments (clear)

  1. I am not rightly able to comprehend... by Man+On+Pink+Corner · · Score: 5, Insightful

    ... the confusion of ideas that would lead someone to treat their live web server as their primary/master data repository.

    I guess I'm still stuck in Commodore 64 World, or something..

    1. Re:I am not rightly able to comprehend... by MichaelSmith · · Score: 4, Informative

      It took something pretty catastrophic to bring it down and cause data lass

      Catastrophic would be an earthquake, tsunami and meltdown, in that order. From my reading of the situation amazon stuffed up their own replication mechanism and it recursively replicated the system to fill up the available hardware. Thats just bad design. Its obvious they did no testing under realistic conditions.

      --
      http://michaelsmith.id.au
    2. Re:I am not rightly able to comprehend... by Yvanhoe · · Score: 4, Funny

      That is still better than Amazon's plan actually.

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    3. Re:I am not rightly able to comprehend... by wvmarle · · Score: 4, Informative

      From a look at the linked article, it seems that one of the issues is data generated by these web sites. Such as user statistics, or user uploaded content, etc. That naturally lives primarily on the live web server and is also data that you don't want to lose. Also as other commenters mentioned as well the EC2 service is not a cloud-storage server, it's a web hosting service, and web hosts tend to indeed generate their own data.

      This data of course needs to be backupped actively, and one would expect a web host to include that in its service. That's one of the reasons to pay for such a service, instead of doing it yourself.

      Besides relying on their backups it's of course a good idea to regularly take backups yourself. But even if you do this daily, it means you may lose up to a day's worth of data. And that's (partly) what happened here. It's similar to someone who takes a photo on a digital camera, and subsequently loses that camera and the photo with it. You don't say "they shouldn't use a camera as primary data repository". It isn't. It's a temporary repository, and when the data is generated it's the one and only repository, simply pending copying to backup media.

    4. Re:I am not rightly able to comprehend... by teh+kurisu · · Score: 3, Interesting

      That depends. Only a couple of our servers in that availability zone were actually affected, but we're apparently being compensated as though all of them were. Bonus for us.

  2. What is S3? by badran · · Score: 5, Informative

    EC2 is not meant to be used for data storage, that is what S3 is designed for. You store data and backups on S3, and use EC2 to serve high bandwidth websites to the masses.

  3. Re:Clouds are ephemeral by mini+me · · Score: 5, Informative

    Cloud applications hosted on Amazon survived this incident without issue, as expected. Only the regular old hosted applications had problems with the outage. They were never "the cloud" to begin with, so I'm not sure why the term even comes up in this discussion.

    The cloud represents a black box that hides the underlying network topology so that there are no single points of failure. Cloud applications are tolerant because they are spread through different datacenters across multiple points of in world. A catastrophe at one or more datacenters will have no noticeable effect on the availability of a cloud application because it continues to run in many more.

    Amazon offers a few cloud applications: S3 comes to mind. But Amzon's EC2/EBS hosting service is a plain old hosting service like any other. The EC2 topology is not hidden away from you. You have to make active decisions about where you want your EC2 instance to live. That goes against the idea of the cloud. What Amazon does offer in EC2 is the tools necessary for you to build a cloud application, but not everything hosted on EC2 is a cloud application by default.

  4. Post morten Amazon explanation by nereid666 · · Score: 5, Informative

    Post morten Amazon explanation:
    http://aws.amazon.com/message/65648/

    --
    Damia