Slashdot Mirror

← Back to Stories (view on slashdot.org)

Major Outage At the Amazon Web Services

Posted by ryuzaki0 on from the but-the-cloud-fixes-everything dept.

ralphart writes "The Northern Virginia datacenter for Amazon Web Services appears to be having a major outage that affects EC2 services. The Amazon Forums are full of reports of problems. Latest update from the status page: 2:49 AM PDT We are continuing to see connectivity errors impacting EC2 instances, increased latencies impacting EBS volumes in multiple availability zones in the US-EAST-1 region, and increased error rates affecting EBS CreateVolume API calls. We are also experiencing delayed launches for EBS backed EC2 instances in affected availability zones in the US-EAST-1 region. We continue to work towards resolution."

12 of 247 comments (clear)

  1. No Way! by Frosty+Piss · · Score: 5, Funny

    But how can this be possible? It's The Cloud . This sort of this simply doesn't happen.

    --
    If you want news from today, you have to come back tomorrow.
    1. Re:No Way! by 0123456 · · Score: 4, Informative

      A major outage on most professional cloud setups means it is down for a few hours. A major outage at work means the full day. It is like saying driving my car is so much safer then flying because I never got into an accident.

      Last time I remember a day-long outage at work was 1994, and that was because the license server failed so we couldn't run our own software (we couldn't recompile it to remove the DRM because the compiler also needed a license to run).

      I seem to remember that the Mac guys at the company also had a long outage when they couldn't connect to one of their Mac servers, but eventually someone actually went to the server room and discovered that it had been stolen.

      Back on topic, I just don't see all these day-long outages that apparenty seem to happen all the time in companies that haven't moved their servers to The Cloud(tm).

    2. Re:No Way! by lgw · · Score: 4, Insightful

      his will become quite the event in data warehouse circles I bet, because the cost of 'being in the cloud' just doubled; it's not enough to buy storage from one provider. The "always there" quality that's supposedly the benefit of cloud storage is a facade.

      The cloud doesn't have to be perfect - it just has to be as good in the eyes of VPs as the contractors they'd otherwise hire to run their internal datacenter. What's the value of an IT guy in the eyes of an MBA? Yeah, this sort of reality check wont phase them at all.

      --
      Socialism: a lie told by totalitarians and believed by fools.
  2. Re:Reddit is down because of this by cobrausn · · Score: 5, Funny

    You're posting on Slashdot, so I believe you already found the answer.

    --
    How does it feel to be a liar with pants constantly on fire?
  3. Severe weather in Virginia likely the culprit by stopacop · · Score: 3, Informative

    Severe weather hit the area. They shutdown Surry Power Station in Surry County, Virginia after a tornado took the power out that powers the power station.

    --
    http://www.stopacop.so -- You have rights. How about standing up for them before they go away?
    1. Re:Severe weather in Virginia likely the culprit by getagrip · · Score: 4, Informative

      I am in Northern Virginia. There is no power outage or severe weather here.

  4. Emergency Plan by sycorob · · Score: 4, Interesting

    I didn't even realize that one of our partners was using Amazon EWS until suddenly they were down all day. Amazon is really stable historically, but it's frustrating when you're out of business and all you can do is wait and see if Amazon will fix it soon.

    In the "old school" thinking, smart companies have a redundant data center somewhere, humming along and waiting to be switched on if the main data center ever goes down. "The cloud" was supposed to solve that - massive redundancy within Amazon's services were supposed to protect you from outages. Not the case, apparently, since it looks like Amazon is going to fall below their promised 99.95% uptime (4.38 hours per year downtime).

    I think the answer is to have redundant cloud services online, so you could switch from Amazon to Google or DevGrid if you had issues. The problem is, there's nothing quite like Amazon right now, it's not easy to switch from Amazon to some random service. This might be the biggest argument against virtual services - lack of standardization makes it hard to move from one to another, and hard to set up backup services in case of emergency.

    1. Re:Emergency Plan by MariusBoo · · Score: 3, Insightful

      Actually in the case of EC2 the smart thing would have been to have your instances spread over different availability zones...

    2. Re:Emergency Plan by Alarash · · Score: 3

      Even by using only AWS you can set up redundancy across multiple North America's regions. Even across continents, with one data center in Ireland and one in Singapore. But obviously it costs extra as they bill you the bandwidth between the regions. That's how you use The Cloud (c) (tm) (R). Using a single data center to set up redundancy is dumb because it's not redundancy. You need high availability for your VMs, but also for your data center.

      This is why banks or large businesses, for instance, have two or more data centers they always keep synchronized and have at least 50 kilometers between them. Thinking "well it's in one AWS data center so it's safe" is wrong, and this incident is a fine example of that.

  5. 6 weeks before the AWS summit 2011 by grapeape · · Score: 3, Interesting

    Gotta wonder what kind of flack Amazon is going to take for this one. I've had a couple clients looking into cloud services including moving to AWS and have already had one of them call me and cancel a meeting about it. While I understand stuff happens, the entire sales pitch for AWS was redundancy and build as you grow. Redundancy has obviously not worked in this case, while I usually support cloud services, this is definitely going to be a hard example to counter when trying to sell it to potential customers.

    1. Re:6 weeks before the AWS summit 2011 by TooMuchToDo · · Score: 4, Informative

      It's not short sighted at all. When someone else runs your gear, all you can do is sweat until they get things back online, and they can take their time under what's known as "commerically reasonable SLAs". When you own your own gear, your own colo, etc., how much effort you use to get back up and running is up to you.

      "The Cloud" for mission critical businesses is a joke.

  6. Re:Reddit is down because of this by Richard_at_work · · Score: 4, Informative

    Don't worry - Slashdot just did something similar. When I try and reply to comments through my accounts comments history page, its horribly horribly broken. Each attempt to click in the reply box loads a new comment further up in the comment tree, and scrolls the page to the newly loaded comment. Scroll back down, click in the box again and it loads anotehr comment and shunts me back up the page. It can get really fucking annoying when you are trying to reply to a comment thats quite a way down a long tree.