Slashdot Mirror

← Back to Stories (view on slashdot.org)

Amazon's Cloud Service Has Outage, Disrupting Sites (usatoday.com)

Posted by msmash on from the outage-report dept.

An anonymous reader shares a report on USA Today: Portions of Amazon Web Services, the nation's largest cloud computing company, went offline Tuesday afternoon, affected multiple companies across the United States but especially on the east coast. The outage appeared to have begun around 12:45 pm ET. It was centered in AWS' S3 storage system on the east coast. Many of the services that firms use AWS are for back-end processes, and therefore not immediately visible to consumers, though the outage could disrupt customer-facing activities like logins and payments. At least some websites that appear to be affected are: Airbnb, Down Detector, Freshdesk, Pinterest, SendGrid, Snapchat's Bitmoji, Time, Buffer, Business Insider, Chef, Citrix, CNBC, Codecademy, Coursera, Cracked, Docker, Expedia, Expensify, Giphy, Heroku, Home Chef, iFixit, IFTTT, isitdownrightnow.com, Lonely Planet, Mailchimp, Medium, Microsoft's HockeyApp, News Corp, Quora, Razer, Slack, Sprout Social, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), and Zendesk.

The dashboard of Amazon Web Services, which tracks the status of the service, is unable to change color, Amazon said. It is because the status dashboard also runs on the service that is down.

161 comments

  1. I guess that explains Strava by dugancent · · Score: 2

    I couldn't upload or view my run this morning. I was a bit upset, but I guess it can wait.

    --
    SJWs are the new boogeyman. -Me
    1. Re:I guess that explains Strava by xevioso · · Score: 2

      You could go for a run while you wait...

    2. Re:I guess that explains Strava by Anonymous Coward · · Score: 0

      MapMyRun as well.

    3. Re:I guess that explains Strava by Anonymous Coward · · Score: 0

      For some that would last about 30 seconds before getting winded....

    4. Re:I guess that explains Strava by Liquid+Len · · Score: 1

      Yes, it was posted on the Strava website yesterday that an outage of AWS was responsible for their own outage.

    5. Re: I guess that explains Strava by Anonymous Coward · · Score: 0

      I listened to those bsrds at reInvent:2016 and tabled a BlueMix initiative started two weeks earlier. They got up on stage and said their only naysayers were "infrastructure type people" and then had the head of the SEC's inside trading detection division declare AWS as secure and reliable. In other words, you could get fired for not going with AWS. Those freaking aholes! I'm going to load up OpenWisk.

  2. Also affected... by wbr1 · · Score: 1

    CNN is slow
    Hosted connectwise - broken
    Solarwinds/logic now MSP services - broken
    Imgur is down
    Amazon itself (music app will not connect, viewing past orders broken, probably more)

    --
    Silence is a state of mime.
    1. Re:Also affected... by s.petry · · Score: 1

      CNN is slow

      No need for cheap shots.

      Humor aside, https://www.minds.com/ was having issues also.

      --

      -The wise argue that there are few absolutes, the fool argues that there are no probabilities.

    2. Re:Also affected... by FatdogHaiku · · Score: 1

      Heck Amazon is having issues. I've placed two orders today and so far only one confirmation email has arrived, two hours later than normal. So I went to my orders page and got a not available message along with a zero purchase history for the last six months, and that's just wrong. Going to the actual item DOES show I just bought it and allow viewing the order, but that is the long way around...

      --
      You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
  3. Too bad ... by Anonymous Coward · · Score: 1, Funny

    Too bad it doesn't disrupt the ads on this site

    1. Re:Too bad ... by Oswald+McWeany · · Score: 4, Insightful

      Meh. Finally gave in and installed Ad Blocker. Slashdot easily has the most invasive ad structure of any web site I go to.

      --
      "That's the way to do it" - Punch
  4. Can confirm by trevc · · Score: 1

    My static website is on S3 and is down

    1. Re:Can confirm by Anonymous Coward · · Score: 0

      Our website is on West S3 and it is not down.

    2. Re:Can confirm by MachineShedFred · · Score: 1

      They are specifically saying the issues with S3 are in the US-Standard and US-East-1 regions. If you have specifically put your stuff in a bucket in another region, you are probably fine.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    3. Re:Can confirm by Anonymous Coward · · Score: 0

      I don't have a site at all, and the sites I do not have are obviously not on Amazon S3, and they are not down either.

    4. Re:Can confirm by ranton · · Score: 1

      They are specifically saying the issues with S3 are in the US-Standard and US-East-1 regions. If you have specifically put your stuff in a bucket in another region, you are probably fine.

      Which is exactly why this isn't some catastrophic situation, since anyone who needed a highly available site would have it hosted in more than one availability zone.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
    5. Re: Can confirm by Anonymous Coward · · Score: 0

      More than one availability zone was down. You had to be in more than two or picked the right two. Do you feel lucky...

    6. Re:Can confirm by MachineShedFred · · Score: 1

      They've also had cross-region S3 bucket syncing since 2015. If you really need it to be up, spend a few more bucks on automatically syncing between say US-East-1 and US-West-2.

      If both of those go down at the same time, it's a very bad day for Amazon.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  5. even downdetector.com is down by gd2020 · · Score: 4, Funny

    irony

    1. Re:even downdetector.com is down by Daetrin · · Score: 2, Funny

      Are you sure it's down? I tried to check with IsItDownRightNow.com, but am having difficulties with the site for some reason...

      --
      This Space Intentionally Left Blank
    2. Re:even downdetector.com is down by gd2020 · · Score: 1

      http://downforeveryoneorjustme... is up for saying its down =)

    3. Re:even downdetector.com is down by tsqr · · Score: 1

      The first time I looked at that URL (all lowercase, as in TFS), I read it as I Sit Down Right Now. I'm sitting down right now, too, so what's the big deal?

    4. Re:even downdetector.com is down by gd2020 · · Score: 1

      as long as your chair is not hosted at AWS you are probably fine, but if it is then get up and fix it please.

    5. Re:even downdetector.com is down by bondsbw · · Score: 1

      PEAC

      Problem Exists At Chair

      --
      All my liberal friends think I'm a conservative, all my conservative friends think I'm a liberal.
    6. Re:even downdetector.com is down by fnj · · Score: 1

      The first time I looked at that URL (all lowercase, as in TFS)

      Domain names are not case sensitive (RFC 4343). The remainder of the URL may be (and frequently, not not always, is), depending on the host software. This is also true for SMTP addresses.

    7. Re:even downdetector.com is down by tsqr · · Score: 1

      The first time I looked at that URL (all lowercase, as in TFS)

      Domain names are not case sensitive (RFC 4343). The remainder of the URL may be (and frequently, not not always, is), depending on the host software. This is also true for SMTP addresses.

      It's a sad commentary on the current state of /. that you felt compelled to explain what should be (and would have been just a few years ago) common knowledge to the site's readership. Here's the reason I mentioned the casing of the URL as presented in TFS: as you may have noticed, domain names are frequently presented in mass marketing with casing in order to clarify/emphasize what the site is about or to make it easier to remember what may be a long string of characters. "IsItDownRightNow.com" and "ISitDownRightNow.com" are both easy to remember, but lead the average person to different conclusions; "isitdownrightnow.com" leaves it to the parsing peculiarities of the reader.

  6. partly cloudy today by turkeydance · · Score: 1

    overcast tomorrow

    1. Re:partly cloudy today by Anonymous Coward · · Score: 0

      Amazon cloud...obscured by sky

  7. isitdownrightnow.com by Anonymous Coward · · Score: 0

    Yup.

  8. So why use these large cloud services? by xevioso · · Score: 0

    Seriously...I would suspect this is due to an attack of some sort. Just a hunch. And if so, isn't it the case that the larger the company, the bigger the target?
    Why not use smaller services? What's the real benefit of using AWS when you could use someone smaller that may be less likely to be a target for an attack?

    1. Re:So why use these large cloud services? by Anonymous Coward · · Score: 1

      Cost and capability.

    2. Re:So why use these large cloud services? by Lisandro · · Score: 1

      This. All jokes aside S3 is normally a great service at a very competitive price.

    3. Re:So why use these large cloud services? by ScentCone · · Score: 5, Informative

      Seriously...I would suspect this is due to an attack of some sort. Just a hunch.

      The last time Microsoft's Azure platform had a huge, sustained failure, it was just an internal screw-up, not an attack. I've got no reason to think Amazon's east coast problems are any different. Not to say it couldn't be an attack, but no reason to think one way or the other, and lots of reasons to think "screwed up" - because that has happened at Amazon and elsewhere in Big Cloud many times.

      And if so, isn't it the case that the larger the company, the bigger the target?

      Yeah, but they've also got the enormous resources to help fend off problems that would crush a smaller provider. Works both ways.

      --
      Don't disappoint your bird dog. Go to the range.
    4. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      Economies of scale. There is a reason why a lot of small companies either have, or in the process of moving to AWS, especially serverless services like Amazon Lambda. It means they can fire their OS guys, DBA, and middleware people, and focus on being a dev house. AWS means no worries about warranties, hardware, data centers, or any of that stuff.

    5. Re:So why use these large cloud services? by Anonymous Coward · · Score: 1, Insightful

      You're right. I'll have my devs spend a year learning and implementing some small cloud provider only for them to go under.

      Also, the smaller ones go down too, it's just not noticeable or newsworthy.

    6. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      About 80% of all downtimes are caused by manipulation errors of employees.

      Downtimes due to attacks are actually very rare.
      Attackers mostly steal information/data and don't want to ring the alarmbells by creating a downtime.

    7. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      This. All jokes aside S3 is normally a great service at a very competitive price.

      I thought you said 'all jokes aside'?

      Amazon is not cost-competitive.

      I get more value and redundancy out of spinning up dozens of 1-2 GB Digital Ocean droplets than I do with Amazon--especially now that they have their LBS service and an equivalent to EBS.

    8. Re:So why use these large cloud services? by hawguy · · Score: 1

      Amazon is not cost-competitive.

      I get more value and redundancy out of spinning up dozens of 1-2 GB Digital Ocean droplets than I do with Amazon--especially now that they have their LBS service and an equivalent to EBS.

      A 2GB RAM digital ocean droplet gives you 40GB of disk space for $20/month, or around 50 cents/GB. To get the same triple redundancy that S3 gives you, you're paying $1.50/GB, compared to $0.023/GB that S3 costs. Plus you've got to manage the redundancy yourself. Or, better availability, you can have S3 mirror your data between regions, and you're paying $0.046/GB.

      I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my app to the us-west region (it could be done automatically, but full-region outages like this are so rare, I haven't bothered)

      (you could store your data on Digital Ocean LBS and pay $0.10/GB (or $0.30/GB for the same triple redundancy S3 provides).

      How is it that you pay more for storage, have more hardware to manage, yet you get more value and redundancy than S3?

    9. Re:So why use these large cloud services? by Lisandro · · Score: 1

      It is kinda cute when people try to think "cloud storage" in terms of HDDs alone.

    10. Re:So why use these large cloud services? by phantomfive · · Score: 1

      Who would you you use? Serious question. Between AWS, Google, and Azure, AWS still seems better to me.

      --
      "First they came for the slanderers and i said nothing."
    11. Re:So why use these large cloud services? by phantomfive · · Score: 1

      I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my app to the us-west region (it could be done automatically, but full-region outages like this are so rare, I haven't bothered)

      This is important: if you need HA, you should mirror across regions. Because regions go down, and this is not the first time. They will continue to go down in the future. Amazon.com is still up because they know that this is a problem, and they prepared for it.

      --
      "First they came for the slanderers and i said nothing."
    12. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      ...because today it was nobody's fault. If you host on your own hardware and it goes down, you fucked up. If you host at a small hosting provider and they go down, you fucked up. If you host in the cloud and the cloud goes down, management won't admit fault, so nobody is to blame. Besides, everybody else is down too, so you won't get singled out for bad press. Nobody ever got fired for...

    13. Re:So why use these large cloud services? by BenFranske · · Score: 1

      That's easier said than done though. See for example the Amazon AWS status portal which stated things were just fine for far longer than they were because Amazon couldn't update it. Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.

    14. Re:So why use these large cloud services? by Lisandro · · Score: 1

      In my own personal experience, good failover is not as easy as expected. Never.

    15. Re:So why use these large cloud services? by BenFranske · · Score: 0

      Google is pretty much an also-ran at this point so the question is AWS vs. Azure (or of course self-hosting but we'll assume you really want to do cloud and I can't talk you out of it). In my experience the answer depends on your application. If you're building a new from the ground up web-based application AWS is probably the front-runner. If you're migrating an existing in-house system and want to do things more incrementally, do something hybrid with your in-house stuff, etc. Azure is far simpler to get that going with. Azure feels designed to be familiar and comfortable for traditional enterprise IT people, AWS feels designed by/for the Silicon Valley startup crowd.

    16. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      Google's cheaper than AWS until you're dealing with serious scale.

    17. Re:So why use these large cloud services? by phantomfive · · Score: 1

      Yeah, that's why AWS doesn't do it for you automatically.
      You gotta test it.

      --
      "First they came for the slanderers and i said nothing."
    18. Re:So why use these large cloud services? by ranton · · Score: 1

      So why use these large cloud services?

      Because this outage doesn't affect anyone who cared about ensuring their cloud systems had high availability. If they did they would have had servers hosted in multiple availability zones. No cloud provider, or home-brew solution, is going to be highly available if you only have servers in one server farm.

      So it appears either us-east-1 or us-east-2 was down. If you wanted 99.9%+ up-time you should have servers in us-west-1 or us-west-2 also. This isn't rocket science.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
    19. Re:So why use these large cloud services? by phantomfive · · Score: 1

      Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.

      I just think of it as a normal data center, except you don't have to drive down there to install a new box or figure out what's wrong.
      So AWS is just more convenient, that's all. Still requires basically the same expertise as a datacenter, minus wiring.

      --
      "First they came for the slanderers and i said nothing."
    20. Re:So why use these large cloud services? by ranton · · Score: 1

      ...because today it was nobody's fault.

      If your boss wanted a highly available site and you only hosted it in a single availability zone, it would still be your fault if it goes down (unless you have records of him denying funds for the second zone). Hosting it in the cloud doesn't mean you can start being stupid.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
    21. Re:So why use these large cloud services? by phantomfive · · Score: 1

      Google's cheaper than AWS until you're dealing with serious scale.

      Has Google started running their own websites on their cloud yet?

      --
      "First they came for the slanderers and i said nothing."
    22. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      Um, why? They certainly have the infrastructure.

    23. Re:So why use these large cloud services? by userw014 · · Score: 1

      Seriously ... these outages are usually caused by the organization itself, either by an immediate technical mistake, or when a minor glitch cascades into something major due to a design flaw.

      I'm sure Amazon is a constant target of hackers, both pimply faced youths and of shady state-sponsored black-hats. But taking out Amazon isn't a very interesting goal.

    24. Re:So why use these large cloud services? by gravewax · · Score: 1

      Yep, these are the people that really should be in services like AWS or Azure as they simply don't have a good understanding of technology or what it takes to provide HA/DR (not that AWS seems to have done a good job either with this mess).

    25. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      These are regions not AZs.

    26. Re:So why use these large cloud services? by Anonymous Coward · · Score: 0

      You completely missed GP's point. Congratulations.

    27. Re:So why use these large cloud services? by BenFranske · · Score: 0

      Your question is unclear. Are you asking why Google is an also-ran or why AWS/Azure split the way I suggested. I'm going to assume the former. It's got nothing to do with the infrastructure, it's about the business, offerings, service/support, and wherewithal. Simply put Google doesn't have 1) the cloud services customer base that either of the other two do, and 2) the cloud product variety/maturity that the others do. Google also has shown no serious interest in improving their cloud services and/or doing a better job of selling them and competing with the others. Google has pretty much zero track record of selling services (other than ads) to businesses. About the only thing you can come up with is "Google Apps for Business". Compared to other offerings like Office365 that is a joke. Their customer support in all products is terrible. Need I go on? There's just no evidence that they will be a player in outside cloud services, sure they run a huge amazing infrastructure internally but they just don't play well with others.

    28. Re:So why use these large cloud services? by mspohr · · Score: 1

      Nobody expects the Spanish Inquisition.

      --
      I don't read your sig. Why are you reading mine?
    29. Re:So why use these large cloud services? by DogDude · · Score: 1

      Well, obviously, Amzaon's redundancy is either useless on non-existent, so I don't know what you think you're paying for.

      --
      I don't respond to AC's.
    30. Re:So why use these large cloud services? by hawguy · · Score: 1

      Well, obviously, Amzaon's redundancy is either useless on non-existent, so I don't know what you think you're paying for.

      There's a difference between durability and availability. I'm mostly interested in durability for objects I store in S3, but for when I care about availability, I mirror them across regions.

    31. Re:So why use these large cloud services? by ebrandsberg · · Score: 1

      I was working with a customer today that was using opsworks to provision into us-west-2, and it was failing as well. Did the issue show up on the dashboard? Nope. Were they impacted in a DIFFERENT zone than what had the issue? Yep. As much as they would like to say, there were issues across the board as a result of the dependency on us-east-1.

    32. Re: So why use these large cloud services? by Anonymous Coward · · Score: 0

      3 nines allows you to have more than 8 hours of downtime in a given year. So even an all-eggs-in-one-AZ deployment would still be okay for an SLA that required only 3 nines.

  9. The solution by Ol+Olsoc · · Score: 4, Funny

    This wouldn't happen if we put the cloud in the cloud.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    1. Re:The solution by wonkey_monkey · · Score: 1

      Yo dawg, I heard ah fuck it.

      --
      systemd is Roko's Basilisk.
    2. Re:The solution by Ol+Olsoc · · Score: 1
      With apologies to God

      Ezekial saw the cloud

      Ezekiel saw the cloud

      Way up in the middle of the air.

      Now Ezekiel saw the cloud was great.

      Way in the middle of the air.

      Chorus:

      And the big cloud is run by Bezos, good Lordy

      And the little cloud run by Sundar Pichai

      In the cloud in the cloud in the cloud good Lord

      Way in the middle of the air.

      Who's that yonder backed up tonight?

      Way in the middle of the air

      . It must be the internet of the people that’s right,

      Way in the middle of the air.

      Chorus:

      And the big cloud is run by Bezos, good Lordy

      And the little cloud run by Sundar Pichai

      In the cloud in the cloud in the cloud good Lord

      Way in the middle of the air.

      Who's that yonder dressed in red?

      Way in the middle of the air.

      stored their data local, now they dead

      Way in the middle of the air.

      Chorus:

      And the big cloud is run by Bezos, good Lordy

      And the little cloud run by Sundar Pichai

      In the cloud in the cloud in the cloud good Lord

      Way in the middle of the air.

      Who's that yonder dressed in black?

      Way in the middle of the air.

      must be data lost in an attack

      Way in the middle of the air.

      Chorus:

      And the big cloud is run by Bezos, good Lordy

      And the little cloud run by Sundar Pichai

      In the cloud in the cloud in the cloud good Lord

      Way in the middle of the air.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  10. Yeah, this got me as well by supremebob · · Score: 1

    Our mobile app hosts most of it's images in S3. We're basically displaying blank screens to our customers right now.

    1. Re:Yeah, this got me as well by DontBeAMoran · · Score: 1

      Does it also host your spell checker?

      --
      #DeleteFacebook
    2. Re:Yeah, this got me as well by Paradise+Pete · · Score: 2

      Does your spell checker really understand when to use its or it's? Mine doesn't. Not an easy problem for computers (nor for humans, apparently). Or were you talking about something else?

    3. Re:Yeah, this got me as well by DontBeAMoran · · Score: 0

      "Our mobile app hosts most of it is images in S3."

      That's what you wrote above. I guess you're the one who doesn't understand when to use its vs it's.

      --
      #DeleteFacebook
    4. Re:Yeah, this got me as well by Osgeld · · Score: 2

      shut up, dickhead

    5. Re:Yeah, this got me as well by Known+Nutter · · Score: 3, Informative

      Your and idiot.

      --
      Beware of the Leopard.
    6. Re:Yeah, this got me as well by Gabest · · Score: 1

      That's nice, but you can't blame a 3rd party, your users don't care about excuses.

    7. Re:Yeah, this got me as well by Anonymous Coward · · Score: 0

      The more people read badly written languages, the more people copy the same mistakes.

      Be my guest trying to read english ten years from now.

    8. Re:Yeah, this got me as well by sexconker · · Score: 1

      Yore*

    9. Re:Yeah, this got me as well by Anonymous Coward · · Score: 0

      "Shut up, dickhead."

      FTFY

    10. Re:Yeah, this got me as well by thegarbz · · Score: 1

      In most cases users don't care about outages either. It takes actual dataloss / monetary loss for users to care about something. Being inconvenienced will be brought back to "oh these silly computers are being silly again".

    11. Re:Yeah, this got me as well by Anonymous Coward · · Score: 0

      Yeah, drop dead, asshole.

    12. Re:Yeah, this got me as well by phantomfive · · Score: 1

      You can mirror S3 across regions, just FYI. It's a bit more expensive though, of course.

      --
      "First they came for the slanderers and i said nothing."
    13. Re:Yeah, this got me as well by Anonymous Coward · · Score: 0

      Do you know the difference between a spell and grammar checker? :-)

    14. Re:Yeah, this got me as well by DontBeAMoran · · Score: 1

      Of course I do. They're different from a spell and grammar chess.

      --
      #DeleteFacebook
    15. Re:Yeah, this got me as well by Paradise+Pete · · Score: 1

      First of all I didn't write it. Second, obviously I knew what the problem was because, ya know, I talked about it. Third, you did not answer the question. Does your spell checker understand when to use it's or its? Why are you being so snarky when so far neither of your posts have actually demonstrated a full understanding of the thread?

  11. Wow AWS Goes down also? by gods_design · · Score: 1

    Amazing AWS goes down randomly like Azure does? Huh?

    --
    -- David inquired...
    1. Re:Wow AWS Goes down also? by Nutria · · Score: 1

      More amazingly, Microsoft runs a system on the competitor's cloud instead of Azure. (I presume that HockeyApp is an acquisition that they haven't migrated, but still it's amusing.)

      --
      "I don't know, therefore Aliens" Wafflebox1
    2. Re:Wow AWS Goes down also? by MachineShedFred · · Score: 1

      Yeah, it's almost like it's a service run by humans, and humans are fallible no matter who their employer is.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    3. Re:Wow AWS Goes down also? by BenFranske · · Score: 2

      I caught that too and thought it was interesting. One other interesting thing about an application that really uses cloud services to their full potential (rather than just as an expensive VM/VPS) is that since the services are not commoditized/standardized there is a lot of cloud vendor lock-in. E.g. if you build a huge web app around AWS you're going to have a lot of rework to do (to some extent depending on how well you modularized your code) to migrate to another cloud provider.

    4. Re: Wow AWS Goes down also? by Anonymous Coward · · Score: 0

      Interestingly, AWS allows you to scale-out Docker deployments, now. So, actually, as long as you don't consider Docker to be a vendor to be locked-into, you can potentially use Docker as your "platform" and just coincidentally deploy it into AWS.

      I have no idea if any other cloud providers support Docker, but theoretically it would be a good move if they wanted to lure customers away from AWS in the future.

  12. Ahh, the cloud. by Lisandro · · Score: 4, Insightful

    AKA "just someone's else computer".

    This outage is being going for over an hour now but, according to Amazon, their services are green all across the board with "increased error rates". Almost feels like they're trying to cheat out their own SLAs.

    1. Re:Ahh, the cloud. by Lisandro · · Score: 2

      Grrr, it's "someone else's". Thanks phone.

    2. Re:Ahh, the cloud. by MachineShedFred · · Score: 1

      Nah, their service health dashboard was also affected by the issues. They've fixed it (the status page) and it's showing wider ranging issues.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  13. Thingiverse by DontBeAMoran · · Score: 1

    This explains why a lot of images won't load on thingiverse.com

    --
    #DeleteFacebook
  14. losing nines by teefal · · Score: 1

    nine-nines
    eight-nines
    seven-nines
    six-nines ...

    In a few minutes, it'll be two hours down.

    1. Re:losing nines by Anonymous Coward · · Score: 0

      Well, it wasn't a good week for Price Waterhouse Coopers either. At least their problem was a momentary lapse, not an hours-long one.

    2. Re:losing nines by Anonymous Coward · · Score: 0

      Well, it wasn't a good week for Price Waterhouse Coopers either. At least their problem was a momentary lapse, not an hours-long one.

      They were down/wrong for at least 3.5 hours though. Just because they didn't realize it wasn't working doesn't mean it was. the other 3:29:59

  15. as jason scott said by Anonymous Coward · · Score: 0

    FUCK THE CLOUD

  16. Why are they lying? by KlomDark · · Score: 1

    Their whole says GREEN and "No Recent Events". What a bunch of liars.

    "Increased Error Rates" my ass

    This will lead to massive lost respect for AWS!

    'Fess up, don't be lyin.

    1. Re:Why are they lying? by KlomDark · · Score: 1

      blew my href on the word 'dashboard', sorry

    2. Re:Why are they lying? by Anonymous Coward · · Score: 0

      Not everything green for me, some yellow and some red - mostly N. Virginia

    3. Re:Why are they lying? by Anonymous Coward · · Score: 2, Informative

      Ha!

      Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

  17. Coffee Doesn't Meet Bagel by Anonymous Coward · · Score: 0

    CMB messaging has silently stopped.

    1. Re:Coffee Doesn't Meet Bagel by ColdWetDog · · Score: 5, Funny

      CMB messaging has silently stopped.

      The Cosmic Microwave Background has stopped?

      Jeez. I didn't realize that Amazon was that important.

      --
      Faster! Faster! Faster would be better!
    2. Re:Coffee Doesn't Meet Bagel by Anonymous Coward · · Score: 0

      This reminds me of the time I edited the Wikipedia page for the three-torus model and linked "CMB" to the page for Color Me Badd. It stayed up there for like 6 months, fun times.

    3. Re:Coffee Doesn't Meet Bagel by Anonymous Coward · · Score: 0

      CMB messaging has silently stopped.

      The Cosmic Microwave Background has stopped?

      Jeez. I didn't realize that Amazon was that important.

      With the cosmic microwave turned off we won't need to worry about global warming anymore. Thanks Amazon! You guys are the best.

  18. Great time to handle failure cases... by swan5566 · · Score: 1

    To make them fail gracefully with this sort of outage. These opportunities don't come by very often.

    --
    In debates about Christianity, there are two groups: those looking for answers, and those looking to just ask questions.
    1. Re:Great time to handle failure cases... by Dunbal · · Score: 3, Insightful

      Why bother. People are used to mediocrity. Doing what you suggest would cost money and eat into profits. Fuck 'em. The TOS are quite clear, this will happen once in a while and you'll just have to put up with it. By the way, don't forget your next payment is due.

      --
      Seven puppies were harmed during the making of this post.
  19. The irony of this... by Anonymous Coward · · Score: 0

    Here is the irony of this. A while back, when I was interviewing for jobs, A CxO of a firm wanted to have his round with me face to face. When I asked him how he did backups, his reply was, "Asking about 'backups' or 'downtime' is like asking a Tesla owner what type of buggy whip they use. We are cloud based, with the only hardware other than workstations being the pipe to AWS."

    I decided to go elsewhere.

    1. Re:The irony of this... by MachineShedFred · · Score: 3, Informative

      Yeah, that guy is a douche, and you don't want to work for him.

      I work for a company that hosts stuff in AWS, and we are doing cross-region backups, as well as a weekly dump to our on-prem servers. Yes, AWS has a good amount of redundancy between the multiple availability zones per region, etc. But why take the chance, when it's trivial to dump your stuff on a scheduled basis to another region, or even to something as cheap as a Synology box or one of those workstations being recycled into a FreeNAS box in the office?

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    2. Re:The irony of this... by the_Bionic_lemming · · Score: 1

      I decided to go elsewhere.

      You missed a great opportunity!

      CxO : Dammit! The Cloud is Down - Do something!

      You: "Right, I'm on it" //heads for exit

      CxO: Yells "Where are you going at a time like this"???

      You: "To buy a buggy whip For that Horse you're going to be needing".

      --
      _ _ _ Go for the eyes Boo! GO FOR THE EYES!
  20. Multi-region design++ by Anonymous Coward · · Score: 0

    For critical applications of course, due to the expense.

  21. Dashboard was dependent on S3 US-EAST-1 =) by Anonymous Coward · · Score: 0

    That's why

    1. Re:Dashboard was dependent on S3 US-EAST-1 =) by KlomDark · · Score: 1

      Just saw that and came here to update the info. That's pretty funny!

  22. internet images down... by s1d3track3D · · Score: 1

    Browsing in general I'm getting a kick out of actually seeing how many sites use S3 as an image server.

  23. Not a complete outage by Anonymous Coward · · Score: 0

    Sorry, but if your mission critical sites/services are down from this then it's your own fault for not implementing a robust enough disaster planning. Amazon provides its services in different zones with isolation specifically to address this sort of issue. If you have redundancy implemented on the West coast, and your DR is well designed, you should be able to switch over to your West coast resources and keep operating during the east coast outage with the only impact being a few extra milliseconds of latency for users located in the east. The only reason this is even a story is that it's so rare for AWS to even have this level of outage.

    1. Re: Not a complete outage by Anonymous Coward · · Score: 0

      Seeing how Amazon has trouble updating their own status page because it's in the affected region shows how hard that is to do in practice. Inter-DC failovers are usually not undertaken lightly and running Active-Active from multiple DC's is only possible for specific applications.

    2. Re:Not a complete outage by swan5566 · · Score: 1

      Depends on your business model. If you're big on data, having redundancy might not make economic sense. If it's mostly compute, then yeah, have a backup region.

      --
      In debates about Christianity, there are two groups: those looking for answers, and those looking to just ask questions.
    3. Re: Not a complete outage by BenFranske · · Score: 2

      This. Clearly a lot of people jumped into cloud "to save boatloads of money" (same reason so many jumped into outsourcing. Saving tons of money is not often a good reason to do something. Usually you can make incremental savings but it's never what the salespeople or service promises because those prices ignore things like redundancy, etc. In the end doing it right ends up costing about the same as you were paying before, maybe a little less or a little more and maybe you gain some more features, but it's also probably more complex.

  24. Increased productivity! by Photonmaker · · Score: 1

    This might actually impact the Federal employee productivity metric - bring down Quora alone has increased my work output today.

  25. Irony by jargonburn · · Score: 1

    isitdownrightnow.com

    Yep.

  26. All good by Anonymous Coward · · Score: 4, Insightful

    My website is on my own server network and is working fine.

    Because I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates...

    Oh yeah, and my costs are far less than the monthly dollop of blood extracted by cloud services.

    "Live by the cloud... die by the cloud."

    Carry on, suckers.

    1. Re:All good by Anonymous Coward · · Score: 1

      Serious comment here:

      The only problem with running your own infrastructure is load estimation. When preparing a business case for running your own infrastructure, any deficit in load has to be paid for by the business. With using cloud services, the load deficit does not exist, and the company only pays for what it uses.

      It is for this reason that cloud services are always cheaper than in-house infrastructure.

      If your company decides that the many advantages (listed in the post above) of in-house infrastructure are worth the added cost, and the few disadvantages then congratulations, in-house is the way for you. As it happens, in-house is also the winner for me and my requirements also.

      However, for many, the cloud does win out for many.

    2. Re: All good by Anonymous Coward · · Score: 0

      I'm interested; tell me more.

      How much did that peering agreement with Worldcom cost when you had them attach you to the internet backbone?

    3. Re:All good by Anonymous Coward · · Score: 0

      ...where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates...

      I see you know nothing about cloud computing or AWS. You do control every one of those things. You pick the OS, you update it, you install what you want, you schedule backups, you can set up a worldwide network of servers for fail over, etc. And if you know what you're doing, all of that takes about 15 minutes. And it's cheap as dirt. You are absolutely doing it the dumb way.

  27. what the fuck is with these ads? by koreanbabykilla · · Score: 1, Troll

    whats with the shitty half page ads that wont go away /.?

  28. Phil Hendrie Show by Anonymous Coward · · Score: 0

    http://www.philhendrieshow.com/ multimedia content is unavailable! I'm about to go Bakersfield chimp.

  29. Russia by geek · · Score: 1, Funny

    Russia did it

  30. Start time by jamesivie · · Score: 1

    The outage started at 12:38 ET.

    --
    "O'Connor, smash the window." "Why me, Bigboote?" "It might be boobie-trapped!" "Oh!"<smash> -Buckaroo Banzai
    1. Re:Start time by Anonymous Coward · · Score: 0

      There was a 3 in the number?
      HL3 confirmed!

  31. All Eggs, One Basket by Anonymous Coward · · Score: 0

    AWS East would have fewer outages if only idiots would stop putting absolutely everything in AWS East. There are other regions. Fucking use them, you motherfucking idiots.

    1. Re:All Eggs, One Basket by Anonymous Coward · · Score: 0

      Fucking idiot talking about fucking idiots

    2. Re:All Eggs, One Basket by Anonymous Coward · · Score: 0

      West coast represent! What outage? There was an outage? Fucking idiots overloaded the east coast again? Fuck them.

  32. ONLY EAST COAST by Anonymous Coward · · Score: 0

    EAST COAST IS ONLY COAST

    1. Re:ONLY EAST COAST by sexconker · · Score: 4, Funny

      West coast best coast, east coast least coast.

    2. Re:ONLY EAST COAST by KlomDark · · Score: 1

      Omaha: We Don't Coast

      (Somebody actually got paid to think up that stinker.)

  33. Amazon Images by Anonymous Coward · · Score: 0

    Some cover images from my Amazon author's page are missing.

  34. Amazon FireStick down for me in Germany by Anonymous Coward · · Score: 0

    That thing is basically bricked right now..

    1. Re:Amazon FireStick down for me in Germany by Anonymous Coward · · Score: 0

      Have you tried chanting "FireStick walk with me" ?

  35. Who knew by Anonymous Coward · · Score: 0

    It wasnt really a cloud after all... just a bunch of smog

  36. GitHub by thebullshitpatrol · · Score: 1

    No GitHub icons either.

  37. Soundcloud by Anonymous Coward · · Score: 0

    You can add soundcloud to the list too!

  38. Shattered SHA1? by pHZero · · Score: 2

    Surprised no one has mentioned that a possible root cause is that someone uploaded the two PDFs from the Shattered attack, causing a SHA1 collision on S3

  39. It's not all bad by DontBeAMoran · · Score: 1

    At least some websites that appear to be affected are: [...] Pinterest

    At least there's something positive about this outage.

    --
    #DeleteFacebook
  40. It's not "no dependencies" as much as "fewer" by tepples · · Score: 3, Insightful

    I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates

    I'm interested in how you eliminated dependencies on your home ISP's DHCP server, backbone routers, the DNS, the OCSP server of the CA that provided your site's TLS certificate, etc. And without advertisement exchanges or subscription payment servers, how do you afford to keep your server powered on and connected to the Internet?

    1. Re:It's not "no dependencies" as much as "fewer" by Anonymous Coward · · Score: 0

      I'm interested in how you eliminated dependencies on your home ISP's DHCP server

      I have static IPs. As far as I know (could be wrong), the ISPs DHCP server or servers has nothing to do with me. Certainly my IP never changes. If it did, I'm sure I'd be warned, and ten seconds at the DNS server record editor would take care of that.

      backbone routers, the DNS

      Everyone who serves web pages via a DNS-translated name uses DNS. Just as you said: much fewer dependencies.

      One can also serve web pages from an IP, and I've done that on occasion as well. Works fine. Just not very friendly.

      the OCSP server of the CA that provided your site's TLS certificate, etc

      I don't serve HTTPS. I'm not collecting information about you, or serving illegal or secret information.

      And without advertisement exchanges or subscription payment servers, how do you afford to keep your server powered on and connected to the Internet?

      I work.

      There's another thing, too. When many people concentrate their dependencies on something such as Amazon's cloud, when the cloud dies or is successfully attacked, everyone goes down. If someone attacks my site, or it has a problem that knocks it off the web, all the world loses is my one site. Distributed is better. That was a useful design goal of the Internet. It's both amazing and depressing to see people undermining that any more than absolutely necessary.

    2. Re:It's not "no dependencies" as much as "fewer" by tepples · · Score: 2

      I have not built up an infrastructure depending on computers I don't own, don't control

      I'm interested in how you eliminated dependencies on your home ISP's DHCP server

      I have static IPs.

      First, most home ISPs charge extra per month for that. Second, Comcast requires subscribers with a static IP to either forfeit the static IP or rent and use Comcast's modem, which reintroduces "depending on computers I don't own, don't control", and moving to an area whose cable company is not Comcast can prove cost prohibitive, particularly if your work is unrelated to your website.

      I don't serve HTTPS. I'm not collecting information about you, or serving illegal or secret information.

      Making even a completely public, completely static website available over cleartext HTTP and not HTTPS has three consequences. First, the most popular western web search engine will demote your site. Second, you run the risk of a rogue ISP injecting pop-up advertisements and other malware into your pages, as Comcast has done (source 1; source 2). Third, you cannot make resources on your site available for transclusion into sites that do use HTTPS due to the mixed content policy.

      And without advertisement exchanges or subscription payment servers, how do you afford to keep your server powered on and connected to the Internet?

      I work.

      Is your work related to your website? If not, what benefit do you gain from having a website, and do you ever have to take time off your day job to keep it updated?

      When many people concentrate their dependencies on something such as Amazon's cloud, when the cloud dies or is successfully attacked, everyone goes down.

      The same is true of DNS or a major home ISP. Or is your argument "only those dependencies that are provably absolutely necessary, and not a single one more"?

    3. Re:It's not "no dependencies" as much as "fewer" by Anonymous Coward · · Score: 0

      I'm interested in how you eliminated dependencies on your home ISP's DHCP server

      dyndns

      Many home gateways (routers) have dyndns functionality built in.

    4. Re:It's not "no dependencies" as much as "fewer" by Anonymous Coward · · Score: 0

      dyndns

      DynDNS has nothing to do with dependencies on a DHCP server.

      The DHCP server gives out IP addresses on a lease.

      A DNS server associates names with IP addresses.

      Having DynDNS does not remove the dependency on a DHCP server.

      Quite the opposite, actually: The very fact that one uses a DHCP server in the first place is the primary reason for even needing something like DynDNS.

      You appear to be confused.

    5. Re:It's not "no dependencies" as much as "fewer" by tepples · · Score: 1

      Dyn is just as vulnerable as AWS.

  41. AMZN by MillionthMonkey · · Score: 1

    I'm seriously wondering if I should unload my AMZN stock for a couple days... before the geniuses who bought Nintendo stock during the Pokemon Go craze realize why all these websites are down.

  42. Problems with the Cloud by Cro+Magnon · · Score: 4, Insightful

    If it rains, you're in trouble.

    --
    Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
  43. Restrospective by Anonymous Coward · · Score: 0

    This should be an interesting post-mortem on the the failure.

    As a dev-ops guy, I'd be interested in the "lessons learned".

  44. gay hookup apps too by Anonymous Coward · · Score: 0

    Can't get on Scruff or Growlr today.

  45. Re: for many by hackwrench · · Score: 1

    Brought to you by the department of redundancy department.

  46. The internet is down by Anonymous Coward · · Score: 0

    So much the internet routing around blockages.

  47. Something else going on? by Majestic+Fear · · Score: 1

    Interestingly around the same time the company I work at which does not use AWS had several of its sites go down. The site I work out of was down for a few minutes, unsure how long the others were down.

  48. hmm... by Anonymous Coward · · Score: 0

    haven't noticed anything. I don't use amazon and as it seems nor sites hosted there lol..

  49. That's what you get by Khyber · · Score: 1

    When you rely upon someone else to handle the shit you should be handling, this is your just reward.

    --
    Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
  50. The victim list is very long by sentiblue · · Score: 1

    Apple is one of the largest customers being hit...
    Apple iCloud and iTunes services were intermittently going offline the whole morning yesterday. I had trouble viewing my TV episodes.

  51. pCloud by cherishjoo · · Score: 1

    I only use pCloud.

  52. Cost per minute of AWS/S3 downtime? by Anonymous Coward · · Score: 0

    This makes me more than happy that I decided on a hybrid cloud strategy. We use AWS/S3 but have a local fallback environment. Worked without a hitch while others were dead in the water. I wonder what the cost per minute was of the AWS/S3 downtime to those who depend 100% on it.