Slashdot Mirror
Amazon's Cloud Service Has Outage, Disrupting Sites (usatoday.com)
An anonymous reader shares a report on USA Today: Portions of Amazon Web Services, the nation's largest cloud computing company, went offline Tuesday afternoon, affected multiple companies across the United States but especially on the east coast. The outage appeared to have begun around 12:45 pm ET. It was centered in AWS' S3 storage system on the east coast. Many of the services that firms use AWS are for back-end processes, and therefore not immediately visible to consumers, though the outage could disrupt customer-facing activities like logins and payments. At least some websites that appear to be affected are: Airbnb, Down Detector, Freshdesk, Pinterest, SendGrid, Snapchat's Bitmoji, Time, Buffer, Business Insider, Chef, Citrix, CNBC, Codecademy, Coursera, Cracked, Docker, Expedia, Expensify, Giphy, Heroku, Home Chef, iFixit, IFTTT, isitdownrightnow.com, Lonely Planet, Mailchimp, Medium, Microsoft's HockeyApp, News Corp, Quora, Razer, Slack, Sprout Social, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), and Zendesk.
The dashboard of Amazon Web Services, which tracks the status of the service, is unable to change color, Amazon said. It is because the status dashboard also runs on the service that is down.
The dashboard of Amazon Web Services, which tracks the status of the service, is unable to change color, Amazon said. It is because the status dashboard also runs on the service that is down.
I couldn't upload or view my run this morning. I was a bit upset, but I guess it can wait.
SJWs are the new boogeyman. -Me
CNN is slow
Hosted connectwise - broken
Solarwinds/logic now MSP services - broken
Imgur is down
Amazon itself (music app will not connect, viewing past orders broken, probably more)
Silence is a state of mime.
Too bad it doesn't disrupt the ads on this site
My static website is on S3 and is down
irony
overcast tomorrow
This wouldn't happen if we put the cloud in the cloud.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
Our mobile app hosts most of it's images in S3. We're basically displaying blank screens to our customers right now.
Amazing AWS goes down randomly like Azure does? Huh?
-- David inquired...
AKA "just someone's else computer".
This outage is being going for over an hour now but, according to Amazon, their services are green all across the board with "increased error rates". Almost feels like they're trying to cheat out their own SLAs.
This explains why a lot of images won't load on thingiverse.com
#DeleteFacebook
nine-nines ...
eight-nines
seven-nines
six-nines
In a few minutes, it'll be two hours down.
Cost and capability.
This. All jokes aside S3 is normally a great service at a very competitive price.
Seriously...I would suspect this is due to an attack of some sort. Just a hunch.
The last time Microsoft's Azure platform had a huge, sustained failure, it was just an internal screw-up, not an attack. I've got no reason to think Amazon's east coast problems are any different. Not to say it couldn't be an attack, but no reason to think one way or the other, and lots of reasons to think "screwed up" - because that has happened at Amazon and elsewhere in Big Cloud many times.
And if so, isn't it the case that the larger the company, the bigger the target?
Yeah, but they've also got the enormous resources to help fend off problems that would crush a smaller provider. Works both ways.
Don't disappoint your bird dog. Go to the range.
Their whole says GREEN and "No Recent Events". What a bunch of liars.
"Increased Error Rates" my ass
This will lead to massive lost respect for AWS!
'Fess up, don't be lyin.
To make them fail gracefully with this sort of outage. These opportunities don't come by very often.
In debates about Christianity, there are two groups: those looking for answers, and those looking to just ask questions.
You're right. I'll have my devs spend a year learning and implementing some small cloud provider only for them to go under.
Also, the smaller ones go down too, it's just not noticeable or newsworthy.
Browsing in general I'm getting a kick out of actually seeing how many sites use S3 as an image server.
This might actually impact the Federal employee productivity metric - bring down Quora alone has increased my work output today.
isitdownrightnow.com
Yep.
My website is on my own server network and is working fine.
Because I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates...
Oh yeah, and my costs are far less than the monthly dollop of blood extracted by cloud services.
"Live by the cloud... die by the cloud."
Carry on, suckers.
whats with the shitty half page ads that wont go away /.?
Depends on your business model. If you're big on data, having redundancy might not make economic sense. If it's mostly compute, then yeah, have a backup region.
In debates about Christianity, there are two groups: those looking for answers, and those looking to just ask questions.
Just saw that and came here to update the info. That's pretty funny!
Russia did it
Yeah, that guy is a douche, and you don't want to work for him.
I work for a company that hosts stuff in AWS, and we are doing cross-region backups, as well as a weekly dump to our on-prem servers. Yes, AWS has a good amount of redundancy between the multiple availability zones per region, etc. But why take the chance, when it's trivial to dump your stuff on a scheduled basis to another region, or even to something as cheap as a Synology box or one of those workstations being recycled into a FreeNAS box in the office?
Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
Amazon is not cost-competitive.
I get more value and redundancy out of spinning up dozens of 1-2 GB Digital Ocean droplets than I do with Amazon--especially now that they have their LBS service and an equivalent to EBS.
A 2GB RAM digital ocean droplet gives you 40GB of disk space for $20/month, or around 50 cents/GB. To get the same triple redundancy that S3 gives you, you're paying $1.50/GB, compared to $0.023/GB that S3 costs. Plus you've got to manage the redundancy yourself. Or, better availability, you can have S3 mirror your data between regions, and you're paying $0.046/GB.
I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my app to the us-west region (it could be done automatically, but full-region outages like this are so rare, I haven't bothered)
(you could store your data on Digital Ocean LBS and pay $0.10/GB (or $0.30/GB for the same triple redundancy S3 provides).
How is it that you pay more for storage, have more hardware to manage, yet you get more value and redundancy than S3?
The outage started at 12:38 ET.
"O'Connor, smash the window." "Why me, Bigboote?" "It might be boobie-trapped!" "Oh!"<smash> -Buckaroo Banzai
It is kinda cute when people try to think "cloud storage" in terms of HDDs alone.
Who would you you use? Serious question. Between AWS, Google, and Azure, AWS still seems better to me.
"First they came for the slanderers and i said nothing."
No GitHub icons either.
I've done the latter -- mirrored my data between regions, so the us-east-1 outage took a single parameter change and service restart to point my app to the us-west region (it could be done automatically, but full-region outages like this are so rare, I haven't bothered)
This is important: if you need HA, you should mirror across regions. Because regions go down, and this is not the first time. They will continue to go down in the future. Amazon.com is still up because they know that this is a problem, and they prepared for it.
"First they came for the slanderers and i said nothing."
This. Clearly a lot of people jumped into cloud "to save boatloads of money" (same reason so many jumped into outsourcing. Saving tons of money is not often a good reason to do something. Usually you can make incremental savings but it's never what the salespeople or service promises because those prices ignore things like redundancy, etc. In the end doing it right ends up costing about the same as you were paying before, maybe a little less or a little more and maybe you gain some more features, but it's also probably more complex.
I decided to go elsewhere.
You missed a great opportunity!
CxO : Dammit! The Cloud is Down - Do something!
You: "Right, I'm on it" //heads for exit
CxO: Yells "Where are you going at a time like this"???
You: "To buy a buggy whip For that Horse you're going to be needing".
_ _ _ Go for the eyes Boo! GO FOR THE EYES!
That's easier said than done though. See for example the Amazon AWS status portal which stated things were just fine for far longer than they were because Amazon couldn't update it. Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.
Surprised no one has mentioned that a possible root cause is that someone uploaded the two PDFs from the Shattered attack, causing a SHA1 collision on S3
At least there's something positive about this outage.
#DeleteFacebook
In my own personal experience, good failover is not as easy as expected. Never.
I have not built up an infrastructure depending on computers I don't own, don't control, have no ability to see to the physical and network security of, where I don't have any control of reliability, redundancy, backup, availability of resources, longevity, OS level, OS and other software updates
I'm interested in how you eliminated dependencies on your home ISP's DHCP server, backbone routers, the DNS, the OCSP server of the CA that provided your site's TLS certificate, etc. And without advertisement exchanges or subscription payment servers, how do you afford to keep your server powered on and connected to the Internet?
Yeah, that's why AWS doesn't do it for you automatically.
You gotta test it.
"First they came for the slanderers and i said nothing."
So why use these large cloud services?
Because this outage doesn't affect anyone who cared about ensuring their cloud systems had high availability. If they did they would have had servers hosted in multiple availability zones. No cloud provider, or home-brew solution, is going to be highly available if you only have servers in one server farm.
So it appears either us-east-1 or us-east-2 was down. If you wanted 99.9%+ up-time you should have servers in us-west-1 or us-west-2 also. This isn't rocket science.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
Doing cloud based applications right is neither as inexpensive or as simple as a lot of people were led to believe/preached.
I just think of it as a normal data center, except you don't have to drive down there to install a new box or figure out what's wrong.
So AWS is just more convenient, that's all. Still requires basically the same expertise as a datacenter, minus wiring.
"First they came for the slanderers and i said nothing."
I'm seriously wondering if I should unload my AMZN stock for a couple days... before the geniuses who bought Nintendo stock during the Pokemon Go craze realize why all these websites are down.
...because today it was nobody's fault.
If your boss wanted a highly available site and you only hosted it in a single availability zone, it would still be your fault if it goes down (unless you have records of him denying funds for the second zone). Hosting it in the cloud doesn't mean you can start being stupid.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
If it rains, you're in trouble.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
Google's cheaper than AWS until you're dealing with serious scale.
Has Google started running their own websites on their cloud yet?
"First they came for the slanderers and i said nothing."
CMB messaging has silently stopped.
The Cosmic Microwave Background has stopped?
Jeez. I didn't realize that Amazon was that important.
Faster! Faster! Faster would be better!
West coast best coast, east coast least coast.
Seriously ... these outages are usually caused by the organization itself, either by an immediate technical mistake, or when a minor glitch cascades into something major due to a design flaw.
I'm sure Amazon is a constant target of hackers, both pimply faced youths and of shady state-sponsored black-hats. But taking out Amazon isn't a very interesting goal.
Yep, these are the people that really should be in services like AWS or Azure as they simply don't have a good understanding of technology or what it takes to provide HA/DR (not that AWS seems to have done a good job either with this mess).
Brought to you by the department of redundancy department.
Nobody expects the Spanish Inquisition.
I don't read your sig. Why are you reading mine?
Well, obviously, Amzaon's redundancy is either useless on non-existent, so I don't know what you think you're paying for.
I don't respond to AC's.
Well, obviously, Amzaon's redundancy is either useless on non-existent, so I don't know what you think you're paying for.
There's a difference between durability and availability. I'm mostly interested in durability for objects I store in S3, but for when I care about availability, I mirror them across regions.
Omaha: We Don't Coast
(Somebody actually got paid to think up that stinker.)
I was working with a customer today that was using opsworks to provision into us-west-2, and it was failing as well. Did the issue show up on the dashboard? Nope. Were they impacted in a DIFFERENT zone than what had the issue? Yep. As much as they would like to say, there were issues across the board as a result of the dependency on us-east-1.
Interestingly around the same time the company I work at which does not use AWS had several of its sites go down. The site I work out of was down for a few minutes, unsure how long the others were down.
When you rely upon someone else to handle the shit you should be handling, this is your just reward.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Apple is one of the largest customers being hit...
Apple iCloud and iTunes services were intermittently going offline the whole morning yesterday. I had trouble viewing my TV episodes.
I only use pCloud.