Slashdot Mirror
Dark Day In the AWS Cloud: Big Name Sites Go Down
An outage of one company's servers might only affect that company's customers — but when a major data center for Amazon hits kinks, sites that rely on the AWS cloud services all suffer from the downtime. That's what happened today, when several major sites or online services (like Instagram and AirBnB) were knocked temporarily offline, evidently because of problems at an Amazon data center in Northern Virginia. From TechCrunch's coverage of the outage: "The deluge of tweets that accompanied the services’ initial hiccups first started at around 4 p.m. Eastern time, and only increased in intensity as users found they couldn’t share pictures of their food or their meticulously crafted video snippets. Some further poking around on Twitter and beyond revealed that some other services known to rely on AWS — Netflix, IFTTT, Heroku and Airbnb to name a few — have been experiencing similar issues today."
How is it that AWS is less reliable than the 4 Windows machines I get stuck managing? One of which has had a failed CPU for a few years now ... yet its still going.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
I thought this might already exist, but I'm not finding it with a quick Google search. Seems like it's a thing that could get ad views from some decent IT audiences.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
When morons can't watch TV (or equivalent) they fuck. 9 months later you'll see a birth rate spike.
In Soviet Russia, company's customers go down on YOU!
One of the features of AWS was supposed to be the ability to reroute everything to a different datacenter if one goes down. I know I read that somewhere back when AWS was first starting up. You don't think they lied, do you?
In Soviet Russia, company's customers go down on YOU!
so dirty...
Website Hosting
That went down and I think it ate some files with it. Just before the crash my client reported 103 files being removed. They weren't by me.
Anonymous Cowards generally receive no replies because you're a coward and I'm a bitch
That's expensive. "Cloud" hosting services cost about 1.5x traditional hosting. When you want multiple locations("regions" in aws) you need to pay for resources in each additional region, then pay another cost to provide that failover. Cloud hosting is great, but it's nothing it does is new or cheaper than hosting 10 years ago.
Website Hosting
You can do it with AWS, no problem. Only one region was affected this time, other regions are OK.
That kind of redundancy is great, but not if you have a connectivity issue and your load balancers are impacted which is what happened here. Also, moving all traffic from one DC to another is a major shift; so depending on the problem and how long it may take to fix, it might not be worth it. Shifting everything over and back is a great feature to have, but it does come at a cost.
I thought for sure the first comment would be "I'm on to you NSA...down time for service "upgrades" " I'm disappointed in you my tin foil hat wearing brethren.
You just have multiple DNS records for each service, and the client should move on to the next if one is down.
Sig: I stole this sig.
I've run servers on both Amazon and Rackspace for several years now and I can't recall a single instance of Rackspace having an outage. On the other hand, Amazon seems to have major issues at least 2 or 3 times a year. Is this stuff tracked anywhere?
"Don't teach a man to fish, feed yourself. He's a grown man. Fishing's not that hard." - Ron Swanson
Were there any problems with Amazon.com? You'd assume they use their own service.
Unfortunately, "should" is rarely "does". If a brower receives multiple IP addresses for a name, it doesn't try them in turn, it just tries one.
The real "Libtards" are the Libertarians!
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again. It's foolish-but a lot of companies act this way.
Somehow cloud hosting is taken as the silver bullet to prevent outages-it isn't. You still have to architect things the way you would normally if you're looking for things like disaster recovery, high availability, etc...etc..
Assuming you mean traditional round-robin A records, the timeout(s) you still have to suffer through would kill your latency.
If your talking about DNS providers (disclaimer, I work for Dyn) with advanced features that detect a failover event occurring and will only serve healthy A records, then that is a different story.
What is going on? I don't buy it. While I get that you can't tell when the NSA has tapped the line I would imagine that things might go down in such instances. Something has to go down before they cut the line unless there are multiple entry points maybe. However I wonder if this has something to do with the way things are being done. That is there not tapping the line any more. Rather they are force implementing taps that provide access to specific data types. For instance now they can do more than just search for strings in users email. Now they can see a users facebook page as they user sees it for example instead of just a series of texts.
I gues stuff goes down. But not at the rate in which major sites are going down. If there is a logical explanation of some kind that impacts everybody (sun bursts radiation type thing) then please... provide it. But they all seem to be giving vague answers to the reasons the sites have gone down. Ebay it was 'regular maintenance' gone amok (I do believe that was scheduled, but others haven't been, I don't think).
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again. It's foolish-but a lot of companies act this way.
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
If * I * have to worry about that stuff then I might as well just do it myself and not give my money to Amazon.
How do you do it automatically? It's simply not possible to transparently replicate arbitrary VMs across geographically distant datacenters (lightspeed and all that...).
However, AWS provides tools for developers to do it.
Re you need to pay for resources in each additional region. ...the regional services should be good?
Why the lack of power and real optical links that where regional, power distinct.
Is this like the idea of linking to a site/city/state/regional 'ring' many times? Very safe from any local cut/drop, cheap, but still very dependant on one geographic provider?
You also have a submarine communications cable (France to the USA) on the way for that State??
Domestic spying is now "Benign Information Gathering"
Middle management in their luxury SUV/sedans sit in daily commutes behind buses with descriptive 'cloud' ads. The upgrade message filters back to their bosses over time?
Domestic spying is now "Benign Information Gathering"
Yes. Rackspace even has an outage on their main website that lasted *days* just few months ago, if you wanted to access it via IPv6. Sadly, there was not easy place to report the outage. The technical contact in whois is something at netnames.com? So I just ignored it.
Anyway,
https://status.rackspace.com/
lots of reports of small issues. You should know this stuff if you are running an instance on their hardware!!
That functionality is there -- I use it in my own deployments. The thing is, it's not automagic. You have to actually architect your application to take advantage of AWS features.
TANSTAAFL
That has never been the "contract" with cloud. Amazon does not, and cannot, understand the architecture of every one of the multitude of applications running in their cloud. You can pay to have that kind of support from various companies (maybe even including Amazon) but it's not what "the cloud" is.
The functionality is there for you to make an extremely robust application in AWS -- if you actually take advantage of it and if it's necessary for your business needs to do to that much effort/expense.
yeah, but cloud is sold as this super cheap way to compute and have five nines reliability
Outages with AWS and cloudy friends are becoming so common it's almost a non-story at this point.
this is my sig
Oh it's possible to do just rather expensive to do well. Disk based bits don't work as sync writes past a region take far to long. Higher up the stack you can deal with 35-70ms of network latency. But now it's not mysql with any old crappy php code.
AWS is PHB buzzword like IBM a decade and a half ago it makes the VC guys happy that you fixed your scaling issue. In reality everybody else's scaling issues now impact you.
No sir I dont like it.
...nothing it does is new...
Ahem, it siphons additional funds from customers ;-)
Well, right now I have 500 machines running some heavy calculations in multiple AZs. Works perfectly fine, we have noticed the recent problems but simply stopped using the affected region (us-east-1) for the time being, shifting our calculations to other regions.
AWS is really great at scaling. It's better than anything else on the market, but it does require a lot of work.
Most clients either won't move on to a second IP at all or will only move on once the OS times out the first TCP connection. And OS TCP connection timeouts are long enough that most users won't put up with them for interactive services.
A better strategy is to put a DNS server in each datacenter, make the TTLs short and set things up to automatically remove records if a server goes offline. This works much better because DNS fallback timeouts are much shorter than TCP connection timeouts.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
No, you have to manage your own redundancy and failover on AWS. Look at all the effort Netflix has put into programming failover and stress testing and yet they still have frequent outages with AWS.
this is my sig
Would make a good site, a historic long term heat map of server outages. A lot of tech press to search back into, thankfully you can buy into digital press databases :)
Domestic spying is now "Benign Information Gathering"
either you don't speak English or you need to take your meds. no offense. so i'll try muddling a reply together for you.
There are many ways to setup remote failover systems. Most of them rely on some type of heartbeat system where there's a "heartbeat message" which they all send each other periodically, and if the current Active goes out of response for too long the others choose one to take over. So it doesn't matter if they're all in one room connected with a single switch, or spread all over the planet.
The real rub for any mechanism is DNS... if the primary server your FQDN points at drops then you might have redundancy but most people won't be able to take advantage of it. With more manual mechanisms (such as telling users "If our primary site goes down, try here instead!") that's not as much of a concern, just a PITA to keep track of.
Isn't this why AWS offers multiple regions?
Such large sites should understand that having multiple availability zones means nothing if the zones are all in the same region. Oh, and your application would need to be designed for failover.
In addition, when looking for high-availability, you don't segregate your audience to individual regions. You let the working regions take over for you.
Or spend the extra money and set up your own co-lo arrangement.
Kriston
Most modern browsers do, indeed, try the next address. It' s a browser feature, though, not an official standard.
AWS Status Dashboard?
I know this is /., and people here don't like to read, but did anyone actually read the status dashboard posts?
This issue was limited to a single AZ, effected only a small number of machines, and was specifically an issue with added latency in EBS volumes. And Amazon completely resolved the issue in 4 hours.
So, call me crazy, but didn't they do exactly what they are supposed to do? Also, AWS quite clearly states that any given AZ *might* fail. Hence, if you want any sort of high-availability, you replicate across different AZs.
Plus, I have 10+ EC2 instances, and a number of other resources with AWS, and none of them were effected by this outage.
"cloud" is sold as a *convenient* way to compute, where it's quick to add resources when needed so you can start small and scale up (and down) with demand.
It is *not* generally considered a cheap or particularly reliable solution. So far at least none of the cloud providers are offering five nines--if you want that, you should (for now at least) jbe looking at enterprise/telecom gear.
As a cloud customer, reliability (currently at least) is up to you. If you want the extra reliability of running instances in multiple availability zones then it's up to you to pay for it.
The point of the cloud as it stands currently is not that it's cheap or reliable, but that it's easy to scale up/down with demand.
That things like this will happen with a cloud infrastructure are obvious. That the reliability claims made by the cloud providers are fantasy is also obvious. As soon as they start to do "uptime or else" (meaning you get tons of money as downtime compensation), things may be different. but they will not do that. At this time, the only thing you can do is change to a different cloud provider, which will have the same issues. Uptime guarantees without penalties when failed to meet them are worthless.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
It depends which data center you're in. PortableApps.com has been hosted at Rackspace for years and we had multiple major outtages due to ongoing power issues in the Dallas data center in 2009. The switch from grid to ups was failing and would take the whole wing of the data center out with every server crashing hard. It would take quite a while to come back up. Then we'd have to wait hours for the Rackspace folks to rebuild our corrupted database (fully managed account on a dedicated server). It happened two weekends in a row in June and one other time if I recall correctly, basically costing us a full day of downtime each time.
Portable versions of Firefox, GIMP, LibreOffice, etc
public cloud services as "the future". I will never risk my corporate data uptime and reliability to some "location in the cloud". I'll stick to private clouds (VMWare/VCenter) where I have control of both hardware and software and reliable failsafe systems. At least then if I have downtime I also have accountability and predictability. They same cannot be said for cloud providers and no matter what anyone says once the data leaves your hardware, you have lost that control.
The first rule of diversification: don't put your eggs into correlating baskets.
In this context it means that if your primary is on AWS, then your secondary must be on Rackspace or whatever - NOT other AWS.
Netcraft?
If you want news from today, you have to come back tomorrow.
Supposedly the load balancer problem did not affect LBs that have backing hosts in two availability zones according to the article. The major question is... who runs everything in one availability zone? You're not supposed to do that for high availability sites.
Shouldn't this, technically speaking, be a "bright day" or a "sunny day"? After all, that's what I call it when the cloud-coverage breaks around here.
I had an outage in IAD just two weeks ago. Connectivity failure on several aggregates affecting many customers. Rackspace shill much?
For believing and investing in some handwavy concept called 'cloud' where you abrogate responsibility take the iOS view (it Just Works) of technology.
I want to delete my account but Slashdot doesn't allow it.
Depends on which "future" you are talking about. The future where the bulk of personal data is stored on the cloud to be shared across devices and with friends, family and authorized services is one I think is bound to come to fruition.
The future where Corporations put their core infrastructure into the Cloud is not one I ever recall anyone talking about.
When you want multiple locations("regions" in aws) you need to pay for resources in each additional region, then pay another cost to provide that failover.
Or you can have storage in those regions prepped to failover, with no other resources provisioned. When failure needs to occur, you start spinning up the instances in the other region.
It does require planning; you can reroute But you don't get that automatically; it requires work and preparation.
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again.
But that was one of the big promises of "the cloud": that you'd never have to worry about the nitty-gritty of network administration again, your provider would handle all that for you. If that isn't the case, then you gain nothing and might as well host the data yourself.
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
Boy have you been fed a line. Read the SLA. If it's not in there; then you don't get it.
If you think the cloud provider is clustering your instance and giving you HA; then AWS is not for you.
Amazon provides availability zones you can provision separate instances storage and networks in. If your application cannot survive the failure of an instance and the failure of an entire availability zone, then you don't have HA, and Amazon won't give it to you -- your app may be inappropriate for AWS, if HA is required.
"In Soviet Russia, company's customers go down on YOU!"
Now we know the truth about why Snowden went there...
I love stacking my barbecues in the shed at the end of summer - you can't beat a bit of grill on grill action.
Now add cooling, power, generators, physical security, a SAN, a virtualization platform, and multiple failover sites.
"nothing it does is new or cheaper than hosting 10 years ago."
Welcome to the wonderful world of marketing. Sell people what they already have for 50% more.
I love stacking my barbecues in the shed at the end of summer - you can't beat a bit of grill on grill action.
Configure the warm resources in the other region to constantly monitor the primary. If the primary goes down, they automatically activate the secondary.
now we need to go OSS in diesel cars
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again.
But that was one of the big promises of "the cloud": that you'd never have to worry about the nitty-gritty of network administration again, your provider would handle all that for you.
There are many different flavors of "cloud" computing - if you throw your app at a cloud provider and blindly expect them to make it highly available, then you'll get what you deserve. There is no end of cloud solution providers that will be happy to help you architect your app for whatever level of redundancy you want. But it's not going to be free.
Amazon does let you get rid of your network admin and concentrate on managing the servers. No need to worry about BGP, buying bandwidth from multiple redundant providers, buying and administering your own firewalls, network switches, routers, etc.
But you still have to manage your servers. Amazon will help you with multi-AZ redundancy for things like MySQL.
If that isn't the case, then you gain nothing and might as well host the data yourself.
That's depends heavily on your use case. If you have a relatively small number of servers, or have large demand spikes, Amazon can be much more cost effective than hosting your own servers. If you have hundreds of servers and keep them busy all the time, you can probably save money by doing it yourself.
But if you have dozens of servers, then it's likely that you'll save money with Amazon over buying your own servers, network gear, a SAN, backup solution, hardware service contracts, etc.
But you have to architect your application properly. We have our core servers split across multiple AZ's with the database replicated across those AZ's. We don't trust our failover/failback scripts enough to make it automatic, so we have a simple web interface to let anyone on the tech team do the failover. The only impact we saw in this outage was higher latency and timeouts to some of our app servers, but our database was not in the affected zone, and Amazon's load balancer correctly routed traffic to the servers in the good AZ.
Additionally, we have a warm spare running in a different region - the servers are kept up to date with data, but they are running in smaller instance types than we need to run our app, do to a regional failover, we'd have to reboot them into larger instance types (our app startup scripts already tune memory parameters to take advantage of the greater amounts of RAM in the larger instances), then repoint DNS.
Chances are that there are no providers that offer a true 99.999% uptime. If you demand that, you need to be building your code to run in a HA cluster with nationwide dispersion. (For reference, you get 5.25 minutes of downtime across a whole year).
99.999% uptime is also completely unnecessary, but sounds really good to management until you talk cost.
If people can connect to one another even the smallest of voices will grow loud.
--Serial Experiments Lain
The right wing talking heads on TV would be squealing like stuck pigs. They would be screaming about "gubment" waste and incompetence, and start floating bills to privatize the FAA (or whomever). You'd get the same response on Slashdot as well.
Meanwhile in real life AWS, Google, and NASDAQ have all had dramatic failures in recent weeks. Although NASDAQ got a fair amount of coverage, and Google got some mention, AWS has been pretty much below the radar for the mainstream media. No one is making dramatic statements on TV about how Google is run by a bunch of idiots, or NASDAQ, a quasi-governmental entity, should be nationalized, because when it fails the entire economy is as risk. As far a critical comments, it's the sound of crickets.
Clearly, there is a double standard. When there are problems with technology in the public sector, it's all hostility and table thumping. Similar failures in the private sector are treated like natural disasters completely beyond human control. According to common rhetoric, the private sector is always better then the public sector. Yet when the private sector fails, no one ever compares it to the well functioning public sector.
There is clearly a lot of hypocrisy in bashing the government. A lot of political power is at stake, and along with that goes a lot of money. This situation makes some people very happy, because they are getting what they want, both in public policy and private profit.
Why is Snark Required?
> Amazon provides availability zones
Not always. Sometimes they provide *unavailability* zones. That's the problem. And that's what makes people think they're deceptively named.
Your head of state is a corrupt weasel, I hope you're happy.
and the operators realizing this tried to deactivate it?
> Now add cooling, power, generators, physical security, a SAN, a virtualization platform, and multiple failover sites.
6 of those things we've been doing all along anyway (OK, we non zed-heads have only been virtualising for half a decade). The remaining one AWS doesn't do for you unless you pay significantly more than the simple hosting service.
Your head of state is a corrupt weasel, I hope you're happy.
I've had tons of outages on rackspace cloud. Including systemwide networking outages.
Not always. Sometimes they provide *unavailability* zones.
That's just a play on words: they are availability zones. Just because you don't like how frequently they have a zone-wide outage doesn't change the name.
Another word for their availability zones is fault domain or failure domain.
If you don't want two things to fail together, you run them in a different region, using only resources in separate availability zones in separate regions.
So, call me crazy, but didn't they do exactly what they are supposed to do? Also, AWS quite clearly states that any given AZ *might* fail. Hence, if you want any sort of high-availability, you replicate across different AZs.
For whatever reason, many of AWS's biggest flameouts have happened at the Virginia datacenter.
Between bad weather, rickety power infrastructure, bad hardware components, poorly configured software/hardware, etc etc etc
It's like setting up your data center in the Bermuda Triangle.
[Fuck Beta]
o0t!
Except Amazon is acting as a datacenter.
It wouldn't surprise me that these outages prompt one of three decisions from their customers: 1. Switch providers, 2. Setup internal servers (internal to the company), or 3. Ghost the entire setup from one region to another.
That 3rd option is what Amazon hopes their customers do in every single one of these outages. Even though they taut 99.95% uptime (43.8 min according to Wiki). Which they've fail at least once the last 3 years. That is for EC2, I won't assume they use the same SLA for all of their services.
Cooling? Everyone has that.
Power? Yeah, you pay for that anyway.
Generators? Not as expensive as you think. I lived on a farm in the middle of nowhere once, and we had our own water supply and back-up generators.
Physical security? If you think that you're less likely to enjoy security with servers on your own site than controlled by some random third party who won't let you onto their site, who won't let you audit any of their processes, and who is almost certainly happily giving over information on request to the authorities, you're insane.
SAN/virtualisation? So, what every decent IT person has been handling since the '60s.
Multiple failover sites? Well, that would make us better than Amazon already...
I believe that claim only applied within one data center so if zone a went down zone b should pick it up. Not AWS-West. even their cloud load balancers cant balance between 2 DCs.
Sanity is the trademark of a weak mind. -- Mark Harrold
Please please don't do this. A $50 router/Linux box should be able to serve millions of people, if DNS is reasonably TTL'd. But when everyone sets TTL to 5 min, you have to buy expensive hardware or run dedicated boxes because the number of queries is so high.
Learn to love Alaska
Don't set your TTLs short. The recommended values are good enough. Short TTLs are good only for changes (i.e. TTL at 24 hours, then set it to 12 hours 24 hours before an IP migration, then 4 hours 12 hours before a migration, then 1 hour 4 hours before a migration, then 5 minutes 1 hour before a migration, then to 24 hours after the migration is done), obviously set in seconds, 86,400 for a day, 300 for 5 minutes (don't use).
For what you are stating, DNS servers at both using anycast, so the one that's up says it's the one up is a much better way to do it. The down one won't ever announce itself, and the up one will respond. That's how the Big Boys do it. No 5 minute wait, and no extra load on DNS caching servers.
Learn to love Alaska
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
Then either you're incredibly naive or you've never looked at what you get with most cloud providers.
Those £15/month virtual servers? You don't get any redundancy on those. If you're lucky, the provider will move it to a new physical host if the one it's living on breaks down, but they won't make any guarantees regarding how quickly that will happen or how automated and transparent that process is.
IME, the pile-it-high, sell-it-cheap brigade are punting exactly this. It's a whole bunch of physical boxes running something like Xen with a web-based front end but none of the work necessary to make it truly highly available has been carried out.
You want true high availability in the cloud - where even an entire datacentre going dark won't affect you? Well, then you have two choices:
- Architect your own. This means you will need several cheap virtual servers and you'll have to write your own software that accounts for all the various failure modes. Yes, this is difficult. Yes, this means you can't just fire up an Ubuntu image with Apache preinstalled on AWS and forget about it. Yes, this means it's a hell of a lot more expensive because suddenly you need to pay for lots of virtual servers rather than just one or two and you need to put a hell of a lot more work into the development process. But that was a choice you made when you went for the cheap option. Oh, you thought that because they used the word "cloud" in their marketing, that meant they'd already done all that for you? Ah.... no. Sorry.
- Contract it out to a company that has already built all this at the virtualisation level so you don't need to worry about it at the OS level. They operate a highly-available infrastructure with redundant everything and guarantees that even if something does fail, the redundancy will kick in automatically and you'll see no downtime. There are companies that offer this, but you might want to sit down with a strong drink before you look at their pricing structure. Clue: It's a hell of a lot more than £15/month for a basic virtual server.
Short TTLs are good only for changes (i.e. TTL at 24 hours, then set it to 12 hours 24 hours before an IP migration, then 4 hours 12 hours before a migration, then 1 hour 4 hours before a migration, then 5 minutes 1 hour before a migration, then to 24 hours after the migration is done), obviously set in seconds
Which works fine for planned migrations, useless for unplanned ones.
The down one won't ever announce itselft
Sure once a DNS server goes down it will stop sending responses regardless of whether you used anycast or the traditional system of just listing multiple DNS server IPs but responses it has already sent will continue to be used by clients until their TTL expires.
To make anycast useful without low DNS TTLs you would need to anycast not just the DNS servers but the servers they point to as well. However anycasting TCP based services causes broken connections if routing changes.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Netflix did NOT go down.
Well I've wondered about that myself. Yes, US-East is not only less expensive than the other zones for quite a few services, there's also AWS pricing policies that make it more attractive to use US-East for a lot of things. For example, S3 ingress data transmission charges are waved in US-East but not in the other AZs so I have other AZs replicating to US-East for disaster purposes just because of that. If US-East goes down it doesn't matter because I have apps leveraging multi-AZ availability and when US-East recovered, those services hosted there recovered just fine and replication continued just as if I had built this out in self-provisioned or co-located data centers. Application tools and services for managing this have become better and will keep getting better and better because of the economies of scale.
Sure, there's a lot of hype around cloud services but there's also a lot of benefits too for businesses looking to become more agile in responding to new opportunities or looking at ways to respond to changes in demand. It doesn't relieve you of doing your homework and making sure that the services you choose can support your requirements including those key non-functional areas of Availability and Reliability.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
One of the features of AWS was supposed to be the ability to reroute everything to a different datacenter if one goes down. I know I read that somewhere back when AWS was first starting up. You don't think they lied, do you?
That all works just fine - if you build your application to use it.
However, nobody does this, because when your coworker is working on a new feature he can show the boss when it comes time for bonuses, do you want to spend your time working on something that you can only show off when Netflix goes down? Oh, and the way that you know that your work did anything is because your coworker's new feature still works.
Reliability is a hard personal sell in IT, and that is why there is so little of it. Anybody who set up real AWS redundancy didn't bat an eye at this outage. Their load balancers were not exclusively in one data center, and when they spotted the outage they sent all the work to other data centers where they spun up a ton of capacity.
If you just deploy all of your stuff and point your DNS at one Amazon datacenter, and that datacenter goes down, well so does everything you could have used to fix the problem. Even so, if you had control over your DNS outside of Amazon you could spin up another instance elsewhere manually if your tools are designed to handle that. If you hard-coded us-east-1 or whatever into all your scripts or didn't even have scripts to completely rebuild your instance from nothing, not so much.
I was a former Slicehost user in the St. Louis data center and then was moved to Chicago after the Rackspace acquisition. Even so, there's never been so much as a blip from there in the last 5 years. Probably is data center dependent, I just never remember hearing about anything.
Friend of mine here in town owns a web business using about 9 Rackspace servers to host 700 websites and he said they hadn't had an outage in the last 8 years.
"Don't teach a man to fish, feed yourself. He's a grown man. Fishing's not that hard." - Ron Swanson
That's not a feature, and I don't recall it ever being advertised as one. You get to decide which zone and which datacenter(datacenters contain multiple zones) your instances/data lives in. If you want to replicate those to other zones and datacenters, it would be a very good idea. Amazon can't automagically do that for you though, and does not advertise that it can.
Trackball users will be first against the wall.
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff.
They are, if you pay for "all that stuff". If you only pay for some of that stuff, you don't get HA.
They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service.
Again, AWS does "worry" about such things, and yes, they do have systems in place so that there's no loss of service - for customers who have purchased that level of service.
Amazon provides availability zones you can provision separate instances storage and networks in. If your application cannot survive the failure of an instance and the failure of an entire availability zone, then you don't have HA, and Amazon won't give it to you...
Not for free, they won't, but you most certainly can buy an HA solution from them.
entertainment and social media down? who gives a shit? grow up.
I have to say with all of the big names having problems recently this has been one of the best weeks ever for the lowly corporate sys admin. Now if the company's email, file or web server--or even the coffee machine goes down, they can point to the big names that also have problems. It's great to be able to say that even at companies like Amazon, Google or Microsoft with all of their talents their servers also have problems. It's the greatest excuse ever for tripping over the power cord. And if that doesn't work, you can always blame the NSA for the typo in your email or the late TPS reports.
Thanks everyone and happy SysAdmin day! (which isn't today, but due to the unexpected outage is running late)
"no need for capital expenditure" and "minimal start-up costs" are not the same as "cheap". All it means is that you don't need to pay up-front.
It's like renting a car for a day vs buying one. If you only need a car a few times a year, renting is cheap. If you need a car every day for a decade, you should probably buy one.
There are some things where five-nines makes sense.
Disclaimer...I have worked in the telecom industry in the past.
Is that recursion or just self documenting?
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
To make anycast useful without low DNS TTLs you would need to anycast not just the DNS servers but the servers they point to as well. However anycasting TCP based services causes broken connections if routing changes.
Huh? It seems either you don't know how it's used, or you think I don't know how it's used. You over-simplified your answer to the point of technical falseness. Anycast is a routing trick to turn a unicast into a multicast-analogue (that is multi-unicast). As it is tecnically a routing function, yes, routing changes can break it. But pointing that out is like saying "breaking your car can break your car". It's a tautology. Changing routing between the anycast server and the recipient devices doesn't "break" it (though it could break certain sub-functionality, like load balancing, if you are using anycast for it), as it's essentially still a unicast to the destination at that point. It just lets you dynamically select a single "best candidate" for the DNS querry, or send it to multiple for the one that's up to respond.
You don't unicast the servers they point to, the DNS servers hold different tables. The DNS1 holds entries for WWW1 and MX1, and DNS2 holds entries for WWW2 and MX2. One should assume that WWW1 and WWW2 replicate or otherwise communicate with each other. If DNS1 is down, then DNS2 will respond. All load will go to WWW2. You don't need to anycast the servers they point to. Anycast isn't a DNS "trick". It's a routing trick used primarily for DNS (it can be used for other services, but, in practice, isn't).
Learn to love Alaska
If DNS1 is down, then DNS2 will respond.
Sure it will
All load will go to WWW2.
Not immediately it won't
Consider this scenario.
1: Initially all servers are up
2: user starts browsing your site. His DNS packets go to DNS1 since that is the closest DNS server to his recursive resolver. DNS1 returns the IP of WWW1 which is cached by the user's recursive resolver and possibly his OS.
3: site1 (including DNS1 and WWW1) goes down.
4: The user clicks a link within your site. The browser asks the OS to look up the name, the OS either looks in it's DNS cache and returns the IP of WWW1 or contacts it's local recursive resolver which looks in it's cache and returns the IP of WWW1.
5: The browser tries and fails to connect to WWW1, the user eventually gets a timeout message if they can be bothered to wait long enough.
6: The user hits the refresh button and the browser tries again but continues to get connection timeouts until the DNS cache expires the record, contacts DNS2 and gets the IP for WWW2.
(it can be used for other services, but, in practice, isn't).
And there is a reason for that. Using anycast for a stateful protocol (e.g. anything TCP based) means that routing changes that would normally be harmless (that is routing changes which mean the packets still get to you but enter your network on a different interface) can break open connections even when all your systems are up. In the worst case if routing is really unstable or if someone has done some dumb load balancing then they may not be able to complete a connection at all.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Netflix routinely touts all the stuff they do to make sure they have high availability - including loss of a whole zone - yet they are still often impacted by AWS choking. Something's not right.
this is my sig
I thought one of the pro's of using "the cloud" was that these events would not happen, as you are no longer relying on a single datacenter?
If AWS is putting several "clouds" in a single datacenter, what's the use of AWS?
/ The Arrow
"How lovely you are. So lovely in my straightjacket..." - Nny
Chelsea Manning
"The future where Corporations put their core infrastructure into the Cloud is not one I ever recall anyone talking about."
Microsoft tried to push that very concept during the Windows 2000 launch tour, back in 1999. At the presentation I went to in Los Angeles, the audience, ~1000 professional IT types, all developed identical angry scowls.
~REZ~ #43301. Who'd fake being me anyway?
You have to PAY for that. You're allowed to spawn instances and pay-per-use with righteously low rates on miriads of servers and managed services, and some of the default setups include proxies and load balancers and failovers. So, assuming you've set things up properly, and have failover resources ready to go and programmed to take over when shit hits the fan, then yes you can reroute everything to a different datacenter if one goes down.
It's not the default tho. AWS is very technical. Not like some service you can pay for and have email accounts and storage space and web space and so on. There's a full API, technical control panels, ready to use scripts, help services, and RSA keypair manager, as well as a price fluctuation setting to nab the cheap prices for your deferrable data processing, but mainly it runs on top of VM and DS instances. Not exactly your turn-key operation, but quite powerful.
Sadly, a Libertarian cannot force his views on another, and freedom cannot spread as does the cancer known as religion.
UM, do you mean Brandi Womanthing? :-)
The Virginia region simply dwarfs the other AWS regions, likely by an order of magnitude, so there's a correspondingly bigger chance of failures. No magic required.
No, not possible. A geographically remote link has latency too big to be useful for automatic replication. And that's even without considering the way to solve the 'split brains' problem.