Slashdot Mirror
Amazon Explains Why S3 Went Down
Angostura writes "Amazon has provided a decent write-up of the problems that caused its S3 storage service to fail for around 8 hours last Sunday. It providers a timeline of events, the immediate action take to fix it (they pulled the big red switch) and what the company is doing to prevent re-occurrence.
In summary: A random bit got flipped in one of the server state messages that the S3 machines continuously pass back and forth. There was no checksum on these messages, and the erroneous information was propagated across the cloud, causing so much inter-server chatter that no customer work got done."
a single bit?! I think there are some serious design deficiencies ...
S3 is a total slut.
They need to start using Erlang more. It's designed specifically for building highly-distributed, concurrent systems that must scale to millions of transactions per minute. So it's a natural fit between Erlang and what Amazon is trying to offer with their S3 service.
I think Erlang's cool and all, but it's not the magical bullet that will solve this. It's still possible to have information corrupted during message passing between processes in Erlang (say, as the result of an intermittently failing network switch) as it is in any language.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
For those who don't know what you're referring to, like the AC who commented: search in this for "evil bit".
"National Security is the chief cause of national insecurity." - Celine's First Law
Cosmic Rays perhaps? I guess they could line the room with lead, or simply re-market S3 as a Neutrino detector. :-)
It must have been something you assimilated. . . .
By 11:05am PDT, ..., the system's state cleared.
At 2:57pm PDT, Amazon S3's EU location began successfully completing customer requests.
So WTF happened during the four hours between 11:05AM and 2:57PM?
And... have they learned nothing from the TIBCO fiasco?
Other large businesses could learn a lot from Amazon's example.
How often do you have the problem really explained to you, an apology, and a reasonable set of changes to stop it occurring again?
Most businesses would never explain the root of any problem. They simply list "hardware issues." And they NEVER say sorry anymore - supposedly it opens them up to more liability or something.
If I was an Amazon customer I would be happy with their explanation and apology even if obviously the downtime is still an issue.
was trying to hold onto a man?
I'm just guessing here.
I remember at least one similar incident:
Early (or earlish) arpanet, the network controllers did not implement checksum on messages. A bit flip caused the whole shebang to keep on forwarding the same message over and over, to the point that nothing else would flow, not even a reset command. Some one had to be sent to the culprit box and manually reset it.
I remember this being discussed on some technical circles for a while (I think Software Engineering Notes from the ACM had a go on it).
It has been generally well-known for a number of years now that any time you have a large cluster you cannot count on hardware checksums to catch every bit flip that may occur during copies and transmission, particularly with consumer hardware which has many internal paths with no checksums at all. Google learned this the hard way, like the supercomputing people before them, and now like Amazon after Google. And some of the better database engines also do their own internal software checksums as well to catch uncaught errors introduced as the data gets copied across the silicon, disks, and network -- it is one way they get their very high uptime and low failure rate.
It does not reflect well on the software community that most people *still* do not know to do this for very large scale system designs. The performance cost of doing a software CRC on your data every time it is moved around is low enough that it is generally worth it these days. If your system is large enough, the probability of getting bitten by this approaches unity. Very fast implementations of Adler-32 and other high-performance checksum algorithms are widely available online.
In 99.9% of cases it isn't the protocol that causes a server crash and instead it is the way that the protocol is implemented.
This story is another example of that. Although they're fixing it by changing the protocol spec' that is really just a much cheaper way of resolving the core issue (e.g. That any input shouldn't ever cause a crash - corrupt or otherwise).
Programmers need to learn something but I think the real lesson here is simply that input over the network cannot ever be trusted. You should assume that it is corrupt, untrusted, or wrong.
This message is written by one that writes real parallel, distributed and concurrent code (they are not all the same):
Erlang or any other functional language will not account for lack of good design. If you have a good design with the right concerns you can implement in Java, C, Fortran, ASM and if done right, it will work.
I'm sick of hear "Erlang is THE solution". It is not. Good design and implementation practices are.
Vulnerabilities of Network Control Protocols: An Example, published in January 1981.
What do they say about those who ignore history?
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
It provideRS? PROVIDERS?!?
I'TS PROVIDED!
Comment removed based on user account deletion
Programmers need to learn something but I think the real lesson here is simply that input over the network cannot ever be trusted. You should assume that it is corrupt, untrusted, or wrong.
Well, I have to say I'm guilty. I just assume that all the checksums at the various levels of TCP/IP make sure the data that comes over a socket is the same as what went in.
Of course, if I was working on a critical/must-never-go-down system, I'd maybe be a little more paranoid.
Get your own free personal location tracker
No. Even if the end user is not malicious or stupid, the network itself will not always transmit accurately. Interference or signal degradation on the lines, or a dodgy switch can screw things up.
which is totally what she said
I think the real lesson here is simply that input over the network cannot ever be trusted.
It is their network, and S3 is built on tech which is explicitly designed to not have any kind of security built-in. The security is applied at the API level, but any misbehaving machine within the S3 cluster could cause some serious damage.
I actually agree with this philosophy, to an extent. After all, this is essentially a large number of computers acting as a hard disk. How would you approach talking to a hard disk in your own machine? Do you assume that everything is corrupt, untrusted, or wrong?
Don't thank God, thank a doctor!
What Solid Snake Simulation?
"sudo rm -rf your-face"
Anybody else noticed that these major problems always occur when no one is around to fix them?
As someone who worked at Amazon as a software engineer for over three years in various backend areas, I can say that without a doubt, Amazon's code and production quality is so horrible, that it's hard to believe.
Engineers carry pagers and, in many groups, are constantly paged. The only thing that keeps the systems running is a bunch of junior engineers responding in the middle of the night, fixing databases, bouncing services, etc., etc. Engineers are rarely, if ever, given the chance to actually *fix* things, they're just supposed to band-aid them up.
And, here's a big secret for you: When I left Amazon a little over a year ago, no development groups internally were even using EC2, S3, SQS, or any of the other web services they sell to you. They make it sound like you're using the same high-end services they use to satisfy tens of millions of customers. They're not.
I hafta wonder if the bit flipped due to a bad RAM stick?
Nothing specific about *what* caused the bit to flip.
This comes to mind only because bad RAM on a new server at work caused installation of a stock Perl module to throw excessive errors during the XS compile phase - the same package installed without error on an identical machine 20 minutes earlier. Took over an hour before we realized it was probably hardware. Memtest86 quickly turned up the problem.
Would hashes and the like protect against RAM suddenly going south? Wouldn't any piece of data that passes through main memory be vulnerable to corruption? Makes me wonder why ECC memory isn't being used much anymore... we have various flavors of RAID to protect slow memory from corruption, but not many machines I see have ECC anymore.
O lord, bless this thy holy hand grenade, that with it thou mayest blow thine enemies to tiny bits, in thy mercy.
See RFC 3514 http://www.ietf.org/rfc/rfc3514.txt?number=3514
So the whole cloud is in trouble if one node starts spewing nonsense? So much for redundancy. Amazon developers would be well advised to read up on the "Byzantine Generals" problem.
I work for a company that uses Amazon S3 for our customer's data storage for the same reasons that many other companies do - they're reliable and inexpensive. We have a couple hundred terabytes of data stored on Amazon's servers and, aside from this one instance, we haven't had a major problem in three years.
Because we're in Seattle and a few blocks from Amazon's headquarters, we got a personal visit last week from one of the senior managers of Amazon's hosted platforms group. In addition to being able to ask him all kinds of great questions about how they do their business and what technologies they employ that we could also use, we got to ask him about what happened.
He was completely open and honest about it. He knew that we, like every other Amazon S3 customer, had suffered and that some of us had lost a Sunday to dealing with customer complaints. He apologized and told us that they were taking steps to make sure it wouldn't happen again.
Amazon has handled this very well and we will continue to be a customer of theirs.
---
Five nines allows for over eight hours of downtime a year.
I thought that was caused by the bouncy-ball "gift" from the Great Collector. (He thought it was funny as hell....)
"The server is down, purple monkey dishwasher"
We reserve the right to serve refuse to anyone. -management
Yes, or at least we should. Both zfs and ANSI T10-DIF (hardware) assume that there will be silent errors at the storage level and make changes to accommodate them. The funny thing to me is that ANSI T10-DIF takes the same approach the mainframe made 28+ years ago and use 520B blocks instead of 512B blocks.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Yes, or at least we should.
Let me be more specific:
ZFS assumes everything could be corrupt. It makes no assumptions about wrong or untrusted. ("Wrong" being defined here as human error, so that it's not just "corrupt".) And I think that's about all a storage layer should do.
ZFS also isn't the only way to do this. It's perhaps the slickest, but there's still things like the bad-block relocation layer in Linux, and there are higher-level software layers, like Git.
Now, yes, S3 dropped the ball -- there was a point where they missed "corrupt" on that checklist. But I can see how they might have made the simple leap from "trusted" to "assumed non-corrupt".
Also: Consider that this is not the data. This is the communication layer. You might well assume that your data is fine, but what about your northbridge? How do you know the data in RAM is good? (Does ZFS make any allowances for that?)
Don't thank God, thank a doctor!
I thought that was caused by the bouncy-ball "gift" from the Great Collector. (He thought it was funny as hell....)
He apparently thought we WERE hosting an intergalactic kegger down here.