Slashdot Mirror
An Incorrect Command Entered By Employee Triggered Disruptions To S3 Storage Service, Knocking Down Dozens of Websites, Amazon Says (amazon.com)
Amazon is apologizing for the disruptions to its S3 storage service that knocked down and -- in some cases affected -- dozens of websites earlier this week. The company also outlined what caused the issue -- the event was triggered by human error. The company said an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended," the company said in a press statement Thursday. It adds: The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.
>> wrong command
Sure, blame the intern actually typing the command.
More seriously, perhaps it's time that utility Clippy-ed up. As in "I see you're about to kill thousands of servers. Type YES to proceed."
The worker called the AWS internal helpdesk and the BOFH on the other end said "Okay, log in as root and type this... rm -rf slash... that'll fix it"
playbook?? This is my data not a football!
Well that all sounds easy enough
Enter command: DELETE ALL SERVERS
Confirm that you wish to delete all servers: YES
Are you sure? YES
You really wish to delete all servers? YES
I cannot find a predefined scenario under which all servers are removed. Do you wish to abort? NO
Please enter administrator command override to begin deletion of all servers: ZERO ZERO ZERO DESTRUCT ZERO
#DeleteChrome
It's the "-f" that's scary. Heck, I "rm -r" at least once a month, but when that "-f" is needed, then it's double and triple check time, proceeded by a feeling of dread as my finger depresses the return key.
Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws-Plato
The big problem is not the US-EAST-1 S3 outage.
The big problem is all the other Amazon "special sauce" that blew up when US-EAST-1 S3 went down, which means Amazon has not adequately made their own services reliable with multi-AZ/multi-region resiliency.
Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.
So Echo/Alexa was down because it depends on Lambda, new subscriptions to AMI software, Simple Email Service, etc.
... we never, NEVER typed such critical commands. They were always entered into a script, and the script double-checked by a second set of eyes. While we did have some minor inconsequential errors, we never had a major error because of mis-typed commands.
Speaking as a Sysadmin that has been there, nothing compares to the horror of realizing that the split second it took between hitting [Enter] and aborting with [Ctrl-c] was enough to blow up half the production environment.
This is why all potentially very dangerous commands should default to "--dry-run" and only execute with a "--force" switch.
Realy going to claim redundant sites for static data is hard? Eventually consistent databases are a thing has been for a long time outside of some very specific niches how much stuff really needs ACID transactions.
And yes I've built these many times well before the cloud was a "thing". Using a single cloud provider for anything is a risk the same reasons we use multiple data centers in different parts of the country/world since before the internet allowed commercial traffic and probably before that (no direct experience but the greybeards of my youth told stories).
No sir I dont like it.
i've seen someone once change the usable memory of SQL server down to 1/10 the physical RAM by accident cause he thought he was so awesome and only used sql for changing configuration options
why i like the almost dumb proof GUI where you can double and triple check visually before you do something that can take a dozen applications offline
"I must have put a decimal point in the wrong place or something. I always do that! I always mess up some mundane detail!"
Nobody expects The Spanish Inquisition!
His name was Robert `); DROP TABLE S3-subsystem; --
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Before my company moved from internally managed email to office365 managed email, the email service was highly reliable. But now, Microsoft unilaterally deleted half of our entire corporate email history due to some internal mistake. It was able to restore most (if not all) of the deleted email, so we narrowly avoided a disaster.
But this kind of stuff is a disaster waiting to happen that too many management-level boneheads seem to either not understand or not care about until it's too late.
Anyone relinquishing control over their infrastructure to unaccountable third parties needs to fired ASAP, and be replaced with someone who isn't a complete and utter moron. The mine is littered with dead canaries, and too many responses are of the line, "that won't happen to our canaries. Let's forge ahead."
I would say most stuff requires ACID or at least continuously consistent databases (you don't always need transactions or atomicity) and eventually consistent is a niche. Most 'eventually consistent' systems I've seen have an entire layer on top to make sure the data is consistent.
Anytime you do a financial transaction of any sorts (free or not), you need a consistent system or risk someone being able to manipulate the data. Obviously, some developers don't really care at first since eventually consistent updates are fast enough initially. But once they realize the mistake they made, an entire layer of patchwork gets written to make it behave like a rational database again.
Custom electronics and digital signage for your business: www.evcircuits.com
Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended
oh, nevermind.
What do color blind admins do?
AWS should use the cloud, that way when one server goes down the load is picked up seamlessly by another one with no downtime ...... Oh Wait. Never Mind
**Life is too short to be serious**
Now Amazon can sell AWS prime. If you are a susbcriber of AWS prime we will check "Twice" before removing your servers. That should boost the profit somewhat
**Life is too short to be serious**
> For reliability/safety, you automate only that which is guaranteed to be safe. The more reliability/safety you want, the less you can automate.
My experience is the exact opposite. When I write software to automate something, that automated procedure is planned and reviewed, then undergoes unit testing, integration testing, and acceptance testing. When I do something by hand - well you better hope the phone doesn't ring while I'm in the middle of it because if I lose concentration for a moment mistakes are quite possible. My boss agrees; the other day I mentioned I was doing something manually and he cocked his head and asked "manually? Isn't that subject to typos and other errors?"
Oh, be careful, friend! I made the grave mistake of suggesting on Reddit that we've kinda/sorta/maybe become too enamored of CLIs and that just MAYBE a GUI MIGHT have prevented this, and I got hammered mercilessly.
You don't want to say anything that doesn't equate to worship at the feet of the almighty, great and awesome CLI around the wrong people.
If a pion (n-) collides with a proton in the woods & noone is there to hear it, does lamdba decay into the source pa
Maybe 'remove one server' should be a big green button, and 'remove many servers' should be a big red button.
What do color blind admins do?
Take down all the S3 services in the US-EAST-1 Region :P
Well that all sounds easy enough
Well, computers are easy. A child can program one. That's why you should always hire the cheapest IT workers you can get.