An Incorrect Command Entered By Employee Triggered Disruptions To S3 Storage Service, Knocking Down Dozens of Websites, Amazon Says (amazon.com)

Amazon is apologizing for the disruptions to its S3 storage service that knocked down and -- in some cases affected -- dozens of websites earlier this week. The company also outlined what caused the issue -- the event was triggered by human error. The company said an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended," the company said in a press statement Thursday. It adds: The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.

Comments

Fucking interns

[xxxJonBoyxxx]

>> wrong command

Sure, blame the intern actually typing the command.

More seriously, perhaps it's time that utility Clippy-ed up. As in "I see you're about to kill thousands of servers. Type YES to proceed."

Re:Fucking interns

[squiggleslash]

"alexa take down s3 servers a b and c"
"OK, taking down s3"

AWS Internal Help Desk

[EmagGeek]

The worker called the AWS internal helpdesk and the BOFH on the other end said "Okay, log in as root and type this... rm -rf slash... that'll fix it"

Re:AWS Internal Help Desk

[Lisias]

No. He is splitting the command in more than one line. The next input would trigger the killing frenzy.

Not enough UNIX, as it appears.

playbook?? This is my data not a football!

[Joe_Dragon]

playbook?? This is my data not a football!

Transcript

[93 Escort Wagon]

Enter command: DELETE ALL SERVERS
Confirm that you wish to delete all servers: YES
Are you sure? YES
You really wish to delete all servers? YES
I cannot find a predefined scenario under which all servers are removed. Do you wish to abort? NO
Please enter administrator command override to begin deletion of all servers: ZERO ZERO ZERO DESTRUCT ZERO

Re:Transcript

[sky_khan72]

Once I worked in a software firm. They said they got rid of the one feature after they got support calls for that irreversible destructive operation which you must enter something like "YES I UNDERSTAND. DELETE ANYWAY" to proceed.

The employee name was revealed.

[140Mandak262Jamuna]

Looks like Amazon has very strict sign off requirement. After entering the command, the system asked for his name to be logged for the audit trail.

His name was Robert `); DROP TABLE S3-subsystem; --