The Simian Army and the Antifragile Organization

CowboyRobot writes "ACM has an article about how Netflix conducts its resilience testing. Instead of the GameDays used by sites such as Amazon and Google, Netflix uses what they call The Simian Army, based on the philosophy that 'Resilience can be improved by increasing the frequency and variety of failure and evolving the system to deal better with each new-found failure, thereby increasing anti-fragility.' While GameDay exercises are like a fire-drill, with scheduled exercises where failure is manually introduced or simulated, the Simian Army relies on failure in the live environment induced by autonomous agents known as 'monkeys.' Chaos Monkey randomly terminates virtual instances in a production environment that are serving live customer traffic. Chaos Gorilla causes an entire Amazon Availability Zone to fail. And Chaos Kong will take down an entire region of zones. 'What doesn't kill you makes you stronger' and Netflix hopes that by constantly protecting itself from internal onslaught, they will become increasingly 'anti-fragile — growing stronger from each successive stressor, disturbance, and failure.'"

Comments

no wonder nobody takes Netflix seriously

[ebno-10db]

No wonder nobody takes Netflix seriously. What kind of tech company worries about things like reliability and robustness? That's soooo 20th century. Everyone knows that if you have more than 90% availability or too low of a bug rate it means you're not agile enough and you can't be one of those amazingly innovative social networking outfits.

Mongolian Horde

[girlintraining]

The problem with this, is that it's still programmed failure. In my experience, hardware or software faults, or combinations of both, are not nearly as effective as plain old human stupidity. Oh, and government action. There is no disaster recovery plan for "Here's a warrant. Give us all your shit." There is a similar lack of recovery options for human stupidity. And let's be honest: It's more abundant in the universe than hydrogen, and infinitely harder to defend against, precisely because stupidity is far more cunning and unpredictable than intelligence could ever hope to be.