Slashdot Mirror
Facebook Engineers Crash Data Centers In Real-World Stress Test (ieee.org)
An anonymous reader writes: In a report via IEEE Spectrum, Facebook's VP of Engineering Jay Parikh described the company's "Project Storm" -- regular takedowns of Facebook's data center intended to stress test the company's disaster recovery efforts. The first few didn't go so well, he reports. (Perhaps doing a test during a World Cup final was not such a good idea). Months and months of planning went into the initial effort, though up until the actual moment, other Facebook leaders didn't think he'd actually take out an active data center. "In 2014, Parikh decided Project Storm was ready for a real-world test: The team would take down an actual data center during a normal working day and see if they could orchestrate the traffic shift smoothly," reports IEEE Spectrum. Parikh recalls: "I was having coffee with a colleague just before the first drill. He said, 'You're not going to go through with it; you've done all the prep work, so you're done, right?' I told him, 'There's only one way to find out'" if it works. (Parikh made the remarks at this week's @Scale conference in San Jose.) Parikh says there never seemed to be a good time to perform the live takedowns. "Something always ended up happening in the world or the company. One was during the World Cup final, another during a major product launch." The report adds, "The live takedowns continue today, with the Project Storm team members coming up with crazier and crazier ambitions for just what to take offline, Parikh says.
This is totally worth it. What would happen if there was a REAL disaster (like a nuclear strike) and people couldn't check their facebook feed and post "thoughts and prayers" messages? Too terrible to think about.
Real world stress test! See if the other satellites can pick up the slack.
Good for him! Most DR exercises I've seen are planned weeks, if not months in advance. They are more of a scheduled fail-over to a redundant site and not an actual disaster recovery test.
In the event of an actual disaster, there would be no recovery.
I'm heartened to see SOMEONE does it right.
Learning HOW to think is more important than learning WHAT to think.
Thats how you should do it. Disasters dont wait around untill your not busy doing something else, they hit when they want to. Thats why i think live testing is important to see if recovery plan works or needs iteration of improvement... Best test is usually when disaster strikes right when you have hands full of something else... No time to dig up manuals, etc...
My last DR Test Plan only had 2 lines in it
Power=OFF
Execute DR Plan.
facebook and engineering aren't something I'd put into a sentence together.
And how do you think they got to handle the amount of data they did? The same comments rolled in when Walmart released its cloud service. Of course they have an engineering department.
Facebook engineers are working on a scale that most people will never see.
What's this World coming to when twitterbook has to be protected from natural disasters.
A ubiquitous service that most people have access to is one that speeds up disaster recovery. People have already used groups to organize disaster recovery efforts on small and large scales.
No one says you have to use it to upload food selfies.
I didn't know that Facebook still exists. Haven't most people gone somewhere else for their 'waste of time' needs already? But I suppose it's still important to stress test FB. Someone's grandma would be really sad if she couldn't post her cookie recipe after Yellowstone had blown up or an accidental thermonuclear war had broken out.
So Netflix has been doing this for years now... it's called the Chaos Monkey and part of their "Simian Army" that performs this kind of function but *all the time* with no schedule. This is not something FB came up with but this post seems to give them credit for this innovation. More interesting than the lack of credit to Netflix though is this adoption of a method that heavily favors a Microservice Architecture. Seeing more and more of this flexible, scalable, and highly resilient architecture and methodology being put out in industry is certainly encouraging.
Pity.
Just when you thought your privacy could be restored by the massive failure of a data center, a new center rises up from the ashes of the other to take the helm of stealing every personal detail about your life and the life of people you love.
Facebook is still up and running.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
It would be very useful for Facebook to stop announcing ASN 32934 for a few centuries as an experiment just to see what would happen.
Or permanently remove authority records for facebook.com.
You know just to see what would happen.
Months and months of planning went into the initial effort, ...
Into the take down or recovery? 'Cause the former just requires pulling a cable of some sort. :-) TFS says the team would take down a site and try to migrate the traffic, but wouldn't it be better if the disaster group and the recovery group were different teams for a "real world" stress test?
It must have been something you assimilated. . . .
Actually, this is one of the better story summaries I've seen here.
I knew what it was talking about (no unexplained mayfly buzzwords), I knew who the protagonists were, and I knew what was at stake. The only implied innovation was one of personal chutzpah, against the backdrop of an organization notorious for taking all things in collective stride (these being very, very short strides).
Working at Facebook Sounds Like Joining a Cult
At some level, I think we do indigenous people a disservice by referring to them as First Nations, freezing them in the amber of the era, as if they couldn't (and hadn't) kicked the shit out of their neighbours every bit as ruthlessly as the Spanish, the Dutch, or the British (secretly, it's a badge of honour, isn't it, to have kick-ass forbears?)—the main difference being that the European cultures brought with them a written language—so long, illiterate heathens—then, however they found the table set is assigned a positive integer (let's not even grant them "zero") to functionally signify that no form of tomahawk displacement came before.
So of course Netflix didn't invent this technique.
Clubbing a bunch of strange-looking men and taking their women is an idea that never owed much to the example of recent history, no matter how grand and savage an example your nearest neighbour might have set in the memorable recency.
First day at the new job they turned me loose on the product, told me to poke around the software and get familiar with the GUI. I took down the entire system within minutes after running a report that didn't place limits on how far back you could search. Needless to say, that got fixed fairly quickly.
There it's called "DiRT" (stands for "Disaster Recovery Test").
O btw, the Simian Army is open sourced!
But at least HP really did blow something up:
https://www.youtube.com/watch?v=bUwthF9x210
here's an idea..... how about the WHOLE FUCKING THING
and we really don't give a shit if you ever get it all back online either, so.. no pressure at all. just DOIT
Reminds me of "Chaos Monkey" from the netflix simian army.
As does pretty much any large company. I''m honestly not sure why this is worthy of being news? Any organisation that relies on IT for its business will have appropriate DR facilities and be regulary tesying. Financials are required by regulation to perform such tests. What makes Farcebooks test any different?
This reminds me of when Pee-wee Herman fell off his bike and then got up and said: "I meant to do that"