Slashdot Mirror
Facebook Engineers Crash Data Centers In Real-World Stress Test (ieee.org)
An anonymous reader writes: In a report via IEEE Spectrum, Facebook's VP of Engineering Jay Parikh described the company's "Project Storm" -- regular takedowns of Facebook's data center intended to stress test the company's disaster recovery efforts. The first few didn't go so well, he reports. (Perhaps doing a test during a World Cup final was not such a good idea). Months and months of planning went into the initial effort, though up until the actual moment, other Facebook leaders didn't think he'd actually take out an active data center. "In 2014, Parikh decided Project Storm was ready for a real-world test: The team would take down an actual data center during a normal working day and see if they could orchestrate the traffic shift smoothly," reports IEEE Spectrum. Parikh recalls: "I was having coffee with a colleague just before the first drill. He said, 'You're not going to go through with it; you've done all the prep work, so you're done, right?' I told him, 'There's only one way to find out'" if it works. (Parikh made the remarks at this week's @Scale conference in San Jose.) Parikh says there never seemed to be a good time to perform the live takedowns. "Something always ended up happening in the world or the company. One was during the World Cup final, another during a major product launch." The report adds, "The live takedowns continue today, with the Project Storm team members coming up with crazier and crazier ambitions for just what to take offline, Parikh says.
This is totally worth it. What would happen if there was a REAL disaster (like a nuclear strike) and people couldn't check their facebook feed and post "thoughts and prayers" messages? Too terrible to think about.
Good for him! Most DR exercises I've seen are planned weeks, if not months in advance. They are more of a scheduled fail-over to a redundant site and not an actual disaster recovery test.
In the event of an actual disaster, there would be no recovery.
I'm heartened to see SOMEONE does it right.
Learning HOW to think is more important than learning WHAT to think.
Thats how you should do it. Disasters dont wait around untill your not busy doing something else, they hit when they want to. Thats why i think live testing is important to see if recovery plan works or needs iteration of improvement... Best test is usually when disaster strikes right when you have hands full of something else... No time to dig up manuals, etc...
facebook and engineering aren't something I'd put into a sentence together.
And how do you think they got to handle the amount of data they did? The same comments rolled in when Walmart released its cloud service. Of course they have an engineering department.
Facebook engineers are working on a scale that most people will never see.
What's this World coming to when twitterbook has to be protected from natural disasters.
A ubiquitous service that most people have access to is one that speeds up disaster recovery. People have already used groups to organize disaster recovery efforts on small and large scales.
No one says you have to use it to upload food selfies.
So Netflix has been doing this for years now... it's called the Chaos Monkey and part of their "Simian Army" that performs this kind of function but *all the time* with no schedule. This is not something FB came up with but this post seems to give them credit for this innovation. More interesting than the lack of credit to Netflix though is this adoption of a method that heavily favors a Microservice Architecture. Seeing more and more of this flexible, scalable, and highly resilient architecture and methodology being put out in industry is certainly encouraging.
Pity.
And this is, sir, why IT is its current lame shape: allowing incompetent people taking key decisions just because they happen to relate to the right people.
Now, for a real world working DR plan:
10 REM 'DR Master Plan'
20 LET Power = OFF
30 PRINT 'AIIIIEEEEEEEE!'
40 GOTO 30
Now: *THAT'S* how professionalism looks like.
Just when you thought your privacy could be restored by the massive failure of a data center, a new center rises up from the ashes of the other to take the helm of stealing every personal detail about your life and the life of people you love.
It would be very useful for Facebook to stop announcing ASN 32934 for a few centuries as an experiment just to see what would happen.
Or permanently remove authority records for facebook.com.
You know just to see what would happen.
Months and months of planning went into the initial effort, ...
Into the take down or recovery? 'Cause the former just requires pulling a cable of some sort. :-) TFS says the team would take down a site and try to migrate the traffic, but wouldn't it be better if the disaster group and the recovery group were different teams for a "real world" stress test?
It must have been something you assimilated. . . .
Actually, this is one of the better story summaries I've seen here.
I knew what it was talking about (no unexplained mayfly buzzwords), I knew who the protagonists were, and I knew what was at stake. The only implied innovation was one of personal chutzpah, against the backdrop of an organization notorious for taking all things in collective stride (these being very, very short strides).
Working at Facebook Sounds Like Joining a Cult
At some level, I think we do indigenous people a disservice by referring to them as First Nations, freezing them in the amber of the era, as if they couldn't (and hadn't) kicked the shit out of their neighbours every bit as ruthlessly as the Spanish, the Dutch, or the British (secretly, it's a badge of honour, isn't it, to have kick-ass forbears?)—the main difference being that the European cultures brought with them a written language—so long, illiterate heathens—then, however they found the table set is assigned a positive integer (let's not even grant them "zero") to functionally signify that no form of tomahawk displacement came before.
So of course Netflix didn't invent this technique.
Clubbing a bunch of strange-looking men and taking their women is an idea that never owed much to the example of recent history, no matter how grand and savage an example your nearest neighbour might have set in the memorable recency.
There it's called "DiRT" (stands for "Disaster Recovery Test").
https://www.youtube.com/watch?...
Reminds me of "Chaos Monkey" from the netflix simian army.
This reminds me of when Pee-wee Herman fell off his bike and then got up and said: "I meant to do that"