Slashdot Mirror
Netflix Gives Data Center Tools To Fail
Nerval's Lobster writes "Netflix has released Hystrix, a library designed for managing interactions between distributed systems, complete with 'fallback' options for when those systems inevitably fail. The code for Hystrix—which Netflix tested on its own systems—can be downloaded at Github, with documentation available here, in addition to a getting-started guide and operations examples, among others. Hystrix evolved out of Netflix's need to manage an increasing rate of calls to its APIs, and resulted in (according to the company) a 'dramatic improvement in uptime and resilience has been achieved through its use.' The Netflix API receives more than 1 billion incoming calls per day, which translates into several billion outgoing calls (averaging a ratio of 1:6) to dozens of underlying systems, with peaks of over 100,000 dependency requests per second. That's according to Netflix engineer Ben Christensen, who described the incredible loads on the company's infrastructure in a February blog posting. The vast majority of those calls serve the discovery user interfaces (UIs) of the more than 800 different devices supported by Netflix."
Not only have you created an amazing tool, it is open source and the best part...it's actually well documented! Christmas came early this year!
s/black/pot/
Dolt.
I hate to say it, but the only thing I take away from this is that Netflix's software is such an unwieldy mess that they need a library just to enforce application separation and provide default fall-backs when a service call does fail.
FWIW, my preferred "circuit breaker" is a load balancer... All possible requests are network calls that go through load balancers, where it goes to the most responsive server, and if your admins screw up and none of the servers are responding quickly enough to answer the health check in time, a standard HTTP service unavailable response is supplied, without hammering the busy back-end.
And with all that complexity and effort, Netflix still can't handle two movies in your queue being assigned the same number, or a mixture of reordering and deleting titles at the same time... Things it handled just fine years ago, when it was a much smaller, web-1.0 service that didn't even require javascript.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
But they can't possibly manage to bring it to Linux.
One of the best changes in "design philosophy" that has happened in the past 20 years is that instead of the idea of any product as a fortress that cannot fail, products are designed to expect their components to fail, and to recovery gracefully from it.
This leads to a more flexible and resilient product. It reminds me of the military approach, where every system has at least two backups or alternates.
I read that Hysterix:
One becomes Hysterixical when their data center components fail.
Hystrix does not include Chaos Monkey, but Chaos Monkey was opensourced some time ago.
(I work at Netflix)
Hey! Let the Guru meditate in peace.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
"Your education gives you tools to win. An unwillingness to further self-educate yourself gives you unnecessary roadblocks to fail." I admit the second sentence sounds more natural if it used "failure" instead of "fail". Netflix was just trying to be too cutesy and took poetic license, but I agree, it is still confusing and clunky.
"Love heals scars love left." -- Henry Rollins
Nike gives Olympian shoes to run.
It's not great but i was smart enough to parse it fairly easy.
Paying taxes to buy civilization is like paying a hooker to buy love.
Still doesn't make sense. In your example, running is what the Olympian wants to do, and Nike is giving him the shoes to do it. This stupid headline makes it sound like failing is the goal, and Netflix is helping them accomplish failure.
OK, that gets me far enough to understand that the entity Netflix is performing the action of giving to the target of a data center, and the object its giving is tools. I'm still stumped at the "To Fail". Like the sibling reply says, this makes it sound like Netflix wants the data center to use its tools to accomplish failure. Should the title say "To Handle Failure" instead of "To Fail?"
Broke that for you. Now it's more like the title of this story and you may be able to see where everyone else's complaint is coming from.
If I have been able to see further than others, it is because I bought a pair of binoculars.
Fail is a verb. Give is another verb. Give is not being 'targeted'. Netflix is giving data centers tools to fail. "Fail" describes what the data centers are doing with the tools in which they are being given. Fail is being used in the infinitive here.
From the wiki page on Infinitives, as an example: "The letter says I'm to wait outside". You understand that in that sentence, "says" isn't being a 'target' of "wait", right? Why would you think otherwise in the summary? I mean, the only difference I can see there is the addition of a preposition to the end of the sentence. Perhaps you could parse the summary as "Netflix gives data center tools to fail [with]"?
Support the EFF and Creative Commons. The war is coming, and they're supporting you...
Perhaps you don't understand the headline because you don't understand the topic being discussed. Netflix has given data centers tool to fail (and recover effeciently). If Nike had made some shoes that made it easier to recover from tripping, then "Nike gives Olympian shoes to trip." would be like the title of this story and you would understand how appropriate it really is. If it helps you, take some of the irony out of the titles by adding the word 'gracefully' or 'efficiently' to the end of each.
So what you are saying is that in the absence of these tools the data centers would NOT fail? That is just stupid. With the tools, the data centers are RECOVERING from failure, or AVOIDING failure, or some such. Take out those important words, and you convey the exact opposite meaning from that which was intended. At that is pretty much the definition of a really crappy headline.
At first I thought "Data Center" was being used as an adjective, which was part of my problem. I thought they weren't just regular tools, I thought they were Data Center tools being given to something else. That's why I found it so confusing (to answer the question, "why would you think otherwise?").
Ah, that makes more sense then.
Support the EFF and Creative Commons. The war is coming, and they're supporting you...
In response to your question, I don't think an if and only if relationship was proposed, but I'm not the writer.
:-P
In response to the rest of your statment, well, I never said any part of it WAS written well, only that I understood what the writer was trying to say.
Support the EFF and Creative Commons. The war is coming, and they're supporting you...
... that goes down every time someone breaks wind in an AWS datacenter, right?
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
How about "to fail gracefully."
Ignorance killed the cat. Curiosity was framed.
I wonder how many pointy-haired bosses have used Chaos Monkey to load test their own Amazon setup but accidentally hit someone else's servers (or is it somehow PHB proof?)
It depends on your definition of "someone else's". It uses your AWS credentials to kill an instance, so in the worst case, the PHB of group A in company Z could kill instances of group B in company Z; company Y would still be safe.
I thought the same as you on reading the headline. I thought perhaps Netflix's code had caused data centers to fail.
Often in publications, this kind of word misuse is caught and corrected by someone called an editor.
Everything I've ever learned the hard way was based on a statistically invalid sample.
Good point, sounds like it might be PHB-proof