Netflix Gives Data Center Tools To Fail

← Back to Stories (view on slashdot.org)

Netflix Gives Data Center Tools To Fail

Posted by timothy on Tuesday November 27, 2012 @05:01AM from the because-the-fan-is-dirty dept.

Nerval's Lobster writes "Netflix has released Hystrix, a library designed for managing interactions between distributed systems, complete with 'fallback' options for when those systems inevitably fail. The code for Hystrix—which Netflix tested on its own systems—can be downloaded at Github, with documentation available here, in addition to a getting-started guide and operations examples, among others. Hystrix evolved out of Netflix's need to manage an increasing rate of calls to its APIs, and resulted in (according to the company) a 'dramatic improvement in uptime and resilience has been achieved through its use.' The Netflix API receives more than 1 billion incoming calls per day, which translates into several billion outgoing calls (averaging a ratio of 1:6) to dozens of underlying systems, with peaks of over 100,000 dependency requests per second. That's according to Netflix engineer Ben Christensen, who described the incredible loads on the company's infrastructure in a February blog posting. The vast majority of those calls serve the discovery user interfaces (UIs) of the more than 800 different devices supported by Netflix."

75 comments

Min score:

Reason:

Sort:

FAIL INTERNAL SERVER ERROR !! by Anonymous Coward · 2012-11-27 05:08 · Score: 0

Happens all the time here, so kettle, meet black !!
1. Re:FAIL INTERNAL SERVER ERROR !! by Anonymous Coward · 2012-11-27 05:22 · Score: 1
  
  s/black/pot/
  Dolt.
2. Re:FAIL INTERNAL SERVER ERROR !! by X0563511 · 2012-11-27 05:57 · Score: 1
  
  Hey! Let the Guru meditate in peace.
  
  --
  For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Thank you Netflix! by siDDis · 2012-11-27 05:16 · Score: 4, Interesting

Not only have you created an amazing tool, it is open source and the best part...it's actually well documented! Christmas came early this year!
1. Re:Thank you Netflix! by timeOday · 2012-11-27 06:27 · Score: 1
  
  Yeah, dumb question, but why would they choose to share it? Are they building on GPL code?
2. Re:Thank you Netflix! by Anonymous Coward · 2012-11-27 06:31 · Score: 5, Insightful
  
  (Netflix employee here, so forgive the AC)
  We don't use GPL code (and, assuming we were using GPLv2 code, given that we don't ship out server code, we wouldn't need to share it anyway), but:
  1. Netflix uses a ton of open-source technology. It's nice to be polite and give back;
  2. It's good publicity, which helps when we recruit people (which is something we do all the time);
  3. If it's good, then we'll have other people contribute to the software engineering efforts, which lowers the cost we pay to maintain and improve the software;
3. Re:Thank you Netflix! by Anonymous Coward · 2012-11-27 07:10 · Score: 0
  
  So Netflix doesn't use Linux, then?
4. Re:Thank you Netflix! by helix2301 · 2012-11-27 07:50 · Score: 1
  
  Its great there planning for the 3% of possible down time. Shows how great Netflix really is when you own 27% of internet traffic I guess you need to be prepaired.
  
  --
  http://www.thetechnologygeek.org
5. Re:Thank you Netflix! by Anonymous Coward · 2012-11-27 09:02 · Score: 0
  
  Probably Windows and maybe BSD - it needs Silverlight after all..!
6. Re:Thank you Netflix! by Anonymous Coward · 2012-11-27 09:22 · Score: 0
  
  Original AC responding here:
  I'm not sure if you or I were unclear, or if you're simply being difficult.
  Netflix does, of course, use Linux in the sense of "our cloud instances run on a distribution of Linux." That's not sufficient for requiring source code disclosure -- you have to actually use Linux source code, and/or use Linux libraries (I'm handwaving here because I'm not a GPL expert). Netflix doesn't do either.
7. Re:Thank you Netflix! by Anonymous Coward · 2012-11-27 09:38 · Score: 0
  
  There is a difference between using software and using code. Their servers probably run Linux, but they may not be rolling their own distribution.
Sounds like Netflix is a mess by evilviper · 2012-11-27 05:29 · Score: 1

I hate to say it, but the only thing I take away from this is that Netflix's software is such an unwieldy mess that they need a library just to enforce application separation and provide default fall-backs when a service call does fail.
FWIW, my preferred "circuit breaker" is a load balancer... All possible requests are network calls that go through load balancers, where it goes to the most responsive server, and if your admins screw up and none of the servers are responding quickly enough to answer the health check in time, a standard HTTP service unavailable response is supplied, without hammering the busy back-end.
And with all that complexity and effort, Netflix still can't handle two movies in your queue being assigned the same number, or a mixture of reordering and deleting titles at the same time... Things it handled just fine years ago, when it was a much smaller, web-1.0 service that didn't even require javascript.

--
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
1. Re:Sounds like Netflix is a mess by curunir · 2012-11-27 05:48 · Score: 4, Insightful
  
  Your critique seems overly simplistic. An HTTP load balancer is great for HTTP calls, but not everything in a complex infrastructure is HTTP. There's queues, data stores, caches, RPC, FileSystem access (SAN, NAS or local) and more that shouldn't run behind an HTTP interface. This tool helps solve the problem and gives you health check monitoring and metrics in the process. On initial inspection, my only complaint is that it requires too much modification of application code, however it seems like it should be pretty simple to integrate with the various IoC frameworks to use AOP proxies to apply the tool declaratively based on annotations.
  And you do realize that you followed up a weak critique of a backend scalability tool with a critique about a failing of their front-end application, right? What relevance does that have?
  
  --
  "Don't blame me, I voted for Kodos!"
2. Re:Sounds like Netflix is a mess by Synerg1y · 2012-11-27 06:24 · Score: 1
  
  Load balancing & Fail over serve two entirely different purposes. What happens when your load balancer has nowhere to balance to (we're talking about streaming HD media after all), or something in the cluster fails? Not a whole lot of places have instant fail-over, so it's going to be interesting to see who takes advantage of this.
3. Re:Sounds like Netflix is a mess by Black-Man · 2012-11-27 06:58 · Score: 1
  
  Load balancing is a mess. The vast majority of load balancing algorithms on VIPs are round robin-based. Fine for http requests from a single UI and that's about it.
  This application allows balancing of the services - not the servers.
4. Re:Sounds like Netflix is a mess by evilviper · 2012-11-27 07:39 · Score: 1
  
  An HTTP load balancer is great for HTTP calls, but not everything in a complex infrastructure is HTTP
  A common load balancer isn't restricted to HTTP. Any TCP connection can be load balanced quite well.
  
  And you do realize that you followed up a weak critique of a backend scalability tool with a critique about a failing of their front-end application, right? What relevance does that have?
  No, I followed-up by saying the old API was superior, while less taxing on the back-end... A common issue with "Web-2.0" interfaces.
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
5. Re:Sounds like Netflix is a mess by evilviper · 2012-11-27 07:41 · Score: 1
  
  All I get from your comment is the fact that you've NEVER used a load balancer before, and don't really know what it is and does.
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
6. Re:Sounds like Netflix is a mess by evilviper · 2012-11-27 07:47 · Score: 1
  
  The vast majority of load balancing algorithms on VIPs are round robin-based.
  A fair point... Stay away from the cheaper units, and make sure you use something that uses latency to make proper decisions.
  
  This application allows balancing of the services - not the servers.
  Services these days are damn near all TCP/IP based... Your front-end is making a network request to a Tomcat backend, which is running an app that's make JSON requests to some other service, which is pulling some data from some database, which is... you get the point. Every single one of those can go through a load-balancer, even if they all happen to be running on the same local machine. Of course the onus is on the admins and devs not to use 127.0.0.1 everywhere, and do proper networking, instead.
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
7. Re:Sounds like Netflix is a mess by Anonymous Coward · 2012-11-27 07:48 · Score: 0
  
  Your critique seems overly simplistic. An HTTP load balancer is great for HTTP calls, but not everything in a complex infrastructure is HTTP.
  So that's why there are ports other than 80! Can you explain this to consumer router manufacturers?
8. Re:Sounds like Netflix is a mess by ahem · 2012-11-27 08:01 · Score: 1
  
  Not to pick too nittily, but your assertion that a load balancer should just "go[es] to the most responsive server" is kind of simplistic. When I was working there, we had a failure mode where the most responsive server was one that had tripped over a subtle bug causing all subsequent requests to that instance of the service to almost immediately respond with an http 200 and empty content (it was a bug, after all, and it was compounded by this failure mode of returning a 200). Because we used weighted round-robin, we were able to diagnose that only some of the servers were behaving badly and could focus on reproducing the problem, finding the root cause and fixing it. The short-term mitigation was to bounce the ones in a bad state as they got into the bad state. This was at the cost of a poor user experience for 1/n of current traffic. In your proposed model, all traffic would have black-holed into this highly responsive server possibly making diagnosis and mitigation more difficult.
  As far as having to abstract away the heavy lifting of service management into a dedicated layer, I'm in violent agreement with that architectural decision. Without providing service management as a service, you put the burden on everyone's shoulders to manage it themselves. I'd rather have 98% of the developers focused on application logic and a specialized 2% of the developers providing productivity improvements to the core platform.
  While it might be more rewarding for some to re-blaze trails that have already been explored (and improvements do come out of parallel efforts) it's a much better business decision to put as much weight as possible behind moving the product forward and leave the task of lifting the efficiency of the platform to a dedicated team.
  
  --
  Not A Sig
9. Re:Sounds like Netflix is a mess by evilviper · 2012-11-27 11:25 · Score: 1
  
  we had a failure mode where the most responsive server was one that had tripped over a subtle bug causing all subsequent requests to that instance of the service to almost immediately respond with an http 200 and empty content
  
  Load balancers can be configured to verify the checksum of the content returned, and not just assume the return code is accurate.
  
  I'd rather have 98% of the developers focused on application logic and a specialized 2% of the developers providing productivity improvements to the core platform.
  The whole idea of writing crap code, and "optimizing" it later, whether with automated tools or by handing it off to others, works very, very poorly in practice. Putting a little effort in, at the start, to architect services properly, and keeping an eye on the design through the coding process, pays off in spades later on.
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
10. Re:Sounds like Netflix is a mess by ahem · 2012-12-06 13:59 · Score: 1
  
  I'd rather have 98% of the developers focused on application logic and a specialized 2% of the developers providing productivity improvements to the core platform.
  The whole idea of writing crap code, and "optimizing" it later, whether with automated tools or by handing it off to others, works very, very poorly in practice. Putting a little effort in, at the start, to architect services properly, and keeping an eye on the design through the coding process, pays off in spades later on.
  I never suggested that it was a good idea to write crap code. I suggested that it's a good idea to have some developers focused on things that all developers need to be taking care of (e.g. a platform that supports universal tasks). In that way, you raise everyone's efficiency with a single core effort and the vast majority of the team can focus on implementing features that move the business forward.
  
  --
  Not A Sig
800 devices supported by interkin3tic · 2012-11-27 05:30 · Score: 2, Insightful

But they can't possibly manage to bring it to Linux.
1. Re:800 devices supported by Githaron · 2012-11-27 05:45 · Score: 1
  
  But they can't possibly manage to bring it to Linux.
  Probably has something to with the Silverlight deal with Microsoft.
2. Re:800 devices supported by Anonymous Coward · 2012-11-27 05:48 · Score: 0
  
  Yea, those devices outnumber by magnitudes and make them more money than the whiny desktop Linux users.
3. Re:800 devices supported by Anonymous Coward · 2012-11-27 06:03 · Score: 0
  
  Yea, nevermind that most of those 800 devices manage to run some form of Linux...
4. Re:800 devices supported by Anonymous Coward · 2012-11-27 06:29 · Score: 0
  
  most of those 800 devices manage to run some form of Tivo-ized Linux...
  It's all about the DRM. It doesn't matter what OS runs on an unhackable appliance.
5. Re:800 devices supported by interval1066 · 2012-11-27 06:30 · Score: 1
  
  Srsly, does this thing use silverlight?
  
  --
  Python: 'And then suddenly you have a language which says "we're all stuck with whatever the whiniest coder wants".'
6. Re:800 devices supported by msk · 2012-11-27 06:30 · Score: 1
  
  I read that as a lack of will on the part of Netflix, who is likely Microsoft's single largest user of Silverlight.
  They have the leverage to push Microsoft on this.
  But don't.
7. Re:800 devices supported by Anonymous Coward · 2012-11-27 06:32 · Score: 0
  
  The point. You fail to grasp it.
8. Re:800 devices supported by Anonymous Coward · 2012-11-27 06:32 · Score: 0
  
  Netflix runs on plenty of Linux-based devices, just not generic, desktop distributions via a web browser.
  They technically could do it, but they can't for IP/DRM reasons, whether you think that is a good reason or not is irrelevant.
9. Re:800 devices supported by msk · 2012-11-27 06:33 · Score: 2
  
  They also haven't had a version of the Android player that's worked well on the LG Optimus V since v1.2 and that version won't work any more.
10. Re:800 devices supported by bill_mcgonigle · 2012-11-27 06:39 · Score: 3, Informative
  
  Probably has something to with the Silverlight deal with Microsoft.
  Close, but 'confusing cause and effect'.
  Silverlight was a facet of the DRM deal that Netflix made with the Studios. So is not releasing a Linux client (because then, y'know, there would be Netflix rippers and movies on bittorrent...).
  Amazon plays movies on Flash on Linux, so Netflix made a bad deal (or perhaps Amazon benefited from not being 'first', same as when Apple pioneered online music with iTunes and got AES AAC while Amazon later had plain MP3). There's also a libnetflixplayer.so ELF-32 on Chromebook, so there's no technical obstacle.
  Presumably those contracts have a renewal period. Accept that there's no technical problem and focus on the legal (government) problems instead.
  
  --
  My God, it's Full of Source!
  OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
11. Re:800 devices supported by interkin3tic · 2012-11-27 06:48 · Score: 2
  
  I'd say it actually has more to do with their CEO being on the board of MS until recently.
12. Re:800 devices supported by interkin3tic · 2012-11-27 06:50 · Score: 1
  
  It's subscription based, so I really don't see how you get to that. I have a netflix subscription, I pay money to netflix reguardless of whether I'm watching it on my PC, wii, or pocketwatch.
13. Re:800 devices supported by interkin3tic · 2012-11-27 06:52 · Score: 1
  
  They technically could do it, but they can't for IP/DRM reasons, whether you think that is a good reason or not is irrelevant.
  No, they could do it aside from the CEO of netflix having ties to MS. Whether I think it is a good reason or not is irrelevant to what netflix does, you're right, though I think implying my opinion is completely irrelevant is a bit harsh. It's relevant to ME!!! (sniff sniff).
14. Re:800 devices supported by Anonymous Coward · 2012-11-27 08:25 · Score: 0
  
  On the PC it uses Silverlight. On other devices it does not.
  As far as I understand it, Silverlight provides very close to the metal DRM support OOB which is required by the content owners to have in their IP streaming. If this sort of DRM is not supported OOB, netflix has to develop it as well as the app for the platform. Windows Phone supports a very similar DRM to silverlight so every windows phone has a netflix app. iOS does not, but only has 3(?) active versions so developing them is negligible. Android doesn't either,and because if it's open source nature and every provider distributing essentially their own version of Android, a netflix app has to be developed with DRM for each and every phone (same would go for TVs).
  Went on a bit of a tangent there, but I thought it was interesting when I was told about it. Please speak up if I'm wrong about this.
15. Re:800 devices supported by Anonymous Coward · 2012-11-27 09:02 · Score: 0
  
  The idea that Netflix streaming would be available on Linux desktops if not for Microsoft ties is ludicrous.
  None of the studios are willing to sign deals to let Netflix stream onto a platform that is under user control.
16. Re:800 devices supported by mattack2 · 2012-11-27 09:13 · Score: 1
  
  (or perhaps Amazon benefited from not being 'first', same as when Apple pioneered online music with iTunes and got AES AAC while Amazon later had plain MP3)
  Based on lots of reviews, AAC sounds better at the same bitrate. How is that being worse?
17. Re:800 devices supported by interkin3tic · 2012-11-27 10:36 · Score: 1
  
  And we know that how? Because Netflix, helmed by the microsoft board member said so?
  
  Do you commonly take company statements at face value?
18. Re:800 devices supported by Anonymous Coward · 2012-11-27 11:39 · Score: 0
  
  Based on lots of reviews, AAC sounds better at the same bitrate. How is that being worse?
  He's discussing DRM. Apple didn't remove DRM until the Amazon MP3 Store showed that's what people wanted.
Plan for failure of components. by concealment · 2012-11-27 05:31 · Score: 4, Insightful

One of the best changes in "design philosophy" that has happened in the past 20 years is that instead of the idea of any product as a fortress that cannot fail, products are designed to expect their components to fail, and to recovery gracefully from it.
This leads to a more flexible and resilient product. It reminds me of the military approach, where every system has at least two backups or alternates.
1. Re:Plan for failure of components. by medcalf · 2012-11-27 05:59 · Score: 1
  
  Except for the Internet of course. The Internet was designed with resiliency in mind, but the practical implementation of it has ended up being highly prone to failure, mostly because effectively all routing is over a limited number of backbone networks, sharing a small number of interconnects. It doesn't help practical resiliency to be able to take any route from A to B, when only one possible route is provided.
  
  --
  -- Two men say they're Jesus. One of them must be wrong. - Dire Straits
2. Re:Plan for failure of components. by mcgrew · 2012-11-27 07:44 · Score: 1
  
  It reminds me of the military approach, where every system has at least two backups or alternates.
  Millitary geat is FAR more robust than its civilian counterparts. I don't know how many winter coats I've bought that have worn out in the last forty years, I finally quit buying coats and now just wear the USAF field jacket I was issued forty years ago, that is still in good shape. I was always lucky to get three years out of a civilian coat (I'm rough on clothing).
  
  --
  Free Martian Whores!
3. Re:Plan for failure of components. by RabidReindeer · 2012-11-27 09:15 · Score: 1
  
  One of the best changes in "design philosophy" that has happened in the past 20 years is that instead of the idea of any product as a fortress that cannot fail, products are designed to expect their components to fail, and to recovery gracefully from it.
  This leads to a more flexible and resilient product. It reminds me of the military approach, where every system has at least two backups or alternates.
  I think that comes more from the fact that whereas in ancient times, re-IPLing an IBM mainframe was horribly expensive and something to be prevented wherever possible, doing the 3-finger salute on a Windows computer was infinitely cheaper than paying someone to write reliable software. Especially since Windows wasn't.
  As to the "recover gracefully" part....
Hystrix by poofmeisterp · 2012-11-27 05:44 · Score: 1

I read that Hysterix:
One becomes Hysterixical when their data center components fail.
1. Re:Hystrix by poofmeisterp · 2012-11-27 06:15 · Score: 1
  
  Cool story, brah.
  cool sense of humor and wordplay there, brah.
2. Re:Hystrix by Anonymous Coward · 2012-11-27 07:00 · Score: 0
  
  My sense of humor is fine. You're just not funny.
3. Re:Hystrix by poofmeisterp · 2012-11-27 11:12 · Score: 1
  
  My sense of humor is fine. You're just not funny.
  then why did you even bother commenting on it?
4. Re:Hystrix by Anonymous Coward · 2012-11-27 11:13 · Score: 0
  
  Because I wanted to.
5. Re:Hystrix by poofmeisterp · 2012-11-27 11:17 · Score: 1
  
  Because I wanted to.
  and you're still following it anonymously. wow, I feel flattered. thank you. :-D
Re:But does it include the chaos monkey? by CrankyFool · 2012-11-27 05:46 · Score: 5, Informative

Hystrix does not include Chaos Monkey, but Chaos Monkey was opensourced some time ago.
(I work at Netflix)
Confusing Title by BlueMonk · 2012-11-27 05:52 · Score: 0

I can't be the only one having trouble parsing the title of this article "Netflix Gives Data Center Tools To Fail". What does it mean to "give something to fail?" I thought "fail" was a verb and doesn't make sense as the target of the verb "give". I've heard of the phrase "given to failure", but that doesn't seem what's being implied here.
1. Re:Confusing Title by Baby+Duck · 2012-11-27 06:00 · Score: 1
  
  "Your education gives you tools to win. An unwillingness to further self-educate yourself gives you unnecessary roadblocks to fail." I admit the second sentence sounds more natural if it used "failure" instead of "fail". Netflix was just trying to be too cutesy and took poetic license, but I agree, it is still confusing and clunky.
  
  --
  "Love heals scars love left." -- Henry Rollins
2. Re:Confusing Title by ArsonSmith · 2012-11-27 06:01 · Score: 1
  
  Nike gives Olympian shoes to run.
  It's not great but i was smart enough to parse it fairly easy.
  
  --
  Paying taxes to buy civilization is like paying a hooker to buy love.
3. Re:Confusing Title by bws111 · 2012-11-27 06:10 · Score: 1
  
  Still doesn't make sense. In your example, running is what the Olympian wants to do, and Nike is giving him the shoes to do it. This stupid headline makes it sound like failing is the goal, and Netflix is helping them accomplish failure.
4. Re:Confusing Title by BlueMonk · 2012-11-27 06:22 · Score: 1
  
  OK, that gets me far enough to understand that the entity Netflix is performing the action of giving to the target of a data center, and the object its giving is tools. I'm still stumped at the "To Fail". Like the sibling reply says, this makes it sound like Netflix wants the data center to use its tools to accomplish failure. Should the title say "To Handle Failure" instead of "To Fail?"
5. Re:Confusing Title by Qzukk · 2012-11-27 06:39 · Score: 1
  
  Nike gives Olympian shoes to trip.
  Broke that for you. Now it's more like the title of this story and you may be able to see where everyone else's complaint is coming from.
  
  --
  If I have been able to see further than others, it is because I bought a pair of binoculars.
6. Re:Confusing Title by dyingtolive · 2012-11-27 06:49 · Score: 1
  
  Fail is a verb. Give is another verb. Give is not being 'targeted'. Netflix is giving data centers tools to fail. "Fail" describes what the data centers are doing with the tools in which they are being given. Fail is being used in the infinitive here.
  
  From the wiki page on Infinitives, as an example: "The letter says I'm to wait outside". You understand that in that sentence, "says" isn't being a 'target' of "wait", right? Why would you think otherwise in the summary? I mean, the only difference I can see there is the addition of a preposition to the end of the sentence. Perhaps you could parse the summary as "Netflix gives data center tools to fail [with]"?
  
  --
  Support the EFF and Creative Commons. The war is coming, and they're supporting you...
7. Re:Confusing Title by DaTrueDave · 2012-11-27 06:53 · Score: 1
  
  Perhaps you don't understand the headline because you don't understand the topic being discussed. Netflix has given data centers tool to fail (and recover effeciently). If Nike had made some shoes that made it easier to recover from tripping, then "Nike gives Olympian shoes to trip." would be like the title of this story and you would understand how appropriate it really is. If it helps you, take some of the irony out of the titles by adding the word 'gracefully' or 'efficiently' to the end of each.
8. Re:Confusing Title by bws111 · 2012-11-27 07:14 · Score: 2
  
  So what you are saying is that in the absence of these tools the data centers would NOT fail? That is just stupid. With the tools, the data centers are RECOVERING from failure, or AVOIDING failure, or some such. Take out those important words, and you convey the exact opposite meaning from that which was intended. At that is pretty much the definition of a really crappy headline.
9. Re:Confusing Title by BlueMonk · 2012-11-27 07:23 · Score: 1
  
  At first I thought "Data Center" was being used as an adjective, which was part of my problem. I thought they weren't just regular tools, I thought they were Data Center tools being given to something else. That's why I found it so confusing (to answer the question, "why would you think otherwise?").
10. Re:Confusing Title by dyingtolive · 2012-11-27 07:56 · Score: 1
  
  Ah, that makes more sense then.
  
  --
  Support the EFF and Creative Commons. The war is coming, and they're supporting you...
11. Re:Confusing Title by dyingtolive · 2012-11-27 07:59 · Score: 1
  
  In response to your question, I don't think an if and only if relationship was proposed, but I'm not the writer.
  
  In response to the rest of your statment, well, I never said any part of it WAS written well, only that I understood what the writer was trying to say. :-P
  
  --
  Support the EFF and Creative Commons. The war is coming, and they're supporting you...
12. Re:Confusing Title by steveg · 2012-11-27 08:09 · Score: 1
  
  How about "to fail gracefully."
  
  --
  Ignorance killed the cat. Curiosity was framed.
13. Re:Confusing Title by abirdman · 2012-11-27 12:11 · Score: 1
  
  I thought the same as you on reading the headline. I thought perhaps Netflix's code had caused data centers to fail.
  
  Often in publications, this kind of word misuse is caught and corrected by someone called an editor.
  
  --
  Everything I've ever learned the hard way was based on a statistically invalid sample.
Wow, backhanded summary title. by Anonymous Coward · 2012-11-27 06:22 · Score: 0

Yeesh.
Re:But does it include the chaos monkey? by Anonymous Coward · 2012-11-27 06:40 · Score: 0

Hystrix does not include Chaos Monkey, but Chaos Monkey was opensourced some time ago.
(I work at Netflix)
It was even covered on Slashdot.
This is the same NetFlix ... by eyegone · 2012-11-27 08:08 · Score: 1

... that goes down every time someone breaks wind in an AWS datacenter, right?

--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
Re:But does it include the chaos monkey? by Pollardito · 2012-11-27 08:18 · Score: 1

I wonder how many pointy-haired bosses have used Chaos Monkey to load test their own Amazon setup but accidentally hit someone else's servers (or is it somehow PHB proof?)
Re:But does it include the chaos monkey? by CrankyFool · 2012-11-27 09:20 · Score: 1

It depends on your definition of "someone else's". It uses your AWS credentials to kill an instance, so in the worst case, the PHB of group A in company Z could kill instances of group B in company Z; company Y would still be safe.
Re:But does it include the chaos monkey? by Pollardito · 2012-11-29 06:05 · Score: 1

Good point, sounds like it might be PHB-proof