Slashdot Mirror
Making Facebook Self Healing
New submitter djeps writes "I used to achieve some degree of automated problem resolution with Nagios Event Handler scripts and RabbitMQ, but Facebook has done it on a far larger scale than my old days of sysadmin. Quoting: 'When your infrastructure is the size of Facebook's, there are always broken servers and pieces of software that have gone down or are generally misbehaving. In most cases, our systems are engineered such that these issues cause little or no impact to people using the site. But sometimes small outages can become bigger outages, causing errors or poor performance on the site. If a piece of broken software or hardware does impact the site, then it's important that we fix it or replace it as quickly as possible. ... We had to find an automated way to handle these sorts of issues so that the human engineers could focus on solving and preventing the larger, more complex outages. So, I started writing scripts when I had time to automate the fixes for various types of broken servers and pieces of software.'"
We had to find an automated way to handle these sorts of issues so that the human engineers could focus on solving and preventing the larger, more complex outages.
This seems backwards to me. Surely the "larger, more complex outages" are caused by an accumulation of, or interaction between, the smaller, less complex problems/situations. If all of the smaller problems are well understood and dealt with, then those more complex problems should not arise. I think it's dangerous to assume that because the smaller problems can be transiently resolved by a script with minimal human intervention that the more complex problems need less exploration. Sure, scripts to handle the less complex issues are great, but this should not shift the focus of the human engineers to "focus on solving and preventing complex outages"; solving those often (always?) means solving the less complex issues.
How are we supposed to kill it if it's self-healing? Now it will never die!
"None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
We had to find an automated way to handle these sorts of issues so that the human engineers could focus on solving and preventing the larger, more complex outages.
Given how glitchy Facebook was in the past, I can't help but be reminded of this comic.
I was rolling out Big Brother Network Monitor a decade ago. It was well capable of doing this.
Today, I'd use an RDB that stored output from perl:DBI cronjobs running on each machine, and another job that checked the db and made sure all that ought to be happening had reported in successfully recently. Anything that hadn't would trigger an email to someone to look into it.
Easy to develop, implement, extend, and maintain.
No, I don't want to connect to FB just to read the article. Post it somewhere else if you want it read.
"Tongue tied and twisted, just an Earth bound misfit
Damn, if I weren't so adverse to soul crushing rejection, I'd apply.
This guy was insightful and informative, so I believe what is quoted above.
And I'm surprised: I figured Facebook would be either more bureaucratic (like MS) or kinda dickishly autocratic (like Zuckerberg is rumoured to be).
Auto ticketed errors, I am Amazed. If you did not detect sarcasm, please enter a problem ticket. You don't think that shit's automated do you?
I was thinking more in terms of "assisted suicide".
If you want news from today, you have to come back tomorrow.
I guess upstart was written by the Winklevoss twins.
Did you even read the article? It talks about things like broken hard drives.
"Today, the FBAR service is developed and maintained by two full time engineers, but according to the most recent metrics, itâ(TM)s doing the work of approximately 200 full time system administrators"
Which doesn't really tell anyone anything. Who expresses amount of work done in terms of number of full time workers? In case anyone had failed to get the message, the above shows that such a metric isn't very useful. Perhaps the message here is that 2 really effective people can do the work of 200 not so effective people - but that has been known for a long time. Still, the more this message is spread, the better.
Please correct me if I got my facts wrong.
Part of the reason Facebook and Google can "self heal" is because failures are mostly not noticeable by end users. If a Facebook or Google machine fails, unless you are getting a 404 or a service failure message there is little to no way for you to know that the web page you have been served up is wrong, partial or out of date. This failure ambiguity provides a lot of leeway on the methods and speed required to fix a failure.
For most other services where there is a definite correct and incorrect output - like file systems or financial services - a broken service has immediate impact and fixing it is much harder.
How come friends keep disappearing only to request again, saying I dropped them. Either it's buggy or broke....
From the sounds of this article, Facebook and Google go about this VERY differently.
The Facebook way, it seems, is that every node in the infrastructure is possibly important. So they write and maintain all these healing scripts to deal with problems like broken processes or failed hard drives.
Google goes about the same problem in a very different way. Google's system is architected such that no node is important. Everything is massively parallel and redundant - such that you could take and destroy any server, any set of servers, even an entire data centre and blow it up with a bomb, and side from performance issues, no one would notice.
From an admin's point of view - I would much prefer Google's system. Something doesn't look right on a box? Yank it out TOTALLY, put in a new one, investigate some other time.
I would like to suggest a subtle change to the posting system: Make it so the first post on any article cannot be done as "Anonymous Coward".
I know Slashdot has a tradition of being a "free-for-all, run through a blender" but I don't think there has ever been an AC first post that has ever been anything but either:
- So lame that you wonder how a person manages to survive such a terminal case of lack of personality or creativity... or
- There is no real reason it couldn't have been posted under a login.
Stupidity really should be viciously stamped out but if we can use automated steps to reduce the "background stupid" we can then focus more energy on more invasive cases of dumb.
s/cosmonaut/confidant/
Maybe You have confused Zuckerberg with Guy Laliberté or Mark Shuttleworth. Or perhaps with Richard Branson, who builds space tourism vehicles.
The song, however, has nothing to do with space travelers.
"Facebook devotes over 100 physical servers to every 35,000 users. That is incredibly inefficient"
Absolutly yes! If they only managed to serve 350 users per server, that, that would be a neat thing.