How Google Routes Around Outages

1sockchuck writes "Making changes to Google's search infrastructure is akin to 'changing the tires on a car while you're going at 60 down the freeway,' according to Urs Holzle, who oversees the company's massive data center operations. In a Q-and-A with Data Center Knowledge, Holzle discusses Google's infrastructure, how it has engineered its system to route around hardware failures, and how it responds when something goes awry. These updates usually go unnoticed, but during system maintenance last month a software bug triggered an outage for Gmail."

Comments

Just me?

[Anonymous Coward]

Was it just me or did anyone else spend a few minutes contemplating how you actually could make a car that did allow you to change a flat while moving?

Re:Just me?

[Anonymous Coward]

I thought about it for approximately 30 seconds. Then I realized that it is a bad analogy. A Google car would have hundreds of redundant wheels, changing one is easy.

Re:Just me?

[Slumdog]

Just you. I kept thinking about how I could use a car metaphor to describe how google...oh wait.

I kept thinking about derailing a car, before I realized I was on the wrong track.

Re:Just me?

[Saerko]

That's what I was thinking too; and probably just function like an 18-wheeler where a tire can blow out and there's so much support that the load is still distributed adequately.

Basically, all this means is Google designs like Mack while everyone else designs like Chrysler...

Re:Just me?

[tux0r]

I kept thinking about derailing a car, before I realized I was on the wrong track.

I was going to reply about mixing metaphors, but then I lost my train of thought.

I know! I know! Pick Me!!!

[fuzzyfuzzyfungus]

It just treats the damage as censorship and routes around it, right?

Google File System Paper

[Anonymous Coward]

To those looking for a more in-depth description, check out the technical paper on the google file system:

http://labs.google.com/papers/gfs.html

Had to read it for a search engines course in college, it's pretty darn spiffy.

Re:Google File System Paper

[SeePage87]

Why would we need a more in-depth description? We already got our car metaphor!

Video of the car analogy

[Anonymous Coward]

Excellent use of the car analogy, especially since it is possible to change a tire while driving a car. Youtube video at 1:48.

Slightly..ahem... OT so posting anon.

Article doesn't really say anything.

[girlintraining]

You know, the article read like a press release. Hasn't slashdot whored itself out enough lately on these kinds of things? Google is so ultra-reliable, blah blah, 24x7, blah blah, commitment, blah blah, premier service partner, blah blah... I get that kind of talk enough in staff meetings. Where's the meat already!?

Why not write an article with some nice graphics saying what happens to my request from the time I hit "Search" to the time I click a result. List off all the servers it goes through, their roles, how they're monitored, etc. Give examples of failure and show the mode decisions the software makes (and where this software is running) -- show the latencies and other performance impacts as my request bounces over failure after failure. That's what I expect when I pull up an article entitled "How Google Routes Around Outages". Something useful, professionally enriching, intellectually stimulating, etc. In short, tell me why I (should) never see a "500 Internal Server Error" from Google, but I do from just about every other major website I've used.

Inspiration taken from the same thing it runs on!

[Thantik]

Isn't this how the *internet* is (at least in theory) supposed to work anyhow? Instead we have 90% of the cables that route the middle-east/europe running through the same canal. And I know of VERY few ISPs who actually make their systems redundant anymore. /sadface