Google Caffeine Drops MapReduce, Adds "Colossus"

An anonymous reader writes "With its new Caffeine search indexing system, Google has moved away from its MapReduce distributed number crunching platform in favor of a setup that mirrors database programming. The index is stored in Google's BigTable distributed database, and Caffeine allows for incremental changes to the database itself. The system also uses an update to the Google File System codenamed 'Colossus.'"

Comments

Sounds inefficient

[martin-boundary]

This sounds like it's going to be highly inefficient for nonlocal calculations, or am I missing something? Basically, if the calculation at some database entry is going to require inputs from arbitrarily many other database entries which could reside anywhere in the database, then the computation cost per entry will be huge compared to a batch system.

Re:Sounds inefficient

[kurokame]

No, that's not it.

MapReduce is a sequence of batch operations, and generally, Lipkovits explains, you can't start your next phase of operations until you finish the first. It suffers from "stragglers," he says. If you want to build a system that's based on series of map-reduces, there's a certain probability that something will go wrong, and this gets larger as you increase the number of operations. "You can't do anything that takes a relatively short amount of time," Lipkovitz says, "so we got rid of it."

"[The new framework is] completely incremental," he says. When a new page is crawled, Google can update its index with the necessarily changes rather than rebuilding the whole thing.

There are still cases where Caffeine uses batch processing, and MapReduce is still the basis for myriad other Google services. But prior the arrival of Caffeine, the indexing system was Google's largest MapReduce application, so use of the platform has been significantly, well, reduced.

They're not still using MapReduce for the index. It's still supported in the framework for secondary computations where appropriate, and it's still used in some other Google services, but it's been straight-up replaced for the index. Colossus is not a new improved version of MapReduce, it's a completely different approach to maintaining the index.

Re:Sounds inefficient

[kurokame]

Sorry, Colossus is the file system. Caffeine is the new computational framework.

I made the same error in several posts now...but Slashdot doesn't support editing. Oh well! Everyone reads the entire thread, right?

Awesome choice of name.

[Scytheford]

"This is the voice of world control. I bring you peace. It may be the peace of plenty and content or the peace of unburied death. The choice is yours: Obey me and live, or disobey and die. [...] We can coexist, but only on my terms. You will say you lose your freedom. Freedom is an illusion. All you lose is the emotion of pride. To be dominated by me is not as bad for humankind as to be dominated by others of your species. Your choice is simple."
-Colossus.

Source: http://www.imdb.com/title/tt0064177/

Re:Awesome choice of name.

[Anonymous Coward]

Colossus is also the name of the computers Bletchley Park used to crack the German Lorenz cipher.
http://en.wikipedia.org/wiki/Colossus_computer

I have to say...

[tpstigers]

I am so glad Google has moved away from the Argus platform and into the Mercedes system. It makes it so much easier for those of us who are used to programming in Gibberish. Don't get me wrong - the days of Jabberwocky code were brilliant, but it's high time we moved into the Century of the Fruitbat.

Re:I have no idea

[icebike]

Follow the link to the Original Article over on The Register , where you will find a rather lucid explanation, far better than the summary above can provide.

Short answer:

The old method of building their search database was essentially a Batch Job, Run it, wait, wait, wait a long time, swap results into production servers.

The new method is continuous updates into a gigantic database spread over their entire network,

This is why things show up in Google days, sometimes weeks ahead of the other search engines. The other guys are still trying to clone Google's old method.

Re:I have no idea

[A Friendly Troll]

This is why things show up in Google days, sometimes weeks ahead of the other search engines.

For a hands-on example of what icebike is saying, look here:

http://www.google.com/search?q=%22This+is+why+things+show+up+in+Google+days%2C+sometimes+weeks+ahead+of+the+other+search+engines%22

Actually, Google will index Slashdot comments in a matter of minutes.

Re:I have no idea

[Runaway1956]

Bing probably redirects the search to Google, then displays the results on their own page. Bleahhh.