Google's Technology Explored

RobotWisdom writes "Internetnews offers a moderately detailed peek at Google's technology. For example, they use stripped-down Red Hat on a massively redundant network, and they're starting to have success with automatic clustering of concepts, so that pages can match even if none of the words in your query actually appear on the page." Additional analysis on InformationWeek and C|Net. From the article: "As a search query comes into the system, it hits a Web server, then is split into chunks of service. One set of index servers contains the index; one set of machines contains one full index. To actually answer a query, Google has to use one complete set of servers. Since that set is replicated as a fail-safe, it also increases throughput, because if one set is busy, a new query can be routed to the next set, which drives down search time per box."

Comments

PigeonRank(TM)

[Kimos]

That's now how google does it! This is their REAL secret:
http://www.google.com/technology/pigeonrank.html

Re:PigeonRank(TM)

[Tackhead]

> That's now how google does it! This is their REAL secret: http://www.google.com/technology/pigeonrank.html

That was pre-IPO.

We'd like you to meet Bubba. Bubba's fully vested, and as this article says, he's, uh... he's grown somewhat.

Re:PigeonRank(TM)

[eric_brissette]

Their technology for waste management alone must be revolutionary.

Truly Amazing.

[iibbmm]

It really is amazing to think of the amount of information and data that we can access so quickly these days. When I stop and think about what my little search query goes through to bring me an almost instant response, it almost seems impossible. Of course the search engine side of this is only one example, but it's a nifty insight into how powerfull our infrastructure is these days. Bravo, mankind.

Whats really impressive

[mattmentecky]

The technology that is truly asstounding, is Google's ability to cache itself. Yeah, think about THAT one for a while.

Also Amazing: How much we miss

[Ieshan]

It's also amazing how much of the general knowledge of the world we *can't* access, because it's unconnected or unpublished.

Just think about how vast and extensive Google's search is, and then think about how little of the World's knowledge and creative achievement it actually can access.

The quantity and breadth of human knowledge is breathtaking, no?

Re:Also Amazing: How much we miss

[iibbmm]

That's why projects like wikipedia are so important, and so impressive.

Only a few years ago it could take forever to find any kind of decent information on some topics online or even in libraries. Today, I go to wiki and I'm almost assured to have a FAIRLY reliable source for information, as it's cross checked by peers who have some kind of a personal interest in the subject.

However, there's a downside.

Back when I was in school, researching a subject typically meant going through encyclopedia after encyclopedia, which wasn't a bad thing. I learned quite a bit by being FORCED to over-research topics. Today, I can generally straight-shoot to whatever I need to find, giving my brain a good set of blinders to everything else along the way.

Re:/. effect

[SmokeHalo]

It's been tried. From TFA:

One literal meltdown -- a fire at a datacenter in an undisclosed location -- brought out six fire trucks but didn't crash the system.

Oops

[Daedala]

Theoretically, he said, if someone searches for "Bay Area cooking class," the system should know that "Berkeley courses: vegetarian cooking" is a good match even though it contains none of the query words.

One word: cooking.

I'm sure the principle is sound. I just think the example is a leetle bit flawed.

Video about some of the backend stuff

[otisg]

Here it is, from one of the Google guys:
Google: A Behind-the-Scenes Look.

Sure?

[ferar]

I always thougth that they used NT + Access Database.

gCluster

[RobiOne]

They should make a googleCluster Live CD.. ala clusterKnoppix.. ..or perhaps use more of clusterKnoppix features or openmosix..share cpu/mem..
sourceforge is begging for something like this..

Their engineer desktops have special google builds of linux which help them compile things insanely fast with g4, ie hacked p4 (Perforce).

They also have one of the best intranet sites I've seen. Lots of info and services the employees can use, apart from email.

The internal blogs really help with keeping track of projects you're not working on, and what others are doing. Their mailing lists are often usefull too, for example there's a lost and found, for sale, and biking partners list. All kinds of usefull little stuff, taking care of the people with little nice things. Lots of reading too.

-- Robi

Google Maps - Designed to protect data centres

[Matt Clare]

Google's redundancy theory works on a meta level, as well, according to Hoelzle. One literal meltdown -- a fire at a datacenter in an undisclosed location -- brought out six fire trucks but didn't crash the system.

"You don't have just one data center," he said, "you have multiples."

The real idea behind Google Maps is so that as the server catches fire it use it's last cycles to send an eMail to the nearest fire cheif and include a map. I think it would also throw in a GMail invite for incentive.

Re:/. effect

[Anonymous Coward]

Computer programming languages are great, and I love them, but that does not mean that you have to use them for everything

open browser at www.google.com
get a drinking duck thing that bobs up and down hitting F5 every second

seems better to me.

Re:Question...

[limbostar]

They're not obligated to share unless they are planning on redistributing the software. They are perfectly free to patch their own software and use the patched versions for their servers without sharing those modifications.

The GPL does not force them to do anything unless they wish to redistribute the software.

Re:define: cheap machines

[canadiangoose]

I read somewhere that early Google datacentres were built by filling their racks with plywood shelves, then filling each shelf with one power supply running four motherboards each with one HDD. They didn't even use cases. This allowed them to build massively dense datacentres very cheaply. At one point they decided it wasn't worth it to replace dead hardware, so they started placing the racks too close together to be accessible. Why dig through and replace things when you can just keep adding more?

Anyhow, the article mentioned that in these early datacentres they experienced something like a 25% hardware failure rate, but that it didn't matter because the software worked around it and the hardware was cheap.

Here's a link to the page where I read all this neat stuff. It's probably mostly about the same stuff as the article we've all just slashdotted, but I won't be albe to tell for a while....