Google's 4000 Node Linux Cluster

Check out the Red Hat press release running at LWN, or the news article at techweb about Google's 4000 Node Linux Box. Both articles are basically Red Hat commercials, but there's some interesting bits like the fact that they have a terebyte index of 300 million Web pages, and that they might expand their cluster to 6000 nodes in the future.

Comments

Good comparison

[JamesSharman]

It's nice to see some good Linux publicity happening, Google is fast becoming the most respected search engine around, their clean and uncluttered interface is drawing people away from the more traditional search engines where it seems you have to download more portal c$&p every day. It seems poetic the google is becoming an ambassador for linux by showing up their bloat laden competitors in the search engine market, while linux does the same in the OS market.

Re:Good comparison

[Gurlia]

Yeah, all the other popular search engines nowadays seem to be ridden with banner ads, promotions, and all kinds of useless fluff on their pages. Google is nice and simple, doesn't clutter the screen, and in general makes everything easier on the eyes. I think this is part of the attractiveness of Google -- you're not flooded with irrelevant info and pictures, but just the stuff you're looking for.

One thing I have against Google though -- I wish they had an advanced search where you can specify to search for exact phrases, etc., or perhaps even a full boolean search. I don't know how Google works, so I can't tell if these features are left out because of design issues. But, being the "hacker's search engine" and everything, it really should support more advanced searches. If they can find a way to implement this well, it may even become a deciding factor against other search engines. (I hardly know any search engine out there that can handle full boolean search, and certainly Google's speed will be a great advantage.)

---

Probable source of their inspiration...

[Carnage4Life]

I can just see it now. A manager at Google walking over to a developer's PC and seeing this sticker and saying,"Why not?"

Now all that's needed is for thinkgeek to claim responsibility for this action. :)

hey

[jbarnett]

So this "super computer" will be used for Total World Domination? Oh, can we use it atleast to take over some small thrid world countries? I promise to have it back by six tonight.

The Google crew must have some killer Seti@home stats.

I would like to put one of these in my basement and finally disprove the "7 steps to Kevin Bacon" theory everyone seems to buy into.

google uses RAIP technology

[aozilla]

redundant array of inexpensive processors

Re:Very Smart *NOT*

[cybrthng]

Well, as you are all well aware of, dot.com's are going through money like nothing. Sure it is *great* publicity to have 4,000 servers witn another 2,000 coming online.

But damn, that takes a staff of 200 people to manage the security/connectivity/accounts/space and other duties just for the cluster.

The Power bill has to be outrageous!

The Cabling/switching/routing mess has to be totally unmanageable

What happens when you reach a buck in the hardware or have to patch the system or replace a kernel because of a hack that came about? It is costly and hellish to work on 4-6,000 pcs

I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper. Sure your upfront bill may be more, but only have to worry about 8-16 power connections (redudancy) is alot easier then 6,000 power cords/strips/racks/floor space/cooling/maintenance.

Sure it is one hell of a beast to be proud of, but one hellova costly beast to work with.

Just my 2 cents

Google is driven by python not by perl

[segmond]

just my own 10cents, The google guys use python over perl, hrmmm, i wonder why. :D by the way their paper is a good read. http://www7.scu.edu.au/programme/fullpapers/1921/c om1921.htm

Re:Very Smart *NOT*

[heimdall]

I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper.

Last I checked (this was about a year or so ago) a fully loaded (64/64) E10K ran around $12M and the base (2psr) system was running around $800,000. Even if that's off by a factor of 3 or 4, you're still talking $3-$4M a piece... at three of them, you're looking at between $12-$48M. On the other hand, the typical white box PC will run between $800-$1500. That amounts to $3.68M-$6.9M for 4600 nodes. This doesn't include the network infrastructure or administration costs, however, as someone who has administered large clusters (largest was an 80 node SP/2), it actually becomes easier to administer that many nodes in a cluster than it would that many servers. Keep in mind that there most certainly are groupings of nodes where they are kept identical except for IP.

Another significant expense is that hardware support costs associated with such systems. If you have 4600 nodes, it's trivial to simply keep (MANY) spare systems floating around. Also, you can disable a node with negligible impact. Even if you're subdomaining an E10K, there are (a small few) single points of failure on the platform (regardless of what Suns documentation says). If you're not subdomaining it, you're simply talking a 32way SMP box (might as well just use a 6500 for that configuration). If you were to lose the backplane for whatever reason, you've lost a singificant portion of your compute resources.