Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google's 4000 Node Linux Cluster

Posted by CmdrTaco on from the now-that's-distributed dept.

Check out the Red Hat press release running at LWN, or the news article at techweb about Google's 4000 Node Linux Box. Both articles are basically Red Hat commercials, but there's some interesting bits like the fact that they have a terebyte index of 300 million Web pages, and that they might expand their cluster to 6000 nodes in the future.

3 of 158 comments (clear)

  1. Re:Very Smart *NOT* by cybrthng · · Score: 5
    Well, as you are all well aware of, dot.com's are going through money like nothing. Sure it is *great* publicity to have 4,000 servers witn another 2,000 coming online.

    But damn, that takes a staff of 200 people to manage the security/connectivity/accounts/space and other duties just for the cluster.

    The Power bill has to be outrageous!

    The Cabling/switching/routing mess has to be totally unmanageable

    What happens when you reach a buck in the hardware or have to patch the system or replace a kernel because of a hack that came about? It is costly and hellish to work on 4-6,000 pcs

    I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper. Sure your upfront bill may be more, but only have to worry about 8-16 power connections (redudancy) is alot easier then 6,000 power cords/strips/racks/floor space/cooling/maintenance.

    Sure it is one hell of a beast to be proud of, but one hellova costly beast to work with.

    Just my 2 cents

  2. Google is driven by python not by perl by segmond · · Score: 5

    just my own 10cents, The google guys use python over perl, hrmmm, i wonder why. :D by the way their paper is a good read. http://www7.scu.edu.au/programme/fullpapers/1921/c om1921.htm

    --
    ------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
  3. Re:Very Smart *NOT* by heimdall · · Score: 5

    I would have thought it to be wiser to setup Sun E10000's or something like that.. having 4 32 proc e 100000's in a cluster is a hell of alot easier to manage and cheaper.

    Last I checked (this was about a year or so ago) a fully loaded (64/64) E10K ran around $12M and the base (2psr) system was running around $800,000. Even if that's off by a factor of 3 or 4, you're still talking $3-$4M a piece... at three of them, you're looking at between $12-$48M. On the other hand, the typical white box PC will run between $800-$1500. That amounts to $3.68M-$6.9M for 4600 nodes. This doesn't include the network infrastructure or administration costs, however, as someone who has administered large clusters (largest was an 80 node SP/2), it actually becomes easier to administer that many nodes in a cluster than it would that many servers. Keep in mind that there most certainly are groupings of nodes where they are kept identical except for IP.

    Another significant expense is that hardware support costs associated with such systems. If you have 4600 nodes, it's trivial to simply keep (MANY) spare systems floating around. Also, you can disable a node with negligible impact. Even if you're subdomaining an E10K, there are (a small few) single points of failure on the platform (regardless of what Suns documentation says). If you're not subdomaining it, you're simply talking a 32way SMP box (might as well just use a 6500 for that configuration). If you were to lose the backplane for whatever reason, you've lost a singificant portion of your compute resources.