Drooling Over VA Tech's 1100-Node G5 Cluster

Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"

Comments

It's slow now. Complete mirror.

[Anonymous Coward]

here.

Re:Why G5s?

[2nd Post!]

Here's why. Some of the more pertinent points:

Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]

Sun (sparc) - required too many processors, also too expensive

IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available

HP (itanium) - ditto

Apple (IBM PPC970) - system available with chassis for lowest price

drool over this, baby!

[F2F]

Pink at LANL has the following:

1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive

Not only do they have pictures of its assembly, they have movies.

Check the web page for more stats and better quality movies.

Oh, yes, it's unclassified :)

Re:Expensive processor vs. inexpensive processors

[90XDoubleSide]

If you read the link in one of the earlier comments, you would see that:

Slide Four
Choosing the Right Architechture

cost vs. performance (purely)
total cost $5.2 million includes system itself, memory, storage, and communication fabrics
one of the cheapest systems of its kind

Slide Five
Architectural Options

Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price

Re:Hm.

[Benley]

I run a lab with about 50 macs (assorted models, from 350mhz iMacs through 800mhz eMacs, and a few 1ghz G4's) - I spent a good bit of time on a solution, and it's really not as hard as this thread makes it sound.

First, I build one system and set it up *Exactly* the way I want all the others to be. I have some run-once script voodoo to set the IP address of each machine based on its Mac address, and to munge some ByHost user preferences for the built-in guest account. Then, I use Carbon Copy Cloner">Carbon Copy Cloner to create an image of that machine's hard drive.

Once I have an image of the machine, I use NetRestoreNetRestore (by the same guy as CCC) to create a netboot image that will automatically install the master machine's HD image onto each client.

I am fortunate to have a MacOS X Server machine on which to run the NetBoot server - which is independent of the subnet's master DHCP server, I might add - but it is possible to netboot macs from other Unix machines with a bit of patching to dhcpd.

Anyhow, all in all I don't find it any more difficult to netinstall Macs than it is to do the same for Windows machines. Building the master clone image is time consuming and annoying, but it always will be for any platform.

Feel free to email me if you are interested in my machine setup voodoo script. I had to borrow some binaries from OS X Server in order to make it work. It's slowly turning into something useful as I add more functionality to it.