Drooling Over VA Tech's 1100-Node G5 Cluster

Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"

Comments

It's slow now. Complete mirror.

[Anonymous Coward]

here.

1100 reality distortion field generators

[jtnishi]

Oh God.

Imagining each one of those came with just a little bit of Steve Job's Reality Distortion Field, someone from NASA might want to head over there and make sure that some kind of tear in space/time doesn't occur right there. With that many G5s, we don't know what level of destruction could happen.

Hm.

[ixt]

What I really would like to know is how they install and configure all those machines. Their method of doing that will be very useful for even the (relatively) smaller networks that don't necessarily have to be clusters.

For example, I've yet to figure out a way to effectively get a computer lab with 30 eMacs installed and configured the same way. DHCP/Netboot is slow because we only have 100mbit switches. Split CD images are slow, and Jaguar doesn't yet have free software that does that yet (besides the dd of course). I'm not sure how to keep them all updated either.

I really hope they describe how they maintain the operating system on them.

Re:Hm.

[Benley]

I run a lab with about 50 macs (assorted models, from 350mhz iMacs through 800mhz eMacs, and a few 1ghz G4's) - I spent a good bit of time on a solution, and it's really not as hard as this thread makes it sound.

First, I build one system and set it up *Exactly* the way I want all the others to be. I have some run-once script voodoo to set the IP address of each machine based on its Mac address, and to munge some ByHost user preferences for the built-in guest account. Then, I use Carbon Copy Cloner">Carbon Copy Cloner to create an image of that machine's hard drive.

Once I have an image of the machine, I use NetRestoreNetRestore (by the same guy as CCC) to create a netboot image that will automatically install the master machine's HD image onto each client.

I am fortunate to have a MacOS X Server machine on which to run the NetBoot server - which is independent of the subnet's master DHCP server, I might add - but it is possible to netboot macs from other Unix machines with a bit of patching to dhcpd.

Anyhow, all in all I don't find it any more difficult to netinstall Macs than it is to do the same for Windows machines. Building the master clone image is time consuming and annoying, but it always will be for any platform.

Feel free to email me if you are interested in my machine setup voodoo script. I had to borrow some binaries from OS X Server in order to make it work. It's slowly turning into something useful as I add more functionality to it.

Re:Why G5s?

[2nd Post!]

Here's why. Some of the more pertinent points:

Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]

Sun (sparc) - required too many processors, also too expensive

IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available

HP (itanium) - ditto

Apple (IBM PPC970) - system available with chassis for lowest price

Re:Interesting... A Light Just Clicked On...

[entartete]

we got the dual processor g5 we ordered in at the university I work at and have had it for a while. I guess apple is really pushing to make the institutional customers happy. and if a university gets a g5 dozens of students can play with it and drool over it and become filled with g5 lust while if one regular customer gets one they'll just hide in their room mumbling about 'my preciousssss' and fondling it and that's not very good advertising and is sorta creepy.

drool over this, baby!

[F2F]

Pink at LANL has the following:

1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive

Not only do they have pictures of its assembly, they have movies.

Check the web page for more stats and better quality movies.

Oh, yes, it's unclassified :)

Re:Expensive processor vs. inexpensive processors

[90XDoubleSide]

If you read the link in one of the earlier comments, you would see that:

Slide Four
Choosing the Right Architechture

cost vs. performance (purely)
total cost $5.2 million includes system itself, memory, storage, and communication fabrics
one of the cheapest systems of its kind

Slide Five
Architectural Options

Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price

Re:Why G5s?

[McAddress]

Truly amazing, how many of you ever thought you would live ling enough to see Apple win a contract based on price?

Re:Expensive processor vs. inexpensive processors

[gerardrj]

- It's pretty clear to me that Apple didn't divert anything. If you look at the numbers, the VT order accounts for about 1% of all Dual G5 orders. That's hardly enough to cause the delays that people are seeing in their ship dates. Notice the slide states Apple offered an "early september" ship date, but Apple initially promised customers a mid-late August date. Given when those talks between VT and Apple were likely taking place, that means that Apple had intended to fill other orders first, and had a special allotment for the VT order.

- I don't know a whole lot about a blade center, but there doesn't seem to be a place to plug in the high-speed interconnects. Also, it runs on Intel chips that run hotter and do less work than the G5, especially when AltiVec gets involved, which is usually why you build a computer this size; vector processing. I'm also guessing the required configuration needed resale value to students at the end of life for the project/system.

- That's absoloutly true. When you need technical details about Linux you have to dig. When you have a question about OS X's guts, I'd guess you call Apple and have a conference call with all the coders (at least at this level of purchase/prestige). Could you imagine trying to get Linus, and all the other code writers for Linux and the supporting libraries and utilites on the phone at once?

Tshirts given to Terascale Volunteers

[Kirby-meister]

I volunteered time to help get some G5's ready for this baby, and I think my favorite moment was getting the tshirt all volunteers received:

It is quite the fashion statement :)

(Excuse the blurriness and poor lighting - crappy cam and crappy dorm lighting)

Re:think massive

[lastninja]

They should have bought 237 more G5s, that would have been so 1337.