Slashdot Mirror
Drooling Over VA Tech's 1100-Node G5 Cluster
Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"
here.
Here's why. Some of the more pertinent points:
Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price
GPL Deconstructed
Pink at LANL has the following:
:)
1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive
Not only do they have pictures of its assembly, they have movies.
Check the web page for more stats and better quality movies.
Oh, yes, it's unclassified
One of the reasons VTech went for a G5 based cluster WAS price-performance...The mac option is cheaper then a PC aption and easier to install and maintain then Linux says the slide show. I might not fully agree, but thats their reasoning.
Sig (appended to the end of comments you post, 120 chars)
"Reality is just a convenient measure of complexity" -Alvy Ray Smith
Actually, I just ran into this problem with a new eMac lab at my school, we set it up in about 4 hours. :)
Easy way to do it.
1. Set up 1 machine how you want it.
2. Get a bunch of firewire cables.
3. Hook the eMac's together using the cables. (If you can't reach with the cables, get some portable firewire drives, iPods work well with this too.)
4. Use Carbon Copy Cloner 2.2 (http://www.bombich.com/software/ccc.html) and move down the line of machines until they're all the same.
5. Go in and change HD, Network, etc names.
6. Smile because you just did something in 4-5 hours that it would take Windows users a week to do.
If you have questions, feel free to email (sethmath @ mac.com) me about it. I can walk you through if necessary.
http://www.apple.com/xserve/
I run a lab with about 50 macs (assorted models, from 350mhz iMacs through 800mhz eMacs, and a few 1ghz G4's) - I spent a good bit of time on a solution, and it's really not as hard as this thread makes it sound.
First, I build one system and set it up *Exactly* the way I want all the others to be. I have some run-once script voodoo to set the IP address of each machine based on its Mac address, and to munge some ByHost user preferences for the built-in guest account. Then, I use Carbon Copy Cloner">Carbon Copy Cloner to create an image of that machine's hard drive.
Once I have an image of the machine, I use NetRestoreNetRestore (by the same guy as CCC) to create a netboot image that will automatically install the master machine's HD image onto each client.
I am fortunate to have a MacOS X Server machine on which to run the NetBoot server - which is independent of the subnet's master DHCP server, I might add - but it is possible to netboot macs from other Unix machines with a bit of patching to dhcpd.
Anyhow, all in all I don't find it any more difficult to netinstall Macs than it is to do the same for Windows machines. Building the master clone image is time consuming and annoying, but it always will be for any platform.
Feel free to email me if you are interested in my machine setup voodoo script. I had to borrow some binaries from OS X Server in order to make it work. It's slowly turning into something useful as I add more functionality to it.
Depends what you mean by "as well as"... That only applies if you aren't talking about heat output, power requirements, cooling required, decent case design, ease of servicing. Then, for the programs you are using, things like an extra-fast bus, large CPU cache, and posibility of huge ammounts of RAM, must not be important at all to you.
So, sure, if those 8 things are not to be considered at all, then sure, you can say that the x86 option will run just as well.
And before you start calling me an Apple zealot, I do not, nor have I ever owned a single Apple or Mac-compatible computer. I do not work for Apple or any associated companies. Additonally, I do not common use Apple computers for any purpose.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Re: Processors
Perhaps for their benchmarks, the G5 was 2x the performance of the Opteron. Have you taken into consideration the Altivec processor, which happens to be 128bit in size? Any vector processing will be enhanced greatly by the powerful nature of the G5 in general, and especially when using Altivec optimized code. Couple this with IBM's XLC auto-vectorizing C compiler, and I wouldn't be surprised if Altivec did wipe SSE2/3D!Now; it's been discussed before that Altivec is a superior solution to MMX/MMX2/SSE, and SSE2, so there's no reason to doubt that when you pump up the FSB from 167MHz->1GHz, pump up the CPU from 1.4GHz->2.0GHz, on the PowerPC architecture, that Altivec doesn't become the most powerful SIMD solution in commodity computing.
Re: Chassis
It may be a time of research vs time to market discrepancy; IE, at the time VT was requesting bids, there were no Opteron chassis announced or available, whilst Apple may have had at 95% completion, barring an actual press release and announcement. Like, simultaneous to the release of the G5 there are no IBM PPC 970 machines, yet both companies use the same CPU.
Re: OS X
Yeah, there is a 64 bit X. It's called OS X Panther, and there's a 64 bit aware X called 10.2.7, and the libraries for Altivec have been 128bit for years now, so all 10.2.7 really added was... 64 bit pointers and memory addresses, really.
To recap: Altivec makes a big difference. Having immediately available machines makes a difference. Having a lower price point per performance per machine makes a difference (each node, including AC + networking + ram only costs about $4,727, which is $1,600 lower than an identically specced stock dual G5 with 4GB of ram!), as well as supportability of OS X vs Linux or, heaven forbid, Windows 2k... And yes, OS X for these machines are at least 64 bit enough to address 8GB of ram, and the OS has *always* been able to manipulate 128 bit data, as well as 64 bit data.
GPL Deconstructed
Or do you just want to bitch?
The real answer is that the problems that are going to be solved with this cluster are easily parallelizable. That's the IDEA, right? 1100 machines, each running one chunk. Well, the G5, and more specifically the Altivec vector processing section of it, is SO MUCH better for processing big bites of easily parallelizable data at a time than any of the alternatives that it can run rings around any Intel or AMD machine you care to name with fewer than double the number of processors. (And in the cases of some particular kinds of calculations, it beats those, too. But you can't count on that for all your problems.)
We've seen this before a number of times... I seem to recall a gene sequencing program that was running five or six times faster on a G4 than it was on a Pentium IV of the same speed. And then there's SETI@home, which runs much faster, cycle-for-cycle, on the Mac, and doesn't even USE altivec. (Though I believe it does take advantage of the 'multiply-and-add' instruction of the PPC, which is another nice little feature.)
Altivec is an astonishingly clean and usable interface for an amazingly powerful vector processor that is, in 99% of the Macs out there, underutilized to the point that if it suddenly disappeared, most people wouldn't notice any difference at all. It's kind of a pity, really.
Basically, Intel came out with MMX (and all the later developments) in order to have a talking point on a slide presentation about their processors, about the time when competitors like AMD were starting to come forward: functionally, an awful mess, and impossibly difficult to program. (In fact, for the first few years, Intel would send programmers out to work with companies to implement MMX, because otherwise none of them would bother.
AMD came up with something that was a little less hacked together in a very short period of time, as a response to Intel. But it still wasn't pretty, at least partially because of the limitations of the archetecture, and the performance wasn't *that* much better than just doing without.
Apple (who really designed a lot of the basics themselves when it comes to Altivec, so don't think this was a Motorola invention) said, 'Hey, wow, we need something like that, in order to compete.' First they decided on a coprocessor, but that didn't fly any better with the PPC than it did with the older Macs (840av, 660av) with DSPs in them. So they sat down and came up with a really *good* spec for a set of multimedia extensions. And they've only gotten better since.
I've toyed with altivec code, and I can tell you that in one application that I wrote, one instruction (vector permute) did the work of ten or more non-altiveced instructions on four times the data per cycle. Mind you, I just did it for fun, I don't know enough about parallel computing problems to come up with anything useful... but there's some interesting stuff under the hood.
Of course, nobody is going to believe this, because as fashionable as it is to like MacOS X on slashdot these days, nobody wants to admit that, for *some* subset of problems, Mr. Jobs's reality distortion field might not be quite as much of a distortion as you might think...
-fred
Sign #11 of Slashdot overdose: You see the phrase 'moderate Republican' and you wonder if that would be a +1 or a -1.
Exactly.
Except, with Ghost, you could install 1000 machines in 30 minutes - using multicast.
A couple might fail and you'd have to redo them, but if you have a 100Mbit switched network (or gig, even better) then its about 30 minutes to blast a Windows 2000 install to any number of machines.
I love macs, typing this on a PB17", but all the apple zealots out there really make me ashamed sometimes.
Macs are strong in some areas, and weak in others. If it wins in something, DONT RUB PEOPLES FACES IN IT. They don't care.
Get over it.