Slashdot Mirror
Drooling Over VA Tech's 1100-Node G5 Cluster
Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"
Imagine a beowulf clu...oh, wait.
here.
Oh God.
Imagining each one of those came with just a little bit of Steve Job's Reality Distortion Field, someone from NASA might want to head over there and make sure that some kind of tear in space/time doesn't occur right there. With that many G5s, we don't know what level of destruction could happen.
Using full sized cases seems like a rather inefficient use of space to me. But I guess those cases are all fairly full - the heatsinks in those things are enormous. Wish PCs had heatsinks like that, then maybe mine wouldn't be so noisy.
-kidlinux.
This presentation contains content that your browser may not be able to show properly. This presentation was optimized for more recent versions of Microsoft Internet Explorer.
Why make a website "optimized for IE", when the content of the said website is of interest to people who are probably not running IE or Windows?
"Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." -- Linus Torvalds
They've got 1100+...where's mine? I ordered a Dual 2.0 GHz G5 in July....still no sight. Supposed to ship on Tuesday....but online time will tell....
Sigh...Maybe they'll loan me one if mine gets delayed!
PS--anyone got the rest of these pics? There were a TON of them...Mirror? COMPLETE?
What I really would like to know is how they install and configure all those machines. Their method of doing that will be very useful for even the (relatively) smaller networks that don't necessarily have to be clusters.
For example, I've yet to figure out a way to effectively get a computer lab with 30 eMacs installed and configured the same way. DHCP/Netboot is slow because we only have 100mbit switches. Split CD images are slow, and Jaguar doesn't yet have free software that does that yet (besides the dd of course). I'm not sure how to keep them all updated either.
I really hope they describe how they maintain the operating system on them.
He had to admit they didn't have any in stock, and weren't expecting to get any from Apple for some time.
I guess I know where the dual-G5 systems are all going. Ah, well, it's all for a good cause. I hope.
Welcome to the Panopticon. Used to be a prison, now it's your home.
Here's why. Some of the more pertinent points:
Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price
GPL Deconstructed
It's a bit of a cocktease to post this link right now...Most of the mac community sites linked to the pictures at Virginia Tech's site but brought it down. Try clicking on the "pictures" link on their site and you'll se that they chmod 0'd the whole site so that the bandwidth usage won't peak out again
The pics at chaosmint are a small selection of what was originally on the site.
But to be on topic I'm suprised that Apple didn't get them Xserve G5's for the cluster. While the desktop G5's look cool it's really unneeded to use up all that space.
Pink at LANL has the following:
:)
1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive
Not only do they have pictures of its assembly, they have movies.
Check the web page for more stats and better quality movies.
Oh, yes, it's unclassified
I haven't actually tried it yet since I don't have access to enough Macs, but I imagine it's something you would start and let happen overnight... I mean, that's more or less how Apple does it in their own stores, wipe and restore overnight, I think. Or at least after the store closes and before the next opening day.
GPL Deconstructed
I hope they were able to run these without video cards. I can't imagine 1100 brand-new sweet ATI video cards sitting idle for years...
One of the reasons VTech went for a G5 based cluster WAS price-performance...The mac option is cheaper then a PC aption and easier to install and maintain then Linux says the slide show. I might not fully agree, but thats their reasoning.
Sig (appended to the end of comments you post, 120 chars)
"Reality is just a convenient measure of complexity" -Alvy Ray Smith
Reliability.
No one in their right mind would try to argue that one couldn't build a home-grown system for less. But with optical ports? FW 400 and 800? Gigabit ethernet? USB 2.0? And with said home-grown machines, when the NIC goes bad in one, or a memory slot goes bad in another, who do you call? The NIC or mainboard manufacturer? So you what, keep a list of all your machines, give 'em i.d. numbers or whatever, itemize the guts and who made what (mainboard, NIC, RAM, CPU, HDD, etc.) of each, and hope to make sense of it all when stuff starts to fail? Me, if I was in charge of it, it would make sense to me to farm it all out to one company, and then when something breaks there is one number that I have to call.
Also, lets not forget that this is probably going to be used for research, and if it involves vectors, then AltiVec is the SIMD for you.
Of course, being human, my opinion is suspect.
(tig)
Ignorance and prejudice and fear
Walk hand in hand
Ah, vivid memories of the cover of Softtalk magazine, with a picture of the Apple II assembly line with hundreds of machines. Just imagine... 200 * 64k = 12.5 MEGABYTES! That would take 90 floppies to store all that data!
Now some statistic pr0n:
There were about 5 1/2 million Apple IIs sold, so at an average of 64k each (just a guess), that would be 343 GB of memory total. Adding up the couple of computers in the office (it's a 4 person company), we're about 1/70 of the way there. Assuming 2 140K floppy drives per computer, that would be 1.5TB of disk storage -- that would be 6 hard drives, and they would occupy less space than a single pair of old floppy drives.
HIV Crosses Species Barrier... into Muppets
Truly amazing, how many of you ever thought you would live ling enough to see Apple win a contract based on price?
It doesn't take pull... it takes money. Small schools have other significant advantages.
When you choose between going to a large, research-oriented school and going to a smaller school, you're essentially making a trade-off between resources and personal attention. Bigger schools have more and deeper resources, but it can be tough for undergrads to have much significant interaction with professors, particularly in the first year or two. Smaller schools may not offer the same variety of courses, or get huge research funding, or field a championship football team, but as an undergrad your chances of not just interacting with but really getting to know the faculty members (and not just the ones in your major department) are much better.
Most schools are happy to collaborate with others, so if you've got an idea that you think is well suited to Virginia Tech's cluster, talk to your advisor about submitting a proposal to VT. If it really is a good idea, your advisor may help you refine it and ultimately turn it into a research project.
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
Y'know, I saw this presentation a few days ago. I wasn't there, I saw it on the net. Anyway, this bullet point stuck out then - like, what are they talking about?
For one, how come it required twice the number of processors? From the benchmarks I've seen Opterons normally whup the G5, or are at least very competitive on paticularly G5 optimised code. Certainly not out by a factor of two, anyway.
And no chassis? What the hell does this mean? You can get 1U, 2U and 4U beast Opteron boxes from the likes of, well, IBM for one. As mentioned above.
It's not even like the kinda ropey nature of 64 bit Linux comes into play either because, well, there is no 64 bit OS X - unless VT know something we don't (which is always possible).
So, yeah, I think someone decided to buy all the G5's made for a month and just set up the project to make it happen. This "achitectural options" thing is horseshit.
Dave
I write a blog now, you should be afraid.
talk to VA tech, i'm pretty certain they'll have 1099 of them to spare right about now.
- It's pretty clear to me that Apple didn't divert anything. If you look at the numbers, the VT order accounts for about 1% of all Dual G5 orders. That's hardly enough to cause the delays that people are seeing in their ship dates. Notice the slide states Apple offered an "early september" ship date, but Apple initially promised customers a mid-late August date. Given when those talks between VT and Apple were likely taking place, that means that Apple had intended to fill other orders first, and had a special allotment for the VT order.
- I don't know a whole lot about a blade center, but there doesn't seem to be a place to plug in the high-speed interconnects. Also, it runs on Intel chips that run hotter and do less work than the G5, especially when AltiVec gets involved, which is usually why you build a computer this size; vector processing. I'm also guessing the required configuration needed resale value to students at the end of life for the project/system.
- That's absoloutly true. When you need technical details about Linux you have to dig. When you have a question about OS X's guts, I'd guess you call Apple and have a conference call with all the coders (at least at this level of purchase/prestige). Could you imagine trying to get Linus, and all the other code writers for Linux and the supporting libraries and utilites on the phone at once?
Article X: The powers not delegated... by the Constitution...are reserved...to the people
It is quite the fashion statement :)
(Excuse the blurriness and poor lighting - crappy cam and crappy dorm lighting)
http://www.apple.com/xserve/
Depends what you mean by "as well as"... That only applies if you aren't talking about heat output, power requirements, cooling required, decent case design, ease of servicing. Then, for the programs you are using, things like an extra-fast bus, large CPU cache, and posibility of huge ammounts of RAM, must not be important at all to you.
So, sure, if those 8 things are not to be considered at all, then sure, you can say that the x86 option will run just as well.
And before you start calling me an Apple zealot, I do not, nor have I ever owned a single Apple or Mac-compatible computer. I do not work for Apple or any associated companies. Additonally, I do not common use Apple computers for any purpose.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Re: Processors
Perhaps for their benchmarks, the G5 was 2x the performance of the Opteron. Have you taken into consideration the Altivec processor, which happens to be 128bit in size? Any vector processing will be enhanced greatly by the powerful nature of the G5 in general, and especially when using Altivec optimized code. Couple this with IBM's XLC auto-vectorizing C compiler, and I wouldn't be surprised if Altivec did wipe SSE2/3D!Now; it's been discussed before that Altivec is a superior solution to MMX/MMX2/SSE, and SSE2, so there's no reason to doubt that when you pump up the FSB from 167MHz->1GHz, pump up the CPU from 1.4GHz->2.0GHz, on the PowerPC architecture, that Altivec doesn't become the most powerful SIMD solution in commodity computing.
Re: Chassis
It may be a time of research vs time to market discrepancy; IE, at the time VT was requesting bids, there were no Opteron chassis announced or available, whilst Apple may have had at 95% completion, barring an actual press release and announcement. Like, simultaneous to the release of the G5 there are no IBM PPC 970 machines, yet both companies use the same CPU.
Re: OS X
Yeah, there is a 64 bit X. It's called OS X Panther, and there's a 64 bit aware X called 10.2.7, and the libraries for Altivec have been 128bit for years now, so all 10.2.7 really added was... 64 bit pointers and memory addresses, really.
To recap: Altivec makes a big difference. Having immediately available machines makes a difference. Having a lower price point per performance per machine makes a difference (each node, including AC + networking + ram only costs about $4,727, which is $1,600 lower than an identically specced stock dual G5 with 4GB of ram!), as well as supportability of OS X vs Linux or, heaven forbid, Windows 2k... And yes, OS X for these machines are at least 64 bit enough to address 8GB of ram, and the OS has *always* been able to manipulate 128 bit data, as well as 64 bit data.
GPL Deconstructed
Funniest thing I've seen all day.
And I'm a Mac guy, too. I wouldn't mind wandering through that room for a while myself... though I probably would keep my pants on.
-fred
Sign #11 of Slashdot overdose: You see the phrase 'moderate Republican' and you wonder if that would be a +1 or a -1.
Or do you just want to bitch?
The real answer is that the problems that are going to be solved with this cluster are easily parallelizable. That's the IDEA, right? 1100 machines, each running one chunk. Well, the G5, and more specifically the Altivec vector processing section of it, is SO MUCH better for processing big bites of easily parallelizable data at a time than any of the alternatives that it can run rings around any Intel or AMD machine you care to name with fewer than double the number of processors. (And in the cases of some particular kinds of calculations, it beats those, too. But you can't count on that for all your problems.)
We've seen this before a number of times... I seem to recall a gene sequencing program that was running five or six times faster on a G4 than it was on a Pentium IV of the same speed. And then there's SETI@home, which runs much faster, cycle-for-cycle, on the Mac, and doesn't even USE altivec. (Though I believe it does take advantage of the 'multiply-and-add' instruction of the PPC, which is another nice little feature.)
Altivec is an astonishingly clean and usable interface for an amazingly powerful vector processor that is, in 99% of the Macs out there, underutilized to the point that if it suddenly disappeared, most people wouldn't notice any difference at all. It's kind of a pity, really.
Basically, Intel came out with MMX (and all the later developments) in order to have a talking point on a slide presentation about their processors, about the time when competitors like AMD were starting to come forward: functionally, an awful mess, and impossibly difficult to program. (In fact, for the first few years, Intel would send programmers out to work with companies to implement MMX, because otherwise none of them would bother.
AMD came up with something that was a little less hacked together in a very short period of time, as a response to Intel. But it still wasn't pretty, at least partially because of the limitations of the archetecture, and the performance wasn't *that* much better than just doing without.
Apple (who really designed a lot of the basics themselves when it comes to Altivec, so don't think this was a Motorola invention) said, 'Hey, wow, we need something like that, in order to compete.' First they decided on a coprocessor, but that didn't fly any better with the PPC than it did with the older Macs (840av, 660av) with DSPs in them. So they sat down and came up with a really *good* spec for a set of multimedia extensions. And they've only gotten better since.
I've toyed with altivec code, and I can tell you that in one application that I wrote, one instruction (vector permute) did the work of ten or more non-altiveced instructions on four times the data per cycle. Mind you, I just did it for fun, I don't know enough about parallel computing problems to come up with anything useful... but there's some interesting stuff under the hood.
Of course, nobody is going to believe this, because as fashionable as it is to like MacOS X on slashdot these days, nobody wants to admit that, for *some* subset of problems, Mr. Jobs's reality distortion field might not be quite as much of a distortion as you might think...
-fred
Sign #11 of Slashdot overdose: You see the phrase 'moderate Republican' and you wonder if that would be a +1 or a -1.
Yeah man I don't understand the big deal with G5s for this kind of application. I'm sitting here in front of my 1100-unit dual-G5 cluster at my freelance gig trying to copy a 17M file from one folder to another and it's taking over 20 nanoseconds. My Cray at home would be done with this already, and even my beowulf cluster of TRS-80s wouldn't take this long....
Between the on-campus nuclear reactor and the supercomputer cluster, I'd keep an eye out if I were Tech's cross-state rival, University of Virginia. I'd say the Hokies are just one diabolical dean away from becoming an evil university bent on world domination. And five bucks says they start in Charlottesville.
DecafJedi
my weblog: apropos of something
Maybe they can put those G5s to use as desktop computers after the cluster has been "retired".
Irene KHAAAAAAN!