Slashdot Mirror
Drooling Over VA Tech's 1100-Node G5 Cluster
Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"
Imagine a beowulf clu...oh, wait.
here.
Oh God.
Imagining each one of those came with just a little bit of Steve Job's Reality Distortion Field, someone from NASA might want to head over there and make sure that some kind of tear in space/time doesn't occur right there. With that many G5s, we don't know what level of destruction could happen.
Using full sized cases seems like a rather inefficient use of space to me. But I guess those cases are all fairly full - the heatsinks in those things are enormous. Wish PCs had heatsinks like that, then maybe mine wouldn't be so noisy.
-kidlinux.
This presentation contains content that your browser may not be able to show properly. This presentation was optimized for more recent versions of Microsoft Internet Explorer.
Why make a website "optimized for IE", when the content of the said website is of interest to people who are probably not running IE or Windows?
"Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." -- Linus Torvalds
do you think it would take this nifty cluster to correct the barrel distorition from their wide angle lesnse?
FooGoo
People who bite the hand that feeds them usually lick the boot that kicks them
They've got 1100+...where's mine? I ordered a Dual 2.0 GHz G5 in July....still no sight. Supposed to ship on Tuesday....but online time will tell....
Sigh...Maybe they'll loan me one if mine gets delayed!
PS--anyone got the rest of these pics? There were a TON of them...Mirror? COMPLETE?
> Surely it's a pretty expensive option Apple won the contract based on being the lowest priced option to give them the power they wanted.
insert all your g5 are belong to us joke here
What I really would like to know is how they install and configure all those machines. Their method of doing that will be very useful for even the (relatively) smaller networks that don't necessarily have to be clusters.
For example, I've yet to figure out a way to effectively get a computer lab with 30 eMacs installed and configured the same way. DHCP/Netboot is slow because we only have 100mbit switches. Split CD images are slow, and Jaguar doesn't yet have free software that does that yet (besides the dd of course). I'm not sure how to keep them all updated either.
I really hope they describe how they maintain the operating system on them.
He had to admit they didn't have any in stock, and weren't expecting to get any from Apple for some time.
I guess I know where the dual-G5 systems are all going. Ah, well, it's all for a good cause. I hope.
Welcome to the Panopticon. Used to be a prison, now it's your home.
Here's why. Some of the more pertinent points:
Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price
GPL Deconstructed
It's a bit of a cocktease to post this link right now...Most of the mac community sites linked to the pictures at Virginia Tech's site but brought it down. Try clicking on the "pictures" link on their site and you'll se that they chmod 0'd the whole site so that the bandwidth usage won't peak out again
The pics at chaosmint are a small selection of what was originally on the site.
But to be on topic I'm suprised that Apple didn't get them Xserve G5's for the cluster. While the desktop G5's look cool it's really unneeded to use up all that space.
Pink at LANL has the following:
:)
1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive
Not only do they have pictures of its assembly, they have movies.
Check the web page for more stats and better quality movies.
Oh, yes, it's unclassified
I haven't actually tried it yet since I don't have access to enough Macs, but I imagine it's something you would start and let happen overnight... I mean, that's more or less how Apple does it in their own stores, wipe and restore overnight, I think. Or at least after the store closes and before the next opening day.
GPL Deconstructed
I hope they were able to run these without video cards. I can't imagine 1100 brand-new sweet ATI video cards sitting idle for years...
Yep. From the slideshow: Secondary Communications Gigabit Ethernet Fast Ethernet management backplane. Will carry NFS, control, job startup and "typical" IP traffic. Based on five Cisco 4500 series switches. 240 Gigabit Ethernet ports/switch Managed fabric with integrated wire-speed IP routing engine.
As a Professional * Information Technology Location Analyst and Physical Security Specialist I need to use my professional abilities to make a professional analysis of the situation for my professional collegues so that we may put forth a professional solution to this problem.**
* - I really, really hate people who make gratutious use of the word "professional" as some sort of elitist mark of supremecy
** - I would like to run in there, see if the machines are locked down, and grab as many as I can hold.
(And yes, I'm just joking, I don't want to steal anything from them and I neither have the plans nor the means to do so, it's just a joke people)
One of the reasons VTech went for a G5 based cluster WAS price-performance...The mac option is cheaper then a PC aption and easier to install and maintain then Linux says the slide show. I might not fully agree, but thats their reasoning.
Sig (appended to the end of comments you post, 120 chars)
"Reality is just a convenient measure of complexity" -Alvy Ray Smith
Reliability.
No one in their right mind would try to argue that one couldn't build a home-grown system for less. But with optical ports? FW 400 and 800? Gigabit ethernet? USB 2.0? And with said home-grown machines, when the NIC goes bad in one, or a memory slot goes bad in another, who do you call? The NIC or mainboard manufacturer? So you what, keep a list of all your machines, give 'em i.d. numbers or whatever, itemize the guts and who made what (mainboard, NIC, RAM, CPU, HDD, etc.) of each, and hope to make sense of it all when stuff starts to fail? Me, if I was in charge of it, it would make sense to me to farm it all out to one company, and then when something breaks there is one number that I have to call.
Also, lets not forget that this is probably going to be used for research, and if it involves vectors, then AltiVec is the SIMD for you.
Of course, being human, my opinion is suspect.
(tig)
Ignorance and prejudice and fear
Walk hand in hand
I'm really feeling for the poor slobs who have to lift 1100 of those beasts onto shelves. G5s are heavy!!!
Ah, vivid memories of the cover of Softtalk magazine, with a picture of the Apple II assembly line with hundreds of machines. Just imagine... 200 * 64k = 12.5 MEGABYTES! That would take 90 floppies to store all that data!
Now some statistic pr0n:
There were about 5 1/2 million Apple IIs sold, so at an average of 64k each (just a guess), that would be 343 GB of memory total. Adding up the couple of computers in the office (it's a 4 person company), we're about 1/70 of the way there. Assuming 2 140K floppy drives per computer, that would be 1.5TB of disk storage -- that would be 6 hard drives, and they would occupy less space than a single pair of old floppy drives.
HIV Crosses Species Barrier... into Muppets
Truly amazing, how many of you ever thought you would live ling enough to see Apple win a contract based on price?
I suppose I'm just anti-Mac trolling, but reading their slides, I can't help but get the impression that this university's RFP was intentionally slanted toward apple. Some things that seem interesting to me:
- It's clear from other posters that Apple did some shuffling to get inventory that wasn't available to the general public to this school. Not surprising, but indicates Apple was making considerable accomodations.
- I simply don't understand how full-size "commodity" Mac's could be cheaper than something like an IBM BladeCenter (especially in infrastructure (Switch, Rack, Space) costs). There's a lot of talk in the slides about the "required configuration" and the chassis... it sounds as though they placed some kind of hardware requirement on the proposal that could only be met by Apple. It just doesn't *sound* right, you know?
- One of the slides implies that they chose OSX over Linux because there isn't enough support for Linux. But this is a supercomputer! They're not running Photoshop on this thing. right? Aren't most supercomputer apps written fairly specifically for the machine they're running on? I really am asking the question here... is there something I don't understand about high-end cluster computing? I again simply don't understand how, at the super-cluster level, one could say Linux is poorly supported.
Of course, the university is well within their rights to buy from whoever they want. Their claims just seem questionable to me, at least from the brief slides.
It doesn't take pull... it takes money. Small schools have other significant advantages.
When you choose between going to a large, research-oriented school and going to a smaller school, you're essentially making a trade-off between resources and personal attention. Bigger schools have more and deeper resources, but it can be tough for undergrads to have much significant interaction with professors, particularly in the first year or two. Smaller schools may not offer the same variety of courses, or get huge research funding, or field a championship football team, but as an undergrad your chances of not just interacting with but really getting to know the faculty members (and not just the ones in your major department) are much better.
Most schools are happy to collaborate with others, so if you've got an idea that you think is well suited to Virginia Tech's cluster, talk to your advisor about submitting a proposal to VT. If it really is a good idea, your advisor may help you refine it and ultimately turn it into a research project.
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
Y'know, I saw this presentation a few days ago. I wasn't there, I saw it on the net. Anyway, this bullet point stuck out then - like, what are they talking about?
For one, how come it required twice the number of processors? From the benchmarks I've seen Opterons normally whup the G5, or are at least very competitive on paticularly G5 optimised code. Certainly not out by a factor of two, anyway.
And no chassis? What the hell does this mean? You can get 1U, 2U and 4U beast Opteron boxes from the likes of, well, IBM for one. As mentioned above.
It's not even like the kinda ropey nature of 64 bit Linux comes into play either because, well, there is no 64 bit OS X - unless VT know something we don't (which is always possible).
So, yeah, I think someone decided to buy all the G5's made for a month and just set up the project to make it happen. This "achitectural options" thing is horseshit.
Dave
I write a blog now, you should be afraid.
talk to VA tech, i'm pretty certain they'll have 1099 of them to spare right about now.
- It's pretty clear to me that Apple didn't divert anything. If you look at the numbers, the VT order accounts for about 1% of all Dual G5 orders. That's hardly enough to cause the delays that people are seeing in their ship dates. Notice the slide states Apple offered an "early september" ship date, but Apple initially promised customers a mid-late August date. Given when those talks between VT and Apple were likely taking place, that means that Apple had intended to fill other orders first, and had a special allotment for the VT order.
- I don't know a whole lot about a blade center, but there doesn't seem to be a place to plug in the high-speed interconnects. Also, it runs on Intel chips that run hotter and do less work than the G5, especially when AltiVec gets involved, which is usually why you build a computer this size; vector processing. I'm also guessing the required configuration needed resale value to students at the end of life for the project/system.
- That's absoloutly true. When you need technical details about Linux you have to dig. When you have a question about OS X's guts, I'd guess you call Apple and have a conference call with all the coders (at least at this level of purchase/prestige). Could you imagine trying to get Linus, and all the other code writers for Linux and the supporting libraries and utilites on the phone at once?
Article X: The powers not delegated... by the Constitution...are reserved...to the people
Learn to read, VT told you
Because the G5 systems were the cheapest AND fastest
There is no rackmount version of the G5 yet. That would be an upcoming G5 Xserve that has not been announced yet.
Plus, I guess when this cluster needs upgrading, they can sell off the Dual G5s which should hold their value for quite a while, as they are just stock G5 workstations.
It is quite the fashion statement :)
(Excuse the blurriness and poor lighting - crappy cam and crappy dorm lighting)
http://www.apple.com/xserve/
Configure one machine, use that one to produce a dupe, repeat with all currently configured machines as parents until out of unconfigured machines.
Still a huge job.
Depends what you mean by "as well as"... That only applies if you aren't talking about heat output, power requirements, cooling required, decent case design, ease of servicing. Then, for the programs you are using, things like an extra-fast bus, large CPU cache, and posibility of huge ammounts of RAM, must not be important at all to you.
So, sure, if those 8 things are not to be considered at all, then sure, you can say that the x86 option will run just as well.
And before you start calling me an Apple zealot, I do not, nor have I ever owned a single Apple or Mac-compatible computer. I do not work for Apple or any associated companies. Additonally, I do not common use Apple computers for any purpose.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
The error comes from the fact that the calculations are being done in floating point, and that some of the quantities involved cannot be represented exactly as a base 2 floating point number.
We run into the same problem using decimal notation in base 10. For example, 1/3 is 0.333... (repeating forever). If you only use a finite number of digits, then whatever number you write down in decimal notation will be a little bit smaller than 1/3. Now multiply that number by 3 and subtract 1:
3 * 1/3 - 1 = 0
But if we use a finite number of decimal digits (say 4) then we get 3 * 0.3333 - 1 = -0.0001.
What throws most people is that although they are used to 1/3 being a repeating decimal, they think 0.1 should be an exact number in floating point. However, computers generally use base 2 instead of base 10, and 1/10 happens to be a repeating fraction in base 2, so all of those decimal numbers become inexact in floating point calculations.
Most of the time this inaccuracy is hidden by performing the calculation with extra digits and rounding the results. Often the errors are rounded away before displaying the result, but they are still lurking in the floating point values. Take your example: 3.083-3.014. In most programs, (Calculator apps, Excel, etc) the result is probably displayed as 0.069. However, if you calculate 3.083-3.014-0.069 you will not get 0. You will see the rounding error.
The bottom line is that floating point calculations are inherently inexact. Most programs (in most situations) do a good job hiding this, but the error is always there.
That argument is mute. Motorolla fucked Apple over bigtime. The G4 can not handle ddram ram. THe chipset slows the speed down to 133 sdram because the cpu can not handle anything faster. It created a huge bottleneck.
The g5 is the IBM power4 in a lighter configuration. ITs the fastest desktop level cpu.
You could probably build a dual smp pIV which will perform close but the G5 has better fpu's and is cheaper in an smp system then xeon based top of the line PIV. This is why they chose the G5.
Also I do not like Windows and like MacOSX. You may want to test the latest from Apple. They really are leaps and bounds faster then the obsolete G4.
http://saveie6.com/
All it took was Apple's deep, deep discount for the marketing hoopla this represents for them.
This is gonna be a bullet point in every Apple advertisement for quite some time. It's damned cheap publicity.
A Good Intro to NetBS
Re: Processors
Perhaps for their benchmarks, the G5 was 2x the performance of the Opteron. Have you taken into consideration the Altivec processor, which happens to be 128bit in size? Any vector processing will be enhanced greatly by the powerful nature of the G5 in general, and especially when using Altivec optimized code. Couple this with IBM's XLC auto-vectorizing C compiler, and I wouldn't be surprised if Altivec did wipe SSE2/3D!Now; it's been discussed before that Altivec is a superior solution to MMX/MMX2/SSE, and SSE2, so there's no reason to doubt that when you pump up the FSB from 167MHz->1GHz, pump up the CPU from 1.4GHz->2.0GHz, on the PowerPC architecture, that Altivec doesn't become the most powerful SIMD solution in commodity computing.
Re: Chassis
It may be a time of research vs time to market discrepancy; IE, at the time VT was requesting bids, there were no Opteron chassis announced or available, whilst Apple may have had at 95% completion, barring an actual press release and announcement. Like, simultaneous to the release of the G5 there are no IBM PPC 970 machines, yet both companies use the same CPU.
Re: OS X
Yeah, there is a 64 bit X. It's called OS X Panther, and there's a 64 bit aware X called 10.2.7, and the libraries for Altivec have been 128bit for years now, so all 10.2.7 really added was... 64 bit pointers and memory addresses, really.
To recap: Altivec makes a big difference. Having immediately available machines makes a difference. Having a lower price point per performance per machine makes a difference (each node, including AC + networking + ram only costs about $4,727, which is $1,600 lower than an identically specced stock dual G5 with 4GB of ram!), as well as supportability of OS X vs Linux or, heaven forbid, Windows 2k... And yes, OS X for these machines are at least 64 bit enough to address 8GB of ram, and the OS has *always* been able to manipulate 128 bit data, as well as 64 bit data.
GPL Deconstructed
Except VT isn't going to be running SPEC[int|fp]+[_rate]*2000, so optimizing against those benchmarks isn't sufficient.
When all is said and done, it's been shown that for many vectorizable programs, Altivec still spanks SSE/SSE2/3d!Now, and anything else offered in the commodity market. Couple dual 2GHz G5s with Altivec and IBM's XLC autovectorizing compiler, and I think VT probably does have quite a powerful machine, more so than SPEC scores can quantify.
GPL Deconstructed
Funniest thing I've seen all day.
And I'm a Mac guy, too. I wouldn't mind wandering through that room for a while myself... though I probably would keep my pants on.
-fred
Sign #11 of Slashdot overdose: You see the phrase 'moderate Republican' and you wonder if that would be a +1 or a -1.
Hrm, this same logic should apply to medicine, cars, houses, and just about anything else that we as people have access to Original Source... yet notice how as a society and culture we tend to specialize and rely on experts?
I would be that, due to economics and division of labor, it is more productive and cost effective for VT, and many other places, to rely on a third party (IE Apple or Red Hat) to support their OS, while they themselves support, say, their business/product/venture... in this case, VT's research, while Apple supports the hardware/software. Makes sense, doesn't it? Each party does what it does best, so that the end result is more spectacular than if VT decided to, I dunno, devote resources to replicate what Apple already knows. Or at least has the resources to know.
GPL Deconstructed
Or do you just want to bitch?
The real answer is that the problems that are going to be solved with this cluster are easily parallelizable. That's the IDEA, right? 1100 machines, each running one chunk. Well, the G5, and more specifically the Altivec vector processing section of it, is SO MUCH better for processing big bites of easily parallelizable data at a time than any of the alternatives that it can run rings around any Intel or AMD machine you care to name with fewer than double the number of processors. (And in the cases of some particular kinds of calculations, it beats those, too. But you can't count on that for all your problems.)
We've seen this before a number of times... I seem to recall a gene sequencing program that was running five or six times faster on a G4 than it was on a Pentium IV of the same speed. And then there's SETI@home, which runs much faster, cycle-for-cycle, on the Mac, and doesn't even USE altivec. (Though I believe it does take advantage of the 'multiply-and-add' instruction of the PPC, which is another nice little feature.)
Altivec is an astonishingly clean and usable interface for an amazingly powerful vector processor that is, in 99% of the Macs out there, underutilized to the point that if it suddenly disappeared, most people wouldn't notice any difference at all. It's kind of a pity, really.
Basically, Intel came out with MMX (and all the later developments) in order to have a talking point on a slide presentation about their processors, about the time when competitors like AMD were starting to come forward: functionally, an awful mess, and impossibly difficult to program. (In fact, for the first few years, Intel would send programmers out to work with companies to implement MMX, because otherwise none of them would bother.
AMD came up with something that was a little less hacked together in a very short period of time, as a response to Intel. But it still wasn't pretty, at least partially because of the limitations of the archetecture, and the performance wasn't *that* much better than just doing without.
Apple (who really designed a lot of the basics themselves when it comes to Altivec, so don't think this was a Motorola invention) said, 'Hey, wow, we need something like that, in order to compete.' First they decided on a coprocessor, but that didn't fly any better with the PPC than it did with the older Macs (840av, 660av) with DSPs in them. So they sat down and came up with a really *good* spec for a set of multimedia extensions. And they've only gotten better since.
I've toyed with altivec code, and I can tell you that in one application that I wrote, one instruction (vector permute) did the work of ten or more non-altiveced instructions on four times the data per cycle. Mind you, I just did it for fun, I don't know enough about parallel computing problems to come up with anything useful... but there's some interesting stuff under the hood.
Of course, nobody is going to believe this, because as fashionable as it is to like MacOS X on slashdot these days, nobody wants to admit that, for *some* subset of problems, Mr. Jobs's reality distortion field might not be quite as much of a distortion as you might think...
-fred
Sign #11 of Slashdot overdose: You see the phrase 'moderate Republican' and you wonder if that would be a +1 or a -1.
Yeah man I don't understand the big deal with G5s for this kind of application. I'm sitting here in front of my 1100-unit dual-G5 cluster at my freelance gig trying to copy a 17M file from one folder to another and it's taking over 20 nanoseconds. My Cray at home would be done with this already, and even my beowulf cluster of TRS-80s wouldn't take this long....
Between the on-campus nuclear reactor and the supercomputer cluster, I'd keep an eye out if I were Tech's cross-state rival, University of Virginia. I'd say the Hokies are just one diabolical dean away from becoming an evil university bent on world domination. And five bucks says they start in Charlottesville.
DecafJedi
my weblog: apropos of something
If you believe 100mbits is too slow why not use IP over Firewire. I use it on my MacOSX server and it is screaming fast. You get 3200 mbits.
There is really nothing to do just daisy chain your mac with Firewire cables and configure the new network. On MacOSX client you will need to install IP over Firewire manually.
Maybe they can put those G5s to use as desktop computers after the cluster has been "retired".
Irene KHAAAAAAN!
I was one of the hordes of CS majors who helped setup the super computer (grunt work is fun!). VT is using inifiniband cards w/extremely low latency copper cable (forget the name) which acheives the same bandwidth as fiber optics.
Loads of cisco catalyst switches are involved also.
This gives them 1101 good computers - a kickass cluster now, and 1100 workstations later.
[100% ISO 646 Compliant]
SVM, ERGO MONSTRO.
Nope.
The *management* net is gigabit ethernet.
The actual clustering will be done over 10-gigabit Infiniband. (For a good time, figure out what the maximum bandwidth of a PCI-X slot is, and compare..)
In case you're curious to see what the educational discount price actually is, you can do so through the Apple Store very easily:
- go to http://store.apple.com/
(get redirected to the store home page)
- on the left side under "Interests...' click Education
- under "Shop for your School" pick "Find Your College or University"
- Pick "Virginia" and enter "Blacksburg", then click "Find"
- Select "Virginia Polytechnic Inst. & State Univ" (the only option) and click Continue
Voila, you get redirected to the Apple Store home page, but this time you are seeing the educational discount prices that VT departments would get.
Pick the G5 dual-CPU, no customization, and put Qty 1100 in your shopping cart. Click Update Subtotal. $2,968,900.00, will that be Visa or Mastercard?