Slashdot Mirror
Virginia Tech to Build Top 5 Supercomputer?
hype7 writes "ThinkSecret is running a story which might explain exactly why the Dual 2GHz G5 machines have been delayed to the customers that ordered them minutes after the keynote was delivered. Apparently, Virginia Tech has plans to build a G5 cluster of 1100 units. If it manages to complete the cluster before the cut-off date, it will score a Top 5 rank in the Linpack Top 500 Supercomputer List. Both Apple and the University are playing mum on the issue, but there's talk of it all over the campus."
"Both Apple and the University are playing mum on the issue, but there's talk of it all over the campus."
Must be a pretty boring campus...
~ "When I'm of that age I'm just going to live up a tree."
Are they gonna run Photoshop on that supercomputer ?
but there's talk of it all over campus.
Funny, I haven't heard anything about it prior to today. Guess I'm just out of the loop then...
-Julius X
remove "-whatkindofspamdoyoutakemefor-" from email to send
And it'll get skunked by 40 teraflops by Duke's supercomputer every year!
I don't want to start a holy war here, but what is it with you with you G5 zealots? Ive been sitting at my 1100 CPU G5 supercomputer for 20 minutes as it computers a fast fourier transform of an 8Ghz guassian. 20 minutes! At home, on my 60 cpu linux beowulf cluster, the same operation would take 2 minutes if that. Also, while this operation is takiing place, Doom III won't start, and everything else grinds to a halt, even my multithreaded emacs is struggling to keep up as i type this.
My Sun Enterprise 5000 is faster than this machine at times. Super computer addicts, flame me if you want, but I'd rahter hear some inteligent reasons why I should use the G5 supercomputer over cheaper, faster clusters.
I take it you don't look at Think Secret on a regular basis. It is, easily, the most accurate Mac rumors site out there. In fact, they have posted info on numerous occasions that has caught the attention of Apple's lawyers, and have been forced to pull down and issue their standard disclaimer. Say what you will about other rumors sites (most of them simply feed off each other) but there are some startlingly reliable sources informing Think Secret. Frankly, I don't recall the last time they were wrong about anything they've posted.
--Rick "If it isn't broken, take it apart and find out why."
Virginia Tech placed the dual-2GHz G5 order shortly after the G5 was announced. Multiple sources said Virginia Tech has ordered 1100 units
:-)
...)
Wow, that'll make Apple's quarter for sure
Seriously though, why PowerMacs ? I've always been under the impression that intelloid machines are the cheapest commodity hardware around for equivalent processing power, if not the most exciting. Would anybody know why Powermac G5s are a better choice here?
(Note to computer zealots: it's not a flamebait, it's a genuine question, from someone who is rigorously ignorant of the Mac world. And just in case, the first sentence is a joke, too
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
I got the following email the other day:
Virginia Tech is in the process of building a Terascale Computing Cluster which will be housed in the Andrews Information Systems Building (AISB). For those who are interested in learning more about this project, we will host an information session on Thursday, September 4th from 11 a.m. to noon in the Donaldson Brown Hotel and Conference Center auditorium.
We look forward to seeing you there
Terry Herdman Director of Research Computing.
I'll try to remember to take notes on this and let you all know if there's anything interesting...
Friends help you move. Real friends help you move bodies.
Yeah, chicks dig massive...computers.
No wait, no they don't!
Several years ago I did some work on some Virginia Tech "supercomputers" (actually, baby versions of ones on campus that were the same as huge ones they leased time on elsewhere), and I think the people talking about Altivec are on track. I never knew exactly what they did, but at that time the Math, CS, and Engineering groups were working together to simulate wing designs for the YF-22 jet figher prototype. Since I was more of a "sysadmin" (althoug h with a math and CS background) I ignored most of what was going on, but one thing I can tell you was vectors, vectors, and more vectors. The vector is king. It's an assumption, but I'll bet they are still working on similar type studies, and if built, this will be just the beast for it.
Exactly, Virginia tech has a goal to become a top 30 research university. Having known about the plan for some time, this makes perfect sense. The departments who are building the cluster, have gotten very large grants and donations from our great alumni to build this, and become a better university for it. This construction can be compared to the stadium expansions. The stadium expansion is paid out of a different set of funds, as is research. Academic fund is hurting because alumni rarely give money for academic reasons, but more for football or research.
The grant money that flows into a public research and occasionally teaching institution can be stagering, and absolutely dwarf the money students pay in tuition (sometimes by a factor of 10!). A better question might be, why don't the gradstudents donating their labor, possibly to patents that will be controlled by the university, recieve more consideration, and fair labor law protections.
But I would bet this will be not too dissimilar in use from the HP Itanium2 referenced earlier on slashdot. I would bet one of the paramount concerns this cluster would look at is the effect of farm runoff, and probably climatology too among other things.
--Jimmy has fancy plans; and pants to match.
>> A box designed to be separate just will not have the latency advantage of a supercomputer designed from the ground up.
I suggest you look at the list of the top supercomputers in the world. Most are clusters, ie. separate, distinct machines (just a quick glance shows the top 25 all are). It's just too darn hard to make a shared memory computer with 1000's of processors. So the common architecture is to make a cluster of smaller shared memory machines.
Besides, most clusters built utilize special interconnects like Myrinet that offer low latency connections. They're more expensive than ethernet, but it's a supercomputer so you spend it.
>> All this "the internet is one giant distributed computer" doesn't acknowledge this.
On the contrary... people know this very well. That's why we see rendering and SETI processing as distributed. They don't really need to communicate with others often.
C'mon guys, don't forget to say hi to the Olson twins when you see them on campus next year!
Wow! Imagine a Beowulf cluster of...
Er, ah, forget I said anything.
I watched C-beams glitter in the dark near the Tannhauser gate.
There are tradeoffs actually. This isn't like distributed.net or seti@home, this is a controlled network. They have complete control over the network switches, technology, and topology used and can design the network to accomodate tho problems the cluster will be designed to solve.
For example, you could use Myrinet to get 2 Gigabit, super low latency connectivity, or Quadrix, or Infiniband, or just a well laid out Gigabit Ethernet with high end switches.
In multiple processors in a box, the processors have to fight for the resources that box has to offer. NUMA alleviates demand on the memory, but IO operations (when writing to disk or to network) in a multiprocessor box block a good deal as the processor count in a node rises.
The idea with clusters is that inter-node communication in most cases can be kept low. Each system can work on a HUGE chunk of a problem on its own, with its own dedicated hard drive, memory subsystem, and without having too much competition for the network card. A lot of problems are really hard to solve computation wise, but are *very* well suited to distributed computing. A prime example of this is rendering 3D movies. Perhaps oversimplifying things, but for the most part, a central node divides up discrete parts (a segment of video), and each node works without talking to others until done, so the negative impact is minimal. Certain problems (i.e. nuclear explosion simulations where time and spacial chunks interact more with one another) are much more sensitive to latency/throughut. Seti@Home and distributed.net are *extremely* apathetic to throughput/latency issues (not much traffic and very infrequent communication).
XML is like violence. If it doesn't solve the problem, use more.
It all depends on what you need the cluster for. Some computations need constant communication, others can go on for hours, days or even weeks without feedback. If you're smart, you use supercomputers for the first kind of tasks and clusters for the second kind.
Universities (and big business) often work together and exchange resources. Virginia Tech gets a large amount of bargaining power by having control over a large amount of processing power. They can easily trade CPU time on their cluster for CPU time on a low-latency supercomputer.
The Drowned and the Saved - Primo Levi
While the AltiVec unit is very impressive, The SSE2 unit on the P4 or the Opteron would have nearly the same performance and cost a whole heck of a lot less (I am betting if this rumor is true at all, then Apple has given the units to the school).
Real world numbers don't bear this out. Check out the Photoshop and other application performance numbers for this. The gcc version used by the SPEC benchmarks used by Apple didn't even take advantage of AltiVec. When accounted for, and any institution making such a purchase would definitely have considered this, the AltiVec-enabled PowerPC chips totally spank x86 and others in number crunching tasks.
What I am wondering is, what OS is this cluster going to run? I mean, have the BSD folks figured out how to scale? No chance it will be OS X...maybe AIX?
An OS doesn't need to 'scale' to be a member of a cluster. It just needs to run the code locally and send the result back to the cluster master node.
___
Cogito cogito, ergo cogito sum.
With 1100 machines in the cluster, there must be _at least_ 2200 DIMMs. Since these must be 400MHz (PC3200) DDR, they can't be on a large 0.15 micron DRAM process, but most likely between 0.11 and 0.13u.
d /d imm_results.htm
Who cares?
APPLE G5'S DO NOT SUPPORT ECC.
The random bit error rate for 2200 DIMMs with 0.13u cells is roughly one '1' bit dropped to '0' every 9 hours. In other words: good luck getting any reliable, large-scale computation done with this cluster. (And I do mean "good luck" - they might get a run of two or three days without any problems once in a while.)
Now if only Apple would support PC3200 ECC DIMMS, which certainly do exist:
http://www.intel.com/technology/memory/ddr/vali
this cluster might be a bit more useful for real work.
I wonder if any universities have tried to write a distributed computing app along the lines of seti. Require it to connect to the university network, it grabs itself maybe 50 megs of hd space, and a fraction of all the new computers people bring to campus, in addition to all the computer lab gear belong to their massive number crunching problems. Make another version available to alumni, or even institutions as some form of corporate sponsorship.
Then if it got popular, and they were really clever, they could sell off a part of that computational power they amassed to solve other peoples problems providing for funding for new versions and new supercomputing clusters.
--Jimmy has fancy plans; and pants to match.
-I'm about to buy another in a day or two...
... there is a pretty big gap between asking for one and actually getting one. Tell you what, let me finish this gig I got happening in Virginia and then we can start dealing with the customers that want to order G5 machines in the onsies - twosies quantities.
... now THAT is hardcore.
You mean you are about to order another in a day or two
Sincerely,
Steve.
sjobs@apple.com
PS - Seven Mac's in ten years isn't hardcore. 1100 units ordered on one purchase order before they even ship (August, 2003)
Glonoinha the MebiByte Slayer
The SSE2 unit on the P4 or the Opteron would have nearly the same performance and cost a whole heck of a lot less.
Uh, no. 2 years ago, my roommate and I were both running the distributed.net client. I have a 500 Mhz Powerbook G4 (100Mhz bus). He had a 1.4GHz P4 with rambus RAM. I got 4Million keys/sec. He got 2MKeys/sec.
So clock for clock, my machine was nearly 4 times faster.
I have a shitty sig!
Latency is paramount for some tasks, less important for those that *can* make a good distributed project over the Internet of today.
Now, since today's supercomputers are *all* massively parallel constructions, the difference between a commercial design and an off-the-shelf cluster is in the quality and speed of the interconnects. NEC's Earth Simulator, the prime example of 'custom' supercomputer architecture, puts many processor units on *ridiculously* fast 'local' buses, and its racks are all interconnected with still_pretty_insanely_fast (and rather expensive) custom links.
Meanwhile, more 'commercial' designs use various interconnects. IIRC, NEC's 'regular' supercomputers, which formed the design basis for the Earth Simulator architecture, use Fibre Channel 'mesh' networks between racks. The Opteron - sure to be an up-and-coming player in this market - offers HyperTransport, which it looks like Cray will be stretching to its limits on Red Storm; I'm not sure *how* long an HT bus can be, but one gets the impression they'll be stretching it as far as possible, and it's certainly high throughput/low-latency versus the technologies you'd usually find in use for 'networking.'
Anyhow, point is, those designs pack a lot of CPUs together with *very* fast interconnects (equivalent to 16, 32, 64+-way SMP), and have lots and lots of racks of those. (The Opteron/Red Storm approach sounds sexy to me, because I think Hypertransport should let them pack 'lots and lots' of CPUs together versus existing designs. I've yet to read anything about what they're actually doing with it, though.)
Now.. In contrast, an 'off the shelf' cluster is usually going to stick with Ethernet, and will only have 1 to perhaps 4 processors per [node-unit-where-the-CPUs-are-connected-on-a-fast- local-bus], depending how affordable 'cheap' multiprocessor systems are at the time. But *everyone* building supercomputers bumps up against the latency/routing problem; it's just a question of whether it's a problem for, say, 50 Earth Simulator racks (aren't there quite a few more?) vs. 1100 PowerMacs. Experimenting with 'lots of little nodes' has led us to better understand the problem, and learn how to produce tuned topologies that can compete favorably with 'purpose-built' hardware. See: http://aggregate.org/KASY0/
Now, the question *is* one of cost-benefit. Large supercomputers tend to be built with maintenance features and power efficiency in mind. In turn, a totally 'off the shelf' cluster like KASY0 has some advantages because each machine is a cheap, practically disposable 'module' unto itself, and can doubtless be downed off the cluster, pulled out and replaced with another while being easily bench-repaired (since, after all, it's a self-contained PC, rather than a CPU blade or some other random card that would require an expensive test rack to troubleshoot). Meanwhile, if you absolutely demand low-latency, you want one sort of design (Red Storm seems to be acheiving it 'on the cheap,' by combining off-the-shelf - and thus cheap - chips and buses with smart 'custom-design' engineering) while if you can sacrifice some for throughput (jobs with few conditionals), you want another... (like 1100 G5 Macs on a shelf, wired with 'boring' gigabit ethernet, especially if Apple is giving you a bulk discount on the hardware).
So what I'm trying to say is... this is a *combination* of PR stunt and intelligent planning, and there's certainly a lot of 'good science' they could do with the beast - both in number-crunching and 'computer science' a-la cluster topologies. Whether they'll actually *use* it for such, or if it'll be solely a topology toy is anyone's guess.
I think there's some hope that it'll be the "Real Thing," though, since this would explain some of the weird rumors about FC-on-the-mainboard Macs. So they get a Real Monster, made of what will be revealed as "the new G5 Xserves" at the unveiling. The best of COTS *and* fresh d
i would take this story to imply that a G5 powered Xserve is not going to be shipping anytime soon..... the Xserve is made to cluster and run in situations like this. i guess the rumor sites can speculate if it's G5 parts available or some other holdup on a G5 Xserve.
/. a year or so ago about a group that went from building a rack and unboxing their G4s to a running cluster in part of a day. i really don't remember the specifics but i think it was something like 30 G4s? i would guess the G5 is not that much harder... and they seem to have Apple helping. maybe they hooked up the optical cards from the Xserve...... we'll see i guess.
unless there is some reason the desktops are better for this project that i did not pick up on?
as for the above question about Macs.... depending on what they want to really do with this, Altivec is really efficient for some computations. all flame wars aside there have always been people clustering Macs for certain uses. i do not know how much of it was user preference or the software they wanted to run or the simplicity of getting the cluster running.
it is supposedly VERY simple to cluster Macs. there was a story on
I've been tempted to order a dual G5. I've resisted the temptation by realizing that my only real reason for wanting it would be to awe friends and co-workers. Pretty shallow. I was ashamed.
What a surprise to find that the folks who buy multi-million dollar supercomputers seek some of the same shallow satisfaction that moves me--bragging rights.
Still, if a single order for 1100 units causes significant delays filling orders for other customers, Apple must not have been expecting to sell many of these things. Maybe I should place an order just to help out.
When all you have is an axe, everything looks like a grindstone.
As Zack pointed out, iWalk was not a Think Secret report; in fact, we debunked it. For WWDC, we reported that Apple would announce 64-bit Power Macs as well as a videoconferencing camera that we said would be called "iSight," -- I think we're in the clear there. iWorks? I maintain that it is still a future Apple release. As for 12-inch and 17-inch PowerBooks, while we raised the possibility of a release that week, we specifically said we couldn't confirm the delivery date: "It's unclear when Apple plans to announce the upgrades..."
Bottom line? Like any other news organization, Think Secret has occasional misses. But those misses don't appear to include any of the items mentioned here. I think our record speaks for itself.
Nick dePlume
Publisher and Editor in Chief
Think Secret
For the ones who are questioning this existence, the order is shipping, the racks (a ton of them) are there in the main Computing Center server room. First they required all servers to be moved innto racks. Then they started moving servers around, including removing the Petaplex. The power has been upgraded in the server room (the UPS backup generator actually). This caused a morning of basically all the important servers on campus having to go down for one day in the summer - I hated waking up to go switch off machines for that one. The AC has been upgraded to accomidate the huge amount of heat to be put out. It was't until I heard about the cluster that all the chages in the Machine Room made sense. Now they're recruiting help to do the grunt work of putting all the machines in the racks.
/. link to the campus geek list (If someone hasn't beaten me to it).
/. since a cluster this size is noteworthy of the frontpage. (Rumor - and this is rumor sice I haven't goe to direct sources on this - is that it will not be running OS X, and probably BlackLab or YellowDog or SuSE.)
The stated objective was to be on the next 500 list. Dell and HP were considered, but they couldn't fill the order in time (possibly as they have made announcements of other large clusters recently) and Apple promised delivery after someone leaked the story of the cluster meetign with Dell and HP to Apple and Apple jumped at the chance.
Basically, the story is not a rumor from the point of view of the geeks on campus who have been effected by the preperations. I'll probably post the
I'm disapointed about this being only on the Apple section of
I am, and always will be, an idiot. Karma: Coma (mostly effected by
You got a link for dual Opterons for 1500 bucks with all the goodies in the G5? If yes, Id love to see it.
:) )
(seriously - I can take out a loan
Obviously you haven't looked at VT recently. Tuition and fees is only $7,500, out of state. I can only wish that my tuition were that low. Hell, for in-state students, the room and board is the same price as tuition (around $2,000). But of course, you're modded up insightful, because you pulled a random idea out of your ass and presented it as fact.
I'm a lawyer, but not yours. I wouldn't represent someone who thinks taking legal advice from Slashdot is a good idea.
No kidding. My 667Mhz Powerbook G4 gets 6.8 MKps, my friend tried it on his 2.53Ghz P4 and got 3.4 MKps, and my 1.4Ghz Athlon XP 1600+ gets aroung 4.5 MKps. I calculate a Dual 2Ghz G5 around 40 MKps, probably more. Imagine them running distributed.net on this bad*** cluster!!!! YOW baby!
Ron Paul 2012
And 8 hours@12.4GFlops...damn you Virginia Tech, you owe me a third of a quadrilion floating point multiplies!
1. The PPC970 draws from the Power4 lineage, which I have used for a long time. The PPC970 has 2 double precision FPUs, each capable of fused multiply add instructions leading to 4 flops/cycle/processor (2 units*2flops/cycle). This is identical to the Itanium2 FPU microarchitecture. The Opteron on the other hand can only do 2 double precision flops/cycle, which makes it only half as powerful on matrix heavy scientific computations, when compared to the PPC970 or the Itanium 2. The PPC970 should really be compared in FP terms to the Itanium2 at 1/10th of the cost, and at 2GHz it is clocked higher than the top-end 1.5GHz Itanium2 Madison. Moral of the story, read thy arstechnica. 2. The standard benchmarking process (LINPACK) only uses double precision FP. If this rumor is true, then this machine is capable of an Rpeak (LINPACK) of 17.6 Teraflop, which those of you who follow top500 will realize is quite substantial. 3. If they are really using Infiniband, this should be a nice machine. Infiniband provides 10 Gbps (20 Gbps full duplex) of bandwidth, which is much faster than either Myrinet or Quadrics. Also Infiniband latency is 10us and the benchmarking process is bandwidth not latency sensitive. On the other, this stuff is really expensive. If all of this is true, this would be a major engineering endeavor. Also, it is probably cheap. However, all in all, this could well just be a rumor (come on it is thinksecret - remember iWorks). If not, this should be a fairly substantial machine.
It's just too darn hard to make a shared memory computer with 1000's of processors. So the common architecture is to make a cluster of smaller shared memory machines.
It's hard, but not too hard or impossible. The Silicon Graphics Origin 3000 supports 512 processors in a single image system with the stock IRIX kernel and 1024 processors with the "XXL" kernel.
Rumor has it Origin 4000 will support 2048 processors, as will Altix once SGI has done some major work with their kernel patches. (Altix is currently limited to 64 processors per system image).
Damn, ethernet controllers must really piss you off then, huh?