10-TFlop Computer Built from Standard PC Parts

OrangeTide writes "Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply. More information can be obtained from this article at EETimes." Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply.

Comments

imagine the future

[ryochiji]

From the article:
>The 1- to 10-teraflops processing range is opening up a revolutionary capability for scientific applications

In the not too distant future, that kind of processing power could very well be available in home PCs. Imagine what that would do to...well, I mean, dang it, what the heck will we do? Game frame rates can only go so high. Even realism of 3D graphics may have it's limits. Oh sure, we'll find something, but it's difficult for us to imagine now...

Re:Connections through PCI bus?

[tap]

There are chips designed to connect two PCI busses together, called PCI-PCI Bridges. For instance, I have an Intel dual port ethernet card with one:

Bus 0, device 12, function 0: PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 3). Master Capable. Latency=64. Min Gnt=4.

But you can't use this to connect a rack of computers. For one thing the max cable length for connecting two busses would be just a few inches. For putting PCI cards in 1.75" high 1U rackmount cases, there are PCI risers with a short ribbon cable that connects to the PCI slot. Even these short cables often cause timing problems. For instance, with the riser, cards may only work in the first one or two slots that will otherwise work in all the slots.

But even if you could cable all the computers together on one giant PCI bus, it would still be a bad idea. A good 24 port gigabit ethernet switch (~$2000) has a 480MB/sec switching fabric, to support full speed full duplex on each port. 32 bit 33Mhz PCI is only about 132 MB/sec, not nearly as fast. You'd need a 64 bit 66 Mhz PCI bus to keep up. And there are more expensive gbit switches with more ports that have 100 Gbit/sec fabric. And this is just gbit ethernet, the slowest and cheapest of the high speed interconnects used in modern Beowulf clusters.

There are faster ways to connect computers than gigabit ethernet. The EE times article is very untechnical, but this one has some more information. LLNL has used a very fast and very expensive interface called quadrics. This is probably the fastest way to connect computers in a Beowulf. People like Cray/SGI and IBM have faster things still, but they cost real big bucks. Other ways to connect a Beowulf are the above mentioned gigabit ethernet (~$100-$250 a node for up to 24 nodes), myrinet (~$1400-$2000 /node up to 128 nodes), and SCIhardware and software (~$1400-$2100 /node). Myrinet uses a switch like gigabet ethernet and the largest switch they have is 128 ports. SCI is switchless, each card has multiple cables (1-3), and is connected in into a ring, 2D or 3D torus.

This is not "Big Iron"

[sdeath]

The title says it all. Big Iron is _engineered_. No matter how big or how spiffy a Beowulf cluster is, it's still just a bunch of PC motherboards kludged together with a bunch of network cards. There is a reason Crays are expensive - they are _worth it_ from a performance standpoint, because not every problem lends itself easily to the solution of a Beowulf cluster. Some problems require the exchange of a lot of data between a lot of nodes, and a little math will show that it won't take much data interchange to saturate even a GigE switch. Adding more machines is not going to help; craftily designing and overengineering the network _might_, but by the time you get this whole damned thing glued together well enough to approximate a Cray's performance, you'll have spent enough to have just flat-out bought a Cray in the first place.

As others have noted, while this thing may have a theoretical peak performance of 10 TFLOPS, I'm willing to bet that number goes down like Monica Lewinsky on Quaaludes when you feed this magical supercomputer a problem that's _not_ suitable for distributed.net (i.e. one where computations on one node are dependent on computations on another node, like fluid-dynamics problems, turbulence, etc.)

Yeah, it's interesting as a curiosity, but this is by no means spectacular. Beowulf is good for what it's good for, which is a "poor-man's supercomputer" that works well for coarsely-parallel problems that don't require a lot of internode communication. It's not the Philosopher's Stone, folks.

-SD

Re:Processing power

[FuzzyDaddy]

So the question is (and I don't know, I didn't study nuclear physics beyond A-level), are the significant computational problems associated with the development of nuclear weapons easy to parallelize, or do they require a real supercomputer [sgi.com]?

I believe the calculations needed are massive finite element calculations. And I would imagine that things happen quickly enough in a nuclear explosion that there's a lot of significant stuff going on over a time period much shorter than it takes for any change to move from one side of the simulated device to the other.

As an analogy, suppose you wanted to simulate a large number of gravitating bodies. You would break the problem up into sections. Even though each body acts on every other, bodies outside a certain distance can be treated by their average force. So you can simulate things near each other on the same node, and have the nodes talk to pass the information about the "average" field. It requires some communication between nodes, but a large amount of work can be done on an individual nodes.

Or for your gas example, if you broke the problem up into boxes, you would have to "hand off" a particle as it passed from one box to another, and perhaps pass off information about forces close to the box boundaries. But if a lot of stuff is happening in a single box (like, say, chemical reactions), you can still get a big benefit out of parallalization.

Also, if designing nuclear bombs is anything like designing microwave components, you would have several simulations going at the same time, to try different variations on one design. Or you would design several subparts and have them running at the same time.

In short, I think that the problem very much lends it self to parallel computing.

Re:Processing power

[LWATCDR]

Since the first Atomic Bomb was made in 1944-45 and worked the first time. All you need is a computer equal to what they had in 1944.
To make a small portable nuke is harder.