10-TFlop Computer Built from Standard PC Parts

OrangeTide writes "Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply. More information can be obtained from this article at EETimes." Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply.

Comments

Imagine...

[Anonymous Coward]

A commodity supercomputing cluster of these! (There has to be a better name for it, but I'm new here on Slashdot).

Supercomputer developed for...

[jabex]

... which was specifically developed for running Doom III.

Re:Supercomputer developed for...

[Charles Dodgeson]

... Doom III

Well, considering who's building it and running it, it's reasonable to guess that some form of doom will be simulated on it.

Howlingly funny?

[noitalever]

As long as it doesn't see a full moon, then it won't turn into...

Beowulf!

zzziiiippp!!!!(I love the smell of nomex in the morning!)

imagine the future

[ryochiji]

From the article:
>The 1- to 10-teraflops processing range is opening up a revolutionary capability for scientific applications

In the not too distant future, that kind of processing power could very well be available in home PCs. Imagine what that would do to...well, I mean, dang it, what the heck will we do? Game frame rates can only go so high. Even realism of 3D graphics may have it's limits. Oh sure, we'll find something, but it's difficult for us to imagine now...

Re:imagine the future

[Charles Dodgeson]

In the not too distant future, that kind of processing power could very well be available in home PCs. [...] Oh sure, we'll find something [to do with it], but it's difficult for us to imagine now

the Search for Extremely Trivial Iterations at home.

Re:imagine the future

[jericho4.0]

I know exactly what I want to do with this kind of power. I want a massive FPS with a wicked frame rate. I want every object and material in that world to react with me. Newtownian physics down to grains of sand. Nothing short of movie quality realisim.I also want it to be massivly multiplayer. That will require some huge bandwidth and IO. Get on it Livermore!

Re:imagine the future

[darkgreen]

Actually, it would be pretty cool to see a community that contributed to all the little details - kind of like a co-op world.

then you'd have this huge pool of resources from which to draw - no time to model and texture that bookcase you need for a room? just go and buy it.

grin, there could be auctions, just like real life, and since the 'structures' and objects you'd be buying have taken time and effort to create, and will have varying degrees of craftsmanship, you'd have a chance that it would actually turn into a market.

the value, of course, would be that everybody would have to be able to use the resources, resell and develop their own parts of the world.

we've seen something vaguely similar in the mod community since Doom (the original, kiddies), but it's not really there yet.

Re:imagine the future

[commodoresloat]

Oh sure, we'll find something, but it's difficult for us to imagine now...

Come on; you know what we'll do with this; what we do with every major technological advance: new, more realistic, and more processor-intensive pr0n.

We use it!

[e8johan]

Were I work (I can't say where... I've signed papers...) we have replaced an SGI 'super computer' (or mini computer, whatever, a big number crunching beast of silicon!) with a Beowulf cluster. This not only gives us great scaleability, but also lots of FLOPS per dollar (or rather, krona :^P).

Re:We use it!

[Anonymous Coward]

I know you "signed papers", but all McDonald's employees have to sign that application before they can start working. And just because you FLIPS a lot of burgers doesn't make you an expert on supercomputers.

Re:We use it!

[ceejayoz]

Were I work (I can't say where... I've signed papers...)

Are you referring to this part of your CV?

Gardening
Juli 7 - 27 1986 i managed a group of tomato plants.

Re:We use it!

[e8johan]

Yes! I used the cluster to compute the correct amount of water for each plant, considering cloudiness, temperature, time in direct sun light and how far each particular plant had grown. :)

Re:We use it!

[darkgreen]

Mini computers are the dinosaurs of old.

Hey, wouldn't dinosaurs be the dinosaurs of old? Mini computers would be the dinosaurs of kinda recent. =)

Wait until the weapons inspectors get to Iraq!

[joeflies]

Then the world will finally see the 4000 Playstation 2's that Saddam used to build a supercomputer

Re:Wait until the weapons inspectors get to Iraq!

[7-Vodka]

or maybe just one apple mac. Sitting all alone in a big empty room because saddam bought the marketing spiel.

zerg

[Lord Omlette]

Look, all the 'cool' people are doing Linux, right? But BIG IRON is clearly trying to suck you and your money in. Oh, it's so cheap, but they don't tell you about the hidden costs, do they?

Just say no to BIG IRON!

the world's fastest machines

[Wild Bill Hickock]

can be found here.

Parallel computing

[vlad_petric]

The difficulty is not to conglomerate processing power ... you can do that relatively easily with Benjamins ... the real difficulty is in either parallelizing your computations, or making a single processor work faster.

So the Teraflops they're mentioning are just a theoretical upper bound, don't get too aroused when you see it.

The Raven.

Re:Parallel computing

[Hawaiian Lion]

I've done a little parallel programming working at my college for a professor as well as an internship for Gemini Observatories in Hawaii.

From what I've seen, just about any simulation involving large systems of particles can be fairly easily parallelized code-wise. These are mostly the sort of problems that require massive processing power in the first place.

I can't think of a reason why we shouldn't be getting hyped about these teraflops. We use a 8 node AppleSeed cluster at work and I've seen that thing hump out 4-6 gigaflops of crunching power. It takes as long as a week to run some of our molecular dynamics simulations. If we had 10 teraflops of power in our hands those simulations could take somewhere on the order of minutes instead of days.

Re:Parallel computing

[sql*kitten]

I can't think of a reason why we shouldn't be getting hyped about these teraflops. We use a 8 node AppleSeed cluster at work and I've seen that thing hump out 4-6 gigaflops of crunching power. It takes as long as a week to run some of our molecular dynamics simulations. If we had 10 teraflops of power in our hands those simulations could take somewhere on the order of minutes instead of days.

As an aside, I have to wonder whether or not that's a good thing. I have noticed in myself and almost everyone I've worked with that having massive amounts of CPU at your disposal makes you sloppy - people tend to take a "shotgun" approach, rather than thinking through a problem, they just "try something" until it works. Of course in some cases, CPU really is cheaper than developer time, but in just as many cases, it's an excuse for laziness. I see this all the time, people will build an over-complex solution using technologies like J2EE and EJBs when something much simpler and more efficient would suffice. For another example, every Slashbot who has complained about bloat in MS Office knows exactly what I mean.

Roll on the teraflops, but not before developers have the self-discipline to use them well.

Is this a big deal?

[Ed Avis]

The important part isn't the number of FLOPS (to get those you can just keep buying more PCs until you reach the desired number) but the performance in applications which are not 'embarassingly parallel'. In other words how good is the interconnect between machines? The article talks about a new network to replace Gigabit Ethernet.

Interesting Approach on Network

[jki]

Selected clips:

The system has a few unique features that the lab says will facilitate applications performance, including a fast, custom-made network that taps into an enterprisewide file system.

"This network approach is nice because we can use a standard PCI slot on each processor node, which gives a 4.5-microsecond latency," he said, as opposed to 90-s latency for Gigabit Ethernet."

The boards are linked by a network assembled by Linux Networx into a clustered system that will have 960 server nodes.

The file system, called Lustre, uses a client/server model. Large, fast RAM-based memory systems support a metadata center, and data is represented across the enterprise in the form of object-storage targets. "Being able to share data across the enterprise is an exciting new capability

I think this is especially interesting, because it seems to glue together pieces from traditional clustering and distribted or metacomputing. Is there some site for this project with more details?

Re:Interesting Approach on Network

[hawkstone]

More details -- probably more than you want :) -- here:
http://www.llnl.gov/linux/mcr/.

who ever said ...

[valmont]

... that penguins couldn't do steroids?

Connections through PCI bus?

[Dr. Spork]

Do I understand correctly that they just wired PCI slots from different motherboards together, instead of running the data around over ethernet (which probably would have been plugged into a PCI slot anyway)? If so, I mean, if there's nothing more to it than that, it seems like this will be a kickass way of clustering. But there must be something more to it than I realize, because if there wasn't, there wouldn't be so many ethernet-based beowulf systems.

So please explain this. I mean, I have two linux boxes in my room and each has a free PCI slot. What do I need to to to network them over directly over PCI?

Re:Connections through PCI bus?

[Dr. Spork]

Wouldn't they have bus timing issues? There must be some extra piece of hardware involved... If that's the case isn't it basically just a new kind of NIC?

I don't know much about this type of stuff, but wouldn't it be awesome if the way they made it work is through software? If they did, Linux/Beowulfing would be about to take a huge leap forward.

Anyway, maybe this is not the sort of thing you can solve with software. Whatever they're using to connect the computers sounds very practical though, not just for supercomputers but for general fast networking. When can we do this at home?

Re:Connections through PCI bus?

[tap]

There are chips designed to connect two PCI busses together, called PCI-PCI Bridges. For instance, I have an Intel dual port ethernet card with one:

Bus 0, device 12, function 0: PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 3). Master Capable. Latency=64. Min Gnt=4.

But you can't use this to connect a rack of computers. For one thing the max cable length for connecting two busses would be just a few inches. For putting PCI cards in 1.75" high 1U rackmount cases, there are PCI risers with a short ribbon cable that connects to the PCI slot. Even these short cables often cause timing problems. For instance, with the riser, cards may only work in the first one or two slots that will otherwise work in all the slots.

But even if you could cable all the computers together on one giant PCI bus, it would still be a bad idea. A good 24 port gigabit ethernet switch (~$2000) has a 480MB/sec switching fabric, to support full speed full duplex on each port. 32 bit 33Mhz PCI is only about 132 MB/sec, not nearly as fast. You'd need a 64 bit 66 Mhz PCI bus to keep up. And there are more expensive gbit switches with more ports that have 100 Gbit/sec fabric. And this is just gbit ethernet, the slowest and cheapest of the high speed interconnects used in modern Beowulf clusters.

There are faster ways to connect computers than gigabit ethernet. The EE times article is very untechnical, but this one has some more information. LLNL has used a very fast and very expensive interface called quadrics. This is probably the fastest way to connect computers in a Beowulf. People like Cray/SGI and IBM have faster things still, but they cost real big bucks. Other ways to connect a Beowulf are the above mentioned gigabit ethernet (~$100-$250 a node for up to 24 nodes), myrinet (~$1400-$2000 /node up to 128 nodes), and SCIhardware and software (~$1400-$2100 /node). Myrinet uses a switch like gigabet ethernet and the largest switch they have is 128 ports. SCI is switchless, each card has multiple cables (1-3), and is connected in into a ring, 2D or 3D torus.

I need one!

[Newer Guy]

I have a lot of movies to convert to DIVX...

Re:Does that mean...

[Dr. Spork]

You know, it wouldn't be stupid of Apple to try to build in some code for arbitrarily large clusters into Darwin. It would really be a prestige coup if a mac cluster became a top-500 computer.

I told you....

[stephenisu]

I told you my other Boxen was a 1000 node beowulf cluster... But no one believed my sticker...

yeah, well..

[radon28]

"There is no reason anyone would want a computer in their home." - Ken Olson, President, chairman and founder of Digital Equipment Corporation, 1977.

Processing power

[rovingeyes]

Actually, your statement made me wonder for a while. I remember that till not long ago, US wouldn't let other countries buy latest super computers becoz they feared it'd be used to do those nuclear explosion simlations. Now I'm not sure if it still is the case.

Anyways, what I'm trying to point out is that it is actually becoming very convinient to build a super computer with lots of PCs that just lie idle. I am not sure if Saddam has heard about cheap linux systems. But what if he could build a super computer cluster?

Boy this gets interesting and scarier at the same time.

Re:Processing power

[sql*kitten]

Anyways, what I'm trying to point out is that it is actually becoming very convinient to build a super computer with lots of PCs that just lie idle. I am not sure if Saddam has heard about cheap linux systems. But what if he could build a super computer cluster?

Well, it depends. A Linux cluster is a good way to render a movie, because you can easily parallelize that task - send a frame to each node you've got, wait for it to come back, send out the next one, then when you're done composite them into an animation. That's easy, because you can make each task essentially stateless. For example, you don't have to wait for frame 1 to rasterize before you know how to light frame 2.

But in many scientific computations, there is a limit to how you can subdivide a task. Say you are modelling the movement of a gas in 3 dimensional space, you cannot partition your space 3x3x3 and send it to 27 compute nodes, because what happens in each partition both influences and is influenced by what happens in adjacent partitions. If you did try to do something like this on a cluster designed for rendering movies (or brute forcing a cipher, or serving web pages) performance would be terrible because of the overhead of communication between nodes. For that, a Single System Image machine has a vast advantage.

So the question is (and I don't know, I didn't study nuclear physics beyond A-level), are the significant computational problems associated with the development of nuclear weapons easy to parallelize, or do they require a real supercomputer?

Re:Processing power

[FuzzyDaddy]

So the question is (and I don't know, I didn't study nuclear physics beyond A-level), are the significant computational problems associated with the development of nuclear weapons easy to parallelize, or do they require a real supercomputer [sgi.com]?

I believe the calculations needed are massive finite element calculations. And I would imagine that things happen quickly enough in a nuclear explosion that there's a lot of significant stuff going on over a time period much shorter than it takes for any change to move from one side of the simulated device to the other.

As an analogy, suppose you wanted to simulate a large number of gravitating bodies. You would break the problem up into sections. Even though each body acts on every other, bodies outside a certain distance can be treated by their average force. So you can simulate things near each other on the same node, and have the nodes talk to pass the information about the "average" field. It requires some communication between nodes, but a large amount of work can be done on an individual nodes.

Or for your gas example, if you broke the problem up into boxes, you would have to "hand off" a particle as it passed from one box to another, and perhaps pass off information about forces close to the box boundaries. But if a lot of stuff is happening in a single box (like, say, chemical reactions), you can still get a big benefit out of parallalization.

Also, if designing nuclear bombs is anything like designing microwave components, you would have several simulations going at the same time, to try different variations on one design. Or you would design several subparts and have them running at the same time.

In short, I think that the problem very much lends it self to parallel computing.

Re:Processing power

[LWATCDR]

Since the first Atomic Bomb was made in 1944-45 and worked the first time. All you need is a computer equal to what they had in 1944.
To make a small portable nuke is harder.

PCI Null-Modem

[Bios_Hakr]

Uuh, I mean null-card connection. I have never really looked at the PCI spec from an eletrical engineer standpoint, but there are probably power leads, data leads, timing leads, and ground leads on there.

The data leads should be easy...TX to RX. Although they may use a full-duplex lead where the data shares the bus based on clock pulses.

The power could be dropped, as both machines already have the proper power requirements. The ground leads could be tied together if you wanted, but dropping them shouldn't have too much impact on the final outcome.

The tricky part would be the clock pulses. In order to keep the data integrity, you need to have both bachines on the same clock. The easy way would be to take the crystal from one motherboard and wire it to the other. Same crystal, same clock pulse.

Then drivers would be needed to make the other computer look like an attached device. Shouldn't be too difficult. Just take a NIC driver and modify it...heavily.

I think an easier option would be to share data across the IDE bus. Make an IDE driver look like a NIC driver and send IP across IDE. In fact, I remember Linux Journal publishing an article about someone doing IP over SCSI about 2 years ago. Get some SCSI cards and make your own version of a CDDI network ring.

Re:PCI Null-Modem

[Bios_Hakr]

AGP is only useful for one-way communication. Great for shoving data to the monitor, but sucks for pulling data from an outside source. I'd say if you really wanted a challenge, use the memory bus for networking. The prob would be timeouts while waiting for the data to be pulled across the wire from the other machines' memory bank.

Or, go for broke and use the second processor slot in a dual mobo.

On the cheap end, you could use the USB and a null-modem cable there to link 2 boxes.

I can see the adds now..

[PerryMason]

"I was doing my nuclear simulations on the ASCII White and it was like BEEP BEEP BEEP...and like half my work was gone..."

Re:Why XEONs?

[tap]

If you check current prices, the Xeon isn't much more expensive than the AthlonMP. Pricewatch has the 2.2Ghz xeon at $245 and the athlonMP 2200+ at $204. Each of these machines is interconnected with a Quadrics board that probably costs more than $2000, so an extra $80 for CPUs isn't much.

Why not use AMD anyway? There are xeon motherboards with chipsets like the Intel E7500 and ServerWorks GC-HE that have greater memory bandwidth and PCI bandwidth than the AMD 760MPX. For many problems in scientific computing, memory bandwidth is what is important, not CPU speed.

(Open)MOSIX?

[Jeppe Salvesen]

Anyone have any experience using (Open)MOSIX? I have a partially CPU-bound application (automatic part is IO-bound, manual part is CPU-bound) in Perl, Apache and MySQL. Anyone got experience with this stuff?

For those who don't even know what MOSIX is, it is a kernel patch that essentially creates a virtual computer out of several boxes. They claim they will scale your application as long as you have multiple processes (they migrate them as needed) - without any coding on your part.

Since I'm looking for extra performance with limited resources, this looks like a potentially easy way out :)

This is not "Big Iron"

[sdeath]

The title says it all. Big Iron is _engineered_. No matter how big or how spiffy a Beowulf cluster is, it's still just a bunch of PC motherboards kludged together with a bunch of network cards. There is a reason Crays are expensive - they are _worth it_ from a performance standpoint, because not every problem lends itself easily to the solution of a Beowulf cluster. Some problems require the exchange of a lot of data between a lot of nodes, and a little math will show that it won't take much data interchange to saturate even a GigE switch. Adding more machines is not going to help; craftily designing and overengineering the network _might_, but by the time you get this whole damned thing glued together well enough to approximate a Cray's performance, you'll have spent enough to have just flat-out bought a Cray in the first place.

As others have noted, while this thing may have a theoretical peak performance of 10 TFLOPS, I'm willing to bet that number goes down like Monica Lewinsky on Quaaludes when you feed this magical supercomputer a problem that's _not_ suitable for distributed.net (i.e. one where computations on one node are dependent on computations on another node, like fluid-dynamics problems, turbulence, etc.)

Yeah, it's interesting as a curiosity, but this is by no means spectacular. Beowulf is good for what it's good for, which is a "poor-man's supercomputer" that works well for coarsely-parallel problems that don't require a lot of internode communication. It's not the Philosopher's Stone, folks.

-SD

Networks and parallel algorithms are the key

[dsfd]

The distributed memory Crays (T3D, T3E) are just the same: boards and network cards. The processors they use are not faster than the last generation PC processors. The difference are the NICs, that have about 10 times more bandwith and 10 times less latency (compared with standard fast ethernet cards).

There is the difference. As you say, for certain problems, this means that the whole machine is about 10 times faster than a Beowulf.

However, if/when conventional NICs are fast enough, specially in terms of latency, both systems can be equivalent again. In the meantime, a lot of people are trying to develop parallel algorithms that minimize the number and size of the messages, allowing to use cheap PCs as supercomputers.

Re:The real question is

[mpe]

Answer: Depends on his intent. If he is using it for finding extra terrestrial life, by all means he can go ahead, but if he is using it to test one of his biological weapons then he is obviously bad.

What if he finds some ETs who can help him out with some guy, known as GW Bush, who wants to invade his country.

"total cost of ownership" against off-the-shelf

[peter303]

2/3rds the cost of the three year computer lifetime is the electricity and cooling system. When TOC is counted a transmeta based cluster or the super-dese SGI cluster announced yesterday is cheaper.

Babelfish [Engineer - Geek]

[grub]

"We have been using the File Transfer Protocol over Gigabit Ethernet, but now we will be able to read files directly from any available disk."

translation
We used to use FTP over Gig-E but came up with something more L337.

Re:Why XEONs?

[fitten]

- Depends on the Xeons they are using. The 'old' Xeons are around the same cost as their AMD counterparts. The 'new' Xeons have large L3 caches (1M and 2M).

- The AMD SMP chipset is slow (memory bandwidth) compared to the newer Intel chipsets.

- IIRC, the P4s use less power than the Athlons, probably this is not as important but it is there.

I'd like to see a comparison of a newer dual Xeon machine vs. a good dual AMD to see the performance difference. I would suspect that the dual Xeon machine would be a bit faster.

Computer engineering is.....

[Conspiracy_Of_Doves]

a race between engineers trying to make faster and better computers and Microsoft trying to make more bloated and processor-heavy operating systems. So far, Microsoft is winning.

On the other hand, it beats using a slide rule!

[SwedishChef]

One of the benefits of computers is the ability to solve a problem with iteration rather than trying to come up with a classic "equation" and solve it. When I first entered the job market I had a trusty Pickett N4ES slide rule (and an N600-ES pocket slide rule) and had to first explain a problem with an equation and then solve the equation (from the "inside out" which was why HP calculators with RPG were so popular with engineers when they first came out versus the TI models... but I digress).

With the introduction of the HP-35 calculator (the "electronic slide rule") we could solve problems by just crunching the numbers at our desks. With the availability of programmable calculators (HP-67/97 and HP-41 - both of which I still use... but then I still use the slide rules too) we could program them to iterate through problems.

Not as elegant, certainly. But lots more efficient. And I'm sure that most of us have lost some of our old abilities to "see" problems in math... and perhaps some students never really learn that. But the jobs still get done and the tools still keep making it easier. I'm thinking about a Beowulf cluster for our office, actually.

More theoretical than you think

[tyler_larson]

Yes, I saw this computer before it was delivered to LLNL (hardly off-the-shelf parts, BTW). Very sleek looking, though.

The interesting thing about this setup is that it doesn't work like the traditional supercomputer. It's more like a community of totally independant computers all willing to work on the same problem.

The system employs a whole lotta control nodes that spend their whole time trying to assign work out to the worker nodes. The problem then becomes not just parallelizing the work but coordinating the workers. Apparently with this cluster design, it's not all as cut-and-dried as with a "real" supercomputer. They have been able to do some really cool stuff, though. Like, for example, any computer in the cluster can address the memory on any other computer.

The admins I talked to said they weren't really sure just how fast the system could go, because they could never get it to operate at full capacity. They said the fastest they'd gotten it to go was 4T-Flops, but they figured they were only at %40 theoretical capacity.