10-TFlop Computer Built from Standard PC Parts

OrangeTide writes "Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply. More information can be obtained from this article at EETimes." Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply.

Comments

We use it!

[e8johan]

Were I work (I can't say where... I've signed papers...) we have replaced an SGI 'super computer' (or mini computer, whatever, a big number crunching beast of silicon!) with a Beowulf cluster. This not only gives us great scaleability, but also lots of FLOPS per dollar (or rather, krona :^P).

Parallel computing

[vlad_petric]

The difficulty is not to conglomerate processing power ... you can do that relatively easily with Benjamins ... the real difficulty is in either parallelizing your computations, or making a single processor work faster.

So the Teraflops they're mentioning are just a theoretical upper bound, don't get too aroused when you see it.

The Raven.

Re:Parallel computing

[Hawaiian Lion]

I've done a little parallel programming working at my college for a professor as well as an internship for Gemini Observatories in Hawaii.

From what I've seen, just about any simulation involving large systems of particles can be fairly easily parallelized code-wise. These are mostly the sort of problems that require massive processing power in the first place.

I can't think of a reason why we shouldn't be getting hyped about these teraflops. We use a 8 node AppleSeed cluster at work and I've seen that thing hump out 4-6 gigaflops of crunching power. It takes as long as a week to run some of our molecular dynamics simulations. If we had 10 teraflops of power in our hands those simulations could take somewhere on the order of minutes instead of days.

Is this a big deal?

[Ed Avis]

The important part isn't the number of FLOPS (to get those you can just keep buying more PCs until you reach the desired number) but the performance in applications which are not 'embarassingly parallel'. In other words how good is the interconnect between machines? The article talks about a new network to replace Gigabit Ethernet.

Re:Is this a big deal?

[dsfd]

You are right, the network AND the algorithms are the the key in many cases, not just FLOPS.

A good example is solving large linear equation systems, with say 10^7 unkowns or more. This is a central problem in many fields of scientific computing. In our CFD simulations we need to solve 10^6 linear systems with 3x10^6 unkowns each to obtain the final answer.

It is difficult to use large number of processors to do it efficiently, specially if you use a conventional 100 Mbits/s network with high latency. Currently we are using 36 processors to do so, and the solution of each system takes about 4 seconds. Just multiply to have an idea of the total processing time !

But without Beowulf clusters (and GNU/Linux is a central part of them) this kind of problems would requiere conventional, very expensive supercomputers.

Interesting Approach on Network

[jki]

Selected clips:

The system has a few unique features that the lab says will facilitate applications performance, including a fast, custom-made network that taps into an enterprisewide file system.

"This network approach is nice because we can use a standard PCI slot on each processor node, which gives a 4.5-microsecond latency," he said, as opposed to 90-s latency for Gigabit Ethernet."

The boards are linked by a network assembled by Linux Networx into a clustered system that will have 960 server nodes.

The file system, called Lustre, uses a client/server model. Large, fast RAM-based memory systems support a metadata center, and data is represented across the enterprise in the form of object-storage targets. "Being able to share data across the enterprise is an exciting new capability

I think this is especially interesting, because it seems to glue together pieces from traditional clustering and distribted or metacomputing. Is there some site for this project with more details?

Connections through PCI bus?

[Dr. Spork]

Do I understand correctly that they just wired PCI slots from different motherboards together, instead of running the data around over ethernet (which probably would have been plugged into a PCI slot anyway)? If so, I mean, if there's nothing more to it than that, it seems like this will be a kickass way of clustering. But there must be something more to it than I realize, because if there wasn't, there wouldn't be so many ethernet-based beowulf systems.

So please explain this. I mean, I have two linux boxes in my room and each has a free PCI slot. What do I need to to to network them over directly over PCI?

Re:Does that mean...

[Dr. Spork]

You know, it wouldn't be stupid of Apple to try to build in some code for arbitrarily large clusters into Darwin. It would really be a prestige coup if a mac cluster became a top-500 computer.

Re:imagine the future

[jericho4.0]

I know exactly what I want to do with this kind of power. I want a massive FPS with a wicked frame rate. I want every object and material in that world to react with me. Newtownian physics down to grains of sand. Nothing short of movie quality realisim.I also want it to be massivly multiplayer. That will require some huge bandwidth and IO. Get on it Livermore!

yeah, well..

[radon28]

"There is no reason anyone would want a computer in their home." - Ken Olson, President, chairman and founder of Digital Equipment Corporation, 1977.

Processing power

[rovingeyes]

Actually, your statement made me wonder for a while. I remember that till not long ago, US wouldn't let other countries buy latest super computers becoz they feared it'd be used to do those nuclear explosion simlations. Now I'm not sure if it still is the case.

Anyways, what I'm trying to point out is that it is actually becoming very convinient to build a super computer with lots of PCs that just lie idle. I am not sure if Saddam has heard about cheap linux systems. But what if he could build a super computer cluster?

Boy this gets interesting and scarier at the same time.

Re:Processing power

[Duds]

Indeed. Beowolfing (is that a new word) conviently skips a lot of that. The law was actually changed when the US realised that the Playstation 2 was technically a super computer. Not that they had jurstiction on the PS2 itself but it brought home it was daft.

PCI Null-Modem

[Bios_Hakr]

Uuh, I mean null-card connection. I have never really looked at the PCI spec from an eletrical engineer standpoint, but there are probably power leads, data leads, timing leads, and ground leads on there.

The data leads should be easy...TX to RX. Although they may use a full-duplex lead where the data shares the bus based on clock pulses.

The power could be dropped, as both machines already have the proper power requirements. The ground leads could be tied together if you wanted, but dropping them shouldn't have too much impact on the final outcome.

The tricky part would be the clock pulses. In order to keep the data integrity, you need to have both bachines on the same clock. The easy way would be to take the crystal from one motherboard and wire it to the other. Same crystal, same clock pulse.

Then drivers would be needed to make the other computer look like an attached device. Shouldn't be too difficult. Just take a NIC driver and modify it...heavily.

I think an easier option would be to share data across the IDE bus. Make an IDE driver look like a NIC driver and send IP across IDE. In fact, I remember Linux Journal publishing an article about someone doing IP over SCSI about 2 years ago. Get some SCSI cards and make your own version of a CDDI network ring.

Re:PCI Null-Modem

[Bios_Hakr]

AGP is only useful for one-way communication. Great for shoving data to the monitor, but sucks for pulling data from an outside source. I'd say if you really wanted a challenge, use the memory bus for networking. The prob would be timeouts while waiting for the data to be pulled across the wire from the other machines' memory bank.

Or, go for broke and use the second processor slot in a dual mobo.

On the cheap end, you could use the USB and a null-modem cable there to link 2 boxes.

And they said Linux was a toy....

[LazLong]

What is most hilarious about all this is that three years ago the same people at the Lab who put together this cluster, Livermore Computing, insisted that Linux was a toy....That it had no future in scientific computing....That it was a hobbyist's OS.

I sure hope they love the taste of crow....

Re:imagine the future

[darkgreen]

Actually, it would be pretty cool to see a community that contributed to all the little details - kind of like a co-op world.

then you'd have this huge pool of resources from which to draw - no time to model and texture that bookcase you need for a room? just go and buy it.

grin, there could be auctions, just like real life, and since the 'structures' and objects you'd be buying have taken time and effort to create, and will have varying degrees of craftsmanship, you'd have a chance that it would actually turn into a market.

the value, of course, would be that everybody would have to be able to use the resources, resell and develop their own parts of the world.

we've seen something vaguely similar in the mod community since Doom (the original, kiddies), but it's not really there yet.

Why XEONs?

[Anonymous Coward]

Why would they use ridiculously expensive Intel processors instead of equally powerful but much much less expensive AMDs?

This is not a troll. I really want to know. Is it again because "no-one has ever been fired for buying Intel"?

(Open)MOSIX?

[Jeppe Salvesen]

Anyone have any experience using (Open)MOSIX? I have a partially CPU-bound application (automatic part is IO-bound, manual part is CPU-bound) in Perl, Apache and MySQL. Anyone got experience with this stuff?

For those who don't even know what MOSIX is, it is a kernel patch that essentially creates a virtual computer out of several boxes. They claim they will scale your application as long as you have multiple processes (they migrate them as needed) - without any coding on your part.

Since I'm looking for extra performance with limited resources, this looks like a potentially easy way out :)

Re:The real question is

[mpe]

Answer: Depends on his intent. If he is using it for finding extra terrestrial life, by all means he can go ahead, but if he is using it to test one of his biological weapons then he is obviously bad.

What if he finds some ETs who can help him out with some guy, known as GW Bush, who wants to invade his country.

"total cost of ownership" against off-the-shelf

[peter303]

2/3rds the cost of the three year computer lifetime is the electricity and cooling system. When TOC is counted a transmeta based cluster or the super-dese SGI cluster announced yesterday is cheaper.

On the other hand, it beats using a slide rule!

[SwedishChef]

One of the benefits of computers is the ability to solve a problem with iteration rather than trying to come up with a classic "equation" and solve it. When I first entered the job market I had a trusty Pickett N4ES slide rule (and an N600-ES pocket slide rule) and had to first explain a problem with an equation and then solve the equation (from the "inside out" which was why HP calculators with RPG were so popular with engineers when they first came out versus the TI models... but I digress).

With the introduction of the HP-35 calculator (the "electronic slide rule") we could solve problems by just crunching the numbers at our desks. With the availability of programmable calculators (HP-67/97 and HP-41 - both of which I still use... but then I still use the slide rules too) we could program them to iterate through problems.

Not as elegant, certainly. But lots more efficient. And I'm sure that most of us have lost some of our old abilities to "see" problems in math... and perhaps some students never really learn that. But the jobs still get done and the tools still keep making it easier. I'm thinking about a Beowulf cluster for our office, actually.