10-TFlop Computer Built from Standard PC Parts

OrangeTide writes "Using PCI host adapters and Xeon processors, engineers at Lawrence Livermore National Labs have achieved 10-TFlops relatively cheaply. More information can be obtained from this article at EETimes." Lately, Linux seems to be the operating system of choice for new supercomputers, and this one's no different. It's cool to see big iron made cheaply.

Comments

Parallel computing

[vlad_petric]

The difficulty is not to conglomerate processing power ... you can do that relatively easily with Benjamins ... the real difficulty is in either parallelizing your computations, or making a single processor work faster.

So the Teraflops they're mentioning are just a theoretical upper bound, don't get too aroused when you see it.

The Raven.

Interesting Approach on Network

[jki]

Selected clips:

The system has a few unique features that the lab says will facilitate applications performance, including a fast, custom-made network that taps into an enterprisewide file system.

"This network approach is nice because we can use a standard PCI slot on each processor node, which gives a 4.5-microsecond latency," he said, as opposed to 90-s latency for Gigabit Ethernet."

The boards are linked by a network assembled by Linux Networx into a clustered system that will have 960 server nodes.

The file system, called Lustre, uses a client/server model. Large, fast RAM-based memory systems support a metadata center, and data is represented across the enterprise in the form of object-storage targets. "Being able to share data across the enterprise is an exciting new capability

I think this is especially interesting, because it seems to glue together pieces from traditional clustering and distribted or metacomputing. Is there some site for this project with more details?

Connections through PCI bus?

[Dr. Spork]

Do I understand correctly that they just wired PCI slots from different motherboards together, instead of running the data around over ethernet (which probably would have been plugged into a PCI slot anyway)? If so, I mean, if there's nothing more to it than that, it seems like this will be a kickass way of clustering. But there must be something more to it than I realize, because if there wasn't, there wouldn't be so many ethernet-based beowulf systems.

So please explain this. I mean, I have two linux boxes in my room and each has a free PCI slot. What do I need to to to network them over directly over PCI?

Processing power

[rovingeyes]

Actually, your statement made me wonder for a while. I remember that till not long ago, US wouldn't let other countries buy latest super computers becoz they feared it'd be used to do those nuclear explosion simlations. Now I'm not sure if it still is the case.

Anyways, what I'm trying to point out is that it is actually becoming very convinient to build a super computer with lots of PCs that just lie idle. I am not sure if Saddam has heard about cheap linux systems. But what if he could build a super computer cluster?

Boy this gets interesting and scarier at the same time.

PCI Null-Modem

[Bios_Hakr]

Uuh, I mean null-card connection. I have never really looked at the PCI spec from an eletrical engineer standpoint, but there are probably power leads, data leads, timing leads, and ground leads on there.

The data leads should be easy...TX to RX. Although they may use a full-duplex lead where the data shares the bus based on clock pulses.

The power could be dropped, as both machines already have the proper power requirements. The ground leads could be tied together if you wanted, but dropping them shouldn't have too much impact on the final outcome.

The tricky part would be the clock pulses. In order to keep the data integrity, you need to have both bachines on the same clock. The easy way would be to take the crystal from one motherboard and wire it to the other. Same crystal, same clock pulse.

Then drivers would be needed to make the other computer look like an attached device. Shouldn't be too difficult. Just take a NIC driver and modify it...heavily.

I think an easier option would be to share data across the IDE bus. Make an IDE driver look like a NIC driver and send IP across IDE. In fact, I remember Linux Journal publishing an article about someone doing IP over SCSI about 2 years ago. Get some SCSI cards and make your own version of a CDDI network ring.