Slashdot Mirror

Try Dolphin SCI link .

It has latency in the microseconds, not milliseconds.

unless you're including the cost of installation and configuration in the sci stuff, i believe your numbers are pretty far off. i was looking at dolphin sci cards which could do roughly 8-10gb and remote dma. the prices i was discussing were on the order of $1-1.5k / device (dual ported cards, depending on volume) and about $5k for an 8 port switch (cables will add a pretty significant amount to these totals). (check their pricelist at http://www.dolphinics.com/pdf/pricing/Price%20List %2020050701.pdf). the dolphin guys were pretty pleasant to deal with. i would have loved to have used their solution, but power requirements made that essentially impossible (high bandwidth, low latency, and low power communications solutions don't really exist, which is a pity.)

Is SCSI P2P used in real world clusters though? How does it compare to SCI or gigabit ethernet? Price? Performance? Status of the project? No idea...

Is SCSI P2P used in real world clusters though? How does it compare to SCI or gigabit ethernet? Price? Performance? Status of the project? No idea...

NASA and the Department of Energy use Linux, and have for years. They use Linux Beowulf clusters for computational fluid dynamics. Most of them have an SCI backbone for inter-node communications. Although, I guess nowadays for CFD you might as well write multithreaded code that's compatible with OpenMosix and, instead of spending a ton of $ on a cluster, run the simulations on your employees' workstations.

Oh, and you might as well not bother imagining a Beowulf cluster of those babies... it's been done.

Merging n ordered lists into a single list with the same ordering is pretty trivial: consider the first item of each list, and take the first of those. Rinse, repeat. Extremely fast and simple; where the lists cover discrete ranges (e.g. node 1 has the first X items in this ordering, node 2 has the 2nd, etc) it is as simple as appending the lists!

Latency is the name of the game, with Ethernet in 10s of miliseconds and memory in the 10s of nanoseconds, there will always be a huge penalty for sincronization through network. There's alway ways to add throughput (http://geocities.com/feromus/db-scalability.html) but latency will always have to increase...

You can also do much better than Ethernet for latency, although cost explodes as a result; look at Dolphin for example. (Microsecond latency on data transfers between nodes.) Not quite RAM speeds, but much much faster (and with lower CPU usage) than Ethernet.

Normal latency for SCI (Dolphin delivers SCI) is about 1.5 microseconds; for Myrinet, about 8 microseconds. We are talking about latencies that are three order of magnitude (a thousand times) smaller.

FireWire is not enabling for the same range of applications as a full-blown scientific cluster interconnect is. However, few applications need s a fullblown interconnect - most clustering apps I've built could have been tied together by the bandwidth of smoke signals (fortunately,they seldom have been - the one case I can remember was when a customer insisted on dictating hardware...)

Eivind.

When I first read the post, I got pretty excited. Dreams of cheap clustering for scientific applications danced in my head. No more need for Myrinet, no Dolphin, just Firewire and Beowulf!

Then, I read some performance metrics on Firewire. High bandwidth. High latency. Doh! The fairies stopped dancing for joy.

The problem is that in scientific computing, the time it takes for one node to say I need that data to another node, and actually get that data determines the performance of many more apps than does the speed of the CPUs.

So, until a cheap, low latency solution for communications comes by, real clusters will be communicating over Dolphin, Myrinet, or some other propietary technology.

Tony

When I first read the post, I got pretty excited. Dreams of cheap clustering for scientific applications danced in my head. No more need for Myrinet, no Dolphin, just Firewire and Beowulf!

Then, I read some performance metrics on Firewire. High bandwidth. High latency. Doh! The fairies stopped dancing for joy.

The problem is that in scientific computing, the time it takes for one node to say I need that data to another node, and actually get that data determines the performance of many more apps than does the speed of the CPUs.

So, until a cheap, low latency solution for communications comes by, real clusters will be communicating over Dolphin, Myrinet, or some other propietary technology.

Tony

Bus 0, device 12, function 0: PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 3). Master Capable. Latency=64. Min Gnt=4.

But you can't use this to connect a rack of computers. For one thing the max cable length for connecting two busses would be just a few inches. For putting PCI cards in 1.75" high 1U rackmount cases, there are PCI risers with a short ribbon cable that connects to the PCI slot. Even these short cables often cause timing problems. For instance, with the riser, cards may only work in the first one or two slots that will otherwise work in all the slots.

But even if you could cable all the computers together on one giant PCI bus, it would still be a bad idea. A good 24 port gigabit ethernet switch (~$2000) has a 480MB/sec switching fabric, to support full speed full duplex on each port. 32 bit 33Mhz PCI is only about 132 MB/sec, not nearly as fast. You'd need a 64 bit 66 Mhz PCI bus to keep up. And there are more expensive gbit switches with more ports that have 100 Gbit/sec fabric. And this is just gbit ethernet, the slowest and cheapest of the high speed interconnects used in modern Beowulf clusters.

There are faster ways to connect computers than gigabit ethernet. The EE times article is very untechnical, but this one has some more information. LLNL has used a very fast and very expensive interface called quadrics. This is probably the fastest way to connect computers in a Beowulf. People like Cray/SGI and IBM have faster things still, but they cost real big bucks. Other ways to connect a Beowulf are the above mentioned gigabit ethernet (~$100-$250 a node for up to 24 nodes), myrinet (~$1400-$2000 /node up to 128 nodes), and SCIhardware and software (~$1400-$2100 /node). Myrinet uses a switch like gigabet ethernet and the largest switch they have is 128 ports. SCI is switchless, each card has multiple cables (1-3), and is connected in into a ring, 2D or 3D torus.

I must admit that I'm looking toward the future.

No, you're looking toward the past. In the future, multi-CPU machines will become more common, not less, and learning to use them efficiently will also become more important. Within the box, multithreading will perform better than alternatives, even if there's message passing going on between boxes.

These processors will use asynchronous message-passing logic at the very core, and farther out the system will be entirely based on messages. HyperTransport is the writing on the wall.

Yes, I'm somewhat familiar with transitions between message passing and shared memory. Remember that fast interconnect I mentioned, from five years ago? It was at Dolphin. On the wire, it was message passing. Above that, it presented a shared-memory interface. Above that, I personally implemented DLPI/NDIS message-passing drivers because that's what some customers wanted.

The fact is that whatever's happening down below, at the programming-model level it's still more efficient to have multiple tasks coordinate by running in the same address space than by having them spend all their time packing and unpacking messages. The lower-level message-passing works precisely because it's very task-specific and carefully optimized to do particular things, but that all falls down when the messages have to be manipulated by the very same processors you were hoping to use for your real work.

The future of high-performance computing is message passing

...between nodes that are internally multi-processor.

The request-producer doesn't have to block waiting for the first response: it can keep queueing requests.

Yes, yes, using parallelism to mask latency. Yawn. Irrelevant.

Every screen update requires at least one context switch. Does it suck? Not at all.

If context switches aren't all that bad, why were you bitching about context switching in multithreaded applications? Hm. The fact is, a context switch is less expensive that a context switch plus packing/unpacking plus address manipulation plus (often) a data copy. Your proposal is to use multiple processes instead of threads, even within one box. When are you going to start explaining how that will perform better, or even as well? When it won't "suck", to use your own charming phrase.

What about Monte Carlo simulations? Losing the occassional random process is irrelevant. What about artificial intelligences running on really big machines?

Please try to pay attention. I already referred to regularly decomposable applications with high compute-to-communicate ratios, and that's exactly what you're talking about. Yes, what you say is true for some applications, but does it work in general? No. As I said, I've worked in high availability. I've seen database app after database app, all based on message passing between nodes, lock up because one node froze but didn't crash. Everyone's familiar with applications hanging when the file server goes out, and that's not shared memory either. Message passing doesn't make causal dependencies go away.

If the system is built on a foundation of multithreading, the only failure mode is a total crash.

Simply untrue. I've seen (and written) plenty of multithreaded applications that could survive an abnormal thread exit better than most IPC-based apps could survive an abnormal process exit.

Second: SCI is orthogonal to the other two technologies - it is a special hardware network technology (Scalable Coherent Interface), originally made to support distributed shared memory. You may be thinking of the software Dolphin Interconnect Solutions provide with their SCI solutions, but as far as I know, that doesn't directly enter into the same space, either. Their web pages does certainly not indicate that it does, and my discussions with (one of?) their Linux developer(s) implied that it contained somewhat more (lock managers etc), but not in the same space. A technology that compete with SCI, though proprietary, is Myrinet. This has a longer history than SCI, and has been less plagued with problems than SCI (though SCI is supposedly quite stable now).

Third: There are a bunch of other technologies (some cross-platform, some single-platform) that compete in making it easy to build clusters. MOSIX and Beowulf are just two of them. If you give more details of what you want to achieve, I'll dig out references from my collection (made to support the development of FreeBSD-specific clustering improvements, so some types of references may be lacking, but I'll probably be able to come with at least some points to start for any wanted cluster workload.)

Eivind.