Slashdot Mirror
InfiniBand Drivers Released for Xserve G5 Clusters
A user writes, "A company called Small Tree just announced the release of InfiniBand drivers for the Mac, for more supercomputing speed. People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop. A lot. Voltaire also makes some sort of Apple InfiniBand products, though it's not clear whether they make the drivers or hardware."
http://www.oreillynet.com/pub/a/network/2002/02/04 /windows.html
This is a short into to infiband.
"InfiniBand breaks through the bandwidth and fanout limitations of the PCI bus by migrating from the traditional shared bus architecture into a switched fabric architecture."
"Each connection between nodes, switches, and routers is a point-to-point, serial connection. This basic difference brings about a number of benefits:
Because it is a serial connection, it only requires four as opposed to the wide parallel connection of the PCI bus.
The point-to-point nature of the connection provides the full capacity of the connection to the two endpoints because the link is dedicated to the two endpoints. This eliminates the contention for the bus as well as the resulting delays that emerge under heavy loading conditions in the shared bus architecture.
The InfiniBand channel is designed for connections between hosts and I/O devices within a Data Center. Due to the well defined, relatively short length of the connections, much higher bandwidth can be achieved than in cases where much longer lengths may be needed."
"The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. At the time that I am writing this, there are already 1x and 4x adapters available in the market. So, the InfiniBand will be able to achieve must higher data transfer rates than is physically possible with the shared bus architecture without the fan-out limitations of the later."
People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop.
Note that V.T.'s cluster already uses InfiniBand, courtesy of Mellanox.
It's mentioned at V.T.'s pages.
Infiniband is designed to be low latency to the extreme. Their driver software is going to be really sensitive to latency. If they can make their nic driver .5 usec faster than their competition it's a huge change in total latency. Thats only 2000 clock ticks, possibly 30-50 memory pulls. But for scientific computing it makes a huge difference in Computational Fluid Dynamics. The more cpu's you scale to, the more important the latency.
So their driver software is something that they are going to protect. It would be negligent to give it to the competition.
Storm
No offense, but you don't know what you're talking about. IB can sustain tranfer rates of 700 MB/s; the best I've ever seen from GigE was almost an order of magnitude lower, not to mention the two orders of magnitude drop in latency with IB. That might not mean much to you, but I guarantee you it's a big deal for folks with big parallel scientific codes.
Oh, and your pricing's wrong too. In the quantities you'd need it for a decent size cluster, IB gear is about the same cost as its direct competitors (Myrinet and Quadrics).
Hi,
e cts_docu.pdf
where did you get these numbers?
If you really want to compare the latency of actual interconnects you should use the official performance results achieved in real environments using the driver api:
(values from homepages)
1. SCI (dolphinIcs) : 1.4 us
2. Quadrics: 1.7 us
3. Infiniband 4.5 us
4. Myrinet 6.3 us
MPI latency and bandwidth highly depend on the mpi library. I suggest to compare the mpich results.
I rated these interconnects. But I'm sorry, I only have a german version.
http://stef.tvk.rwth-aachen.de/research/interconn