InfiniBand Drivers Released for Xserve G5 Clusters

A user writes, "A company called Small Tree just announced the release of InfiniBand drivers for the Mac, for more supercomputing speed. People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop. A lot. Voltaire also makes some sort of Apple InfiniBand products, though it's not clear whether they make the drivers or hardware."

Comments

Proprietary Crap

[ceswiedler]

The article is still subscriber-only, but Linux Weekly News has a good summary of some discussion on the LKML about InfiniBand. Greg K-H's original posting can be found here. Basically, he feels that it's impossible to implement the specification for InfiniBand in a free/open source product without violating the licensing agreement of the spec, because of patent infringement.

Re:Proprietary Crap

[tempest69]

Infiniband is designed to be low latency to the extreme. Their driver software is going to be really sensitive to latency. If they can make their nic driver .5 usec faster than their competition it's a huge change in total latency. Thats only 2000 clock ticks, possibly 30-50 memory pulls. But for scientific computing it makes a huge difference in Computational Fluid Dynamics. The more cpu's you scale to, the more important the latency. So their driver software is something that they are going to protect. It would be negligent to give it to the competition. Storm

Re:Proprietary Crap

[Durindana]

gig-e can do everything infiniband can, WITH tcp, although without the same low latency of infiniband. infiniband just never caught on when it could, it was ahead of its time, but now gigabit ethernet is cheap, and soon ten gigabit ethernet will strip it dry.

[A 747] can do everything [the Joint Strike Fighter] can, although without the same [supersonic speed, air-to-air combat capability] of [the JSF]. [The JSF] just never caught on when it could, it was ahead of its time, but now [747s are] cheap, and soon [the Airbus A300] will strip it dry.

Smart. "Without the low latency of infiniband"? Idiot, what do you think it's for? We're not talking eDonkeying Halo 2 here... ultra-low latency is THE POINT.

Gee, hard decision, although with that price I can see why mac user's would go for it.

Oh wait, you're just a stupid fucking troll. Why don't you go die?

Re:Proprietary Crap

[LoRdTAW]

Pay $5k for infiniband hardware or $40 for a gig ethernet card?

Where did you get this dollar amount and what exactly is it for, the HCA a switch, cable or all of them? HCA's are about 800-1000 dollars. Switches from Mellanox start at about $8,000 for a 480Gbit backplane 24 port switch. And up to $66,000 for a full 96 port modular switch. Cables though I will admit are costly at 100 bucks for a 4X 2m cable.

Also your statement that it is useless is complete FUD. It certainly will never gain the widespread use of Ethernet, but it serves a niche market for a standard high speed interconnect.

You obviously have no clue as to what Infiniband is or is capable of. First off 4x Infiniband is 10 times faster then Ethernet at 10 gigabits/sec. And second it has lower latency and CPU utilization then common commodity GbE hardware like the $40 GbE adapter you speak of. GigE and TCP are quite inefficient when compared to Infiniband, even if you bought TOE cards like the 7711 from Adaptec you're still paying as much as you would for an Infiniband HCA and have 10 times less bandwidth. Ok so the switches are expensive but still the throughput is incredible with a 24port switch having 480Gbits of bandwidth at about 8000 bucks! More expensive GigE switches commonly used in clusters are almost as costly as the Infiniband switches.

I even read that a 1024 node cluster using GbE was just as fast as a 256 node cluster using IB, mainly because each node in the GbE cluster was mostly dealing with Ethernet and TCP/IP then the actual application.

So before you start talking out of your ass do some research like I did. I might not be 100% correct but I think I am close.

Re:Proprietary Crap

[Kalak]

OK, the "proprietary crap" discussed here is for:
#1 XServes runing (wait for it....) Mac OS X.
#2 Supercomputers

This is not your linux box you're using for a NAT server, or a Beowolf running SETI, so if you're building a super computer or just like drolling over them and thinking of using and expensive interconnect like InfiniBand, you're not looking to compare it to Beowolf over gigabit, and possibly not likely to care about if the drivers are binary only or not.

This article is in no way related to any LKML posting other than it's the same company. This is about OSX Infiniband drivers. RTFA sometime, and you might realize such things.

Welcome to the Apple section. If you're not interested in discussion of things related to Apple, please uncheck the appropriate box in your preferences, and we will all be happier. If you like to run Linux on Apple Hardware, please examine the OS discussed before trolling.

If you want to troll about Infinbands policies effecting Linux, then wait until the LWN article is public ("Alternatively, this item will become freely available on October 21, 2004"), and submit it to /.'s general section (where I would be more than happy to consider it not trolling), and enjoy a livelier discussion there, with a wider, and more appropriate, audience.

Re:Proprietary Crap

[Tiosman]

You obviously have no clue as to what Infiniband is or is capable of. First off 4x Infiniband is 10 times faster then Ethernet at 10 gigabits/sec.

4X infiniband is 10 Gb/s signal rate but actually 8 Gb/s data rate (8b/10b encoding). This is one of many facts that the IB marketing dept. keep forgetting (I keep telling them, but they won't listen for some reasons).

GigE and TCP are quite inefficient when compared to Infiniband

TCP over Infiniband is as inefficient, it has nothing to do with GigE. People use IP over GigE because it's convenient, but you can use GigE without IP if you talk directly to the hardware. Some have tried and are still trying http://www.disi.unige.it/project/gamma/, but the main problem is the lack of hardware documentation from GigE vendors and the short life span of GigE chips.

I even read that a 1024 node cluster using GbE was just as fast as a 256 node cluster using IB

It's interesting to note that there are not many 256 nodes clusters in production with IB at the moment, even less with 1024 nodes. Second, just as fast doing what ? A pointless benchmark specially tuned for Infiniband as the IB supporters are used to publish or real-world applications ? Yes, high speed interconnects make a difference but GigE is just fine for a lot of the HPC applications I have seen so far.

So before you start talking out of your ass do some research like I did.

Don't believe everything you read, and don't drink the cool-aid that fast. Look at the Top500 just to see what machines are out there, not for the ranking (Linkpack is useless). You will see that there are quite a lot of GigE clusters and not that many IB ones. It's a matter of economics: if IB makes sense, people will buy it. These days, they buy much more GigE (or other) than IB.

Re:Proprietary Crap

[Shinobi]

Gig-E can't do half of what IB can in the segment that IB targets, and 10GigE can barely do half of what IB can do. First of all: GigE/10GigE is only practically useful together with TCP/IP. Congratulations, you just killed latency, there's no way you can come down to the 12-40 microseconds latency that IB achieves with real workloads. Second, the IB protocol handles traffic priorization directly in the low-level protocols, same thing with the self-healing aspects, routing data around failures. Third, and this is the most neglected part among those who haven't worked with it: RDMA. Direct in hardware and low-level protocols. It lets your process announce memory space out to the node that is sending it data, so that node can write directly into the memory for example. Allows you to build a system with fake shared memory, and still retain 12-40 microsecond latencies, unlike slow and fugly hacks run on top of TCP/IP that try to achieve the same thing with latencies of up to 10ms.

One example of fake shared memory that I've seen is a cluster with an unusual design: Two IBM P5 570's with a total of 32 cores and 128GB RAM, linked together via IBM's NUMA interconnect. They also had a total of 36 Infiniband HCA's. The slave nodes are Xserve G5's with 2GB RAM and Infiniband, and a Xilinx FPGA-card that has its own memory banks. What the slave nodes do is essentially that they work straight against the RAM on the two 570's, with the local RAM only as a form of cache. The project runs as a multi-threaded app on the 570's, and are slaved out to the nodes. The project was originally meant to be used with some p690's.

Re:Proprietary Crap

[Barto]

You're missing the point: if the spec was made open (NOT the driver software), open source drivers could be developed increasing the demand for Infiniband products, reduce costs to users and Infiniband and improve compatibility.

Re:Proprietary Crap

[Johannes]

What's wrong with OpenIB?

Imagine

[commodoresloat]

installing Infiniband on a single unit G5....

Shocking

[CMiYC]

With so few companies left doing anything Infiniband related, makes you wonder what the thinking is here.

Infiniban into

[hardlined]

http://www.oreillynet.com/pub/a/network/2002/02/04 /windows.html

This is a short into to infiband.

"InfiniBand breaks through the bandwidth and fanout limitations of the PCI bus by migrating from the traditional shared bus architecture into a switched fabric architecture."

"Each connection between nodes, switches, and routers is a point-to-point, serial connection. This basic difference brings about a number of benefits:

Because it is a serial connection, it only requires four as opposed to the wide parallel connection of the PCI bus.

The point-to-point nature of the connection provides the full capacity of the connection to the two endpoints because the link is dedicated to the two endpoints. This eliminates the contention for the bus as well as the resulting delays that emerge under heavy loading conditions in the shared bus architecture.

The InfiniBand channel is designed for connections between hosts and I/O devices within a Data Center. Due to the well defined, relatively short length of the connections, much higher bandwidth can be achieved than in cases where much longer lengths may be needed."

"The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. At the time that I am writing this, there are already 1x and 4x adapters available in the market. So, the InfiniBand will be able to achieve must higher data transfer rates than is physically possible with the shared bus architecture without the fan-out limitations of the later."

Comparison with Myrinet

[Sosarian]

I've always understood that Myrinet is one of the better latency products available.

And it has MacOSX Drivers:
http://www.myri.com/scs/macosx-gm2.html

Myrinet is used by 39% of the Top500 list published in November 2003
http://www.force10networks.com/applications/roe.as p?content=9

Re:Comparison with Myrinet

[Anonymous Coward]

Here's how bandwidth and latency break down for interconnect technologies:

1. Quadrics (EXPENSIVE! and closed standard) sub 4 microsec
2. InfiniBand (Realtively inexpensive, open standard) 4.5 microsec
3. Myrinet (Roughly the same price as IB, but closed standard) sub 10 microsec
4. GigE (cheap) 20+ microsec

All latency numbers are hardware not software latencies. Depending on how good your MPI stack is you can often triple those numbers.

There are so few companies making IB because there is only one chipset manufacturer right now. Mellanox. All the companies making IB products are startups and it will be a while before things get better.

Re:Comparison with Myrinet

[stef716]

Hi,

where did you get these numbers?
If you really want to compare the latency of actual interconnects you should use the official performance results achieved in real environments using the driver api:
(values from homepages)

1. SCI (dolphinIcs) : 1.4 us
2. Quadrics: 1.7 us
3. Infiniband 4.5 us
4. Myrinet 6.3 us

MPI latency and bandwidth highly depend on the mpi library. I suggest to compare the mpich results.
I rated these interconnects. But I'm sorry, I only have a german version.

http://stef.tvk.rwth-aachen.de/research/interconne cts_docu.pdf

Re:Comparison with Myrinet

[Junta]

To say IB network management tools are better is a great understatement. Part of myrinet is that the network topology is forced to be simple and the switches as dumb as possible (distribute the task of routing and mapping the networks to the nodes). IB switches offer a tad more functionality and offload mapping work to the switch, but stays a source-routed network (which is the chief way these technologies acheive low latency while ethernet is switch routed and therefore scales poorly as the switches have more and more work to do.

Of course, until IB over fiber media comes around, myrinet cabling is a hell of a lot easier to deal with, longer lengths, more bendable, and tighter bend radius.

Not 3rd fastest, in fact not on list at all.

[Anonymous Coward]

The Virginia Tech cluster isn't on the top 500 list anymore:

from http://www.top500.org/lists/2004/06/trends.php

* The 'SuperMac' at Virginia Tech, which made a very impressive debut 6 month ago is off the list. At least temporarily. VT is replacing hardware and the new hardware was not in place for this TOP500 list.

Re:Not 3rd fastest, in fact not on list at all.

[derdesh]

Thursdays As the Apple Turns has an episode speculating that Virginia Tech's cluster should come in at number 5 in the new list.

(The link should be good until sometime this weekend, then it will be avaiable in re-runs)

BigMac already has I.B.

[mfago]

People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop.

Note that V.T.'s cluster already uses InfiniBand, courtesy of Mellanox.

It's mentioned at V.T.'s pages.

Well thank goodness....

[Killer Eye]

...Halo and UT2004 were starting to slow down on my 1200 CPU cluster!

Re:Third fastest what?

[ztirffritz]

The BigMac at VA Tech missed the list this year because they were busy switching over to DP G5 Xserves. Last I heard, they had completed the project and were busy re-benchmarking the beast. I I also heard that it was poised to move to number 2 possibly on the list after it was retested officially. The Army's version of the BigMac will probably take that title away though. That then 2 of the top 3 machines will be G5 based. Too Cool!

Proprietary, but definitely not crap

[Anonymous Coward]

gig-e can do everything infiniband can, WITH tcp, although without the same low latency of infiniband.

No offense, but you don't know what you're talking about. IB can sustain tranfer rates of 700 MB/s; the best I've ever seen from GigE was almost an order of magnitude lower, not to mention the two orders of magnitude drop in latency with IB. That might not mean much to you, but I guarantee you it's a big deal for folks with big parallel scientific codes.

Oh, and your pricing's wrong too. In the quantities you'd need it for a decent size cluster, IB gear is about the same cost as its direct competitors (Myrinet and Quadrics).

Re:Proprietary, but definitely not crap

[sjames]

It would be nice to see IB actually come together, but it's an uphill battle. The Spec is a massive tangly mess. Vendor infighting and politics has nearly killed it dead two or three times now. The last thing it needs is for the specs to be priced like they're printed on gold leaf and patent battles to boot.

Meanwhile, I've seen lightweight reliable non-IP protocols over bog standard GigE hardware get 10 microsecond latency and as a result, 90MB/s ACTUAL transfer.

Given that, 10GigE could give IB a real run, especially if it's coupled with an onboard DMA engine (there's no reason it can't be). Consider that with the right protocol, GigE can get a little over 90% of theoretical, if 10GigE manages that, it'll beat IB.

There's a lot of good things about IB, but if the IBTA really wants it to catch on, they'd better start acting like they WANT people to buy it. Right now, IB's best chance looks to be the OpenIB project. However, if the IBTA decides to try locking it up tighter and tighter, OpenIB won't save them, the rest of the industry will do clever things with 10GigE and save itself a bunch of patent headaches.

Re:Third fastest what?

[antifoidulus]

Doubtful. IBM's BlueGene is the king right now(well for the time being), but I don't see Big Mac(either version) beating the earth sim. Still, 2 out of the top 4 isn't bad.

That article is obsolete

[Wesley Felter]

Back in 2002, people were pitching IB as a replacement for PCI. Today, nobody tries to do that -- IB and PCI are used for different purposes (clustering and I/O expansion, respectively).

Re:That article is obsolete

[sirsnork]

Actually IB is VERY closely related to PCI Express. At one point they were the same thing and that was called 3GIO by Intel

Permanent link.

[Xenex]

Yes, Twelve Captures Fifth (10/14/04).