Slashdot Mirror
InfiniBand Drivers Released for Xserve G5 Clusters
A user writes, "A company called Small Tree just announced the release of InfiniBand drivers for the Mac, for more supercomputing speed. People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop. A lot. Voltaire also makes some sort of Apple InfiniBand products, though it's not clear whether they make the drivers or hardware."
With so few companies left doing anything Infiniband related, makes you wonder what the thinking is here.
Of course that misses the point about getting low latencies to improve Beowolf cluster performance by a factor over ethernet.
[A 747] can do everything [the Joint Strike Fighter] can, although without the same [supersonic speed, air-to-air combat capability] of [the JSF]. [The JSF] just never caught on when it could, it was ahead of its time, but now [747s are] cheap, and soon [the Airbus A300] will strip it dry.
Smart. "Without the low latency of infiniband"? Idiot, what do you think it's for? We're not talking eDonkeying Halo 2 here... ultra-low latency is THE POINT.
Oh wait, you're just a stupid fucking troll. Why don't you go die?
properiaty drivers for properiaty os that is run on properiaty hardware(on os that's only legal to run on that hw makers hardware too).
so if you're there you're already pretty deep in "properiaty crap".
world was created 5 seconds before this post as it is.
Back in 2002, people were pitching IB as a replacement for PCI. Today, nobody tries to do that -- IB and PCI are used for different purposes (clustering and I/O expansion, respectively).
OK, the "proprietary crap" discussed here is for:
/.'s general section (where I would be more than happy to consider it not trolling), and enjoy a livelier discussion there, with a wider, and more appropriate, audience.
#1 XServes runing (wait for it....) Mac OS X.
#2 Supercomputers
This is not your linux box you're using for a NAT server, or a Beowolf running SETI, so if you're building a super computer or just like drolling over them and thinking of using and expensive interconnect like InfiniBand, you're not looking to compare it to Beowolf over gigabit, and possibly not likely to care about if the drivers are binary only or not.
This article is in no way related to any LKML posting other than it's the same company. This is about OSX Infiniband drivers. RTFA sometime, and you might realize such things.
Welcome to the Apple section. If you're not interested in discussion of things related to Apple, please uncheck the appropriate box in your preferences, and we will all be happier. If you like to run Linux on Apple Hardware, please examine the OS discussed before trolling.
If you want to troll about Infinbands policies effecting Linux, then wait until the LWN article is public ("Alternatively, this item will become freely available on October 21, 2004"), and submit it to
I am, and always will be, an idiot. Karma: Coma (mostly effected by
Gig-E can't do half of what IB can in the segment that IB targets, and 10GigE can barely do half of what IB can do. First of all: GigE/10GigE is only practically useful together with TCP/IP. Congratulations, you just killed latency, there's no way you can come down to the 12-40 microseconds latency that IB achieves with real workloads. Second, the IB protocol handles traffic priorization directly in the low-level protocols, same thing with the self-healing aspects, routing data around failures. Third, and this is the most neglected part among those who haven't worked with it: RDMA. Direct in hardware and low-level protocols. It lets your process announce memory space out to the node that is sending it data, so that node can write directly into the memory for example. Allows you to build a system with fake shared memory, and still retain 12-40 microsecond latencies, unlike slow and fugly hacks run on top of TCP/IP that try to achieve the same thing with latencies of up to 10ms.
One example of fake shared memory that I've seen is a cluster with an unusual design: Two IBM P5 570's with a total of 32 cores and 128GB RAM, linked together via IBM's NUMA interconnect. They also had a total of 36 Infiniband HCA's. The slave nodes are Xserve G5's with 2GB RAM and Infiniband, and a Xilinx FPGA-card that has its own memory banks. What the slave nodes do is essentially that they work straight against the RAM on the two 570's, with the local RAM only as a form of cache. The project runs as a multi-threaded app on the 570's, and are slaved out to the nodes. The project was originally meant to be used with some p690's.
It would be nice to see IB actually come together, but it's an uphill battle. The Spec is a massive tangly mess. Vendor infighting and politics has nearly killed it dead two or three times now. The last thing it needs is for the specs to be priced like they're printed on gold leaf and patent battles to boot.
Meanwhile, I've seen lightweight reliable non-IP protocols over bog standard GigE hardware get 10 microsecond latency and as a result, 90MB/s ACTUAL transfer.
Given that, 10GigE could give IB a real run, especially if it's coupled with an onboard DMA engine (there's no reason it can't be). Consider that with the right protocol, GigE can get a little over 90% of theoretical, if 10GigE manages that, it'll beat IB.
There's a lot of good things about IB, but if the IBTA really wants it to catch on, they'd better start acting like they WANT people to buy it. Right now, IB's best chance looks to be the OpenIB project. However, if the IBTA decides to try locking it up tighter and tighter, OpenIB won't save them, the rest of the industry will do clever things with 10GigE and save itself a bunch of patent headaches.
SUNET is having problems with 10GigE when they reach around 50-60%
If they're using IP, I'm not at all surprised. IP is designed to provide reasonable performance in a hetrogenous unreliable network. In a cluster environment, you would want the protocol to provide excellent performance in a homogenous and error-free environment and simple correctness (likely at a terrible penelty) in an unreliable environment.
The problem is that IP has to deal with fragmentation, out of order delivery and moderatly frequent lost packets. It has to support TTL so that routing loops don't tear the whole net down. It's designed to sorta work even if the underlying fabric is a mess. None of that makes it a bad protocol, just the wrong choice for a reliable low latency HPC fabric that's under a single administrative authority.
The fact that TCP/IP works as well as it does on a fabric that bis nearly the opposite of what it is designed for speaks volumes. The fact that it is used on so many clusters demonstrates that people are willing to pay a substantial performance penelty in exchange for an open, well understood, and easy to program network. That's a big reason that IP over IB exists at all.
I see about 50-60% for GigE using TCP/IP as well. I only see >90% with alternate protocols. Most I have talked to report that IP over IB where IB appears to the IP stack as an ethernet like device barely outperforms GigE. That's not surprising either. I suspect the IP stack is the problem there as well.
This is the Apple section of slashdot. These sections are present for a reason. Apple policies wern't being discussed at all, but Infiniband policies. Given that drivers are now released for Infiniband for OSX, the question of what this brings to Apple Clusters is something relevant to be discussed here. The reason I would brand you a troll is that you're speaking negatatively about something that is not relevant to this section of /.: How linux clusters are effected by IP issues decided by Infinband. You have yet to frame this in terms of contrasting how their policies effect Linux and how Apple drivers are now released. If you had started that way, then perhaps I would have just watched the discussion.
/. at the appropriate time (when the content on LWN becomes publically available). I all but wrote the submission for you! That hardly sounds like I'm trying to keep you from discussing this at all. I've suggested broadening your audience even.
To say I don't want to hear what you have to say has been already been proven wrong, as I have suggested previously that you make such a discussion in the appropriate section of
Also to say I've made up my mind about Infiniband or Linux v. Apple clusters couldn't be further from the truth. I haven't said anything about what I prefer, what is best, or anything to indicate my opinion. In fact, I have no informed opinion on the subject at all, much less have I expressed one. As a matter of fact, what is my direct put down of Linux-based clusters? Any comment I made was that this is that clusters likely to look at Infiband are not small scale, hobbyist clusters, but more likely larger clusters with larger budgets, so if the specs are free or not will be less of a factor.
I am, and always will be, an idiot. Karma: Coma (mostly effected by