Slashdot Mirror
InfiniBand Drivers Released for Xserve G5 Clusters
A user writes, "A company called Small Tree just announced the release of InfiniBand drivers for the Mac, for more supercomputing speed. People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop. A lot. Voltaire also makes some sort of Apple InfiniBand products, though it's not clear whether they make the drivers or hardware."
The article is still subscriber-only, but Linux Weekly News has a good summary of some discussion on the LKML about InfiniBand. Greg K-H's original posting can be found here. Basically, he feels that it's impossible to implement the specification for InfiniBand in a free/open source product without violating the licensing agreement of the spec, because of patent infringement.
It's an interconnect technology which was stillborn.
installing Infiniband on a single unit G5....
With so few companies left doing anything Infiniband related, makes you wonder what the thinking is here.
http://www.oreillynet.com/pub/a/network/2002/02/04 /windows.html
This is a short into to infiband.
"InfiniBand breaks through the bandwidth and fanout limitations of the PCI bus by migrating from the traditional shared bus architecture into a switched fabric architecture."
"Each connection between nodes, switches, and routers is a point-to-point, serial connection. This basic difference brings about a number of benefits:
Because it is a serial connection, it only requires four as opposed to the wide parallel connection of the PCI bus.
The point-to-point nature of the connection provides the full capacity of the connection to the two endpoints because the link is dedicated to the two endpoints. This eliminates the contention for the bus as well as the resulting delays that emerge under heavy loading conditions in the shared bus architecture.
The InfiniBand channel is designed for connections between hosts and I/O devices within a Data Center. Due to the well defined, relatively short length of the connections, much higher bandwidth can be achieved than in cases where much longer lengths may be needed."
"The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. At the time that I am writing this, there are already 1x and 4x adapters available in the market. So, the InfiniBand will be able to achieve must higher data transfer rates than is physically possible with the shared bus architecture without the fan-out limitations of the later."
It's an interconnect technology which was stillborn.
My thoughts are with the parents.
This is cool. The Xserve is a great server. We got one at work and we used it as a mirror for a while before switchover. This thing never crashes. according to one of the articles these drivers will optimize the power of these beasts...
Has anyone here any experience with those "supercomputers"? It sounds kinda sci-fi! I'm curious, what language is the best for writing programs for such machines? Can one program them in Python or Perl, or only "Real Programmers(tm)" languages like Java and C++?
I've always understood that Myrinet is one of the better latency products available.
s p?content=9
And it has MacOSX Drivers:
http://www.myri.com/scs/macosx-gm2.html
Myrinet is used by 39% of the Top500 list published in November 2003
http://www.force10networks.com/applications/roe.a
The Virginia Tech cluster isn't on the top 500 list anymore:
from http://www.top500.org/lists/2004/06/trends.php
* The 'SuperMac' at Virginia Tech, which made a very impressive debut 6 month ago is off the list. At least temporarily. VT is replacing hardware and the new hardware was not in place for this TOP500 list.
People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop.
Note that V.T.'s cluster already uses InfiniBand, courtesy of Mellanox.
It's mentioned at V.T.'s pages.
...Halo and UT2004 were starting to slow down on my 1200 CPU cluster!
"Microsoft killed my company, I hold a personal grudge. I don't use Microsoft products and neither should you."-JWZ
This is what I've been waiting for, to be able to multi-thread applications with zero latency.
Doom 3 in a window? Halo in a window, UT2004 in a window?
No problem, I'm all goosebumps!!!
Not a gaming machine? My @$$!!!
I sure dont see it given that the 'official' word back in 2003 is that it was 3rd fastest. The Top 500 list (June of 2004) I can't even find it on that page. And last, if it did reach the 10.6TFlops it'd be #5 after the 11.6TFlop BlueGene/I
Your hair look like poop, Bob! - Wanker.
See what happens when your Windows computer drives you to madness?
The VT cluster dropped off the list because they didn't have the XServe upgrade running in time. It will be back in November's list. It was #3, but you have to remember the list is a moving target. Nobody stays in place for very long. It was shocking that the Earth Simulator stayed as long as it did!
No offense, but you don't know what you're talking about. IB can sustain tranfer rates of 700 MB/s; the best I've ever seen from GigE was almost an order of magnitude lower, not to mention the two orders of magnitude drop in latency with IB. That might not mean much to you, but I guarantee you it's a big deal for folks with big parallel scientific codes.
Oh, and your pricing's wrong too. In the quantities you'd need it for a decent size cluster, IB gear is about the same cost as its direct competitors (Myrinet and Quadrics).
Imagine a BeoWulf cluster with this installed... Ooh...!
Word on the street is that VT never managed to get the InfiniBand network working correctly. It was certainly waaaay more ambitious than any other IB installation around at the time.
Most of the parallel applications on our clusters are scientific simulation codes, written in Fortran, C, or C++ using MPI for inter-process communication.
"My life's work has been to prompt others... and be forgotten." --Cyrano de Bergerac
As The Apple Turns has a nice little article about Virginia Tech's newly Xserved system: http://appleturns.com/scene/?id=4980
Note that they got special Dual 2.3Ghz Xserves rather than the usual Dual 2Ghz ones.
They've already improved their previous third-place worthy score of 10.28 (admittedly no longer third-place worthy) to their current 12.05 teraflops, which could still put them in the top five. And they have a while to tweak their system until the final measurements will be taken for the Top500.
Back in 2002, people were pitching IB as a replacement for PCI. Today, nobody tries to do that -- IB and PCI are used for different purposes (clustering and I/O expansion, respectively).
I got to play with an Infiniband network at work, we use it in the testing lab to basically remove network latency from the testing equation. And I'll be honest, it's faster than anything I've ever seen.
Small Tree also makes cool multiport gigabit ethernet cards that support 802.1ad bonding. Really, the gigE cards are the more interesting thing for most of us who don't have a supercomputing cluster to run. The two-port version is less than $300. They work on Linux as well.
http://small-tree.com/mp_cards.htm
Gigabit has a latency of about 100 microseconds and realistic throughput of about 50MB/s. Infiniband has a latency of about 15 microseconds and a throughput of about 500MB/s.
I mostly sell small Apple workgroup clusters of 16 nodes, and these are almost always just a gigE backbone. There are certain classes of problems that can benefit from Infiniband at low node counts, but for the most common apps, like gene searching using BLAST, gigE is just fine.
- "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
Try clustering windows.. what you get is a pile of broken glass. Good for Steve, keeping his hardware limited is the key. Apple will never be able to compete for the home user but with quality in hardware and software, he will take over just about everything else. Penguins not included, we have to put up with MS (P)iece of (C)rap hardware.
October 14, 2004 Pg. 54
http://www.netlib.org/benchmark/performance.pdf
http://appleturns.com/scene/?id=4980
"Calm down, Beavis; take a closer look at the third and fourth entries and you'll realize that they're the same exact cluster, before and after its owners added another 64 processors to it. In much the same way, System X is also listed in the seventh, ninth, and eleventh slots, with scores taken at various points along its journey to life as a complete 1,100-Xserve system. Factor out the doubles and, barring an "October Surprise," System X ought to sit in fifth place, under an Alpha cluster, a new Itanium2 system, the once-mighty Earth Simulator, and the new top dog, that chunk of IBM's unfinished BlueGene. Woo-hoo, PowerPCs in two of the top five! No other chip can say that."
~hylas
NASA speculated that they would have their Altix cluster in place in time for the next top500 list. If they do manage that, it should easily take first until the final BlueGene/L system comes online next year. That system isn't on the As The Apple Turns list because they are using a list that is between official versions, but NASA did think they would have it in place for the next official list.
NASA said that they expect their Altix cluster to be online for the next top500 list. If it is, it should easily take first place until another version of BlueGene/L is released. That would put Itanium 2 in two of the top 5 positions, and knock PowerPC down to one.
Yes, Twelve Captures Fifth (10/14/04).
Anyone know if we'll ever see a Quad G5 in a Mac? Probably an Xserve, but even an IBM workstation using a quad G5 would be nice. Comments?
Welcome to the Apple section. If you're not interested in discussion of things related to Apple, please uncheck the appropriate box in your preferences, and we will all be happier.
I don't get this. Apple advocates constantly feel the need to point out in every other discussion about how much better Macintosh does everything. For example, pretty much like clockwork, Macintosh advocates will post to any discussion of Linux UIs some claim or other that the Macintosh is "the" system to use if you want to have a high quality UI on a UNIX-like system.
Slashdot is, after all, not a site where every group of people can retreat into their little corner, it's a site where people from different communities talk to each other.
OK, the "proprietary crap" discussed here is for: #1 XServes runing (wait for it....) Mac OS X. #2 Supercomputers This is not your linux box you're using for a NAT server, or a Beowolf running SETI
So what is your point? There are lots of 64-bit Linux clusters with high-speed interconnects. Much of the software used on those clusters was created in non-proprietary environments. It certainly makes sense to ask the question of what those proprietary systems bring to the table. I'm sorry if you are offended when people start asking hard questions.
I think the thinking from Apple on the current configurations is that a dual 2.5Ghz is going to be better than the fastest available Intel-based system with a single 3.xGhz P4. There's no need to make a 4-way box because the 2-way box already beats the best P4 because 2.5+2.5=5. Or something like that. For clusters, who cares how many cores in a single box? Just link a bunch of 2-way systems together.
But, once the G5 goes dual-core, I would expect to see a dual dual-core G5 machine out there somewhere. Does that count?
How does InfiniBand compare to Xsan? Are they different systems altogether, do they work in conjunctioin with one another, or are they competing standards?
The Big Mac may have been in third place, but things in supercomputing have been changing dramaticly.
There are now several Linux clusterss (in addition to the Linux cluster that was already faster) that are faster then the Apple cluster.
Also the new Blue Gene technology from IBM (using PowerPC embedded style proccessors modified for large image clustering) has everybody spanked. Including the now infamious Earth Simulator from Japan. The US is now back in number one on super computing (and number 3, 4, 5, 6, 7... etc etc).
Now all those IBM execs may breath a sigh of releif.
But for the rest of us, I think bonded RDMA Gig ethernet will be fine. Anybody seen RDMA ethernet cards in the wild yet?
That's a blatant lie, without MACs ethernet simply wouldn't work.
I bet your computer is using a MAC right now!
This is not stuff cobbling 2-4 PCs together. Its for people who want the ultimate Xserve solution. I have a 16-processor Xserve G5 with Gig-E and Myrinet. My next solution will be some 96 processors and all InfiniBand.
1) Macs only has 3% of the market...so who cares?
2) Macs are only for designers...so who cares?
3) Macs cost more than PCs...so who cares?
I'm surprised we haven't seen the usual, eight year old "facts" as to why this is a fruitless effort. Slowly but surely, Apple is making its way back into the limelight. After being the whipping boy for so long for a variety of reasons (no market share, higher outright cost, stability issues, etc), Apple is proving itself to be cheaper, more stable, and damn powerful. Enterprises, educational institutions, the government, and more are all starting to see the benefits and overcome the usual excuses.
Just for fun, go back ten years and compare Apple products with the products of today. Also compare the general opinion and stereotypes of Apple from then to now. Anyone see a major difference?
"He uses statistics as a drunken man uses lampposts...for support rather than illumination." - Andrew Lang
Parent could be modded funny.
Voltaire's okay, but you'll notice that Small Tree isn't reselling their gear, they're reselling Infinicon's gear. ICS sells the switches and shared IO gear you need to put it all together.
As I understand it, the advantages of IB over gig-E are lower latency and scalability.
Clear, Dark Skies
but IIRC, SmallTree wrote the software for them; all the IB vendors use Mellanox's chips for their HCAs, they differentiate themselves with the software the lay on top of them.
Clear, Dark Skies
First, it's NOT on the Top 500 list, goddammit!
Will you ever stop repeating that lie?
Second, it is under testing (not even in production).
(Third - not as relevant but still - why is a driver release still news? Topspin et al have been offering infinband drivers for Linux for a while; who wants
unidirectional bandwidth of 931 million bytes per second is equal to 887 MegaBytes per second. More than an entire CD-ROM per second.
But what about a quad dual core? Liquid cooling might not be enough? : D
I bet they saved a buttload of space in the process too.
We're only coming out with 144 node switches now. BigMac must have used a pile of 32 or 16 node switches.
Clear, Dark Skies