Slashdot Mirror
InfiniBand Drivers Released for Xserve G5 Clusters
A user writes, "A company called Small Tree just announced the release of InfiniBand drivers for the Mac, for more supercomputing speed. People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop. A lot. Voltaire also makes some sort of Apple InfiniBand products, though it's not clear whether they make the drivers or hardware."
The article is still subscriber-only, but Linux Weekly News has a good summary of some discussion on the LKML about InfiniBand. Greg K-H's original posting can be found here. Basically, he feels that it's impossible to implement the specification for InfiniBand in a free/open source product without violating the licensing agreement of the spec, because of patent infringement.
installing Infiniband on a single unit G5....
With so few companies left doing anything Infiniband related, makes you wonder what the thinking is here.
http://www.oreillynet.com/pub/a/network/2002/02/04 /windows.html
This is a short into to infiband.
"InfiniBand breaks through the bandwidth and fanout limitations of the PCI bus by migrating from the traditional shared bus architecture into a switched fabric architecture."
"Each connection between nodes, switches, and routers is a point-to-point, serial connection. This basic difference brings about a number of benefits:
Because it is a serial connection, it only requires four as opposed to the wide parallel connection of the PCI bus.
The point-to-point nature of the connection provides the full capacity of the connection to the two endpoints because the link is dedicated to the two endpoints. This eliminates the contention for the bus as well as the resulting delays that emerge under heavy loading conditions in the shared bus architecture.
The InfiniBand channel is designed for connections between hosts and I/O devices within a Data Center. Due to the well defined, relatively short length of the connections, much higher bandwidth can be achieved than in cases where much longer lengths may be needed."
"The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. At the time that I am writing this, there are already 1x and 4x adapters available in the market. So, the InfiniBand will be able to achieve must higher data transfer rates than is physically possible with the shared bus architecture without the fan-out limitations of the later."
This is cool. The Xserve is a great server. We got one at work and we used it as a mirror for a while before switchover. This thing never crashes. according to one of the articles these drivers will optimize the power of these beasts...
I've always understood that Myrinet is one of the better latency products available.
s p?content=9
And it has MacOSX Drivers:
http://www.myri.com/scs/macosx-gm2.html
Myrinet is used by 39% of the Top500 list published in November 2003
http://www.force10networks.com/applications/roe.a
The Virginia Tech cluster isn't on the top 500 list anymore:
from http://www.top500.org/lists/2004/06/trends.php
* The 'SuperMac' at Virginia Tech, which made a very impressive debut 6 month ago is off the list. At least temporarily. VT is replacing hardware and the new hardware was not in place for this TOP500 list.
People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop.
Note that V.T.'s cluster already uses InfiniBand, courtesy of Mellanox.
It's mentioned at V.T.'s pages.
...Halo and UT2004 were starting to slow down on my 1200 CPU cluster!
"Microsoft killed my company, I hold a personal grudge. I don't use Microsoft products and neither should you."-JWZ
The BigMac at VA Tech missed the list this year because they were busy switching over to DP G5 Xserves. Last I heard, they had completed the project and were busy re-benchmarking the beast. I I also heard that it was poised to move to number 2 possibly on the list after it was retested officially. The Army's version of the BigMac will probably take that title away though. That then 2 of the top 3 machines will be G5 based. Too Cool!
Why doesn't anything interesting happen when I have mod points?
You needn't be so snippy about it. If you had done any research at all into it you'd know that it was, indeed, in the #3 position but wasn't ranked at all last time around because it was down for an upgrade. They're moving from dual processor G5 desktop machines in the cluster to all G5 Xserves and since all the nodes weren't up during the official ranking period it doesn't appear on the list. Look for it to make a strong appearance again in the near future.
You seem like the type that needs proof, so here's the previous list.
No offense, but you don't know what you're talking about. IB can sustain tranfer rates of 700 MB/s; the best I've ever seen from GigE was almost an order of magnitude lower, not to mention the two orders of magnitude drop in latency with IB. That might not mean much to you, but I guarantee you it's a big deal for folks with big parallel scientific codes.
Oh, and your pricing's wrong too. In the quantities you'd need it for a decent size cluster, IB gear is about the same cost as its direct competitors (Myrinet and Quadrics).
Most of the parallel applications on our clusters are scientific simulation codes, written in Fortran, C, or C++ using MPI for inter-process communication.
"My life's work has been to prompt others... and be forgotten." --Cyrano de Bergerac
Doubtful. IBM's BlueGene is the king right now(well for the time being), but I don't see Big Mac(either version) beating the earth sim. Still, 2 out of the top 4 isn't bad.
Monstar L
Back in 2002, people were pitching IB as a replacement for PCI. Today, nobody tries to do that -- IB and PCI are used for different purposes (clustering and I/O expansion, respectively).
Small Tree also makes cool multiport gigabit ethernet cards that support 802.1ad bonding. Really, the gigE cards are the more interesting thing for most of us who don't have a supercomputing cluster to run. The two-port version is less than $300. They work on Linux as well.
http://small-tree.com/mp_cards.htm
Gigabit has a latency of about 100 microseconds and realistic throughput of about 50MB/s. Infiniband has a latency of about 15 microseconds and a throughput of about 500MB/s.
I mostly sell small Apple workgroup clusters of 16 nodes, and these are almost always just a gigE backbone. There are certain classes of problems that can benefit from Infiniband at low node counts, but for the most common apps, like gene searching using BLAST, gigE is just fine.
- "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
October 14, 2004 Pg. 54
http://www.netlib.org/benchmark/performance.pdf
http://appleturns.com/scene/?id=4980
"Calm down, Beavis; take a closer look at the third and fourth entries and you'll realize that they're the same exact cluster, before and after its owners added another 64 processors to it. In much the same way, System X is also listed in the seventh, ninth, and eleventh slots, with scores taken at various points along its journey to life as a complete 1,100-Xserve system. Factor out the doubles and, barring an "October Surprise," System X ought to sit in fifth place, under an Alpha cluster, a new Itanium2 system, the once-mighty Earth Simulator, and the new top dog, that chunk of IBM's unfinished BlueGene. Woo-hoo, PowerPCs in two of the top five! No other chip can say that."
~hylas
Yes, Twelve Captures Fifth (10/14/04).
an one program them in Python or Perl, or only "Real Programmers(tm)" languages like Java and C++?
They can be programmed in any language. Fortran and C are by far the most common choices. It's common to see perl and shell scripts used as glue between standalone modular programs. It's about the only place where you'll still occasionally find hand assembly in the inner loops, though that's becoming less common as more compilers support MMX,SSE, etc instructions.
You won't find a lot of interpreted languages doing heavy lifting in HPC. While a typical server is I/O bound (disk or net) and so can spare CPU cycles on an interpreted language, in HPC the CPU is normally pegged.
Anyone know if we'll ever see a Quad G5 in a Mac? Probably an Xserve, but even an IBM workstation using a quad G5 would be nice. Comments?
If you look at the PDF, NASA is 4th with 8 nodes. If they want to reach #1 they'll need to get 8 more nodes online pronto.
I think the thinking from Apple on the current configurations is that a dual 2.5Ghz is going to be better than the fastest available Intel-based system with a single 3.xGhz P4. There's no need to make a 4-way box because the 2-way box already beats the best P4 because 2.5+2.5=5. Or something like that. For clusters, who cares how many cores in a single box? Just link a bunch of 2-way systems together.
But, once the G5 goes dual-core, I would expect to see a dual dual-core G5 machine out there somewhere. Does that count?
How does InfiniBand compare to Xsan? Are they different systems altogether, do they work in conjunctioin with one another, or are they competing standards?
That's a blatant lie, without MACs ethernet simply wouldn't work.
I bet your computer is using a MAC right now!
This is not stuff cobbling 2-4 PCs together. Its for people who want the ultimate Xserve solution. I have a 16-processor Xserve G5 with Gig-E and Myrinet. My next solution will be some 96 processors and all InfiniBand.
1) Macs only has 3% of the market...so who cares?
2) Macs are only for designers...so who cares?
3) Macs cost more than PCs...so who cares?
I'm surprised we haven't seen the usual, eight year old "facts" as to why this is a fruitless effort. Slowly but surely, Apple is making its way back into the limelight. After being the whipping boy for so long for a variety of reasons (no market share, higher outright cost, stability issues, etc), Apple is proving itself to be cheaper, more stable, and damn powerful. Enterprises, educational institutions, the government, and more are all starting to see the benefits and overcome the usual excuses.
Just for fun, go back ten years and compare Apple products with the products of today. Also compare the general opinion and stereotypes of Apple from then to now. Anyone see a major difference?
"He uses statistics as a drunken man uses lampposts...for support rather than illumination." - Andrew Lang
Apple advocates constantly feel the need to point out in every other discussion about how much better Macintosh does everything.
/. and the OS of the drivers released, I'd say it brings something to the table.
Calling me an Apple advocate because I see announcement of OS X drivers for a product as unrelated to a discussion about compatability with open source projects is assuming too much. In fact I'm about as cross platform as someone can get. I personally use Linux, OS X and Windows for both work and home. The count of servers at work, where I'm a sysadmin, is 12 Linux, 6 Windows, and 1 Mac OS X server (not counting development servers). I use what bests gets the job done. Sound like a advocate to you?
I don't inject Apple topics into Linux discussions, and I don't appreciate others injecting Linux into Apple related discussions. It's as simple as that. If I was making the comment asking why someone is spouting about OS X in a Linux discussion, would that make you feel better? Usually, that comment is made by the time I read the posts, so I don't bother to say it again.
It certainly makes sense to ask the question of what those proprietary systems bring to the table.
No such question was asked. No question was asked at all. It sounded a lot like complaining about a company's policies in the wrong venue. I even suggested how to adress it in a larger, and more effective venue. If you want a discussion about what companies bring to the table, you don't start it by complaining about IP issues whe a company announces a driver release for a different platform. The post on what the latencies for a few interconnects I found to be the most useful of the discussion, and that is addressing the question that wasn't asked.
If you want to know what Infiniband brings to the table of Linux supercomputing, the answer may be nothing, for reasons cited. But for what it brings to OS X, the topic of the article, the section of
I am, and always will be, an idiot. Karma: Coma (mostly effected by
Voltaire's okay, but you'll notice that Small Tree isn't reselling their gear, they're reselling Infinicon's gear. ICS sells the switches and shared IO gear you need to put it all together.
As I understand it, the advantages of IB over gig-E are lower latency and scalability.
Clear, Dark Skies
but IIRC, SmallTree wrote the software for them; all the IB vendors use Mellanox's chips for their HCAs, they differentiate themselves with the software the lay on top of them.
Clear, Dark Skies
If we're assuming the Army's COLSA MACH5 will meet the October deadline, then that would make it 2 out of the top 6 machines being G5 based. But if we're counting installed but not-yet-operational supercomputers, then we should probably include Red Storm (41.5 Rpeak). Does anyone know if MACH5 and Red Storm will make the deadline?
TO START
PRESS ANY KEY
Where's the 'ANY' key? I see Esk, Kitarl, and Pig-Up...
First, it's NOT on the Top 500 list, goddammit!
Will you ever stop repeating that lie?
Second, it is under testing (not even in production).
(Third - not as relevant but still - why is a driver release still news? Topspin et al have been offering infinband drivers for Linux for a while; who wants
Calling me an Apple advocate
/. and the OS of the drivers released, I'd say it brings something to the table.
I didn't call you anything. Macintosh advocates in general regularly post all sorts of Apple trivia to Linux and X11 discussions. And Macintosh advocates in general make all sorts of claims about Linux and X11 in Apple forums on Slashdot. And that's OK: that's what Slashdot is about.
I don't inject Apple topics into Linux discussions, and I don't appreciate others injecting Linux into Apple related discussions.
Then you're on the wrong site. This is "News for Nerds", not "Macintosh driver central", "Macintosh advocacy", or "News for Apple Fans"; there are enough other sites and mailing lists where Apple fans can retreat in isolation, without the inconvenience of having their world view challenged.
No such question was asked. No question was asked at all. It sounded a lot like complaining about a company's policies in the wrong venue.
Slashdot is, in fact, exactly the right site to discuss Apple policies and contrast them with open source because this is a site where lots of people from all sorts of different backgrounds come. If not here, where else? What other site is there where people knowledgeable in different platforms can discuss these issues? The only problem is that people like you are quick to brand anybody as a "Troll" when you don't want to hear what they have to say.
If you want to know what Infiniband brings to the table of Linux supercomputing, the answer may be nothing, for reasons cited. But for what it brings to OS X, the topic of the article, the section of
I want to know what Macintosh+Infiniband brings to the table of supercomputing in general, relative to, say, Linux+Myrinet. We know that it may give you 20% lower latency in some caes if one post is to be believed, but little else, and even that didn't talk about cost/benefit. The matter seems to be settled in your mind, given your put-down of Linux-based cluster, but, sorry, you haven't made your case.
This is the Apple section of slashdot. These sections are present for a reason. Apple policies wern't being discussed at all, but Infiniband policies. Given that drivers are now released for Infiniband for OSX, the question of what this brings to Apple Clusters is something relevant to be discussed here. The reason I would brand you a troll is that you're speaking negatatively about something that is not relevant to this section of /.: How linux clusters are effected by IP issues decided by Infinband. You have yet to frame this in terms of contrasting how their policies effect Linux and how Apple drivers are now released. If you had started that way, then perhaps I would have just watched the discussion.
/. at the appropriate time (when the content on LWN becomes publically available). I all but wrote the submission for you! That hardly sounds like I'm trying to keep you from discussing this at all. I've suggested broadening your audience even.
To say I don't want to hear what you have to say has been already been proven wrong, as I have suggested previously that you make such a discussion in the appropriate section of
Also to say I've made up my mind about Infiniband or Linux v. Apple clusters couldn't be further from the truth. I haven't said anything about what I prefer, what is best, or anything to indicate my opinion. In fact, I have no informed opinion on the subject at all, much less have I expressed one. As a matter of fact, what is my direct put down of Linux-based clusters? Any comment I made was that this is that clusters likely to look at Infiband are not small scale, hobbyist clusters, but more likely larger clusters with larger budgets, so if the specs are free or not will be less of a factor.
I am, and always will be, an idiot. Karma: Coma (mostly effected by
unidirectional bandwidth of 931 million bytes per second is equal to 887 MegaBytes per second. More than an entire CD-ROM per second.
We're only coming out with 144 node switches now. BigMac must have used a pile of 32 or 16 node switches.
Clear, Dark Skies