Slashdot Mirror

What about this? This could help you maybe. https://community.mellanox.com...

Let me google that for you: http://www.mellanox.com/page/p...

PCIe 3.0 16x card. That's 135Gb/s of bandwidth via PCIe, so you at least know the IO can handle it.

most large systems dont use IB at all

InfiniBand is the only Petascale-proven, standard interconnect solution, connecting 24 out of the 48 Petaflop capable systems on the [TOP500]

Say, does Windows support zero-copy Infiniband links?

Yes, it does.
Click Here

http://www.mellanox.com/pdf/whitepapers/SDPCluster2006.pdf

How about MPI performance?

Yes, it does.
http://www.purempi.net/

How about fire&forget clustered processes?

Yes, it does.
http://blogs.msdn.com/distributedservices/default.aspx

There are already10Gbt copper cards from Mellanox. They can tolerate cable runs of 55 meters using cat5e and cat 6 or the full 100 meters using cat6A. That might be a problem for current cabling infrastructure but backbones most likely use fiber it will be no problem. Work stations and desktops wont need 10gbt any time soon so the existing 1gbt over cat 5 is fine.

And as for 40/100 and beyond Ethernet who cares what cable media it will use. Most likely will be fiber and most likely wont really be necessary for the desktop/workstation any time soon. Gigabit should last us another 10 years for desktop and laptop use. 10 gig would be good for use on workstation and server boards.

Yes, yes it does.

Yes, yes it does.

Let me try and jerk out into reality. Here is the press release from Mellanox announcing $69 per IB HCA chip in volume. http://www.mellanox.com/news/press_releases/pr_030 105.php
Agreed its not widespread yet. But there are systems manufacturers actively looking at deploying this silicon on the mother board. IWILL is one company thats already started. Asus, Tyan are next.

The reality is that there is only one *correct* way to do a fast interconnect, and that is to build it into the CPU itself. Oh wait, AMD intends to do just that!

You do realize on board interconnects can only scale so much .. before you run into serious data coherency issues. Intel Xeon's can scale up to 4 processors per board. IBM has tweaked so they can scale up to 8 procs on their proprietory motherboards. AMD Hypertransport allows you to scale upto 8 sockets per motherboard. To scale beyond that you have to go use NUMA like interconnect (pricey) or push it on a fast pipe to other boxes (not so pricey). AMD recommends this. AMD specifically recommends to use InfiniBand to push high bandwidth. Go talk to AMD. Hypertransport is available *today* in the market. Go talk to supermicro. You don't have to wait for it. All this other junk has a shelf life of maybe a few years at best (as does pretty much all networking gear, but the difference is that this stuff costs 10 times as much). Pretty much everything in computers has a shelf life of few years hardware and software .. what is your point ? It's a huge waste of time for all but the most extreme clustered applications for which there is no algorithmic solution to the latency issue (read: people like to throw hardware at badly written programs more often then they should). People are not complete idiots when they are spending millions of dollars setting up 100's or even 1000's of nodes of beowulf clusters over IB. There are specific applications that are latency sensitive and bandwidth hungry. And that is the segment that IB addresses very well and where ethernet is considered an outsider. Tell me you want to use 1G ethernet to process a satellite feed coming in at 3.2 Gbps .. again, I think you are either clueless or ignorant .. just because IB doesn't fit your bill doesn't mean it doesn't have its play.

People have already been making supercomputer clusters for the Mac, including Virginia Tech's third-fastest supercomputer in the world, but InfiniBand is supposed to make the latency drop.

Note that V.T.'s cluster already uses InfiniBand, courtesy of Mellanox.

It's mentioned at V.T.'s pages.

Infiniband. Infiniband. Infiniband.

The Infiniband NICs are made by Mellanox, and IB switches are made by Mellanox, Voltaire, and one other company whose name eludes me. I don't know who developed the OSX drivers (maybe Mellanox, maybe Apple, maybe both), but AFAIK the high-level MPI library that pretty much everybody (including Va. Tech) uses for IB is a channel driver for MPICH developed by D.K. Panda's group at Ohio State. (I know this because Pete Wyckoff, a co-worker of mine at OSC, has done a whole bunch of debugging and stress testing for Panda's crew; his name's on most of their IB-related papers too.)

In short, the only thing Apple might have bought to the table as far as the IB interconnect on the Va. Tech machine is some driver development.

--Troy

Cost of this 1 teraflop Mellanox machine is less than US$1e6 according to this brochure.

That's considerably less than the US$50e6 that the first teraflop machine cost (Sandia's ASCI Red see this SC1996 flier) 7 years ago.

I don't have a spare million, either, but that kind of 98% price reduction is still fairly impressive.

The interconnect is Infiniband by Mellanox. These things get 10Gbps bandwidth with 6us latency under MPI. Very decent stuff. There is more information at the site above.

Note that 1100*$3000 = $3M. This doesn't include the 4GB RAM, but also doesn't include any volume discounts. Thus the interconnect may cost about $2M.

Oh, and to the guy who said "4 Athlons + Myranet is the same price as one G5" -- can I have some of what you're smoking?