Slashdot Mirror

That IBM whitepaper link was supposed to be: Performance of HPC Applications over InfiniBand, 10 Gb and 1 Gb Ethernet

I'll assume you know more about this than me, but he did say that the nodes are going to be wired with 4x GigE. Might there be a penalty bridging from that to IB rather than 10GigE?

Anyway, to get low latency those GigE links to the nodes need to be optimized. I thought this was interesting:

High performance network technologies such as InfiniBand use a kernel by-pass method to improve performance. This capability is also available for Ethernet, but is not widely used outside of the HPC community. One such methodology is Intel® Direct Ethernet Transport (DET), which works by providing a User Direct Access Programming Library (uDAPL) interface like InfiniBand. uDAPL defines a single set of user APIs for all Remote Memory Direct Access (RDMA)-capable transports. DET includes a kernel module and an uDAPL library for Ethernet and will work on almost any Ethernet NIC. It can be linked with any software requiring a uDAPL library, such as an MPI version.

Another popular kernel by-pass effort is the Open-MX project. Open-MX is based on the Myrinet MX protocol. Essentially, any software that links to the Myricom MX library should be able to link with Open-MX. Currently, Open MPI, MPICH2, and the PVFS2 file system have all been shown to work with Open-MX. While Open-MX will work with almost all GigE and 10-GigE chip-sets without modifying drivers, it does require kernel 2.6.15 or higher to work. Depending on the chip-set Open-MX latencies as low as 10 seconds for GigE have been reported.

(From The Ethernet Cluster

For 10GigE here's a recent low-latency benchmark:
Audited STAC-M2 Benchmark of IBM LLM on an IBM-BNT G8264 switch, using IBM x3550 servers and Mellanox MNPH29C-XTR ConnectX®-2 EN with RoCE
"Using standard Ethernet and RoCE protocols, at the base message rates set by the specs, the mean latency of the solution did not exceed 7 microseconds, while standard deviation of latency was measured at 1 microsecond. At the highest tested rate of 2.3 million messages/second, the mean latency of the solution was just 13 microseconds while the standard deviation of latency was measured at 2 microseconds."

Chelsio claims 3 microsecond latency using RDMA over 10G Ethernet on their "T4" model: "Chelsio T4 Unified Wire adapters can run iWARP RDMA, TCP, iSCSI and FCoE simultaneously with full offload and deliver full wire speed throughput and extremely low latency between the computing nodes, the storage resources, and the user and cluster management nodes in any HPC environment." Not sure how much that really costs compared to IB, though.
They also say:

"Since IB lacks congestion management and adaptive routing, it quickly hits hot spots even in clusters of moderate size. iWARP over Ethernet, in contrast, achieves reliability via TCP, which results in a lower effective latency for useful applications."

*"10Gb IB link is effectively 8Gb. Furthermore, InfiniBand cards, like Ethernet cards, are limited by PCIeGen2 x8. Independently of how many 10Gb or 40Gb ports an adapter exposes, the aggregate bandwidth is limited to about 26Gbps in each direction. Therefore, Chelsio’s T4 based adapters and the leading IB adapters offer the SAME bandwidth."

*"Ethernet switch port prices have reached parity. The same can be said about adapter prices. However, an IB cluster further requires an Ethernet switch for management, a gateway for routing, and expensive IB storage available from a limited set of suppliers, as well as specialized IT personnel."

I'll assume you know more about this than me, but he did say that the nodes are going to be wired with 4x GigE. Might there be a penalty bridging from that to IB rather than 10GigE?

Anyway, to get low latency those GigE links to the nodes need to be optimized. I thought this was interesting:

High performance network technologies such as InfiniBand use a kernel by-pass method to improve performance. This capability is also available for Ethernet, but is not widely used outside of the HPC community. One such methodology is Intel® Direct Ethernet Transport (DET), which works by providing a User Direct Access Programming Library (uDAPL) interface like InfiniBand. uDAPL defines a single set of user APIs for all Remote Memory Direct Access (RDMA)-capable transports. DET includes a kernel module and an uDAPL library for Ethernet and will work on almost any Ethernet NIC. It can be linked with any software requiring a uDAPL library, such as an MPI version.

Another popular kernel by-pass effort is the Open-MX project. Open-MX is based on the Myrinet MX protocol. Essentially, any software that links to the Myricom MX library should be able to link with Open-MX. Currently, Open MPI, MPICH2, and the PVFS2 file system have all been shown to work with Open-MX. While Open-MX will work with almost all GigE and 10-GigE chip-sets without modifying drivers, it does require kernel 2.6.15 or higher to work. Depending on the chip-set Open-MX latencies as low as 10 seconds for GigE have been reported.

(From The Ethernet Cluster

For 10GigE here's a recent low-latency benchmark:
Audited STAC-M2 Benchmark of IBM LLM on an IBM-BNT G8264 switch, using IBM x3550 servers and Mellanox MNPH29C-XTR ConnectX®-2 EN with RoCE
"Using standard Ethernet and RoCE protocols, at the base message rates set by the specs, the mean latency of the solution did not exceed 7 microseconds, while standard deviation of latency was measured at 1 microsecond. At the highest tested rate of 2.3 million messages/second, the mean latency of the solution was just 13 microseconds while the standard deviation of latency was measured at 2 microseconds."

Chelsio claims 3 microsecond latency using RDMA over 10G Ethernet on their "T4" model: "Chelsio T4 Unified Wire adapters can run iWARP RDMA, TCP, iSCSI and FCoE simultaneously with full offload and deliver full wire speed throughput and extremely low latency between the computing nodes, the storage resources, and the user and cluster management nodes in any HPC environment." Not sure how much that really costs compared to IB, though.
They also say:

"Since IB lacks congestion management and adaptive routing, it quickly hits hot spots even in clusters of moderate size. iWARP over Ethernet, in contrast, achieves reliability via TCP, which results in a lower effective latency for useful applications."

*"10Gb IB link is effectively 8Gb. Furthermore, InfiniBand cards, like Ethernet cards, are limited by PCIeGen2 x8. Independently of how many 10Gb or 40Gb ports an adapter exposes, the aggregate bandwidth is limited to about 26Gbps in each direction. Therefore, Chelsio’s T4 based adapters and the leading IB adapters offer the SAME bandwidth."

*"Ethernet switch port prices have reached parity. The same can be said about adapter prices. However, an IB cluster further requires an Ethernet switch for management, a gateway for routing, and expensive IB storage available from a limited set of suppliers, as well as specialized IT personnel."

The limitations were more the frame size (not using jumbo frames on that network) along with the read/write speeds of the system on each end.

Umm... probably not. Jumbo frames aren't a panacea, and rarely help much in practice. Our own internal benchmarking showed jumbo frames providing at most a 10% boost in TCP throughput for iSCSI/GbE. Since enbaling Jumbo end-to-end in a large network comes with significant configuration and testing costs, it's really not worth the hassle.

I used to be really against iSCSI, as the native stacks on various OSes just did not deal with it well. By that I mean that a 50 MB/s file transfer would consume almost 100% of a 3ghz CPU. Also, the hard limit on gig-e transfers of 85 MB/s (TCP/IP overhead + iSCSI overhead) was just too low.

Now, that has all changed. Not only can you get TCP/IP Offload Engines for just about every OS (I don't work with Windows, so I don't know what the status of that is). Also, 10 gigabit ethernet has become financially reasonable.

For instance, the T210-cx is around $800, and will deliver a sustained 600 MB/s (not peak or any other crap). Also, the latency on a 1500 MTU 10-gbs ethernet fabric is something to behold.

I think by the end of this year, we will see iSCSI devices on 10gbe that out-perform traditional SAN equipment in the 2gbs evironment, in every respect (including price), by a large margin. 4gbs SAN could come close, but I still think hardware accelerated iSCSI has a _ton_ of potential.

If I were starting a storage company today, I would be focusing exclusively on the 10gbs iSCSI market. It is going to explode this year.

People should have been using FBSD for wifi access points and aggregation long ago. Glad to see the corporate folks have finally figured out how linux errodes the bottom line.

The
stupider
ones
haven't
figured
it
out
yet.
You'd
better
go
tell
them!

Accelerating Ethernet in hardware, while remaining 100% compatible with the standard protocols on the wire, isn't all that new. Just over 2 years ago, I worked on a TOE (TCP offload engine) card at Adaptec.

http://www.adaptec.com/worldwide/product/prodfamil ymatrix.html?cat=%2fTechnology%2fNAC+Cards%2fNAC+C ards

It was a complete TCP stack in hardware (with the exception of startup/teardown, which still was intentionally done in software, for purposes of security/accounting).

Once the TCP connection was established, the packets were completely handled in hardware, and the resulting TCP payload data was DMA'ed directly to the application's memory when a read request was made. Same thing in the other direction, for a write request. Very fast!

I'm not sure of the exact numbers but we reduced CPU utilization to around 10%-20% of what it was under a non-accelerated card, and were able to saturate the wire in both directions using only a 1.0Ghz CPU. This is something that was difficult to do, given the common rule of thumb that you need 1Mhz of CPU speed to handle every 1Mbit of data on the wire.

To make a long story short, it didn't sell, and I (among many others) was laid off.

The reason was mostly about price/performance: who would pay that much for just a gigabit ethernet card? The money that was spent on a TOE-accelerated network card would be better spent on a faster CPU in general, or a more specialized interconnect such as InfiniBand.

When 10Gb Ethernet becomes a reality, we will once again need TOE-accelerated network cards (since there are no 10GHz CPU's today, as we seem to have hit a wall at around 4Ghz). I'd keep my eye on Chelsio: of the Ethernet TOE vendors still standing, they seem to have a good product.

BTW, did you know that 10Gb Ethernet is basically "InfiniBand lite"? Take InfiniBand, drop the special upper-layer protocols so that it's just raw packets on the wire, treat that with the same semantics as Ethernet, and you have 10GbE. I can predict that Ethernet and InfiniBand will conceptually merge, sometime in the future. Maybe Ethernet will become a subset of InfiniBand, like SATA is a subset of SAS....