Slashdot Mirror
Is There a Place for a $500 Ethernet Card?
prostoalex writes "ComputerWorld magazine runs a story on Level 5 Networks, which emerged from the stealth startup status with its own brand of network cards and software called EtherFabric. The company claims they are reducing the load on the servers CPUs and improve the communications between the servers. And it's not vaporware: 'The EtherFabric software shipping Monday runs on the Linux kernel 2.4 and 2.6, with support for Windows and Unix coming in the first half of next year. High volume pricing is $295 for a two-port, 1GB-per-port EtherFabric network interface card and software, while low volume quantities start from $495.'"
Plain old 32-bit PCI has a bandwidth of 32 bits * 33 MHz = 1.056 Gbps. A 64-bit bus would obviously be twice that. Unless there is a lot of other traffic on the PCI bus, I suspect the limitation is the driver, not the bus.
Peripherals like that built into the motherboard are generally on a PCI bus segment anyway. You can see by looking at the device manager in Windows or by using lspci in Linux. In both cases you will see a vendor ID, bus number, and slot number.
With newer Linux-kernels this is quite simply not the case.
To avoid the torrent of interupts from a fast nic the Linux-kernel detects that the card gets packets so often that essentially there's "always" one or more packets waiting in the cards buffers.
It responds to this condition by disabling interupts for the card in question and switch to polling it regularily.
Normally polling is inefficient, because it amounts to asking over and over again "got anything for me now?", where in most situations the answer is "no" 99.99% of the time, which makes it a waste of resources to ask in the first place.
This changes when the answer is "yes" basically all the time. There's no need for the network card to tell the cpu over and over and over "I got a packet for you", instead the cpu collects packets regularily.
It's sorta like having your lawyer receive legal letters for you.
If you get very few, it'd be a waste for you to drop by him every day and ask if he's gotten any for you (polling) most of the days you'd be making the trip for naught. In this situation interupts (i.e. having your lawyer call you and inform you on the rare occasions when a letter *does* arrive) makes more sense.
But if your letter-flow increases to the point where there's normally 3-5 letters every day and it's rare that no letter arrives, then it no longer makes sense that the lawyer calls you every day to tell you you got letters (interupts), you already assumed that. In this scenario it makes more sense for you to come by regularily and pick up letters without being prompted to do so. (polling)
Thus, the flow of interupts from a Gb-nic being flooded with 100byte small-packets (say a loaded dns-server) is not 1 million interupts every second -- it is zero interupts.
(allthough what you write is correct for kernels older than 2.6.10 and for less clever OSes.)
Accelerating Ethernet in hardware, while remaining 100% compatible with the standard protocols on the wire, isn't all that new. Just over 2 years ago, I worked on a TOE (TCP offload engine) card at Adaptec.
l ymatrix.html?cat=%2fTechnology%2fNAC+Cards%2fNAC+C ards
http://www.adaptec.com/worldwide/product/prodfami
It was a complete TCP stack in hardware (with the exception of startup/teardown, which still was intentionally done in software, for purposes of security/accounting).
Once the TCP connection was established, the packets were completely handled in hardware, and the resulting TCP payload data was DMA'ed directly to the application's memory when a read request was made. Same thing in the other direction, for a write request. Very fast!
I'm not sure of the exact numbers but we reduced CPU utilization to around 10%-20% of what it was under a non-accelerated card, and were able to saturate the wire in both directions using only a 1.0Ghz CPU. This is something that was difficult to do, given the common rule of thumb that you need 1Mhz of CPU speed to handle every 1Mbit of data on the wire.
To make a long story short, it didn't sell, and I (among many others) was laid off.
The reason was mostly about price/performance: who would pay that much for just a gigabit ethernet card? The money that was spent on a TOE-accelerated network card would be better spent on a faster CPU in general, or a more specialized interconnect such as InfiniBand.
When 10Gb Ethernet becomes a reality, we will once again need TOE-accelerated network cards (since there are no 10GHz CPU's today, as we seem to have hit a wall at around 4Ghz). I'd keep my eye on Chelsio: of the Ethernet TOE vendors still standing, they seem to have a good product.
BTW, did you know that 10Gb Ethernet is basically "InfiniBand lite"? Take InfiniBand, drop the special upper-layer protocols so that it's just raw packets on the wire, treat that with the same semantics as Ethernet, and you have 10GbE. I can predict that Ethernet and InfiniBand will conceptually merge, sometime in the future. Maybe Ethernet will become a subset of InfiniBand, like SATA is a subset of SAS....
Dr. Demento On The 'Net!