"... In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.
Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"
What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.
The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach
1. User-space network processing means more efficient resource usage and accounting 2. User-space network processing means more efficient data transfer between application and stack. 3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing. 4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card. 5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.
Overall, it seems to have a lot of benefits -- a)Because it's Ethernet based no need to rip out existing infrastructure b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology c)because it's a drop in solution no need for application or kernel redesign and re-engineering.
IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricperf.htm [level5networks.com] ] seem to indicate just this.
As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.
Slashdot Mirror
It surprises me how many people on slashdot either cannot or will not read. From the official website:m [level5networks.com]
f .htm [level5networks.com] ] seem to indicate just this.
http://www.level5networks.com/prod_etherfabric.ht
"...
In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.
Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"
What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.
The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach
1. User-space network processing means more efficient resource usage and accounting
2. User-space network processing means more efficient data transfer between application and stack.
3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing.
4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card.
5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.
Overall, it seems to have a lot of benefits --
a)Because it's Ethernet based no need to rip out existing infrastructure
b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology
c)because it's a drop in solution no need for application or kernel redesign and re-engineering.
IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricper
As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.