Slashdot Mirror

The white-paper and the web pages on the company site which describe the implementation talk about how this is done.

This card, and the software which drives it, differ from traditional ethernet accellerator cards and from alternative network protocols (like myrinet and iWarp) in several ways.

Alternative protocols not only require using a different software API but also require custom hardware at both communication endpoints.

Traditional hardware TCP/IP accelerators run the bottom half of the stack in custom silicon. This does tend to help reduce host CPU load but suffer from a number of problems. Since host CPU speeds have tended to increase regularly, they often helped for only a brief period of time. They also tended to help most for large packets but helped little or not at all for small packets.

This technology claims to help large and small packets equally well, and also claims to reduce packet latency across the board. It does so by running the bulk of the TCP/IP stack in user space rather than via system calls. The hardware runs ethernet Rx and Tx processing but does not implement the higher level IP protocol processing. Instead, once connections are established, the ethernet frames coming from the hardware, are fed via a system call interface to the application process to which they belong. Then, no further context switching between kernel and the process are required. The top end of the hardware driver and all of the subsequent IP layers, are executed in the context of the user space process. They are linked to the app via shared libraries.

Basically, instead of the linking the IP calls against code which requires frequent switching between user and kernel space, the entire upper half of the stack is run by the application sending and receiving the packets. This offers uniform benefits in packet latency across all packet sizes, and offers improvement in throughput as well.

I assume that all that is required is to link against a different set of shared libraries to gain these benefits (and of course to have the custom hardware on at least one side of the comm. link). This looks very good in principle.

The following page provides an overview of the technology and compares it to each of the competing mechanisms.
http://www.level5networks.com/sol_approaches.htm

It surprises me how many people on slashdot either cannot or will not read. From the official website:
http://www.level5networks.com/prod_etherfabric.htm [level5networks.com]

"...
In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.

Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"

What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.

The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach

1. User-space network processing means more efficient resource usage and accounting
2. User-space network processing means more efficient data transfer between application and stack.
3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing.
4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card.
5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.

Overall, it seems to have a lot of benefits --
a)Because it's Ethernet based no need to rip out existing infrastructure
b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology
c)because it's a drop in solution no need for application or kernel redesign and re-engineering.

IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricperf .htm [level5networks.com] ] seem to indicate just this.

As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.

It surprises me how many people on slashdot either cannot or will not read. From the official website:
http://www.level5networks.com/prod_etherfabric.htm [level5networks.com]

"...
In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.

Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"

What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.

The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach

1. User-space network processing means more efficient resource usage and accounting
2. User-space network processing means more efficient data transfer between application and stack.
3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing.
4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card.
5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.

Overall, it seems to have a lot of benefits --
a)Because it's Ethernet based no need to rip out existing infrastructure
b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology
c)because it's a drop in solution no need for application or kernel redesign and re-engineering.

IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricperf .htm [level5networks.com] ] seem to indicate just this.

As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.

It surprises me how many people on slashdot either cannot or will not read. From the official website:
http://www.level5networks.com/prod_etherfabric.htm

"...
In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.

Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"

What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.

The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach

1. User-space network processing means more efficient resource usage and accounting
2. User-space network processing means more efficient data transfer between application and stack.
3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing.
4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card.
5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.

Overall, it seems to have a lot of benefits --
a)Because it's Ethernet based no need to rip out existing infrastructure
b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology
c)because it's a drop in solution no need for application or kernel redesign and re-engineering.

IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricperf .htm ] seem to indicate just this.

As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.

It surprises me how many people on slashdot either cannot or will not read. From the official website:
http://www.level5networks.com/prod_etherfabric.htm

"...
In the conventional network I/O architecture, applications make Sockets or MPI function calls to interact with the network, and the Sockets library software converts these to system calls into the operating system kernel TCP/IP stack. With the EtherFabric architecture, Sockets and MPI function calls from applications invoke EtherFabric library routines that perform the necessary TCP/IP protocol processing in the user, or application, space. They move data directly to or from the virtual hardware interface that has been assigned to the specific user context. This approach eliminates context switching and greatly improves cache locality for both instructions and data, producing substantial reductions in latency and improvements in CPU efficiency.

Furthermore, EtherFabric hardware notifies each user process about incoming packets by placing events on the event queue for the relevant virtual interface. EtherFabric software de-queues these events and performs protocol processing on incoming packets without the need for interrupts. Substantial further savings in CPU cycles are made by avoiding the inefficiencies of interrupt processing"

What they are saying here is that ALL TCP/IP Processing is done in USER-SPACE. From what the blurb says I would assume the actual stack is in user-space and that only interaction with the kernel would be on connection setup/teardown. They don't say how they actually move data in and out of the application but I would bet my bottom dollar they probably do memory mapping or some other trick to allow data movement in and out of the application without corssing the kernel/user application boundary.

The actual design seems like a combination of a very smart ethernet NIC and a kick-ass user-space application space IP stack. I can see a lot of advantages with this approach

1. User-space network processing means more efficient resource usage and accounting
2. User-space network processing means more efficient data transfer between application and stack.
3. By doing the network processing in user-space on the host as opposed to a typical TOE, you are not constrained by the NIC. A faster CPU means you benefit from faster TCP/IP processing.
4. This also makes upgrades and software bugfixes much easier and more maintainable -- replace a library as opposed to having to replace the card or firmware of the card.
5. By using an efficient NIC with hardware support for virtual channels you have very efficient demultiplexing, you dont waste cycles in software oding this.

Overall, it seems to have a lot of benefits --
a)Because it's Ethernet based no need to rip out existing infrastructure
b)Because it's normal tcp/ip as opposed to some weirdo network protocol it's well understood and compatible with existing technology
c)because it's a drop in solution no need for application or kernel redesign and re-engineering.

IMHO, if this does what it says it does, it will revolutionise traditonal ethernet cluster/data centre processing. It's analagous to a turbo-booster on a car, just switch it on and hold on to your hat. Their benchmarks [ http://www.level5networks.com/prod_etherfabricperf .htm ] seem to indicate just this.

As faster networking technologies become more prevalent (eg 10 Gig Ethernet) the problem of network cpu consumption will become a bigger and bigger issue for server type applications. The toss up will be between this approach and the TOE approach -- coupled with the advent of multi-core CPU's I can see this being a much more flexbile solution than TOE's. However, time will tell.

The best explanation of what they've done is a bit hidden away on their site:

http://www.level5networks.com/sol_approaches.htm

This has some block diagrams comparing conventional ethernet, other things like infiniband, and their technology.

The main reason why it's fast is that it reduces the number of context switches, since less work is done by the kernel, and reduces the number of times that the data is copied from one buffer to another.

These people have been working on this for some time and really know what they are doing. I'm suprised that it's as cheap as $500. If you need to accelerate a file server, web server, etc., then this is almost certainly a very cost-effective way of doing it.

And please be happy that they've released it for Linux *first*! This really shows how important Linux is to server applications.

I looked at their benchmark web page http://www.level5networks.com/prod_etherfabricperf .htm where they claim that a typical PC with "conventional" ethernet burns 83.5% of CPU for communication overhead while only 16.5% remain to the application.
But they don't say which CPU was used - probably an 850 MHz Pentium III or something similar outdated.

Fact is, on a current 3.x GHz Pentium IV or an equivalent Athlon or Opteron the communication overhead is in one digit range, percentage-wise.

A famous computer science quote is:
"Lies, damned lies and benchmarks"
and another one is
"Don't trust any statistics that you haven't forged yourself."

From their website:

Low Latency - EtherFabric provides sub-10 usec (micro-second) latencies between application instances on different servers, improving application-to-application inter-server communication by 3-5x over conventional Ethernet.

http://www.level5networks.com/prod_etherfabric.htm