Slashdot Mirror


Is There a Place for a $500 Ethernet Card?

prostoalex writes "ComputerWorld magazine runs a story on Level 5 Networks, which emerged from the stealth startup status with its own brand of network cards and software called EtherFabric. The company claims they are reducing the load on the servers CPUs and improve the communications between the servers. And it's not vaporware: 'The EtherFabric software shipping Monday runs on the Linux kernel 2.4 and 2.6, with support for Windows and Unix coming in the first half of next year. High volume pricing is $295 for a two-port, 1GB-per-port EtherFabric network interface card and software, while low volume quantities start from $495.'"

19 of 423 comments (clear)

  1. Is There a Place for a $500 Ethernet Card? by Anonymous Coward · · Score: 5, Funny

    Yes, there is a place for a $500 ethernet card, far, far away from this guy.

  2. A look into the past by bananahead · · Score: 5, Insightful
    This sounds very similar to the 'smart card' concept back in the late 80's and early 90's. Intel had the 586-driven smart-cards, and I believe 3Com had them as well. They were intended to offload the CPU by putting parts of the stack on the card. They failed because the performance gain and CPU offload numbers were never enough to justify the price difference.

    I wonder what has changed? I have never known the CPU to get dragged down by network traffic, but maybe in the network server markets it is different, However with the Ethernet chipsets being designed into the motherboard and integrated into the tight circle of RAM and CPU, it isn't clear there is a need for this.

    How long before the network control is put into the CPU? It is going to be tough to beat that type of performance.

    --
    A most overlooked advantage to owning a computer is if they foul up there's no law against wacking them around a bit.
    1. Re:A look into the past by bluelip · · Score: 5, Interesting

      I've noticed a slowdown in computer response when using gig cards and moving lotsa' data. I thought the bottleneck may have moved to the file systems. Didn't seem to be the case as pumping dummy data throught the nic also caused issues.

      I didn't pursue it far enough to see where where the actual problem was. These cards may help, but my money is on a faster cpu.

      --

      Yep, I never spell check.
      More incorrect spellings can be found he
    2. Re:A look into the past by Anonymous Coward · · Score: 5, Informative

      Plain old 32-bit PCI has a bandwidth of 32 bits * 33 MHz = 1.056 Gbps. A 64-bit bus would obviously be twice that. Unless there is a lot of other traffic on the PCI bus, I suspect the limitation is the driver, not the bus.

      Peripherals like that built into the motherboard are generally on a PCI bus segment anyway. You can see by looking at the device manager in Windows or by using lspci in Linux. In both cases you will see a vendor ID, bus number, and slot number.

    3. Re:A look into the past by Fweeky · · Score: 4, Interesting

      I expect you can get an Intel 1000/Pro for around $30; full TCP/IP checksum offloads in both directions, interrupt moderation, jumbo frames, and Intel even write their own open source drivers.

      Heh, my on-board Realtek GigE chip has checksum offloads too, but even with them on, 300Mbps would have me up to 70% system/interrupt CPU load (and I hear the checksumming is a bit.. broken); I barely scrape 30% with a PRO/1000GT.

    4. Re:A look into the past by Phil+Karn · · Score: 4, Insightful
      And how long ago was that? What kind of servers had loads increase by 20% when you dumped the "smart" NICs? How much faster have general purpose CPUs gotten since then? And whose unusually inefficient TCP/IP stack and/or Ethernet driver were you running?

      "Smart" network cards are one of those bad ideas that keep coming back from the grave, because computer science seems to lose its collective memory every decade or so.

      Fifteen years ago, Van Jacobsen did a wonderful presentation at SIGCOMM 1990 on just why they were such a bad idea. The reason is very simple. A modern, well-tuned and optimized TCP/IP stack can process a packet with only about 20 instructions on average. Very few "smart" controller cards have host interfaces that can be spoken to with so few instructions! The switch to and from kernel context will usually cost you more than TCP/IP.

      Not only that, but the coprocessor on the "smart" controller card inevitably ends up being considerably slower than the host CPU, because typical host CPUs are made in much larger quantities, enjoy large economies of scale, and are updated frequently. So you often have the absurd situation of a blazingly fast and modern host CPU twiddling its thumbs waiting for some piss-poor slow CPU on a "smart" controller to execute a protocol handler that could have been done on the host with fewer instructions than it takes just to move a packet to or from the "smart" card.

      And if that weren't enough, rarely do these "smart" network controllers come with open source firmware. Usually the company that makes them obsoletes them quickly (because they don't sell well) and/or goes out of business, and you're left with a very expensive paperweight.

      Since his talk, Ethernet interfaces have totally obsoleted "smart" network cards. They now come with lots of internal buffering to avoid losing packets when interrupt latencies are high, and they take relatively few instructions per byte of user data moved. What more could you want?

    5. Re:A look into the past by klui · · Score: 4, Insightful
      It would depend on the implementation. Not all mobos with built-in ports have "direct access." Some of them go through a shared bus or worse, the PCI bus.

      Intel's implementation for the 865P/875P chipset goes through the memory hub directly http://www.intel.com/design/chipsets/schematics/25 281202.pdf while the i845 chipset has the ethernet interface connected to the ICH4 controller hub that is shared among other devices like the PCI bus http://www.intel.com/design/chipsets/datashts/2519 2401.pdf. VIA's PT894/PT880 ethernet connection goes through a "VIA Connectivity" bus much like the Intel 845 http://www.via.com.tw/en/products/chipsets/p4-seri es/pt894pro and http://www.via.com.tw/en/products/chipsets/p4-seri es/pt880. There were some value motherboards that although I recall that they use good/decent chipsets, their designers decided to connect the built-in gigabit ethernet ports off the PCI bus. I cannot recall what these were but I read about them in anandtech several years ago.

    6. Re:A look into the past by 0rbit4l · · Score: 4, Informative
      The reason is very simple. A modern, well-tuned and optimized TCP/IP stack can process a packet with only about 20 instructions on average.
      No, Van Jacobson showed that the fast-path receive could be done in 30 instructions. This doesn't include ("smart"-NIC features, which you ignorantly deride) TCP/IP checksum calculation, nor is it even remotely realistic for a modern TCP/IP stack that supports modern RFCs. You're not including timeout code, connection establishment, state updates, or reassembly on the receive side, and you conveniently completely ignore segmentation issues on the send side. If "smart" NICs are such a bad idea, then I guess Intel and Sun are really up a creek - Intel currently supports TCP segmentation offload (pushing the packet segmentation task from the TCP stack onto the hardware), and is moving to push the entire TCP stack to a dedicated processor + NIC combo.
      Since his talk, Ethernet interfaces have totally obsoleted "smart" network cards.
      You couldn't be more wrong. Since the 90s, the boundary of what the NIC should do and what the OS should do has been repeatedly re-examined, and industry leaders in networking have successfully deployed products that big-iron servers rely on.
    7. Re:A look into the past by Eivind · · Score: 5, Informative
      Smart cards do more than just offload the CPU from handling TCP data. They offload the PCI bus from handling interrupts. Every packet of data triggers an interrupt.

      With newer Linux-kernels this is quite simply not the case.

      To avoid the torrent of interupts from a fast nic the Linux-kernel detects that the card gets packets so often that essentially there's "always" one or more packets waiting in the cards buffers.

      It responds to this condition by disabling interupts for the card in question and switch to polling it regularily.

      Normally polling is inefficient, because it amounts to asking over and over again "got anything for me now?", where in most situations the answer is "no" 99.99% of the time, which makes it a waste of resources to ask in the first place.

      This changes when the answer is "yes" basically all the time. There's no need for the network card to tell the cpu over and over and over "I got a packet for you", instead the cpu collects packets regularily.

      It's sorta like having your lawyer receive legal letters for you.

      If you get very few, it'd be a waste for you to drop by him every day and ask if he's gotten any for you (polling) most of the days you'd be making the trip for naught. In this situation interupts (i.e. having your lawyer call you and inform you on the rare occasions when a letter *does* arrive) makes more sense.

      But if your letter-flow increases to the point where there's normally 3-5 letters every day and it's rare that no letter arrives, then it no longer makes sense that the lawyer calls you every day to tell you you got letters (interupts), you already assumed that. In this scenario it makes more sense for you to come by regularily and pick up letters without being prompted to do so. (polling)

      Thus, the flow of interupts from a Gb-nic being flooded with 100byte small-packets (say a loaded dns-server) is not 1 million interupts every second -- it is zero interupts.

      (allthough what you write is correct for kernels older than 2.6.10 and for less clever OSes.)

  3. Sure there's a place for them by grub · · Score: 5, Insightful

    Is There a Place for a $500 Ethernet Card?

    Of course there is, assuming the card performs as advertised. Sheer conjecture: the card likely has a lot of the smarts onboard. Maybe it has some of the TCP and IP stuff on board too (checksum, etc). Compare that to a crapbox $10.95 RealTek[a] card which generates interrupts like mad because it has no smarts and you'd probably be very suprised. (Think of comparing a decent hardware modem to a software based WinModem.)

    [a] I had a sales-drone at Computer Boulevard here in Winnipeg just RAVE about RealTek cards. I said I really wanted 3 Intel or 3COM cards for a new work proxy server and he said 'Why? RealTeks are way cheaper and run at the same speed!' Retard.

    --
    Trolling is a art,
  4. Knock-Offs by randomErr · · Score: 4, Insightful

    I give Realtek 6 months tops to make thier own knock-off of the card for $24.95.

    --
    You say things that offend me and I can deal with it. Can you?
  5. I think there is definately a market for this... by Famanoran · · Score: 5, Insightful

    But not necessarily where the vendors think it is.

    Back when I was working at a startup developing anti-DDoS technology, one of the biggest problems we were faced when implemented GigE, was the load on the PCI bus. (This was before we started using PCI-X).

    It depends on exactly how customisable the network card software is, but if you could plonk a couple of those into whatever system you wanted - and if the cards themselves could do, say, signature detection of various flood types, or basic analysis of traffic trends then that is a very definite market.

    I realise the core issue is not addressed (if your physical pipe is full, then you're fucked), but it takes the load of dropping the malicious packets off the host CPU so it can attempt to service whatever valid traffic actually gets through.

    And then there is IP fragmentation. Bad fragments? Perhaps a dodgy fragmentation implementation in the stack? (you know which OS I mean) Lets just drop that before the host sees it and crashes.

    I don't know, I can't find any real information describing what they do, but I can certainly see uses for this.

  6. Re:What good is such a fast Ethernet card... by ScentCone · · Score: 5, Insightful

    if your internet connection is anything less than fiber, which is about 99.9% of all connections? Not to mention the fact that not many computers can actually handle that much data at once anyway

    Listen, when I've got 30 web servers banging away on a single database server, I want each web server in and out as quickly as possible. Every bit of the handshake, query, and results is going to wrap up that much faster if things are faster, period. When you're dealing with a huge data-driven e-commerce site, where every page renders around a hundred or more queries, and there are dozens or hundreds of concurrent page views, this stuff really counts in the aggregate.

    If you sell one more widget per day, all year long, because your web presentation layer is just a little more snappy, that's sure as hell going to pay for a $500 NIC.

    --
    Don't disappoint your bird dog. Go to the range.
  7. A rose by any other name... by Ingolfke · · Score: 4, Insightful

    The name Level 5 refers to the network protocol stack where level 5 delivers data from the network to the application, according to Karr. The company isn't concerned about any potential confusion with Internet Protocol telecom Level 3 Communications Inc. On the contrary, he quipped, "It's working in our favor. People say, 'Yes, we've heard of you. You're a big company.'"

    As lawyers at Level 3 begin salivating at thought of all of the potential lawsuits.

  8. There is a place in an NFS environment by crusty_architect · · Score: 4, Insightful

    We use Filers for storage at Gigabit speeds. Compared to our SAN/FC evironments, we see much higher CPU utilisation on our Sol 8 boxes, especially when attempting to get to Gigibit speeds.

  9. Re:What good is such a fast Ethernet card... by njcoder · · Score: 4, Insightful
    " When you're dealing with a huge data-driven e-commerce site, where every page renders around a hundred or more queries, "

    Each page renders a hundred or more queries? Sounds like you're better off investing in a better design than better hardware.

  10. spend the money on more CPU, not specialized stuff by SuperBanana · · Score: 4, Interesting
    . Intel had the 586-driven smart-cards, and I believe 3Com had them as well. They were intended to offload the CPU by putting parts of the stack on the card.

    You're probably thinking of the i960-based cards, though Intel's PRO series adapters (not i960 based) do something similar (TCP checksumming is now builtin to the chipset and most OS drivers now know how to take advantage of that). That processor, and variants, were used in everything from network cards to RAID controllers.

    They failed because the performance gain and CPU offload numbers were never enough to justify the price difference.

    Ding ding ding. I forget who said it (maybe Alan Cox, but I'm REALLY not sure about that), but the opinion was along the lines that it would always be more benefitial to throw the money at a faster processor (or a second processor etc), because you'd get a performance boost everywhere. $300 buys quite a bit extra CPU horsepower these days, and there's no need for the hassles of custom drivers and such. Nowadays CPUs are just so damn fast, it's also not really necessary.

  11. Um, no. by holysin · · Score: 5, Interesting

    If you have a machine (say on a machine running linux kernel 2.4.20-30.9smp) with a built in gig port (say with eth0 identified as eth0: Tigon3 [partno(BCM95704A6) rev 2003 PHY(5704)] (PCI:66MHz:64-bit) 10/100/1000BaseT) connected to a decent gigabit switch, and another machine (same card, same os)with a gigabit card, those two machines will achieve 940Mbps talking to each other (results via iperf, 0.0-10.0 sec 1.09 GBytes 940 Mbits/sec).

    However, if you plug a windows box (2000 or xp, didn't have a 2003 handy) with either an add on card, OR built in gig (2000 vs xp) you get a rather less impressive figure of 550-630. Coincidentally, you'll get the same basic number if you run two instances of iperf on the same computer... This tells me the bottleneck isn't the PCI bus, it's the OS. If you can prove me wrong please do so...

  12. I worked on TCP offload card at Adaptec by Krellan · · Score: 4, Informative

    Accelerating Ethernet in hardware, while remaining 100% compatible with the standard protocols on the wire, isn't all that new. Just over 2 years ago, I worked on a TOE (TCP offload engine) card at Adaptec.

    http://www.adaptec.com/worldwide/product/prodfamil ymatrix.html?cat=%2fTechnology%2fNAC+Cards%2fNAC+C ards

    It was a complete TCP stack in hardware (with the exception of startup/teardown, which still was intentionally done in software, for purposes of security/accounting).

    Once the TCP connection was established, the packets were completely handled in hardware, and the resulting TCP payload data was DMA'ed directly to the application's memory when a read request was made. Same thing in the other direction, for a write request. Very fast!

    I'm not sure of the exact numbers but we reduced CPU utilization to around 10%-20% of what it was under a non-accelerated card, and were able to saturate the wire in both directions using only a 1.0Ghz CPU. This is something that was difficult to do, given the common rule of thumb that you need 1Mhz of CPU speed to handle every 1Mbit of data on the wire.

    To make a long story short, it didn't sell, and I (among many others) was laid off.

    The reason was mostly about price/performance: who would pay that much for just a gigabit ethernet card? The money that was spent on a TOE-accelerated network card would be better spent on a faster CPU in general, or a more specialized interconnect such as InfiniBand.

    When 10Gb Ethernet becomes a reality, we will once again need TOE-accelerated network cards (since there are no 10GHz CPU's today, as we seem to have hit a wall at around 4Ghz). I'd keep my eye on Chelsio: of the Ethernet TOE vendors still standing, they seem to have a good product.

    BTW, did you know that 10Gb Ethernet is basically "InfiniBand lite"? Take InfiniBand, drop the special upper-layer protocols so that it's just raw packets on the wire, treat that with the same semantics as Ethernet, and you have 10GbE. I can predict that Ethernet and InfiniBand will conceptually merge, sometime in the future. Maybe Ethernet will become a subset of InfiniBand, like SATA is a subset of SAS....