Slashdot Mirror
TCP Equipped Ethernet Card
Josh Baugher writes
"
A 100 megabit ethernet card
with a TCP/IP stack built in. They claim
to be able to do 9 megabytes/second with only 2% CPU load (compared to
4.5 megabytes/second at 98% receiving CPU load using Windows NT TCP/IP
( read about this on "geeks" mailing list.) "
Remember to keep in mind the differences between bits and bytes, while its probably true that you don't _notice_ a performance hit at 900K/s, what you really mean is 900 kilobits/s. That's nearly 80 times less than 9 megabytes/s. So if the kernel spends 1% of CPU time on 900kb then its easy to see why it might spend a whole lot more time on 80 times more data.
My two cents.
oh yeah, interested people should take a look at
D. Clark, V. Jacobson, J. Romkey, and H. Salwen,
An analysis of TCP processing overhead, IEEE Communications, Vol. 27, No. 6, pp. 23--29,
June 1989.
Classic paper! (well, kinda)
You must be *very careful* assessing the load on a Linux system, as the measurement of the load is far from optimal or complete!
/proc/stat are also useless for the kind of tests required in this case, because the load imposed by interrupt handling is added to the process that happens to be running when the interrupt comes in. When the interrupt comes when the idle process is active, the interrupt handling time is counted as idle time.
The "load average" you see displayed in the output of the "w" or "uptime" commands is quite useless in this case, it shows the average number of user processes that are "ready to run", i.e. they would take the processor if it were available. This number in no way reflects the load by interrupt handlers. It is also badly computed, as it includes processes that are waiting for the disk.
(this error was introduced into the kernel long, long ago and I have mailed Linus about it, but he did not want to change it as this figure better reflected the general feel of load on the system, in his opinion. In my opinion, it makes the whole "load average" useless, so I always patch this away when I compile a kernel)
The numbers in
This means that when a test is run that does not use much user-space processing and heavily relies on interrupt handling (like a networking test where a test process sends useless data over a TCP socket! same when it tries to use a serial port), the load results obtained from Linux will be far off the realistic load on the processor.
Rob
Although writting a driver would be fairly easy, this would break all sorts of features that have come into existance due to open source protocol stacks (not being able to do IP chains stuff to an external IP stack is a definite step backwards).
They would either need to open source the stack and make it downloadable from an OSS driver (like some of the SCSI cards out there) or the card will never get within 10' of my boxen.
Somone already pointed out the security implications. Personally, I don't want to be yanking a card in and out of production just becuase someone built the next teardrop and the vendor is slow to fix it.
As for NT, well, this is obviously the tact that MS is pursuing to gain equal performance with other OSS operating systems, but it has certain implications that will keep NT in the second fiddle chair. The card will probably weaken NT security initially by breaking fixes that are covered by current hotfixes and service patches. Additionally, I can think of no better way to fill up somone's disks than by having improved transfer retes across the net, operating system and disks. Worm designers should re-joice as well. Now, you can design worms that can consume more resources without being noticed. If you're really tricky-trick you could design a worm that existed only within the context of the the TCP/IP stack and if the board has NVRAM... well, a box could stay compromised for years. How long before a Microsoft Weapons System sees daylight?
-- "Most decently written TCP/IP stack applications have NO buffer overrun problems" - an anonymous programmer at Fort Mead
-- "ALL TCP/IP stack applications are a long way from being mathmatically correct." -- A mathmatician's retort.
-- "Our job is to find the differences between one and the other and keep this information from the public as long as possible." -- A manager who successfully defused the situation..
Heh, the sites been slashdotted, they should use their own cards or something... :)
Moving the stack into hardware is an interesting idea, though. Unfortunately it has some negative (and admittedly positive) implications for those concerned about security.
First, it will be impossible to tell what operating system a computer is running by using TCP fingerprinting. This is both good and bad in that it will thwart script kiddies to some extent by not revealing the platform, thus making it more difficult to take advantage of well known exploits. On the other hand things like Netcraft and the Internet OS counter will also not be able to take surveys properly.
Second, and entirely negative, is the possibility that their hardware implmentation of TCP/IP may be sub-standard. It may have scads of DOS loopholes and other weaknesses. Unless they make the thing software upgradeable as holes are found, and make the software Open Source, I don't see it gaining much marketshare against the cheap and plentiful cards we have now.
- A.P.
--
"One World, One Web, One Program" - Microsoft Promotional Ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
- A.P.
--
"One World, One Web, One Program" - Microsoft Promotional Ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
I worked with an expensive Intel NIC 9 years ago that had an i960(I think) and an OSI protocol stack on board. Never did any benchmarks, but I'm guessing the complex OSI protocol stack plus wimpy ISA '386 boxes made putting intelligence on the NIC a good idea at the time.
I figure there must be a good reason these things haven't gone mainstream in almost a decade. The proliferation of simple TCP/IP plus faster CPUs might be one reason.
It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. - Abraham Maslow
Of course, I tried to post this to /. a few times and it didn't make it. Oh well. Maybe people will find it appropriate to this thread.
When 3COM first started making Ethernet boards, they tried putting the protocol processing software on a smart Ethernet board. It never worked too well. Unless you had a slow system, with a brain dead operating system or small address space. The processors on the "smart" boards were cheap and slow. The on-board software tended to be buggy, limited and out-of-date. The cards were expensive. You still had to have some sort of light weight protocol for communication between the operating system and the card, adding another layer of software. With well written software, a 25 MHz 68020 host, could run TCP/IP at wire speed on a "dumb" 10 MBPS Ethernet card. The "smart" cards quickly disappeared.
Putting the protocol processing in Silicon will burn you when you need new features and algorithms in your networking stack. What happens if you need large windows, SACK, IPV6, IPSEC, QOS?
The right solution is to use an operating system that doesn't suffer from MBD and use decently designed network cards on a fast bus. 100 MBPS shouldn't be a problem for a decent system. 1 GBPS is where current hardware and operating systems fall down and need improvement.
Mea navis aericumbens anguillis abundat
- There are many hardware patents out there that don't affect us that much, the question is -- is this one of them?
- is the "Silicon TCP" is indeed a step forward for NICs, and if so
- is the patent based on "prior art", and therefore unenforceable?
I would hate to see something that would work well for us regular (e.g., don't have unlimited funds to buy the next new great hardware) folks trapped within a "can only get it from one company" patent....Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
They make it sound like they're using fancy servers.
:) Secondly, 48 megs of RAM seems a bit low for NT. Thirdly, the hard drive systems in there are probably also low quality to say the least.
:), although not a Linux box... and it was a server. I don't call a home system a server that will be pumping 90 megabits/sec.
They're not, they're using two proprietary IBM Aptivas, low end machines. (I should know, I have one of them, the same model they used). Firstly, the E56 has a 266 mhz K6, unless IBM lied to me too
Couldn't they have borrowed Mindcraft's server or something? At least THEY could tune an NT box
This is ESR's cycle of reincarnation in action.
Putting TCP on the ether card should allow some serious savings in CPU time, regardless of OS, if it's bus-mastering. For ftpd, the CPU should (in theory) set up a TCP connection with the card, open up a set of blocks (or a file; I'm no expert on this particular protocol) with a bus-mastering SCSI, and just let the disk flow to the card, or the card flow to the disk. Voila, file transfers that eat only the bus and not the processor.
I think that, in the cycle of reincarnation, we have been going from everything-on-the-processor to everything-on-separate-chips. We have separate video accelerator cards (ignoring the video cards themselves), we have SCSI to drop the I/O load off of the chip. People still haven't folded sound into the processor.
Between all this and the dearth of processor vendors, offloading TCP onto a card makes sense. The less load one places on one's processor, the less tied one is to the processor vendor. Today, people talk about having Intel or not. Imagine a world where people describe their computers by their net card type or memory vendor...
--The basis of all love is respect
The web page describing this NIC is really unimpressive. What I'd really like to know is:
How many simultaneous connections can this thing support, and is it slower when multiple connections are used? How's the performance when one of these cards is hooked up to a machine with a standard software TCP/IP?
Putting TCP/IP in hardware is nice, I guess, but then nasty real-world issues crop up. What happens if there's a bug in the implementation? There's also the nasty challenge of writing a driver for a card like this, but I won't claim that's a defect of this NIC design.
the big push for gig ether is campus trunks between buildings. goes up to 10km (single mode) and Cisco's Gigabit EtherChannel can aggregate up to 16 links. Really great if your org grew strangely, and you have departments in two buildings. It also helps in linking switches in a internet server fanout. Either way, it's more of a backbone technology than anything else. In fact, PCI32's theorietical peak (132MB/s) doesn't match the throughput of full duplex 1000Base-SX. And NT's networking core prevents running the network over 400Mbit peak.
Interestingly enough, on most tested unices, i think they're getting around 800-900Mbit. It'd be interesting to see how fast ftp.cdrom.com would be with gig ether to the backbone... Maybe then i'd see more than 10KBps...
First off, to all those engaged in the "I get more bandwidth than you" pissing contest: try beating 560Mb/s sustained application-to-application bandwidth between two P2/450GX systems (running NT, BTW, but not NT's TCP/IP). So you think you can beat that, eh? Try beating 2us application-to-application latency for zero-length messages. Can't do it, even with buzzwords like VIA and I2O, can you? OK, next topic.
Regarding TCP/IP on the card: as another poster pointed out, this is not a new thing but a very old thing. I once worked on putting DECnet on old 3Com "smart" cards...ick. There are all sorts of problems with doing this sort of thing on the card. First is upgradability of the network stack. You immediately become dependent on the manufacturer for upgrades - don't expect open-source firmware any time soon, even if you had the tools to compile and load it. FPGAs aren't really a good choice here because they increase the component and design costs too much. You'd be much better off using a commodity embedded microcontroller with "firmware" stored in flash memory, although this may still increase the cost unacceptably and for _real_ speed you just plain have to chuck all this stuff out the window and go ASIC. As it turns out, most systems in most uses have more CPU power to burn than any other type of resource. Some guys at HP several years ago took this observation and ran with it; they designed a card that was even more stripped down than the typical Ethernet card, doing even more of the work in software, and they actually got excellent results.
Lastly, the conversation about NT's networking code reminds me of an exchange I had with an engineer at MS a couple of years ago. He was saying that they had to sacrifice a little on TCP/IP features and error checking (e.g. not crashing if sent a source-routed frame, or something like that) to get speed. My response was that (a) not checking unusual conditions in incoming network packets is just unacceptable, and (b) NT's TCP/IP performance is piss poor, indicating that they have bigger issues to worry about than shaving a few instructions by not checking packet headers. In the time since then I have found no reason to change either observation.
Slashdot - News for Herds. Stuff that Splatters.
Exactly, this card is of interest because the Microsoft network stack is terribly slow, and costs an awful lot of CPU time.
The main reason for this is: Poor design. (What did you expect?)
The M$ network stack is (as with most M$ device driver architectures) way more complex to deal with then for example Linux and as a result many of the network drivers are not well written. E.g. they are not optimized for performance. I'm sure the writers are happy if the damn thing works at all.
The network stack itself also has much more involvement with each packet going out or coming in then with Linux. In some cases packets are actually copied in RAM. This is what causes the higher CPU overhead.
We have run several tests and the results where depressing. I think we probably lost the source code, but if I can find it I will post it somewhere.
Here's some of the figures I remember: (all tests where raw network tests, RAM to RAM, no hard drives involved)
Two P90 systems using 100Mb/s Full Duplex (DEC 21140) cards. We where unable to sustain more then 35Mb/s using UDP in one direction only.
Linux on this configuration: close to 100Mb/s.
Using a raw packet driver on a 200MMX notebook with a SMC 100Mb/s Full Duplex card and using remote loopback we are unable to get much more then 15Mb/s sustained. (That's 15Mb/s going out and 15Mb/s comming in)
This was using raw Ethernet packets, no TCP/IP or other protocol.
This same configuration in a normal network is unable to receive more than around 4Mb/s UDP multicast packets, or packets will be dropped (thus not received)
I'm surprised that people have kept up with the poor raw network performance of our Redmondians. It's a disgrace, and I will NEVER use a M$ product if serious networking is to be done. Even if the Ethernet adapter handles the protocols.
Breace.
One thing is becoming obvious.
We need some serious network benchmark tools that are cross-platform usable between Linux and Windows and maybe more.
Strangely enough not too many seem to exist. Neither does someone seem to have some hard data on network performance. It entirely useless to say 'I get 1MB/s using FTP'.
A good network benchmark tool will be able to test raw Ethernet performance as well as performance through protocol layers.
Of course it would only test the network and not use the file system or hard drive. It should be very clear what sort of configuration is supposed to be used: two systems running full-duplex, one system using remote loopback or whatever. It could also be interesting to have a > 2 system test to show what collisions do.
And most important as far as I'm concerned: Open Source. I don't believe in closed source benchmarks.
The difference between raw Ethernet and TCP/IP protocol would show us how badly we need hardware assistance on what platform.
Maybe we should try to port netperf ( www.netperf.org) to Windows and add raw Ethernet to it.
Well, maybe I'll be a bit more serious about this if there's an interest.
Breace.
Hmmm...
When I read the slogan under the Slashdot logo at the top of this page it says "News for Nerds. Stuff that Matters". I don't see "LINUX ONLY" stamped anywhere up there. I don't see anything wrong with the posters giving us articles concerning platforms or OSes other than Linux and the hardware it runs on. In fact even though I'm quite a Linux supporter, I'm sick of the people who think this must be a Linux only site. My understanding is that Linux is on the minds of the nerds at this point of history so we get alot of stories about Linux (if I'm wrong here, well, sorry... somebody should make that clear). That's cool, but why are the posters slammed when they post an article about something other than Linux? That's not cool, or uncool, or whatever.
bah