Mixing Gigabit, Copper, and Linux

iampgray writes: "With copper-based gigabit cards selling for less than $36 these days, what kind of performance can you expect -- especially in the often-overlooked Linux market? We sought out to test exactly what you could expect from copper-based gigabit solutoins for the desktop interface through the cluster-targeted products. Name brands and off-brands were put through the wringer. How'd they fare? Interesting results to say the least."

Comments

I was able to get 800MB/sec on a Linux 2.4 kernel

[Anonymous Coward]

Between 2 Tyan/AMB 1.2Ghz machines I was able to pull 800 mega bits per second on old copper.

Huh.

[autopr0n]

Are we even to the point when a normal PC could handle Gigabyte? And if so, why not use optical? I mean, saying I've got a fiber optic home network is a lot cooler then saying I've got a gigabyte eth home network. I mean to a geek, (to anyone else, that would just be lame... er...)

How much more expensive is the optical stuff for GigE? I'm mostly using optical audio connections from my home sterio, and that's not to much money

Re:Huh.

[bleckywelcky]

Gigabit, not GigaByte. Gigabit = 1000000000 bits. GigaByte = 8000000000 bits. A 1 GBps connection is 8 times faster than a 1 Gbps connection.

Re:Huh.

[Gojira Shipi-Taro]

Well... copper is cheaper than fiber for the moment. I'd hate to think what my 50 meter run from my router to the second floor of my townhouse would cost if it was fiber.

I use optical runs for my audo as well, but those are all under a meter, for the most part, and around $30 or so a piece. Not too much money for the purpose, but I dont' think I'd enjoy paying for a 50 meter run. Never mind the cost of devices with optical interfaces.

That said, I guess the only reason I'd consider GB copper is that it's no more expensive than 100 base-T...

Re:Huh.

[WolfWithoutAClause]

> Are we even to the point when a normal PC could handle Gigabyte?

Yes. Some memory parts are 333 Mhz and are 4 bytes parallel and instructions/s (as opposed to the clock rate) is over 1 GIP I think. So a PC can just about knock out a gigabyte/s if it has to, but it hasn't got much time to think about anything.

But this article is talking about gigaBITs/s. That's 8x slower. So that too.

Re:Huh.

[Christopher Thomas]

Are we even to the point when a normal PC could handle Gigabyte? And if so, why not use optical?

A 32-bit 33 MHz PCI bus can support one (1) gigabit ethernet card at full capacity (card's bandwidth is about 100 Mbytes/sec, PCI 32/33 is 133 Mbytes/sec).

If you want to stick multiple cards in (e.g. for a small hypercube-style cluster), buy motherboards that support 64/33 or 64/66 (I was drooling over the dual-processor 64-bit-PCI AMD boards a little while back).

Gigabit ethernet over copper has the advantage of running over your existing cabling (i.e. cat-5 is fine). This avoids having to muck about with fiber, as fiber is a PITA to maintain yourself (getting optically perfect connections for the fiber jacks is picky).

The way gigabit ethernet is encoded on cat-5 cable is both sneaky and elegant.

Re:Huh.

[megabeck42]

>> Gigabit ethernet over copper has the advantage of running over your existing cabling (i.e. cat-5 is fine). This avoids having to muck about with fiber, as fiber is a PITA to maintain yourself (getting optically perfect connections for the fiber jacks is picky).

Actually, the siecor unicam series work really, really well. They use a index of refraction matching gel inside the factory polished terminators. All you have to do is cut'n'crimp. They work great. I haven't ever had to do any splicing though - but, given how well the siecor stuff works, I can't see it being a remarkable problem.

Re:Huh.

[WhaDaYaKnow]

A 32-bit 33 MHz PCI bus can support one (1) gigabit ethernet card at full capacity (card's bandwidth is about 100 Mbytes/sec, PCI 32/33 is 133 Mbytes/sec).

Almost. 133MBytes/sec = 1064Mbit/sec. This means that it could only in theory keep up, if all bandwidth on the PCI bus was available for data. But, this number includes the overhead of setting up transfers and arbiting for devices that want to transfer, and these operations are fairly expensive.

Also, the PCI device needs to obtain descriptors etc. (which indicate where to put the Ethernet packets in RAM) over the same bus, costing more valueable cycles.

If you did have only the one device on the PCI bus (which is very unlikely), with a good chipset, you'd probably get over 100MB/sec, but not much more. So you'd never be able to actually get full Gb Ethernet. (as the test results show, things are MUCH worse then this, but this is probably caused by multiple devices on the PCI bus).

Talking about chipsets, a long time ago we had a board with an OPTi chipset. They ran out of silicon when designing the chip, so they couldn't implement the Bus Master FIFO, so they decided to just abort every BM cycle after each 32 bit transfer, yielding a max transfer rate of 4MB/sec!! For weeks, I couldn't figure why my network driver wouldn't send packets faster than 30 Mbit/sec, until my boss flew to California (where OPTi was located) to find out what we where doing wrong.

Back to the tests: for some reason they failed to mention the chipsets used on the motherboards which really is VERY important if you want to use a gigabit Ethernet card in a 32/33 PCI system. The fact that the Dell has 5 PCI slots probably means that it has an integrated PCI-PCI bus (not many chipsets support 5 PCI slots, unless one or more of them do not support Bus Mastering), which would certainly not improve things.

I think this is important to mention, because most systems today, at least desktops, will only have 32/33 PCI, and as the test results show, with a presumably shitty chipset, you only get marginally better performance than 100Mb Ethernet...

Re:Huh.

[drmofe]

Err,, yes. In data communication, it's base-10. In data storage, it's base-2. Therefore 1Gbps = 1, 000, 000, 000 bits per second. 1Gbyte on your hard disk or any other storage medium is 2^30 or 1073741824 bits. Why? Different standards. 1024 is just close enough to 1000 that computer scientists adopted K as their measure for 2^10 or 1 kilobyte. Scientists and engineers work in base-10, hence the reason why data communications standards use that. This is the same reason that some of us call hard drive manufacturers unethical for quoting storage capacity on hard drives using data communications standards. STF

Re:Huh.

[dakoda]

however, these cards plug into a pci slot, not a memory slot. pci is still 66mhz max (unless you oc it). so even if your memory could dump over a gigabyte per second (which it most likly can, by a factor or two or more these days), you can't sent out all that directly to the nic.

Now, if they were to make nics that plugged into agp slots, it might be a little different (they are clocked higher and transmit more data per clock), but iirc it's still slower than the memory bus.

so we clearly need a nic that plugs into a dimm slot to utilize all this bandwidth =)

Re:Huh.

[pyite]

You're paying for termination. Fiber is not very expensive. A meter of MM zipcord is about $1. So let's say you're paying $29 for termination. A 50m cable would be maybe $79.

Re:Huh.

[Fnord]

The fastest pci gets is 66mhz 64bit. Thats 64 bits per clock cycle, 66M clocks per second....4.224 Gigabits. I'd say thats a little higher than 1 Gb.

Re:Huh.

[dakoda]

very tru. i gotta stop posting when half asleep. my math is so bad =(

Re:Huh.

[fbw]

If a gigabit network card can supply a data stream of roughly 125 megabytes per second, and the pci bus is able to sustain 133 megabytes per second...

You are still forgetting that this data of 125 megabytes goes off the PCI bus, is processed by the CPU (the OS handles network and disk I/O), then goes back onto the PCI bus towards either another network card perhaps, or the storage controller, either IDE or SCSI, which rest on the very same PCI bus.

So your 133 megabytes/second is actually shared, so even without calculating in the overhead that exists, the maximum speed possible is half of that, which would amount to 66,5 megabytes/second.

You simply need a 64bit and/or 66mhz PCI bus to take *full* advantage of a gigabit card, but it is almost always faster than a 100megabit card.

Re:Huh.

[Kizeh]

First off, building a home network and building an enterprise network are quite different things. In the enterprise arena, fiber isn't that much more expensive cable cost wise. It's still more expensive labor-wise, but proper Cat-5(e) installation isn't cheap either. The problem with fiber has been the price of electronics, which have been prohibitively expensive.

Secondly, switches are currently the main problem. We can get gig cards cheap, and in fact many servers come standard with them. However, finding switches that offer any kind of density of gig copper ports at a sane cost seems to be impossible. Even if there were such cards available for, say, the Cat 6500 platform, the problem is that practically none of the enterprise class switches currently in use has a backplane capacity sufficient to handle that kind of traffic. Further, aggregating a switchful of gig connections is a problem. 10 gig is the only realistic solution (Sonet/ATM is way expensive at high speeds, and not well suited for switched LANs to begin with), but it's hideously expensive, and not quite there yet standards-wise or equipment-wise.

Finally, a few notes on what I mean when I say enterprise class switch: modular, chassis based, build with line cards. Completely hot-swappable, with full redundancy of critical components. Support for large numbers of MAC addresses (16k), QoS, VLANs and VLAN trunking (with more than 200 concurrent VLANs), configurable spanning tree, security features, very thorough remote management features, good vendor support so replacements can be had (guaranteed) within 24 hours or faster if something breaks. Most of these things don't matter for the average nerd, but when you build a network with hundreds of switches, the requirements suddenly change, and usually you pay for it.

I didn't even notice 1000bT was so cheap...

[Geek In Training]

I checked out the cards, and yes you can get them cheap, but what about switches? You figure they're still uber-pricey too, right?

Nope... apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150! (I've been most happy with my D-Link DI-804 Router/firewall/switch for $79.)

Is this the normal "cheaper as tech gets more widespread and easier to manufacture," and do you think maybe Apple making gigabit ethernet a standard feature had something to do with it? :)

Re:I didn't even notice 1000bT was so cheap...

[jhunsake]

Are you sure all 8 ports support gigabit? Most likely it has a gigabit uplink port and the rest are 10/100. The switches with all ports supporting gigabit are expensive.

Re:I didn't even notice 1000bT was so cheap...

[fatphil]

You wouldn't be confusing a D-Link 9-port 8-port 10/100 1-port giga switch for $150 with a D-Link 8-port giga for $600 would you. If you aren't, then the site that's selling the 8-port giga for that price is going to change its prices (upwards) _very_ soon, when they realise their mistake.

FP.

Re:I didn't even notice 1000bT was so cheap...

[Graymalkin]

Having a couple hundred thousand Macs with gigabit ethernet probably did have something to do with it. Many vendors now offer gigabit ethernet NICs in their professional series systems which means for network infrastructure folks there's a high demand for equipment leading to increased production and a lower cost. Apple's been selling gigabit ethernet standard for almost two years now which amounts to lots of gigabit ethernet cards floating around in Macland.

Re:I didn't even notice 1000bT was so cheap...

[Fastolfe]

10 100Mbit clients can pull 100Mbit each through 1 Gbit uplink port. With only a 100Mbit uplink port, one client trying to pull all 100Mbit through it will thus degrade service for all the other clients.

Re:I didn't even notice 1000bT was so cheap...

[ncc74656]

Nope... apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150!

D-Link's site is nearly impossible to navigate (maybe it requires JavaScript, which I've shut off), but the Pricewatch description of the DES-1009G indicates that Gigabit Ethernet is only available on one port as an uplink connection; the rest of the switch is your run-of-the-mill 10/100 job. The DGS-1008T is D-Link's 8-port unmanaged 10/100/1000 switch; the cheapest entry on Pricewatch for that is $595.

BTW, I have the entire site downloaded. Maybe I'm insane to even think about mirroring a /.'ed article on my home cable-modem link, but here it is. I've converted all the charts to PNG so they'll load slightly faster, and I got rid of most of the godawful "super-31337" yellow-on-black text to improve readability. You can also choose this link to download the entire page (images and all) in one shot.

Re:I didn't even notice 1000bT was so cheap...

[crudeboy]

I'm assuming that you would uplink to a server with a 1Gb card

Or use the uplink to connect to a backbone network. That's really what the uplinks are there for.

Re:I didn't even notice 1000bT was so cheap...

[Transcendent]

If you followed the links on pricewatch, you'd see that only ONE port is a gigabit port.... and even then it's probably an uplink port...

Keep dreaming for you l33t gigabit lan party...

Re:I didn't even notice 1000bT was so cheap...

[Transcendent]

Maybe I'm insane to even think about mirroring a /.'ed article on my home cable-modem link

Yea, you're pretty f*ckin insane... heh...

well at least its not a link on the main page ::phew::

Re:I didn't even notice 1000bT was so cheap...

[DrSkwid]

50+3+1-1+2-1=49?

50 + 3 = 50
50 + 1 = 50
50 - 1 = 49
49 + 2 = 50
50 - 1 = 49

It doesn't matter how much water you pour into a pint glass, it can only hold a pint. This is what an upper bound is. jeesh, get over it

Hoboy.

[Soko]

How'd they fare?

Not terribly well against the /. effect.

Interesting results to say the least.

Lessee, a story about increasing bandwidth on a server /.ed to oblivion? That's not interestng, that's anti-climatic - I know what happens before I get to the story. Oh well...

Soko

Re:Hoboy.

[digitalunity]

anti-climactic, I hope you mean. anti-climatic? Is that some sort of freezer?

Has anybody read the article yet?

[BeBoxer]

Could you post a summary? That must be about the fastest /.-ing I've seen. What'd that take, about 5 minutes?

Re:Text of Article (second section)

[__aanonl8035]

D-Link DGE-500T

D-Link DGE-500T was the first of the gigabit cards tested. This card is based on SMC's dp83820 chipset and is designed for a 32bit bus. The chipset in this card turned out performance nearly identical to the two Ark cards and the GigaNIX cards tested in our test suite, since all utilize the dp83820 chipset from SMC. The Linux driver used was the ns83820 as included in the 2.4.17 kernel. Latency on both platforms was .0002 seconds.

Peak throughput while operated in a 32bit bus was 192.21 Mbps. This was achieved in the Dell systems. The Athlon systems only obtained a peak of 172.21 Mbps when these cards were inserted into the 32-bit bus. Both systems show a slight drop in throughput but eventually level out. Peak throughput while operated in a 64bit bus running at 33Mhz was 315.96 Mbps.

When the bus was jumpered to autoselect 66/33Mhz, the performance increase was negligible. Peak throughput was 316.40 Mbps. Comparing the plots of the 66Mhz and 33Mhz run reveals that they are essentially identical.

For complete testing results, click here.

Price: $45

The cost per Mbps is as follows:

32bit 33Mhz: $45 /((192.21+172.21) / 2) = $.25>

64bit 33Mhz: $45 / 315.96 = $.14

64bit 66Mhz: $45 / 316.40 = $.14

Ark Soho-GA2500T

The Ark Soho-GA2500T is also a 32-bit PCI card design. Like the D-Link DGE-500T and the Asante GigaNIX cards, this card is based on the SMC dp83820 chipset. With that in mind the performance was estimated to be close to the D-Link DGE500T. The driver used was the generic ns83820 included the 2.4.17 kernel. The latency for both test systems was .0002 seconds.

The peak throughput achieved while in a 32bit 33Mhz bus was in the Dell system: 192.62 Mbps. While the Athlon system in the same bus setup only reached 172.19 Mbps. As before, there is a performance drop at the 1Kb and 5-10Kb packet sizes.

Peak throughput while operated in a 64bit bus running at 33Mhz was 610.83 Mbps and 609.98 Mbps when running at 66Mhz respectively. As with the Soho-GA2000T, there is no noticeable difference between a 33Mhz and a 66Mhz bus.

For complete testing results, click here.

Price: $44

The cost per Mbps is as follows:

32bit 33Mhz: $44 / ((192.62+172.19) / 2) = $.24

64bit 33Mhz: $44 / 610.83 = $.07

64bit 66Mhz: $44 / 609.98 = $.07

Ark Soho-GA2000T

Our transition into cards designed for a 64-bit PCI bus began with the Ark Soho-GA2000T. Like it's 32-bit counterpart, this card was designed around the ns83820 chipset, which will allow us to examine the performance benefits, if any, in moving from a 32-bit As

Designed to run in a 64bit 66Mhz slot, this card is backwards compatible to 32bit and 33Mhz slots. This card is based off of SMC's dp83820 chipset so performance was expected to be similar to the DGE500T and the Soho-GA2500T. The driver used was the generic ns83820 included in the 2.4.17 kernel. Latency was .0002 seconds on both test platforms.

Peak throughput for a 32bit 33Mhz slot was 189.93 Mbps in the Dell system. The Athlons were only able to reach 172.26 Mbps.

Peak throughput for 64bit 33Mhz was 665.06 Mbps with an MTU of 6000. Peak throughput while running at 66Mhz was 640.60 Mbps. With the exception of the 6000MTU tests, there is no noticeable difference between bus speeds of 33 and 66Mhz.

For complete testing results, click here.

Price: $69

The cost per Mbps is as follows:

32bit 33Mhz: $69 / ((172.26+189.93)/2) = $.38

64bit 33Mhz: $69 / 665.06 = $.10

64bit 66Mhz: $69 / 640.60 = $.11

Re:Text of Article (third section)

[__aanonl8035]

Asante GigaNIX

The second 64bit card tested was Asante's Giganix. This card is designed for a 64bit bus but, is backwards compatible to 32bit and 33Mhz configurations. Giganix is based off of the dp83821 chipset. The driver supplied by Asante was unable to compile due a bug in the code. In order to get the card to work the generic ns83820 driver was used again. Performance was expected to be similar to the GA2000T. Latency was .0002 seconds on both systems.

Peak throughput for a 32bit 33Mhz configuration was 238.75 Mbps in the Dell systems, with a peak of 172.19 in the Athlons. When comparing to the GA2000T, the Athlon results stay about the same whereas the Dell systems increase by 50Mbps.

Peak throughput for 64bit 33Mhz 641.02 Mbps with an MTU of 6000. When running at 66Mhz, the peak is 651.51 Mbps with the MTU at 6000.

An interesting spike in throughput on the 64bit 66Mhz tests was when the MTU was set to 3000. Aside from the 40Mbps difference between the two bus speeds, the plots look very similar. The main difference is the spike at 8KB packets.

For complete testing results, click here.

The cost per Mbps is as follows:

32bit 33Mhz: $138 / ((238.75+172.19) / 2) = $.67

64bit 33Mhz: $138 / 641.02 = $.22

64bit 66Mhz: $138 / 651.51 = $.21

Syskonnect SK9821:

The first of the Syskonnect cards tested was the SK9821. This card is designed for a 64bit bus. The SK9821's are backwards compatible to 32bit and 33Mhz configurations. The driver used was sk98lin from the kernel source. Latency was .000048 on the Dells and .000025 seconds on the Athlons. Of all the 64bit cards tested, the SK9821 is the first to have a noticeable difference in performance between the two bus speeds.

Of all cards tested, the Syskonnect SK9821 gave the most consistent throughput over all packet sizes, and was far-and-away the overall performance leader.

In the server-class testing environment, peak throughput in our 64 bit 33Mhz setup was 782.27Mbps with the MTU set to 9000. The peak for 66Mhz tops off at roughly 940Mbps with jumbo frame MTU sizes of 6000 and 9000.

Peak throughput on 32bit 33Mhz was 365.27 Mbps on the Dells. After the peak, is reached there is a noticeable drop in throughput as it levels off to the 330Mbps range.

For complete testing results, click here.

Price: $570

The cost per Mbps is as follows:

32bit 33Mhz: $570 / ((365.27+163.97) / 2) = $2.15

64bit 33Mhz: $570 / 782.27 = $.73

64bit 66Mhz: $570 / 938.97 = $.61
Syskonnect SK9D21:

The second card tested from Syskonnect was the SK9D21. The SK9D21 is aimed at the desktop/workstation market. While support for this card under Windows environments appears to be solid, there were too many technical issues. The testing environment's mix of kernel, motherboard, Athlon chipset, and Syskonnect drivers made for too many components to successfully debug the problems with this card thoroughly. This card is designed for a 64bit bus the card is backwards compatible with 32bit and 33Mhz configurations. While an exhaustive analysis of the cards was unavailable, it should be noted that the latency was successfully determined at .000123 seconds.

Our difficulties with this card were limited to the 64-bit bus. Our tests were successful in analyzing the performance in both the QLITech Linux Computers Athlon-based systems and the Pentium-based systems in 32-bit busses.

When drivers issues for this card are resolved, performance evaluations in this section will be amended.

Peak throughput in the Dell system was 377.53 Mbps. As with the SK9821, there is a drop off after the peak is reached.

For complete testing results, click here.

Price: $228

The cost per Mbps is as follows:

32bit 33Mhz: $228 / 377.53 = $.60
3Com 3c996BT:

The next card in the test suite was the 3Com's 3c996BT. This card is designed as a 64bit 133Mhz card, but is backwards compatible to 32 bit, 33 and 66Mhz configurations. The driver used was the bcm5700, version 2.0.28, as supplied by 3Com. Latency was .000103 in the Dells and .000078 in the Athlons.

The peak throughput achieved in this card while in a 32bit 33Mhz slot was 436.23 Mbps in the Dell systems. In the Athlon system, the same bus configuration only reached 184.02 Mbps.

Peak throughput while running in a 64bit 33Mhz slot was 884.09 Mbps this was with an MTU of 4000. While running at 66Mhz, the peak was only 546.16 Mbps with an MTU of 6000. These plots are all relatively smooth when compared to the other plots for this card.

Performance in a 66Mhz slot is actually lower for all MTU sizes as compared to a 33Mhz slot.

For complete testing results, click here.

Price: $138

The cost per Mbps is as follows:

32bit 33Mhz: $138 / ((436.23+184.02) / 2) = $.44

64bit 33Mhz: $138 / (884.09) = $.16

64bit 66Mhz: $138 / (546.16) = $.25

Intel Pro 1000/XT:

The final 64bit card tested was Intel's E1000 XT. As with the 3c996BT this card is designed for future PCI-X bus speeds running at 133Mhz. It is compatible with a variety of configurations running at 33 and 66Mhz as well as 32bit. The card uses Intel's e1000 module, version 4.1.7. Latency in the Athlon systems was .000091 seconds. Due to time constraints, we have yet to test this card in the Dell testbed.

Peak throughput achieved was 743.14 Mbps while running in a 64-bit 66Mhz slot with the MTU set to 9000. Performance in a 32-bit configuration turned out the lowest throughput for all cards tested coupled with the most erratic throughput. During the throughput tests, the card would drop 100% of packets for extended lengths of time. Initial testing in the 64-bit setup showed performance similar to the Giganix card with regards to a 64-bit bus. Once the MTU was set to 9000 performance became very erratic, stagnated several times, then stabilized once the packet size reached an upper threshold peak. Note that the drop in performance was not associated with the (expected) phenomena of packet reassembly when the TCP packet size exceeds the MTU.

As testing continued the the 66Mhz phase things only got worse. Once the MTU exceeded 3000, performance was no longer predictable. During the 4000 MTU tests, the throughput plummeted to around .4 Mbps for several TCP packet sizes. At an MTU of 6000 and at 9000 the same problem occurred as before in the 64-bit 33Mhz test.

For visual clarity of this phenomena, see the ''Complete Test Results'' link for the Intel Pro 1000/XT below.

For complete testing results, click here.

Price: $169

The cost per Mbps is as follows:

32bit 33Mhz: $169 / 142.02 = $1.18

64bit 33Mhz: $169 / 624.41 = $.27

64bit 66Mhz: $169 / 743.14 = $.22

Re:i'm sorry

[WolfWithoutAClause]

Mutually Exclusive? Nope, just limited range. Fiber can go 1000km or more.

Still, sometimes you only want to go a few feet between two servers or something and there you can't really argue with the price.

Obligatory Mac Plug

[Lally Singh]

Just fyi, Macintosh 1000BaseT ethernet controllers go directly to the memory controller, bypassing PCI altogether..

Re:Obligatory Mac Plug

[Daniel Wood]

I find this even more interesting:

Note: When connecting a Power Mac G4 computer directly to another computer without using an Ethernet hub, a crossover cable is not required; circuits in the PHY detect the type of connection and switch the signal configuration as required.

Re:Obligatory Mac Plug

[Lally Singh]

This feature has saved me many a time on my PowerBook G4. When it works & nothing else does, I've got the wrong cable! :-)

Re:Obligatory Mac Plug

[Skuld-Chan]

Yeah - my rca cable modem does that too - seriously!

Re:Obligatory Mac Plug

[KILNA]

Lack-Of-Vavcabulary Nazi?

Re:Obligatory Mac Plug

[NevDull]

It's not uncommon. It's called Auto MDI-X.

Clusters

[jhunsake]

Stay away from cards that don't have PXE and cards in which the driver won't compile into the kernel (as opposed to a module) if you plan to do easy installations or mount root off the network. In other words, stay away from Netgear and some 3Com cards (I haven't tested others), and play it safe with Intel.

Re:Clusters

[jhunsake]

Yes, we use these in our cluster right now, I said if you want PXE and drivers that compile into the kernel, then avoid them.

Re:Clusters

[klui]

But according to the article, the Intel G-bit card dropped 100% of the packets in spurts for no explanable reason. I'd stay from Intel cards for now.

Re:Clusters

[PSC]

Stay away from cards that don't have PXE

Gross overgeneralization. Since 1000BastT switch ports are still pretty expensive, you will most likely only equip servers with Gigabit/copper. And servers don't usually boot off the net.

stay away from [...] 3Com cards

Our 3com 996T worked pretty well with Redhat 7.1. Just get the driver from 3com, the vanilla kernel
sources and off you go.

Even more comfortable with D-Link DGE-550T which has a driver in current kernels.

BUT BEWARE: both 3com and D-Link won't work on at least the early Dual Athlon Tyan ThunderK7 boards.

Re:Clusters

[squeegee_boy]

>>...I'd stay from Intel cards for now...

...under Linux, with that particular version of the drivers.

We use the Pro/1000T cards exclusively (when we need GigE) around here under Windows. W2K exhibits no such behaviour, and nicely tops out at 796kbps on a memory to memory copy between 2 Dell PE2500's using a 4K MTU.

The article (as of Monday) didn't have the scores for this card under any other OS, at least not that I saw.

Re:Clusters

[squeegee_boy]

Woops, that should be 796Mbps.

Re:Clusters

[jhunsake]

I was specifically targeting my comments to clusters in which the nodes boot and/or mount root off the network. Next time, pay attention.

Gigabit and Linux

[GigsVT]

Well, check out the docs first off. It's hard to get much out of GBit, since most of the utilities don't call the socket open with properly sized buffers/window/whatever.

I set up optical gigabit for some NAS type things at work, and out of the box, GBit performed maybe 30% better than 100 Mbit. We are talking about 110Mbit peaks, compared to 80Mbit peaks with 100Mbit switched.

Setting the MTU to 6144 (max that I could set it to with the ns83820.o) I started to get peaks around 300Mbit/sec.

I tried recompiling the module for higher limits, since in the source it has:

#define RX_BUF_SIZE 6144 /* 8192 */

But if I put in 8192, or 9000 like I wanted it to be, it would crask or lock up.

Anyway, it's not trivial to get good performance out of GBit, and definitely don't expect anywhere near 10X gain.

Re:Gigabit and Linux

[GigsVT]

a) I wrote my comment before I read the report because I thought the site was slashdotted, but it did eventually load.

b) The only card they got 900Mbps out of cost $570, the lower cost ones got no where near that, especially with 1500 MTUs

Also, I forgot to mention, even if your switch can handle jumbo frames, if you use UDP for anything, trying to talk to a 1500MTU host from a jumbo frames host in UDP won't work (i.e. NFS).

Re:Gigabit and Linux

[tcc]

It's all about the card, cheap cards will perform terrible, like I posted a bit lower, the Dlink cards are TERRIBLE, but if you give a shot to an Intel Pro 1000T, these are the best cards on the market for gigabit ethernet over copper. 3Com is also good, but with my dlink cards I was getting HALF the bandwidth that I would get with my pro1000T.

Re:Text of Article (fourth section)

[__aanonl8035]

Comparisons and Observations

In this section, we compare performance differences between cards in like environments , provide some general performance observations, and examine the cost per megabit as determined by the operating environment.

Head-to-head throughput results

While the results obtained in this study clearly show that peak performance is not a complete indicator of peak performance, in this section we examine the peak performance results amongst all cards under common environments.

32-bit, 33MHz PCI Bus, 1500 MTU
64-bit, 33MHz PCI Bus, 1500 MTU
64-bit, 66MHz PCI Bus, 1500 MTU
64-bit, 33MHz PCI Bus, 3000 MTU
64-bit, 66MHz PCI bus, 3000 MTU
64-bit, 33MHz PCI bus, 4000 MTU
64-bit, 66MHz PCI bus, 4000 MTU
64-bit, 33MHz PCI bus, 6000 MTU
64-bit, 66MHz PCI bus, 6000 MTU
64-bit, 33MHz PCI bus, 9000 MTU (Note: Drivers for the dp83820 chipset were limited to around 6000 MTU)
64-bit, 66MHz PCI bus, 9000 MTU (Note: Drivers for the dp83820 chipset were limited to around 6000 MTU)

General Observations
Of the eight cards tested, the clear performance champion was the SK9821 with regard to throughput and consistency. The 3Com 3c996BT has a modest price tag and respectable performance for the entry-level server configuration. If price per megabit is the main concern, the Ark Soho-GA-2500T has the lowest cost per Mbps, making it a viable solution for entry-level systems requiring higher throughput than fast ethernet.

The D-Link DGE500T and the Soho-GA2500T show nearly identical peaks, which is to be expected since the drivers and the chipsets were the same.

The 3Com 3C996BT has results when compared to the 64-bit 33MHz results were surprising inasmuch as these cards showed better performance at 33MHz bus than at the higher 66MHz bus.

Of all of the cards tested, the Intel E1000 TX proved to be comparable to the comparable to the Asante GigaNIX card in peak performance, but the erratic overall performance proved too much to overcome.

In referring to the Complete Test Results sections for the 3C996BT and the SK9821 cards, one sees a very consistent and smooth transition to the peak throughput of the cards over the complete range of packet sizes.

Some general comparisons that can be derived from the above results include the notion of ''cost per peak megabit. Depending upon the environment that the network device is to be installed, the cost per peak megabit varies greatly. For example, if one would wish to upgrade their P-III-based desktop system with a 32-bit, 33MHz PCI, the GA25000T is the clear cost-effective solution, but would not be able to provide throughput at the level of the 3Com 3C996BT.

In an HPC environment, where sustained throughput is critical and the switch is capable of Jumbo frames, the SK9821 would be the best performer. In light of gigabit switching hardware that lacks Jumbo Frame support, a comparison of the 1500MTU results shows the SK9821 is still a viable choice, as is the 3Com 3C996BT which provides a more cost-effective solution.

Paul Gray
323 Wright Hall
University of Northern Iowa

Switches aren't cheap.

[Christopher Thomas]

apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150!

These are for 8x100-base-T with a gigabit uplink. I researched this a while ago, when speccing out my dream network ;).

The cheapest full-gigabit switch D-link sells is about $1500.

Re:Switches aren't cheap.

[Bobzibub]

One could go "token ring" style with 2 cards per machine and the network being a ring not star.
As long as it is a small network and no machine goes off line it should be ok.

Perhaps an expert knows if you could have two virtual IP's per card, a simple "Y" splitter plus two cross-over cables running into each machine via the splitter?? That might be a cheaper ring configuration.

Cheers,
-B

Re:Switches aren't cheap.

[Sycraft-fu]

One could go "token ring" style with 2 cards per machine

That would give real below average performance, even if you did get it to work. IT would be a nightmare getting all the routing setup properly so the computers would pass packets around the ring correctly. And once you got it working, speed would suck. Every packet would have to be evaluated in software and then routed to the next interface if necessary. This would be slow as hell, neurtalising the speed advantage.

Perhaps an expert knows if you could have two virtual IP's per card, a simple "Y" splitter plus two cross-over cables running into each machine via the splitter??

That would work like not at all.

Re:Switches aren't cheap.

[nelsonal]

Dell has an eight port Gb switch for about 500 not too long ago. I think they are loss leading their networking stuff to get you to pay up for their more expensive severs and storage. Here is a link so you can spec it yourself. You have to drop the support option to 1 year and shipping is extra but its an 8 port gigabit switch over coper.

Re:Switches aren't cheap.

[stickyc]

So I could run gigabit to my cable modem for better downloads, ya?

Heh.

[autopr0n]

I knew that actualy, and even flamed someone for typing "MB" when he meant "Mb". Oh well...

Re:Text of Article (fourth section)

[Slash Veteran]

you, sir, are taking a huge slurp from the karma tit today. Congratulations to you! Cheerio.

$30 for 1m!?

[autopr0n]

Wow, you are paying way to much. a 12ft optical cable I had used to connect my PC to my sound system broke a couple of days ago, and I thought it was gonna cost me $40 to replace it. Radio shack sold em for $44, but sears had a 12footer for just $20.

Re:$30 for 1m!?

[Gojira Shipi-Taro]

Cool, thanks. I'll look around more next time. I still think it's more expensive than copper, I'd love to find out differently.

switches?

[scharkalvin]

The same site lists several 8 port switches for gigibit copper. Those with ONE 1000mhz port and 8 10/100 are low cost ($150) but those with 8 1000mhz ports are a bit more (about $600). Add the cost of the switch to your cards and it's probably not cost effective for the HO yet. I'm happy with my 100base T network, my 8 port switching hub was less than $40. I AM using CAT 5E so I can upgrade to 1000baseT someday, just not today!

Re:switches?

[DarkEdgeX]

I'd be happy with an 8 + 2 switch from someone-- i.e. 8 10/100 ports with 2 10/100/1000 ports for my main file sharing boxen (and I imagine these would be a hit at LAN parties, so the server running the game could have a gigabit connect to the switch, allowing most of the 10/100 connections to saturate it with updates (and vice versa)). The trend of hardware makers (Netgear has done this too, FS309T is an 8 port 10/100 with a SINGLE 10/100/1000 copper gigabit port) to make these 8 + 1 solutions just sucks (since you can't really test the faster speed of gigabit with JUST one port).

Of course in a perfect world, I'd agree that 8 port gigabit switches being $200 or less would be about near perfect, especially if higher port counts weren't unrealistically high.

Cheap NICs, costly switches

[Jah-Wren Ryel]

The cards are well priced for home use, and CAT5E cabling is cheap too. The problem with gigabit ethernet is not the cards, it is the lack of switches or even plain hubs at an affordable price point. There are lots of switches out there with a single gigabit port, but even those are a couple of hundred dollars. If you want multiple gigabit ports, you are looking at more than $600 for the bottom rung products.

Re:Cheap NICs, costly switches

[Geek In Training]

The cards are well priced for home use, and CAT5E cabling is cheap too. The problem with gigabit ethernet is not the cards, it is the lack of switches or even plain hubs at an affordable price point...

Funny, this gets modded up as Informative, while my earlier post listing inexpensive 8-port gigabit switches languishes unmoderated.

Slashdot moderation, yet another mystery of the universe. Even after reading the guidelines twice, I can't figure out how other people manage to interpret them the way they do.

Re:Cheap NICs, costly switches

[Geek In Training]

Please disregard the parent post. It is WRONG, the hubs/switches are not cheap, I have been edumacted by my peers. :)

The flogging will happen later, when my karma gets updated.

Re:Cheap NICs, costly switches

[Luminous Coward]

The problem with Gigabit Ethernet is not the cards, it is the lack of switches or even plain hubs at an affordable price point.

To quote from Ethernet: The Definitive Guide (February 2000):

Currently, all Gigabit Ethernet equipment is based on the full-duplex mode of operation. To date, none of the vendors have plans to develop equipment based on half-duplex Gigabit Ethernet operation.

Which would explain why there are no Gigabit Ethernet hubs available (hubs aka repeaters are half-duplex devices). Carrier extension and frame bursting are not needed in full-duplex mode, which would make the design of full-duplex devices simpler, I guess.

On a side note, in the article, they used Ethernet Jumbo Frames which were not part of the official IEEE standard as of the writing of the book.

Re:Cheap NICs, costly switches

[PhotoGuy]

If you want multiple gigabit ports, you are looking at more than $600 for the bottom rung products.

Hey, don't knock it. Two years ago a gigabit switch would run you $5000-$1000! Things are definitely getting more reasonable. Maybe not for the average (non-geek :-) home, but for business, it's getting very accessible.

Re:Cheap NICs, costly switches

[Master Bait]

I'm looking at upgrading the Linux server to act as a giga switch. We've got two Macs and another Linux workstation. That means four $40 Ark cards (both Macs already have giga-nics)and a changing over to a $65 SiS745-based motherboard. SiS claims they have a concurrent line (1.2GB/second bus total) to each of six PCI masters. The server we're using now does file and print serving for only two people. I think it will be able to handle giga-switching quite well.

Re:Cheap NICs, costly switches

[bryan1945]

Hello Mr. Obvious! 2 years ago 802.11 was mucho expensive (even $5000-$1000!; nice backwards price range). Someday maybe people will even have *CABLE* TV! Or digital viagra.

Re:Cheap NICs, costly switches

[peter]

Full duplex means no collisions, so you don't have the minimum packet length requirement that the round-trip time between the two most distant points in the LAN is shorter than the time to transmit the minimum sized packet. (You need that so all hosts know about the collision and can start transmitting again without getting in a mess.) If you had gig hubs, you would be limited to maybe 10 meter cables or something? (that kind of ballpark)

Re:Cheap NICs, costly switches

[kyrre]

This guy claims "Full duplex is bad"

Re:Cheap NICs, costly switches

[NevDull]

I got a Farallon 4-port GigE switch for $210, and don't have to use some kind of bullshit bridging and fuck with an in-place server just to get switching capabilities.

Consider looking for such a thing.

Re:Cheap NICs, costly switches

[phaze3000]

That guy obviously has no clue as to how TCP/IP works. Whilst what he's saying might be correct if you were just shoving ethernet frames out of the network card, instead in the real world people have TCP/IP on top of that, which won't just keep sending packets as fast as it can no matter how many are dropped.

Re:Cheap NICs, costly switches

[Luminous Coward]

Full-duplex means no collisions [...]

Actually, full-duplex means a point-to-point link with separate receive and transmit paths. Hence, no collision can occur.

If you had gig hubs, you would be limited to maybe 10 meter cables or something?

Half-duplex Gigabit Ethernet was designed for a 200-meter network diameter. That's why the designers came up with carrier extension. If a frame is smaller than 512 bytes, it is extended to meet the 512-bytes minimum.

Re:Transfer speeds

[TheOnlyCoolTim]

Someone needs to learn the difference between a gigabit and a gigabyte....

Doing Math we can calculate that a full gigabit of transfer is 125 Megabytes a second. I think this is possible with high end hard drive technologies like SCSI RAID and speeds like this will probably show up in the desktop in a few years.

And, of course, not all data has to be written on Hard Drives. You could have a router or switch that will pass along a gigabit of packets a seconds but it certainly doesn't write them to the hard drive. You could for example but in a Gigabit Ethernet connection between two nearby buildings.

Tim

I have Paul Gray for a professor

[XBL]

and I took Networking from him last semester. He did a preliminary demo for the class, and I think that on the 32 bit PCI Gigabit cards, the effective throughput was around 250Mbps. Of course, the PCI bus was the limitation.

A 64 bit PCI card was getting significantly higher throughput. I don't remember the exact numbers, but it was much closer to 1000Mbps (maybe 800?).

Re:Transfer speeds

[Kwikymart]

I dont mean to be a nitpicking fucktard, but it is 128MB/s. 1 Gb/s == 1024Mb/s, divide that by 8 and you get 128MB/s. Of course, you would never get 128MB/s nor would you get 125MB/s either...... damn theoretical limits not living up to the real world.

My experiences with DGE500T

[redelm]

I bought a pair of DLink DGE500T's about 6 months ago, just to see what I could wring out of them.

I got about 32 MByte/s one-way with `ttcp` [UDP] between a 1.2GHz K7 and 2*500 Celeron (BP-6) through a plain crossover cable.

Not bad, but only 25% of wirespeed (125 MByte/sec). I figured the main limit was the PCI bus, which would only burst at 133 MByte/s, and I strongly suspected that the bursts were too short to achieve anything like this speed. I have yet to play with the PCI latency timer.

One thing for sure -- it isn't the CPU speed or Linux network stacks. The K7 will run both ends of ttcp through the localhost loopback at 570 MByte/s, and the BP6 around 200 MB/s.

Re:My experiences with DGE500T

[cperciva]

Don't be so sure about it not being the network stack. Localhost connections are often special cased because all the mechanisms for packet loss/reordering recovery are unnecessary over the loopback device.

Re:My experiences with DGE500T

[jmauro]

By default the MTU size on a gigabit card is way too low. The efficency sucks because of that. Increase the MTU size and see how much better things become.

Re:My experiences with DGE500T

[cheese_wallet]

There are basically two types of latency in PCI. The first latency is the amount of time it takes for a target to return a requested word. This is 16 33MHz clocks, according to the PCI 2.2 spec.

The second type of latency is the amount of time it takes a target to return a second word in a Burst transaction. This is 8 33MHz clocks, according to the PCI 2.2 spec.

The setting you are playing with in BIOS is probably the first latency... which is basically a setting in the PCI master, deciding how long to wait for data from a target before deciding to change the transaction to a delayed read. A delayed read basically frees the bus, and the master will check back with the peripheral at a later time to see if it has the data ready yet or not.

delayed reads slow down access to that peripheral, because no other transaction is allowed to take place with that peripheral until that delayed read is finished.

Older PCI cards didn't have the 16 clock limit on returning the first word of data, and they usually took longer. On new systems that try to be pci 2.2 compliant, to prevent a bunch of delayed reads from taking place, you have the option of increasing the latency timer in bios, so that it won't time out exactly on the 16 clock boundry, thereby speeding up access to that peripheral, at the cost of hogging the bus.

So anyway, adjusting the latency timer isn't likely to have an effect on newer peripherals... unless you make it too short, causing a bunch of delayed reads, and then your system will slow down.

--Scott

Re:My experiences with DGE500T

[redelm]

This sounds interesting. I definitely want the GigE to hog the bus -- there's no other way. The only other really active device on the PCI bus should be the EIDE, and they should have at least two 512byte buffers, so could wait ~500 PCI clocks (20 MB/s disk).

But I thought there was a register (oddly named latency) that governed how long a busmaster could burst when someone else wanted the bus.

Re:My experiences with DGE500T

[Luminous Coward]

By default the MTU size on a gigabit card is way too low.

Hmmm... The MTU for Ethernet is 1500 bytes. AFAIK, that's what the 802.3 standard says. Perhaps you're talking about Jumbo Frames? Are they supported by every vendor?

Re:My experiences with DGE500T

[Rolo Tomasi]

Actually, the EIDE ports on modern chipsets are not connected to the PCI bus anymore. They use proprietary high-speed buses (V-Link, HyperTransport, etc.), which is a good thing, because one Ultra-ATA133 Port has already the same bandwidth as the whole PCI(-32/33) bus.

Re:My experiences with DGE500T

[jmauro]

The 1500 is only defined for 10Mb/100Mb ethernet. It's higher. Check your equipment, but most switches can handle much larger packets than 1500. And if you have to leave the local network you're probably going to be fragmenting anyway, so the impact won't be too bad.

Re:My experiences with DGE500T

[sysboy]

You should becareful here as you may be confusing IP fragmentation with Ethernet fragmentation. Unfortunately lots of network boxes just truncate large frames.

Nopey

[Rolo Tomasi]

You got it all wrong dude, it's the LINPACK 1000x1000 double precision benchmarks that count.

You are paying too much for your cable

[Sycraft-fu]

Get a cheaper brand of cable, something tells me you really won't be able to hear the difference. All that regardless, the kind of fibre used for eithernet is not nearly so expensive. I can get 12-strand (6 single mode, 6 multi mode) fibre at around $1.00-$1.50/foot. That has enough for 6 different connections, three of them single mode (which costs more). For a short run of premade multi-mode fibre with the ends on it I'd think you shouldn't pay much more than $1/foot and perhaps less. At a length of 50 metres it should be around $0.50/foot.

IF Eithernet fibre was as expensive as you suggest, the university I work at would be bankrupt. Just last week I laid about 50 30-metre patch cables in a closet. This is not to mention the thousands already in place and the millions of metres of fibre that connects the buildings together.

Re:You are paying too much for your cable

[Gojira Shipi-Taro]

Thanks for the info. To be honest, I didn't shop much. I didn't expect the cost to be linear, so I wasn't expecting $1500 for 50 meters or anything, just figured it would cost more than copper.

How about a price/performance comparison then?

[NZheretic]

Anybody out there running Linux on a G4 using 1000BaseT ethernet?

Another evaluation of GigE performance

[sstammer]

There was another review of GigE performance in the IEEE Network Magazine last year.

Re:Transfer speeds

[packeteer]

hoe often is data written to the hd? not as often as you would think... your computer does NOT stream right to the hardrive... most of the time your transfers are done using ram which is up to 5x times the speed of your network depending on your ram...

AGP NICs

[Rolo Tomasi]

Are there any NICs using the AGP? Not many boards have 64bit PCI yet, let alone PCI-X, but every board has an AGP slot. This would be great for cheap 1U cluster nodes, with an appropriate riser card of course.

Re:AGP NICs

[Jonny 290]

AGP = Accelerated Graphics Port. Useless for anything except video cards at this point.

Re:AGP NICs

[Rolo Tomasi]

Well AFAIK AGP is PCI on steroids, it uses the same signals as PCI, the difference is the timing and extra signal lines that allow the AGP card to read and write directly to system memory, so graphics cards would be able to store big textures directly in main RAM (which of course is a few orders of magnitude slower than the graphics card's RAM, so this functionality isn't used at all). That's why an AGP card shows up in the PCI device listing, and why there are AGP and PCI graphics cards using the same chipsets.

But what the hell do I know.

Re:AGP NICs

[Rolo Tomasi]

Cluster nodes (I'm talking about number crunching) need two things: CPU power and network bandwidth. With modern CPUs, the network bandwidth is the limiting factor. And yes, I know that the theoretical bandwidth of PCI is 133 MB/s, and one Gb/s is 125 MB/s. The emphasis is on theoretical. If you had read the article, you would have found that the tested NICs were more than 50% faster in 64-bit PCI slots.

Re:AGP NICs

[Jonny 290]

Interesting. I do remember reading that one of the core functionalities of AGP was that low-end card could store textures in system memory.

I don't think this would make a communications-based card any faster, however. Unless you want to cache commonly-used packets for faster sending, heheh.

Re:AGP NICs

[Fulcrum of Evil]

Most server boards lack AGP and have multiple 64-bit PCI slots. Also, i've never seen an AGP riser.

Re:AGP NICs

[Paul Jakma]

i think the biggest difference is that AGP specifies that there /must/ be an IO-MMU to translate between bus addresses (which the AGP card specifies in requests) and physical addresses (ie of RAM). Ie the AGP host bridge can make system memory appear wherever it wants from the POV of the AGP card.

Note that PCI too can access memory "directly" (ie it can initiate a transfer) and that PCI chipsets used in Alpha's (21174 and up) and Sun have IO-MMU's. However, there is no requirement for PCI host bridges to be able to translate memory accesses.

IO-MMU's are useful on 64bit machines. Without them data held outside of the 4GB of adddress space that PCI can reference must be first copied into address space accessible to the PCI card (bounce buffering) - which is bad for performance. With an IO-MMU you can map the PCI address space to wherever you want in system address space.

Good IO-MMUs (eg the DEC 21x7x) can map very specific ranges of bus addresses to multiple ranges of system address space (ie hardware scatter gather).

Re:Text of Article (fourth section)

[autocracy]

Wow, that is the best fucking karma whoring mess I've ever seen. Drink it up dude...

Stuff about Gbit....

[NerveGas]

First, you can't just stick a gigabit card in a machine and expect it to work at full capacity. The basic design of ethernet was not really designed for gigabit speeds, but we've managed to squeeze it out - barely.

With 10 mbit cards, having the card generate an interrupt with ever incoming frame wasn't too bad. And on 100-mbit, it's still managable - but at a full gigabit, it really, really starts to bog down the machine. Some cards get around that by using interrupt coalescing, where they buffer up a few frames before they trigger an interrupt. That has a drawback, though: It increases latency. The trade-off has to be at some point, and not choosing the RIGHT point can affect either throughput or latency.

Furthermore, to get the full benefit out of your card, you generally need to enable jumbo-frames on both the card and the switch - and of course, your switch has to support that feature.

To make matters even worse, you can't always pump out (or receive) a full gigabit in any other than testing situations. Say you're receiving a large incoming file via FTP, NFS, or the protocol of your choice. Can your machine *really* write data to the disk at over 100 megabytes per second? And if it can, can it really handle both receiving a gigabit from the card, processing it, and writing the gigabit out to the disk? Unless you've got a very large amount of money in the machine, it probably won't.

steve

Re:Stuff about Gbit....

[Omnifarious]

Why not trigger the interrupt immediately, but continue to buffer frames and let the CPU grab all the frames in one go when it gets around to servicing the interrupt?

Actually, with a sophisticated card, driver, and OS, the range of systems which could pull in a full gigabit/sec would be vastly increased. It requires some careful thought and programming.

I sat down and sketched out a sample of how things might go, and realized I was missing some important details. On detail being the fact that it takes time to copy the data from the card to memory, and the CPU can be doing other things in that time. So, it requires more than 10 minutes thought, but I'm sure given a day or two, and access to relevant documentation about how various bits work, I could sketch out a driver design that made near optimal use of the available hardware, and wouldn't be that hard to implement in hardware either.

Re:Stuff about Gbit....

[Anonymous Coward]

If that was so easy for you to do, don't you think it's been done?

Re:Stuff about Gbit....

[Omnifarious]

From the comments posted by the guy I replied to, no, I don't think so, though I could very well be wrong. Caring about this kind of thing in userland has been my obsession for years now. :-) It's actually quite surprising how many mistakes people make because they don't think the problem through carefully, or don't have a good design background in solving those kinds of problems.

Re:Stuff about Gbit....

[Hadlock]

there's a reason why the patent office is still active today... ;)

30% vs 30%

[simpl3x]

one can make decent gains with a good managed switch and optimizing the workstations. i'm not really ready to invest in gig-e as of yet, since the internal transfer speeds are so determinent on the result. by the time general pc i/o speeds are up to par, 10gig-e will be the cost of gig-e today. a much better scenario. now, we simply ensure that certain workstations have precedence over the general pc population. hypertransport... yummy!

Re:Transfer speeds

[Kwikymart]

Technically we are both "right". However, when you use the term "bits" you mean binary digits. emphasize binary. We are not using base 10, but base 2.

The Free On-line Dictionary of Computing (13 Mar 01)

prefix

1. The standard metric prefixes used in the SI
(Syst`eme International) conventions for scientific
measurement. With units of time or things that come in powers
of 10, such as money, they retain their usual meanings of
multiplication by powers of 1000 = 10^3. When used with bytes
or other things that naturally come in powers of 2, they
usually denote multiplication by powers of 1024 = 2^(10).

Here are the SI magnifying prefixes, along with the
corresponding binary interpretations in common use:

prefix abr decimal binary

yocto- 1000^-8
zepto- 1000^-7
atto- 1000^-6
femto- f 1000^-5
pico- p 1000^-4
nano- n 1000^-3
micro- * 1000^-2 * Abbreviation: Greek mu
milli- m 1000^-1

kilo- k 1000^1 1024^1 = 2^10 = 1,024
mega- M 1000^2 1024^2 = 2^20 = 1,048,576
giga- G 1000^3 1024^3 = 2^30 = 1,073,741,824
tera- T 1000^4 1024^4 = 2^40 = 1,099,511,627,776
peta- 1000^5 1024^5 = 2^50 = 1,125,899,906,842,624
exa- 1000^6 1024^6 = 2^60 = 1,152,921,504,606,846,976
zetta- 1000^7 1024^7 = 2^70 = 1,180,591,620,717,411,303,424
yotta- 1000^8 1024^8 = 2^80 = 1,208,925,819,614,629,174,706,176

------

BINARY BINARY BINARY! WE USE BINARY! Take a look to the right under "mega". mega- M 1000^2 1024^2 = 2^20 = 1,048,576. Therefor, 2^30 / 2^20 = 2^10 = 1024 megabits in 1 gigabit.

Now, what part of that dont you understand?

copper vs fibre...

[thepoolguy]

Coolness aside, the market for gigabit ethernet is a lot larger for the corporate user than it is for the home user. One of the primary drivers for the fast adoption of gig-e in corporate environments is the ability to use the existing copper infrastucture by using an additional 2 pairs (copper gig-e uses 4 pairs).

The problem with fiber vs copper isn't really the cost of the medium, it is the cost of laying the infrastructure. If I remember correctly, the cost of the cable is about 1/10 of the total cost.

Part of the reason gig-e has become so cheap so quickly is that it has been able to ride the ethernet adoption curve to make the MACs and copper transcievers cheap because of the huge volumes. These volumes will never be reached by fiber.

-tpg

Re:copper vs fibre...

[GSloop]

Have you ever heard of cable testers?

Sheesh - a TDMA for fiber can tell you not only the quality of the cable and terminations, but also the distance to any faults!

(Believe me, I've got a OmniScanner for copper and I'm itching for Fiber - i just can't justify the cost yet. Troubleshooting cable run problems is really a breeze! And no guessing either. If the "Joes" you got installing fiber don't give you full certification results, you've not done your job in setting the specs for the job. And if your cable is getting damaged after install, either the cable didn't get installed right (protected runs etc) or you've got very careless people running around where they shouldn't.

Fiber is more difficult, but that's really because it hasn't reached critical mass. Once it starts getting installed in higher quantities, we'll see easy termination kits (there are some already).

Cheers!

yet more Excel graphs

[mrm677]

Seems like every graph I look at these days in research papers are the same styles and colors (Microsoft Excel defaults).

Too bad the open-source community doesn't have a better alternative. I've tried Grace...the learning curve was a little steep. Guppi is not ready, not is KChart. The best I've found so far is Octave, a open-source Matlab clone . That's because it provides an interface to GNU plot and Matlab is very familiar to me.

Re:yet more Excel graphs

[joib]

Also check out R. It has, IMHO, somewhat more advanced graphing stuff than gnuplot.

gig-e hubs

[thepoolguy]

When I hear the word hub, I think of half duplex shared medium. Although the gig-e spec (802.3z) contains support for half duplex, I don't know of any vendor that has implemented and tested it, espicially in a hub product.

-tpg

Re:The problem is gigabit SWITCHES are a fortune

[thepoolguy]

The problem is integration of multiple MACs and PHYs into a single chip. If you take apart a 4 or 8 port 10/100 switch, you will find that you have a singe chip product with some small components surrounding it. If you take a multiport 10/100/1000 switch apart, you will find many more chips because the integration hasn't happened there yet. Give it a few more years for the technology to mature.

-tpg

Interrupt coalescing and latency...

[thepoolguy]

You make a good point about interrupt coalescing affecting latency, but that was mostly on first generation gig-e products. It also had a lot to do with how the products were built. Newer NIC architectures have learned to balance this better.

During heavy loads, with interrupt coalescing enabled, latency is not an issue, getting the most throughput per cpu cycle is. If the NIC and driver together can determine system and network load, then it can make an intelligent decision when to delay presenting an interrupt to the host and when it should do so right away. To do this correctly, the NIC should also be able to determine when the host is in the interrupt service routine, so work can just be placed on the queue without requiring any interrupt. In cases like this, the work gets done almost for free.

A similar thing occurs in TCP with delayed ACKs.

-tpg

Re:Interrupt coalescing and latency...

[Omnifarious]

IMHO, the 'intelligent descision' should just fall out of how it's all designed instead of being an explicit part of the low level design.

Re:Interrupt coalescing and latency...

[thepoolguy]

I agree, but optimizing CPU utilization by reducing interrupts (and thus context switches) needs to be done carefully in order to avoid the latency problems mentioned by the previous poster. The proper heuristics for this require more information that older NICs and drivers provided to one another.

It actually pretty nice seeing evolution of some fairly fundamental things like interface definitions as these can really impact the performacne of the final product. A poor NIC interface design (from the host's memory and OS perspectives) can cripple an otherwise good design.

After re-reading what I just wrote, I think I may have just justified your comment. :-)

-tpg.

Re:Do Not Click Link!!!

[jhunsake]

Idiot moderators. The parent of this post had a disguised link. Next time I'll let you have a large open anus in your mind forever. :)

You get what you pay for...

[eplese]

I recently bought a pair of the DLink DGE-500T gigabit ethernet cards. I knew not to expect anywhere near the full gigabit speeds, but the performance I actually got was horrible. I'm not sure what the cause was, but connecting a Linux box to a Windows 2000 box directly with a 3 foot cable, all I got was right around 12MB/sec from Samba. I was quite disappointed with this because that isn't even twice the performance I get with 100mbps cards. It's not even giving me another 50% performance.

I tried tweaking all sorts of stuff, with no luck whatsoever. Samba performance settings, MTU sizes, but nothing helped at all. I couldn't seem to pinpoint any bottlenecks in the systems either.

One of the main reasons that these cards are so cheap is that they have a very small cache on them. The more expensive cards come with much more, but at much more cost also.

If anyone has any suggestions as to how to get any reasonable performance out of these cards, please let me know, but from my experience with these cards, I'd not recommend them at all.

Why copper?

[spiro_killglance]

Gigabit optical network cards a only a little
over a 100$ now, are full duplex and faster
than copper in most cases. We've just installed
4 Dual Athlon 2000MP linux boxes, with gigabit
optical cards, pretty damn fast as you can
imagine.

yea.... ok.....

[Transcendent]

Wow... ethernet cards are on sale from 100 bucks to 36 bucks.... but a gigabit switch is around 1,600 bucks!

You go ahead and buy that NIC, but it'll be a while before some l33t h4x0r geek buys that $1600 switch for his lan party...... well.... i do know a couple morons....

A real measurement problem

[GlobalEcho]

The authors of the study write:

the results obtained in this study clearly show that peak performance is not a complete indicator of peak performance

Wow. That makes any analysis tough, when performance measures fail to satisfy the Reflexive Property!

Brian

So how much fits into Gigabit/s?

[Dr. Spork]

I was wondering whether it was possible to pipe uncompressed video through a gigabit ethernet--say 720p? I ask because I still have this dream that I will be able to buy (make?) a component for my living room entertainment system that logs into my main computer as a user and plays back media files (both audio and video) on my fancy living room equipment. I think that system would be much more elegant than what I have now (analog RCA cables for audio and S-video running into my living room--it's an ugly hack).

The much nicer interface would be to have a living room box join my ethernet LAN. The box would just receive uncompressed audio and video from the computer over gig ethernet. That way, all the decompressing would be done by the fancy CPU in my bedroom, and the box would not become obsolete when new/more CPU-intensive codecs came out. (Because the alternative is, of course, to have the box do the decompression, but I don't like that.) Somebody please make one of these (or explain why it would be a bad idea).

Re:So how much fits into Gigabit/s?

[Hadlock]

well let's just start with the concept, and you can work it from there. first off, it'll most likely saturate the gigabit eithernet. your best bet is to go with mpeg decoding, but enough about that... first, figure out how many pixels you are going to have. this is going to be a number like 1028x768, or 1600x1200. multiply the two numbers together, and then multiply THAT number by how much color depth you're planning on having (16, 24, 32 bit). that gives you your bandwidth req's. 1600x122 @ 24 bit color @ 60 fps is somthing like 3.2 gigabits per second. now you know why those video cards cost so much :-D MPEG-2 encode/decode in either software or hardware isn't a whole lot. most every digital video system uses it, including your DVD player. you could probably hack somthing together using firewire, but wait till this summer when apple updates firewire to an 800mbps/gigabit. i might suggest nan apple cube. it's slow, but it's silent, and does firewire, i think. also has eithernet onboard, but not gigabit. shrug.

Re:So how much fits into Gigabit/s?

[Carl Drougge]

I can't say I understand what 720p is, but I'm going to guess it's resolution in some direction or other. Let's say the smaller one. That gets me 960x720. Say 60fps, even if TVs are only 60 (or 50 here..) half frames per second. That gets us:

$ dc
5k
960 720 60**1024 1024*/p
39.55078

(39.5MB/s for Y data.)

1 3/1+*p
52.73424

(53MB/s when we have C data too.)
^D

Seems to get you 422Mbit/s for the picture. Well within 1000Mbit, except real live throughput isn't there yet. But if you only send half frames you get well withing what you can actually get.. The sudden little stops TCP get you will kill the enjoyment I'd expect though. (Murphy says it'll happen..)

Great uses for gigabit Ethernet on Linux

[IGnatius T Foobar]

Gigabit Ethernet comes in really handy on Linux when you add 802.11q VLAN tagging.

For those of you who don't know how this works, here's a bit of a primer: basically, you set the port on your big data center grade switch to "trunk" and then you enable 802.1q on your Linux box. Then you don't just have one Ethernet interface with one address --- you have up to 4096 virtual ones, each on its own VLAN and each with an IP address that's valid on that VLAN. So you'd have eth0.1, eth0.2, eth0.3, etc... each talking to the machines on that VLAN.

Once you've got that running, you can do all sorts of neat stuff, including:

• A router! You're on every VLAN anyway, so why not? It's not nearly as fast as a hardware-based Layer 3 switching module, but it's several orders of magnitude cheaper.
• Really complex firewalls. You could put different parts of your organization (or whatever) on different VLAN's and then use your nifty Linux box to dictate what kind of policy is used to route between them.
• If you're in a big building with multiple tenants, each with their own VLAN on a shared network, you can reduce the number of Internet access NAT/firewall boxes. Instead of one for each tenant, you've got a single one.
• How about a VPN gateway that can place the caller directly on his or her department's own VLAN instead of having to route to it?

As you can see, it's limited only by your imagination. And with that much stuff potentially running through the box, you're going to need that 1 Gbps of speed. Happy hacking!

Re:Great uses for gigabit Ethernet on Linux

[Strog]

I've really taken an interest in the 802.11q stuff on Linux with all the VLANs here at the college. I was wondering about compatibility issues with Cisco equipment. Does anyone have some experiences to share with this?

Can someone use a linux box w/ 802.11q enabled try to spoof into another VLAN they aren't supposed to have access to?

I'm really thinking about those kids in the electronics lab. Their lab is physically segmented but they have other access in other labs and the dorms.

Re:Great uses for gigabit Ethernet on Linux

[IGnatius T Foobar]

I've tested it with a Cisco switch (specifically, a Catalyst 4006). It does work. If you don't want a port to have access to "every" VLAN then you have to restrict it at the switch. Otherwise, anyone with root on the Linux box plugged into a trunk port can simply define additional eth0.xxx interfaces on whatever VLAN they want.

NetPipe numbers are wrong, megabits in 1024*1024

[jelle]

So I also ran a netpipe test to see what it thought of my NICs.

It gives you a NetPIPE.out. According to the man page, they contain: "time to transfer the block, bits per second, bits in block, bytes in block, and variance."

First of all, the manpage is wrong because the second column gives a number much closer to megabits per second, and after numerical verification, I found that it's giving the value of 1024*1024 bits per second and not 1000*1000 bits per second.

In NIC-talk, when we say gigabit, we mean 1000,000,000 bits, not 1000*1024*1024 bits.

So when benchmarking your gigabit network card with netpipe, please remember that you're looking at speed results "1024*1024"-megabits, so your NIC is really only 953.6 megabits, which immediately gives a much better insight into the speed achieved by the Syskonnect card.

Did anyone else find it disturbing?

[licketyspit]

I was noticing that the pentium platform consistently outperformed amd. Does anyone know of a good reason why this would happen? I'm currently in the process of building my first dual athlon machine, and would like to run linux on it.

Re:Did anyone else find it disturbing?

[pmsr]

Yes, and it has to be expected, at least in machines with VIA chipsets. VIA pci implementation is something out of a comic book, to say the least. Refer to http://www.tecchannel.de/hardware/817/index.html for some light on the problem.

/Pedro

Re:Did anyone else find it disturbing?

[KILNA]

I'll take a stab at it. You're not seeing much in terms of off-the-shelf, branded AMD-based server systems... since the gigabit ethernet market is pretty much the same as the supported/branded server market, I would guess it would make sense for any optimizations for vendor-provided drivers to be Intel-centric. I did notice that one table that would indicate a bottleneck on the AMD side, but it could also just be that there was a peculiarity with throughput on that one motherboard as well.

Any experience with 2 cards in one cheap system?

[hamjudo]

What happens to the throughput when two links are active on one system?

Pretend I have 3 cheap Athlon based systems in one building. Assume one is acting as a server, and the other two are clients that aren't talking to each other. Because these are the cheaper cards, I only expect 300Mbps when one client is active. What happens when both clients are active?

Ideally, throughput would be no worse than 150Mbps/per card. I suspect it would be much worse.

If multiple cards did work well, then you could buy 6 cards to directly connect 3 machines. Much cheaper than 3 cards and a GigE switch.

I think I'll have to wait until even cheap machines have 64bit/66Mhz PCI busses. I know I'll have to wait until I get all my machines into one building.

Re:Transfer speeds

[TeraCo]

No, giga, mega, and kilo have always meant 'thousand'.. until programmers came along that messed everything up to make things easier for them.

Examples of this are: Kilograms, Megalitres, Gigawatts.. none of these things are 1024 times anything.

Re:Transfer speeds

[TeraCo]

Ultra-secret rot13 message: v whfg jnfgrq n srj frpbaqf bs lbhe zrnavatyrff yvsr

Y'AI 'NG'NGAH, YOG-SOTHOTH H'EE-L'GEB F'AI THRODOG UAAAH!

Yeah, I know it's off topic, but hey, I don't care.

Re:Transfer speeds

[Tremul]

Not only that, there exist Protocol limitations which only arise under a gigabit situation. These occur in the TCP header.

• The first is the flow window for TCP is somewhat inssuficient to express the window which could possibly be used under gigabit assuming a fast enough processor to empty the buffers at a fast enough rate.
  
• Current implmentations of TCP congestion control increase at a relatively slow rate when compared to the speeds possible with Gigabit.
  
• Because of the amount of packets that can be on the network at one time under gigabit it gives rise to possible sequence number problems. Packets laying around on the net from a previous connection could arrive "late" and appear to be a part of the current connection. This is an unlikely occurrence however it has a much greater probability of happening under gigabit.
  

I am somewhat tired and sleep deprived so take my comments with a grain of salt.

e1000 XT: I get 1000Gbs with 1500 MTU :-P

[buck68]

We've just setup up a gigabit cluster with new
1u servers from supermicro:

http://www.supermicro.com/PRODUCT/SUPERServer/Su pe rServer6012P-6.htm

I've tested it using a CISCO catalyst 3500XL switch. Our new 4000 switch goes online this week.

I can get a fully saturated link using a program I wrote:

http://mxtraf.sf.net

I'm using a 2.4.18 kernel with the latest drivers from Intel.

BTW, anybody know how to enable jumbo frames in IOS? :-} I havn't been able to test how much more CPU headroom they give me.

-- Buck

PCI 32/133?

[Sensei_knight]

Why hasn't the PCI bus been gradually upgraded like MB memory?

Re:PCI 32/133?

[robhancock]

It has been - we already have 64 bit and/or 66 MHz versions which together would quadruple the speed of 32/33. They're just not used on machines other than heavy-duty servers, etc. at present because most regular uses of PCI just don't need that much bandwidth. I imagine this will start to change, however, because of such things as gigabit Ethernet.

Binary metric prefixes

[peter]

read this and learn how kiB means 1024 bytes, kB means 1000 bytes, etc.

Re:Binary metric prefixes

[Kwikymart]

Cool, thanks for the link. However, those standards were created only four years ago. Standards take much more time than that to be fully adopted by the general public. Up until now, I guess, the same prefix was used for both binary and decimal. That is very interesting.

Re:Binary metric prefixes

[peter]

It's used in the Linux kernel, for example:
Linux version 2.4.18 (root@yeti) (gcc version 2.95.4 (Debian prerelease)) #15 Wed Apr 3 02:12:16 AST 2002
hda: 60046560 sectors (30744 MB) w/2048KiB Cache, CHS=29785/32/63
hdc: 25429824 sectors (13020 MB) w/418KiB Cache, CHS=25228/16/63

Re:comparison from a can

[KILNA]

Because the hardware on the cards themselves have a maximum MTU.

Intel e1000 adapter

[geirt]

There must be something wrong with the graphs for the e1000 packet size vs. throughput plot, I believe the axis are reversed.-

Also Intel acknowledges that their e1000 adapter have driver issues under linux. This text is from: ftp://aiedownload.intel.com/df-support/2897/ENG/re adme.txt

Known Issues
============
Driver Hangs Under Heavy Traffic Loads


Intel is aware that previously released e1000 drivers may hang under very
specific types of heavy traffic loads. This version includes a workaround
that resets the adapter automatically if a hang condition is detected. This
workaround ensures network traffic flow is not affected when a hang occurs.

This is for the driver verion 4.1.7, released 3/23/2002 (ie. quite new). Older versions had even bigger problems. This might explain why the Intel adapter does so bad in this test. I wish that Intel gets a clue and releases all card specs and GPLs the existing driver so that a true (stable) open source driver could be written and included in the linux kernel. I think the hardware is OK, but the drivers sucks.

Intel NIC's

[DarkEdgeX]

Does Intel's desktop cards support PXE (or rather, have the correct support so as not to be lumped in with Netgear's cards (bleh.. when I first got into networking I bought some Netgear cards because I had such great success with their switches/hubs-- NEVER AGAIN; this is the company that accidentally setup their PCI ID (or whatever it is that allows Win9x to autodetect and load drivers for devices) incorrectly as ANOTHER card, thus allowing Windows to load the WRONG driver for the card-- nightmare!)?

I've had really good experiences with Intel NIC's, and in fact have two Pro 100/S Server Adapters and two Pro/1000 T Server Adapters (the forefather to the newer 'server' class models) for use in my systems-- Intel's driver support is absolutely amazing, and incredibly stable/friendly. The fact that they offer up alternate platform drivers is just another bonus.

sortof

[poemofatic]

but according to the IEC it's better to say "Gibibit" or "Gibibyte", etc. when referring to binary numbers.

Look at the national institute of standards' web page dealing with base-2 units.

So the hard-drive makers are right, but can still be flamed for a lot of other reasons:)

Build your own Gigabit Switch! Fun Project?

[jbridges]

As many have said, Gigabit switches are priced WAY out of proportion to the price of Gigabit NICs.

So how about filling up a cheap PC with cheap NICs and using it as a switch?

Granted as others have said, the PCI bus is a limiting factor. But it will certainly blow away any 100mbit switch.

Another possibility is to put two Gigabit NICs in every machine, and run a daisy chain or even a ring type network.

Sounds like a fun project!

sounds like fun!

[nege]

I cant wait to download this and start sending in sites like amazon and microsoft.com!! w00t!

So what?

[purduephotog]

The DLink Gigabit ethernet cards I threw into the small file storage machine that hangs off of my computer don't care what cable you stick in them. As long as the wires come out, it figures out what's the correct 'routing' - crossover or regular.

So does that mean my $99 Gigabit Ethernet Card is at least as special as your $3000 Mac? :P

Re:Transfer speeds

[mesa]

IIRC gigabit ethernet and fibre channel use 10-bit bytes giving just 100MB/second.

Don't get a cheezy 1000T card (read on)

[tcc]

I needed gigabit bandwidth at work because I am moving 100GB files.

I went on reading about it on the net, on sites like www.3wire.com for example, and to make a long story short, Fiber optic yeilds the best results (obviously) but are way to expensive. Next are some 1000T copper cards that are almost doing the job, but then again, after getting 5 different cards, I can tell you right away that you can have a BIG difference from a board to another.

The best card I've got so far performance-wise are the Intel Pro 1000T-based adapters, with no optimization card to card running netcps, I'd get twice as much speed out than with the Dlink counterpart (DGE500T). They are a bit more expensive, but if you want more than 3x increase over 100Mbits, you need something a tad more expensive.

The other thing is you see card with 70Megs/second bandwidth tests on some websites, with jumbo packets turned on. You need a jumbo-frame capable switch (read: Expensive) to be able to turn that on. The cheapest gigabit switch I've found that could take an aweful lot of load without costing me an arm was the Netgear GS508T, but if you are used to managed switch, that one isn't.

Also you might be tempted to get a Gigabit card as upling with let's say 8 ports @ 100Mbits, that way you won't waste bandwidth to the server and the 8 of them can crunch it. Well good idea on paper but don't get the Dlink DES-1009G, I had to return 2 of them, and the firmware on that thing truely SUCKS. You can't just leave it there and forget it, you need to cycle the power sometimes so it can "read properly" on the ports wether 100 or 10 or half or full duplex. It's miserable and poor performing. It's cheap though :). If you can afford to do power cycling once and a while and it's not a buisness server with critical uptime, it's not all bad.. (like for a little renderfarm).

For the Intel pro cards, I got both the workstation and server ones, server being 64bits PCI.

There's one thing you want to consider, if you use Gigabit ethernet, you need also to be able to feed it, 50megs/second on the board requires a drive being able to deliver 50 megs a second to the card, and requires a PCI bus able to take the load as well (remember, it's 50megs x 2 bandwidth on the bus that on pci32/33mhz saturates at 128 but in real world 100).

We need something simpler

[billcopc]

Gig-E seems to me like it's DOA. Ethernet is much too complicated to sustain such high speeds. We need something totally redesigned that will take out some of the excess baggage that's been dragged along since the days of 2Mbps glory.

My main gripe is that NICs have always been the most bus-abusive device in a system, unless you pay 5x more for a high-end 3Com card. Heck, we've got Firewire-p2p running circles around 100base-T, partly because the low-level protocol is simpler.

The other catch is that bandwidth is usually inversely proportional to latency. On paper it sounds absurd, since more speed equates to smaller packets that arrive faster. Indeed they do get 'here' faster, but they take longer to process because the CPU is constantly being harassed by the network card running 10x faster than typical 100bT. To cope with this we need fatter packets.

My temporary solution : have two NICs, one Gig-E, the other 100bT or even 10bT, and have them on separate PCI buses. The Gig-E can handle fileserving and other heavy tasks, while the 100bT will take care of low-latency low-throughput tasks, that way you can crank up the MTU size on the Gig-E side to an absurd number on your LAN, and keep your sub-1500 MTU on the internet side for fast pings in Q3A.

The hard part is finding an affordable motherboard with dual PCI buses.

cpu utilization - how big is the pig?

[mcv]

How much of the CPU was used during the benchmarks? It would be nice if the benchmark reported it. I don't know if netpipe can do this, but if it cannot, measuring it externally will have to do.

Knowing this value can help measure the efficiency of the hardware and the device driver. For all things equal (data throughput / price ) -- a solution that uses the least amount of CPU is the most attractive to me. In a server environment this may be even more important than the raw throughput #'s.

Re:Text of Article (fourth section)

[__aanonl8035]

When it was all said and done,
I lost karma posting this.

It was posted in sections because the
article was huge and I could only format
the HTML to text after working on it for
a bit, so in order to get it up as fast
as possible....

I would work on a section and post it...
so readers could see (the site was slashdotted)

So much for trying to help.