Slashdot Mirror


Mixing Gigabit, Copper, and Linux

iampgray writes: "With copper-based gigabit cards selling for less than $36 these days, what kind of performance can you expect -- especially in the often-overlooked Linux market? We sought out to test exactly what you could expect from copper-based gigabit solutoins for the desktop interface through the cluster-targeted products. Name brands and off-brands were put through the wringer. How'd they fare? Interesting results to say the least."

73 of 243 comments (clear)

  1. Huh. by autopr0n · · Score: 2, Interesting

    Are we even to the point when a normal PC could handle Gigabyte? And if so, why not use optical? I mean, saying I've got a fiber optic home network is a lot cooler then saying I've got a gigabyte eth home network. I mean to a geek, (to anyone else, that would just be lame... er...)

    How much more expensive is the optical stuff for GigE? I'm mostly using optical audio connections from my home sterio, and that's not to much money

    --
    autopr0n is like, down and stuff.
    1. Re:Huh. by bleckywelcky · · Score: 2, Informative


      Gigabit, not GigaByte. Gigabit = 1000000000 bits. GigaByte = 8000000000 bits. A 1 GBps connection is 8 times faster than a 1 Gbps connection.

    2. Re:Huh. by Gojira+Shipi-Taro · · Score: 4, Informative

      Well... copper is cheaper than fiber for the moment. I'd hate to think what my 50 meter run from my router to the second floor of my townhouse would cost if it was fiber.

      I use optical runs for my audo as well, but those are all under a meter, for the most part, and around $30 or so a piece. Not too much money for the purpose, but I dont' think I'd enjoy paying for a 50 meter run. Never mind the cost of devices with optical interfaces.

      That said, I guess the only reason I'd consider GB copper is that it's no more expensive than 100 base-T...

      --
      "Oh my God. This is terrible. This is the end of my Presidency. I'm fucked."; ~ Donald J. Trump
    3. Re:Huh. by WolfWithoutAClause · · Score: 5, Interesting

      > Are we even to the point when a normal PC could handle Gigabyte?

      Yes. Some memory parts are 333 Mhz and are 4 bytes parallel and instructions/s (as opposed to the clock rate) is over 1 GIP I think. So a PC can just about knock out a gigabyte/s if it has to, but it hasn't got much time to think about anything.

      But this article is talking about gigaBITs/s. That's 8x slower. So that too.

      --

      -WolfWithoutAClause

      "Gravity is only a theory, not a fact!"
    4. Re:Huh. by Christopher+Thomas · · Score: 3, Insightful

      Are we even to the point when a normal PC could handle Gigabyte? And if so, why not use optical?

      A 32-bit 33 MHz PCI bus can support one (1) gigabit ethernet card at full capacity (card's bandwidth is about 100 Mbytes/sec, PCI 32/33 is 133 Mbytes/sec).

      If you want to stick multiple cards in (e.g. for a small hypercube-style cluster), buy motherboards that support 64/33 or 64/66 (I was drooling over the dual-processor 64-bit-PCI AMD boards a little while back).

      Gigabit ethernet over copper has the advantage of running over your existing cabling (i.e. cat-5 is fine). This avoids having to muck about with fiber, as fiber is a PITA to maintain yourself (getting optically perfect connections for the fiber jacks is picky).

      The way gigabit ethernet is encoded on cat-5 cable is both sneaky and elegant.

    5. Re:Huh. by megabeck42 · · Score: 2, Informative

      >> Gigabit ethernet over copper has the advantage of running over your existing cabling (i.e. cat-5 is fine). This avoids having to muck about with fiber, as fiber is a PITA to maintain yourself (getting optically perfect connections for the fiber jacks is picky).

      Actually, the siecor unicam series work really, really well. They use a index of refraction matching gel inside the factory polished terminators. All you have to do is cut'n'crimp. They work great. I haven't ever had to do any splicing though - but, given how well the siecor stuff works, I can't see it being a remarkable problem.

      --
      fnord.
    6. Re:Huh. by WhaDaYaKnow · · Score: 2, Insightful

      A 32-bit 33 MHz PCI bus can support one (1) gigabit ethernet card at full capacity (card's bandwidth is about 100 Mbytes/sec, PCI 32/33 is 133 Mbytes/sec).

      Almost. 133MBytes/sec = 1064Mbit/sec. This means that it could only in theory keep up, if all bandwidth on the PCI bus was available for data. But, this number includes the overhead of setting up transfers and arbiting for devices that want to transfer, and these operations are fairly expensive.

      Also, the PCI device needs to obtain descriptors etc. (which indicate where to put the Ethernet packets in RAM) over the same bus, costing more valueable cycles.

      If you did have only the one device on the PCI bus (which is very unlikely), with a good chipset, you'd probably get over 100MB/sec, but not much more. So you'd never be able to actually get full Gb Ethernet. (as the test results show, things are MUCH worse then this, but this is probably caused by multiple devices on the PCI bus).

      Talking about chipsets, a long time ago we had a board with an OPTi chipset. They ran out of silicon when designing the chip, so they couldn't implement the Bus Master FIFO, so they decided to just abort every BM cycle after each 32 bit transfer, yielding a max transfer rate of 4MB/sec!! For weeks, I couldn't figure why my network driver wouldn't send packets faster than 30 Mbit/sec, until my boss flew to California (where OPTi was located) to find out what we where doing wrong.

      Back to the tests: for some reason they failed to mention the chipsets used on the motherboards which really is VERY important if you want to use a gigabit Ethernet card in a 32/33 PCI system. The fact that the Dell has 5 PCI slots probably means that it has an integrated PCI-PCI bus (not many chipsets support 5 PCI slots, unless one or more of them do not support Bus Mastering), which would certainly not improve things.

      I think this is important to mention, because most systems today, at least desktops, will only have 32/33 PCI, and as the test results show, with a presumably shitty chipset, you only get marginally better performance than 100Mb Ethernet...

    7. Re:Huh. by Fnord · · Score: 4, Informative

      The fastest pci gets is 66mhz 64bit. Thats 64 bits per clock cycle, 66M clocks per second....4.224 Gigabits. I'd say thats a little higher than 1 Gb.

  2. I didn't even notice 1000bT was so cheap... by Geek+In+Training · · Score: 2, Insightful

    I checked out the cards, and yes you can get them cheap, but what about switches? You figure they're still uber-pricey too, right?

    Nope... apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150! (I've been most happy with my D-Link DI-804 Router/firewall/switch for $79.)

    Is this the normal "cheaper as tech gets more widespread and easier to manufacture," and do you think maybe Apple making gigabit ethernet a standard feature had something to do with it? :)

    --
    SlashSigTheorem: Humorous, Political, Critical, Constructive- If you have a .sig, someone WILL complai
    1. Re:I didn't even notice 1000bT was so cheap... by Graymalkin · · Score: 2

      Having a couple hundred thousand Macs with gigabit ethernet probably did have something to do with it. Many vendors now offer gigabit ethernet NICs in their professional series systems which means for network infrastructure folks there's a high demand for equipment leading to increased production and a lower cost. Apple's been selling gigabit ethernet standard for almost two years now which amounts to lots of gigabit ethernet cards floating around in Macland.

      --
      I'm a loner Dottie, a Rebel.
    2. Re:I didn't even notice 1000bT was so cheap... by Fastolfe · · Score: 2

      10 100Mbit clients can pull 100Mbit each through 1 Gbit uplink port. With only a 100Mbit uplink port, one client trying to pull all 100Mbit through it will thus degrade service for all the other clients.

    3. Re:I didn't even notice 1000bT was so cheap... by ncc74656 · · Score: 5, Informative
      Nope... apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150!

      D-Link's site is nearly impossible to navigate (maybe it requires JavaScript, which I've shut off), but the Pricewatch description of the DES-1009G indicates that Gigabit Ethernet is only available on one port as an uplink connection; the rest of the switch is your run-of-the-mill 10/100 job. The DGS-1008T is D-Link's 8-port unmanaged 10/100/1000 switch; the cheapest entry on Pricewatch for that is $595.

      BTW, I have the entire site downloaded. Maybe I'm insane to even think about mirroring a /.'ed article on my home cable-modem link, but here it is. I've converted all the charts to PNG so they'll load slightly faster, and I got rid of most of the godawful "super-31337" yellow-on-black text to improve readability. You can also choose this link to download the entire page (images and all) in one shot.

      --
      20 January 2017: the End of an Error.
  3. Hoboy. by Soko · · Score: 4, Funny

    How'd they fare?

    Not terribly well against the /. effect.

    Interesting results to say the least.

    Lessee, a story about increasing bandwidth on a server /.ed to oblivion? That's not interestng, that's anti-climatic - I know what happens before I get to the story. Oh well...

    Soko

    --
    "Depression is merely anger without enthusiasm." - Anonymous
    1. Re:Hoboy. by digitalunity · · Score: 2

      anti-climactic, I hope you mean. anti-climatic? Is that some sort of freezer?

      --
      You can't legislate goodness. Let each to his own destiny, by will of his freely made choices.
  4. Has anybody read the article yet? by BeBoxer · · Score: 3

    Could you post a summary? That must be about the fastest /.-ing I've seen. What'd that take, about 5 minutes?

  5. Re:Text of Article (second section) by __aanonl8035 · · Score: 2, Informative

    D-Link DGE-500T

    D-Link DGE-500T was the first of the gigabit cards tested. This card is based on SMC's dp83820 chipset and is designed for a 32bit bus. The chipset in this card turned out performance nearly identical to the two Ark cards and the GigaNIX cards tested in our test suite, since all utilize the dp83820 chipset from SMC. The Linux driver used was the ns83820 as included in the 2.4.17 kernel. Latency on both platforms was .0002 seconds.

    Peak throughput while operated in a 32bit bus was 192.21 Mbps. This was achieved in the Dell systems. The Athlon systems only obtained a peak of 172.21 Mbps when these cards were inserted into the 32-bit bus. Both systems show a slight drop in throughput but eventually level out. Peak throughput while operated in a 64bit bus running at 33Mhz was 315.96 Mbps.

    When the bus was jumpered to autoselect 66/33Mhz, the performance increase was negligible. Peak throughput was 316.40 Mbps. Comparing the plots of the 66Mhz and 33Mhz run reveals that they are essentially identical.

    For complete testing results, click here.

    Price: $45

    The cost per Mbps is as follows:

    32bit 33Mhz: $45 /((192.21+172.21) / 2) = $.25>

    64bit 33Mhz: $45 / 315.96 = $.14

    64bit 66Mhz: $45 / 316.40 = $.14

    Ark Soho-GA2500T

    The Ark Soho-GA2500T is also a 32-bit PCI card design. Like the D-Link DGE-500T and the Asante GigaNIX cards, this card is based on the SMC dp83820 chipset. With that in mind the performance was estimated to be close to the D-Link DGE500T. The driver used was the generic ns83820 included the 2.4.17 kernel. The latency for both test systems was .0002 seconds.

    The peak throughput achieved while in a 32bit 33Mhz bus was in the Dell system: 192.62 Mbps. While the Athlon system in the same bus setup only reached 172.19 Mbps. As before, there is a performance drop at the 1Kb and 5-10Kb packet sizes.

    Peak throughput while operated in a 64bit bus running at 33Mhz was 610.83 Mbps and 609.98 Mbps when running at 66Mhz respectively. As with the Soho-GA2000T, there is no noticeable difference between a 33Mhz and a 66Mhz bus.

    For complete testing results, click here.

    Price: $44

    The cost per Mbps is as follows:

    32bit 33Mhz: $44 / ((192.62+172.19) / 2) = $.24

    64bit 33Mhz: $44 / 610.83 = $.07

    64bit 66Mhz: $44 / 609.98 = $.07

    Ark Soho-GA2000T

    Our transition into cards designed for a 64-bit PCI bus began with the Ark Soho-GA2000T. Like it's 32-bit counterpart, this card was designed around the ns83820 chipset, which will allow us to examine the performance benefits, if any, in moving from a 32-bit As

    Designed to run in a 64bit 66Mhz slot, this card is backwards compatible to 32bit and 33Mhz slots. This card is based off of SMC's dp83820 chipset so performance was expected to be similar to the DGE500T and the Soho-GA2500T. The driver used was the generic ns83820 included in the 2.4.17 kernel. Latency was .0002 seconds on both test platforms.

    Peak throughput for a 32bit 33Mhz slot was 189.93 Mbps in the Dell system. The Athlons were only able to reach 172.26 Mbps.

    Peak throughput for 64bit 33Mhz was 665.06 Mbps with an MTU of 6000. Peak throughput while running at 66Mhz was 640.60 Mbps. With the exception of the 6000MTU tests, there is no noticeable difference between bus speeds of 33 and 66Mhz.

    For complete testing results, click here.

    Price: $69

    The cost per Mbps is as follows:

    32bit 33Mhz: $69 / ((172.26+189.93)/2) = $.38

    64bit 33Mhz: $69 / 665.06 = $.10

    64bit 66Mhz: $69 / 640.60 = $.11

  6. Re:i'm sorry by WolfWithoutAClause · · Score: 2

    Mutually Exclusive? Nope, just limited range. Fiber can go 1000km or more.

    Still, sometimes you only want to go a few feet between two servers or something and there you can't really argue with the price.

    --

    -WolfWithoutAClause

    "Gravity is only a theory, not a fact!"
  7. Obligatory Mac Plug by Lally+Singh · · Score: 4, Informative

    Just fyi, Macintosh 1000BaseT ethernet controllers go directly to the memory controller, bypassing PCI altogether..

    --
    Care about electronic freedom? Consider donating to the EFF!
    1. Re:Obligatory Mac Plug by Daniel+Wood · · Score: 3, Insightful
      I find this even more interesting:

      Note: When connecting a Power Mac G4 computer directly to another computer without using an Ethernet hub, a crossover cable is not required; circuits in the PHY detect the type of connection and switch the signal configuration as required.

    2. Re:Obligatory Mac Plug by Lally+Singh · · Score: 2

      This feature has saved me many a time on my PowerBook G4. When it works & nothing else does, I've got the wrong cable! :-)

      --
      Care about electronic freedom? Consider donating to the EFF!
    3. Re:Obligatory Mac Plug by Skuld-Chan · · Score: 2

      Yeah - my rca cable modem does that too - seriously!

    4. Re:Obligatory Mac Plug by NevDull · · Score: 2

      It's not uncommon. It's called Auto MDI-X.

  8. Clusters by jhunsake · · Score: 3, Informative

    Stay away from cards that don't have PXE and cards in which the driver won't compile into the kernel (as opposed to a module) if you plan to do easy installations or mount root off the network. In other words, stay away from Netgear and some 3Com cards (I haven't tested others), and play it safe with Intel.

  9. Gigabit and Linux by GigsVT · · Score: 5, Informative

    Well, check out the docs first off. It's hard to get much out of GBit, since most of the utilities don't call the socket open with properly sized buffers/window/whatever.

    I set up optical gigabit for some NAS type things at work, and out of the box, GBit performed maybe 30% better than 100 Mbit. We are talking about 110Mbit peaks, compared to 80Mbit peaks with 100Mbit switched.

    Setting the MTU to 6144 (max that I could set it to with the ns83820.o) I started to get peaks around 300Mbit/sec.

    I tried recompiling the module for higher limits, since in the source it has:

    #define RX_BUF_SIZE 6144 /* 8192 */

    But if I put in 8192, or 9000 like I wanted it to be, it would crask or lock up.

    Anyway, it's not trivial to get good performance out of GBit, and definitely don't expect anywhere near 10X gain.

    --
    I've had enough abrasive sigs. Kittens are cute and fuzzy.
    1. Re:Gigabit and Linux by tcc · · Score: 2

      It's all about the card, cheap cards will perform terrible, like I posted a bit lower, the Dlink cards are TERRIBLE, but if you give a shot to an Intel Pro 1000T, these are the best cards on the market for gigabit ethernet over copper. 3Com is also good, but with my dlink cards I was getting HALF the bandwidth that I would get with my pro1000T.

      --
      --- Metamoderating abusive downgraders since my 300th post.
  10. Re:Text of Article (fourth section) by __aanonl8035 · · Score: 2, Informative

    Comparisons and Observations

    In this section, we compare performance differences between cards in like environments , provide some general performance observations, and examine the cost per megabit as determined by the operating environment.

    Head-to-head throughput results

    While the results obtained in this study clearly show that peak performance is not a complete indicator of peak performance, in this section we examine the peak performance results amongst all cards under common environments.

    32-bit, 33MHz PCI Bus, 1500 MTU
    64-bit, 33MHz PCI Bus, 1500 MTU
    64-bit, 66MHz PCI Bus, 1500 MTU
    64-bit, 33MHz PCI Bus, 3000 MTU
    64-bit, 66MHz PCI bus, 3000 MTU
    64-bit, 33MHz PCI bus, 4000 MTU
    64-bit, 66MHz PCI bus, 4000 MTU
    64-bit, 33MHz PCI bus, 6000 MTU
    64-bit, 66MHz PCI bus, 6000 MTU
    64-bit, 33MHz PCI bus, 9000 MTU (Note: Drivers for the dp83820 chipset were limited to around 6000 MTU)
    64-bit, 66MHz PCI bus, 9000 MTU (Note: Drivers for the dp83820 chipset were limited to around 6000 MTU)

    General Observations
    Of the eight cards tested, the clear performance champion was the SK9821 with regard to throughput and consistency. The 3Com 3c996BT has a modest price tag and respectable performance for the entry-level server configuration. If price per megabit is the main concern, the Ark Soho-GA-2500T has the lowest cost per Mbps, making it a viable solution for entry-level systems requiring higher throughput than fast ethernet.

    The D-Link DGE500T and the Soho-GA2500T show nearly identical peaks, which is to be expected since the drivers and the chipsets were the same.

    The 3Com 3C996BT has results when compared to the 64-bit 33MHz results were surprising inasmuch as these cards showed better performance at 33MHz bus than at the higher 66MHz bus.

    Of all of the cards tested, the Intel E1000 TX proved to be comparable to the comparable to the Asante GigaNIX card in peak performance, but the erratic overall performance proved too much to overcome.

    In referring to the Complete Test Results sections for the 3C996BT and the SK9821 cards, one sees a very consistent and smooth transition to the peak throughput of the cards over the complete range of packet sizes.

    Some general comparisons that can be derived from the above results include the notion of ''cost per peak megabit. Depending upon the environment that the network device is to be installed, the cost per peak megabit varies greatly. For example, if one would wish to upgrade their P-III-based desktop system with a 32-bit, 33MHz PCI, the GA25000T is the clear cost-effective solution, but would not be able to provide throughput at the level of the 3Com 3C996BT.

    In an HPC environment, where sustained throughput is critical and the switch is capable of Jumbo frames, the SK9821 would be the best performer. In light of gigabit switching hardware that lacks Jumbo Frame support, a comparison of the 1500MTU results shows the SK9821 is still a viable choice, as is the 3Com 3C996BT which provides a more cost-effective solution.

    Paul Gray
    323 Wright Hall
    University of Northern Iowa

  11. Switches aren't cheap. by Christopher+Thomas · · Score: 5, Informative

    apparently Pricewatch.com has D-Link 8-port 10/100/1000baseT auto-detect switches listed for under $150!

    These are for 8x100-base-T with a gigabit uplink. I researched this a while ago, when speccing out my dream network ;).

    The cheapest full-gigabit switch D-link sells is about $1500.

    1. Re:Switches aren't cheap. by Bobzibub · · Score: 2

      One could go "token ring" style with 2 cards per machine and the network being a ring not star.
      As long as it is a small network and no machine goes off line it should be ok.

      Perhaps an expert knows if you could have two virtual IP's per card, a simple "Y" splitter plus two cross-over cables running into each machine via the splitter?? That might be a cheaper ring configuration.

      Cheers,
      -B

    2. Re:Switches aren't cheap. by Sycraft-fu · · Score: 2
      One could go "token ring" style with 2 cards per machine

      That would give real below average performance, even if you did get it to work. IT would be a nightmare getting all the routing setup properly so the computers would pass packets around the ring correctly. And once you got it working, speed would suck. Every packet would have to be evaluated in software and then routed to the next interface if necessary. This would be slow as hell, neurtalising the speed advantage.

      Perhaps an expert knows if you could have two virtual IP's per card, a simple "Y" splitter plus two cross-over cables running into each machine via the splitter??

      That would work like not at all.

  12. Heh. by autopr0n · · Score: 2

    I knew that actualy, and even flamed someone for typing "MB" when he meant "Mb". Oh well...

    --
    autopr0n is like, down and stuff.
  13. Re:Text of Article (fourth section) by Slash+Veteran · · Score: 3, Funny

    you, sir, are taking a huge slurp from the karma tit today. Congratulations to you! Cheerio.

  14. $30 for 1m!? by autopr0n · · Score: 2

    Wow, you are paying way to much. a 12ft optical cable I had used to connect my PC to my sound system broke a couple of days ago, and I thought it was gonna cost me $40 to replace it. Radio shack sold em for $44, but sears had a 12footer for just $20.

    --
    autopr0n is like, down and stuff.
  15. Cheap NICs, costly switches by Jah-Wren+Ryel · · Score: 5, Informative

    The cards are well priced for home use, and CAT5E cabling is cheap too. The problem with gigabit ethernet is not the cards, it is the lack of switches or even plain hubs at an affordable price point. There are lots of switches out there with a single gigabit port, but even those are a couple of hundred dollars. If you want multiple gigabit ports, you are looking at more than $600 for the bottom rung products.

    --
    When information is power, privacy is freedom.
    1. Re:Cheap NICs, costly switches by Geek+In+Training · · Score: 3, Funny

      Please disregard the parent post. It is WRONG, the hubs/switches are not cheap, I have been edumacted by my peers. :)

      The flogging will happen later, when my karma gets updated.

      --
      SlashSigTheorem: Humorous, Political, Critical, Constructive- If you have a .sig, someone WILL complai
    2. Re:Cheap NICs, costly switches by PhotoGuy · · Score: 2
      If you want multiple gigabit ports, you are looking at more than $600 for the bottom rung products.
      Hey, don't knock it. Two years ago a gigabit switch would run you $5000-$1000! Things are definitely getting more reasonable. Maybe not for the average (non-geek :-) home, but for business, it's getting very accessible.
      --
      Love many, trust a few, do harm to none.
    3. Re:Cheap NICs, costly switches by Master+Bait · · Score: 3, Interesting
      I'm looking at upgrading the Linux server to act as a giga switch. We've got two Macs and another Linux workstation. That means four $40 Ark cards (both Macs already have giga-nics)and a changing over to a $65 SiS745-based motherboard. SiS claims they have a concurrent line (1.2GB/second bus total) to each of six PCI masters. The server we're using now does file and print serving for only two people. I think it will be able to handle giga-switching quite well.

      --
      "Only in their dreams can men truly be free 'twas always thus, and always thus will be."
      --Tom Schulman
    4. Re:Cheap NICs, costly switches by NevDull · · Score: 2

      I got a Farallon 4-port GigE switch for $210, and don't have to use some kind of bullshit bridging and fuck with an in-place server just to get switching capabilities.

      Consider looking for such a thing.

    5. Re:Cheap NICs, costly switches by phaze3000 · · Score: 2

      That guy obviously has no clue as to how TCP/IP works. Whilst what he's saying might be correct if you were just shoving ethernet frames out of the network card, instead in the real world people have TCP/IP on top of that, which won't just keep sending packets as fast as it can no matter how many are dropped.

      --
      Blaming GW Bush for the Iraq war is like blaming Ronald McDonald for the poor quality of food.
  16. Re:Transfer speeds by TheOnlyCoolTim · · Score: 4, Informative

    Someone needs to learn the difference between a gigabit and a gigabyte....

    Doing Math we can calculate that a full gigabit of transfer is 125 Megabytes a second. I think this is possible with high end hard drive technologies like SCSI RAID and speeds like this will probably show up in the desktop in a few years.

    And, of course, not all data has to be written on Hard Drives. You could have a router or switch that will pass along a gigabit of packets a seconds but it certainly doesn't write them to the hard drive. You could for example but in a Gigabit Ethernet connection between two nearby buildings.

    Tim

    --
    Omnia vestra castrorum habetur nobis.
  17. I have Paul Gray for a professor by XBL · · Score: 2

    and I took Networking from him last semester. He did a preliminary demo for the class, and I think that on the 32 bit PCI Gigabit cards, the effective throughput was around 250Mbps. Of course, the PCI bus was the limitation.

    A 64 bit PCI card was getting significantly higher throughput. I don't remember the exact numbers, but it was much closer to 1000Mbps (maybe 800?).

  18. My experiences with DGE500T by redelm · · Score: 5, Informative
    I bought a pair of DLink DGE500T's about 6 months ago, just to see what I could wring out of them.

    I got about 32 MByte/s one-way with `ttcp` [UDP] between a 1.2GHz K7 and 2*500 Celeron (BP-6) through a plain crossover cable.

    Not bad, but only 25% of wirespeed (125 MByte/sec). I figured the main limit was the PCI bus, which would only burst at 133 MByte/s, and I strongly suspected that the bursts were too short to achieve anything like this speed. I have yet to play with the PCI latency timer.

    One thing for sure -- it isn't the CPU speed or Linux network stacks. The K7 will run both ends of ttcp through the localhost loopback at 570 MByte/s, and the BP6 around 200 MB/s.

    1. Re:My experiences with DGE500T by cperciva · · Score: 2

      Don't be so sure about it not being the network stack. Localhost connections are often special cased because all the mechanisms for packet loss/reordering recovery are unnecessary over the loopback device.

    2. Re:My experiences with DGE500T by jmauro · · Score: 2

      By default the MTU size on a gigabit card is way too low. The efficency sucks because of that. Increase the MTU size and see how much better things become.

    3. Re:My experiences with DGE500T by cheese_wallet · · Score: 4, Interesting

      There are basically two types of latency in PCI. The first latency is the amount of time it takes for a target to return a requested word. This is 16 33MHz clocks, according to the PCI 2.2 spec.

      The second type of latency is the amount of time it takes a target to return a second word in a Burst transaction. This is 8 33MHz clocks, according to the PCI 2.2 spec.

      The setting you are playing with in BIOS is probably the first latency... which is basically a setting in the PCI master, deciding how long to wait for data from a target before deciding to change the transaction to a delayed read. A delayed read basically frees the bus, and the master will check back with the peripheral at a later time to see if it has the data ready yet or not.

      delayed reads slow down access to that peripheral, because no other transaction is allowed to take place with that peripheral until that delayed read is finished.

      Older PCI cards didn't have the 16 clock limit on returning the first word of data, and they usually took longer. On new systems that try to be pci 2.2 compliant, to prevent a bunch of delayed reads from taking place, you have the option of increasing the latency timer in bios, so that it won't time out exactly on the 16 clock boundry, thereby speeding up access to that peripheral, at the cost of hogging the bus.

      So anyway, adjusting the latency timer isn't likely to have an effect on newer peripherals... unless you make it too short, causing a bunch of delayed reads, and then your system will slow down.

      --Scott

    4. Re:My experiences with DGE500T by redelm · · Score: 2, Interesting
      This sounds interesting. I definitely want the GigE to hog the bus -- there's no other way. The only other really active device on the PCI bus should be the EIDE, and they should have at least two 512byte buffers, so could wait ~500 PCI clocks (20 MB/s disk).

      But I thought there was a register (oddly named latency) that governed how long a busmaster could burst when someone else wanted the bus.

    5. Re:My experiences with DGE500T by Rolo+Tomasi · · Score: 2, Interesting

      Actually, the EIDE ports on modern chipsets are not connected to the PCI bus anymore. They use proprietary high-speed buses (V-Link, HyperTransport, etc.), which is a good thing, because one Ultra-ATA133 Port has already the same bandwidth as the whole PCI(-32/33) bus.

      --
      Did you know you can fertilize your lawn with used motor oil?
  19. You are paying too much for your cable by Sycraft-fu · · Score: 2

    Get a cheaper brand of cable, something tells me you really won't be able to hear the difference. All that regardless, the kind of fibre used for eithernet is not nearly so expensive. I can get 12-strand (6 single mode, 6 multi mode) fibre at around $1.00-$1.50/foot. That has enough for 6 different connections, three of them single mode (which costs more). For a short run of premade multi-mode fibre with the ends on it I'd think you shouldn't pay much more than $1/foot and perhaps less. At a length of 50 metres it should be around $0.50/foot.

    IF Eithernet fibre was as expensive as you suggest, the university I work at would be bankrupt. Just last week I laid about 50 30-metre patch cables in a closet. This is not to mention the thousands already in place and the millions of metres of fibre that connects the buildings together.

  20. How about a price/performance comparison then? by NZheretic · · Score: 2

    Anybody out there running Linux on a G4 using 1000BaseT ethernet?

  21. Another evaluation of GigE performance by sstammer · · Score: 4, Informative

    There was another review of GigE performance in the IEEE Network Magazine last year.

  22. AGP NICs by Rolo+Tomasi · · Score: 2, Interesting

    Are there any NICs using the AGP? Not many boards have 64bit PCI yet, let alone PCI-X, but every board has an AGP slot. This would be great for cheap 1U cluster nodes, with an appropriate riser card of course.

    --
    Did you know you can fertilize your lawn with used motor oil?
    1. Re:AGP NICs by Paul+Jakma · · Score: 2

      i think the biggest difference is that AGP specifies that there /must/ be an IO-MMU to translate between bus addresses (which the AGP card specifies in requests) and physical addresses (ie of RAM). Ie the AGP host bridge can make system memory appear wherever it wants from the POV of the AGP card.

      Note that PCI too can access memory "directly" (ie it can initiate a transfer) and that PCI chipsets used in Alpha's (21174 and up) and Sun have IO-MMU's. However, there is no requirement for PCI host bridges to be able to translate memory accesses.

      IO-MMU's are useful on 64bit machines. Without them data held outside of the 4GB of adddress space that PCI can reference must be first copied into address space accessible to the PCI card (bounce buffering) - which is bad for performance. With an IO-MMU you can map the PCI address space to wherever you want in system address space.

      Good IO-MMUs (eg the DEC 21x7x) can map very specific ranges of bus addresses to multiple ranges of system address space (ie hardware scatter gather).

      --
      I use Friend/Foe + mod-point modifiers as a karma/reputation system.
  23. Re:Text of Article (fourth section) by autocracy · · Score: 2

    Wow, that is the best fucking karma whoring mess I've ever seen. Drink it up dude...

    --
    SIG: HUP
  24. Stuff about Gbit.... by NerveGas · · Score: 4, Interesting

    First, you can't just stick a gigabit card in a machine and expect it to work at full capacity. The basic design of ethernet was not really designed for gigabit speeds, but we've managed to squeeze it out - barely.

    With 10 mbit cards, having the card generate an interrupt with ever incoming frame wasn't too bad. And on 100-mbit, it's still managable - but at a full gigabit, it really, really starts to bog down the machine. Some cards get around that by using interrupt coalescing, where they buffer up a few frames before they trigger an interrupt. That has a drawback, though: It increases latency. The trade-off has to be at some point, and not choosing the RIGHT point can affect either throughput or latency.

    Furthermore, to get the full benefit out of your card, you generally need to enable jumbo-frames on both the card and the switch - and of course, your switch has to support that feature.

    To make matters even worse, you can't always pump out (or receive) a full gigabit in any other than testing situations. Say you're receiving a large incoming file via FTP, NFS, or the protocol of your choice. Can your machine *really* write data to the disk at over 100 megabytes per second? And if it can, can it really handle both receiving a gigabit from the card, processing it, and writing the gigabit out to the disk? Unless you've got a very large amount of money in the machine, it probably won't.

    steve

    --
    Oh, you're not stuck, you're just unable to let go of the onion rings.
    1. Re:Stuff about Gbit.... by Omnifarious · · Score: 2

      Why not trigger the interrupt immediately, but continue to buffer frames and let the CPU grab all the frames in one go when it gets around to servicing the interrupt?

      Actually, with a sophisticated card, driver, and OS, the range of systems which could pull in a full gigabit/sec would be vastly increased. It requires some careful thought and programming.

      I sat down and sketched out a sample of how things might go, and realized I was missing some important details. On detail being the fact that it takes time to copy the data from the card to memory, and the CPU can be doing other things in that time. So, it requires more than 10 minutes thought, but I'm sure given a day or two, and access to relevant documentation about how various bits work, I could sketch out a driver design that made near optimal use of the available hardware, and wouldn't be that hard to implement in hardware either.

  25. Re:Transfer speeds by Kwikymart · · Score: 2, Informative

    Technically we are both "right". However, when you use the term "bits" you mean binary digits. emphasize binary. We are not using base 10, but base 2.

    The Free On-line Dictionary of Computing (13 Mar 01)

    prefix

    1. The standard metric prefixes used in the SI
    (Syst`eme International) conventions for scientific
    measurement. With units of time or things that come in powers
    of 10, such as money, they retain their usual meanings of
    multiplication by powers of 1000 = 10^3. When used with bytes
    or other things that naturally come in powers of 2, they
    usually denote multiplication by powers of 1024 = 2^(10).

    Here are the SI magnifying prefixes, along with the
    corresponding binary interpretations in common use:

    prefix abr decimal binary

    yocto- 1000^-8
    zepto- 1000^-7
    atto- 1000^-6
    femto- f 1000^-5
    pico- p 1000^-4
    nano- n 1000^-3
    micro- * 1000^-2 * Abbreviation: Greek mu
    milli- m 1000^-1

    kilo- k 1000^1 1024^1 = 2^10 = 1,024
    mega- M 1000^2 1024^2 = 2^20 = 1,048,576
    giga- G 1000^3 1024^3 = 2^30 = 1,073,741,824
    tera- T 1000^4 1024^4 = 2^40 = 1,099,511,627,776
    peta- 1000^5 1024^5 = 2^50 = 1,125,899,906,842,624
    exa- 1000^6 1024^6 = 2^60 = 1,152,921,504,606,846,976
    zetta- 1000^7 1024^7 = 2^70 = 1,180,591,620,717,411,303,424
    yotta- 1000^8 1024^8 = 2^80 = 1,208,925,819,614,629,174,706,176

    ------

    BINARY BINARY BINARY! WE USE BINARY! Take a look to the right under "mega". mega- M 1000^2 1024^2 = 2^20 = 1,048,576. Therefor, 2^30 / 2^20 = 2^10 = 1024 megabits in 1 gigabit.

    Now, what part of that dont you understand?

    --

    Buying a Dell computer is equivalent to dropping the soap in a prison shower.
  26. Re:Interrupt coalescing and latency... by Omnifarious · · Score: 2

    IMHO, the 'intelligent descision' should just fall out of how it's all designed instead of being an explicit part of the low level design.

  27. Why copper? by spiro_killglance · · Score: 2


    Gigabit optical network cards a only a little
    over a 100$ now, are full duplex and faster
    than copper in most cases. We've just installed
    4 Dual Athlon 2000MP linux boxes, with gigabit
    optical cards, pretty damn fast as you can
    imagine.

  28. A real measurement problem by GlobalEcho · · Score: 3, Funny
    The authors of the study write:

    the results obtained in this study clearly show that peak performance is not a complete indicator of peak performance


    Wow. That makes any analysis tough, when performance measures fail to satisfy the Reflexive Property!

    Brian
  29. So how much fits into Gigabit/s? by Dr.+Spork · · Score: 2
    I was wondering whether it was possible to pipe uncompressed video through a gigabit ethernet--say 720p? I ask because I still have this dream that I will be able to buy (make?) a component for my living room entertainment system that logs into my main computer as a user and plays back media files (both audio and video) on my fancy living room equipment. I think that system would be much more elegant than what I have now (analog RCA cables for audio and S-video running into my living room--it's an ugly hack).

    The much nicer interface would be to have a living room box join my ethernet LAN. The box would just receive uncompressed audio and video from the computer over gig ethernet. That way, all the decompressing would be done by the fancy CPU in my bedroom, and the box would not become obsolete when new/more CPU-intensive codecs came out. (Because the alternative is, of course, to have the box do the decompression, but I don't like that.) Somebody please make one of these (or explain why it would be a bad idea).

    1. Re:So how much fits into Gigabit/s? by Hadlock · · Score: 2

      well let's just start with the concept, and you can work it from there. first off, it'll most likely saturate the gigabit eithernet. your best bet is to go with mpeg decoding, but enough about that... first, figure out how many pixels you are going to have. this is going to be a number like 1028x768, or 1600x1200. multiply the two numbers together, and then multiply THAT number by how much color depth you're planning on having (16, 24, 32 bit). that gives you your bandwidth req's. 1600x122 @ 24 bit color @ 60 fps is somthing like 3.2 gigabits per second. now you know why those video cards cost so much :-D MPEG-2 encode/decode in either software or hardware isn't a whole lot. most every digital video system uses it, including your DVD player. you could probably hack somthing together using firewire, but wait till this summer when apple updates firewire to an 800mbps/gigabit. i might suggest nan apple cube. it's slow, but it's silent, and does firewire, i think. also has eithernet onboard, but not gigabit. shrug.

      --
      moox. for a new generation.
  30. Great uses for gigabit Ethernet on Linux by IGnatius+T+Foobar · · Score: 5, Interesting
    Gigabit Ethernet comes in really handy on Linux when you add 802.11q VLAN tagging.

    For those of you who don't know how this works, here's a bit of a primer: basically, you set the port on your big data center grade switch to "trunk" and then you enable 802.1q on your Linux box. Then you don't just have one Ethernet interface with one address --- you have up to 4096 virtual ones, each on its own VLAN and each with an IP address that's valid on that VLAN. So you'd have eth0.1, eth0.2, eth0.3, etc... each talking to the machines on that VLAN.

    Once you've got that running, you can do all sorts of neat stuff, including:
    • A router! You're on every VLAN anyway, so why not? It's not nearly as fast as a hardware-based Layer 3 switching module, but it's several orders of magnitude cheaper.
    • Really complex firewalls. You could put different parts of your organization (or whatever) on different VLAN's and then use your nifty Linux box to dictate what kind of policy is used to route between them.
    • If you're in a big building with multiple tenants, each with their own VLAN on a shared network, you can reduce the number of Internet access NAT/firewall boxes. Instead of one for each tenant, you've got a single one.
    • How about a VPN gateway that can place the caller directly on his or her department's own VLAN instead of having to route to it?


    As you can see, it's limited only by your imagination. And with that much stuff potentially running through the box, you're going to need that 1 Gbps of speed. Happy hacking!
    --
    Tired of FB/Google censorship? Visit UNCENSORED!
    1. Re:Great uses for gigabit Ethernet on Linux by IGnatius+T+Foobar · · Score: 2

      I've tested it with a Cisco switch (specifically, a Catalyst 4006). It does work. If you don't want a port to have access to "every" VLAN then you have to restrict it at the switch. Otherwise, anyone with root on the Linux box plugged into a trunk port can simply define additional eth0.xxx interfaces on whatever VLAN they want.

      --
      Tired of FB/Google censorship? Visit UNCENSORED!
  31. NetPipe numbers are wrong, megabits in 1024*1024 by jelle · · Score: 2

    So I also ran a netpipe test to see what it thought of my NICs.

    It gives you a NetPIPE.out. According to the man page, they contain: "time to transfer the block, bits per second, bits in block, bytes in block, and variance."

    First of all, the manpage is wrong because the second column gives a number much closer to megabits per second, and after numerical verification, I found that it's giving the value of 1024*1024 bits per second and not 1000*1000 bits per second.

    In NIC-talk, when we say gigabit, we mean 1000,000,000 bits, not 1000*1024*1024 bits.

    So when benchmarking your gigabit network card with netpipe, please remember that you're looking at speed results "1024*1024"-megabits, so your NIC is really only 953.6 megabits, which immediately gives a much better insight into the speed achieved by the Syskonnect card.

    --
    --- Hindsight is 20/20, but walking backwards is not the answer.
  32. Any experience with 2 cards in one cheap system? by hamjudo · · Score: 2
    What happens to the throughput when two links are active on one system?

    Pretend I have 3 cheap Athlon based systems in one building. Assume one is acting as a server, and the other two are clients that aren't talking to each other. Because these are the cheaper cards, I only expect 300Mbps when one client is active. What happens when both clients are active?

    Ideally, throughput would be no worse than 150Mbps/per card. I suspect it would be much worse.

    If multiple cards did work well, then you could buy 6 cards to directly connect 3 machines. Much cheaper than 3 cards and a GigE switch.

    I think I'll have to wait until even cheap machines have 64bit/66Mhz PCI busses. I know I'll have to wait until I get all my machines into one building.

  33. Re:copper vs fibre... by GSloop · · Score: 2

    Have you ever heard of cable testers?

    Sheesh - a TDMA for fiber can tell you not only the quality of the cable and terminations, but also the distance to any faults!

    (Believe me, I've got a OmniScanner for copper and I'm itching for Fiber - i just can't justify the cost yet. Troubleshooting cable run problems is really a breeze! And no guessing either. If the "Joes" you got installing fiber don't give you full certification results, you've not done your job in setting the specs for the job. And if your cable is getting damaged after install, either the cable didn't get installed right (protected runs etc) or you've got very careless people running around where they shouldn't.

    Fiber is more difficult, but that's really because it hasn't reached critical mass. Once it starts getting installed in higher quantities, we'll see easy termination kits (there are some already).

    Cheers!

  34. Re:switches? by DarkEdgeX · · Score: 2

    I'd be happy with an 8 + 2 switch from someone-- i.e. 8 10/100 ports with 2 10/100/1000 ports for my main file sharing boxen (and I imagine these would be a hit at LAN parties, so the server running the game could have a gigabit connect to the switch, allowing most of the 10/100 connections to saturate it with updates (and vice versa)). The trend of hardware makers (Netgear has done this too, FS309T is an 8 port 10/100 with a SINGLE 10/100/1000 copper gigabit port) to make these 8 + 1 solutions just sucks (since you can't really test the faster speed of gigabit with JUST one port).

    Of course in a perfect world, I'd agree that 8 port gigabit switches being $200 or less would be about near perfect, especially if higher port counts weren't unrealistically high.

    --
    All I know about Bush is I had a good job when Clinton was president.
  35. Re:yet more Excel graphs by joib · · Score: 2

    Also check out R. It has, IMHO, somewhat more advanced graphing stuff than gnuplot.

  36. Intel e1000 adapter by geirt · · Score: 2

    There must be something wrong with the graphs for the e1000 packet size vs. throughput plot, I believe the axis are reversed.-

    Also Intel acknowledges that their e1000 adapter have driver issues under linux. This text is from: ftp://aiedownload.intel.com/df-support/2897/ENG/re adme.txt

    Known Issues
    ============
    Driver Hangs Under Heavy Traffic Loads


    Intel is aware that previously released e1000 drivers may hang under very
    specific types of heavy traffic loads. This version includes a workaround
    that resets the adapter automatically if a hang condition is detected. This
    workaround ensures network traffic flow is not affected when a hang occurs.

    This is for the driver verion 4.1.7, released 3/23/2002 (ie. quite new). Older versions had even bigger problems. This might explain why the Intel adapter does so bad in this test. I wish that Intel gets a clue and releases all card specs and GPLs the existing driver so that a true (stable) open source driver could be written and included in the linux kernel. I think the hardware is OK, but the drivers sucks.

    --

    RFC1925
  37. Intel NIC's by DarkEdgeX · · Score: 2

    Does Intel's desktop cards support PXE (or rather, have the correct support so as not to be lumped in with Netgear's cards (bleh.. when I first got into networking I bought some Netgear cards because I had such great success with their switches/hubs-- NEVER AGAIN; this is the company that accidentally setup their PCI ID (or whatever it is that allows Win9x to autodetect and load drivers for devices) incorrectly as ANOTHER card, thus allowing Windows to load the WRONG driver for the card-- nightmare!)?

    I've had really good experiences with Intel NIC's, and in fact have two Pro 100/S Server Adapters and two Pro/1000 T Server Adapters (the forefather to the newer 'server' class models) for use in my systems-- Intel's driver support is absolutely amazing, and incredibly stable/friendly. The fact that they offer up alternate platform drivers is just another bonus.

    --
    All I know about Bush is I had a good job when Clinton was president.
  38. Build your own Gigabit Switch! Fun Project? by jbridges · · Score: 2

    As many have said, Gigabit switches are priced WAY out of proportion to the price of Gigabit NICs.

    So how about filling up a cheap PC with cheap NICs and using it as a switch?

    Granted as others have said, the PCI bus is a limiting factor. But it will certainly blow away any 100mbit switch.

    Another possibility is to put two Gigabit NICs in every machine, and run a daisy chain or even a ring type network.

    Sounds like a fun project!

  39. So what? by purduephotog · · Score: 2

    The DLink Gigabit ethernet cards I threw into the small file storage machine that hangs off of my computer don't care what cable you stick in them. As long as the wires come out, it figures out what's the correct 'routing' - crossover or regular.

    So does that mean my $99 Gigabit Ethernet Card is at least as special as your $3000 Mac? :P

  40. Don't get a cheezy 1000T card (read on) by tcc · · Score: 2

    I needed gigabit bandwidth at work because I am moving 100GB files.

    I went on reading about it on the net, on sites like www.3wire.com for example, and to make a long story short, Fiber optic yeilds the best results (obviously) but are way to expensive. Next are some 1000T copper cards that are almost doing the job, but then again, after getting 5 different cards, I can tell you right away that you can have a BIG difference from a board to another.

    The best card I've got so far performance-wise are the Intel Pro 1000T-based adapters, with no optimization card to card running netcps, I'd get twice as much speed out than with the Dlink counterpart (DGE500T). They are a bit more expensive, but if you want more than 3x increase over 100Mbits, you need something a tad more expensive.

    The other thing is you see card with 70Megs/second bandwidth tests on some websites, with jumbo packets turned on. You need a jumbo-frame capable switch (read: Expensive) to be able to turn that on. The cheapest gigabit switch I've found that could take an aweful lot of load without costing me an arm was the Netgear GS508T, but if you are used to managed switch, that one isn't.

    Also you might be tempted to get a Gigabit card as upling with let's say 8 ports @ 100Mbits, that way you won't waste bandwidth to the server and the 8 of them can crunch it. Well good idea on paper but don't get the Dlink DES-1009G, I had to return 2 of them, and the firmware on that thing truely SUCKS. You can't just leave it there and forget it, you need to cycle the power sometimes so it can "read properly" on the ports wether 100 or 10 or half or full duplex. It's miserable and poor performing. It's cheap though :). If you can afford to do power cycling once and a while and it's not a buisness server with critical uptime, it's not all bad.. (like for a little renderfarm).

    For the Intel pro cards, I got both the workstation and server ones, server being 64bits PCI.

    There's one thing you want to consider, if you use Gigabit ethernet, you need also to be able to feed it, 50megs/second on the board requires a drive being able to deliver 50 megs a second to the card, and requires a PCI bus able to take the load as well (remember, it's 50megs x 2 bandwidth on the bus that on pci32/33mhz saturates at 128 but in real world 100).

    --
    --- Metamoderating abusive downgraders since my 300th post.
  41. Re:Binary metric prefixes by peter · · Score: 2

    It's used in the Linux kernel, for example:
    Linux version 2.4.18 (root@yeti) (gcc version 2.95.4 (Debian prerelease)) #15 Wed Apr 3 02:12:16 AST 2002
    hda: 60046560 sectors (30744 MB) w/2048KiB Cache, CHS=29785/32/63
    hdc: 25429824 sectors (13020 MB) w/418KiB Cache, CHS=25228/16/63

    --
    #define X(x,y) x##y
    Peter Cordes ; e-mail: X(peter@cordes , .ca)