Finding the Bottleneck in a Gigabit Ethernet LAN?

guroove asks: "I have a small gigabit ethernet network at home, and I spent a lot of money getting gigabit NICs for all my computers and even bought cat 6 cabling. I only have 3 computers on the gigabit network (a Mac, a Windoze machine, and a Linux box) so instead of getting a switch, I triple NIC'd the Linux box, which I use as a gateway and a file server. After the network was complete, I wasn't satisfied with transfer rates, so I started a transfer of a very large file and found that the transfer rate was topping off at just over 145 Mbps (which is a far cry from 1000 Mbps). I'm wondering now where my bottleneck is. Is it the NICs? Are all gigabit NICs really giving us 1000 megabits per second? Is it the driver? Is it Samba? Could it be that the hard drives aren't fast enough? Does anyone have experience with gigabit home networking enough to know where the bottlenecks are? Does the current PCI technology even allow for bandwidth that high"

Comments

The Linux machine is acting as a router ?

[bungeejumper]

It is "entirely possible" that the Linux machine is acting as a router, switching all your traffic in C code. Not to mention it is probably sending traffic up and down the PCI bus, once at ingress and once at egress. The lookup of the IP destination address is probably using a whole lot of memory bandwidth, and if it's at all like a regular router, it's probably doing a full IP header Sanity check (using the IP CRC), version number and TTL decrement. After the TTL decrement, you would need to recompute the CRC. I would say the Linux machine is your bottleneck. Unless you could somehow get it configured as an ethernet switch, rather than a Layer 3 router.

Re:The Linux machine is acting as a router ?

[Jahf]

Agreed.

While there are a number of Linux based routers out there, none that I know of are used in the Gigabit realm. Even if they are, they at the -very- least have recompiled the kernel to switch on a number of router/gateway optimizations ... and quite possibly contain proprietary network / NIC kernel modules to further gain improvements.

Unless you have a VERY modern bus architecture (alot of people using Linux routers do so on old gear), preferably an AMD with hyperthreading (since I doubt you have a non-x86 system or you'd have mentioned it), you will never get close to maximizing not one but -3- Gb NICs.

Take a look at some of the servers that are out there in the x86 realm. They usually require you to use a 100MHz or 133MHz PCI card to get best results from a Gb ethernet NIC. And if you look at the first generation of x86 servers (say, from 2 years ago) that came with Gb ports by default, looking deep into the benchmarks you often find that they never reached their Gb potential with the built-in ports either. The advantage was that it was still better than 100 Megabit.

With a hyperthreaded high-speed bus and some kernel tweaks, I would be quite happy if I could get all 3 NICs to stress-test simultaneously at 300-500Mb/each. Heck, I'd probably be happy around the 250Mb range.

BTW, even a Gb switch, on the home CPE level, is probably never going to send multi-Gb of data (ie, by trying to switch data amongst multiple Gb ports). Often times you are limited to a max of 1Gb total throughput because of the switched backplane. Heck, even then you may max around 900Mb due to network overhead.

Moral is simply to realize that with all networking products, the real speed is usually significantly less than the rated speed.

Maybe something besides samba

[Student_Tech]

I found that unless both machines were of a recent vintage, samba seems to hit a limit. Exmaple being my current computer AMD 2400XP running Linux 2.4.24, to a AMD 500 K6-2 running Linux 2.4.20 tops off about 1 MB sec on a 100 Mb/sec network. Contrast my current computer (2400 one) to a friends 2600XP running Win2K, I was seeing about 6-7 MB/sec. (and a 25% CPU usage...)

I have found that FTP seems to use the bandwidth up better if you want to test it. Computer xbox I can get 7-9 MB/sec on a 100 Mb/sec connection.

You might also look into some network bandwidth tools that just go to and from memory and are designed for testing network speeds.

How about checking the HD's on either end?

[_LORAX_]

Um... how about the obvious. How fast is the Hard Drives in both computers? 145Mbps = ~18MB/s which is approaching the sustained limit for many ATA100/133 drives these day.

So I would start there.

Re:How about checking the HD's on either end?

[Matthias Wiesmann]

Why not use iperf, which is meant for this usage?

Re:Use Iperf to test network bandwidth

[bohnsack]

Using IPerf to test your network bandwidth is easy:

[machine1]# iperf -s
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec

[machine2]# iperf -c machine1
Client connecting to machine1, TCP port 5001
TCP window size: 16.0 KByte (default)
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec

This, ~950 Mb/s, is around what you can expect from a 1500 MTU GigE network.

Samba, hard drives, pci bus

[molo]

A couple likely bottlenecks:

1. samba. The microsoft SMB or CIFS protocol is a big inefficient hog. Try transferring with FTP. The data is piped down a TCP stream, end of story.

2. hard drives. most hard drives can't push a gigabit/second from the platters (let alone write). Check out their sustained transfer speed (not burst cache). Also check out your bus medium. ATA-66 won't push a gigabit.

3. pci bus. Transferring data down the PCI bus from the disk controller and then back out the PCI bus to the network card means you need a 2x effective bandwidth. PCI can't hit 2 gigabit here. You might get better results with PCI Express.

Good luck.
-molo

Gigabit switching not for generic machines...

[Anonymous Coward]

With the increased bandwidth of Gigabit Ethernet, software routing on generic hardware is severely non-optimal these days.

• 32-bit/33Mhz PCI (which is the nice short PCI slots in all NON-server motherboards) is limited to 132MBytes/s transfer. Since a full 1Gbit = 128MBytes, you only can only get a theoretical 66MBytes per GBit port, since ALL traffic has to go back and forth from main memory, and thus has to cross the PCI bus twice.
• For the high-end 64-bit/66Mhz PCI slots available on server motherboards, you get a theoretical 528MBytes/s performance, which should be enough to run 2 simultaneous connections (even with some of the PCI bus collisions).
• The above holds even for dual-port NICs, since the traffic has to go back and forth to RAM, and can't just stay on the NIC.
• The size of the NIC's on-board buffer has a serious impact on performance, as this acts a temporary storage while the CPU deals with the network packet interrupt. If you have a small buffer, then you're going to force a lot of retransmits, as stuff comes in and overwrites the existing data while waiting for the CPU.
• Remember that for every packet incoming, there is an interrupt request sent to the CPU to deal with the incoming data. A rule of thumb from the Sun Solaris side of the house: You dedicated 1 full 400Mhz UltraSPARC II CPU to just servicing the interrupt requests from a single GBit ethernet card. Translated to the x86 world, that generally means that you'll run at least 25% CPU load on a 2GHz CPU while trying to service 1 GBit ethernet's full of network interrupts.
• If you have NICs which can use Jumbo Frames, these improve performance considerably, as they reduce the total number of packets (and thus, overhead) by a factor of 10.
• Linux's network stack is not fully optimized for GBit performance. The BSDs are better, but neither have had the obscene tuning that dedicated router/switch stacks have (such as Cisco's IOS).
• As mentioned above, the non-Server versions of Windows have similar limitations in their network stacks, which seem to limit network throughput to about 200Mbits/s, regardless of hardware. The various Server versions don't have this problem.
• Remember that you are doing ROUTING, when all you really want to do is SWITCHING. Routing is significantly more work for the CPU, since it involves packet inspection, and not just a MAC address table lookup and reforward.

You really need to use dedicated switches (as there are hardware ASICs that do this all at near-wire speeds, and eliminate all the potential problems above).

OP: Answers

[Glonoinha]

On your best day a random IDE drive is going to read or write 30 megs a second (average, on the fairly high side for anything short of SATA or nice SCSI) for completely sequential data in a large contiguous file; that's 240 megabits maximum throughput at the drive heads, or effectively 'wire speed'. That's assuming you are using relatively new hard drives in all these machines.

Throw in all the Samba and other protocol overhead, throw in the fact that you probably aren't running P4 3.2GHz boxes, in fact maybe much less, throw in the lack of a dedicated switch and all of a sudden getting 50% of your theoretical peak throughput (hard drive being the limiting factor, not network) isn't too harsh of a reality.

And it's a 'Windows' box, you stupid fuck. Maybe if Linux users (yea, I'm posting this in Mozilla on a RH9 install) would grow up and learn to spell the word 'Windows', Corporate America wouldn't instantly dismiss Linux users as a bunch of fucking retards. I spend a part of my work day trying to convince my boss that Linux is the choice of a new generation of professionals and every time someone says M$ or 'Windoze' I have to start over from ground zero. If you aren't part of the solution, you are part of the problem.