Finding the Bottleneck in a Gigabit Ethernet LAN?

guroove asks: "I have a small gigabit ethernet network at home, and I spent a lot of money getting gigabit NICs for all my computers and even bought cat 6 cabling. I only have 3 computers on the gigabit network (a Mac, a Windoze machine, and a Linux box) so instead of getting a switch, I triple NIC'd the Linux box, which I use as a gateway and a file server. After the network was complete, I wasn't satisfied with transfer rates, so I started a transfer of a very large file and found that the transfer rate was topping off at just over 145 Mbps (which is a far cry from 1000 Mbps). I'm wondering now where my bottleneck is. Is it the NICs? Are all gigabit NICs really giving us 1000 megabits per second? Is it the driver? Is it Samba? Could it be that the hard drives aren't fast enough? Does anyone have experience with gigabit home networking enough to know where the bottlenecks are? Does the current PCI technology even allow for bandwidth that high"

Comments

The Linux machine is acting as a router ?

[bungeejumper]

It is "entirely possible" that the Linux machine is acting as a router, switching all your traffic in C code. Not to mention it is probably sending traffic up and down the PCI bus, once at ingress and once at egress. The lookup of the IP destination address is probably using a whole lot of memory bandwidth, and if it's at all like a regular router, it's probably doing a full IP header Sanity check (using the IP CRC), version number and TTL decrement. After the TTL decrement, you would need to recompute the CRC. I would say the Linux machine is your bottleneck. Unless you could somehow get it configured as an ethernet switch, rather than a Layer 3 router.

How about checking the HD's on either end?

[_LORAX_]

Um... how about the obvious. How fast is the Hard Drives in both computers? 145Mbps = ~18MB/s which is approaching the sustained limit for many ATA100/133 drives these day.

So I would start there.

OP: Answers

[Glonoinha]

On your best day a random IDE drive is going to read or write 30 megs a second (average, on the fairly high side for anything short of SATA or nice SCSI) for completely sequential data in a large contiguous file; that's 240 megabits maximum throughput at the drive heads, or effectively 'wire speed'. That's assuming you are using relatively new hard drives in all these machines.

Throw in all the Samba and other protocol overhead, throw in the fact that you probably aren't running P4 3.2GHz boxes, in fact maybe much less, throw in the lack of a dedicated switch and all of a sudden getting 50% of your theoretical peak throughput (hard drive being the limiting factor, not network) isn't too harsh of a reality.

And it's a 'Windows' box, you stupid fuck. Maybe if Linux users (yea, I'm posting this in Mozilla on a RH9 install) would grow up and learn to spell the word 'Windows', Corporate America wouldn't instantly dismiss Linux users as a bunch of fucking retards. I spend a part of my work day trying to convince my boss that Linux is the choice of a new generation of professionals and every time someone says M$ or 'Windoze' I have to start over from ground zero. If you aren't part of the solution, you are part of the problem.