Slashdot Mirror
Finding the Bottleneck in a Gigabit Ethernet LAN?
guroove asks: "I have a small gigabit ethernet network at home, and I spent a lot of money getting gigabit NICs for all my computers and even bought cat 6 cabling. I only have 3 computers on the gigabit network (a Mac, a Windoze machine, and a Linux box) so instead of getting a switch, I triple NIC'd the Linux box, which I use as a gateway and a file server. After the network was complete, I wasn't satisfied with transfer rates, so I started a transfer of a very large file and found that the transfer rate was topping off at just over 145 Mbps (which is a far cry from 1000 Mbps). I'm wondering now where my bottleneck is. Is it the NICs? Are all gigabit NICs really giving us 1000 megabits per second? Is it the driver? Is it Samba? Could it be that the hard drives aren't fast enough? Does anyone have experience with gigabit home networking enough to know where the bottlenecks are? Does the current PCI technology even allow for bandwidth that high"
Well, the way to find the bottleneck is obvious. First try a transfer from linux to the mac. Then try a transfer from linux to the peecee. If both of those are fast transfers, then its time to start thinking about your linux box's bus. If one is fast and one is slow, go to town on the slow leg.
Putting 2 peer links in the linux box seems like it might have been a mistake, since you're now not able to add new computers without buying new nics for the linux box. Buying a hub might have been better, but what do I know? Maybe gigabit nics cost $1 and hubs cost $1,000.
I have several harddrives that top out around 14-20Megabytes per second, which turns into roughly the speed rating you are talking about.
I doubt your running out of PCI bandwidth.
It could be the latency, or that you have a poorly tuned network stack. I know that using NFS, getting 12-15Mbit/sec was considered pretty good given the inherient latency of the protocol.
I had similar problems no matter what protocol I was using FTP, HTTP, or scp. What I found was that, I needed to use a network speed tool: NPtcp, which is part of the netpipe tool set.
http://www.scl.ameslab.gov/Projects/old/ClusterCoo kbook/nprun.html
The other thing is to figure out if your cards support Jumbo frames. If they do, it can be a boon to go change your MTU, and modify specific parameters in your TCP stack, and in your application to change the socket options used (specifically, to use a packet size larger then 8k). I'm not sure how to do this under windows, but I've found it readily available under Linux on google searches.
More information is more useful. Knowing what chipset it is based off of, which drivers you are using, what OS, would be mighty helpful to helping you solve your problem.
Kirby
Gigabit ethernet is fast, and it's very easy for your processor, your tcp stack, your driver, your card, your pci bus, etc... to bottleneck at less than gigabit speeds. It's pretty hard to tell you which without seeing the whole setup and analyzing it in place. It's also possible for the tcp protocol itself to bottleneck at a lower-than-gigabit speed if you don't tune various parameters to help it out. You can tell if it's a tcp bottleneck by trying multiple parallel transfers between the same pair of machines and checking to see if the aggregate bandwidth is significantly higher than a single transfer. If this turns out to be the case, you can look at various network tunable like buffer sizes and window sizes. Another related tunable is the MTU of your ethernet network. If ALL your cards (and your switches if you had any) support it, you can turn on Jumbo Frames and push 9000 bytes per ethernet frame instead of 1500, which can make a big difference in transfer speeds over a gigabit network.
11*43+456^2
I just installed gigabit at my home network but sprung for a cheaper switch, the only problem with it is that it doesn't do jumbo framing, and here is a list of jumbo frame compatible hardware
to test your link speeds you should not be using Samba, instead use ttcp (windows version,java version, or your favorite distro should have a copy, I know it's in the ports of FreeBSD)
I created a testfile:
500+0 records in
500+0 records out
Copied the file to another box over NFS:
- sync; THETIME=`date +%M:%S:%N`; cp testfile
/mnt/video/ -v ; sync; THETIME2=`date +%M:%S:%N`; echo -e "$THETIME2\n$THETIME"
`testfile' -> `/mnt/video/testfile' 40:26:402691000 39:26:956812000 = ~60sThen deleted, and sent the same file over samba to the same drive:
`testfile' -> `/mnt/video/testfile'
53:09:727205000
50:53:194652000 = ~136s
I had suspected samba would be slower, but thats an order of magnitude slower. Did I do something wrong, or is the difference really that large? I repeated the test afterwards and got very nearly the same results.
I just read an article in this month's CPU (computer power user) about the limitations of TCP on high-speed networks. Apparently, due to the way TCP adjusts to available bandwidth, it can never exceed 450Mbps or so.
TCP was designed for the 10/100 Mbps ethernet or less. To make effective use of faster networks, you need to use an enhanced version of TCP, of which there are several. I don't think any are mainstream yet, though.
That aside, as others have pointed out, your PCI bus and hard drives will bottleneck sooner than TCP will.
This is not a troll. Workstation versions of Windows, like Windows NT Workstation, Windows 98, Windows ME, Windows 2000 Professional and Windows XP have a crippled TCP/IP stack that make them move data slower than they really should. Using a "Server" edition of windows like Windows NT Server, Windows 2000 Server and Advancd Server and any edition of Windows 2003 will make any network throughput markedly faster.
Give Iperf a try. I used it for benchmarking my home gigabit LAN. It's got multiple versions available for many platforms (as well as source code). It generates data and sends it, not requiring any hard drive access thereby taking drive speed out of the equasion. This blog site also has some more info.