Slashdot Mirror
Finding the Bottleneck in a Gigabit Ethernet LAN?
guroove asks: "I have a small gigabit ethernet network at home, and I spent a lot of money getting gigabit NICs for all my computers and even bought cat 6 cabling. I only have 3 computers on the gigabit network (a Mac, a Windoze machine, and a Linux box) so instead of getting a switch, I triple NIC'd the Linux box, which I use as a gateway and a file server. After the network was complete, I wasn't satisfied with transfer rates, so I started a transfer of a very large file and found that the transfer rate was topping off at just over 145 Mbps (which is a far cry from 1000 Mbps). I'm wondering now where my bottleneck is. Is it the NICs? Are all gigabit NICs really giving us 1000 megabits per second? Is it the driver? Is it Samba? Could it be that the hard drives aren't fast enough? Does anyone have experience with gigabit home networking enough to know where the bottlenecks are? Does the current PCI technology even allow for bandwidth that high"
tell me the linux machine's not an old 486 you had lying under the stairs...
That man tried to kill mah Daddy
It is "entirely possible" that the Linux machine is acting as a router, switching all your traffic in C code. Not to mention it is probably sending traffic up and down the PCI bus, once at ingress and once at egress. The lookup of the IP destination address is probably using a whole lot of memory bandwidth, and if it's at all like a regular router, it's probably doing a full IP header Sanity check (using the IP CRC), version number and TTL decrement. After the TTL decrement, you would need to recompute the CRC. I would say the Linux machine is your bottleneck. Unless you could somehow get it configured as an ethernet switch, rather than a Layer 3 router.
Well, the way to find the bottleneck is obvious. First try a transfer from linux to the mac. Then try a transfer from linux to the peecee. If both of those are fast transfers, then its time to start thinking about your linux box's bus. If one is fast and one is slow, go to town on the slow leg.
Putting 2 peer links in the linux box seems like it might have been a mistake, since you're now not able to add new computers without buying new nics for the linux box. Buying a hub might have been better, but what do I know? Maybe gigabit nics cost $1 and hubs cost $1,000.
Your speed is also dependent on protocol, driver, and OS overhead. Check those things before you worry about such a simple hardware setup.
You didn't give any information about your protocol so that leads me to believe that you haven't considered TCP vs. UDP, for example.
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
a gigabit 128 megabytes
If you are getting 145 megabytes / second, that's damn good.
I found that unless both machines were of a recent vintage, samba seems to hit a limit. Exmaple being my current computer AMD 2400XP running Linux 2.4.24, to a AMD 500 K6-2 running Linux 2.4.20 tops off about 1 MB sec on a 100 Mb/sec network. Contrast my current computer (2400 one) to a friends 2600XP running Win2K, I was seeing about 6-7 MB/sec. (and a 25% CPU usage...)
I have found that FTP seems to use the bandwidth up better if you want to test it. Computer xbox I can get 7-9 MB/sec on a 100 Mb/sec connection.
You might also look into some network bandwidth tools that just go to and from memory and are designed for testing network speeds.
1000 megabits would = about 125 megabytes (1000/8). If you are getting 145 megabits a second that would be about 18 megabytes a second and would be very horrible speeds. If on the other hand you are getting 145 megabytes a second you are doing pretty damn good I think.
Um... how about the obvious. How fast is the Hard Drives in both computers? 145Mbps = ~18MB/s which is approaching the sustained limit for many ATA100/133 drives these day.
So I would start there.
I have several harddrives that top out around 14-20Megabytes per second, which turns into roughly the speed rating you are talking about.
I doubt your running out of PCI bandwidth.
It could be the latency, or that you have a poorly tuned network stack. I know that using NFS, getting 12-15Mbit/sec was considered pretty good given the inherient latency of the protocol.
I had similar problems no matter what protocol I was using FTP, HTTP, or scp. What I found was that, I needed to use a network speed tool: NPtcp, which is part of the netpipe tool set.
http://www.scl.ameslab.gov/Projects/old/ClusterCoo kbook/nprun.html
The other thing is to figure out if your cards support Jumbo frames. If they do, it can be a boon to go change your MTU, and modify specific parameters in your TCP stack, and in your application to change the socket options used (specifically, to use a packet size larger then 8k). I'm not sure how to do this under windows, but I've found it readily available under Linux on google searches.
More information is more useful. Knowing what chipset it is based off of, which drivers you are using, what OS, would be mighty helpful to helping you solve your problem.
Kirby
I might be good to start by measuring your network's performance, without hard drives or application software in the loop. I'd suggest using IPerf to accomplish this. If you measure less than expected performance with IPerf, your problem is with your NICs, switch, or drivers. If IPerf reports OK numbers, start looking at Samba and your hard drives. The bus shouldn't be a problem, because even a lowly 32 bit 33 MHz PCI bus has a theoretical 1.056 Gb/s data rate.
Gigabit ethernet is fast, and it's very easy for your processor, your tcp stack, your driver, your card, your pci bus, etc... to bottleneck at less than gigabit speeds. It's pretty hard to tell you which without seeing the whole setup and analyzing it in place. It's also possible for the tcp protocol itself to bottleneck at a lower-than-gigabit speed if you don't tune various parameters to help it out. You can tell if it's a tcp bottleneck by trying multiple parallel transfers between the same pair of machines and checking to see if the aggregate bandwidth is significantly higher than a single transfer. If this turns out to be the case, you can look at various network tunable like buffer sizes and window sizes. Another related tunable is the MTU of your ethernet network. If ALL your cards (and your switches if you had any) support it, you can turn on Jumbo Frames and push 9000 bytes per ethernet frame instead of 1500, which can make a big difference in transfer speeds over a gigabit network.
11*43+456^2
A couple likely bottlenecks:
1. samba. The microsoft SMB or CIFS protocol is a big inefficient hog. Try transferring with FTP. The data is piped down a TCP stream, end of story.
2. hard drives. most hard drives can't push a gigabit/second from the platters (let alone write). Check out their sustained transfer speed (not burst cache). Also check out your bus medium. ATA-66 won't push a gigabit.
3. pci bus. Transferring data down the PCI bus from the disk controller and then back out the PCI bus to the network card means you need a 2x effective bandwidth. PCI can't hit 2 gigabit here. You might get better results with PCI Express.
Good luck.
-molo
Using your sig line to advertise for friends is lame.
PCI has a transfer peak rate of 133 MB/s, according to Wikipedia. This means you'll never get over 500 Mbps, since PCI is NOT full-duplex.
You could get better results by enabling the "Fast switching" options in the Linux Kernel; this may cut the box out of the net (can't remember), but you can try to see how this affects you network. If numbers rise, the the bottleneck is the CPU. Otherwise it's probably the PCI bus.
I just installed gigabit at my home network but sprung for a cheaper switch, the only problem with it is that it doesn't do jumbo framing, and here is a list of jumbo frame compatible hardware
to test your link speeds you should not be using Samba, instead use ttcp (windows version,java version, or your favorite distro should have a copy, I know it's in the ports of FreeBSD)
To effectively use a gigabit pipe, especially for sustained transfers, you need gigabit integrated into the motherboard and a fast RAID or fast Serial ATA. You also cannot expect commodity hardware to route gigabit through PCI, get a switch.
Make sure you have jumbo frames enabled on all the machines. Note that you need OS X 10.2.4 or newer on your Mac to use the 9K frames.
Also, Samba might not be the best choice for Mac Linux transfers. You'll probably be better off using NFS version 3 between the Mac and Linux box. Both machines should speak NFS natively and not require any additional software.
I just read an article in this month's CPU (computer power user) about the limitations of TCP on high-speed networks. Apparently, due to the way TCP adjusts to available bandwidth, it can never exceed 450Mbps or so.
TCP was designed for the 10/100 Mbps ethernet or less. To make effective use of faster networks, you need to use an enhanced version of TCP, of which there are several. I don't think any are mainstream yet, though.
That aside, as others have pointed out, your PCI bus and hard drives will bottleneck sooner than TCP will.
This is not a troll. Workstation versions of Windows, like Windows NT Workstation, Windows 98, Windows ME, Windows 2000 Professional and Windows XP have a crippled TCP/IP stack that make them move data slower than they really should. Using a "Server" edition of windows like Windows NT Server, Windows 2000 Server and Advancd Server and any edition of Windows 2003 will make any network throughput markedly faster.
With the increased bandwidth of Gigabit Ethernet, software routing on generic hardware is severely non-optimal these days.
- 32-bit/33Mhz PCI (which is the nice short PCI slots in all NON-server motherboards) is limited to 132MBytes/s transfer. Since a full 1Gbit = 128MBytes, you only can only get a theoretical 66MBytes per GBit port, since ALL traffic has to go back and forth from main memory, and thus has to cross the PCI bus twice.
- For the high-end 64-bit/66Mhz PCI slots available on server motherboards, you get a theoretical 528MBytes/s performance, which should be enough to run 2 simultaneous connections (even with some of the PCI bus collisions).
- The above holds even for dual-port NICs, since the traffic has to go back and forth to RAM, and can't just stay on the NIC.
- The size of the NIC's on-board buffer has a serious impact on performance, as this acts a temporary storage while the CPU deals with the network packet interrupt. If you have a small buffer, then you're going to force a lot of retransmits, as stuff comes in and overwrites the existing data while waiting for the CPU.
- Remember that for every packet incoming, there is an interrupt request sent to the CPU to deal with the incoming data. A rule of thumb from the Sun Solaris side of the house: You dedicated 1 full 400Mhz UltraSPARC II CPU to just servicing the interrupt requests from a single GBit ethernet card. Translated to the x86 world, that generally means that you'll run at least 25% CPU load on a 2GHz CPU while trying to service 1 GBit ethernet's full of network interrupts.
- If you have NICs which can use Jumbo Frames, these improve performance considerably, as they reduce the total number of packets (and thus, overhead) by a factor of 10.
- Linux's network stack is not fully optimized for GBit performance. The BSDs are better, but neither have had the obscene tuning that dedicated router/switch stacks have (such as Cisco's IOS).
- As mentioned above, the non-Server versions of Windows have similar limitations in their network stacks, which seem to limit network throughput to about 200Mbits/s, regardless of hardware. The various Server versions don't have this problem.
- Remember that you are doing ROUTING, when all you really want to do is SWITCHING. Routing is significantly more work for the CPU, since it involves packet inspection, and not just a MAC address table lookup and reforward.
You really need to use dedicated switches (as there are hardware ASICs that do this all at near-wire speeds, and eliminate all the potential problems above).Over unencrypted AFP (Mac native filesharing protocol) from one Xserve G5 2.0 DP to another, via a $100 Gigabit switch, I got 33MB/s, or almost exactly 2GB/minute.
I did find that turning on the encryption option blew that number to hell: I didn't play with it long, but it appeared that it cut my speeds to about 10% of that.
The servers in question didn't have much else going on, etc. Not truly scientific, as that rate was good enough for me. And enough to show me that I should not use encryption when I do large size file transfers.
--
$tar -xvf
with jumbo packets off:
- 22 MB/s (~220 Mb/s) disk-to-disk (FTP)
- 35 MB/s RAM-to-RAM (FTP)
with 9k jumbo packets on:
- 22 MB/s (~220 Mb/s) disk-to-disk (FTP)
- 60 MB/s RAM-to-RAM (FTP)
never got more than 60 MB/s sustained
Suggestions:
- get a high speed server
- get a high speed server-grade NIC card, that (at least) offloads checksums from driver to NIC
- turn on jumbo packets (warning: couldn't make it work between AIX-Windows 2000 on a Dell 2650)
- if using a switch, get a good switch that supports jumbo packets
If all your hardware isn't capable of supporting and configured to use jumbo frames it will be hard to come anywhere close to saturation point.
On your best day a random IDE drive is going to read or write 30 megs a second (average, on the fairly high side for anything short of SATA or nice SCSI) for completely sequential data in a large contiguous file; that's 240 megabits maximum throughput at the drive heads, or effectively 'wire speed'. That's assuming you are using relatively new hard drives in all these machines.
Throw in all the Samba and other protocol overhead, throw in the fact that you probably aren't running P4 3.2GHz boxes, in fact maybe much less, throw in the lack of a dedicated switch and all of a sudden getting 50% of your theoretical peak throughput (hard drive being the limiting factor, not network) isn't too harsh of a reality.
And it's a 'Windows' box, you stupid fuck. Maybe if Linux users (yea, I'm posting this in Mozilla on a RH9 install) would grow up and learn to spell the word 'Windows', Corporate America wouldn't instantly dismiss Linux users as a bunch of fucking retards. I spend a part of my work day trying to convince my boss that Linux is the choice of a new generation of professionals and every time someone says M$ or 'Windoze' I have to start over from ground zero. If you aren't part of the solution, you are part of the problem.
Glonoinha the MebiByte Slayer
before you went out and bought all that gear?
Give Iperf a try. I used it for benchmarking my home gigabit LAN. It's got multiple versions available for many platforms (as well as source code). It generates data and sends it, not requiring any hard drive access thereby taking drive speed out of the equasion. This blog site also has some more info.
Most PCs nowdays have the standard 32-bit 66mhz PCI slots and 64-bit PCI. Of course, the 66Mhz PCI slot will top out at transfer speeds of ~266Mbytes/s. 64 bit slots will of course be a bit faster.
Worse yet, if you are running these gigabit nics in a 33Mhz PCI slot, you will get less than 133Mbyte/s transfer rates across the bus.
So my advice to you is that you investigate what kind of speeds and slots your cards use. Are they on their optimal slot type? Are they actually using bus mastering?
You should easily be able to get near top speed if place the cards in their optimal slots and actually configure your drivers right. Also, increasing the MTU to a higher value will probably significantly improve your large data transfer speeds.
Internet traffic is
Voltaire: God is dead.
God: Voltaire is dead!
High latency has nothing to do with bandwidth. TCP easily scales up to 10GbE as long as the latencies are low. (And even with high latencies it's simply a matter of increasing send/receive window sizes). Like the parent said: Learn something before posting.
As a lot of people have pointed out, off-the-shelf PCs aren't a good choice for gigabit Ethernet switching and routing regardless of the OS, and you can't really take advantage of true full-duplex Gigabit Ethernet on a standard consumer PCI bus. Still, you can do better than 145 Mbit/sec.
I've been using a LinkSys EG008W switch on my home network, and it's a real bargain. It is a true switch (not a hub), costs less than $200, and all eight ports are capable of autosensing gigabit-capable hardware. Not all so-called "Gigabit" hubs are created equal; some of them only work in half-duplex mode, some of them only have gigabit capability on their uplink ports; some of them slow down to 100 megabit/sec if any of their ports are connected to 100-megabit devices.
The Linksys's big drawback is its fan noise. It is insanely annoying. I owned mine for about 24 hours before I opened it up and dropped the voltage to the fan with a three-terminal regulator IC. I cut a hole in the top to improve the airflow at the lower fan speed, and it's perfectly unobtrusive now. (No, I don't remember what voltage I ended up running the fan at, unfortunately.) If you're either (a) deaf; (b) located at least a couple of rooms away from your network closet; or (c) handy with a soldering iron and indifferent to manufacturer warranties, the EG008W would be an ideal piece of hardware for your situation.
Dahlmann tightly grips the knife, which he may have no idea how to use, and steps out into the plain.
1. What you are seeing is average rate.
TCP goes into congestion avoidance and fast retransmit and recovery (for example TCP-Reno). The connection might be touching maximum rate but you are not seeing it!
2. If your file transfer is over a large round trip time then TCP rate gets dilated: (File-Size / N*RTT)
where RTT is round trip time and N is the number of round trips required to transport the page.
3. If you are downloading the file, from "somewhere out there" then the bottleneck might be "somewhere out there" and not in your setup. Please recall, the bottleneck will cause TCP to de-accelerate whenever it sees a packet loss.
2/100 dollars.
Voltaire: God is dead.
God: Voltaire is dead!
you could make a 512 MB (or less) ram disk and ... ...
copy i file from one ram disk to another thru
gigabit. this would tell you if the NICs are
really capable to do it
another bottleneck might be the switch/hub
did you do the cableing correctly?
One of the best things you could do is configure SNMP on all 3 boxen. After that, run MRTG to figure out what's happening on the wire. If you made the connectors yourself (as opposed to factory-made cables), doublecheck to see if the connectors fall within the CAT-VI spec. How much of the pair is untwisted? How far into the connector is the shield/plenum seated? Is the wire kinked or does it have sharp bends anywhere? Is the wire running next to power? All these things can cause the signal to be degraded.
Get a good disk benchmark and run that on all 3 boxen. Find out if the disks can sustain traffic at the 1000mb (125MB/s isn't going to come from a single IDE disk) rate. Also, keep in mind that core logic switching from a PCI RAID card to a PCI NIC will eat up some bandwidth.
Finally, benchmark each link individually with a server benchmark tool. Put a 1GB file on the linux box and see how long it takes to transfer to each of the clients. Then do the same file from the clients back to the server.
On a side note, SOHO GB switches shouldn't cost more than $100. But, if your disks cannot keep up with the rate, it won't matter.
On a side, side note, we have tools that show a lot of things about our hardware. Why are there no tools showing the used bandwidth of the PCI/AGP/memory bus. Troubleshooting this prob would be simple if you could see that a specific bus is being saturated.
I'd rather you do it wrong, than for me to have to do it at all.
25 years ago: Finding the Bottleneck in my 4mps Token Ring LAN - Hello, even though I'm supposed to be getting 4mps, I'm lucky when I can see sustained 1mps speeds...
15 years ago: Finding the Bottleneck in my 10mps Ethernet LAN - Hello, even though I'm supposed to be getting 10mps, I'm lucky when I can see sustained 2mps speeds...
10 years ago: Finding the Bottleneck in my Fast Ethernet LAN - Hello, even though I'm supposed to be getting 100mps, I'm lucky when I can see sustained 10mps speeds...
1 year ago: Should we rejoice yet? - Having noted the nearly a decade its been since we've had to explain to yet another ijit why theoretical maximum bandwidth is almost never attained, I say it's time to party. Finally, the deluge is over, what do you think?
I only have 3 computers on the gigabit network (a Mac, a Windoze machine, and a Linux box) so instead of getting a switch, I triple NIC'd the Linux box, which I use as a gateway and a file server.
;-)
Why would anyone put 3 gigabit NIC's in a box and route within a 3-node network when you can buy a low-end gigabit switch for less than $100??? Were you hung over the day you "designed" this??? Also note that using cable of a higher grade than required by the spec typically doesn't give you some magic speed increase
I just installed the 8-port version of that switch (~$110) at home, and I find that I get ~350mbps between a G4 tower and a G4 PowerBook, without tweaking any IP options, and on my existing Cat 5 wiring.
just get a gigabit switch.
i'm not trying to be a dick here, but your a fucking moron if you think you can use elmers glue and duct tape to build a high speed network! gigabit needs gigabit cards and gigabit switches period, not haveing these is effectively taking the giga out of the bit.
secondly, if your saying that it was cheaper for you to get 5 gigabit cards that it was to get 3 gigabit cards and a 4 port gigabit switch, then a lot of your problem is problably weak gigabit cards. you didn't but the 12$ ones on the internet did you? those should be labeled 1/3gigabit, their processors arent capable of enough transactions, and some actually offload onto the CPU like some sick "winethermodem"
i run a gigabit network at my home, i have 4 desktop machines on it, 2 of them with INTEL gigabit built into the motherboard, and the other 2 with intel PCI cards. i can easily transfer using nfs at 700mpbs, which sounds fair to me after TCP/IP overhead. my samba results are a bit less and around 600-650mpbs.
also. every one of these machines is an XP1600+ or faster, except for my notebook, which is a celeron2.4 and is using a PCMCIA gigabit card from 3Com. The laptop is slower on the network with about 400mbps with samba, which is most likely a limitation of the 3com card combined with the PCMCIA bus.
--
i appologize for cursing, but please read that paragraph again. you need to build things within spec(or above) to get the stated performance, gigabit is not made to be strung nic->nic->nic and routed with standard routing software. Your PCI bus, your nics, your memory and CPU, and your un-tuned routing are problably ALL adding up to your week transfer rates.
A noname 5 port gigabit switch costs about 70 EUR (8 ports for 80 EUR, 1 EUR is about 1 US $). A gigabit ethernet adapter costs about 12 EUR. You currently need two extra cards in the Linux box, 24 EUR. To get a "Linux 5 port switch", you need 4 extra cards, 48 EUR. For a "Linux 8 port switch", you need 7 extra cards, 84 EUR, and perhaps another mainboard. If you look only at the hardware costs, the break even is at about 6 ports. But you trade money against performance, as you already know. Your ordinary PC can't handle the bandwidth of two gigabit connections, and it sure can not handle 8 connections. Looking at the AC power, your PC will draw between 50W (idle) and 300W (working), blowing most of it as heat through a lot of perhaps noisy fans. A real standalone switch is usually fanless, needs about 10W, and can handle 3 or more gigabit connections without trouble.
BTW: Why do you think that you would need a gigabit ethernet for just three home PCs? You have no device (except for /dev/null and ramdisks) that can source or sink a gigabit stream. Your internet connection is almost sure slower than good old ethernet (10 MBit/s). And I know from my own experiments that looking big videos on Windows (667 MHz) from a Linux fileserver (2x 233 MHz, ATA software RAID) is no problem at 100 MBit/s with only three PCs running.
Did you know that you can mix 10 MBit/s, 100 MBit/s and 1 GBit/s devices on a gigabit switch? You know that gigabit adapters can fall back to 100 MBit/s to work with 100 MBit/s devices (crossover cable, hub, 100 MBit/s switch, ...) and even to 10 MBit/s for old devices? 100 MBit/s adapters can also fall back to 10 MBit/s.
Tux2000
Denken hilft.