Slashdot Mirror
Gigabit Networking for the Home?
The Clockwork Troll asks: "I've had a whole-house audio/video distribution project on the back-burner for a while now. As gigabit networking hardware prices come down to earth, I'm tempted to jump on the 1000BaseTX bandwagon. As far as I can tell though, the current crop of consumer-priced hardware/software doesn't address a couple key issues, namely: fragmenting jumbo frames for the benefit of legacy clients - this is critical as some of the devices on my network will not tolerate the 9000+ byte Ethernet frames which are needed to get the most out of gigabit; and OS support - do Linux and Windows require much tweaking to take advantage of gigabit? Will most drivers automatically optimize themselves? A Google search didn't reveal too much consensus, especially on hardware choices. What switches and software configurations have Slashdot readers been using for home gigabit networks, in particular mixed ones (100/1000BaseTX?"
In Mac OS X, there's a setting right in the Network Preference Pane that is under "Ethernet" and it allows you to scale up the packet size depending on the immediately aparent network appliances. I haven't been able to use this feature because:
A: Some clients have nice network hardware, but legacy copper
B: Some clients have gig copper, but not enough hardware
I can't wait to see the transfer rates on Gig with Jumbo packets though. *Drool*
I got nothin'.
I've got an Abit motherboard with Intel gigabit built in and WindowsXP loaded on it. My GF has a Powerbook with gigabit built in. We bought the cheapest gigabit switch we could find. We got Cat 6 cable.
:-).
Everything was autodetected and the speed improvement over 100mbit was dramatic. Highest performance increase I've ever gotten for doing basically zero work (I did plug in the cables all by myself
Now, this obviously doesn't answer all your questions, but for anyone out there who doesn't have legacy issues all I can say is go for it, it's a no-brainer.
BTW, I use a Linksys WAP-Router for internet. It didn't so much as burp when we plugged it into the gigabit switch.
TW
I think the biggest thing about gigabit is that PCI isn't really fast enough to support it. You can shovel 133MB/second over a PCI bus, or 1064Mb.... very slightly more than a gigabit, but that's with NOTHING else happening on the bus. Generally, since the hard drive controller is also on the Southbridge, I think about the best you're going to get off most PCs, even very, very fast ones, is about 300 megabits sustained.
To really take advantage, you're going to need machines that run the network card off the Northbridge. Presumably, PCI-Express network cards will also keep up pretty easily. From what I can see, you're probably best to wait another year to eighteen months before upgrading; by then, PCI-X should be pretty common, and gigabit networking shouldn't be very expensive.
Note that I don't have any direct experience with gigabit: these are just back-of-the-envelope calculations. I could be completely off, so pay attention to replies.
I have a mixed network and have not had any problems with speed or the switches flaking out.
I have 3com gigabit cards in three computers and a 3com 100Mb card in one.
One gigabit machine is a redhat 8 machine that is used as the network attached storage (NAS) box feeding media throughout the house and acting as the DNS for the house (This is so much faster than relying on your ISP!) and to filter packets for the kids computer (Damn Pr0N!)
One gigabit machine is my personal desktop.
One gigabit machine is in the family room sucking media from the NAS.
The 100MB machine is upstairs and the kids use that one.
The gigabit machines are plugged into a LanReady gigabit switch that I bought for 60 bucks Ebay.
The 100MB machine is plugged into a 3com superstack.
Both switches are then plugged into the cable router.
Speeds between the gigabit machines average 50 Meg a second depending how large the files are and if it's streaming or copying, The 100Mb box pulls 7-8 MB a sec from the others.
I'm happy with the speed.
Karma means nothing to me, so suck it...
Tests using PCI Gigabit chips (e.g., broadcom, 3com, intel) get around 500Mbps or so.
/. story! Of course, that controller is attached to the processor by a 6.4GB/s link!
Intel CSA attached gigabit chips (on Intel chipset motherboards only) perform better. CSA is a dedicated link from the northbridge to a gigE controller.
Of course, nForce3 250Gb integrates gigE inside, and gets over 800Mbps performance. See the preceeding
Also, PCI-X != PCIe. PCIe (PCI Express) is the upcoming high speed serial version of PCI that operates on a point-to-point basis. PCI-X is the extended faster variant of 64-bit 66MHz PCI running at up to 133MHz (1GB/s PCI essentially) in a bus configuration.
Al Qaeda has ninjas!
We use GigE fiber for our server networks, and pass up between 400Mb/s and 600Mb/s on high traffic days from each one.
The one thing I can say is that you'll probably never use it. There's really no need at this time. most protocols aren't any good at sucking up that much bandwidth on a single stream.
I've had many people prove this to me. They'll transfer files as single transfers. They can use up to about 10Mb/s. But if they transfer lots of files, they can use lots more. Try it through a switch that you can monitor bandwidth on. Through FTP, SMB, SCP, or whatever, you won't use up 100Mb/s. But, running multiple concurrent sessions, you can try to come close.
Heroinewarrior has a library called "firehose", which uses up all the available bandwidth, and will stripe across multiple connections to use up more. So, if you have 3 100Mb/s cards in a machine, you can come close to transfering at 300Mb/s.
You should also consider the other factors. Can your machine really send that fast? Is your hard drive fast enough to send over 100Mb/s ?? A nice fast SCSI drive, or a SATA drive can do it, but most IDE drives will fall short (specs be damned, try it in real life).
I transfer stuff around on the GigE lan all the time. We do exceed 100Mb/s, but it's usually with multiple machines.
The highest bandwidth usage machines we have are voyeurweb.com . They send out 150Mb/s through TEQL (Linux kernel option) combined 100baseTX cards, with several copies of thttpd running.
thttpd is a web server that is very small, and works very efficently. Apache has one process per connection, but thttpd has one process for everyone. Well, at least theoretically. It was around 80Mb/s of regular web site files, that it started flaking out. So, we run 4 copies of it on seperate IP's and let it scream.
As for our network, I'll outline our largest network.
We have a 1Gb/s uplink to Level3. This goes to a Cisco Catalyst 3508 (8 GBIC ports).
The remaining 7 GBIC ports go to 7 switches, mostly Cisco Catalyst 3550-48 (48 100Mb/s ethernet, 2 GBIC), and the servers are attached to the 100Mb/s ports. We have one Dell switch, which does 1000baseTX on all the ports, and a few machines with 1000baseTX cards. They can't pull anything resembling 1000Mb/s between each other. it simply doesn't happen. Honestly, doing transfers through http, ftp, or scp doesn't ever use over 100Mb/s on individual transfers. Sure, we can do it with multiple concurrent transfers, but at home, how many hundred or thousand users are you really trying to supply?
For home, you'll never use it. 100Mb/s is usually overkill. I set up my house with 802.11b, and at 11Mb/s peak, I see no difference than my old house, where we had copper run to every room and a Catalyst 2924 managing it. 11Mb/s is more than sufficent for a home network.
Spend your money on a *GOOD* 100Mb/s switch. I highly recommend Cisco, like a 2924, which you should be able to get relatively cheap used. Even if you put GigE cards in the machines, you can at least monitor your bandwidth now, and see what you really use. If you start flat-lining at 100Mb/s (bandwidth graphs make things really obvious), then you could consider upgrading.
Serious? Seriousness is well above my pay grade.
A friend of mine just went nuts when he found out about a new switch from SMC, the SMC8508T. While it's unmanaged, it offers non-blocking architecture across the entire line as well as support of jumbo frames up to 9K, which is extremely unusual for SOHO stuff. Not even a lot of expensive Cisco stuff does jumbo frames. And he paid $150 for it.
Why should you care about jumbo frames? I found this nice guide about that here.
-R
I wired my whole house with CAT5, which I assume wouldn't handle anywhere near gigabit speeds.
The nice thing about GbE is that you can still use your old CAT5 (if it isn't too low quality).
If you buy new cables, you should get CAT5e - basically the same as CAT5, but tested for 125MHz, while CAT5 is only tested for 100MHz. (GBe combines 4 bi-directional wire-pairs with 125MHz each to achieve 1000 Mb/s)
Boy this turned into a bit of a tome.
/proc/sys/net/ipv4/tcp_rmem /etc/samba/smb.conf
For a switch I went with an 8 port SMC EZSwitch 8508T. I chose it since:
1. It supports jumbo frames. According to my testing it will pass ethernet packets up to 9212 bytes which should correspond to a 9198 byte MTU.
2. It doesn't have a cooling fan. A definate plus since in my experience the little fans in switches such as this can become quite annoying as they age.
3. It comes with rack mount ears.
4. It's affordable. I purchased it from Securemart.com for $139.31 shipped. Ordered it Thursday or Friday, it arrived Monday or Tuesday.
As to NICs, one of my PCs already had an Intel gigabit port on the motherboard. In addition I purchased 4 more Intel Pro 1000/MT Desktop Adapters. Since:
1. They have good driver support on both Linux and Windows.
2. They support jumbo frames. Supposedly up to around 16000 bytes.
3. They're supposed to be pretty fast/efficient. It's kind of dated but you can find a comparison of some 32-bit gigabit NICs here.
4. They'll do 66Mhz if your motherboard supports it and of my systems does.
5. They have DOS NDIS2 drivers so I can use Ghost to make/restore images over the network.
One I purchased through Intel's evaluation program for $35.31 shipped. As I recall it took over a week to show up. The other three I ordered from OnlineMicro for $28 each plus $11.32 shipping. Be sure to change the shipping option from ground to 2 day air if you order more than 1, it's cheaper. They shipped them out the day of my order and they arrived on time.
One of the Intel NICs died about 4 hours after I installed it. I swapped it with another and the replacement has been working fine for a few weeks now. I ran the diagnostics on it and other all but the link test passed. When the OS is booted up the switch shows no link lights but sometimes when the PC is off the link lights do come on. I've also tried it in another PC where it exhibits similar symptoms. I haven't yet contacted Intel about getting it replaced.
I spent a lot of time tweaking various things. Some findings:
1. With default SO_RCVBUF sizes a MTU in the neighborhood of 4000 or so bytes seems to get about the best network/application wide throughput. Specifically the otherwise fast NF7-S system below would lose almost 50% throughput with 9000 byte MTUs with the default SO_RCVBUF size. Linux to Linux lost around 30% as I recall.
In theory you can change the default SO_RCVBUF size on linux by echoing appropriate values to:
Other than that you appear to have to change this setting in each individual application. One application of note that allows you to easily make this change is samba. See your:
2. If you crank the SO_RCVBUF size up to 200ish k or more then a 9000ish byte MTU can eek out another 5ish percent more bandwidth. Thus for the moment I've decided to just stick with 4076.
3. MTUs that are not of a size of the form 8x+4 cause Linux to behave oddly when it performs path MTU discovery. Namely for jumbo sizes that don't fit that form the discovery decides that the PMTU is 1492. You can read more detail about it in a Usenet post I made here. I still don't have a good picture of what'
Gigabit Ethernet actually uses the same frequency (100MHz) as 100Mbit ethernet. Cat5 and Cat5e is both rated for 100MHz. Actually, I wonder if you can get Cat5 but not Cat5e any more. When I wired my house, Cat5e was the minimum spec being sold.
The difference with Gig-E is that it uses all four pairs in the wire (100Mbit only uses 2 pairs) and it has a different linecode that allows more bits per baud.
Oolite: Elite-like game. For Mac, Linux and Windows