Slashdot Mirror
Gigabit Networking for the Home?
The Clockwork Troll asks: "I've had a whole-house audio/video distribution project on the back-burner for a while now. As gigabit networking hardware prices come down to earth, I'm tempted to jump on the 1000BaseTX bandwagon. As far as I can tell though, the current crop of consumer-priced hardware/software doesn't address a couple key issues, namely: fragmenting jumbo frames for the benefit of legacy clients - this is critical as some of the devices on my network will not tolerate the 9000+ byte Ethernet frames which are needed to get the most out of gigabit; and OS support - do Linux and Windows require much tweaking to take advantage of gigabit? Will most drivers automatically optimize themselves? A Google search didn't reveal too much consensus, especially on hardware choices. What switches and software configurations have Slashdot readers been using for home gigabit networks, in particular mixed ones (100/1000BaseTX?"
Of course, if your needs are more extensive you may need something more...
Al Qaeda has ninjas!
Hard as I try, I can't imagine ever having enough stuff in my house to warrant gigabit. Damn.
In Mac OS X, there's a setting right in the Network Preference Pane that is under "Ethernet" and it allows you to scale up the packet size depending on the immediately aparent network appliances. I haven't been able to use this feature because:
A: Some clients have nice network hardware, but legacy copper
B: Some clients have gig copper, but not enough hardware
I can't wait to see the transfer rates on Gig with Jumbo packets though. *Drool*
I got nothin'.
I've got an Abit motherboard with Intel gigabit built in and WindowsXP loaded on it. My GF has a Powerbook with gigabit built in. We bought the cheapest gigabit switch we could find. We got Cat 6 cable.
:-).
Everything was autodetected and the speed improvement over 100mbit was dramatic. Highest performance increase I've ever gotten for doing basically zero work (I did plug in the cables all by myself
Now, this obviously doesn't answer all your questions, but for anyone out there who doesn't have legacy issues all I can say is go for it, it's a no-brainer.
BTW, I use a Linksys WAP-Router for internet. It didn't so much as burp when we plugged it into the gigabit switch.
TW
I think the biggest thing about gigabit is that PCI isn't really fast enough to support it. You can shovel 133MB/second over a PCI bus, or 1064Mb.... very slightly more than a gigabit, but that's with NOTHING else happening on the bus. Generally, since the hard drive controller is also on the Southbridge, I think about the best you're going to get off most PCs, even very, very fast ones, is about 300 megabits sustained.
To really take advantage, you're going to need machines that run the network card off the Northbridge. Presumably, PCI-Express network cards will also keep up pretty easily. From what I can see, you're probably best to wait another year to eighteen months before upgrading; by then, PCI-X should be pretty common, and gigabit networking shouldn't be very expensive.
Note that I don't have any direct experience with gigabit: these are just back-of-the-envelope calculations. I could be completely off, so pay attention to replies.
I've had a good experience with a Dell PowerConnect hub (the 8- or 16-port model, I forget which). It was quite inexpensive and claims to support Jumbo Frames (however I haven't actually gotten this to work; when I enlarge the frame size on Linux it loses the connection). Oh, and I had to disable one default feature on the hub (tree-spanning something or other) to get it to work.
For clients I use Intel gigabit cards (the 64-bit PCI "server" model). I wouldn't skimp here since indications are that cheap gigabit cards don't have any hope of getting wire speed. NFS file copies max out at 20-30MB/sec, but I know that is limited by my server's disk array. I did a test for raw network bandwidth (just sending zero bytes as fast as possible) and got around 60-80MB/sec.
Everything is connected to my existing Cat-5 cable with no problems. This includes several Linux systems, one Mac and one Windows PC.
I will caution you not to expect anything like gigabit wire speeds with typical clients. My Mac G4 in particular seems to have trouble getting good bandwidth (I think the problem is either the network stack or NFS client).
If anyone has a success story with jumbo frames, I'd love to hear about it. The only references I could find are for mega-dollar Cisco/Foundry type equipment.
Now I know this is /. but before everyone says "you don't need gigabit!" and "bah, who needs that kind of speed" gigabit ethernet is genuinely useful. Even copying 500mb files can take intolerably long when you want it done 4 minutes ago. If the poster wanted a bunch of nonsense about why he shouldn't do it and why its a dumb idea, he could have gone to Circuit city (they don't sell gigabit so they would try to sell him 10/100). Instead he asked us for an informed option and information on the matter.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
I have a mixed network and have not had any problems with speed or the switches flaking out.
I have 3com gigabit cards in three computers and a 3com 100Mb card in one.
One gigabit machine is a redhat 8 machine that is used as the network attached storage (NAS) box feeding media throughout the house and acting as the DNS for the house (This is so much faster than relying on your ISP!) and to filter packets for the kids computer (Damn Pr0N!)
One gigabit machine is my personal desktop.
One gigabit machine is in the family room sucking media from the NAS.
The 100MB machine is upstairs and the kids use that one.
The gigabit machines are plugged into a LanReady gigabit switch that I bought for 60 bucks Ebay.
The 100MB machine is plugged into a 3com superstack.
Both switches are then plugged into the cable router.
Speeds between the gigabit machines average 50 Meg a second depending how large the files are and if it's streaming or copying, The 100Mb box pulls 7-8 MB a sec from the others.
I'm happy with the speed.
Karma means nothing to me, so suck it...
Tests using PCI Gigabit chips (e.g., broadcom, 3com, intel) get around 500Mbps or so.
/. story! Of course, that controller is attached to the processor by a 6.4GB/s link!
Intel CSA attached gigabit chips (on Intel chipset motherboards only) perform better. CSA is a dedicated link from the northbridge to a gigE controller.
Of course, nForce3 250Gb integrates gigE inside, and gets over 800Mbps performance. See the preceeding
Also, PCI-X != PCIe. PCIe (PCI Express) is the upcoming high speed serial version of PCI that operates on a point-to-point basis. PCI-X is the extended faster variant of 64-bit 66MHz PCI running at up to 133MHz (1GB/s PCI essentially) in a bus configuration.
We use GigE fiber for our server networks, and pass up between 400Mb/s and 600Mb/s on high traffic days from each one.
The one thing I can say is that you'll probably never use it. There's really no need at this time. most protocols aren't any good at sucking up that much bandwidth on a single stream.
I've had many people prove this to me. They'll transfer files as single transfers. They can use up to about 10Mb/s. But if they transfer lots of files, they can use lots more. Try it through a switch that you can monitor bandwidth on. Through FTP, SMB, SCP, or whatever, you won't use up 100Mb/s. But, running multiple concurrent sessions, you can try to come close.
Heroinewarrior has a library called "firehose", which uses up all the available bandwidth, and will stripe across multiple connections to use up more. So, if you have 3 100Mb/s cards in a machine, you can come close to transfering at 300Mb/s.
You should also consider the other factors. Can your machine really send that fast? Is your hard drive fast enough to send over 100Mb/s ?? A nice fast SCSI drive, or a SATA drive can do it, but most IDE drives will fall short (specs be damned, try it in real life).
I transfer stuff around on the GigE lan all the time. We do exceed 100Mb/s, but it's usually with multiple machines.
The highest bandwidth usage machines we have are voyeurweb.com . They send out 150Mb/s through TEQL (Linux kernel option) combined 100baseTX cards, with several copies of thttpd running.
thttpd is a web server that is very small, and works very efficently. Apache has one process per connection, but thttpd has one process for everyone. Well, at least theoretically. It was around 80Mb/s of regular web site files, that it started flaking out. So, we run 4 copies of it on seperate IP's and let it scream.
As for our network, I'll outline our largest network.
We have a 1Gb/s uplink to Level3. This goes to a Cisco Catalyst 3508 (8 GBIC ports).
The remaining 7 GBIC ports go to 7 switches, mostly Cisco Catalyst 3550-48 (48 100Mb/s ethernet, 2 GBIC), and the servers are attached to the 100Mb/s ports. We have one Dell switch, which does 1000baseTX on all the ports, and a few machines with 1000baseTX cards. They can't pull anything resembling 1000Mb/s between each other. it simply doesn't happen. Honestly, doing transfers through http, ftp, or scp doesn't ever use over 100Mb/s on individual transfers. Sure, we can do it with multiple concurrent transfers, but at home, how many hundred or thousand users are you really trying to supply?
For home, you'll never use it. 100Mb/s is usually overkill. I set up my house with 802.11b, and at 11Mb/s peak, I see no difference than my old house, where we had copper run to every room and a Catalyst 2924 managing it. 11Mb/s is more than sufficent for a home network.
Spend your money on a *GOOD* 100Mb/s switch. I highly recommend Cisco, like a 2924, which you should be able to get relatively cheap used. Even if you put GigE cards in the machines, you can at least monitor your bandwidth now, and see what you really use. If you start flat-lining at 100Mb/s (bandwidth graphs make things really obvious), then you could consider upgrading.
Serious? Seriousness is well above my pay grade.
A friend of mine just went nuts when he found out about a new switch from SMC, the SMC8508T. While it's unmanaged, it offers non-blocking architecture across the entire line as well as support of jumbo frames up to 9K, which is extremely unusual for SOHO stuff. Not even a lot of expensive Cisco stuff does jumbo frames. And he paid $150 for it.
Why should you care about jumbo frames? I found this nice guide about that here.
-R
"The same goes for switches. You'll be doing good to get 400 mbps out of a cheap gig switch."
40MB is a hell of a lot better than 10MB. I don't know why everyone keeps saying he won't be able to saturate the line. He doesn't need to max it out in order to enjoy the benefits over 100Mb ethernet. Who knows what kind data we will be dealing with in 5 years? Seems like going 1000 is a smart investment.
I had no idea Gb Ethernet switches had dropped so much in price. If I was buying a new switch today I'd definitely be buying one of those $100 Linksys switches. Considering the cost is so cheap why even bother with 100MB if you think you'll be using bandwidth hungry apps?
If you wanna get rich, you know that payback is a bitch
As for fragmenting down, it might be easier to do that with a router that you actually have software control over (i.e. an old, low power linux box). I don't really have any experience with this on a home network, so...
Sujal
politics, food, music, life: FatMixx
I went through this... I bought netgear gs105 and netgear nics, all really cheap at amazon.
Like me you'll probably find you don't get a 10x increase in speed, but maybe 25-50%, like from 8 MB/s to 13 MB/s when you transfer stuff between two computers.
This is because your hard drive is fragmented, and this will completely, and drastically affect performance when you copy stuff. You don't realize it, but you will take a massive hit when you try to copy your isos, movies, etc across the LAN.
I went from 13 MB/s to like 30 MB/s after i defragmented my source and destination drives.
The main thing is that with Gigabit Ethernet, you have to think of the entire network as a system that works completely together. There has to be a complete unity between all components on your network because you will see the bottlenecks a lot easier.
Also, none of the netgear cheap stuff support jumbo frames. The more expensive NICs do, but the gs10X ports do *not* support jumboframes.
As well, they get really, really, really hot. Unnecessarily hot if you ask me, like burning to the touch, and could really heat up the inside of your CPU. In fact, even the gs105 switch is hot to the touch, too.
I instead bought 2 Intel Pro 1000 MTs. They are much more reliable, they do support jumbo frames (but I can't use it until I actually get a jumob frame compatible switch) and they don't get hot at all.
I wired my whole house with CAT5, which I assume wouldn't handle anywhere near gigabit speeds.
The nice thing about GbE is that you can still use your old CAT5 (if it isn't too low quality).
If you buy new cables, you should get CAT5e - basically the same as CAT5, but tested for 125MHz, while CAT5 is only tested for 100MHz. (GBe combines 4 bi-directional wire-pairs with 125MHz each to achieve 1000 Mb/s)
Boy this turned into a bit of a tome.
/proc/sys/net/ipv4/tcp_rmem /etc/samba/smb.conf
For a switch I went with an 8 port SMC EZSwitch 8508T. I chose it since:
1. It supports jumbo frames. According to my testing it will pass ethernet packets up to 9212 bytes which should correspond to a 9198 byte MTU.
2. It doesn't have a cooling fan. A definate plus since in my experience the little fans in switches such as this can become quite annoying as they age.
3. It comes with rack mount ears.
4. It's affordable. I purchased it from Securemart.com for $139.31 shipped. Ordered it Thursday or Friday, it arrived Monday or Tuesday.
As to NICs, one of my PCs already had an Intel gigabit port on the motherboard. In addition I purchased 4 more Intel Pro 1000/MT Desktop Adapters. Since:
1. They have good driver support on both Linux and Windows.
2. They support jumbo frames. Supposedly up to around 16000 bytes.
3. They're supposed to be pretty fast/efficient. It's kind of dated but you can find a comparison of some 32-bit gigabit NICs here.
4. They'll do 66Mhz if your motherboard supports it and of my systems does.
5. They have DOS NDIS2 drivers so I can use Ghost to make/restore images over the network.
One I purchased through Intel's evaluation program for $35.31 shipped. As I recall it took over a week to show up. The other three I ordered from OnlineMicro for $28 each plus $11.32 shipping. Be sure to change the shipping option from ground to 2 day air if you order more than 1, it's cheaper. They shipped them out the day of my order and they arrived on time.
One of the Intel NICs died about 4 hours after I installed it. I swapped it with another and the replacement has been working fine for a few weeks now. I ran the diagnostics on it and other all but the link test passed. When the OS is booted up the switch shows no link lights but sometimes when the PC is off the link lights do come on. I've also tried it in another PC where it exhibits similar symptoms. I haven't yet contacted Intel about getting it replaced.
I spent a lot of time tweaking various things. Some findings:
1. With default SO_RCVBUF sizes a MTU in the neighborhood of 4000 or so bytes seems to get about the best network/application wide throughput. Specifically the otherwise fast NF7-S system below would lose almost 50% throughput with 9000 byte MTUs with the default SO_RCVBUF size. Linux to Linux lost around 30% as I recall.
In theory you can change the default SO_RCVBUF size on linux by echoing appropriate values to:
Other than that you appear to have to change this setting in each individual application. One application of note that allows you to easily make this change is samba. See your:
2. If you crank the SO_RCVBUF size up to 200ish k or more then a 9000ish byte MTU can eek out another 5ish percent more bandwidth. Thus for the moment I've decided to just stick with 4076.
3. MTUs that are not of a size of the form 8x+4 cause Linux to behave oddly when it performs path MTU discovery. Namely for jumbo sizes that don't fit that form the discovery decides that the PMTU is 1492. You can read more detail about it in a Usenet post I made here. I still don't have a good picture of what'
First, it seems many people around here are not THAT up to date on what you can actually buy right now. It is correct that Gigabit is not really THAT useful when you're using a PCI card stuck to the 133MB/s PCI bus (although I would not consider around 60-70MB/s THAT bad compared to a standard 100MBit network card, it's still 8-9 times faster...). But you CAN buy motherboard integrated GBit cards that ARE on their separate bus right now, at consumer prices. Just look for an Intel 875P board with Intel CSA GBit, e.g. an ASUS P4C800E Deluxe. German c't magazine tested various home GBit solutions and they got around 110 MB/s over consumer priced hardware, if you just choose the right components.
:)
Second, the speed depends of course mainly on what the two sides of the connection are capable of in read speed (from disk) and write speed (to disk). If you copy files from A to B and one side is only using a cheap-ass 10 MB/s hard disk, you won't get anywhere near the theoretical maximum network speed.
I have a LAN here with my main machine being a machine with Intel CSA, and then there are three other machines - two with a PCI GBit card and one with a motherboard-integrated PCI 3com NIC. Depending on which machine copies to which machine, I get transfer speeds of 30 MB/s (copying to my old Celeron PC) to about 70 MB/s (the last only when I copy files from a machine with a fast hard drive to my main machine, which is using the CSA GBit and the SATA stripe set, which is also using a separate bus away from PCI - in this case the network speed seems to be limited by the read speed of the other machine).
So I would say that right now the home GBit is limited mainly a.) by the combined speed of hard disk and PCI GBit card being smaller than 133MB/s in the case of a machine with a PCI network card and b.) the hard disk read/write speed being slower than the max GBit speed in the case of a machine with CSA GBit. I would guess that if I had a second machine like my fastest one (both hard disk and GBit away from PCI and the hard disk stripe set being able of read/write speed greater than 100MB/s) I would finally be in GBit heaven
As far as components go - look, as was said, for the motherboard integrated, non-PCI solutions if you buy a new PC. If you're upgrading an old PC, PCI cards are OK - they are a DEFINITE improvement over 100MBit cards, even if you just read 30MB/s. As for the switch - don't buy the cheapest one, the Realtek chips (they're the ones most likely using in there) seem to have some real issues. Also, if you are noise sensitive, look for one without a fan, those little buggers can get pretty annoying real soon. I bought a 3com 5 port 10/100/1000 switch for (half a year ago) 150 Euros, and I'll probably stick another one on top of it pretty soon. That thing (3C1670500) is small, has no fan and simply does what you want it to do. And it's pretty cheap for a brand name product. And all the components which don't use GBit (like the print server, the DSL router and the Access Point) I simply left on the old 100MBit switch, so the five ports limitation wasn't really one.
I had the same problem. My core routers had difficulty handling the traffic to my branch office networks in the garage and woodshed, so I upgraded to new Juniper kit.
Everything is running smooth now, with the exception of the bathroom subnet. The Juniper gear doesn't like the moisture.
Conformity is the jailer of freedom and enemy of growth. -JFK