Slashdot Mirror
Gigabit Networking for the Home?
The Clockwork Troll asks: "I've had a whole-house audio/video distribution project on the back-burner for a while now. As gigabit networking hardware prices come down to earth, I'm tempted to jump on the 1000BaseTX bandwagon. As far as I can tell though, the current crop of consumer-priced hardware/software doesn't address a couple key issues, namely: fragmenting jumbo frames for the benefit of legacy clients - this is critical as some of the devices on my network will not tolerate the 9000+ byte Ethernet frames which are needed to get the most out of gigabit; and OS support - do Linux and Windows require much tweaking to take advantage of gigabit? Will most drivers automatically optimize themselves? A Google search didn't reveal too much consensus, especially on hardware choices. What switches and software configurations have Slashdot readers been using for home gigabit networks, in particular mixed ones (100/1000BaseTX?"
Check out the 8 port Asante GX5-800P. You can find them for ~ $160.
That's gigabyte. Bit prefixes go by 1000 (10^2), byte prefixes go by 2^10.
Gotta shuttle all that porn around the home network huh? ;-)
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
Of course, if your needs are more extensive you may need something more...
Al Qaeda has ninjas!
Hard as I try, I can't imagine ever having enough stuff in my house to warrant gigabit. Damn.
In Mac OS X, there's a setting right in the Network Preference Pane that is under "Ethernet" and it allows you to scale up the packet size depending on the immediately aparent network appliances. I haven't been able to use this feature because:
A: Some clients have nice network hardware, but legacy copper
B: Some clients have gig copper, but not enough hardware
I can't wait to see the transfer rates on Gig with Jumbo packets though. *Drool*
I got nothin'.
I've got an Abit motherboard with Intel gigabit built in and WindowsXP loaded on it. My GF has a Powerbook with gigabit built in. We bought the cheapest gigabit switch we could find. We got Cat 6 cable.
:-).
Everything was autodetected and the speed improvement over 100mbit was dramatic. Highest performance increase I've ever gotten for doing basically zero work (I did plug in the cables all by myself
Now, this obviously doesn't answer all your questions, but for anyone out there who doesn't have legacy issues all I can say is go for it, it's a no-brainer.
BTW, I use a Linksys WAP-Router for internet. It didn't so much as burp when we plugged it into the gigabit switch.
TW
I think the biggest thing about gigabit is that PCI isn't really fast enough to support it. You can shovel 133MB/second over a PCI bus, or 1064Mb.... very slightly more than a gigabit, but that's with NOTHING else happening on the bus. Generally, since the hard drive controller is also on the Southbridge, I think about the best you're going to get off most PCs, even very, very fast ones, is about 300 megabits sustained.
To really take advantage, you're going to need machines that run the network card off the Northbridge. Presumably, PCI-Express network cards will also keep up pretty easily. From what I can see, you're probably best to wait another year to eighteen months before upgrading; by then, PCI-X should be pretty common, and gigabit networking shouldn't be very expensive.
Note that I don't have any direct experience with gigabit: these are just back-of-the-envelope calculations. I could be completely off, so pay attention to replies.
I've had a good experience with a Dell PowerConnect hub (the 8- or 16-port model, I forget which). It was quite inexpensive and claims to support Jumbo Frames (however I haven't actually gotten this to work; when I enlarge the frame size on Linux it loses the connection). Oh, and I had to disable one default feature on the hub (tree-spanning something or other) to get it to work.
For clients I use Intel gigabit cards (the 64-bit PCI "server" model). I wouldn't skimp here since indications are that cheap gigabit cards don't have any hope of getting wire speed. NFS file copies max out at 20-30MB/sec, but I know that is limited by my server's disk array. I did a test for raw network bandwidth (just sending zero bytes as fast as possible) and got around 60-80MB/sec.
Everything is connected to my existing Cat-5 cable with no problems. This includes several Linux systems, one Mac and one Windows PC.
I will caution you not to expect anything like gigabit wire speeds with typical clients. My Mac G4 in particular seems to have trouble getting good bandwidth (I think the problem is either the network stack or NFS client).
If anyone has a success story with jumbo frames, I'd love to hear about it. The only references I could find are for mega-dollar Cisco/Foundry type equipment.
Now I know this is /. but before everyone says "you don't need gigabit!" and "bah, who needs that kind of speed" gigabit ethernet is genuinely useful. Even copying 500mb files can take intolerably long when you want it done 4 minutes ago. If the poster wanted a bunch of nonsense about why he shouldn't do it and why its a dumb idea, he could have gone to Circuit city (they don't sell gigabit so they would try to sell him 10/100). Instead he asked us for an informed option and information on the matter.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
I have a mixed network and have not had any problems with speed or the switches flaking out.
I have 3com gigabit cards in three computers and a 3com 100Mb card in one.
One gigabit machine is a redhat 8 machine that is used as the network attached storage (NAS) box feeding media throughout the house and acting as the DNS for the house (This is so much faster than relying on your ISP!) and to filter packets for the kids computer (Damn Pr0N!)
One gigabit machine is my personal desktop.
One gigabit machine is in the family room sucking media from the NAS.
The 100MB machine is upstairs and the kids use that one.
The gigabit machines are plugged into a LanReady gigabit switch that I bought for 60 bucks Ebay.
The 100MB machine is plugged into a 3com superstack.
Both switches are then plugged into the cable router.
Speeds between the gigabit machines average 50 Meg a second depending how large the files are and if it's streaming or copying, The 100Mb box pulls 7-8 MB a sec from the others.
I'm happy with the speed.
Karma means nothing to me, so suck it...
No, really. I'm serious. Not at home, anyway.
Unless you get a very hot, brand new PC with motherboard integrated gigE, your PCI bus can't push the bandwidth. The same goes for switches. You'll be doing good to get 400 mbps out of a cheap gig switch.
Even if you have a $5000 gigE switch and a PC that can handle it, what are you going to talk to, your cable modem? The only place gigabit ethernet makes sense is when you are aggregating traffic from multiple computers to a centralized server or set of servers, and are using applications that actually require that kind of bandwidth. Even if you want to move that much data around, and have a way to do it (hint - neither scp nor samba can talk that fast), the best benefit you'll see is about double the performance you get with 100.
Here in the networking world (where I live and play), recent advances in traffic management systems have begun to punch holes in the time-worn theory that throwing bandwidth at a network problem = fixed. If you really want network performance, go check out the Linux advanced router/ traffic control site. (lartc.org) There, you'll learn to get lightning response from ssh and your first person shooters, all while running a 2gig/month web server through your home dsl's 256K uplink. And it won't cost you a dime.
who are those slashdot people? they swept over like Mongol-Tartars.
Tests using PCI Gigabit chips (e.g., broadcom, 3com, intel) get around 500Mbps or so.
/. story! Of course, that controller is attached to the processor by a 6.4GB/s link!
Intel CSA attached gigabit chips (on Intel chipset motherboards only) perform better. CSA is a dedicated link from the northbridge to a gigE controller.
Of course, nForce3 250Gb integrates gigE inside, and gets over 800Mbps performance. See the preceeding
Also, PCI-X != PCIe. PCIe (PCI Express) is the upcoming high speed serial version of PCI that operates on a point-to-point basis. PCI-X is the extended faster variant of 64-bit 66MHz PCI running at up to 133MHz (1GB/s PCI essentially) in a bus configuration.
This is the major point that is overlooked when people talk Gb networks. Only with PCI-X slots do you see a major improvement in performance, and I would doubt that a home network contains even one PCI-X slot.
once you get around the IDE or SATA, the audio, the USB2 or Firewire (if we're talking video editing) etc etc etc, you would be better adding another standard network card and teaming them for your major data stores in the network and leave everything else as it is.
Also on a side note a 1 X PCI Express slot is ~250MB in each direction (about ~500MB total) so yes a 1 X PCI-E slot will do Gb ethernet fine
Normal people worry me!
We use GigE fiber for our server networks, and pass up between 400Mb/s and 600Mb/s on high traffic days from each one.
The one thing I can say is that you'll probably never use it. There's really no need at this time. most protocols aren't any good at sucking up that much bandwidth on a single stream.
I've had many people prove this to me. They'll transfer files as single transfers. They can use up to about 10Mb/s. But if they transfer lots of files, they can use lots more. Try it through a switch that you can monitor bandwidth on. Through FTP, SMB, SCP, or whatever, you won't use up 100Mb/s. But, running multiple concurrent sessions, you can try to come close.
Heroinewarrior has a library called "firehose", which uses up all the available bandwidth, and will stripe across multiple connections to use up more. So, if you have 3 100Mb/s cards in a machine, you can come close to transfering at 300Mb/s.
You should also consider the other factors. Can your machine really send that fast? Is your hard drive fast enough to send over 100Mb/s ?? A nice fast SCSI drive, or a SATA drive can do it, but most IDE drives will fall short (specs be damned, try it in real life).
I transfer stuff around on the GigE lan all the time. We do exceed 100Mb/s, but it's usually with multiple machines.
The highest bandwidth usage machines we have are voyeurweb.com . They send out 150Mb/s through TEQL (Linux kernel option) combined 100baseTX cards, with several copies of thttpd running.
thttpd is a web server that is very small, and works very efficently. Apache has one process per connection, but thttpd has one process for everyone. Well, at least theoretically. It was around 80Mb/s of regular web site files, that it started flaking out. So, we run 4 copies of it on seperate IP's and let it scream.
As for our network, I'll outline our largest network.
We have a 1Gb/s uplink to Level3. This goes to a Cisco Catalyst 3508 (8 GBIC ports).
The remaining 7 GBIC ports go to 7 switches, mostly Cisco Catalyst 3550-48 (48 100Mb/s ethernet, 2 GBIC), and the servers are attached to the 100Mb/s ports. We have one Dell switch, which does 1000baseTX on all the ports, and a few machines with 1000baseTX cards. They can't pull anything resembling 1000Mb/s between each other. it simply doesn't happen. Honestly, doing transfers through http, ftp, or scp doesn't ever use over 100Mb/s on individual transfers. Sure, we can do it with multiple concurrent transfers, but at home, how many hundred or thousand users are you really trying to supply?
For home, you'll never use it. 100Mb/s is usually overkill. I set up my house with 802.11b, and at 11Mb/s peak, I see no difference than my old house, where we had copper run to every room and a Catalyst 2924 managing it. 11Mb/s is more than sufficent for a home network.
Spend your money on a *GOOD* 100Mb/s switch. I highly recommend Cisco, like a 2924, which you should be able to get relatively cheap used. Even if you put GigE cards in the machines, you can at least monitor your bandwidth now, and see what you really use. If you start flat-lining at 100Mb/s (bandwidth graphs make things really obvious), then you could consider upgrading.
Serious? Seriousness is well above my pay grade.
On a 100BaseT NIC the theoretical max transfer rate is 12.5 MBps with a realistic speed of 8 MBps. Multiply that by 10 to get a rough estimate of Gigabit speed. Most ATA HDD's can transfer around 40-60 MBps. You can easily saturate a 100BaseT network with bargain basement machines.
Gigabit Ethernet is faster than what your typical ATA drive will absorb, but it is still going to be quite a bit faster than 100BaseT.
Spend the Money on a nicer HDD or a decent RAID setup and you will be able to make full use of a Gigabit pipe.
A friend of mine just went nuts when he found out about a new switch from SMC, the SMC8508T. While it's unmanaged, it offers non-blocking architecture across the entire line as well as support of jumbo frames up to 9K, which is extremely unusual for SOHO stuff. Not even a lot of expensive Cisco stuff does jumbo frames. And he paid $150 for it.
Why should you care about jumbo frames? I found this nice guide about that here.
-R
Uh, have you ever tried copying 3 hours worth of HD video from your capture system to your main workstation over 100 Mb. After 15 minutes you will be begging for Gb.
Even though I would probably only get double the speed (disk bottlenecks, one is a slow system) I still am thinking it might be worth it.
Q.
I have a NetGear 4 port gigabit switch. I have found I can transfer files about 2.5x as fast as with 100mbit (without jumbo frames). In my book, that's worth the few extra bucks a gigabit switch will cost you.
A warning though, I've heard most of the cheap gigabit switches have fans in them. Fans reduce the reliability of a switch many fold and make them LOUD. I like my 4 port Netgear and they now make an 8 port version which is also fanless and very reasonably priced.
Does anyone have a Linksys or D-Link gigabit switch who can confirm or deny the presence of a fan?
One note I'd like to throw in: Gigabit ethernet requires Cat-5 cable. Not Cat-5e, Not Cat-6, Cat-5. Better cables may be less prone to issues but they aren't part of the gigabit ethernet standard so don't go out and re-cable your house just for a little Gig-E.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
That is why Intel i865/i875 has the option of direct connect e1000 gigabit (CSA) to the northbridge. Most motherboards with gigabit built on that uses either of those chipsets use the e1000 CSA gigabit chips.
As for fragmenting down, it might be easier to do that with a router that you actually have software control over (i.e. an old, low power linux box). I don't really have any experience with this on a home network, so...
Sujal
politics, food, music, life: FatMixx
I went through this... I bought netgear gs105 and netgear nics, all really cheap at amazon.
Like me you'll probably find you don't get a 10x increase in speed, but maybe 25-50%, like from 8 MB/s to 13 MB/s when you transfer stuff between two computers.
This is because your hard drive is fragmented, and this will completely, and drastically affect performance when you copy stuff. You don't realize it, but you will take a massive hit when you try to copy your isos, movies, etc across the LAN.
I went from 13 MB/s to like 30 MB/s after i defragmented my source and destination drives.
The main thing is that with Gigabit Ethernet, you have to think of the entire network as a system that works completely together. There has to be a complete unity between all components on your network because you will see the bottlenecks a lot easier.
Also, none of the netgear cheap stuff support jumbo frames. The more expensive NICs do, but the gs10X ports do *not* support jumboframes.
As well, they get really, really, really hot. Unnecessarily hot if you ask me, like burning to the touch, and could really heat up the inside of your CPU. In fact, even the gs105 switch is hot to the touch, too.
I instead bought 2 Intel Pro 1000 MTs. They are much more reliable, they do support jumbo frames (but I can't use it until I actually get a jumob frame compatible switch) and they don't get hot at all.
I wired my whole house with CAT5, which I assume wouldn't handle anywhere near gigabit speeds.
The nice thing about GbE is that you can still use your old CAT5 (if it isn't too low quality).
If you buy new cables, you should get CAT5e - basically the same as CAT5, but tested for 125MHz, while CAT5 is only tested for 100MHz. (GBe combines 4 bi-directional wire-pairs with 125MHz each to achieve 1000 Mb/s)
Boy this turned into a bit of a tome.
/proc/sys/net/ipv4/tcp_rmem /etc/samba/smb.conf
For a switch I went with an 8 port SMC EZSwitch 8508T. I chose it since:
1. It supports jumbo frames. According to my testing it will pass ethernet packets up to 9212 bytes which should correspond to a 9198 byte MTU.
2. It doesn't have a cooling fan. A definate plus since in my experience the little fans in switches such as this can become quite annoying as they age.
3. It comes with rack mount ears.
4. It's affordable. I purchased it from Securemart.com for $139.31 shipped. Ordered it Thursday or Friday, it arrived Monday or Tuesday.
As to NICs, one of my PCs already had an Intel gigabit port on the motherboard. In addition I purchased 4 more Intel Pro 1000/MT Desktop Adapters. Since:
1. They have good driver support on both Linux and Windows.
2. They support jumbo frames. Supposedly up to around 16000 bytes.
3. They're supposed to be pretty fast/efficient. It's kind of dated but you can find a comparison of some 32-bit gigabit NICs here.
4. They'll do 66Mhz if your motherboard supports it and of my systems does.
5. They have DOS NDIS2 drivers so I can use Ghost to make/restore images over the network.
One I purchased through Intel's evaluation program for $35.31 shipped. As I recall it took over a week to show up. The other three I ordered from OnlineMicro for $28 each plus $11.32 shipping. Be sure to change the shipping option from ground to 2 day air if you order more than 1, it's cheaper. They shipped them out the day of my order and they arrived on time.
One of the Intel NICs died about 4 hours after I installed it. I swapped it with another and the replacement has been working fine for a few weeks now. I ran the diagnostics on it and other all but the link test passed. When the OS is booted up the switch shows no link lights but sometimes when the PC is off the link lights do come on. I've also tried it in another PC where it exhibits similar symptoms. I haven't yet contacted Intel about getting it replaced.
I spent a lot of time tweaking various things. Some findings:
1. With default SO_RCVBUF sizes a MTU in the neighborhood of 4000 or so bytes seems to get about the best network/application wide throughput. Specifically the otherwise fast NF7-S system below would lose almost 50% throughput with 9000 byte MTUs with the default SO_RCVBUF size. Linux to Linux lost around 30% as I recall.
In theory you can change the default SO_RCVBUF size on linux by echoing appropriate values to:
Other than that you appear to have to change this setting in each individual application. One application of note that allows you to easily make this change is samba. See your:
2. If you crank the SO_RCVBUF size up to 200ish k or more then a 9000ish byte MTU can eek out another 5ish percent more bandwidth. Thus for the moment I've decided to just stick with 4076.
3. MTUs that are not of a size of the form 8x+4 cause Linux to behave oddly when it performs path MTU discovery. Namely for jumbo sizes that don't fit that form the discovery decides that the PMTU is 1492. You can read more detail about it in a Usenet post I made here. I still don't have a good picture of what'
First, it seems many people around here are not THAT up to date on what you can actually buy right now. It is correct that Gigabit is not really THAT useful when you're using a PCI card stuck to the 133MB/s PCI bus (although I would not consider around 60-70MB/s THAT bad compared to a standard 100MBit network card, it's still 8-9 times faster...). But you CAN buy motherboard integrated GBit cards that ARE on their separate bus right now, at consumer prices. Just look for an Intel 875P board with Intel CSA GBit, e.g. an ASUS P4C800E Deluxe. German c't magazine tested various home GBit solutions and they got around 110 MB/s over consumer priced hardware, if you just choose the right components.
:)
Second, the speed depends of course mainly on what the two sides of the connection are capable of in read speed (from disk) and write speed (to disk). If you copy files from A to B and one side is only using a cheap-ass 10 MB/s hard disk, you won't get anywhere near the theoretical maximum network speed.
I have a LAN here with my main machine being a machine with Intel CSA, and then there are three other machines - two with a PCI GBit card and one with a motherboard-integrated PCI 3com NIC. Depending on which machine copies to which machine, I get transfer speeds of 30 MB/s (copying to my old Celeron PC) to about 70 MB/s (the last only when I copy files from a machine with a fast hard drive to my main machine, which is using the CSA GBit and the SATA stripe set, which is also using a separate bus away from PCI - in this case the network speed seems to be limited by the read speed of the other machine).
So I would say that right now the home GBit is limited mainly a.) by the combined speed of hard disk and PCI GBit card being smaller than 133MB/s in the case of a machine with a PCI network card and b.) the hard disk read/write speed being slower than the max GBit speed in the case of a machine with CSA GBit. I would guess that if I had a second machine like my fastest one (both hard disk and GBit away from PCI and the hard disk stripe set being able of read/write speed greater than 100MB/s) I would finally be in GBit heaven
As far as components go - look, as was said, for the motherboard integrated, non-PCI solutions if you buy a new PC. If you're upgrading an old PC, PCI cards are OK - they are a DEFINITE improvement over 100MBit cards, even if you just read 30MB/s. As for the switch - don't buy the cheapest one, the Realtek chips (they're the ones most likely using in there) seem to have some real issues. Also, if you are noise sensitive, look for one without a fan, those little buggers can get pretty annoying real soon. I bought a 3com 5 port 10/100/1000 switch for (half a year ago) 150 Euros, and I'll probably stick another one on top of it pretty soon. That thing (3C1670500) is small, has no fan and simply does what you want it to do. And it's pretty cheap for a brand name product. And all the components which don't use GBit (like the print server, the DSL router and the Access Point) I simply left on the old 100MBit switch, so the five ports limitation wasn't really one.
Ok, here's the deal with jumbo frames.
Don't worry about them. Only very, very expensive systems will be able to take advantage of them.
If you have 32/33 pci, you arent going to get max throughput from GbE anyway. I've managed to get around 90mbyte/sec using ttcp, which is about 750mbit/s.
Because the hardware does all the work for you (hardware checksum, interrupt mitigation, etc). the cpu usage is very low even at that rate. And thanks to polling, the interrupt rate isnt an issue either.
Your bottleneck will be your PCI bus, plain and simple. You arent going to get the full 132mbyte/s from 32/33 pci, period.
Unfortunately 64bit/66mhz PCI motherboards are somewhat expensive and 64/66 cards are 3-4x the cost of 32/33 ones.
I've been using GbE for home LAN for about a year now. Here's the hardware I use:
h ttp://yoda.uvi.edu/InfoTech/rj45.htm
Switch:
Linksys Instant Gigabit 10/100/1000 8-port switch
I think I paid ~$200 for this.
Cards:
Intel PRO/1000 MT Desktop Adapter (~$50 ea)
Use the e1000 driver in 2.4.x or 2.6.x.
Netgear GA302T Copper Gigabit Adapter (~50 ea)
Use the tg3 driver in 2.4.x or 2.6.x
The tg3 chipset runs rather hot, the e1000 is tiny and runs cool. I havent noticed a performance difference between either, and both chipsets run fine regardless of whatever PC I put them in.
Motherboards with embedded GbE typically use e1000 (if theyre good), or realtek (if theyre cheap).
Jumbo frames:
See my post on that here.
Cabling:
Hand crimped cat5e. Works fine. One interesting note about GbE, you no longer have to worry about crossover cables -- the GbE spec requires that devices autodetect crossover. You can make all your GbE cables "straight through" cables.
Do pay careful attention to following strict T568 wiring code though. You can no longer get away with incorrectly wired cables which just happened to work for 100bt. Since all pairs are now used in GbE, your wiring order must be 100% spec.
Here's some wiring guides:
http://www.lanshack.com/make-cat5E.asp
Because you'll find that you can't write to a filesystem on a single disk much faster than 100mbit anyway. Gigabit is significantly faster than the I/O that a single drive can provide.
Government of the people, by corporate executives, for corporate profits.