Slashdot Mirror
10GbE: What the Heck Took So Long?
storagedude writes "10 Gigabit Ethernet may finally be catching on, some six years later than many predicted. So why did it take so long? Henry Newman offers a few reasons: 10GbE and PCIe 2 were a very promising combination when they appeared in 2007, but the Great Recession hit soon after and IT departments were dumping hardware rather than buying more. The final missing piece is finally arriving: 10GbE support on motherboards. 'What 10 GbE needs to become a commodity is exactly what 1 GbE got and what Fibre Channel failed to get: support on every motherboard,' writes Newman. 'The current landscape looks promising. 10 GbE is starting to appear on motherboards from every major server vendor, and I suspect that in just a few years, we'll start to see it on home PC boards, with the price dropping from the double digits to single digits, and then even down to cents.'"
Everyone's still running off of ancient Cat3 wiring laid down when telephones were still analog.
10GE Motherboards are still pointless when 10G routers & switches are still way too expensive.
Biggest reason I can remember from when we were looking at upgrading SAN and LAN equipment in our data center was the price/performance point. We didn't need 10 GbE performance yet and the price was pretty far above what we were using. That was 3 years ago though, so I'd have to poke around some of the newer equipment to see if we have any boxes with it. I just took a gander through the HP and Dell offerings and it's not even an option on anything but the top tier equipment. I think that pretty much explains the situation itself.
Of course its growth was going to be lower.
The primary use of 10GbE is virtualization. The use of network cards are a function of the number of chassis, not the number of hosts. Numerically, 10GbE is not 10 1GbE cards. You can split the 10GbE between a lot of hosts. You can easily double, triple, or even quadruple that to making that 10 GbE card the equivalent of 1 GbE cards on 40 servers, depending of their load and use. Instead of buying 40 servers and associated cards, you're buying one larger chassis with larger pipes. In a large farm environment, and it makes sense.
Throw in the fact that network is only as fast as its narrowest choke point, there is no reason to put in a 10 GbE card behind a 7MB DSL connection.
What 10GbE needs to become a commodity is a) end of any data caps, b) data to put down that pipe, and c) a pipe that can handle it.
Show me fiber to my door and then, it will be a commodity.
Ten gigabits per second is 1,250 megabytes per second. High-end consumer SSDs are advertising ~500 MB/sec. A single PCIe 2.0 lane is 500 MB/sec. Then there's your upstream internet connection, which won't be more than 12.5 MB/sec (100 megabits/sec), much less a hundred times that. I guess you could feed 10GbE from DDR3 RAM through a multi-lane PCIe connection, assuming your DMA and bus bridging are fast enough...
I'm sure a data center could make use of 10GbE, but I don't think consumer hardware will benefit even a few years from now. Seems like an obvious place to save some money in a motherboard design.
Visit the
What you describe exists. It's not uncommonly used for IP cameras outside the 100m limit of TP Ethernet (on perimeter fences, etc.). The problem with fibre is that it's a bitch to terminate compared to copper, and therefore quite a bit more expensive to install on a large scale. Fibre still only makes sense when you need the long cable runs.
For what? What's the application? Way too expensive to run to my IP Phone or Desktop PC (could juse use fiber or copper, why both?). Unnecessary in the datacenter (we don't need PoE). What's the use case?
The fact that you can use existing commodity Cat 6 cables for up to 55m with 10GigE will help a lot too. Yes, Cat 6a cables are required for the full 100m distance, and yes, Cat 6a cables are themselves cheap ($4 for that 6 feet you mention costing $80 with Twinax), but for lengths under 55m, the cabling that you've already got will continue working at the higher speeds. I think that will be a big factor, especially for consumers, where cables longer than 10 or 15 metres are incredibly rare anyhow.
Right now, I think the NIC/port price is the major roadblock. Netgear made a big deal about dropping below $125 per 10gig port on their switches a while ago, and that's a step forward, but it's still a thousand bucks for an 8-port switch (probably on the larger end of what you'd find in a home network), and the cheapest 10 gig NIC on NewEgg is $345. That's in the "affordable for enthusiasts" range, though. If you're willing to spend seven hundred bucks, you could connect your desktop to your home file server over 10 gigabit, for example, and two or three thousand bucks could get a bunch of devices on a network. Expensive, but not "mortgage your house, this is only for multi-billion dollar enterprise use" levels like it was a few years ago.
The best reason I can think of not to buy a 10-gigabit Ethernet card is simple: The cheapest ones go for $351 on Newegg. Want an Ethernet switch to go with that? That will be $1036.
So once again, the answer is simple, and it has to do with a dollar sign.
Gigabit equipment got really cheap fairly quickly, but not so much for the 10-gigabit equipment.
We have some of these at work where we do have the need for moving massive volumes of data around. We can get about 99.6% of theoretical throughput in actual use, thanks to the hardware offloading and large frame support. Besides the 10x faster to start with, that's way above any efficiency we get from the 1 GbE ports, though I expect if 10 GbE went commodity you'd lose all the hardware support and you'd be back to 80-90% range.
Note to sustain a data feed to one of these you at least need two SATA 6 gbps SSD drives in RAID0. On the receiving end we're not writing to disk, or you'd need ~3-4 RAIDed.
In our case we're feeding 4 10GbE ports on the same machine and using a 10 SSD RAID0 to supply the data with some headroom (we don't care if we lose the data if one fails, these aren't the master copies). We're just using software RAID, but thanks to all the DMA and offloading the CPU usage is quite low.
Now do I need this at home? Well, SSD speeds are far above the ~85 MB/sec 1GbE delivers, but so far the cost hasn't made it worth it. If I'm copying a gigabyte it takes 12 seconds, which I can live with.
it's trivial to enable LACP to bond several 1 gbps links. no new equipment, no new cabling. that would have slowed down my 10 gbps deployment.
10x1gb != 1x10gb. Your LACP bond still limits a single stream to a single link. Even with multiple streams, you would have to have a lot in order of them to hash out to all the links.
For many things you do, you find 1gbit is enough. More doesn't really gain you anything. It is enough to stream even 4k compressed video, enough such that opening and saving most files is as fast as local access, enough that the speed of a webpage loading is not based on that link but something else.
Every time we go up an order of magnitude, the next one will matter less. There will be fewer things that are bandwidth limited and as such less people that will care about the upgrade.
As you say, 10gbit, or even more, is useful in many datacenters. But at home? What the fuck would I do with it? I guess I could... copy files faster from my desktop to server? Well except my server uses a magnetic drive that is slower than gigabit.
And, of course, you get to re-run all your cables. Gig works over Cat-5e, of course, which has been used for awhile, and with ASICs on smaller processes it actually usually works over Cat-5. So you can have some pretty old wiring and just knock in a gig switch and cards and call it good. 10gig needs Cat-6a. That is new, expensive, and a pain in the ass to work with.
Bandwidth is not something where we need "MOAR ALL OF THE TIMES!!" it isn't something we need to just seek to increase at any cost. Rather it is something that we need to have enough of to make it not a bottleneck for whatever it is we are doing. Well, for a lot of network stuff these days, gig is that. It is fast enough that it doesn't slow things down, at least not a significant amount. So that's all you need.
Same shit with BW anywhere else. You find that increasing memory bandwidth past a point with current CPUs is useless. Like with a Core i7-2600 increasing memory speed up to 1600MHz seems to help, however past that, doesn't matter except in synthetic benches. The memory bandwidth isn't an issue. With graphics cards the PCIe 3.0 upgrade did fuck-all since it turn out PCIe 2.0 4x is almost always enough bandwidth, and 8x is more than enough so PCIe 3.0 16x is doubling something you already have more than enough of.
As things progress we'll probably see more uses for 10gig, and thus it'll get rolled out wider. However it is the kind of thing that'll happen as needed, not that'll happen just because it can. We'll upgrade our building when it needs to be. When our uplinks are getting saturated, we'll take those to 10gig. When there is a reason to get it to the desktop, we will. However we aren't going to run out and drop 6 figures to go 10gig right now just for the sake of doing it.
That's been true at this point at each jump in speeds (well, other than the details of the connection). The Ethernet chip-on-motherboard heralds the price fall on the switch - at the scale the 10GbE chips will soon be made, their price will fall (and thus the price of port-specific electronics in the switch will fall too), and then reasonably-priced unmanaged switches from low-end vendors follow soon after.
Socialism: a lie told by totalitarians and believed by fools.
I think the main reason is cost. I have been working with 10Gbe for several years writing drivers for PHYs and MACs. I've worked with a number of PHYs and 10Gbe is a lot more complex. For example, the SFP+ cables and modules each have a serial EEPROM that contains parameters needed to program the PHY. It's not just a simple RJ45 CAT5/CAT6 cable. As someone who has worked in 10Gbe drivers there's a lot more complexity. With some PHYs I have to query the serial EEPROM to make changes based on things such as cable length and whether or not it's active or passive or if it's copper or optical. Distances over copper are also usually limited to much shorter distances unless active cabling is used.
In terms of cost, a 1 meter copper cable is around $43 from www.cablesondemand.com. A 12 meter cable is $189. It's not like gigabit where you just plug in a CAT5 cable and go.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
10GBe won't be as fast as a nice cheap SSD
It doesn't matter, remote storage is faster than gigabit. I don't need to hit 10 to get a benefit.
but not even an SSD can keep up with an avalanche of data requests from multiple systems unless that remote server is pretty damn beefy by home standards.
What? That's the whole point of fileservers. They need to meet the usage, of course, but that's an always increasing spec.
Simpler to keep your OS local, and trivial as far as cost.
Consolidating is always cheaper (per unit of storage) and it's easier to back up and manage, keep on UPS power, etc.
You'll need a heck of a RAID array for that, but it's buildable. Or, you could just stick with GigE, since that still tops out at 125MB/s and that pushes local (non-SSD) storage.
eh, my current central storage is 5 hard drives in a ZFS raidz2 with one SSD split up for L2ARC (cache) and ZIL (write cache). The entirely of the setup difficulty is:
cd /etc/yum.repos.d
wget url-to-repo
yum install zfs
(reboot or modprobe)
zpool create home raidz2 sda sdb sdc sdd sde cache sdf6 log sdf7
Oh, I had to plug in 6 SATA cables. Typical throughput is about 340MB/s. The only reason they're not all SSD's is because SSD's are expensive and unreliable. If it wasn't a home machine, the ZIL would be on a mirror of SSD's.
Correct me if I'm wrong, but my understanding is that this requires a lot more than just a high-speed connection. High-end connection + craptastic router = terrible latency when dealing with high load.
Switch, not router. There are problems with current buffer management techniques that effectively means that higher ceiling room means latency improvements. Google 'bufferbloat'. Things like CoDel will make this better when the pipes are more full, but they're not widely deployed yet.
"I have my home wired up like a datacenter. Everyone else should want a huge amount of network capacity and capability so that it makes my already extravagant costs slightly cheaper."
JHFCOAS - this is Slashdot. What we're doing now is what will be sold in a box for $200 at WalMart in five years. I'm amazed to find tech geeks who don't even know that normal people have been buying inexpensive Buffalo and WD SAN solutions at the office supply store since 2008. And with all this shit going on about the NSA, you can bet people are going to be pulling some of their stuff back out of the cloud.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)