Slashdot Mirror
10GbE: What the Heck Took So Long?
storagedude writes "10 Gigabit Ethernet may finally be catching on, some six years later than many predicted. So why did it take so long? Henry Newman offers a few reasons: 10GbE and PCIe 2 were a very promising combination when they appeared in 2007, but the Great Recession hit soon after and IT departments were dumping hardware rather than buying more. The final missing piece is finally arriving: 10GbE support on motherboards. 'What 10 GbE needs to become a commodity is exactly what 1 GbE got and what Fibre Channel failed to get: support on every motherboard,' writes Newman. 'The current landscape looks promising. 10 GbE is starting to appear on motherboards from every major server vendor, and I suspect that in just a few years, we'll start to see it on home PC boards, with the price dropping from the double digits to single digits, and then even down to cents.'"
Everyone's still running off of ancient Cat3 wiring laid down when telephones were still analog.
I think its a combo of the crappy economy, but then again maybe the need for wide adaptation just wasn't there. I would think it is like any other thing, if the demand was there the supply would have ramped up and the costs would have gone down.
Yay, I have a sig.
But hey - we had to wait for the system bus, right?
10GE Motherboards are still pointless when 10G routers & switches are still way too expensive.
My idea of the perfect cable:
Four strands, two copper, two fiber.
The two fiber strands enable redundancy (ring topology all the way to the end-point);
The two copper strands for being able to provide power to devices.
That's it. That's all that's needed.
My karma ran over your dogma
probably needed for 4k HD..
Of course its growth was going to be lower.
The primary use of 10GbE is virtualization. The use of network cards are a function of the number of chassis, not the number of hosts. Numerically, 10GbE is not 10 1GbE cards. You can split the 10GbE between a lot of hosts. You can easily double, triple, or even quadruple that to making that 10 GbE card the equivalent of 1 GbE cards on 40 servers, depending of their load and use. Instead of buying 40 servers and associated cards, you're buying one larger chassis with larger pipes. In a large farm environment, and it makes sense.
Throw in the fact that network is only as fast as its narrowest choke point, there is no reason to put in a 10 GbE card behind a 7MB DSL connection.
What 10GbE needs to become a commodity is a) end of any data caps, b) data to put down that pipe, and c) a pipe that can handle it.
Show me fiber to my door and then, it will be a commodity.
[...] and I suspect that in just a few years, we'll start to see it on home PC boards, with the price dropping from the double digits to single digits, and then even down to cents.
*pfffft* HA ha ha ha ha HA... oh, man, that's a good one. The idea that home PC boards will continue to exist in a few years! And that consumer-level devices will have wired networks to begin with! That's hilarious! Thanks, I needed a good laugh today!
Ten gigabits per second is 1,250 megabytes per second. High-end consumer SSDs are advertising ~500 MB/sec. A single PCIe 2.0 lane is 500 MB/sec. Then there's your upstream internet connection, which won't be more than 12.5 MB/sec (100 megabits/sec), much less a hundred times that. I guess you could feed 10GbE from DDR3 RAM through a multi-lane PCIe connection, assuming your DMA and bus bridging are fast enough...
I'm sure a data center could make use of 10GbE, but I don't think consumer hardware will benefit even a few years from now. Seems like an obvious place to save some money in a motherboard design.
Visit the
Until SSD in large capacities are available at reasonable prices, I suspect the demand for 10Gbe will remain low.
Don't count on the price of 10gigE dropping to cents. Unlike gigE, 10gigE has really very little 'enterprise' competition technologies. Fibre channel, infiniband, etc. - if you want more than gigE speeds, it's going to cost you. Those were costly technologies then - but back then, they offered significantly more performance (and thus value) than gigE. With 10gigE, there is no financial incentive to drop costs.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
Most of my customers are still running 100base-T and see little reason to upgrade since their networks primarily exist to distribute Internet access. What took so long? Nobody seems to really want it. Slashdot crowd not withstanding.
I would argue that part of the issue is that 10GigE connections have limited use. Not that they're not useful, but at this point, with the amount of data we're moving around, most people aren't going to see a huge benefit over existing solutions. It's a little like why desktop computer sales have slowed in general: what people have now is kind of working "well enough".
Of course, part of the problem is that a lot of what people are doing now is over the Internet, which means that you're bottlenecked by your ISP. It doesn't matter as much if you have a 100Mb or 1Gb or 10Gb adapter if you're doing an Internet transfer bottlenecked to 8Mb.
The main reason why 10GbE took time to arrive is simple : connectors are not the good-old RJ45 used for 10Mb, 100Mb and 1GbE. The RJ45 connector is small, cheap and backward compatible. The 10GbE connectors are deep, expensive and not RJ45-compatible, hence cannot be used as a 1GbE port.
10GbE is appearing on servers because the price order is compatible with the expensive and deep connector. It won't appear on commodity motherboards until a smaller connector is designed.
it's trivial to enable LACP to bond several 1 gbps links. no new equipment, no new cabling. that would have slowed down my 10 gbps deployment.
The best reason I can think of not to buy a 10-gigabit Ethernet card is simple: The cheapest ones go for $351 on Newegg. Want an Ethernet switch to go with that? That will be $1036.
So once again, the answer is simple, and it has to do with a dollar sign.
Gigabit equipment got really cheap fairly quickly, but not so much for the 10-gigabit equipment.
Please stop talking like your desktop defines IT. 10 Gb ethernet has been around for years for Sun/Oracle servers, IBM servers, Cisco switches, storage arrays, etc. Hell, I could even get 10Gb for my Mac. It hadn't made it into the PC world yet due to office wiring to the desk still being Cat 5. It's hard enough to get 1 Gb connections for the general user.
The only thing worse than a Democrat is a Republican.
What does the average user (or even the most extreme power user) do today that would require 10gb/sec? Even SSDs can't read or write data that fast yet. The only place I can think of where 10GbE is of any use today is in high end server and networking equipment, maybe linking multiple switches together, for connections onto big internet backbones for an entire site, streaming uncompressed video in a TV station or for connecting up a large NAS.
We have some of these at work where we do have the need for moving massive volumes of data around. We can get about 99.6% of theoretical throughput in actual use, thanks to the hardware offloading and large frame support. Besides the 10x faster to start with, that's way above any efficiency we get from the 1 GbE ports, though I expect if 10 GbE went commodity you'd lose all the hardware support and you'd be back to 80-90% range.
Note to sustain a data feed to one of these you at least need two SATA 6 gbps SSD drives in RAID0. On the receiving end we're not writing to disk, or you'd need ~3-4 RAIDed.
In our case we're feeding 4 10GbE ports on the same machine and using a 10 SSD RAID0 to supply the data with some headroom (we don't care if we lose the data if one fails, these aren't the master copies). We're just using software RAID, but thanks to all the DMA and offloading the CPU usage is quite low.
Now do I need this at home? Well, SSD speeds are far above the ~85 MB/sec 1GbE delivers, but so far the cost hasn't made it worth it. If I'm copying a gigabyte it takes 12 seconds, which I can live with.
For many things you do, you find 1gbit is enough. More doesn't really gain you anything. It is enough to stream even 4k compressed video, enough such that opening and saving most files is as fast as local access, enough that the speed of a webpage loading is not based on that link but something else.
Every time we go up an order of magnitude, the next one will matter less. There will be fewer things that are bandwidth limited and as such less people that will care about the upgrade.
As you say, 10gbit, or even more, is useful in many datacenters. But at home? What the fuck would I do with it? I guess I could... copy files faster from my desktop to server? Well except my server uses a magnetic drive that is slower than gigabit.
And, of course, you get to re-run all your cables. Gig works over Cat-5e, of course, which has been used for awhile, and with ASICs on smaller processes it actually usually works over Cat-5. So you can have some pretty old wiring and just knock in a gig switch and cards and call it good. 10gig needs Cat-6a. That is new, expensive, and a pain in the ass to work with.
Bandwidth is not something where we need "MOAR ALL OF THE TIMES!!" it isn't something we need to just seek to increase at any cost. Rather it is something that we need to have enough of to make it not a bottleneck for whatever it is we are doing. Well, for a lot of network stuff these days, gig is that. It is fast enough that it doesn't slow things down, at least not a significant amount. So that's all you need.
Same shit with BW anywhere else. You find that increasing memory bandwidth past a point with current CPUs is useless. Like with a Core i7-2600 increasing memory speed up to 1600MHz seems to help, however past that, doesn't matter except in synthetic benches. The memory bandwidth isn't an issue. With graphics cards the PCIe 3.0 upgrade did fuck-all since it turn out PCIe 2.0 4x is almost always enough bandwidth, and 8x is more than enough so PCIe 3.0 16x is doubling something you already have more than enough of.
As things progress we'll probably see more uses for 10gig, and thus it'll get rolled out wider. However it is the kind of thing that'll happen as needed, not that'll happen just because it can. We'll upgrade our building when it needs to be. When our uplinks are getting saturated, we'll take those to 10gig. When there is a reason to get it to the desktop, we will. However we aren't going to run out and drop 6 figures to go 10gig right now just for the sake of doing it.
satellite tv has to compress att does even fios hit the wall in QAM space.
The current high-cost item is the 10GbE switch. Those things are way too expensive (10+K range, and the plus goes way up)
Also, for flexibility, you want SFP+ ports and adapters for each port. None of those are cheap.
Cisco and most other vendors have made 10Gb ports too expensive and/or don't have a backplane that can effectively support 10Gb across all the ports. This is pretty ridiculous given how cheap processors have gotten. Even when they do support it, the licensing and maintenance costs can be crazy.
For that reason we're currently deploying several 1Gb connections to our VM servers through various switches (depending on costs per port, reliability needed and location).
I've been hoping that late 2013 is when 10Gb will finally appear for us on our campus trunks at least.
----- obSig
What took it so long? Likely because gigabit ethernet, and it's 100MB/sec transfer speed, is 'good enough' for the vast majority of users. You'd need a pretty substantial RAID array to come close to saturating a 10GbE link. And if you're moving 10 gig of files, taking 10 seconds rather than 100 seconds is a much less meaningful speed up than taking 100 seconds rather than 1000.
I needed, or at least I thought I needed, a very high speed connection between some VMWare servers and a backup server, all Dell PowerEdge with PERC controllers and 15k RPM drives. I installed 10Gbe links between them and thought I was the shit.
But, I'm only getting ~600Mbps of real throughput on those 10Gbe links, that's the same as a 1Gbe link. I checked and troubleshot the issue till I was blue in the face and I still couldn't get any more speed. The problem is that it seems that this is as much as the server, its bus and its disk subsystem can push.
I suspect that iperf would get multigigabit performance, but I'm not running a switch, I'm running servers and they can't push the data any faster. :(
10 gigabit ethernet is hard to do. I was bitching about this back in 2008, and 2009. 10 gigabit ethernet was supposed to take off in adoption, and prices were to plummet. They didn't. In the past, industry has been able to power through predicted technical challenges. Not this time.
In 2006, Broadcom released some 2.5 gigabit ethernet hardware, because it was easier to implement, and each XAUI interface could handle a 2.5 gigabit port. A group of 4 XAUI interfaces could be 4 2.5 gigabit ports, or a future 10 gigabit port. 2.5 gigabit ethernet did not take off, but neither has 10 gigabit ethernet. I hope 2.5 gigabit gets a second look, at becoming the cheap, good enough ethernet out there. Since physics has become more difficult, the high bandwidth networking should go to a more complex, and optimizable networking standard, like Infiniband.
Oh, sorry, I thought you said you could torrent the shit out of Good Times.
Never let a lack of data get in the way of a good rant.
it's taking so long because 10GBase-T is $$$ compared to 1GBase-T $.
I think the main reason is cost. I have been working with 10Gbe for several years writing drivers for PHYs and MACs. I've worked with a number of PHYs and 10Gbe is a lot more complex. For example, the SFP+ cables and modules each have a serial EEPROM that contains parameters needed to program the PHY. It's not just a simple RJ45 CAT5/CAT6 cable. As someone who has worked in 10Gbe drivers there's a lot more complexity. With some PHYs I have to query the serial EEPROM to make changes based on things such as cable length and whether or not it's active or passive or if it's copper or optical. Distances over copper are also usually limited to much shorter distances unless active cabling is used.
In terms of cost, a 1 meter copper cable is around $43 from www.cablesondemand.com. A 12 meter cable is $189. It's not like gigabit where you just plug in a CAT5 cable and go.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
And if you've ever looked at a NIC, you can see why. You get a modern gig server class NIC and it has this tiny little ASIC on it that does everything and draws less than a watt. Heck it'll probably drive two ports, if the hardware is on it. Then you get a 10gig NIC and it has a much larger ASIC with a big heatsink on it, and perhaps another chip as well. Guess what? That extra silicon cost extra money, as well as all the other related shit. And it just gets more and more expensive as you want more ports, like on a switch, and have to have a faster fabric. 48 ports of gig needs a 96gbps fabric, 48 ports of 10gig needs a 960gb fabric and costs don't scale linearly.
If you want a quick visual look up the Intel X540 and i350 sometime. They are Intel's current gen 10gig and 1gig 2 port cards. You can see a pretty substantial difference in the amount that goes in to them.
When you talk 10gig these days, you have to want it pretty bad to bear the cost. It is WAY more. As such it is very much a "only if you need it" kind of thing. We just bought a Dell Equallogic setup and opted for the 1gig model for that reason. We only have gig in the building, so the 10gig would only be for "future proofing" and it was a ton more.
Also often if you are over a gig, but not much, it is cheap and easy to just bond ports. Have two ports and get 2gig. Cheaper and works on existing equipment. The aforementioned Equallogic does that, the NAS has 8gig out to the client network and to the SAN (the 10gig model has 40gig to each). That can go a long way because it turns out that just because 1 gig isn't enough doesn't mean 10 is necessary and Cat-5e cable is cheap.
I don't really see mainstream boards with 10GbE. I see server boards and workstation boards mostly from around $400. A little more than 6 months ago but only a gradual increase over th past 3 years.
Ok #1 who does that? I mean that is not a very "home user" application in general. However #2 is gig is plenty for that. 1920x1080 24/30fps AVCHD PH video is 24mbps max. Blu-rays can in theory be 50mbps (between audio and video) mostly for MPEG-2 though in practice it is usually more like 25mbps AVC. Youtube is 6mbps for 1920x1080.
So even with the max Blu-ray rate you are good for two streams. Realistically you can do 4 streams at most data rates. Even when 4k stuff starts to happen, it'll be fine to do one, maybe two streams, as 64mbps is looking like the max for that on Blu-ray (and less on the Internet of course).
Thus where, precisely, do you need this 10gig bandwidth? Sure maybe some time in the future but, well, then get it at that point. Saying "well some day a gig won't be enough!" is silly. That's not the point, the point is that it is NOW.
Also, I'd argue that in your situation, in the unlikely event you needed more streams, a much, MUCH cheaper alternative would be to just put a dual or quad port NIC in your server and do a port channel to the switch. You bandwidth is increased out of the server, the clients and switching hardware are on gig, and a 4 port gig NIC is less than a 1 port 10 gig one.
My God, the luddites have taken over Slashdot tonight.
When I have 10Gb at home, I'll:
* Boot every PC from a remote server. No need even for local swap.
* Have a much better time doing backups. When I get a new computer, the first thing I do is boot a liveOS and run:
gzip /dev/sda | nc home.server 12345
and on the other end store the image. That way it can in theory go back for factory service if needed. The bottleneck is completely the network here, and it's slow even at gigabit speeds.
* Never worry about multiple devices maxxing out the switch trunks.
* Never worry about traffic spikes increasing latency on sensitive protocols.
* Etc.
You guys and your 640K lawns...
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Well, my hard drive can't write data at 1250MB/s so why send it over the network that quickly? So...there's that.
Most people I know barely benefit from gigabit Ethernet. Most people I know are not running Exchange servers and huge file sharing projects on their LANs but hosting their data on their local PC and using their network for E-mail and the web.
While 10-100Mbit made a huge difference to peoples' networking abilities, and going from that to gigabit helps with smooth transfers of larger files, there's still a lot of people running 100Mbit and quite happy with it because modern switches are pretty good at what they do.
Sure, as a geek, I'd love to have 1Tb/s streaming but its really not that relevant to most small business or home users.
For reference, at 12MB/s (100Mbit), you can transfer a 9GB DVD in about 12.5 minutes. At gigabit speeds, you can do it in just over one minute instead. Jumping up to 10GbE brings that down to around 7 seconds or so ... but why are you moving a 9GB DVD any faster than one minute? It takes longer than that to burn or watch.
- Michael T. Babcock (Yes, I blog)
Why do we still even need ethernet? How often do you need something other than IPv4/6 which could be done as its own layer since no one uses a bus topology anymore (did that ever get faster than a total shared 10 megabit capacity?).
now we need to go OSS in diesel cars
I wired my house some years ago with Cat5 cable. If I wish to create a 1GB/10GB home network, then obviously I need to replace my switches, but do I also need to pull all the cables and put some new ones in, or will my existing cable do the job (perhaps with a few packet errors)?
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
Here is a presentation that is almost 10 years old. http://www.ieee802.org/3/minutes/nov03/1103_CFI_2_5G_report.pdf
They note that "Broad level of interest in a solution between 1Gbps and 10Gbps in speed and cost."
and "A Wide belief that 2.5Gbps can provide good cost/performance"
The study group was defeated. Today, 10 gigabit ethernet still has a small market share. Maybe the industry should reconsider 2.5 gigabit.
Now I can have the OS maxed out handling network interrupts even after coalescing from an MTU of 1500 on 10Gb instead of 1Gb. Won't it be nice when stone-age vendors like Cisco allow jumbo frames to enabled by default, or at least not require a whole switch reboot to accomplish.
10GbE has been in major use at most ISPs for years now - there's almost no 1 GbE left. End users just don't see it because this connectivity is mostly in core networks and data centers. With the explosion of data traffic due to smart phones, wider adoption of 3G+ technologies and permanent connectivity, Mobile Backhaul networks also moved to 10GbE years ago (that's your local antenna fibers going to the core network).
The reason is not only the higher throughput per port, but what comes from that: you can get more throughput per U (the standard unit for racks) than with 1 GbE ; so, more throughput per rack. Given the price of a data center, you sure as hell want to make the best possible use of the space. Also, It's better to have 1x 10GbE than 10x1GbE in a trunk, obviously, because a simpler configuration is always more desirable as long as it fits the technical requirement. It also takes less energy to power 1x 10 GbE port than 10x1GbE ports.
What is true, however, is that 10 GbE is just recently making it into *Enterprise*, which I reckon is where most of the posters here work at so they might have a different perspective for 10GbE adoption. Of course, very large enterprises like banks and Facebook-like have had 10GbE in their data centers for ages - just not so much in their local agencies or offices. As for ISPs, they are currently starting the testing and deployment of 40/100GbE.
The big exception are Universities and Research Centers, as some of them require the highest possible throughput. We sold a 100GbE test equipment to the CERN, for instance, because they have so much datamining to do from the LHC that they have to offload some of the analysis to a university in Lyon, France, and another one in Amsterdam, Netherlands (that's 1600 Km of 100 GbE and that's awesome). To my knowledge it's the only non-ISP entity in Europe that has that kind of requirements.
If you want to play with fast (10G+) networking at home, the smart way is to buy infiniband gear on ebay. There's quite a supply from compute clusters being torn down. Older SDR (10G) cards run $30-50. DDR (20G) a bit more and QDR (40G) for a few hundred per card. Buy a cheap copper cable for cross connect and you're done. Or preterminated fiber cables if you need distance, the cards usually handle that too. Some cards also handle 10G and 40G ethernet as well. Need a switch? 36 port QDR switches typically go for $1000. That's 1.4 Tbps worth of bandwidth.
I bought a couple of Mellanox cards that do both 40G ethernet and FDR (56G) infiniband. Between my two linux servers, I get about 37Gbps when using 2+ tcp connections. While bandwidth is about the same, infiniband latency is about half that of ethernet, so I run IP over infiniband.
Apart from being fun (this is slashdot after all), why would you want this? Because it remove the network as a bottleneck and changes the way I think about resources. File transfers are limited by disk performance, there's never network congestion, etc. The only thing that could saturate the link would be memory to memory copying (think VM migrations). Either way, it will be a long time before I worry about network performance again...
What's the point of having 10GbE if disk performance hasn't gotten much better and, at least in the US, average speed for Internet connectivity has stalled over the past decade? At $WORK we use 10GbE almost exclusively on the backbone, and a few hot-spot servers like tape backup systems. Gigabit speeds (and less) for other systems are completely adequate.
???? What?
There are a variety of 10GbE cables and connectors. I have the same connector on my 10GbE cables as on my 1Gb and 100Mb before that.
You CAN get it in fiber if you have some need, but for a small net.. standard 5e cable works fine. Interrupts ... not so well either on windows or linux - over 500MB/s, and one side or the other starts getting cpu bound on samba transfers... windows file transfer protocol doesn't scale beyond 1 processor per user (all IO is over 1 tcp/ip connection/user)...
I have Cat7 fitted to my home. And in the last one as well. Not everybody is unprepared for the future.
10 GbE support on motherboards is not what's needed to increase adoption. Decent prices on 10GbE switches is what's needed. For small companies, 10 GbE is just not practical. You can't get any kind of decent port density without a chassis based switch and they're priced out of our reach. Even if I just wanted 12-24 10 GbE ports, there's not really a cost effective way to get that. Sure, I could look at a Nexus 3000 series at a cost of $30k + SFPs, but then I've got probably $40-45k in a single switch when it's all said and done. Talk to me when I can get that for $10-15k and I might buy one.
A short non-exhaustive list of reasons:
- New tech typically has to burn in for while before people trust it.
- Many companies who needed that level of speed were already invested in 4Gb and/or 8Gb Fiber.
- The cost wasn't dropping very quickly
- Slow desktop hard drive speeds meant 1Gb to the desktop was more than sufficient
Now two of the big reasons it's more prevalent now:
- Virtual desktops and application virtualization have significantly increased the bandwidth and low latency needs on your network
- More (and somewhat cheaper) onboard availability in new servers makes the transition decision easier, especially in smaller budget operations