10-Gigabit Ethernet Standard Approved

A little birdie brings news that that 802.3ae standard for 10 Gigabit/second Ethernet has been approved. Everyone out there with Gigabit Ethernet - you are now officially obsolete. The new standard is fiber only, no more of that nasty copper stuff.

Comments

Approved != Affordable

[CodeMonky]

Approved or not it will still be some time before costs come down enough so that comapnies can justify replacing their gig backbone with 10gig.

not obsolete

[Mortin]

considering hdds can hardly transfer at 1gbps, gigabit is hardly obsolete... yet :)

Re:not obsolete

[larien]

One word: striping. If you put enough disks in, you can get more than 1gbps out of a disk array. Realistically, though, you're limited to using this in two places:

1. Large server with many, many disk controllers and even more disks
2. Network backbones

It'll creep in to the second quickly enough (once Cisco et al support it in hardware), I'd imagine (we already have a 4gbps backbone using 4 gigabit lines in our site) and the former will start happening at the top-end installations of E15K's and the like.

Re:not obsolete

[GeckoUK]

You are correct, but only in the case were you have a network of two computers.

In the real world a company deploying this is likely to have hundreds if not thousands of machines all connected at once.

Re:not obsolete

[GigsVT]

We're going to need something to replace PCI before we can use 10Gbit ethernet fully though. Even 64bit 66Mhz PCI has a max (in theory) of 528 Megabytes/sec.

On a side note, I have sucessfully pulled 130Mbytes/sec out of 5400 RPM IDE Disks on 3ware controllers, with a cost less than $9000. 3 controllers, 24 disks, 64 bit 33Mhz PCI. RAID 0 over 5. So the potential is there to exceed current GigE, without too many disks or controllers, or getting too expensive.

It would also help a lot if we could get regular gigabit ethernet working well first. I think there was a story here on Slashdot not long ago that showed that most GigE cards had trouble pushing over 400Mbits even with large frames. Only the expensive $500 one came close to it's full potential (900Mbits). My experience is that without jumbo frames, there is hardly any advantage with lower end GigE cards.

Re:not obsolete

[Ogun]

Oh, you mean like this:
Cisco 12000 10Gb line card
or like this:
Catalyst 6500 10Gb line card

Cisco did announce these a while ago.

Re:not obsolete

[questionlp]

The highest speed PCI-X (64-bit @ 133Mhz) is capable of reaching ~1GByte/sec which is just about the speed of 10 Gig Ethernet. There was/is the promise of Araphoe (sp?) that resembles AMD's HyperTransport but would be used for expansion cards rather than a chip-to-chip pathway.

The other bottleneck with even high-end Intel-based servers could easily choke when dealing with not only 10 Gig Ethernet but also add Fiber Channel, multiple channels of Ultra 160 or Ultra 320 SCSI RAID, etc., since the memory bandwidth (and processor bus speed?) would then become the possibly the next bottleneck. RISC servers don't have that much of a problem just yet, but sooner or later it will be.

Re:not obsolete

[monkeydo]

Please put down the CCNA study guide and back away slowly. Ever since the invention of switches most people haven't had to worry about collisions. We're talking about full duplex, 2 hosts on a collision domain. In other words no collisions. If you are getting collisions on GigE or switched FastE you are doing something very wrong.

What an informative link.

[taliver]

Here's one that might be a little more informative. I leave the google link to someone else.

In meaningful terms

[Throatwarbler Mangro]

10Gigabit/sec = 1.25Gigabytes/sec

1 LoC (Library of Congress) = 10 Terabytes = 10,000 Gigabytes

That's 0.000125LoC/sec, or roughly 2.22 hours to transfer the entire contents across 10GigE.

Wow.

Re:In meaningful terms

[glwtta]

I thought the time honoured (and extremely relevant) measure of LoC/s has been officially replaced with HG/s (Human Genomes per second)? Mostly because it allowed for a lot more flexibility in making up figures that don't really tell you much, if I remember correctly.

Re:In meaningful terms

[Dark Nexus]

As a slight correction, when it comes to baud ratings, 10 Gigabit/sec = 1 Gigabyte/sec

It's 8:1 for storage, but generally 10:1 for network ratings (an example - more for serial ports, but it still applies), thanks to a header and a footer bit sent with every byte. Sometimes (rarely), throw in a parity bit for good measure.

Mind you, that's still only 2.78 hours.

Re:In meaningful terms

[psychos]

This is incorrect. Low speed serial interfaces do tend to use a start bit and a stop bit, but higher speed interfaces generally do not.

I'm not very familiar with 10gige technology yet, but my brief research shows that it uses 64B/66B coding (e.g., 2 overhead bits out of every 66). Running at a clock rate of 10.3125GHz, that gives you a full 10Gbps of throughput, or 1.25 GB/sec.

100baseT uses 5B/4B coding, which does result in 2 overhead bits out of every 10 just like your serial line example. However, 100baseT actually runs at 125MHz so you do get a real 12.5 MB/sec out of it.

Of course, if you really want to be picky about "LoC/sec" or whatever pointless measure the popular media has latched onto this week, you need to consider the overhead of TCP headers, whether or not you want to allow jumbo frames in your calculations, and so on.

Its not fair!

[GnomeKing]

*looks at his 14.4k modem*

*looks at the article*

*looks at his modem*

*cries*

Re:Its not fair!

[Jugalator]

Upgrade to 56k, you cheap, whining bastard! ;-)

With this annoucement

[SkyLeach]

It should be obvious that to burry copper is completely obsolete. Per yard, fiber should be cheaper to manufacture and bury.

10Gb speeds should be enough for anybody, so start building the infrastructure now and leave the telcos in the dust.

Will they do it? No. Why not? Because they think that they should bury the copper/fiber hybrid cable that they have been burying and come back and do it again later.

Burying cable is the most expensive part of telecomm.... retards.

Re:With this annoucement

[Hieronymus Howard]

10Gb speeds should be enough for anybody

Just like 640K is enough for anybody.

HH
--

Re:With this annoucement

[stevelinton]

Actually, burying pipe is the most expensive part..... I worked briefly for a university computing service a few years ago and they spent an absolute fortune to buy a network of yellow plastic pipe connecting all their buildings. A relatively trivial incidental expenditure was to pull some cable through it. When that sort of cable is obsolete, a further trivial expenditure will replace it, etc....

Copper vs. Fiber

[jandrese]

IIRC the original Gig-E hardware (if not the original spec) was Fiber only as well. Eventually people started coming out with copper hardware to save on costs. In most cases, the only real advantages to fiber are the long cable runs and the immunity to interference in noisy EM environments (like your typical computer room). The downside is the cost.

Oh the possibilities!

[Yoda2]

Lynx will rock!!!

wither Cat6 ?

[green pizza]

My building recently had new copper installed. Previously had Cat5 (great for 100BaseT) but was upgraded to cable meeting the specs for the latest Cat6 draft spec (rather than just Cat5e).

Is 1000BaseTX the end of the line for copper? Or will there eventually by a 10000BaseT that will run on Cat6?

Re:wither Cat6 ?

[Barche]

This thing runs at 10 Gbps. Ethernet uses Manchester encoding (+-=1, -+=0), which means you have to double the bps to obtain the bandwidth. So you need (about) 20GHz of bandwidth on the cable. At that frequency, losses in a copper cable are just too high. You'd need to use either wave conduits (big metal pipes, not an option) or optical fiber.

Re:wither Cat6 ?

[ivan256]

Gigabit ethernet took all it's electronic specifications from fibre channel. When Gigabit came out, there was already copper available for fibre channel, and there was nothing stopping you from using those GBICs. The recent development was that they figured out how to get the signal over regular CAT5.

I'm sure that there will be a copper spec for 10 gigabit too, it's probably just not ready yet. Consider that people will be wanting to use this on the backplane of embedded network hardware, and blade servers.

What you can do with 10Gbps

[Jugalator]

Time to...

Download a typical 100K pr0n JPG: 0.00001 s
Umm...
Download a 650Mb ISO: 0.52 s
Hm...
Download 2 650Mb ISO's: 1.04 s
Eeh?!
Download 100 650Mb ISO's: 52 s
Wow!
Download 1000 650Mb ISO's: 8.7 min
Jeez!
Download an image of CowboyNeal: 12.31 hours
Bah... Tech still need to catch up.

Hit me with the clue-stick please!

[Andy_R]

Can someone bring me up to speed?

1) The link shows it has been approved by "Revcom" - who are Revcom, and why should I be interested in their approval?

2) Seeing as ethernet seems to speed up by an order of magnitude each time, why does the standard not allow for many more x10 jumps?

3) How far is 10Gb Ethernet from getting to the consumer/business market?

Re:Hit me with the clue-stick please!

[GigsVT]

revcom

IEEE

Consider yourself hit with clue-stick.

As if 1000BaseT didn't suck enough CPU cycles

[green pizza]

I'm guessing 10GbitE will be used for inter-switch and inter-router connections long before it gets to the desktop. Ever looked at performance comparisions between 100BT and 1000BT between just two PCs? A couple years ago the difference wasn't much... NICs weren't efficent enough and the host PC's didn't have enough CPU power to handle that many tiny packets per second. Jumboframes and faster CPUs have helped a lot since then, but we're still a long ways away from even 90% utilization between two PCs with 1000BT. And here we are with 10GigE, with 10x as many packets per second.

I'm I the only one that thinks the only efficent 10GigE NICs are going to be PCI-X cards with an onboard 2.6 GHz P4 co-processor and 512 MB of buffer?

Re:As if 1000BaseT didn't suck enough CPU cycles

[Enry]

There was an article in the Linux Journal a few months ago (February issue I think) that talked about intelligent network cards. They had an onboard XScale CPU and its own OS and TCP/IP stack.

What would happen is the OS (Linux) would get intercepted at the socket layer and pass the data to the network card. The card would then handle the process of building the packet and all the remaining layers of communication.

This allowed for a high amount of main CPU time left over for actually doing processing while the network card CPU was focused on handling the TCP/IP packet work. IIRC, you could saturate a 1Gb line with data at only 5% main CPU usage.

Re:As if 1000BaseT didn't suck enough CPU cycles

[bluGill]

My company once did this for a 25Mhz dec machine. We discovered that you now need a new protocol to the adaptor card, and the overhead of that protocol is equal to a well tuned tcp/ip stack. So if they can actually make this work, what it really means is that linux hackers should spend some time to tune the stack.

Note though that tuning the stack may come at the expense of maintainability, or flexability.

Re:Why Bother ?

[CausticPuppy]

Actually, physics won't allow you to use "unlimited bandwidth."

Re:Can I actually use that?

[larien]

As others have pointed out, this is irrelevant for "average" PCs. Where it comes in useful is for high-end servers, network backbones and (possibly) clusters which throw a lot of data around.

Slashdot Effect

[WolfDeusEx]

10Gigabit still woun't standand up to the slashdot effect

Switch prices

[stevelinton]

Switches for these speeds are still kind of large, awkward and pricy. We had a visiting lecturer from one of the major players in this level of kit talking here about 6 months ago, and their top-end product (he showed a photo) was a 48-way full bandwidth 10Gb switch, It filled two full height 19" racks, consumed 20kW and cost upwards of $2M.

Of course they've probably come down a but in the last few months...

10Gb over copper? Won't happen!

[xt]

The reason is material properties.

Six months ago, I had the chance to talk with the 3Com technical manager who was on the board drafting the spec.

What he said was very simple; all tests indicated that the only way to have 10Gb over copper is to limit the connection distance to centimeters!

1Gb already pushed the envelope for copper, using all pairs, multiplexing, and error correction; 10Gb is just not possible.

Orbital RAM

[Rupert]

You thought the latency was bad with RDRAM!

The speed of light will be a hindrance to your plan.

Fiber only - for the moment

[Ashurbanipal]

Every time they come out with a new standard for ethernet it's the same old schpiel - "you need this special expensive coax/shielded-pair/fiber-optic etherhose to make it work; you canna change the laws o' physics Cap'n!"

Then eight months later somebody figures out how to run it on old lamp cords and string.

Don't rush out to buy fiber unless you need the noise isolation (glass is great for that!) and don't care about the cost.

Many are missing the point here

[buss_error]

Sure, you can't use all 10G on ONE machine. Even a server can't use all that speed. Even using many NICs. (Buss congestion, ya know.) That isn't the point here. The point is that instead of having to segment a lot of traffic off to a vlan or other workaround, that traffic can be supported on one lan. This reduces equipment, interconnections, configuration, and alot of other headaches. In short, you can reduce the total points of failure.

And remember, Intel isn't the only hardware platform out there. While I don't know of a hardware platform that can make fully support the speeds needed, there are some that can support better than 4000 Kbps now.

Arg.

[be-fan]

Now my PC133 RAM is *really* obsolete. It can't even handle an ethernet connection!

What use is this? I can think of a few...

[sacremon]

I work in a data center for a major ISP/backbone provider. While we've got OC48's coming in from the backbone, it's 1Gb ethernet to the LAN distribution routers. With 10Gb ethernet, we can finally fully utilize that incoming bandwidth without having to use a lot of ports.

Another good use is the emerging use of iSCSI, or SCSI over ethernet. 1Gbps ~ 100MBps, but more likely around 60-80Mbps. With 10Gbps, a SAN based on iSCSI will actually be able to use the throughput of those SCSI drive arrays.

Eventually this will trickle down to the desktop, but not right now. So it doesn't really matter what PCI can handle - this isn't presently meant for it. BTW, 133MHz PCI-X will give 10Gbps, so if you have a dedicated PCI-X bus to that adapter, you can handle it will today's technology.

Obsolete?

[mindstrm]

I know you say this jokingly.. or do you?

This is not THE new standard, it is A new standard.

It is THE standard for 10Gbps ethernet. Nothing more.

Gigabit is hardly obsolete when a) very few corporate networks are using Gigabit outside the server room, and...

Your average workstation can probably not even push 10Gbps, or anywhere near it in the first place. (Of course, that's not as big a deal, because it's ethernet, right? A single host can't max it out anyway.. the higher capacity means more hosts with lower latency.)

Bandwidth bottleneck on motherboard - AGP?

[Anonymous Coward]

Okay, now PCI is a bottleneck. Even 64bit PCI, quad-pumped, would still only support around 8Gb/sec... So I suggest that we repurpose the AGP port. We can go back to boring old PCI for the graphics card, so lets implement AGP network cards! Of course, it won't be the "Accelerated Graphics Port" anymore, it will be the... "Always Generous Pornography" "Accelerated Game Piracy" "Automatic Grits-to-Pants"

Anyone knows the MTU?

[Oestergaard]

Does anyone know how big packets one can send thru such a pipe ?

100MBit maintained the same MTU as 10MBit, 1GBit maintained the same MTU too - leading to severe problems with performance. It's bad enough on 100Mbit, it's horrible on 1Gbit, to think that they maintained the 1500 byte limit on 10Gbit gives me the shakes...

Yes, I know about "jumbo frames", and I challenge you to find an affordable 1Gbit switch that actually supports it.

Anything below 64KByte packets would be insane as I see it.

Anyone knows ?

Hairy smoking golfballs

[alienmole]

(come to think of it, this implies that physical size is a fundamental limiting factor on the speed of computers - it does no good to have an infinitely fast CPU if its parts can't communicate rapidly due to speed-of-light delays...)

Researchers have realized this for decades. Before enormous silicon chip densities became ordinary, engineers at IBM (IIRC) used to say that the future of computers was "hairy smoking golfballs". This captured a number of important characteristics of very fast computers:

• Hairy because of all the wiring that would be involved (like the rubber strings in the core of old golfballs)
• Smoking because of all the heat that would be generated
• Golfball-sized because of latency issues, as you said
• Golfball-shaped because a sphere is the most efficient use of space for components that need to be interconnected.

Since those days, Intel and its competitors have fulfilled all of these predictions except for the spherical shape, which is much more difficult and not as important as the other characteristics.

A Pentium 4 is hairy - those 55 million transistors have a lot of connections; and smoking, as anyone whose CPU fan has broken can attest. It's smaller than a golfball in cross-sectional area. That size isn't just to make them more convenient! If a physically bigger CPU would be faster, you can bet someone would be building them.

What's so bad about copper?

[Rorschach1]

I can strip copper wire with my teeth, and terminate it with a Leatherman tool. Until I can do that with fiber, my network's sticking to good old-fashioned electrons.