Ethernet at 10 Gbps

An anonymous reader writes "This article talks about 10 Gigabit Ethernet and asks, 'But just how much data can a person consume?' Currently at work, we're working on a major project to re-architect our core application platform so that the different systems can be de-coupled and hosted separately. The legacy design implicitly relies on systems being in the same LAN due to bandwidth-expensive operations (e.g., database replication). Having this much bandwidth would change the way we design. What would you do with this much bandwidth?"

Comments

Hmm...

[ravenspear]

What would you do with this much bandwidth?"

Check out more unusual positions.

What would I do?

[Anonvmous Coward]

The company I used to work for was sending very high resolution images from multiple cameras uncompressed from one unit to another to perform analytical operations on them. I think they manged to work at a gigabit, but 10 would be much nicer for them.

What would Joe Sixpack do with it? I'm not sure at the moment. Thing is, since we're working within our limitations today it's hard to concieve of whta use it'd be. However, what happens when it becomes commonplace? It does open doors. Imagine if cable companies traded in coax for ethernet. They could easily send uncompressed HDTV. That'd be pretty slick.

Re:What would I do?

[nine-times]

Thing is, since we're working within our limitations today it's hard to concieve of whta use it'd be.

Isn't that always the way? I remember having a 20Mhz IBM PS/2 and wondering "How am I going to use all this power?" And the 30 MB hard drive- how would I ever use all that space?

It seems like when we have the capabilities, we find something to do with the extra. HDTV sounds probable, and more bandwidth can only help working over networks on a mass scale (remote home folders and roaming profiles, VNC/Citrix), but you never know. When processors were getting to the 1Ghz point, a bunch of industry analysts were predicting "Now that we have enough power to make working speech-recognition software, we can finally ditch those keyboards!" Yeah, right.

The big concern is, with the extra bandwidth, will Microsoft see this as an opportunity to release new, extra-inefficient network protocols?

Re:What would I do?

[egomaniac]

In your world, 'hdtv' might be 1280x1024, but in the world at least a few of us live in, HDTV (notice the caps) isn't, it's 960x540(assuming square pixels) and I imagine less than 24 bit color (All this analog stuff!). I don't know, but I would also guess that it's the same frame rate as regular old TV (~30 fps).

There are two HDTV resolutions in current use, known as 720p and 1080i. 720p is 1280x720 60fps, and 1080i is 1920x1080 30fps (60 interlaced fields). Both of them are 24-bit truecolor.

I have no idea where you got 960x540 from, as it does not correspond to any HDTV resolution. I'm also not sure what the reference to "all this analog crap" is supposed to mean, as HDTV broadcasts are entirely digital.

Reminds me of an old quote...

[Anonymous Coward]

Just get us the bandwidth - we'll manage to piss it all away. Easy!

1G should be enough for anyone.
-- Nicholas Cravotta, 2004

640K should be enough for anyone.
-- Bill Gates, 1981

Remote Virtual Immersion

[weston]

But just how much data can a person consume?

If I was going under the knife remotely, I'd want the surgeon to have as much bandwidth as possible (and very, very, very low latency).

Re:Remote Virtual Immersion

[MP3Chuck]

Imagine a malpractice lawsuit!

Exhibit A: Surgery Log
[DR]Surgeon opened Xx[Patient]xX's abdomen with a scalpel.
[DR]Surgeon punctures Xx[Patient]xX's stomach with forceps.
Xx[Patient]xX: OMGWTF??!!
[DR]Assistant: ROFL PWNED!!1
[DR]Surgeon: STFU N00B i ping 350
Xx[Patient]xX: w/e

Re:silly question

[hunterx11]

In the future, spelling will not be a part of the post at all, but will be defined by external Cascading Spelling Sheets. Of course, in order to implement this we all have to learn Chinese.

What would I do?-Horror-vision.

[Anonymous Coward]

"What would Joe Sixpack do with it? I'm not sure at the moment. Thing is, since we're working within our limitations today it's hard to concieve of whta use it'd be."

The Goatse.cx experience in holographic, 5.1 surround-sound, smello-tactile-vision.

The Network Is The Computer (tm)

[gusnz]

OK, so for stuff like streaming MP3s and so forth, this is a little overkill for the current style of usage. However, where I think this will come in useful is for stuff like remote disk and memory access over IP.

With a 10G LAN, you'd be able to come up with a great distributed computer system (e.g. for compiling software). IIRC protocols are in the works now for native-ish memory access over networks, turning a network into one huge computer, and you can already access remote disks with the right software. Imagine the simultaneous distributed encoding of several HDTV streams to redundant archives on several different computers, and you'll probably find that more bandwidth = better.

So yeah, there'll definitely be possibilities for this sort of stuff, even if it is only as a base requirement for the post-Longhorn Windows version :).

NC-PC-NC

[basking2]

So, we used to have little dumb terminals that talked to the big smart backend. Then computer became cheaper and we had Personal Computers, but we have to manage and distribute all these updates and it's a real pain and it sometimes destroys your computer during the upgrade/install process. Now we can swing the pendulum back towards the Network Computer a little more.

This isn't a new idea. Software companies like MS would love to sell you a subscription to MS Office which you renew and they in turn patch and maintain the software on your company's server or on the MS servers. It's a neat idea for sure. Companies like Novel have made some interesting claims about Network Computers.

There is also the whole Plan9 type of mentality too.

Re:10 gigabit is kinda much

[cynical kane]

PCI is 33Mhz, not 33 MB/sec. 33 X 32-bit-bus = 133 MB/sec. PCI-X goes up to 133 Mhz and 64-bit, so that's 800 MB/sec.

It's not the bandwidth

[overshoot]

It's the latency. No matter what your bandwidth may be, some tasks (e.g. file servers) need to be "close" to keep latency from being nasty.

"Close" applies both in physical distance (I have to count picoseconds for the kind of stuff I do) and in network distance, since every router adds considerably.

For some jobs (backup is a classic) latency is relatively tolerable. However, even for those you have to watch out because one error can cause the whole process to back up for retries. Ten to the minus fifteen BER sounds good until you look at what it can do to your throughput in a long-latency environment.

Re:It's not the bandwidth

[kasperd]

I have to count picoseconds for the kind of stuff I do

Unless you are working with individual gates inside a chip, I doubt picoseconds really matters. On ethernet we are certainly not talking picoseconds. We are still limited by the speed of light, so it would take the signal 100 picoseconds just to get through the RJ45 connector. With a 1.5m ethernet cable there will be at least 10 nanoseconds of roundtrip time, because that is the time it takes light to travel 3m.

Re:It's not the bandwidth

[Some Dumbass...]

I have to count picoseconds for the kind of stuff I do

Unless you are working with individual gates inside a chip, I doubt picoseconds really matters.

I think you're missing something. If the cabling adds a constant delay to any times this guy's measuring, then he can still measure times in picoseconds (assuming his timer is accurate enough, of course). The fact that network cabling would add nanoseconds to a recorded time is irrelevant. Just as long as it doesn't add a variable delay (I wouldn't recommend doing this timing through any sort of switch or router, for example).

Not that this guy is necessarily using ethernet for what he's doing. Note that he didn't actually say that -- he just said that you had to be close for the kind of stuff he does.

One possibility is that the guy's a physicist working with a particle detector. He's could be talking about detecting the exact timing of the decay of various particles. If these decays occur on the order of picoseconds, and his equipment can accurately keep time in picoseconds, then the fact that the cabling adds, say, 5ns to all of the measured times is no big deal. Just subtract 5ns from everything. That's good enough to get the relative times of all the measured events, e.g. the amount of time between the detection of emissions created by the initial collision (and thus presumably particle creation) and the decay of the various particles.

Lots

[BrookHarty]

We use lots of shared drives, remote desktop applications, X traffic, moving core files, database dumps, email with very large attachments (exchange to boot).

We migrated to 100meg, it was like night and day, and we still need more. We finally got 1gig to IT's network, and still to slow to push files with lots of users.

We have a burstable OC192 to our 2nd remote datacenter, OC48/12's to the smaller datacenters. But this is for production networks that need bandwidth, not desktop usage.

Also, my buddy in Japan just told me he got 100Meg DSL, the stuff you can do when bandwidth isn't a concern. Already Internet TV stations popping up there, amazing. Can't wait for this to catch on in the US. I just upgraded to 6M DSL from speakeasy, and its too fast for fileplanet.

Speed kills :)

Presidential Bioinformatics

[buckhead_buddy]

But just how much data can a person consume?'

100 Megabytes per chromosome
x 23 chromosomes per gamete
x 20 million gametes per ejaculation

Therefore Ms. Lewinsky can consume roughly 46,000,000,000 megabytes
(assuming that there is no overflow to a dress)

How much can you consume?

Re:Presidential Bioinformatics

[Anonymous Coward]

Close, but not entirely accurate.

A single gamete has 1.5 billion individual base pairs. Of course, that's base-4, since DNA doesn't work off of binary. ACGT is what you're made of. :) In other words, you have 3 billion bits per DNA strand. The average male ejaculation contains around 150 million sperm. This means that there is a total of 450,000,000,000,000,000 bits of information, which turns into 56.25 petabytes of information. That number is close to yours, but your information is wrong.

The fact that I just corrected you is pretty sad as well.

Way overkill

[JRHelgeson]

As a CCIE, I have been designing networks for years. I have analyzed traffic to/from desktops and watched traffic to the average desktop never even get above 27mbps. This is due to the average file size of the transfer which is rarely above 10 megabytes. At 10 megs, it only takes a few seconds to get it transfered and it only has a few seconds to get up to speed, by the time it gets all revved up, the file transfer is complete.

High-end workstations such as CAD with gigabit connections, working with 500 mb files, or multi-gigabit video files will occasionally reach 500 to 600 mbps, and even then only for a couple of seconds. At these speeds, power users can use that network connection as if it were a local drive, because at those speeds you are matching the speed at which you're reading/writing data to your local hard drive.

The only time I've ever seen near gigabit traffic at a steady pace was at network servers, where traffic can reach a steady 600mbps on a single gig link - which is maxing out the speed at which the server drive can read/write data to its hard drive. Think of it this way, a 1 gigaBIT link can transfer a 1 gigaBYTE file in about 10 seconds, that's FAST! Conversely, it takes nearly 20-30 seconds just to write that large a file to the hard drive.

Now, on a Cisco 6500 core switch, or a Cisco GSR 12000 where traffic is aggregated, these are the only places where I've actually seen multi-gigabit traffic rates, and that was across the switch fabric - not all directed to a single interface.

The 12000 GSR already has a 10gb interface, it is a single line card that takes up a full slot. It sells for about $60,000 and is used to move data from the switch fabric of one GSR to another GSR, which means you need to put in 2 of them at a mere $120,000 to get the two connected.

Moving to optical links, you can get up to 36Gbps using Dense Wavelength Division Multiplexing on multimode fiber. This uses several colors of laser light to transmit multiple 'channels' across a single fiber link.

Even at these tremendous speeds, they are only used at traffic aggregation points, again because any network device, even a turbocharged SAN couldn't handle reading/writing at those speeds for anything longer than a quick burst.

I say this: If you think that 10gig/sec is your answer, you're looking at the wrong problem. You can get the performance you need at gigabit rates.

I'm not saying that we'll never need 10gigabit to the desktop, just not until we solve the hard drive bottleneck. Solid state storage could solve the problem, but we'd need to have solid state drives that store 100gb of data in order to match the throughput of the network.

Re:Way overkill

[dbarclay10]

Most of your argument rests on people not being able to read/write data from hard drives fast enough to use the network bandwidth. Some examples:

The only time I've ever seen near gigabit traffic at a steady pace was at network servers, where traffic can reach a steady 600mbps on a single gig link - which is maxing out the speed at which the server drive can read/write data to its hard drive. Think of it this way, a 1 gigaBIT link can transfer a 1 gigaBYTE file in about 10 seconds, that's FAST! Conversely, it takes nearly 20-30 seconds just to write that large a file to the hard drive.

More:

Even at these tremendous speeds, they are only used at traffic aggregation points, again because any network device, even a turbocharged SAN couldn't handle reading/writing at those speeds for anything longer than a quick burst.

And lastly, your conclusion:

I say this: If you think that 10gig/sec is your answer, you're looking at the wrong problem. You can get the performance you need at gigabit rates.

Given your premise, you argue for your conclusion quite well. I don't, however, think your premise is accurate. Or perhaps better, I don't think it's relevant. First and foremost, there's all sorts of storage mechanisms which can transfer data as fast or faster than 10Gbps. Think solid-state drives and some decent-sized drive arrays (they don't need to be *that* large, we're talking roughly 1 gigabyte per second; that can be done with 5-10 consumer-grade drives, let alone the arrays of hundreds of high-end 15kRPM SCSI drives and the like). So on the basis of storage speed alone, your argument fails.

Second, what does storage speed have anything to do with it? You mention servers not needing this - a *huge* number of servers never touch their drives to read the data they're serving. Drive access == death in most Internet services, and people invest thousands of dollars in huge RAM pools to cache all the data (they used to invest tens of thousands, but now RAM is cheap :). So for a huge number of servers, drive speed is simply irrelevant; it's all served from RAM and generated by the CPU, so unless you're trying to say that CPUs can't deal with 10Gbps (which you aren't, and quite rightly), the conclusion falls down again.

Do desktops need this? No, of course not. If that's what you're really trying to say, then all fine and dandy, just say it. Acceptable reasons would be "people don't need to be able to transfer their 640MB files in less than 10 seconds" and "their Internet connections aren't even at 10Mbps yet, they certainly don't need 10Gbps!" However, you'll find that this technology quickly percolates downwards, so at some point in the future people will be able to transfer their 4GB (not 640MB at this point) files in a few seconds, and their "little" 640MB files will transfer near-instantaneously.

Argh! No more!

[namespan]

Dude, what *is* it with the porn meme? There's like 15 comments to that effect already in this thread, before it broke 50 comments. If every slashdotter was as dedicated/addicted to downloading pr0n as is stereotype goes, the whole freakin' internet would have been DDOSd long ago.

Yeah, I know it's popular, but geez. Not all of us are spending our time gazing and wanking. Some of us actually code (and even talk to women!)

I hereby banish this to the Beowulf cluster of memes, along with Soviet Russia/Hot Grits/Profit!

Re:Argh! No more!

[optikSmoke]

I hereby banish this to the Beowulf cluster of memes, along with Soviet Russia/Hot Grits/Profit!

Umm, ya. Well done. The, um, banishing of things into..... popularity. That'll be effective. We all know how unused each of those oft-repeating jokes are. Oh, wait......

damn.

All your base are belong to porn?

Re:That's exactly the quote I remembered

[SillyNickName4me]

Well..
I wish my compu had less ram... so that a system dump takes a bit shorter.. ;)

oooh

[iamdrscience]

IDE over IP. Yes, it does exist.

The Copenhagen Metro

[empaler]

Had some pretty slick security cams installed in them from the beginning (~3-4 years ago) - but they couldn't use them. Why? Not enough bandwidth to send the images uncompressed. Which was what they had set them up to do. Solution? Turn off cameras. Wait a few years for more funding.

Re:What would I do with this much bandwidth?-Music

[Lord Prox]

Ya know, so far everyone seems to think of this as a long distance pipe. It's not, it ethernet. RTFA useful distance is in meters *NOT* kilometers. This is an intraoffice connection not a WAN pipe.

Patches.

[empaler]

Several hundred megabyte patches.

Oh.

Re:What would I do with this much bandwidth?

[leapis]

This much bandwidth isn't going to help you do any of these things. I upgraded my network to gigabit ethernet about a year ago (from 100 mbit), and much to my surprise, the speed increase was only about 3 times when copying files from one machine to another. I did a little math, and found the answer. Your average ATA hard drive, even at max bus speed, only delivers 0.8 Gbps. And in the real world, you are lucky to get half that from a single drive. In my own test transfers from RAID1 and RAID5 arrays, my transfer rates never once exceeded 0.70 Gbps. Until there is a fundamental increase in the amount of data you can get off a spinning disc, its not likely that a home user is going to saturate a 1 Gbps line, much less a 10 Gbps line.

Dumb terminals? Cluster computing?

[Entropius]

Let's see. There are about a million pixels on my screen (1280 x 800). Assume 24 bit color, so that's 24 megabits per frame.

This at 60 fps will be 1.44 Gbps.

So 10-Gbps ethernet is enough to stream the output of a monitor, *uncompressed*, at full framerate, to either a dumb terminal or another computer. Even the most elementary compression (only reporting changed pixels, or PNG/jpeg techniques) could cut this to a fraction of 1.44Gbps.

More generally, it could allow more of the things that are currently on the PCI/USB bus to become external, and could become a more flexible replacement for USB. Scanners, cd writers, audio devices, you name it ... lots of things could be externalized and generalized. This would also allow more devices to be shared across networks more easily, since they're *on* the network in any case. With the Internet, nobody cares about the physical location of the machines they access; likewise, with this system peripherals aren't associated as strongly with one specific computer.

This sort of thing might also have applications for cluster computing, allowing more sorts of things to be done with clusters since you have higher inter-node bandwidth.

Re:That's exactly the quote I remembered

[TheToon]

I haven't said that, but a columnist in Byte Magazine in the mid-80s had a rant about this.

He programmed on a Mac, and the compilation took typically 5 to 10 minutes. Enough to get a cup of coffee, check the newspaper and have a quick chat with a cow-orker. Then he got a new Mac, and it compiled the program in a minute or so. No time for coffee, no time for news, no time for smalltalk.

So the new, faster computer was too fast... he had to wait at his desk more with the new computer.

10 Gb/s thats it.

[jellomizer]

Remember this is 10 Gigabits per second that is only 1.25 Gigabytes per second (Assuming 100% speed which I never seen happen). Although Right now that is faster then most computers can handle data internally but there are its uses.

1. System to System Backups. If you have seen the prices of memory and Hard drives space $/GB dropping at a fast rate while magnetic tape remains near constant. Soon the price/GB of memory and Hard drives will be lower then tape so it would be cheaper to backup your data on other Systems and Removable hard drive. So if you have a 3 terrabytes of data it can still take up to 40 minutes.
2. Imagine a Beowolf Cluster connected at 10Gbs still right now the main slow point with a Beowolf cluster is the network bandwidth. at 10Gbs you are getting closer to the speeds of a supercomputer bus.
3. Uncompress Video and Sound. No more lossy compression needed with a bunch of people fighting over compression standards. And we always get high quality Audio and Video realtime off the network
4. 3D Now with 3D displays starting to become available there now will be more data needed to send 3D information. Over the network.

And thats just the tip of the iceberg. Back when the 300bps modem came out they figured the speed was as fast as anyone needed because it was near impossible for anyone to type more then 30 characters per second. Then the 1200 and 2400 bps modem cam out and they though those were as fast as anyone needed because almost no one can read at that rate. Then the 9200 and 14.4k because it takes almost no time to go to the next page 80x25 of colored text. then the 33.6k and 56k modems (Still the fastest modems for 1 normal telephone line) you can now download a 300x200x256 colors picture in no time. As bandwidth increases we find new ways to max it out and also with increased bandwidth we come with new methods of using the computer because it can now do it.