Slashdot Mirror
An Application For 10-Gigabit Networking
Chip Smith sent us a short excerpt from a news article on Supercomputing Online:
"Just yesterday Lawrence Berkeley National Laboratory and several key partners put together a demonstration system running a real-world
scientific application to produce data on one cluster, and then send
the resulting data across a 10 Gigabit Ethernet connection to another
cluster, where it is then rendered for visualization." Here's the link to follow if you'd like to read more on this experiment.
Their first project is to create a virtual dictionary and spell checker so that the Slashdot editors make sure that their posts to the front page are spelled properly. As an added bonus, it'll even check grammar! Unfortunately, the scientists aren't sure if there's enough bandwidth available yet to correct all the mistakes.
You're only as smart as your brain.
This experiment shortly followed by a another showing how fast they could move MP3 collections
Jesus saves souls and redeems them for valuable cash prizes
Can you imagine the pings on game servers for Quake3 or counterstrike? 0 to 1 tops!
BEOWULF! Say it with me now...
Who would use 10 Gbit ethernet apart from routers and labs? I know about Moore's law but I also know that 10 Gbit throughput on my hard disk is not coming soon.
Trollem mirabilem hanc subnotationis exigiutas non caperet
Think of the porn I can download now! er, [notices significant other in room] think of the amount of data I could research and the time I'd save....
Heh. I was being silly. Seems like every time there's a story like this, some users here immediately go into "I dont care how painful it is, I'm going to avoid ever needing it" mode. :)
My company has a real use for this, unfortunately NDA prevents me from talking about it. Let's just say that if you ever want to do anything in real time, there's no such thing as 'too much bandwidth'.
"Derp de derp."
would be running a phat VNC cvonnection in your home... wireless 10Gb would be even better, wander around with what amounts to a dummy terminal VNCed to some massive tower. I'd like a cluster of Mac towers serving to a TiBook with lots of ram and video ram and most of the other stuff like the HD stripped away, to keep the price down-ish.
"I would say that 99 per cent of what my father has written about his own life is false." - L. Ron Hubbard Jr.
Real time Special effects processing? Atmosphere / Meterological visualization? Serious 3D game LAN parties?
A fool throws a stone into a well and a thousand sages can not remove it.
I'll go the other way. This 10-gigabit network is going to painfully slim when I have my biological nervous system replaced with an artificial one. Sure would hate to have a saturated bus when I'm walking down the street chewing gum at the same time.
If tits were wings it'd be flying around.
If you're having that problem, I suggest you remove 'NetBEUI' from your nervous system and go for a lighter protocol. ;
"Derp de derp."
It is logically the next step in the evolution of networking, but it currently outstrips even the memory bandwidth of most systems. Even gigabit today is rarely necessary. In both cases, even if you don't use anywhere near your full bandwidth, you still win if you are using more than the 10 times slower generation provides. There are still, however, some compelling applications that immediately present themselves.
As an uplink between network switches, more bandwidth is always needed. This is what was done in the experiment. If you can't imagine a network that busy, consider switches with 256 100MB ethernet ports, or only a few gigabit ports. It is always better, if it all possible, to be able to guarantee that the switch uplink will never become a bottleneck. If they need to do this with many gigabit ports today, be it ethernet or fibre channel, it is done by using multiple gigabit uplinks.
In any cluster where there is substantial communcation between the nodes, more bandwidth and less latency is always useful. Sufficiently low latency and high bandwidth makes many types of computation feasible that would not otherwise be feasible. Shared memory within a cluster can always benefit from decreased latency. A cluster is always better (meaning VASTLY cheaper and VASTLY more reliable) than a many-many CPU system. Anything that makes the cluster nodes seem closer together will make it easier to move existing software onto cluster-based systems.
It is 10 gigabit, not 10.6 gigabit. 10.6Gb/s was the speed they got over a PAIR of 10 gigabit ethernet connections.
What I really want to know is how many LOC's (Library of Congress) this is per second....
10 gig Ethernet is a big problem for all the ATM folks out there. IP6/10Gig Ethernet is a big big problem for ATM. Now carriers have the option of going straight Ethernet throughout the backbone. You say "what about QOS?" well.. IP6 has those bases covered.
Rest In Peace ATM
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
10.6 gigabits is the expected evolution in networking technology. Take a look at the engineers' picture; How about an evelution in fashion and not wearing socks with sandals. That would be impressive.
This reminds of reading about Neural Nets in the various texts on Artificial Intelligence. They always quote Shepherd and Koch 's "The Synaptic Organisation of the Brain": "The brain incorporates 10 billion neurons and 60 trillion connections."
When I think about these new network technologies I can't help to think it's our connections that we lack these days. Hopefully with more and more advanced technology we can utilize these connections to create things more intelligent. This appears to be on the right track.
Maybe "Jane" will finally come out of the closet. Well, actually we got to have Instantaneous (sp) communication before that happens... Doh, well... we're making slight progress.
This sounds nice. The jump from 10 megabits to 100 megabits in the home office was a very substantial improvement. Moving 60 - 100 megabyte scans between machines is much more tolerable.
And all from a 10 times improvement. This 10 gigabit/sec network is 100 times faster.
It will come down to the disk storage. My 100 megabit network already has me waiting on hard disk reads and writes, so I'm not sure if I'd see any more benefits if this were available now.
No. It's a 10.6 gigabit connection over 2 pairs of 10 gigabit Ethernet interfaces. It doesn't matter how it's made up, the connection speed is the overall speed of the connection.
And, if that's boring, think about the military applications. In order to try and cut costs and save on code duplication, the labs are building systems in which part of the application (the secret part) runs on secure systems, whereas non-secret parts run on machines using commercial code. Having a single physical pipe between the areas rather than, as at present, multiple pipes could make the security setup a lot easier, and make the design of the machine considerably cheaper. We will all sleep a lot more securely knowing that LL is able to design lots of new exciting kinds of nuclear weapon at minimal cost.
By the way, if they use any GPLed software, does that mean they have to release the entire source code for the application? Just a question.
Panurge has posted for the last time. Thanks for the positive moderations.
Must be the middle of the night, because I'm struck by the beautiful improbability of the phrase "real world scientific [computing] application". Looking at the article, I note that the phrase is stolen from it, and that the article also mentions that the app is "Cactus". Looking at the pages for that project, I note that Cactus is a computational framework, not an actual application. Further, there seems to be no indication which app was actually run.
However, most of the Cactus apps seem to be in astrophysics. Sigh. Maybe it's "real-world" astrophysics. Or maybe it's just bedtime.
Peer to Car networking?!
Brilliant! Better pattent it now. = )
Let's do the math:
1280x1024 uses 1.25MPixels (that's real Mega as in 2^20) at 32 bpp gives 48MBit/frame. Multiply that with 60 frames/second yields 2880 MBit/second, or just a tad short of 3 GBit/second. That sounds well within the limits, but consider this: Assuming that in a real network this 10 Gbps network will be relatively as efficient as 10 or 100 Mbps networks (some assumption, I know) are right now, you'll get between 30% and 50% of theoretical throughput as the sustained data rate, leaving you with a very narrow margin of bandwidth. And this assumes that you're the ONLY user on the network, which pretty much zeroes out the use for 1280x1024x32bpp, since this is bound to be needed for networked games, which tend to be less than funny when you're the only player.
But ignoring my assumptions and assuming 100% throughput sustained, you'll get 3 players max - still far less than you're likely to really want.
Black holes are where God divided by zero
The throughput from a single hard disk is not that important: in these environments, RAID arrays (typically fiber channel) optimized for read performance allow overall disk performance at e.g. 1.6Gbps. If you have multiple such servers on your network, with many workstations trying to get at the data, 1Gbps Ethernet starts looking a little slow, especially for the backbones.
The main one is for large networks. If you have a large network, gig or dual gig may just not be enough for connections to your major layer-3 switches. Well till now, gig was all you could do on the eithernet spec, you had to go with something like packet-over-sonet to get more bandwidth. 10GigE is nice because you can keep the whole network eithernet, but get more bandiwdth.
At the university where I work I'm sure we'll start using this sonner rather than later. Right now all our distribution routers have dual gig connections to the two backbone routers. Fine, escept that each is feeding 20 to 50 buildings at 100mb each and the redundant set we are going to add will probably be gig. Those gig links to the backbone will fill up fast if each building has a gig to play with. Hence, 10gig eithernet is great since it works with our existing switches and setup, only faster.
Really, desktop or even server use is not the main target of this at least not for a few more years. The main target is removing bottlenecks from the network that supplies those servers.
You're thinking point-to-point, but that's not what networks are for. Imagine the backbone at a hospital with CAT scanners, MRIs, xrays all generating digital images, and doctors around the hospital accessing a database of those images. 10Gbps isn't enough for applications like this. My local dentist's office uses digital xrays, and they complain about the 1Gbps on their little LAN - and they probably don't have more than about 15 workstations.
And as someone else mentioned, rendering and editing of digital video uses up even more bandwidth. You don't have to be Pixar to need to do stuff like this - many companies in the media business can use this.
okay.. I havent' read the 10Gbit standard, so I'm sort of talking out my ass.. but...
if it follows the way 10 & 100 worked..
This is another one of those "Why you can't transmit at 10Gbs from a single computer over 10Gbps ethernet" posts.
First, realize that the 'speed' of ethernet reflects the maximum use of the broadcast medium at maximum capacity. A medium designed for multiple conversations going on at once.
Example: There is a mandatory 9.6 microsecond gap between attemps to transmit frames on the ethernet (this is a minimum limit). In 10Mbps, this is 96 bits worth of time. In 100, it's 960.. if they do the same in 1000, tht's 9600 bits.
So think of that as a per-frame penalty.
If the frame size stays the same.. the mandatory inter-frame gap is about 6 times the size of the data payload now. (okay, so maybe it's different now..). If this IS the case... what's the point of of 10Gbps? More hosts, no switches. That's right.
Bandwidth is the cure for switching.
Even with a big cache (and the 8GB you mentioned isn't much, for this sort of application), if the communications go through such a cache, then the caching box would need 20Gbps total bandwidth to support reading and writing the 10Gbps simultaneously. Even if a bundle of FCAL connections (e.g. 5 x 2Gbps to each host) is capable of that bandwidth in theory, you're going to need a pretty good box in the middle. I've seen SANs that can sustain 5Gbps, but not 10Gbps or 20Gbps. I don't doubt that some might exist, but most real SANs can't do this.
If you're simply talking about bundling FCAL connections to reach the desired bandwidth between two machines or clusters, well duh. But Ethernet has a bunch of advantages over FCAL - in fact, the SAN industry is licking its lips waiting for this tech.
no single peice of commodity hardware would fill that with usefull data.
On the one hand you're talking about what you call "real" SANs that can sustain 20Gbps bandwidth, and on the other hand you're worrying about whether "commodity" hardware can saturate it? Give me a break.
The bottom line is that communicating through a SAN doesn't really make sense today if your goal is 10Gbps point-to-point streaming communication.
I know somebodies going to chime in about how the data they sent isn't as important as how fast they sent it. But just think about these guys being some of the smartest in the world and they can't come up with a decent test other than what amounts to be a poorly designed client/server setup. It would be like having a server send the whole accounting database down to the client just so the CFO can pull up the details of a single transaction. That's not a practical application for that bandwidth.
The whole point of a cluster is to have each node do part of the work, then communicate it's results elsewhere for more processing. Unlike image processing where the data can be divided and processed at a pixel level granularity, many apps will only divide up so far. From there, you can either pipeline the processing over a really fat pipe, or you can accept that you have hit the limits of scalability and buy massively expensive n-way nodes (where n>4).
I can't think of a practical situation, but if somebody could explain why you would need to send a gigabyte of data in one second vs. 8 second I'd be more impressed.
If all you're moving is a GB, it's not that important, if you need to move 10TB, the difference adds up fast. Consider, why would you need to go a mile/minute rather than in 10 min? If you live 40 miles from work it's the difference between spending 1:20 or 12 hours a day commuting to/from work.
In supercomputing terms, whenever it would take longer to send the data elsewhere for further crunching than it would to crunch it locally, you have hit a limit of scalability.
You're right, of course - no disagreement what so ever. I just answered the question (as I understood it, anyway).
[snip] but full-res PAL video is 720x576 is in size, so adjust the math accordingly
I don't know about you, but I'd like better resolution on my monitor than 720x576. Even 720x576 requires a LOT of bandwidth (too lazy to do the math right now - the sun is shining :-) though, so I guess streaming/compression will remain necessary for a LONG time yet...
Black holes are where God divided by zero