Ethernet The Occasional Outsider

Long Live! by Anonymous Coward · 2006-05-25 07:20 · Score: 5, Funny

Long Live the Token Ring!

One Ring to rule them all

Re:Long Live! by MImeKillEr · 2006-05-25 07:23 · Score: 1

Mod parent up as funny!

(I was actually going to post something similar, but this one beat me to the punch).

Do they even make Token Ring anymore? I know the MAUs were hella-expensive.

--
Cruising the internet on my TI-99/4A @ a whopping 300 baud!
Re:Long Live! by Itninja · 2006-05-25 07:23 · Score: 0, Offtopic

I lost my token ring. Now my Interweb is broken.

--
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
Re:Long Live! by silas_moeckel · 2006-05-25 07:32 · Score: 1

Posted from a token ring connected computer. Yes they still make it, you can still tow a car with a station cable (untill the pigtail to get it into the laptop). It's still slow as well.

--
No sir I dont like it.
Re:Long Live! by EnderWiggnz · 2006-05-25 07:35 · Score: 2, Interesting

yes, people (mostly the government) do have token ring setups.

the funnest, is that i've done work for naval ships that required 10base2. You know... CheaperNet!

--
... hi bingo ...
Re:Long Live! by Jon+Luckey · 2006-05-25 07:48 · Score: 1

Don't you mean that you lost the token, so the ring does not work anymore?

(From an old Dilbert comic IIRC)

--
-- 3 events that reshaped the world in the 20th century: WW1, WW2, and WWW
Re:Long Live! by MrSquirrel · 2006-05-25 07:57 · Score: 3, Funny

I saw students bring in computers with token-ring cards when I worked at a University Helpdesk. They would come in and say "My computers broken, I plugged 'the internet' in but it won't connect" (we would troubleshoot over the phone and they would want us to come up to their room, after much repeating our policies they would cave and bring it down because they wanted to download their pr0n). I was baffled when it would turn out to be a token-ring card... I was like "Where the HELL did they get this?". I'm convinced it's part of the worldwide conspiracy to drive me insane.

--
A computer once beat me at chess, but it was no match for me at kick boxing.
Re:Long Live! by myth24601 · 2006-05-25 08:35 · Score: 2

ARCNET is the tank of networks protocols. I was once working on an arcnet system and I tripped over the cable and yanked it out of the wall. Would you believe the token jumped out of the cable and ran accross the floor and jumped into the wall.

Nothing stops ARCNET!

--
No matter where you go, there you are.
Re:Long Live! by default+luser · 2006-05-25 09:34 · Score: 1

Well, my college was connected with token ring up until 2001, when they did a complete network overhaul. Maybe you got one of our transfer students :D

Apparently, my college got a great deal on token ring from IBM in the early 90s, and at the time it was plenty fast. But by the mid 90s, it was showing its age, with no upgrade path. Back when my college still had no clue how to manage their network (read: 1997, pre-Napster), it consisted of a "turbo" (16Mbit) token ring backbone with various 4Mbit and 16Mbit rings. The bridge to the internet (single T1) was 10-base T.

Since token ring cards were really fucking expensive, the college "loaned" out token ring cards to all students. Students could either shell out $300 for a token ring card and do it themselves, or drop their computers off for a few days and get a "loaner" installed. I say "loaner" because when I graduated in 2001, and the entire network was upgraded, the school sure as hell didn't want these old token ring cards back.

Then the students discovered computers, and then discovered Napster and IM. Within a year, the college upgraded to dual T1s, then a fractional T3, and finally got off their ass and designed a better network. Gigabit optical backbones, 100-base T in the rooms, upgradable to gigabit ethernet. Too bad I wasn't around to enjoy it...

--
Man is the animal that laughs.
And occasionally whores for Karma.
Re:Long Live! by Suzuran · 2006-05-25 14:09 · Score: 2

I had a teacher once who ran ARCNET over a section of barbed-wire fence, just to prove it would work. It worked for about a week until he got bored of it and took it apart, even working through a rainstorm (that made it drop some packets though)
Re:Long Live! by tap · 2006-05-25 18:28 · Score: 1

I had an arcnet setup one year in my dorm with the people across the hall. The campus ethernet would only allow one computer to be active per room (switch with 1 MAC address per port limit). We had the linux machines setup as routers, so we could get to the internet via our ethernet, or via the arcnet across the hall to another computer. We could get multiple computers that way, as long the ethernet switch only saw one MAC address, it didn't matter how many IPs you had behind it. We could have used a second ethernet network, but the arcnet cards were like $4 and the switch was real cheap, ethernet was somewhat more at the time, and you get this cool arc0 device in Linux.

My first real kernel programming was trying to network boot a 20 Mhz 386 with no hard drive and just an arcnet card from another linux machine. Put the kernel on floppy, and used a NFS root filesystem. The problem was that RARP in Linux, used to get the IP address, didn't support Arcnet. It only supported ethernet and some ham radio network. So I had to add arcnet support the linux RARP code. Actually got it working and compiled a kernel on a 20 MHz 386 with 9 MB of ram.
Re:Long Live! by MarkGriz · 2006-05-26 02:22 · Score: 1

(From an old Dilbert comic IIRC)

Looking at it on my wall right now....

Dilbert (to PHB): Here's your problem. The connection to the network is broken. Uh-oh. It's a "token ring" LAN. That means the token fell out and it's in this room someplace.
*PHB searches for token under desk*
Dilbert (to Wally): I'll wait a week then tell him the token must be in the "ethernet"
Wally: You are the wind beneath my wings.

--
Beauty is in the eye of the beerholder.
Re:Long Live! by oldwarrior · 2006-05-26 03:34 · Score: 0

Three Rings for the Elven Oracle and Sun under the California sky,
Twelve for the Dwarf Lords in their Redmond halls of stone,
Nine for Compac, DEC, SG, and DG, doomed to die,
One for the Dark Lord, on his Dark Throne,
In the land of Armonk, where the shadows lie...

--
If it were done when 'tis done, then t'were well it were done quickly... MacBeth

Overlords by Zondar · 2006-05-25 07:22 · Score: 0, Offtopic

I, for one, welcome our new non-ethernet overlords.

My idea: a vat of salt water & CAT5 by Anonymous Coward · 2006-05-25 07:23 · Score: 5, Funny

In our Data Center, we have a great big vat of steaming salt water and we drop one end of the cat5 cables from each server into the vat....those packets that can't figure out where they're going just drop to the bottom and die ...we have to drain this packet-goo out once a month. (but we do recycle it...we press it into CDs and sell them on Ebay)

(Seriously, haven't people heard cut-through switches which just look at the first part of the header and switch based on that... store-and-forward switches are so "1990s")

TDz.

Re:My idea: a vat of salt water & CAT5 by Amouth · 2006-05-25 07:33 · Score: 1

that was my thought exactly (not the salt vat.. althought i like it)

we have a small office ~20 computers and 3 servers.. and i refuse to buy switchs that can't do cut through.. store and forward is slow..and very memory entisive for switchs on high speed networks..

--
'...if only "Jumping to a Conclusion" was an event in the Olympics.'
Re:My idea: a vat of salt water & CAT5 by barawn · 2006-05-25 08:13 · Score: 1

(Seriously, haven't people heard cut-through switches which just look at the first part of the header and switch based on that... store-and-forward switches are so "1990s")

Even still - low 100 ms for store-and-forward ethernet switches? That seems really, really high. I would've said more like single milliseconds, which is still high, but it isn't 100 ms.

I know from experience that I've used store-and-forward ethernet switches with much, much better latency than 100 ms.
Re:My idea: a vat of salt water & CAT5 by rekoil · 2006-05-25 08:41 · Score: 1

If you RTFA you'll see that it was a typo - it's microseconds, not milliseconds. You can ping from New York to Seattle in less than a 100 milliseconds if you're on a decent pipe.
Re:My idea: a vat of salt water & CAT5 by barawn · 2006-05-25 08:43 · Score: 1

It says 100 milliseconds in the article.

I don't doubt that it's wrong (like I said, I know from experience that it's of order 1 ms, not 100 ms) but the article is the one that's wrong, not the story summary.
Re:My idea: a vat of salt water & CAT5 by rekoil · 2006-05-25 08:49 · Score: 1

We're both right - there's a reference to microseconds in the first page, but the 100-millisecond figure shows up later on. Mea culpa.

30 GB? Take that NSA and your outdated 622MB! by Marxist+Hacker+42 · 2006-05-25 07:23 · Score: 2, Interesting

The NSA's network sniffer, recently discovered at an AT&T broadband center, can only sniff up to 622MB. Sounds to me like if you use an InfiniBand switch, that would effectively make the output of the NSA's network sniffers complete gibberish.

--
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.

100ms ethernet latency? by victim · 2006-05-25 07:23 · Score: 5, Informative

I don't think I need to read anymore, well, I did verify that the number really appears in the article.
This author does not understand the subject material.

(I suppose you could deliberatly overload a switch enough to get this number, maybe, but that would be silly, and your switch would need 1.25Mbytes of packet cache.)

Re:100ms ethernet latency? by merreborn · 2006-05-25 07:27 · Score: 5, Informative

Looks like the author fucked up the definition of millisecond too:

"By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second"

http://www.google.com/search?hl=en&q=define%3Amill isecond&btnG=Google+Search
"One thousandth of a second"

Seriously. How the fuck does this idiot get published?
Re:100ms ethernet latency? by Solra+Bizna · 2006-05-25 07:31 · Score: 1

Yeah. Even my needlessly complex network setup (we have three routers all routing to the same LAN, long story) I get latencies about 1 millisecond going from a wireless client to the modem (client -> wireless AP -> wired router 1 -> wired router 2 -> modem)...

-:sigma.SB

--
WARN
THERE IS ANOTHER SYSTEM
Re:100ms ethernet latency? by dextromulous · 2006-05-25 07:48 · Score: 1

A better question might be: how bad is the editor if this wasn't noticed?

--
There are two types of people in the world: those who divide people into two types and those who don't.
Re:100ms ethernet latency? by Phreakiture · 2006-05-25 07:52 · Score: 1, Informative

This author does not understand the subject material.

I disagree. The author has simply misplaced his metric units. He used the word "milliseconds", where he should have used the word "microseconds". You can see an example of this where he refers to milliseconds as one millionth of a second, rather than the one thousandth that they actually are.

--
www.wavefront-av.com
Re:100ms ethernet latency? by drinkypoo · 2006-05-25 08:07 · Score: 1

So what you're saying is that the author may understand the source material, but he's an idiot too stupid to proofread, or even worse, too stupid to catch such a mistake if he does proofread? I don't think that's much of an improvement.

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Re:100ms ethernet latency? by Phreakiture · 2006-05-25 08:14 · Score: 1

So what you're saying is that the author may understand the source material, but he's an idiot too stupid to proofread

Yeah, pretty much.

--
www.wavefront-av.com
Re:100ms ethernet latency? by Sirfrummel · 2006-05-25 08:19 · Score: 1

Alright... I looked it up,

millisecond = 1/1,000
microsecond = 1/1,000,000
nanosecond = 1/1,000,000,000
Re:100ms ethernet latency? by jelle · 2006-05-25 08:23 · Score: 1

The problems is that it's uncertain that every mention of 'millisecond' was meant to be 'microsecond'.

For example, I can agree with "When you get into application-layer clustering, milliseconds of latency can have an impact on performance,'"

But s/milliseconds/microseconds in that, and you're talking about a significantly reduced number of applications that need that kind of response times.

Microseconds are short for most processes. For example, Linux task-switches 100, 250 or 1000 times, depending on the processor. That means the timeslices are 1us, 4us, or 10us. Applications already sit still, not doing anything for multiples of the timeslices, so latency in the same order of magnitude often won't be noticed much if at all.

--
--- Hindsight is 20/20, but walking backwards is not the answer.
Re:100ms ethernet latency? by Anonymous Coward · 2006-05-25 08:40 · Score: 0

milli = 10^-3 I've always wondered why english have words with (i suppose) Saxon's roots (thousand) and uses latin prefixes (milli) for adjectives and other speech parts.... Like "moon" --> "lunar" It's the only thing I like in my otherwise hard-as-hell-to-speak mother language (italian, for the records). It helps avoiding this kind of disasters...
Re:100ms ethernet latency? by Anonymous Coward · 2006-05-25 08:53 · Score: 0

What the hell is going on in schools these days? Those are standard SI prefixes. (If you have to look them up, that's what they're called, and shame on your teachers.)
Re:100ms ethernet latency? by Anonymous Coward · 2006-05-25 08:54 · Score: 0

Microseconds are short for most processes. For example, Linux task-switches 100, 250 or 1000 times, depending on the processor. That means the timeslices are 1us, 4us, or 10us. Applications already sit still, not doing anything for multiples of the timeslices, so latency in the same order of magnitude often won't be noticed much if at all.
Linux does task-switch either ~100, 250, or 1000 times a sec which means timeslices are really 10ms, 4 ms, or 1 ms.
Re:100ms ethernet latency? by kurtvs · 2006-05-25 08:56 · Score: 1

I wish I had longer to write this article, but I've got to leave in 10 minutes, so here's what I can do in the time alloted. Ping time is a very coarse measurement of link latency and does NOT give you an absolute number worth anything because it doesn't distinguish between server latency and link latency and for pings, server latency is not consistent from one machine/device to another. Ping time is mostly useful when you have a previous measurement to compare to from the same machine as a relative indicator of link speed and even then, it is still a coarse measurement. For many/most devices, responding to a ping is a very low priority task. A better measure is to look at the latency for a file service request. Just about any PC or Mac made in the last 5 years will turn around a file read request in much less than a millisecond (1/1000 second), so the latency you measure is the latency of the link. Downside, you need to do this with a protocol analyzer. A router will often have a ping turnaround time of 20 milliseconds or greater and this number is variable depending on load, so ping time can be a (very coarse) measurement of router load and then only when you have a reference when the router was not loaded. The latency of Ethernet itself is 9.6 microseconds (one millionth second) for 10 MB, 0.96 microseconds for 100 MB, although that's a theoretical limit -- it's the time that an Ethernet sender has to observe no signal before sending, but this only applies to half duplex. There's no wait time in full duplex. The actual limit is how fast can you get it through the system, across the bus and out the NIC. In most cases, this is less than a millisecond. Token ring has many drawbacks in typical modern network situations. It was designed with a strong and predictable client-server relationship in mind where you would attach to and use a limited number of servers for long periods of time. Its actual link latency is comparable to Ethernet - so minute as to be insignificant. Gotta go, Kurt VanderSluis PS, I just dashed this off, so if there's a typo, sorry, but I didn't have time to do the fine tooth comb thing
Re:100ms ethernet latency? by jthill · 2006-05-25 09:07 · Score: 1

That s/b "1ms, 4ms or 10ms". But your ordinary boxes aren't the kind of systems the article is talking about.
I see NetworkWorld fixed the article.

--
As always, all IMO. Insert "I think" everywhere grammatically possible.
Re:100ms ethernet latency? by ajs · 2006-05-25 09:08 · Score: 1

Problems also include the use of the term "store and forward Ethernet" (WTF does that mean?!) and the fact that Ethernet channel bonding has been around for about 10 years.
Re:100ms ethernet latency? by Anonymous Coward · 2006-05-25 09:09 · Score: 0

In the words of Emily Litella...
Oh, that's quite different...
Never mind!
Re:100ms ethernet latency? by KenSeymour · 2006-05-25 09:29 · Score: 1

Indeed. When I clicked on the article, they had fixed it to read as the author intended.

--
"We can't solve problems by using the same kind of thinking we used when we created them." -- Albert Einstein
Re:100ms ethernet latency? by ctr2sprt · 2006-05-25 09:39 · Score: 1

Store and forward is when the switch reads in the entire packet before making a routing decision. Most protocols, including Ethernet and TCP/IP, send the target address very early in the frame precisely so that store and forward isn't necessary. Instead they use a strategy called cut-through switching, where they read just enough of the frame to determine where to send it and then send the remainder to the destination port as it arrives on the source port. Most home or small office switches use store and forward switching.
Or maybe you were being pedantic and quibbling about calling it store and forward ethernet instead of store and forward switching.
Re:100ms ethernet latency? by jelle · 2006-05-25 10:53 · Score: 1

Oh no! Now I'm doing it too ;-) It's contagious!

--
--- Hindsight is 20/20, but walking backwards is not the answer.
Re:100ms ethernet latency? by Anonymous Coward · 2006-05-25 12:28 · Score: 0

I've personally written a gigabit driver for Linux that achieves less than 20usec latency. Even if the guy just got his units wrong and meant to write 100us, it's still time to buy some up-to-date networking kit.

The speed of data transmission (Ethernet included) is getting scary. At 10G, an entire 64-byte minimum-length packet can fit on the cable; the last bit in the packet will have been transmitted by the sender before the first bit reaches the other end.
Re:100ms ethernet latency? by MarvinTheHobartian · 2006-05-25 12:51 · Score: 1

Dude - that's not what TFA says at all. They have it correct (micro = millionth, nano = billionth).
Re:100ms ethernet latency? by merreborn · 2006-05-25 13:06 · Score: 1

Believe it or not, I emailed them reguarding both the millisecond definition error, and the claim that pings, and they corrected it.

Subject: FW: Feedback http://www.networkworld.com/news/2006/052506-data- center-ethernet.html
Date: Thu, 25 May 2006 17:03:03 -0400
From: "Phil Hochmuth"
To: merreborn@*****.com

Thanks for your correction. We're making that change now. (Journalist math for you).

As for 1 millisecond of delay, are you talking about on the wire (node-to-node), or inside-the-box
(port-to-port) latency? (I've read reports that SONET/SDH has measured latency of around 1
millsec).

Phil.
Re:100ms ethernet latency? by QuesarVII · 2006-05-25 17:08 · Score: 1
A pair of broadcom or intel gige cards running through a regular "netgear|linksys|d-link|smc|hp" switch can get latencies of about 20-30 microseconds for small packets. It creeps up as packets get larger.

The broadcom needs a slight ethtool config change to go that low- it otherwise gets about 45 us with a slightly lower cpu load.

I work in HPC linux clusters. HPC MPI app performance is largely determined by interconnect bandwidth and latency. We use our Link-Checker tool regularly to test latencies and bandwidths of high performance interconnects. Here are some recent results(latency is one way, bandwidth unidirectional):
- Mellanox Infiniband cards with our own IB switch: latency of ~3.7us, bandwidth ~910MB/s
- Pathscale HTX IB cards w/ own IB switch: latency ~1.3 us, bandwidth ~900MB/s
- Myrinet 2000 w/ new mx software: latency ~2.8us, bandwidth ~230MB/s
- GigE: latency ~30us, bandwidth ~110MB/s
Re:100ms ethernet latency? by ajs · 2006-05-26 03:01 · Score: 1

I was not just being pedantic. I come from the old-school world where store-and-forward means something like UUCP, so the phrase "store-and-forward ethernet" sounded like an oxymoron to me.

If store-and-forward is indeed used in two very different contexts, it might be helpful for someone to update the Wikipedia article on store and forward with current, and accurate meanings.

Low-cost options? by sammy+baby · 2006-05-25 07:24 · Score: 1

Ultra-low latency networking is a minor interest of mine, but one I've never had the chance to really pursue. Can anyone familiar with the landscape recommend some low-cost options for experimenting with this stuff? Or maybe just let me down gently. "No, Sammy, there are no low-cost options. And there's no Santa Claus."

Re:Low-cost options? by dlapine · 2006-05-25 07:52 · Score: 2, Informative

Define low cost? Myrinet with less than 10 microsecond latency is normally considered to be the least expensive option. You can check their price lists, but an 8 port solution (with 8 HBA's) will set you back over $8k, not including the fiber.
For some people, that's cheap. If not, sorry.

--
The Internet has no garbage collection
Re:Low-cost options? by XyborX · 2006-05-25 09:42 · Score: 1

I recently discovered Xdmx, something that seems capable of splitting my desktop across multiple machines. Along with it, I was also thinking in lines of some kind of cluster system, like OpenMosix. My goal at the moment is to be able to run Quake 3 on three computers (two of which are laptops = limited expansion possibilites), giving me wider peripheral vision, or at least to be able to move windows around between the display. Unfortunately, I'm quite certain that at least OpenMosix would require more speed than 100Mbit for Quake 3, so I started pondering the same question: What cheap and fast network solutions exist?

So far, the best idea I can come up with is a mesh of those USB-USB link cables. If USB 2.0 is in the ~400Mbit range, it should be faster than 100Mbit, right? But then again, speed and latency aren't the same.. Comments are welcome :)

--
// Just my few cents
Re:Low-cost options? by Anonymous Coward · 2006-05-25 10:48 · Score: 0

Try Dolphin SCI link .

It has latency in the microseconds, not milliseconds.
Re:Low-cost options? by multimediavt · 2006-05-25 11:12 · Score: 1

Myrinet is one option, InfiniBand costs about the same and has similar performance characteristics. Myrinet drivers are more mature though and will give you fewer headaches during config. Moving up the scale, there's InfiniPath, Dolphin, and Quadrics. All are quite fast, sub-10 ms latency in most cases, but are very expensive. This market is certainly not for the "hobbyist" unless you're pulling down over six figures a year, or are a recipient of a large monetary bequest.
Re:Low-cost options? by sammy+baby · 2006-05-25 16:45 · Score: 1

That's pretty much what I was afraid of. Regrettably, I don't make the scratch necessary to dabble.

Oh well. Back to knocking around $300 servers. ;)
Re:Low-cost options? by vyvepe · 2006-05-25 20:36 · Score: 1

Topspin InfiniBand 14 port Switch costs about $7000. It is comparable price. Infinibad latencies are a bit better.
Re:Low-cost options? by Slashcrap · 2006-05-25 22:07 · Score: 1

So far, the best idea I can come up with is a mesh of those USB-USB link cables. If USB 2.0 is in the ~400Mbit range, it should be faster than 100Mbit, right? But then again, speed and latency aren't the same.. Comments are welcome :)

Here's one - if you were seriously considering USB as the transport you are out of your depth and should find another project to pursue.

Sometimes you have to be cruel to be kind. I usually prefer cruelty for its own sake, but another poster beat me to it on that one.
Re:Low-cost options? by matti2 · 2006-05-25 22:09 · Score: 1

Best one using standard GiGE ethernet is GAMMA (http://www.disi.unige.it/project/gamma/). Bit tricky to install but fantastic performance. We get something like 13muS latency through switch.

Not an Auspicious Start by Anonymous Coward · 2006-05-25 07:25 · Score: 5, Informative

From the article, three paragraphs in:
"(By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second)"

That would be one-thousandth, not millionth (aka micro second). Not a good start...

Re:Not an Auspicious Start by Sycraft-fu · 2006-05-25 07:53 · Score: 1

Well that and ethernet gear is measured in miliseconds? That doesn't seem useful. If I run a traceroute, the time is listed as "1ms" for all the internal hops. There are 5 internal hops, all ethernet. In my experience, all modern ethernet gear adds less than a millisecond of latency. Traceroute programs only report milliseconds because there's a useful measure for Internet traffic and anything under 1ms can be safely called "really fast" for normal work.

Seems to me you'd need to measure ethernet gear in microseconds to get a useful number.

When you get to many hops by with_him · 2006-05-25 07:25 · Score: 5, Funny

I just blame it on the ether-bunny.

Re:When you get to many hops by Teun · 2006-05-25 08:07 · Score: 1

Stop using those Duracells...

--
"The likes of Facebook and WhatsApp are free to those whose privacy is of zero value."
Re:When you get to many hops by c0d3h4x0r · 2006-05-25 09:01 · Score: 1

Is that the gay easter bunny?

--
Moderator hint: a comment is neither "Flamebait" nor "Troll" if it is true.
Re:When you get to many hops by doublem · 2006-05-25 09:02 · Score: 1

Thank you for that incredibly bad pun.

I needed a laugh after the work day I've had.

--
"Live Free or Die." Don't like it? Then keep out of the USA

Software design by nuggz · 2006-05-25 07:26 · Score: 2, Interesting

The origional post makes some comments that
sharing memory ... the smallest hiccups can fail a process or botch data results.
Sounds like bad design, or a known design trade off.
Quite reasonable, when on a slow link, until I know better assume the data I have is correct, if it isn't throw it out and start over. Not wildly different than branch prediction or other approaches to this type of information.

'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,'
Faster is faster, not really a shocking concept.

Re:Software design by Anonymous Coward · 2006-05-25 07:39 · Score: 0

If your relying on timing for your application to work, its the wrong design
Re:Software design by Amouth · 2006-05-25 07:41 · Score: 2, Funny

what it looks like to me is.. ok so they set something up using normal 100/1000 ethernet and then realized something was slow and that if they use gbic 30gb ports things run faster... can someone please sent them a cookie?

--
'...if only "Jumping to a Conclusion" was an event in the Olympics.'

Did you mean "microseconds"? by pla · 2006-05-25 07:26 · Score: 3, Interesting

The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range.

I don't know what sort of switches you use, but my home LAN, with two hops (including one over a wireless bridge) through only slightly-above-lowest-end DLink hardware, I consistantly get under 1ms.

When you get into application-layer clustering, milliseconds of latency can have an impact on performance

Again, I get less than 1ms, singular.

Now, I can appreciate that any latency slows down clustering, but the ranges given just don't make sense. Change that to "microseconds", and it would make more sense. But Ethernet can handle single-digit-ms latencies without breaking a sweat.

Re:Did you mean "microseconds"? by Eideewt · 2006-05-25 07:57 · Score: 1

I wonder if his messed up numbers come from his mistaken belief that a millisecond is three orders of magnitude smaller than it is.
Re:Did you mean "microseconds"? by dlapine · 2006-05-25 08:08 · Score: 2, Informative

Sure, for an 8 port switch, where all the computers have a direct connection. Consider the issues involved for a router with a 128 machines all trying to cross-communicate. Or larger collections of computers that might need to use multiple sets of switches to span the entire system.
On a Force10 switch, with 2 nodes on the same blade:
tg-c844:~ # ping tg-c845
PING tg-c845.ncsa.teragrid.org (141.142.57.161) from 141.142.57.160 : 56(84) bytes of data.
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=1 ttl=64 time=0.148 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=2 ttl=64 time=0.146 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=3 ttl=64 time=0.145 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=4 ttl=64 time=0.144 ms

The same nodes using a myrinet connection:
tg-c844:~ # ping tg-c845-myri0
PING tg-c845-myri0.ncsa.teragrid.org (172.22.57.161) from 172.22.57.160 : 56(84) bytes of data.
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=4 ttl=64 time=0.043 ms

The latency gets below 10 usec with the use of special drivers, this is just using the 2.4 Linux tcp stack. What's even scarier about the Myrinet is that I can have all 900+ machines talking at the same time with no drop in latency- we have that network spec'd for full bisection bandwidth. Try that on 900 nodes on a gige network, let alone a 100baseT.
As was mentioned here earlier, ethernet is nice for networks that change. Once you have a significant number of machines attached, and the number of switches and routers gets past 1, ethernet loses it's equivalence in latency.

--
The Internet has no garbage collection
Re:Did you mean "microseconds"? by cgori · 2006-05-25 09:29 · Score: 1

1) Neat stuff in your cluster.

2) A fair number of ethernet switches exist for ~500 nodes @ 1Gbps that will have predictable latency, like the force10 you are describing. 900 nodes would be tough, admittedly, at the moment. Also, I don't think you meant to say "router" -- you almost certainly are switching if it's all configured right.

3) Myrinet is very specialized and uses cut-through switching. Ethernet is a generalized protocol that can be used on a WAN, and is almost always store-and-forward. Store-and-forward scales better to distance, and under massive load. If your input bandwidth is able to oversubscribe your switch fabric in a cut-through switch, the performance will decline horribly, and the distribution of latencies almost random. Store-and-forward will decline gradually and (usually) have monotonic increasing latency under load.
Re:Did you mean "microseconds"? by Lord+Crc · 2006-05-25 11:34 · Score: 1

I don't know what sort of switches you use, but my home LAN, with two hops (including one over a wireless bridge) through only slightly-above-lowest-end DLink hardware, I consistantly get under 1ms.

Wow, you must use some really old hardware. My packets arrive before I send them:
Pinging 10.0.0.1 with 32 bytes of data: Reply from 10.0.0.1: bytes=32 time=-11ms TTL=63 Reply from 10.0.0.1: bytes=32 time=1ms TTL=63 Reply from 10.0.0.1: bytes=32 time=-11ms TTL=63 Reply from 10.0.0.1: bytes=32 time=2ms TTL=63 Ping statistics for 10.0.0.1: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 1ms, Maximum = -11ms, Average = 1073741819ms
(Yes I have a dual-core ;)
Re:Did you mean "microseconds"? by Anonymous Coward · 2006-05-25 12:48 · Score: 0

That's funny (if it is real), I like most the end where it says min, max and average.
Re:Did you mean "microseconds"? by Lord+Crc · 2006-05-25 13:01 · Score: 1

Oh yeah it's real. QueryPerformanceCounter is not guaranteed to return consistent results between cores. So what happens is it queries the start count on core 1, then gets scheduled on core 2, and measures the stop count there. Then it subtracts and voila, minus X msecs. The workaround is to use SetThreadAffinityMask on threads that uses QueryPerformanceCounter, to force it to run on only one core.

Milliseconds? by rubmytummy · 2006-05-25 07:27 · Score: 2, Funny

On my planet, a millisecond is a full thousandth of a second, not just one millionth.

Oh, well. People tell me I'm just slow.

sharing memory over ethernet? by jm91509 · 2006-05-25 07:28 · Score: 0, Flamebait

That just sounds daft. Given the bottle neck harddrives are for cpu's, it doesn't sound like a great shock that when you gotta wait for your data over ethernet you're going to see problems.

Maybe I should RTFA...

Re:sharing memory over ethernet? by AuMatar · 2006-05-25 07:35 · Score: 1

This is a NUMA (non-uniform memory access) cluster. Basicly a bunch of computers woring together that occasionally need to access the same data. If the last process to need that data happens to be on another computer, it needs ot be transfered. The trick to these clusters is writing software so that transfer need is minimal, and that the same data set stays on the same processor, to the best of your ability.

--
I still have more fans than freaks. WTF is wrong with you people?
Re:sharing memory over ethernet? by TheOriginalRevdoc · 2006-05-25 13:26 · Score: 1

Naturally, the article is riddled with errors, because (1) the author isn't a subject matter expert and (2) good sub-editing is no longer considered essential.

However, there are applications that "share memory" over networks; Oracle RAC springs to mind, where the nodes in the cluster share database blocks as required. However, Oracle recommend gigabit point-to-point connections between nodes, rather than a general-purpose network. The latter tends to make the cluster unusable.
Re:sharing memory over ethernet? by mrchaotica · 2006-05-25 16:37 · Score: 2, Interesting

Maybe I should RTFA...
Either that, or you should take the class that I took this past semester. There's a bunch of links to research papers and lecture slides about distributed shared memory (and other kinds of parallel/shared computing issues), if you care to read them.

--
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz

Lesson: Use appropriate Tech. by Daniel_Staal · 2006-05-25 07:32 · Score: 1

Ethernet's strength is it's flexiblity, not it's speed per se. It can handle changing network environments where hardware or software is added and removed continually, and you never know quite where the bandwith is most needed. You just plug it all in, and ethernet does a decent job of neotiating who gets to use the bandwidth.

But it's never been a really high speed protocol. It's easy to beat, speed-wise, as long as you know what the network use looks like ahead of time.

Which of course is a killer for most general use, but for specialty use that's not so much of a problem.

--
'Sensible' is a curse word.

Re:Lesson: Use appropriate Tech. by Anonymous Coward · 2006-05-25 07:45 · Score: 1, Insightful

I assume you mean "It's never been an efficient protocol." This is somewhat true, but relative to what, exactly? TDM? SONET? ATM?

If you did truly intend to state that it's not high speed, you're mistaken. 10 gigabit Ethernet is common, and modern hardware latencies are not significant. The only way you're going to exceed that would be with WDM (that's cheating) or with an OC-768 (good luck finding one outside of a research lab).
Re:Lesson: Use appropriate Tech. by Daniel_Staal · 2006-05-25 07:50 · Score: 1

You're right; the terminology there is weak. I mean that it usually can't get the full potential out of the underlying physical network that others can. 'High speed' is relative, of course, to whatever you are comparing it to.

Basically, at a given level of tech, you should be able to build a network that is faster than an ethernet network at that level of tech. But it will be more complicated to set up and maintain.

--
'Sensible' is a curse word.

Store & Forward Unnecessary? by ljc86 · 2006-05-25 07:32 · Score: 1

The article's a bit lacking on details, but... Isn't store and forward unnecessary? It's definitely possible to get it down to a much lower latency than is stated in the article if you don't use it.

Re:Store & Forward Unnecessary? by Anonymous+Struct · 2006-05-25 07:55 · Score: 1

I think the article just has a typo. 'Imperceptible' is definitely not how I'd describe 100ms latency on a switched LAN. It's also true that switches do not necessarily have to store and forward, and cut-through switching used to be a lot more popular. I believe it's probably less popular now because store and forward performance is more than adequate, and because it offers a handful of advantages (verify FCS before forwarding, for example).
Re:Store & Forward Unnecessary? by Nefarious+Wheel · 2006-05-25 09:48 · Score: 1

Might be talking about queueing systems like MQ-Series. I know the latter is a bit of a store-and-forward buffer; not terribly fast, but it talks to a lot of hosts & you'll see it a lot in retail supply chains.
Hard to credit any article with so little context, though.
Seriously folks, we engine-hearing types had better learn to write, because it's a fair call that the journalists don't understand engineering.

--
Do not mock my vision of impractical footwear

Didn't RTFA? -Infiniband, FC and Myrinet beat Eth0 by hguorbray · 2006-05-25 07:38 · Score: 4, Interesting

Actually, even with Gigabit ethernet availability HPTC and other network intensive data center operations have moved to Fibre Channel and things like:

Infiniband http://en.wikipedia.org/wiki/Infiniband

and Myrinet http://en.wikipedia.org/wiki/Myrinet

http://h20311.www2.hp.com/HPC/cache/276360-0-0-0-1 21.html
HP HPTC site

-What's the speed of dark?

Re:30 GB? Take that NSA and your outdated 622MB! by Anonymous Coward · 2006-05-25 07:38 · Score: 0

622MB

Nice troll.

Store & Forward ONLY for 10 to 100 to 1,000. by khasim · 2006-05-25 07:41 · Score: 3, Informative

There are only TWO reasons to use Store & Forward.

#1. You're running different speeds on the same switch (why?).

#2. You really want to cut down on broadcast storms (just fix the real problem, okay?)

Other than that, go for the speed! Full duplex!

Re:Bluetooth! by MrSquirrel · 2006-05-25 07:42 · Score: 0

What about a co-axial bus connection? VAMPIRE TAPS!!! WEEEEEE. Come on, someone else has to remember those. It was like playing "Operation" except if you "touched the sides" and screwed the tap too far in, you broke the cable. Fun to the max!

--
A computer once beat me at chess, but it was no match for me at kick boxing.

TCP/IP over ultra SCSI do it for ya? by absinthminded64 · 2006-05-25 07:45 · Score: 1

I've never used it but I know it can be done with Linux, a couple scsi controllers and other nifty things. Don't know the speed or latency of it though

Channel Bonding by Perl-Pusher · 2006-05-25 07:46 · Score: 0, Offtopic

I have a cluster of 45 dual Xeon processing nodes. Latencies average about 210 usec the same as could be expected in any 100Mbs connection, but using channel bonding my bandwidth is double that of a single ethernet connection. I don't have the need for faster, all our processes are wholly independent and don't need to do message passing.

Re:Channel Bonding by booch · 2006-05-25 08:36 · Score: 1

I wouldn't expect channel bonding to significantly improve latency. I'd be surprised if you got more than 10% improvement, unless you are bandwidth-limited.

--
Software sucks. Open Source sucks less.
Re:Channel Bonding by kjs3 · 2006-05-25 08:47 · Score: 4, Funny

So you have an environment with requirements totally unlike the ones described in the article and needing none of the solutions illustrated in the article. Hey...thanks for letting us know. Maybe the other million Slashdot users with environments irrelevant to the post can let us know what they have as well.
Re:Channel Bonding by Slashcrap · 2006-05-25 22:11 · Score: 1

Maybe the other million Slashdot users with environments irrelevant to the post can let us know what they have as well.

I don't allow any hardware on my network unless it's fully RFC1149 compliant.

And yes, the latency is a major concern. Thanks for asking.
Re:Channel Bonding by kjs3 · 2006-05-26 02:06 · Score: 1

I tried that. I got tired of having to shovel poop out from under the raised floor.

No kidding by ShakaUVM · 2006-05-25 07:46 · Score: 4, Interesting

Er, yeah. No kidding.

When I was writing applications at the San Diego Supercomputer Center, latency between nodes was the single greatest obstacle to getting your CPUs to running at their full capacity. A CPU waiting to get its data is a useless CPU.

Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

As a side note, how many parallel apps solve the latency issue is by overlapping their communication and computation phases, instead of having them in discrete phases, this can greatly reduce the time a CPU is idle.

The KeLP kernel does overlapping automatically for you if you want: http://www-cse.ucsd.edu/groups/hpcl/scg/kelp.html

Metric System Still Not Clear To Some by Anonymous Coward · 2006-05-25 07:50 · Score: 1, Funny

Maybe the author meant "imperial milliseconds"?

OK article, bad title by Medievalist · 2006-05-25 07:54 · Score: 1

The article's worth reading, if you're not already familiar with currently popular cluster interconnects, but the title of "Data center networks often exclude Ethernet" is totally bogus.

I guess "Some Tiny Percentage of Data Centers use Something Faster than Ethernet in addition to Ethernet" didn't fit on the page.

Metric system bites U.S in ass again by Anonymous Coward · 2006-05-25 07:55 · Score: 0

I'm thinking that the reason the article got the idea that milli = millionth is because the US doesn't use the metric system.

All 7th graders in Canada know that micro means millionth, and milli = thousandth...hopefully the doctors and nurses in the US know the same thing.

Crashed any rockets lately?

Re:Metric system bites U.S in ass again by Anonymous Coward · 2006-05-25 23:56 · Score: 0

Its taught in science classes all the time dipshit. Go choke on a donut while fucking your hockey stick. See generic cheap shots hurt.
Re:Metric system bites U.S in ass again by Anonymous Coward · 2006-05-26 05:04 · Score: 0

So would the splinters.

Milliseconds? 100's of them? by vidarlo · 2006-05-25 08:00 · Score: 1

--- malin.vidarlo.net ping statistics --- 15 packets transmitted, 15 received, 0% packet loss, time 14003ms rtt min/avg/max/mdev = 0.310/0.347/0.375/0.019 ms 2 hops, over 100Mb ethernet with a cheapass switch (8 port unmanaged hp). Seems like he got no grip on numbers...

--
Assembling etherkillers for fun an profit

Re:30 GB? Take that NSA and your outdated 622MB! by drinkypoo · 2006-05-25 08:01 · Score: 1

MB is a measurement of data; in this case 10^6 bytes. (MiB would be 2^10.) I think you want a measurement of data, such as perhaps MBps. Too bad your comment is a) wrong and b) wrong. specifically it's a) just plain wrong and b) fails to take clusters into account. Were you trying to get fp or something? Anyway "this equipment was the Narus ST-6400, a machine that was capable of monitoring over 622 Mbits/second in real time in May, 2000... The latest generation is called NarusInsight, capable of monitoring 10 billion bits of data per second" - how do you know they're not using the current version today?

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"

Someone needs to look at their network... by Arimus · 2006-05-25 08:01 · Score: 1

Just had a quick ping to the beeb... via a wireless hop onto my ethernet network, two hops to my adsl router, then 6 hops around Nildram's network (ATM into their network then god knows, probably some form of gigabit ethernet) and a couple more hops to the bbc.

Average latency is around 20ms.

Now I know this isn't as plain as straight ethernet but I'd have guessed the latency if anything on ATM + the change from 802.11g to ethernet to atm to ethernet to whatever would have been worse.

So either someone is using cheep hardware or has misconfigured their network.

Apart from that if I was running a cluster each machine would probably have two NIC's depending on their use - one using gigabit ethernet to provide the internal network between nodes on the cluster and the other for external use. The external network would be as normal, the internal network I'd ensure had minimal routers/switches between the nodes and any switches/routers where a) good quality and b) correctly configured.

--
--- Users are like bacteria -> Each one causing a thousand tiny crises until the host finally gives up and dies.

Real-World Experience by FooHentai · 2006-05-25 08:02 · Score: 1

All this time I've been playing Quake over LAN and I thought my ping was about 5ms. Silly me, it's clearly in the range of 100ms, even worse than when I take it online!

Whoops...

Re:Real-World Experience by Anonymous Coward · 2006-05-25 08:27 · Score: 0

DDR Infiniband is a great way to push large amounts of throughput with low latency. ~20Gbps with 2.7us latency (not counting switch ASICs)... each switch hop being about .3us latency... You can use the IP-over-IB driver... latency goes up a little, and throughput goes down a little because of the overhead... but for the most part it's quite nice.

Infinipath is even lower latency and more native IP support. Infinipath is only SDR right now so ~10Gbps with sub 1us latency on the card, and the same switch hop latency additions.

Re:Store & Forward ONLY for 10 to 100 to 1,000 by drinkypoo · 2006-05-25 08:04 · Score: 1

People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone... You can run 100Mbps to the clients, and run gig to the switch only. Of course, since just about everything but laptops is coming with gig now (and probably some of them) this is becoming less valuable.

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"

Re:Store & Forward ONLY for 10 to 100 to 1,000 by multipartmixed · 2006-05-25 08:06 · Score: 1

There's plenty of hardware out there that doesn't come gig-e equipped. Hell, I still deploy RS232 terminal concentrators at 10 megs now and then.

--

Do daemons dream of electric sleep()?

Re:Store & Forward ONLY for 10 to 100 to 1,000 by drinkypoo · 2006-05-25 08:09 · Score: 1

While that's true, most of the time those kind of devices would be happiest on their own subnet for security and management reasons - or at least, I'd be happiest with them there. Therefore they can live on different router interfaces, whether the router's from cisco, or a PC from fry's with linux on it. The only time it's really necessary to mix speeds on the same switch is when you have multiple clients accessing a resource and their aggregate speeds make it useful.

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"

Re:30 GB? Take that NSA and your outdated 622MB! by Captain+Splendid · 2006-05-25 08:14 · Score: 1

Dude, you've been really bitchy lately, what's up witht hat?

--
Linux, you magnificent bastard, I read the fucking manual!

Re:Didn't RTFA? -Infiniband, FC and Myrinet beat E by Anarke_Incarnate · 2006-05-25 08:16 · Score: 1

I believe you mean just Fiber cable, as "Fibre Channel" is an interconnect for storage, etc.

http://en.wikipedia.org/wiki/Fibre_Channel

More Dilbert by sconeu · 2006-05-25 08:16 · Score: 1

No, your token is lost in the Ethernet

--
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.

Re:More Dilbert by Anonymous Coward · 2006-05-25 12:08 · Score: 0

No, you use your ethernet to catch an etherbunny.

The worst post! by Anonymous Coward · 2006-05-25 08:22 · Score: 3, Informative

I wonder what's happening to slashdot. That's as bad as technical news can get. Ethernet latency -- 100ms?? Typical Ethernet latencies are around a few hundred microseconds. Even the ping round-trip time from my machine to google.com is about 20ms.

$ ping google.com
PING google.com (64.233.167.99) 56(84) bytes of data.
64 bytes from 64.233.167.99: icmp_seq=1 ttl=241 time=20.1 ms
64 bytes from 64.233.167.99: icmp_seq=2 ttl=241 time=19.6 ms
64 bytes from 64.233.167.99: icmp_seq=3 ttl=241 time=19.5 ms

What a shame that such a post is on the front page of slashdot! Someone please s/milli/micro.

Re:30 GB? Take that NSA and your outdated 622MB! by drinkypoo · 2006-05-25 08:32 · Score: 1

Probably just trying to release job-related stress. I've been missing my primary outlet since I damaged my race-suspension '89 Nissan 240SX and went back to driving my sloppy-suspension '81 M-B 300SD. This is a lot safer, anyway...

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"

For performance, run the same speed. by khasim · 2006-05-25 08:34 · Score: 4, Interesting

People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone...

What's with the "need to"?

I'm talking performance. Store & Forward hammers your performance. In my experience, you get better performance when you run the server at 100Mb full duplex (along with all the workstations) and use Cut Through than if you have the server on a Gb port, but run Store & Forward to your 100Mb workstations.

Re:For performance, run the same speed. by Anonymous Coward · 2006-05-25 09:48 · Score: 0

Which is true for 1 or 2 clients at a time, but not when you have more data streams. Let's see you get an agregate bandwidth of 465 Mbps (actual performance on my network) to 50 clients through you 100 Mbps connection.

Re:30 GB? Take that NSA and your outdated 622MB! by Marxist+Hacker+42 · 2006-05-25 08:40 · Score: 1

Were you trying to get fp or something?

Yes, and you're right, I should have said Mbps and Gbps (30 Gigabit networking is going to create a packet flow that far outstrips the NSA's 622Megabit packet sniffing capability).

--
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.

Re:30 GB? Take that NSA and your outdated 622MB! by Anonymous Coward · 2006-05-25 08:41 · Score: 0

"MB is a measurement of data; in this case 10^6 bytes. (MiB would be 2^10.) I think you want a measurement of data, such as perhaps MBps."
No offense, but, but I think what you wanted to say was "I think you want a measure of data transmission speed", such as perhaps MBps".

Generally, though, it is "Mbps" for "Megabits per second", and not "MBps", which would be "Megabytes per second".

Re:30 GB? Take that NSA and your outdated 622MB! by Anonymous Coward · 2006-05-25 08:42 · Score: 0

The latest generation is called NarusInsight, capable of monitoring 10 billion bits of data per second

Good pedantry, but it's worth pointing out that InfiniBand runs at 30 Gbps, which is in fact faster than the 10 Gbps that you claim the NarusInsight can do.

Slashdot summary wrong, actual article is better by m.dillon · 2006-05-25 08:45 · Score: 3, Interesting

The slashdot summary is wrong. If you read the actual article the author has it mostly correct except for one comment near the end.

Ethernet latency is about 100uS through a gigE switch, round-trip. A full-sized packet takes about 200uS (micro seconds), round-trip. Single-ended latency is about half of that.

There are proprietary technologies that have much faster interconnects, such as the infiniband technology described in the article. But the article also mentions the roadblock that a proprietary technology respresents over a widely-vendored standard. The plain fact of the matter is that ethernet is so ridiculously cheap these days it makes more sense to solve the latency issue in software, for example by designing a better cache coherency management model and by designing better clustered applications, then it does with expensive proprietary hardware.

-Matt

I duhno . . . by JazzLad · 2006-05-25 08:52 · Score: 0

100/1,000,000th sounds ok. That's what, 1/10th a millisecond?

;)

--
"If you have nothing to hide, you have nothing to fear." - Every fascist, ever

Re:30 GB? Take that NSA and your outdated 622MB! by drinkypoo · 2006-05-25 08:52 · Score: 1

Well, more than one of them can do. Is there any reason they can't use three insight boxes? Or maybe four, just to have some slack :) It might require additional hardware to split up traffic, but...

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"

The real title, "Most Data Centers aren't stupid.. by wsanders · 2006-05-25 09:03 · Score: 1

..or cheap enough to use Ethernet for processor interconnect.

SGI had some kind of shared-memory-over-Ethernet protocol back in the day. Worked about as well as a steam-powered ornithopter. It was designed for customers too cheap or unconcerned about performance to use when they had to.

And I dabbled in OpenMP or whateveritwas back at a contract with just one such cheap customer, and they got what they paid for. Here's a nickel, kid.

Ethernet is Ethernet, and Infiniband et.al. is Infiniband et.al., dad-gummit.

--
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"

Ethernet Problems, IB problems, etc by mrjimorg · 2006-05-25 09:20 · Score: 2, Interesting

Note: I do have a dog in this fight.
One thing that isn't mentioned in the article is the amount of CPU power required to send out ethernet packets. The typical rule is 1 GHz of processing power is required to send 1 Gb of data on the wire. So, if you want to send 10 Gbs of data, you'd need 10 GHz of processor - pretty steep price. Some companies have managed to get this down to 1 GHz/3 Gbs of processing, and one startup(NetEffect) is now claiming roughly ~0.1 Ghz for ~8 Gbs on the wire, using iWarp. With this, your system can be processing information rather than creating packets.
The problem with Infiniband, Myranet, etc is that they require another card in your system (and associated heat problems, size issues, etc), special switches and equipment, and new training for your staff on how to get it up and going. However, IWarp, which is based on TCP/IP can use your standard DHCP, ping, tracert, ipconfig, etc and can allow a single card to be used for networking to the outside world (TCP/IP), clustering in the datacenter(IWarp), and storage (IScsi). 1 card, no special new software widgets, 10 Gb speeds.
However, you cant go and buy a iWarp card from Fry's today. Although, you cant buy an infiniband or myranet card there either

That's not his only mistake. by Short+Circuit · 2006-05-25 09:31 · Score: 1

Such a setup requires extremely low latency, as the processors are pulling Linux operating system images over the InfiniBand links, instead of through a local hard drive. Also, processes shared in RAM among the Linux nodes all run through the Voltaire switch.

Loading bulk data over the network (as in BOOTP) suggests high bandwidth, not latency. And it doesn't even require it; high bandwidth for BOOTP is a convenience. My 10Mb/s Ethernet hub could do it.

The author really is clueless...

--
tasks(723) drafts(105) languages(484) examples(29106)

Tolkien ring by Shabazz+Rabbinowitz · 2006-05-25 09:37 · Score: 4, Funny

I had recently considered using this Tolkien ring until I found out that deinstallation is very difficult. Something about having to take it to a smelter.

Re:Didn't RTFA? -Infiniband, FC and Myrinet beat E by Anonymous Coward · 2006-05-25 09:50 · Score: 0

For all you seem to know about Fiber Channel, we'd think you'd read a little further down your Wikipedia article and discover that FC does indeed support both ATM **AND** (this is a critical one) IP for computer-computer communication. Furthermore, you apparently don't know that FC isn't limited to fiber optic cable interconnects, though that's often where it's found. Beyond that, you surely don't know that FC works at 1,2,4,8 Gb/s, and was designed from the ground up to be a low-latency protocol... Naturally, that makes it great at storage, AND communication at large datacenters that already have the need for large amounts of NAS.

Bad Parent Article by Anonymous Coward · 2006-05-25 09:53 · Score: 0, Interesting

Article is retarded/biased/misinformed/didn't even bother to do actual research; my 48 node gigabit switch has 1ms and its quite busy b/w-wise the way we abuse it...has this genius ever heard of things called full duplex and not mixing speeds in a freaking DATA CENTER? Mine sits in a closet, not even on a rack.

Hell, I think most of my gaming servers are all sub 50ms and thats internet via cableco_that_massively_oversells_bandwidth...

Re:Slashdot summary wrong, actual article is bette by trollogic · 2006-05-25 09:58 · Score: 2, Insightful

I think you have no clue about what your saying. 1) InfiniBand is an open standard hosted by IBTA which is a consortium of companies. The spec is available for anyone who wants to understand/build InfiniBand hardware. Not IEEE does not make it proprietary. 2) The major roadblock with 10Gbps is physics. You can only reach so far with copper without retiming the signal. And optics are expensive. 10 GbE has the same problem and it won't be cheap any time soon. 3) InfiniBand has already reached a volume where on-board IB chips are available in $70-80 range .. 10 GbE is no where close. And IB DDR will be shipping next month (20 Gbps wire / 16 Gbps data). 4) Beowulfs are popular for a reason .. Cache Coherency is a bitch. 5) A round trip node-to-node latency in IB is 2.7 usecs (best case of course). With all the optimization in the world, you won't be able to get ethernet anywhere near that number. 6) InfiniBand is being WIDELY deployed. Sandia Thunderbird is a 9216 processor IB fabric in production. NCSA has Tungsten2 which is 1024 processor IB fabric. NCSA also has a Microsoft Windows Cluster running CCE over IB with 880 processors. There are several large firms Oil&Gas, BioTech, Banks, Market Data houses which run several large multi-hundred/multi-thousand processor IB clusters. 7) Just as with any technology it will take time for new technologies to be accessible to the masses .. so don't write off anything yet. 8) Do you research before you open your mouth.

Let's get some real speed by HardwarePeteUK · 2006-05-25 09:59 · Score: 1

I have actually designed a very big Infiniband switch (288 ports, 10Gb/s full duplex each port, fully bisectional), and that was a couple of years back. I also designed other things InfiniBand, but that's another story :) I can assure you, if we can do it, they have some of them. Oh, the throughput numbers (for a 7U rack) are 288 * 8Gb/s * 2 (bidirectional, 8b/10b encoded in the link) = 4.608Tb/s data throughput = 576MByte/sec. PeteS

Maybe useless info: TOP500 interconnect statistics by MojoStan · 2006-05-25 10:07 · Score: 2, Informative

Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

That reminded me of the TOP500's statistics generator, so I just had to look up the current list's (November 2005) statistics for "interconnect family". For those that are curious:

Myrinet is the second most-used interconnect in the TOP500 at 14% (70 out of 500) followed by HyperPlex at 6% (31).
Gigabit ethernet is by far the most used interconnect at 50% (249).

In the TOP500, it looks like ethernet is not yet an "outsider." Perhaps in the "top 100."

--
TO START PRESS ANY KEY

Where's the 'ANY' key? I see Esk, Kitarl, and Pig-Up...

Well yeah... by Junta · 2006-05-25 10:10 · Score: 1

Well, except the oblig. s/ms/us, but pretty much yeah. With Pathscale (now QLogic) Infinipath HTX cards, you can get 1.5 us latency between nodes, Myrinet 10 G PCI-E can get about 2.5 us. Note that there is now 10Gb ethernet making inroads to compete on terms of throughput (which Infinband SDR, Myrinet are roughly 10 Gbps), but latency is of course still problematic. One chief advantage of non-ethernet is those networks are source routed and every node has a full topology map of how to get to their destinations. This has the benefit of distributing the task of routing to more processors, as well as making intelligent routing decisions. With ethernet, switches have a very heavy routing burden in a busy network. Compare this to a Myrinet or Infiniband switch which merely needs to look at the next port tag and send it on. By and large when trying to do benchmarks on these technologies, we generally don't worry too much about which switch is used. Contrast with Ethernet where we have to be mindful of the packets per second capability of the switch...

Of course, on a large scale network, it is much simpler and easier to do switch-routing frames, but for tightly controlled networks, source-routed can be very advantageous.

I will say switch routed frames have the *potential* for much better utilization of multi-port aggregations, but largely the member of a multi-port aggregation used to send a packet is not based on port congestion, but rather on a hash of the MAC address referenced in the packet, which is nothing a source routed network couldn't do.

--
XML is like violence. If it doesn't solve the problem, use more.

Re:Well yeah... by luken · 2006-05-25 18:01 · Score: 1

Infiniband (IB) isn't source routed, though myrinet is. Myrinet has the route built into the packet header and each switch rips off it's instructions as the packet enters the switch.

IB packets are associated with a Queue Pair (QP) which is associated with 2 LIDs, one for each end of the connection. I believe that the switches key off the target machine's LID and use that for a table based lookup. I keep getting my acronyms mixed up, but IB switches do a table lookup to determin the out going port for the packet that is incoming, just like ethernet, though most (if not all) of the information is pre-programmed into the switch chips via the Session Manager (SM) eliminating the need to go and query a higher authority (like ethernet ARP).

Ugh, they must have read an old paper by servanya · 2006-05-25 10:16 · Score: 1

All new switches (that are decent) employ cut-through forwarding.

Re:Ugh, they must have read an old paper by Percy_Blakeney · 2006-05-25 17:15 · Score: 1

No, most new switches use store-and-forward, especially when they have gigabit-or-faster ports. The latency "penalty" involved in store-and-forward switching becomes less and less noticable at faster network speeds, thus making cut-through undesirable (i.e. its costs outweigh its benefits). As an example, look at Foundry's EdgeIron 8X10G:

http://www.foundrynet.com/services/documentation/e dgeiron_install/7_intro_8X10G.html

It employs store-and-forward, as do most new Cisco switches (if I remember correctly). I can understand why certain ultra-latency-sensitive applications may still require cut-through switching, but 95% of all other applications won't gain much from being on a cut-through switch.

To be fair to ethernet and others by HardwarePeteUK · 2006-05-25 10:17 · Score: 1

Ethernet is pervasive. If you have a recent system, you have Gbit ethernet, although the latency kills you in highspeed interconnects, but you have to ask; is it important?

For the majority of applications, the answer is no.

For some (supercomputing really) where all processors in the cluster need to be kept busy and not waste any time getting their data, it is a big deal.

In those applications, you are not going to worry about buying the InfiniBand HCA (or Myrinet system) and the various devices that also speak (IB, Myrinet, your choice of highspeed interconnect here).

In those cases, you use what you need, not what you have.

PeteS

For those who don't understand... by bill_kress · 2006-05-25 10:21 · Score: 3, Informative

Most (all?) Ethernet hardware reads in an entire packet, looks at it, then sends it on to a destination. This makes building routers and switching hardware fairly easy but extremely slow.

If you go to a high-speed network, what you get is a packet being forwarded as it's being read. By the time the first few bits are through the switch, it should be able to figure out the next hop and have the packet moving in that direction. Phone companies have huge problems with the delays in Ethernet. This is why the ATM protocol was invented, it's hard to use, awkward and not too graceful, but it can fly through a switching network like nobody's business.

Ethernet is also extremely sloppy--Any switch along the way is allowed to throw a packet away and wait for the originator to resend causing a HUGE hiccupp in the communication stream (Most if not all routers do this whenever an address is not in it's forwarding table yet).

IIRC the faster protocols see a "Routing" packet in the stream and set up forwarding hardware before getting the actual packet/stream, then wait until the end of the packet (or entire stream) to tear the route down again.

Ethernet, however, due to it's simplicity is bridging the gaps. It's a pretty crappy protocol in general, but we keep throwing better, smarter hardware at it in an effort to brute-force it into the parameters we require. (I work for a company that makes Ethernet over fiber hardware, and have worked for companies based around ATM, SONET and other interesting solutions).

I guess the point of the article was to remind a world that is coming to believe that ethernet is the end-all be-all of networking that it was always just the simplest hack available and therefore the easiest to deal with.

Just like SNMP.

Re:For those who don't understand... by servanya · 2006-05-25 10:33 · Score: 2

Most (all?) Ethernet hardware reads in an entire packet, looks at it, then sends it on to a destination. This makes building routers and switching hardware fairly easy but extremely slow.

First, Ethernet doesn't forward packets. It forwards frames.
Most (all?) ethernet switches read just the destination MAC and start forwarding it, just as you've described in the next paragraph. If it can't, because there's no bridge table entry for the destination, it floods the frame.

Ethernet is also extremely sloppy--Any switch along the way is allowed to throw a packet away and wait for the originator to resend causing a HUGE hiccupp in the communication stream (Most if not all routers do this whenever an address is not in it's forwarding table yet).

Don't start confusing people with L2/L3 comparisons. Routers will drop a packet that it can't forward, but most routers unknowlegable people deal with will have some sort of a default route, so that never happens. In layer 2 land, however, frames are FLOODED when the destination is unknown.

IIRC the faster protocols see a "Routing" packet in the stream and set up forwarding hardware before getting the actual packet/stream, then wait until the end of the packet (or entire stream) to tear the route down again.

I think I might have an idea what you're talking about here, but it's hard to tell.
Re:For those who don't understand... by n8_f · 2006-05-25 18:49 · Score: 1

He's talking about virtual circuits. Before ATM cells (frames) can be sent to their destination, a connection has to be opened to that destination. Each router in the path from the starting point to the end point has to be able to guarantee the level of service the connection needs (Wikipedia explains). So each router knows exactly where each incoming cell is going. This speeds things up quite a bit, but the problem is that the connection is reserved whether it is used or not. Connections can allocate chunks of bandwidth and that bandwidth is tied up whether or not it is being used, because it might be used and the router has committed to a certain level of service. And there is a single route from start to finish, not the multiple routes that an IP packet might get take (Ethernet frames don't get routed). It makes things faster, but it reduces flexibility and it requires greater resources and complexity.
While I'm writing, I didn't get why they used the term "data center." I think of a data center as something like a colo facility or one of Google's server farms, but the article talked exclusively about computer clusters and HPC. Is that just me?

Re:Store & Forward ONLY for 10 to 100 to 1,000 by Sigma+7 · 2006-05-25 11:16 · Score: 1

#1. You're running different speeds on the same switch (why?).

AFAIK, wireless doesn't consistantly support 100Mbps compared to local Ethernet. Usually, I get around 54Mbps, or possibly 10 Mbps on a weak signal.

That's why you still see store-and-forward - Wireless and wired networks are different speeds.

Re:Store & Forward ONLY for 10 to 100 to 1,000 by petermgreen · 2006-05-25 11:49 · Score: 1

#1. You're running different speeds on the same switch (why?).
lets see:

you have an older but still functional and economical to run printer with a 10base2/T combo card in it and for which a replacement card would be either expensive or unobtainable.

you have 100mbit to most of the desktops because your wiring wasn't done well enough for gig-e to cope.

you have gigabit to your servers

you have a 10 gigabit backbone link.

also even if a switch is cutting through a lot of packets its still going to have to queue those that arrive while another packet is going out onto the backbone (assuming you have a hierachical network and most traffic is client-backbone). So i can't imagine the peak memory needs would be that much lower.

--
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register

Re:Store & Forward ONLY for 10 to 100 to 1,000 by daverabbitz · 2006-05-25 12:18 · Score: 1

[quote]
AFAIK, wireless doesn't consistantly support 100Mbps compared to local Ethernet. Usually, I get around 54Mbps, or possibly 10 Mbps on a weak signal.

That's why you still see store-and-forward - Wireless and wired networks are different speeds.
[/quote]

uh, wireless can't be patched into an ethernet segment, an AP is a router, or at the very least a bridge, and generally has a 100Mbit Ethernet interface. that is not the same thing as patching 100M into a Gigabit Switch. But yes, wireless AP's should be on a different segment anyway (and probably be behind a firewall?).

--
What could be better than a jet powered motorcycle? http://www.youtube.com/watch?v=u8l6GTHLSWE

Re:30 GB? Take that NSA and your outdated 622MB! by Savantissimo · 2006-05-25 12:25 · Score: 1

The NSA's network sniffer, recently discovered at an AT&T broadband center, can only sniff up to 622MB. Sounds to me like if you use an InfiniBand switch, that would effectively make the output of the NSA's network sniffers complete gibberish.

My journal has more info on the latest Naurus, the Insight, as well as info gleaned from their website and links from their website on the connections between Naurus and intelligence agencies and contractors. The 6400 was installed 6 years ago at AT&T by the NSA - it has likely been upgraded. The Insight can do 2.5 Gbps (OC-48) at the application layer or 10 Gbps (OC-192) at the transport layer of the encapsulated TCP/IP stream. There are no faster WAN links in use than 2.5 Gbps at the regional carrier level or OC-192 at the continental carrier level AFAIK (last personal knowledge is 2 years old, but the network was at 10% capacity then, so I doubt they have upgraded). For undersea use they might use bigger pipes (I don't know), but generally there is enough dark fiber that the less-exotic equipment is more effective - just use more physical links and get greater redundancy.

--
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry

Passive Backplane... by TheTitan · 2006-05-25 12:48 · Score: 0

You idiots: don't use active backplane switches (aka: Cisco). Drop me an email and I'll fix you up with something that falls into the "doesn't suck" category.

--
-- Sean Chittenden

Not so sure by kilodelta · 2006-05-25 12:54 · Score: 1

But on our network we vlan'd everything out. All servers on one vlan, I.T. on another vlan, and then major groups on their own vlans. Keeps traffic nice and segregated which is why the I.T. shop has iTunes sharing turned on full blast.

But here's where I notice some performance. We've got all the servers on a gigabit vlan. I can shift a 300MB file between servers in under 20 seconds. Transitioning a 5MB link takes five minutes.

So we did what we could to eliminate latency and we see it in the performance of our network.

Re:Not so sure by Slashcrap · 2006-05-25 22:17 · Score: 1

But here's where I notice some performance. We've got all the servers on a gigabit vlan. I can shift a 300MB file between servers in under 20 seconds. Transitioning a 5MB link takes five minutes.

This article is so not about your kind of setup.

Re:30 GB? Take that NSA and your outdated 622MB! by Captain+Splendid · 2006-05-25 13:01 · Score: 1

Fair enough. Hope you get your groove back, though, you're more fun than this usually.

--
Linux, you magnificent bastard, I read the fucking manual!

Here's a comment by Anonymous Coward · 2006-05-25 14:05 · Score: 0

You're retarded.

Could somebody clue me in? by Omega+Blue · 2006-05-25 14:41 · Score: 1

"But in data centers, where CPUs may be sharing data in memory across different connected machines..."

I have re-read this bit like twenty times and still have no idea what it means. The terms used clashes badly, which leads me to believe that the guy has no idea what he was talking about

Re:Could somebody clue me in? by Anonymous Coward · 2006-05-25 15:55 · Score: 0

Maybe they mean something like http://wiki.tangosol.com/ ?
Re:Could somebody clue me in? by Anonymous Coward · 2006-05-25 18:29 · Score: 0

OK.
This article/summary mess up is really ridiculious.

Just a few quick point ignoring corrections people have made already.
100 us is too much latency for gigE NIC -> NIC uncontested LAN latency.
It was 50 us in 1999 without the amazing other work thats been done since.

Here's a more fair comparision ...

http://www.disi.unige.it/project/GAMMA/

10 us (roundtrip though a switch)
6 us (direct)

Ok, so 200 ns still wins, but at least be honest.
There are a lot of complex tradeoffs here, but the
point is the article is a little slanted, IMO.

So a real group would:
- price possible interconnects
- Cpu/Node costs.
- per-node switching fabric costs.

At maximize performance per $ for the money they have.

So there choice depend on:
1) How much money they have.
2) How what your code looks like.

saying everyone _needs_ these special interconnect for high speed is wrong
in fact I think most groups would be better buying more nodes or NICs.

- Garick
Re:Could somebody clue me in? by Anonymous Coward · 2006-05-25 18:50 · Score: 0

They mean generally clusters/grids working running a application and dataset
spread accross nodes, I think. :)

So, programatically High Performance Computing People generally use a layer
like MPI to write this applications and they send the data to each other as
needed.

But, addressing the "in memory bit". I think this is a fixation that pertains
to the products being hyped. The cards can do a lot of the work copying memory
out of packets into to system memory. Now, this isn't exclusively a thing that
is only done by the "fancy" interconnects.

Basically most NICs support some kind of scatter gather DMA.
some NICS are capable to do RDMA (R = Remote).

Basically driver+hardware on NIC A can basically do a DMA to host B (If host B
allows it). The security is complex but this is the same thing that the "fancy"
boys do.

basically its a local memory to/from remote memory fetch/store over the network.

Garick
Re:Could somebody clue me in? by Anonymous Coward · 2006-05-26 01:25 · Score: 0

I have re-read this bit like twenty times and still have no idea what it means. The terms used clashes badly, which leads me to believe that the guy has no idea what he was talking about

It's called Clustering. You might say that clustering is the Quantum Mechanics of the networking world, in that the usual rules are... well... weird.
There are some good tutorials on this at ClusterMonkey's web site.

Oh, and BTW, TFA has been edited to fix the "millisecond" goofiness. Well, some of it anyway.

Re:Slashdot summary wrong, actual article is bette by m.dillon · 2006-05-25 15:40 · Score: 1

Boy, you sure have a foul mouth. I suggest washing it out with soap.

My comments stand. Start posting prices and lets see how your idea of an open standard stacks up the reality. Oh yah, and remember for every $1000 you spend on your interconnect, that's $1000 less you have to spend on cpus and programmers with a clue.

The reality is that there is only one *correct* way to do a fast interconnect, and that is to build it into the CPU itself. Oh wait, AMD intends to do just that! That's what I'm waiting for. A cheap built-in interconnect that doesn't cost an arm and a leg or eat 100 watts of power all by itself. All this other junk has a shelf life of maybe a few years at best (as does pretty much all networking gear, but the difference is that this stuff costs 10 times as much). It's a huge waste of time for all but the most extreme clustered applications for which there is no algorithmic solution to the latency issue (read: people like to throw hardware at badly written programs more often then they should).

-Matt

RADIANT/SyNeRGy and NOWLAB have better information by Anonymous Coward · 2006-05-25 17:03 · Score: 0

It is nice to see people recognizing this, but the fact that data centers are not using ethernet is old news. The latency for tightly coupled distributed applications requires near microsecond latency that only non-ethernet networks could provide. That is until 10 Gigabit Ethernet (10GigE) burst onto the scene.

For several years now, 10GigE has been researched by several teams, most notably by RADIANT at Los Alamos National Lab (www.lanl.gov/radiant) (which is now SyNeRGy at Virginia Tech (website not yet online)) and NOWLAB at Ohio State (nowlab.cse.ohio-state.edu/).

With the inclusing of TCP offload-engines (TOE), ethernet hardware is becoming more similar to the non-ethernet hardware. Performance numbers show that 10GigE both competes with and surpasses non-ethernet networks in different benchmarks. But not in everything, thereby making the cliche, "choose the best tool for the job", more than adequate.

The edge in bandwidth and cost has always gone to ethernet due to economies of scale. However, niche markets that don't care about cost almost always choose non-ethernet to acheive the lowest-latency possible. Check out Top500.org for the break down of the interconnects used in the top 500 supercomputers world-wide (www.top500.org/lists/2005/11/l/Interconnect_Famil y)

For those interested in learning more about high-performance networks such as 10GigE and InfiniBand, as well as how ethernet and non-ethernet technologies are converging, I recommend reading the plethora of technical papers found on the RADIANT and NOWLAB websites.

Re:Maybe useless info: TOP500 interconnect statist by pyite · 2006-05-25 17:17 · Score: 1

In the TOP500, it looks like ethernet is not yet an "outsider." Perhaps in the "top 100."

It depends on what you're doing. If your job is highly parallel, Ethernet is fine. But what happens when every CPU needs access to every other CPUs results in "real time?" Well, low latency is then a must. 1 ms latency is potentially millions of wasted cycles.

--

"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman

Re:Maybe useless info: TOP500 interconnect statist by ShakaUVM · 2006-05-25 17:34 · Score: 1

/shrug

In practice, cheap is the reality. Just like how consumer goods dominate the market, with much less prosumer and professional equipment sold.

Fast interconnects are way more expensive than ethernet. People that want the extra performance, though, pay for it.

Re:Slashdot summary wrong, actual article is bette by trollogic · 2006-05-25 17:54 · Score: 1

Let me try and jerk out into reality. Here is the press release from Mellanox announcing $69 per IB HCA chip in volume. http://www.mellanox.com/news/press_releases/pr_030 105.php
Agreed its not widespread yet. But there are systems manufacturers actively looking at deploying this silicon on the mother board. IWILL is one company thats already started. Asus, Tyan are next.

The reality is that there is only one *correct* way to do a fast interconnect, and that is to build it into the CPU itself. Oh wait, AMD intends to do just that!

You do realize on board interconnects can only scale so much .. before you run into serious data coherency issues. Intel Xeon's can scale up to 4 processors per board. IBM has tweaked so they can scale up to 8 procs on their proprietory motherboards. AMD Hypertransport allows you to scale upto 8 sockets per motherboard. To scale beyond that you have to go use NUMA like interconnect (pricey) or push it on a fast pipe to other boxes (not so pricey). AMD recommends this. AMD specifically recommends to use InfiniBand to push high bandwidth. Go talk to AMD. Hypertransport is available *today* in the market. Go talk to supermicro. You don't have to wait for it. All this other junk has a shelf life of maybe a few years at best (as does pretty much all networking gear, but the difference is that this stuff costs 10 times as much). Pretty much everything in computers has a shelf life of few years hardware and software .. what is your point ? It's a huge waste of time for all but the most extreme clustered applications for which there is no algorithmic solution to the latency issue (read: people like to throw hardware at badly written programs more often then they should). People are not complete idiots when they are spending millions of dollars setting up 100's or even 1000's of nodes of beowulf clusters over IB. There are specific applications that are latency sensitive and bandwidth hungry. And that is the segment that IB addresses very well and where ethernet is considered an outsider. Tell me you want to use 1G ethernet to process a satellite feed coming in at 3.2 Gbps .. again, I think you are either clueless or ignorant .. just because IB doesn't fit your bill doesn't mean it doesn't have its play.

latency vs. bandwith (ms vs. Bps) by Anonymous Coward · 2006-05-25 18:23 · Score: 0

i see alot of posts saying "why not just use gigabit". er...it's not so simple.

bandwidth and latency are not the same thing folks. it doesn't matter sometimes if you can send a full gigabit in one second. if it takes the whole lot an extra millisecond to get there it's no good in huge datacenters and applications mentioned in the story (summary).

latency is how long it takes a packet to actually get across a network.

bandwidth is how many packets you can shovel onto the network in a short period of time.

high bandwidth is good for sending large files and large chunks of data.

low latency is good for sending lots of seperate bits of small information, like simple browsing, or control signals used in file transfers.

Re:Slashdot summary wrong, actual article is bette by Kynde · 2006-05-25 19:18 · Score: 1

They corrected the article. There were more flaws there earlier than there are now.

--
1 Earth is warming, 2 It's us, 3 it's royally bad, 4 we need to take action NOW

The didn't do enough research on the topic by war3rd · 2006-05-26 00:15 · Score: 1

The funny thing is that there is already a solution to their problem out there. Raptor Networks Technologies already has their ethernet switches in a bunch of places and have (so far) proven that their distributed network technology runs circles around Cisco's (and others) centralized architecture and costs even less. They could probably keep up with the needs of these data centers. I've spoken to guys who use their hardware and they all say 'wow.' This sounds like a perfect network for Raptor's hardware. Anyone else ran into it?

--
Got sushi? The Sushi FAQ

100msec latency for Ethernet? by Anonymous Coward · 2006-05-26 01:52 · Score: 0

The original most is in error... the article states that Ethernet latencies are more in the low MICRO SECOND range, not in the low hundreds of milliseconds.

Try it yourself... ping a machine one or two switch hops away on the local LAN.

Re:Didn't RTFA? -Infiniband, FC and Myrinet beat E by Anarke_Incarnate · 2006-05-26 02:05 · Score: 1

Wow, a mistake was made. Flame on, dumbass. At least I am aware it is Fibre Channel and not Fiber Channel. There is a difference. I am also aware of the speeds FC runs on as well as it not being restricted to "glass" cable. I am glad you could make so many ASSumptions based on the tiny snippet of what I said.

Re:Didn't RTFA? -Infiniband, FC and Myrinet beat E by Mysticalfruit · 2006-05-26 03:39 · Score: 1

Actually you can use FC for ATM and IP, but hardly anybody does because of the expense, and for the fact that it's something different from ethernet. It's only great for storage because of the low latency.

However, if FC doesn't embrace higher speeds such as 10gbit I can see 10gigE displacing it, provided that companies can produce switching hareware that'll have the same latency speed as FC.

For what I use FC for it's great (storage), but I'd be much happier if everything was just Ethernet.

--
Yes Francis, the world has gone crazy.

Re:30 GB? Take that NSA and your outdated 622MB! by Anonymous Coward · 2006-05-26 09:28 · Score: 0

Hell, my company has an OC-192 coming in, and we're far from a "continental carrier".

Re:30 GB? Take that NSA and your outdated 622MB! by Savantissimo · 2006-05-27 10:59 · Score: 1

Big carriers use lots of lines going lots of places and won't risk a single point of failure.

--
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry

Blue Gene by lon3st4r · 2006-05-27 15:33 · Score: 1

I went over to the BlueGene/L's page to see how they manage to shuffle data around. They seem to use 1GB Ethernet links for IO nodes.

Any ideas why Ethernet is not an outsider here.

* lon3st4r *

Re:Slashdot summary wrong, actual article is bette by LordMyren · 2006-06-02 12:35 · Score: 1

I think you have no clue whom you were talking to.

Open Standard says nothing about price.
IB HBA's might be cheap, but the switching fabric sure as fuck aint.

As for cache coherency, you were addressing the man trying to change the cache coherency game. Watch out, skinny.

Lastly, there are some proprietary gigabit technologies (non IP based) that, while not 2.7 usec, are very close. Numerous MPI implementations are written with these technologies, although many also require hardware.

I dont think anyone is writing off IB. Its just a long ways away before we see switches that cost less than very nice houses. You can usually buy a house and stuff it with a couple rooms of gigabit switches for cheaper.

Its great tech, just not anywhere near the realm of mortals.

Re:Slashdot summary wrong, actual article is bette by LordMyren · 2006-06-02 12:56 · Score: 1

You have some very good points. Much better than grandfathers blathering, thanks.

I'd just like to quietly point out that costs are often switching costs, not HBA. Even 10GigE is getting reasonable for the adapters, but the switching costs are still completely astronomical.

It will be very interesting to see what the future of interconnects & cache coherency holds. We're definately rapidly approaching a cusp of something new. AMD today announced they're making a new socket for external peripherials, a great little indicator. I personally think the days of concentional HBA's are rapidly coming to a close, thats the 5 year obsolescence Dillon was talking about. Its going to be a drastic change, not just newer better adapters, newer fabrics, as used to be the case. PCI-ASI (switching fabric built around pci-express, very damned cool) and HT are becoming internal and peripherial busses, becoming full fabrics.

Really, I'm just glad things are finally in motion again, and that multiple groups are trying multiple approaches to this huge problem. There's been a lot of stagnation, besides the slow crawl forwards. Cache coherency and interconnection are being completely rethought, from many different angles.

FYI, Dillon is the guy doing DragonFly BSD, which is working on getting high latency cache coherency working in the OS, across systems. He's a little biased towards the "network is the computer", but in the general case he's dead on. There are definately cases where every usec of latency is a usec of 8,000 computers not doing any work whatsover, in which case IB is probably your god. But for most people? I'd wager HT or PCIe based interconnects, already ubiquitous but not used for system interconnect, will probably take over first. Couple million PCIexpress chipsets shipped this year, how many IB chips? Its advancing whats already used.

Good post mate, cheers.

Slashdot Mirror

Ethernet The Occasional Outsider

169 comments