Ethernet The Occasional Outsider

coondoggie writes to mention an article at NetworkWorld about the outsider status of Ethernet in some high-speed data centers. From the article: "The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range. But in data centers, where CPUs may be sharing data in memory across different connected machines, the smallest hiccups can fail a process or botch data results. 'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,' Garrison says. This forced many data center network designers to look beyond Ethernet for connectivity options."

Comments

100ms ethernet latency?

[victim]

I don't think I need to read anymore, well, I did verify that the number really appears in the article.
This author does not understand the subject material.

(I suppose you could deliberatly overload a switch enough to get this number, maybe, but that would be silly, and your switch would need 1.25Mbytes of packet cache.)

Re:100ms ethernet latency?

[merreborn]

Looks like the author fucked up the definition of millisecond too:

"By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second"

http://www.google.com/search?hl=en&q=define%3Amill isecond&btnG=Google+Search
"One thousandth of a second"

Seriously. How the fuck does this idiot get published?

Re:100ms ethernet latency?

[Phreakiture]

This author does not understand the subject material.

I disagree. The author has simply misplaced his metric units. He used the word "milliseconds", where he should have used the word "microseconds". You can see an example of this where he refers to milliseconds as one millionth of a second, rather than the one thousandth that they actually are.

Not an Auspicious Start

[Anonymous Coward]

From the article, three paragraphs in:
"(By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second)"

That would be one-thousandth, not millionth (aka micro second). Not a good start...

Store & Forward ONLY for 10 to 100 to 1,000.

[khasim]

There are only TWO reasons to use Store & Forward.

#1. You're running different speeds on the same switch (why?).

#2. You really want to cut down on broadcast storms (just fix the real problem, okay?)

Other than that, go for the speed! Full duplex!

Re:Low-cost options?

[dlapine]

Define low cost? Myrinet with less than 10 microsecond latency is normally considered to be the least expensive option. You can check their price lists, but an 8 port solution (with 8 HBA's) will set you back over $8k, not including the fiber.

For some people, that's cheap. If not, sorry.

Re:Did you mean "microseconds"?

[dlapine]

Sure, for an 8 port switch, where all the computers have a direct connection. Consider the issues involved for a router with a 128 machines all trying to cross-communicate. Or larger collections of computers that might need to use multiple sets of switches to span the entire system.

On a Force10 switch, with 2 nodes on the same blade:
tg-c844:~ # ping tg-c845
PING tg-c845.ncsa.teragrid.org (141.142.57.161) from 141.142.57.160 : 56(84) bytes of data.
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=1 ttl=64 time=0.148 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=2 ttl=64 time=0.146 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=3 ttl=64 time=0.145 ms
64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161): icmp_seq=4 ttl=64 time=0.144 ms

The same nodes using a myrinet connection:
tg-c844:~ # ping tg-c845-myri0
PING tg-c845-myri0.ncsa.teragrid.org (172.22.57.161) from 172.22.57.160 : 56(84) bytes of data.
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from tg-c845-myri0.ncsa.teragrid.org (172.22.57.161): icmp_seq=4 ttl=64 time=0.043 ms

The latency gets below 10 usec with the use of special drivers, this is just using the 2.4 Linux tcp stack. What's even scarier about the Myrinet is that I can have all 900+ machines talking at the same time with no drop in latency- we have that network spec'd for full bisection bandwidth. Try that on 900 nodes on a gige network, let alone a 100baseT.

As was mentioned here earlier, ethernet is nice for networks that change. Once you have a significant number of machines attached, and the number of switches and routers gets past 1, ethernet loses it's equivalence in latency.

The worst post!

[Anonymous Coward]

I wonder what's happening to slashdot. That's as bad as technical news can get. Ethernet latency -- 100ms?? Typical Ethernet latencies are around a few hundred microseconds. Even the ping round-trip time from my machine to google.com is about 20ms.

$ ping google.com
PING google.com (64.233.167.99) 56(84) bytes of data.
64 bytes from 64.233.167.99: icmp_seq=1 ttl=241 time=20.1 ms
64 bytes from 64.233.167.99: icmp_seq=2 ttl=241 time=19.6 ms
64 bytes from 64.233.167.99: icmp_seq=3 ttl=241 time=19.5 ms

What a shame that such a post is on the front page of slashdot! Someone please s/milli/micro.

Maybe useless info: TOP500 interconnect statistics

[MojoStan]

Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

That reminded me of the TOP500's statistics generator, so I just had to look up the current list's (November 2005) statistics for "interconnect family". For those that are curious:

• Myrinet is the second most-used interconnect in the TOP500 at 14% (70 out of 500) followed by HyperPlex at 6% (31).
• Gigabit ethernet is by far the most used interconnect at 50% (249).

In the TOP500, it looks like ethernet is not yet an "outsider." Perhaps in the "top 100."

For those who don't understand...

[bill_kress]

Most (all?) Ethernet hardware reads in an entire packet, looks at it, then sends it on to a destination. This makes building routers and switching hardware fairly easy but extremely slow.

If you go to a high-speed network, what you get is a packet being forwarded as it's being read. By the time the first few bits are through the switch, it should be able to figure out the next hop and have the packet moving in that direction. Phone companies have huge problems with the delays in Ethernet. This is why the ATM protocol was invented, it's hard to use, awkward and not too graceful, but it can fly through a switching network like nobody's business.

Ethernet is also extremely sloppy--Any switch along the way is allowed to throw a packet away and wait for the originator to resend causing a HUGE hiccupp in the communication stream (Most if not all routers do this whenever an address is not in it's forwarding table yet).

IIRC the faster protocols see a "Routing" packet in the stream and set up forwarding hardware before getting the actual packet/stream, then wait until the end of the packet (or entire stream) to tear the route down again.

Ethernet, however, due to it's simplicity is bridging the gaps. It's a pretty crappy protocol in general, but we keep throwing better, smarter hardware at it in an effort to brute-force it into the parameters we require. (I work for a company that makes Ethernet over fiber hardware, and have worked for companies based around ATM, SONET and other interesting solutions).

I guess the point of the article was to remind a world that is coming to believe that ethernet is the end-all be-all of networking that it was always just the simplest hack available and therefore the easiest to deal with.

Just like SNMP.