Ethernet The Occasional Outsider

coondoggie writes to mention an article at NetworkWorld about the outsider status of Ethernet in some high-speed data centers. From the article: "The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range. But in data centers, where CPUs may be sharing data in memory across different connected machines, the smallest hiccups can fail a process or botch data results. 'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,' Garrison says. This forced many data center network designers to look beyond Ethernet for connectivity options."

Comments

Didn't RTFA? -Infiniband, FC and Myrinet beat Eth0

[hguorbray]

Actually, even with Gigabit ethernet availability HPTC and other network intensive data center operations have moved to Fibre Channel and things like:

Infiniband http://en.wikipedia.org/wiki/Infiniband

and Myrinet http://en.wikipedia.org/wiki/Myrinet

http://h20311.www2.hp.com/HPC/cache/276360-0-0-0-1 21.html
HP HPTC site

-What's the speed of dark?

No kidding

[ShakaUVM]

Er, yeah. No kidding.

When I was writing applications at the San Diego Supercomputer Center, latency between nodes was the single greatest obstacle to getting your CPUs to running at their full capacity. A CPU waiting to get its data is a useless CPU.

Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

As a side note, how many parallel apps solve the latency issue is by overlapping their communication and computation phases, instead of having them in discrete phases, this can greatly reduce the time a CPU is idle.

The KeLP kernel does overlapping automatically for you if you want: http://www-cse.ucsd.edu/groups/hpcl/scg/kelp.html

For performance, run the same speed.

[khasim]

People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone...

What's with the "need to"?

I'm talking performance. Store & Forward hammers your performance. In my experience, you get better performance when you run the server at 100Mb full duplex (along with all the workstations) and use Cut Through than if you have the server on a Gb port, but run Store & Forward to your 100Mb workstations.