Slashdot Mirror
Corporate Data Centers As Ethernet's Next Frontier
alphadogg writes with a story that's about the possibilities for the next generation(s) of Ethernet, stuff far beyond 10base-T: "Ethernet has conquered much of the network world and is now headed deep into the data center to handle everything from storage to LAN to high-performance computing applications. Cisco, IBM and other big names are behind standards efforts, and while there is some dispute over exactly what to call this technology, vendors seem to be moving ahead with it, and it's already showing up in pre-standard products. 'I don't see any show-stoppers here — it's just time,' says one network equipment-maker rep. 'This is just another evolutionary step. Ethernet worked great for mundane or typical applications — now we're getting to time-sensitive applications and we need to have a little bit more congestion control in there.'"
Are there any foreseeable applications for the consumer world?
On second thought, let's not go to the internet. 'Tis a silly place.
Yeah, I can't wait until they rip out all of the Stallion COM port boards from our data center and put in those new-fangled switches and routers! Boy, we's gonna be uppity!
20baseU. Some hotshot near me is trying 30baseV - show off!
FTA: "But in its current state, Ethernet is not optimized to provide the service required for storage and high-performance computing traffic -- speed alone won't cut it, vendors say. Ethernet, which drops packets when traffic congestion occurs, needs to evolve into a low latency, "lossless" transport technology with congestion management and flow control features, CEE and DCE backers say."
If I understand right, they're trying to change Ethernet because of TCP/IP? Isn't that kinda, backwards as a concept?
The derivation of "ethernet" is quite interesting.
Ether - from the chemical compound Diethyl Ether, an anaesthetic.
Net - meaning 'full of holes'.
I think that's right.
http://twitter.com/onion2k
"Ethernet has conquered much of the network world and is now headed deep into the data center to handle everything from storage to LAN to high-performance computing applications."
Ethernet? Used for LAN? Jeepers, who'd ever have though of using Ethernet for THAT! I bet it'll be much faster than my 300-baud modem! And we can even connect storage devices to it!
I could have sworn that ethernet's next big step was going to be home audio. Doh.
Why did I buy this http://www.usa.denon.com/ProductDetails/3429.asp then? >
And to make it easy we could call it "TokenRing".
Or maybe a token passing bus! Maybe call it "ARCnet".
Seriously, if there are problems with Ethernet ... for the usage you envision ... don't try to change Ethernet.
You take the parts you want from Ethernet and you create a NEW standard with the other features you want.
But you leave Ethernet as Ethernet. That way there is no confusion.
This seems like a total kludge being put together by networking equipment vendors to find a way to differentiate their products and return to the days where a 10 Base-T hub was $1000.
Network gear is now mostly a commodity, except at the super high end.
The vendors hate that - so they are trying to push the host's functionality into the LAN gear instead. They don't want to provide "dumb pipes" any longer, they want to monkey around with the traffic and protocols, and provide virtual servers and such in their boxes.
Really, it's just an attempt to make the servers the commodity and their gear the expensive part.
Except... you already can implement this yourself with existing equipment and software, with much better control and no vendor lock-in. For low-end solutions, a Linux cluster works great behind an UltraMonkey front end. For higher-end ones, well, that's what IBM z-series mainframes are for.
What a great solution in search of a problem.
I think Ethernet is going to take over the data traffic for large corporates and client connections. However, there still exists applications worth multi billion $ that can absolutely not use such a horrible non-reliable protocol and will continue to use serial/profibus communications.
From the article:
"Ethernet is not optimized to provide the service required for storage and high-performance computing traffic -- speed alone won't cut it, vendors say. Ethernet, which drops packets when traffic congestion occurs, needs to evolve into a low latency, "lossless" transport technology with congestion management and flow control"
Q: Packet loss and traffic congestion are to Ethernet as:
A) blue screens are to Windows
B) registers are to assembly
C) mustard is to sausages
"Creationists make it sound as though a 'theory' is something you dreamt up after being drunk all night." -Asimov
The IT world went the cheap route and embraced ethernet. FDDI was technically superior, but required at least two brain cells to set up properly. Brain cells are optional with ethernet.
Perhaps instead of building an inverted house of cards, this consortium should re-examine where FDDI left off and pick up from there.
There's a draft of Firewire that hasn't hit yet that uses standard Ethernet cables. The port is supposed to be able to use either Firewire OR Ethernet and be able to switch between them depending on what it's plugged into. This sounds ideal to me.
The preceding post was not a Slashvertisement.
Ethernet is more of a generic name than a specific thing. It's more like "soup" than it is like "VHS".
Ethernet started as a daisy-chained garden-hose-size coax cable with vampire taps. Then RG-58 with BNC connectors, then twisted pairs to a hub, then switching hubs, then wireless... Not much stayed the same, not speed, media, topology,... except maybe carrier-sense. It's basically a comforting name, with the Ethernet-of-the-day varying at the chef's whim.
Keeping the name while tossing out the last remaining bit of commonality is a bit bizarre.
Why do we still use ethernet? Ethernet was designed to work with multiple access cables in 10B2 and 10B5 layouts with backoff algorithms and all the other stuff that goes with detecting and avoiding collisions. With 10BT all that stuff is irrelevant now - what 10 base T is effectively is a highly complex serial cable carrying just one machines data to a router or switch. All the overhead of frame encoding and decoding and the collision system should be ditched and something more appropriate to a 1 -> 1 connection used instead.
Fibre Channel over Ethernet has real promise, but these new requirements are a real stumbling block.
What many of the posters here may not realize is that storage traffic (and the "standard" SCSI it uses) is extremely intolerant of dropped frames. A link that in the TCP/IP world would be specatacular is wholly unsuited for block-level storage, which is too latency sensitive to have time to recover from dropped data.
Since the most common cause of dropped frames within a data center is congestion, FCoE requires your Ethernet to implement frame-based flow control, which prevents the congestion from occuring, along with the resulting frame loss.
SirWired
Lemme see here.. ethernet.. collision detection... vs ATM collision avoidance..
collision avoidance.. imagine that no collisions..
multiple levels of QOS.. more addressable space than IPV6.. AND intelligent, on the fly, reconfigurable routing that supports redundant paths natively.
naw.. ATM.. it's too hard..
we're gonna slap more standards on ethernet to make it do things it can't do inheirently and then slap more stuff on top like mpls and 802.1Q and stuff to make it like atm.. but easy. ya..
laymen are morons.
Ethernet already has flow control at the link-level - they're called stop frames (and since all modern switches give you dedicated links to end workstations and have some amount of hardware buffering, collisions/overrun aren't an issue). Now, since the world really runs on IP (doing raw ethernet would only ever work in the most local of LAN applications which is rather pointless in most deployments), and IP has TOS bits (which every real modern router can classify, queue, and throttle per-queue all in the hardware fast-path with no additional latency), I'm failing to see what they're proposing to solve since the problem is already solved. 1G/10G switches are used all over data centers and in HPC situations today (and have been for years)...
Nope, the CSMA/CA part of Ethernet is gone also. Specs for a GigE hub exist in the standards, but nobody ever implemented them. (Switching got to be too cheap for anybody to bother.)
Obviously it didn't even get specc'd out with 10Gb Ethernet.
Oh, the frame format is still more-or-less the same from Classic Ethernet. Not identical, but still pretty close.
SirWired
Oh golly - the "next generation(tm)", far beyond 10BASE-T!!!
Who knows, maybe they will reach something so fast they could call it 100BASE-T, or even 1000BASE-T - only the future will tell, and the future clearly belongs to this upcoming "next generation technology".
They want a system that SHARES the available resource (bandwidth in this case) ... but that allows each machine on it to behave as if it had EXCLUSIVE access to that resource.
And they want to base that system off of a technology that evolved from a concept based upon COLLISIONS.
And the reason they want to do that is ... because that old technology is ubiquitous.
Not because it is well suited to their needs. Not because it is inexpensive to modify. Not for any good technological reason.
But because everyone already has it.
So they want to make networks more expensive for EVERYONE so that they THEY can sell their products for less.
Fuck that.
You can have Ethernet Classic and New Ethernet!
AND...
If New Ethernet is a dud, you can say it was a marketing gimmick to get people to drink... err, switch back to Ethernet Classic!!!
Given the direction SATA & USB is going, the rate at which its bandwidth has increased relative to traditional CATx ethernet, and the relatively lower cost of interconnection devices, is Ethernet really the best? If we're going to making significant wiring changes in server rooms I'd prefer to just do it once and standardize on the cheapest, fastest "2-wire" solution that makes sense.
Since nobody buys anything not labeled ethernet it's going to be called ethernet anyway. Maybe ethernet+ or ethernet ring or some BS marketing term.
The technology changes have been substantial since Xerox was pumping 3Mbit/s through a coaxial cable but we continue to call it ethernet because PHB's don't want to bother implementing a new networking standard.
Platform advocacy is like choosing a favorite severely developmentally disabled child.
My point wasn't based on Infiniband or Myrinet, or any other interconnect. It just seems to me that all of this is a solution desperately searching for a problem.
Different technologies exist for a reason: they do what they are designed for as optimally as possible. Trying to make one network technology that does "Everything" is doomed to epic failure. It will become a compromise that does lots of things, and few of them well.
I work as a Storage Architect at a large telco. For moving massive quantities of data from point-a to point-b SAN's still rule the day for us and I don't see that changing anytime soon.
Ethernet has several major deficiencies that make it less attractive for being the dump truck of data movement. I'm writing this while thinking of the large enterprise Backup and Recovery environments out there but there are other applications that involve moving massive (think 100+ terabytes nightly) amounts of data around that SAN's are still better suited for.
First of all SAN's by inherent design have the ability to aggregate data across multiple ISL's (trunks) in real-time. If you have 2 pipes between switches your I/O's will be evenly distributed across the links adjusting in real time as needed to fully utilize both links. Need more bandwith? Simply plug in another ISL, done.
Ethernet routing isn't quite as intelligent. Being that data transfers are session based, you can have a completely flooded trunk with the other sitting there idle for endless hours.
While true the next session that starts may choose the 2nd under-utilized path, your pretty much SOL on performance with the first one if it becomes saturated.
This isn't quite as painful if your data transfers involve a vast number of relatively "quick" transactions such as FTP's but with NFS/CIFS mounts these may not be re-evaluated for days, weeks or months. So once a route has been picked you are essentially stuck with that routing decision across your trunk until the session has been re-established.
SAN's were designed for large scale data movement and high IOPs from the beginning. Ethernet on the other hand needs quite a bit of tweaking and still comes up short for some large enterprise applications.
More like $30 for the cable that is needed.
What are ethernet's two biggest strengths? Think about it a moment.
It is high speed at a very low cost, and the ubiquitousness of the technology.
It doesn't really matter that the technology is inferior to the numerous very fast low latency solutions available today because all of those solutions are also very high-cost, low-volume solutions relative to ethernet, and have no ability to be incrementally upgraded. It doesn't matter that there are congestion issues when pushing a packet-switched technology to its limit, because anyone who has lived with this stuff for the last few decades knows that you simply overbuild it for your needs and all those problems disappear. It doesn't matter that the price point is GiGE because the technology has proven over and over again that the price point moves very quickly with adoption. The price point will soon be 10GiGE, and after that 100GiGE. It is unarguable. It doesn't matter that ethernet traditionally has higher latencies then currently existing higher-cost solutions. Latency has never been a show stopper for anyone except the super-computing dweebs.
The price of hardware has dropped around the ears of the higher cost solutions to the point where only a very small application space can actually gain an advantage using them. It will only get worse over time.
So even though, GiGE isn't quite fast enough to match platter rates on the best disks, let alone transfer rates available from SSDs, it doesn't matter. The low cost and clear future trends for the technology will trump everything. And the people making the decisions have the choice: They can be early adopters of 100GiGE, or choose to spend more of a premium on 10GiGE, but with the flexibility of NOT having to rip up their entire infrastructure. The ability to piecemeal-replace infrastructure is its own trump card.
-Matt
And I'll tell you why.
Collisions became a non-issue the moment low-cost 10BaseT switches came into existance, which was, what, over 15 years ago? Don't bother arguing about collisions, not even my home network uses hubs any more. There are no collisions.
Congestion is a lot easier to handle then people seem to realize. All modern switches have packet buffers. Even the CHEAP switches have a few megabytes of buffer. GiGE switches have flow control on top of that.
Buffering and flow control does NOT mean that you can just squirt out packets and let things back up. What it does mean is that a large chunk of the problem space has already been solved. If you use the traditional over-build methodology and gang the links in your trunks you might not even have to worry about congestion at all.
For those situations where you do have to worry about congestion you already have multiple feedback mechanisms available to you to control throttling. Even the simplest algorithms can make a huge difference, and that is the key.
There IS a need for the more complex forms of congestion control. But the simpler forms solve 90% of the problem space. In otherwords, there is an incremental solution space to the general problem and the effect of that incremental solution space will not only be ubiquitous network protocols but also very low costs and the ability to incrementally upgrade your infrastructure as you need it rather then have to spend millions up front for an infrastructure to handle your expected needs for the next X years.
Let me give you an example of a simple form of congestion control for a SAN device. Lets say the device can handle 256 simultanious requests and implementations a reservation protocol for those 256 slots. A simple protocol would be for the host wanting to use the device to request N slots. The device assigns it, say, 3 slots. The host then uses those 3 slots to queue up to 3 simultanious requests, and reuses those slots as long as it needs to or until the device tells it use fewer. Lets say any given request cannot have a payload larger then 8K, just for argument's sake.
Consider the consequences of such a mechanism. For one thing you have immediately put a control on the maximum amount of data that can be backed up in intermediate switches for that device. 3 requests, 8K ... approximately 24K. That is a form of congestion control. For another the device has a limited number of slots for parallel operations (256). That is another form of congestion control. The device and/or host can revoke slots. That is another form of congestion control. The device can order which requests it responds to first. That is another form of congestion control.
Starting to get the idea? There are literally an INFINITE number of ways to control congestion, none of them require anything sophisticated at the physical layer. Even without smart switches you have hundreds of software solutions, all easily tunable.
In otherwords the problem is not only solvable, but even partial solutions will give system operators and network managers plenty of knobs they can tune to make their infrastructure work.
-Matt
640Kfloppies/sec ought to be enough for anybody.
So, basically, what we're looking at is the manufacturer's ability to add hardware DRM to every component of our PC. We need to fight this tooth and nail, folks, or we can say goodbye to all our "illegal" format shifting and document sharing.
Before flaming me for being apparently anti-P2P, please note the quotes around "illegal" in the above statement, and take them for the sarcasm they are.
This has been a public service announcement.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
Bandwidth reservation?
Sounds more and more like an end-run on network neutrality.
I'll stick with your wording, I think it suits my viewpoint best.
Fuck that.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
For "local" devices, it seems to make more sense to use something like AoE vs. iSCSI
From what I can see on SCTP (not too familiar with it... I'm a hard-core FC guy, not a TCP/IP guy), it uses an end-to-end flow control mechanism.
That imposes too much latency of FC/FCP (virtually all FC flow control is done on a port-to-port basis, not end-to-end) and it looks like it would impose substantial overhead and complexity on FC/FCP itself, which would have to be modified to accomodate it. FCoE is a drop-in replacement for layers 1 and 2 of the FC stack, and requires very little in modifications.
I also do not see a way to easily bridge between the FC and SCTP. One of the major strengths of FCoE is the relative simplicity of bridging between the two protocols.
SirWired
From what I can read on RED, it performs congestion control via pro-active frame drop. This won't work with anything vaguely resembling current SCSI stacks.
To put it simply, the acceptable frame loss rate for a production storage network is approx. zero, and the current driver stacks are written with that expectation in mind. For instance, if you lose the SCSI status frame, the box waiting for it must wait at least two seconds, and usually longer, for that frame to arrive before giving up on it. This is simply not something you want to occur very often when you are waiting for, say, a latency-sensitive DB transaction to complete.
The biggest argument for FCoE is the fact that it maps easily on top of existing FC SANs, adapters, and driver stacks, with few modifications. Once you have modified it for those IP extensions and mapping it onto an IP transport (SCTP, TCP, whatever), you end up with something very much like iSCSI, which, for whatever reason, has yet to really take off in the higher-end space. (I think FCoE will be an iSCSI jumping-off point...)
SirWired
It's being phased out on consumer gear. I'll miss it too. Hopefully USB3 will be as dreamy as they say.
Nope, both of which have higher overhead than full-duplex ethernet. They have better throughput than half-duplex ethernet, which is about as useful as being better than avian carriers. Half-duplex ethernet should just be banned entirely. Maybe that would make Linksys wake up.
Half-duplex ethernet corresponds to the way things work on a shared peer-to-peer radio channel. Like WiFi. (Which uses the Ethernet MAC and collision/backoff algorithms - though I think the collision detection is inferred rather than explicit.) (WiMAX, however, is a full-duplex protocol with central stations monopolizing an outbound channel and assigning timeslots for replies from remote stations on an inbound channel.)
Of course that DOES qualify as an "avian carrier", doesn't it?
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
"... Ethernet, which drops packets when traffic congestion occurs, needs to evolve into a low latency, "lossless" transport technology with congestion management and flow control features, ..."
If I understand right, they're trying to change Ethernet because of TCP/IP?
Nope.
They're trying to build a mechanism for bandwidth and latency guarantees into the protocols. That's what you need for reliable streams, efficient delivery of networked storage data, etc. Ethernet doesn't provide such a mechanism: It's first-come-first-served at switches, though routers (and some smart switches) can pick and chose (and both can backpressure with PAUSE frames.)
You could build the mechanism on top of ethernet transport using IP's Quality of Service mechanisms - and that would be the preferred way to do it. But doing so requires treating some packets better than others. And the "network neutrality" push is throwing that baby out with the bathwater.
So it looks like the vendors are trying to build those mechanisms into the underlying protocols - sidestepping the issue (and redefining things so it could also be used to implement backbone packet preferences later without making a non-"neutral" network a violation of the definition of the network protocols).
= = = =
Another change coming down the pike, by the way: One thing Ethernet DOESN'T do now, but COULD, is distribute network timing by synchronizing the carrier to a network clock at a transmitter and recovering it at the receiver. This is the last missing piece for converging the old TDM networks (T1/E1/SONET/SDH/POTS/ADM/...) onto IP-over-Ethernet.
The new thing is called "Synchronous Ethernet" and telecom equipment suppliers (notably Ericsson) are pushing its standardization. (IEEE 1588 tries to do this with messages but is a couple orders of magnitude less accurate and a lot more expensive. For some equipment vendors doing Synchronous Ethernet is a minor tweak to line cards plus a tiny bit of software, and it's as accurate as the decades-proven TDM stuff it replaces.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
"low latency lossless". Geez, like in ATM or SONET??? (or add your favorite GBW fieldbus). Once again, 802.3 is stealing all the good ideas from 802.6 and POTS
Mind you, there's an interesting challenge - implementing distributed queueing over a tree. Hmm, time to look at Dr. Robert M. Newman's papers and patents again.
Then we get to go through VLAN vs RSTP hell again while everyone interprets the standard differently.
Actually a real problem with Wintel based TCP is that you can't set the TCP parameters to values suitable for low latency networks. F'example, if I'm doing transactions several times a second and timing them out in 1/10th, TCP never gets a look in on a windows box. So Windows TCP/IP over ethernet "doesn't work reliably"
-- Butlerian Jihad NOW!
You're taking me back now. I started on "thin" ethernet, the RG-58 version, then later had to adapt to "thick" ethernet, with external tranceivers and cables that had to be, for some reason, exactly some multiple of a meter long. Thick like a garden hose, but less flexible. Fast though -- theoretical maximum of 1 MB/s. You wouldn't get that in practice. All daisy-chained together, with terminators at two ends.
Those were the days, but that performance came at a price -- as I recall about $750 per host.
The subject who is truly loyal to the Chief Magistrate will neither advise nor submit to arbitrary measures (Junius)
The bad thing about the usage of this "new Ethernet" is not so much in the new link level congestion control (presumably for the sake of SCSI) but in "bypassing TCP/IP" - and with it all it's services (naming, routing, service discovery, zero-config, security etc.) and recreating them over Ethernet in proprietary form (at least for now) and not have them widely available. It is also doubtful that the new schemes are as the well thought through and "scale in time and network size" as the layered protocols we have in place now i.e., if they can survive transition to faster and larger network or they are a palliative for a perceived weakness of TCP/IP and a way to extend the life of the highly profitable market for Fiber Channel.