Slashdot Mirror


Corporate Data Centers As Ethernet's Next Frontier

alphadogg writes with a story that's about the possibilities for the next generation(s) of Ethernet, stuff far beyond 10base-T: "Ethernet has conquered much of the network world and is now headed deep into the data center to handle everything from storage to LAN to high-performance computing applications. Cisco, IBM and other big names are behind standards efforts, and while there is some dispute over exactly what to call this technology, vendors seem to be moving ahead with it, and it's already showing up in pre-standard products. 'I don't see any show-stoppers here — it's just time,' says one network equipment-maker rep. 'This is just another evolutionary step. Ethernet worked great for mundane or typical applications — now we're getting to time-sensitive applications and we need to have a little bit more congestion control in there.'"

41 of 152 comments (clear)

  1. what am I missing with this article? by poetmatt · · Score: 2, Insightful

    FTA: "But in its current state, Ethernet is not optimized to provide the service required for storage and high-performance computing traffic -- speed alone won't cut it, vendors say. Ethernet, which drops packets when traffic congestion occurs, needs to evolve into a low latency, "lossless" transport technology with congestion management and flow control features, CEE and DCE backers say."

    If I understand right, they're trying to change Ethernet because of TCP/IP? Isn't that kinda, backwards as a concept?

    1. Re:what am I missing with this article? by LWATCDR · · Score: 5, Informative

      No.
      Ethernet uses collision detection and resending to to manage packets.
      Well it used to anyway. I am not sure about Giga-E
      The way Ethernet used to work is that a sender would listen to see if the line was clear and then send a packet and listen at the same time. If the packet was damaged by a collision the sender would wait a random amount of time and then try to resend.
      This system really bugged a lot of people but it was cheap and it worked.
      This is the actually physical layer and not TCP/IP.

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    2. Re:what am I missing with this article? by afidel · · Score: 4, Interesting

      No, they want Ethernet as a transport to contain a lot of the features of TCP so that they can lay their own protocols on top of it while being able to assume it's a reliable transport. That will increase the cost of ethernet by including that intelligence down the stack. Basically the cost of ethernet ports is plummeting compared to things like fiberchannel due to economies of scale and so cash strapped datacenters are trying to use it for everything, but it's not ideally designed to handle those other protocols so the other technology areas are trying to mold ethernet to meet their needs. Basically the way I see it if the industry does what is right there will be no 100Gbit Fiberchannel, there will be 100Gbit FCOE adapters.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    3. Re:what am I missing with this article? by Paralizer · · Score: 2, Informative

      Yes, that is the case strictly at layer 1 of the OSI model. However at layer 2 we have the switch. By segmenting the collision domain up and creating one for each port rather than the entire unit we no longer have collisions and CSMA/CD is no longer needed. Unfortunately wireless still uses CSMA/CA which operates similar to what you described, except it requests silence of the 'wire' first before trying to send rather than retransmitting when a collision occurs. Switches are still part of ethernet since they operate at layer 2. TCP/IP doesn't come into play until layer 3 when we get TCP/IP IP addresses.

      Ethernet itself is not reliable, which is why we use TCP in TCP/IP as the transport protocol so we know if we need to retransmit due to undelivered packets. I can't imagine how they would go about 'fixing' ethernet though, as the GP pointed out. If you pass something along between a series of switches/routers/nodes there must be the possibility something fubars and you lose that information in transit - be it a noise on the wire or maybe the node runs out of memory and can not process it.

    4. Re:what am I missing with this article? by mikael · · Score: 2, Interesting

      For a high-performance system with a large number of nodes, the cost of the actual network to connect everything together can cost more than the CPU's and servers themselves. To get high performance from this network, everything has to be tied so tightly together, that is is considered a component in itself, the network fabric. Also, the actual communication through the network cables is the slowest part of the system. So this price/performance ratio is what customers will be considering when buying a system.

      The vendors want to keep the cheap network hardware (cables, connectors, switches) because the consumer market has driven down costs down to commodity prices. But Ethernet uses the cheapest method of shared communication - packet collision detection ("keep shouting until someone hears you"). I've read some research papers which say they can get up to 90% efficiency now.

      High performance network architectures (FDDI, token ring) are a bit more civilized - they had a token that was passed around - only the machine with the token could send any data.
      So there was never any lost packets. Other methods give each pair of devices a unique time-slot on a multi-slice basis. Or there are crossbar switch architectures like telephone exchanges that allow multiple connections to exist at the same time.

      So the vendors want it both ways - the cheap commodity prices of Ethernet hardware, combined with the high efficiency of existing high-end network hardware.

      The changes that they want only really affects the Link layer of TCP/IP, where collision detection is currently being performed instead of token passing or sychnronised time-slicing.

      --
      Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
    5. Re:what am I missing with this article? by amorsen · · Score: 2, Informative

      So technically you could argue that there was not really a "collision" but the computer that didn't get its packet through is still told there was one so that it retransmits.

      No. Switches don't notify the source that the packet was dropped. TCP's retransmit works without explicit notification.

      Collisions are a thing of the past. Dropped packets aren't.

      --
      Finally! A year of moderation! Ready for 2019?
    6. Re:what am I missing with this article? by erc · · Score: 3, Informative

      However, with a switch at least one packet always gets through.

      Wrong. There is no "collision detection", the only way to tell that you had a collision is if the packet didn't get there. If two devices transmit at the same time, you get a mangled packet that won't pass CRC and gets dropped.

      What they're really looking for is token ring - which was (and still is) a superior protocol - for Ethernet, as you increase the bandwidth utilization beyond 10%, you get so many collisions that your throughput goes through the floor, while for token ring, the throughput degradation is much more gradual. For bandwidth utilization above 10%, token ring is far superior to Ethernet.

      Why Ethernet was adopted over token ring has more to do with religious issues and who can scream the loudest and bully their way through technical issues with emotion than it has to do with technical superiority.

      --
      -- Ed Carp, N7EKG erc@pobox.com PGP KeyID: 0x0BD32C9B What I'm up to: http://intuitives.mine.nu
    7. Re:what am I missing with this article? by amorsen · · Score: 3, Interesting

      t is possible to have IP on some other network, like token ring or FDDI, bother of which actually achieves higher throughput than ethernet for a given bandwidth.

      Nope, both of which have higher overhead than full-duplex ethernet. They have better throughput than half-duplex ethernet, which is about as useful as being better than avian carriers. Half-duplex ethernet should just be banned entirely. Maybe that would make Linksys wake up.

      --
      Finally! A year of moderation! Ready for 2019?
    8. Re:what am I missing with this article? by LWATCDR · · Score: 2, Informative

      I think it had a lot more to do with cost.
      Ethernet was available first and had more hardware suppliers so the cost went down.
      Token ring was really popular with IBM. It was almost a standard for IBM systems. I have a few microchanel Token Ring adapters if you need them :)
      FDDI is Token Ring on fiber.

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    9. Re:what am I missing with this article? by afidel · · Score: 2, Interesting

      Since 10Gbit Ethernet has the collision domain defined as the two endpoints there IS no longer a collision domain on the wire, just a virtual one in an oversubscribed switch. This isn't about guaranteeing transmission over the internet, it's about guaranteeing reliability in a LAN/MAN/WAN Ethernet network. The idea is you will have one set of wires, one physical protocol with several personalities sitting on top. The biggies are TCP/IP and FCOE but there are other things like remote DMA that can greatly benefit from a reliable network layer transport.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    10. Re:what am I missing with this article? by erc · · Score: 2, Informative

      Haha, thanks for the offer but I don't have anything that will take MCA boards ... that was an interesting attempt by IBM to retake the PC market they lost. Oh, well.

      As to Ethernet being first, I thought it was the other way around?

      --
      -- Ed Carp, N7EKG erc@pobox.com PGP KeyID: 0x0BD32C9B What I'm up to: http://intuitives.mine.nu
    11. Re:what am I missing with this article? by afidel · · Score: 2, Informative

      Huh, in Ethernet which is CSMA/CD you listen to the wire before starting to transmit, this doesn't avoid all possible garbled packets but it does avoid most if things are working to spec. Also because VTT goes higher than normal transmit levels during a collision there IS detection. The reason that ethernet won over tokenring is that IBM charged a hefty fee per port for tokenring. There was also real world reliability problems with tokenring as early designs used a physical ring or string layout instead of the star topology of ethernet (later tokenring switches did allow for physical star topology but this was well past the point where ethernet had won the mass market).

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    12. Re:what am I missing with this article? by vux984 · · Score: 3, Informative

      Collisions occur when there are more than one sender on a collision domain, they don't have to be sending to the same host. Imagine you have four computers on a hub. Computer A sends a message to B while C simultaneously sends a message to D -- this is a collision.

      We are really just talking about how collisions occur on a switch. Technically, they CAN'T occur on a full duplex switched network. The collision domain is the switch port and the PC port, and both can talk at once (full duplex).

      Hypothetically though, if you set aside buffering, a 'collision like' conflict occurs when multiple PCs try to talk to a single port, except that one gets through and the rest are 'blocked' which is what I was trying to say. Of course, due to buffering, this is 'handled' and the conflict is actually pushed back to when the buffer overflows instead.

      And yes, switches do have outbound buffers for each port so that if two sources try to send to the same host they can be done in sequence rather than causing an outbound collision on the destination port's collision domain. I am not sure what happens if this buffer becomes full, I had always assumed the switch would just begin dropping the packets (as indicated by this Cisco document).

      http://www.webopedia.com/TERM/B/backpressure.html

      Dropping packets is one option. The other is to use 'back pressure' to signal to the PC to back off a bit. This can be done by sending 'fake collisions' or via 802.3x Flow Control 'pause' signals. Many switches support these modes including those from intel and cisco.

      Its often better to just dropping the packets and let tcp deal with it, but in some cases you can get better performance by using flow control/back pressure features.

  2. Welcome to 1980! by snarfies · · Score: 4, Funny

    "Ethernet has conquered much of the network world and is now headed deep into the data center to handle everything from storage to LAN to high-performance computing applications."

    Ethernet? Used for LAN? Jeepers, who'd ever have though of using Ethernet for THAT! I bet it'll be much faster than my 300-baud modem! And we can even connect storage devices to it!

  3. We could add a "token" and make it a "ring"! by khasim · · Score: 4, Interesting

    And to make it easy we could call it "TokenRing".

    Or maybe a token passing bus! Maybe call it "ARCnet".

    Seriously, if there are problems with Ethernet ... for the usage you envision ... don't try to change Ethernet.

    You take the parts you want from Ethernet and you create a NEW standard with the other features you want.

    But you leave Ethernet as Ethernet. That way there is no confusion.

    1. Re:We could add a "token" and make it a "ring"! by Legion_SB · · Score: 4, Funny

      I agree. I don't want any of this "doesn't drop packets" Ethernet either. Packet loss is critical to a number of my in-house applications.

      --
      'a';DROP TABLE users; SELECT * FROM DATA WHERE name LIKE '%'... if you're reading this, it didn't work.
    2. Re:We could add a "token" and make it a "ring"! by Bill,+Shooter+of+Bul · · Score: 2, Insightful

      Something new won't sell. People wont adopt revolutionary products as easily as they will adopt incremental upgrades with a known and trusted brand. So calling it "Uber-fiber hyper gylde" won't sell as well as "Ethernet v10".

      People will deal with confusion. They deal with it all the time. Its the only way they know to deal with the walrus.

      --
      Well.. maybe. Or Maybe not. But Definitely not sort of.
    3. Re:We could add a "token" and make it a "ring"! by R2.0 · · Score: 4, Funny

      "Uber-fiber hyper gylde"

      Combination personal lubricant and laxative?

      --
      "As God is my witness, I thought turkeys could fly." A. Carlson
  4. Network Vendors by maz2331 · · Score: 3, Interesting

    This seems like a total kludge being put together by networking equipment vendors to find a way to differentiate their products and return to the days where a 10 Base-T hub was $1000.

    Network gear is now mostly a commodity, except at the super high end.

    The vendors hate that - so they are trying to push the host's functionality into the LAN gear instead. They don't want to provide "dumb pipes" any longer, they want to monkey around with the traffic and protocols, and provide virtual servers and such in their boxes.

    Really, it's just an attempt to make the servers the commodity and their gear the expensive part.

    Except... you already can implement this yourself with existing equipment and software, with much better control and no vendor lock-in. For low-end solutions, a Linux cluster works great behind an UltraMonkey front end. For higher-end ones, well, that's what IBM z-series mainframes are for.

    What a great solution in search of a problem.

  5. Quiz Time by inaneframe · · Score: 2, Funny

    From the article:
    "Ethernet is not optimized to provide the service required for storage and high-performance computing traffic -- speed alone won't cut it, vendors say. Ethernet, which drops packets when traffic congestion occurs, needs to evolve into a low latency, "lossless" transport technology with congestion management and flow control"

    Q: Packet loss and traffic congestion are to Ethernet as:
    A) blue screens are to Windows
    B) registers are to assembly
    C) mustard is to sausages

    --
    "Creationists make it sound as though a 'theory' is something you dreamt up after being drunk all night." -Asimov
  6. Combo Firewire/Ethernet port by pecosdave · · Score: 5, Interesting

    There's a draft of Firewire that hasn't hit yet that uses standard Ethernet cables. The port is supposed to be able to use either Firewire OR Ethernet and be able to switch between them depending on what it's plugged into. This sounds ideal to me.

    --
    The preceding post was not a Slashvertisement.
    1. Re:Combo Firewire/Ethernet port by pecosdave · · Score: 2, Informative

      If anyone is interested, there's info here on that.

      --
      The preceding post was not a Slashvertisement.
  7. Think of "Ethernet" as "Soup" by Ancient_Hacker · · Score: 4, Informative

    Ethernet is more of a generic name than a specific thing. It's more like "soup" than it is like "VHS".

    Ethernet started as a daisy-chained garden-hose-size coax cable with vampire taps. Then RG-58 with BNC connectors, then twisted pairs to a hub, then switching hubs, then wireless... Not much stayed the same, not speed, media, topology,... except maybe carrier-sense. It's basically a comforting name, with the Ethernet-of-the-day varying at the chef's whim.

    Keeping the name while tossing out the last remaining bit of commonality is a bit bizarre.

  8. SAN over Ethernet has real promise, but... by sirwired · · Score: 4, Interesting

    Fibre Channel over Ethernet has real promise, but these new requirements are a real stumbling block.

    What many of the posters here may not realize is that storage traffic (and the "standard" SCSI it uses) is extremely intolerant of dropped frames. A link that in the TCP/IP world would be specatacular is wholly unsuited for block-level storage, which is too latency sensitive to have time to recover from dropped data.

    Since the most common cause of dropped frames within a data center is congestion, FCoE requires your Ethernet to implement frame-based flow control, which prevents the congestion from occuring, along with the resulting frame loss.

    SirWired

    1. Re:SAN over Ethernet has real promise, but... by ToasterMonkey · · Score: 2, Interesting

      Fibre Channel over Ethernet has real promise, but these new requirements are a real stumbling block.

      Something to note is that the Ethernet in FCoE is really not the same Ethernet we use today. The acronym really confuses things. The article offers some better names for the new Ethernet standard, "Converged Enhanced Ethernet (CEE)", "Data Center Ethernet (DCE)." It really is the convergence of Fibre Channel and Ethernet, NOT Fibre Channel glued to the back of Ether. Think of it more like a gigantic leap for Ethernet (and IP/TCP eventually, as functionality is pushed down a few layers), not so much a downgrade of Fibre. Also, this mostly applies to 10Gig Ether, which is already pretty damned different from previous forms of Ethernet.

      These are the new Ethernet standards.

      I think it's necessary to explain all this because while most posters don't know dick about storage and think existing Ethernet is good enough for everything, a good number of them might also be SAN admins that shrug it off without knowing that the "Ethernet" in the acronym has changed. HBA's aren't going anywhere, they will just be running more IP traffic now =) Also, if iSCSI is still around (gag me with a spoon if it is) it will at least have a better foundation to stand on. Damn I really hope FCoE ends its misery though.

  9. This is FUD... by volxdragon · · Score: 4, Interesting

    Ethernet already has flow control at the link-level - they're called stop frames (and since all modern switches give you dedicated links to end workstations and have some amount of hardware buffering, collisions/overrun aren't an issue). Now, since the world really runs on IP (doing raw ethernet would only ever work in the most local of LAN applications which is rather pointless in most deployments), and IP has TOS bits (which every real modern router can classify, queue, and throttle per-queue all in the hardware fast-path with no additional latency), I'm failing to see what they're proposing to solve since the problem is already solved. 1G/10G switches are used all over data centers and in HPC situations today (and have been for years)...

  10. Nope, Carrier Sense is gone too... by sirwired · · Score: 3, Informative

    Nope, the CSMA/CA part of Ethernet is gone also. Specs for a GigE hub exist in the standards, but nobody ever implemented them. (Switching got to be too cheap for anybody to bother.)

    Obviously it didn't even get specc'd out with 10Gb Ethernet.

    Oh, the frame format is still more-or-less the same from Classic Ethernet. Not identical, but still pretty close.

    SirWired

  11. Re:Hmm... by pla · · Score: 3, Interesting

    Are there any foreseeable applications for the consumer world?

    Connect your new keyboard and mouse via ethernet.
    Connect your new HDD* via ethernet.
    Connect your new video card via ethernet.
    Connect your new scanner via ethernet.
    Connect your new CD/DVD/BR via ethernet.
    Connect your new printer* via ethernet.
    Connect your new webcam* via ethernet.

    No more USB cables with a million different connector types. No more PATA or SATA cables. No more serial or parallel cables. No more trying to figure out where to plug a given device in on a motherboard or looking for spare PCI/whatever slots - Just one type of cable and they all plug into a switch-like section of the motherboard.

    Now, some devices (video cards as the most obvious) will still require extra power, but most devices could probably manage with a variant on PoE, meaning the inside of your case goes from rats-nets of assorted cable types, to a half-dozen or so tidy round cables.

    * Yes, you can already get network enabled versions of these, but they count as a real full-fledged network endpoint, not as a slave device "local" to a particular computer.

  12. Re:Hmm... by Yetihehe · · Score: 5, Insightful

    No more USB cables with a million different connector types.

    You realise "no more different connector types" was the reasoning with USB?

    --
    Extreme Programming - Redundant Array of Inexpensive Developers
  13. USB by psbrogna · · Score: 3, Informative

    Given the direction SATA & USB is going, the rate at which its bandwidth has increased relative to traditional CATx ethernet, and the relatively lower cost of interconnection devices, is Ethernet really the best? If we're going to making significant wiring changes in server rooms I'd prefer to just do it once and standardize on the cheapest, fastest "2-wire" solution that makes sense.

  14. Re:Hmm... by Grey_14 · · Score: 4, Insightful

    And sadly, you'd see the same issues it with this standard too, because an ethernet RJ-45 plug isn't appropriate to plug into a cell phone, digital camera or mp3 player, but a 5-pin mini-connector isn't appropriate to run 25 feet to a switch/router either.

  15. Re:So Ethernet's Next Frontier isn't Audio? by jsalbre · · Score: 3, Funny

    Ohh.... That hurts my brain on so many levels. "...directional markings are provided for optimum signal transfer." Wouldn't want the electrons to forget which way they were supposed to go, eh?

  16. Re:Hmm... by Shatrat · · Score: 2, Informative

    I think the biggest reason that there are hundreds of different USB connectors is that standardized plugs don't help sell 30 dollar Apple or Sony branded AC/DC adapters.
    I really loved my old motorola phone with the mini-usb connector, now my LG phone doesn't share it's connector with any other device I own.

    --
    09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
  17. Re:Hmm... by TheRaven64 · · Score: 2, Informative

    * Yes, you can already get network enabled versions of these, but they count as a real full-fledged network endpoint, not as a slave device "local" to a particular computer.

    You can with protocols like iSCSI or ATAoE. A lot of enterprise gear uses iSCSI, which makes a remote device appear like a local SCSI device, and some consumer-grade NAS devices running Linux can act as ATAoE devices, which does the same thing but with ATA instead of SCSI and over raw Ethernet frames rather than over IP.

    --
    I am TheRaven on Soylent News
  18. Re:The core concept is ... problematic. by TheRaven64 · · Score: 2, Insightful

    It sounds like they just want bandwidth reservation and isochronous transfers on Ethernet. Something that would establish a virtual circuit and then not drop frames. Something with an Asynchronous Transfer Mode, perhaps?

    --
    I am TheRaven on Soylent News
  19. Re:Umm by marcosdumay · · Score: 2, Insightful

    You probably don't have a storage area network, running over some proprietary fiber protocol, or some hight performance proprietary cluster, or a supercomputer around, do you? All those things are fading out as Ethernet evolves to do those kinds of jobs, but they didn't disapear yet.

  20. Re:Hmm... by mollymoo · · Score: 3, Interesting

    Why is having the tab on the bottom better?

    --
    Chernobyl 'not a wildlife haven' - BBC News
  21. Re:Hmm... by mollymoo · · Score: 2, Insightful

    Quite a few of the non-standard USB leads use non-standard connectors because they're actually USB-serial converters, not just leads. My previous phone, a Sony Ericsson, was like that.

    --
    Chernobyl 'not a wildlife haven' - BBC News
  22. Re:Hmm... by atomic+brainslide · · Score: 3, Insightful

    Ethernet has nothing to do with the connector type. It is a layer 2 protocol that sits on top of the physical transport medium. There is a little bit of overlap with things like wiring specs for distances and attenuation, but it ethernet itself doesn't really care what plugs or wires you use. even if connectors were in the spect, it would still likely be extended to allow for new connector types to fit the appropriate devices (mobile phones, mp3 players, etc).

    thus, for the consumer world you probably wouldn't see much difference on the user end. developers, on the other hand, would have to start pushing their device drivers into the network stack in order to get them working. say hello to firewalls and IDS/IPS on your HDD and video card.

    --
    check out my comic: Essential Tremors
  23. Half duplex corresponds to wireless. by Ungrounded+Lightning · · Score: 2, Informative

    Nope, both of which have higher overhead than full-duplex ethernet. They have better throughput than half-duplex ethernet, which is about as useful as being better than avian carriers. Half-duplex ethernet should just be banned entirely. Maybe that would make Linksys wake up.

    Half-duplex ethernet corresponds to the way things work on a shared peer-to-peer radio channel. Like WiFi. (Which uses the Ethernet MAC and collision/backoff algorithms - though I think the collision detection is inferred rather than explicit.) (WiMAX, however, is a full-duplex protocol with central stations monopolizing an outbound channel and assigning timeslots for replies from remote stations on an inbound channel.)

    Of course that DOES qualify as an "avian carrier", doesn't it?

    --
    Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
  24. Re:Hmm... by killmofasta · · Score: 2, Interesting

    Actually Not at all.
    The design of RJacks specifically RJ-45, try and mitgate this as much as possible.

    If its properly crimped, wiggling the cable, even pulling it should have no effect on either the cable contact or the jack contact.

    i.e. it has no importance in real life by design.