Fibre Channel Over Ethernet: From Fee To Free

alphadogg writes "With demand for Fiber Channel over Ethernet (FCoE) more sluggish than vendors had hoped, 10 Gigabit Ethernet switch and adapter makers are making it available for free. FCoE is a standard driven largely by Cisco to converge customers' data center LAN and storage fabrics with 10G Ethernet. Industry heavyweights Intel and Brocade are among those now giving away FCoE capabilities. There are several factors prompting vendors to slash FCoE prices or stop charging for it altogether, including market indifference; technological immaturity; competing alternatives, such as virtualized Fibre Channel and Ethernet I/O; the recession; and vendors looking to drive switch volumes. 'When FCoE first came out there used to be a fairly large price premium,' says Alan Weckel, director of Dell'Oro Group. 'Cisco had to give it away for free to drive switch volumes. Users were not adopting as rapidly as thought or that Cisco had hoped for.'"

Comments

CSCO forward guidance is low

[Anonymous Coward]

Their stock was killed last week ... down below $20. I know, I own some. They are just trying to generate business, kind of like those 'tards who are pushing 3D TV.

Too late

[Anonymous Coward]

As network fabric bandwidth continued to increase and latency decrease, FCoE appeared to be a last ditch effort to plug the steady trickle of customers from the highly expensive FC over to the much cheaper to deploy iSCSI. I'm sure the thinking was that by making it routable and with the same semantics as existing FC installs, it could accomplish that task. However, I'm also thinking that in most situations, where there's little to distinguish between iSCSI and FCoE other than the now almost commonplace on-NIC hardware iSCSI acceleration, it's a case of too late.

Re:Too late

[Junta]

Actually, FC isn't routable. FC over *ethernet* has no ip and no provisions to span gateways. The *theory* is that FCoE has fewer layers allowing for higher performance, but it's rare for that difference to be realized in cheap ethernet fabrics (the whole point of FCo*E*) and even rarer to matter relative to storage device performance limitations. iSCSI is much easier to manage with fewer limitations and gets some nice things from being over TCP whether FCoE people will admit it or not.

When FCoE first came to market, vendors had dollarsigns in their eyes with thoughts of extorting customers with FC pricing strategies using 'just' ethernet. You saw people trying to do per-port FC enablement licensing BS and other stuff unheard of in ethernet land.

If FCoE is going to exist long term, it will be as a 'freebie' alternative to iSCSI or as a convenience to build large SANs without a lot of FC switches and HBAs but using existing FC enclosures.

Re:Too late

[afidel]

FC is so routable, there have been FC directors with routing capabilities pretty much since there have been FC directors. That said routed FC is kind of a hack but since lossless iSCSI requires DCB which also isn't really routable to get the same reliability you end up with the same limitations with a much higher complexity and CPU cost.

Re:Too late

[guruevi]

I think he meant that FC is not routable as the standard IP protocol is unless you buy expensive and proprietary solutions (as you call it, a hack).

  iSCSI has the advantage of being able to sustain a packet loss while FCoE can't stand it. FCoE is thus only possible over a small fabric (eg. 1 stack of dedicated switches) while iSCSI can be mixed with other traffic without sustaining any issues. Off course, some people using iSCSI can't sustain any packet losses either (because of latency - eg. streaming video and live video editing) and don't understand that Ethernet is not built for that kind of load - network engineers don't care about packet losses and hope the transport layer will fix them, storage engineers can't have any packet losses and the transport layer relies on it.

That being said, FCoE is similar to ATAoE, never widespread because of it's iSCSI cousin and those that ended up using it, might as well just used a true FC (or Infiniband) fabric.

Re:Too late

[CAIMLAS]

FC is expensive, compared to what?

Unless you're using the (mostly shit) iSCSI software initiators, you'll be using iSCSI hardware initiators on 10gbit Ethernet - which is hardly cheap.

Do the math: to get > 1Gbit/s on your fabric-side links, you need to spend a copious amount if you're going 10gigE. Not only are the switches ungodly expensive compared to 1Gbit, but they're expensive compared to pretty much everything else - Infiniband or Fibrechannel. When it comes down to it, the biggest thing 10gig Ethernet has going for it is compatibility and people understanding Ethernet (it certainly isn't price or availability - it's easier to find high throughput FC and the like).

FC is great - for storage networks. Infiniband is likewise great, though it was way ahead of the curve (and I have no idea what's keeping it from competing now, aside from vendors not wanting to push it). They're comparable on price and similar on functionality. Ethernet keeps getting its push from 35 years of history.

Why anyone would want to encapsulate FC on top of Ethernet (thereby reducing any advantage FC might have), I have no idea. It sounds like one of those "let's fix our poor engineering with creative application of technology; it can bite the next poor fucker in the ass".

Re:Too late

[butlerm]

network engineers don't care about packet losses and hope the transport layer will fix them

That is a bit of a generalization don't you think? Excessive packet loss is death to any real network, which is why there has been a lot of effort to do various forms of explicit congestion notification instead.

On the Internet, that may be hard, but on a local network it is much easier. Some switches even have ECN marking these days, which is a far superior solution to avoiding loss through buffer bloat.

Re:Too late

[grub]

Do the math: to get > 1Gbit/s on your fabric-side links, you need to spend a copious amount if you're going 10gigE. Not only are the switches ungodly expensive compared to 1Gbit

At work the iSCSI chassis we've been buying have 4x 1Gb ports which bind to make a 4 Gb pipe (the initiator has to be capable). Not 10GigE but much cheaper than a 10 Gb-ready switch.

'course, if you need 10 Gbit, you need 10 Gbit but this is a nice trade-off if super-performance isn't critical.

And still, no one buys it.

[7213]

FCoE...

A solution in search of a problem. 10GbE ethernet is really very nice. FC (and FCoE included) have a history of poor vender interop.

So by using FCoE you get the worst of both worlds, 10GbE with vendor lockin at the storage level....

So... NFS anyone (or I guess iScsi)?

Only time i've ever used FCoE was as a WAN tunnel link for asynch rep.... not seeing any other value for this anytime soon.

Planned Obsolescence

[MrQuacker]

Worked wonders for the auto makers, hows that working for you?

/s

Duh

[afidel]

When Brocade introduced their FCoE switch I could pick up two 40 port 8Gbps FC switches and a pair of 48 port GigE switches with 10Gb uplinks for what they were charging for 24 ports of FCoE with 4x FC connections. So instead of going with the switch that probably cost them no more to manufacture I bought a pair of 5100's and bought a pair of stacking HP GbE switches and so had complete redundancy for about the same cost as one FCoE switch.

Re:Duh

[afidel]

Actually FC has an inherent disadvantage and that is they ship about 5% of the ports per year that Ethernet does and so all their R&D has to be spread over 1/20th the ports. The vendors all realized this about 5 years ago and so they started on FCoE to spread their R&D budget over the much larger Ethernet ecosystem. I believe 16Gb FC will be the last standalone standard and that after that they will piggyback on Ethernet for 40Gbps and 100Gbps speeds. Speaking to industry insiders at Storage Networking World it's obvious that the days are numbered for standalone FC.

As to your claims that 10Gb FCoE is slower than 8Gb FC for throughput that's rubbish as the framing overhead for FCoE is nowhere near 20% and they are both lossless protocols, for latency it may or may not be true depending on implementation.

Re:Duh

[drsmithy]

So instead of going with the switch that probably cost them no more to manufacture I bought a pair of 5100's and bought a pair of stacking HP GbE switches and so had complete redundancy for about the same cost as one FCoE switch.

You also had 1/10th the bandwidth and twice as much cabling to each server, higher power draw, more rack space required and more devices to manage.

Re:Forgive my ignorance...

[Junta]

One is most 'FCoE' equipment had FC and ethernet ports, so there was a hardware difference.

Another is FCoE generally means the ethernet switch has some FC layer management features (e.g. looking at WWN, zoning, etc). I think this is the *key* priced modification for most FCoE equipment without FC ports. Basically making a way of dealing with the switches exactly the way storage admins are accustomed to dealing with SAN switches.

Finally, there are some layer 2 features considered essentially mandatory for 'decent' FCoE (more advanced pause frames, for one).

I still haven't found anyone getting excited about FCoE, even if it doesn't cost more. The storage admins I've met by and large hate the way they have been required to deal with their equipment.

Re:Forgive my ignorance...

[quickOnTheUptake]

Apparently the latter. "Since classical Ethernet has no flow control, unlike Fibre Channel, FCoE requires enhancements to the Ethernet standard to support a flow control mechanism (this prevents frame loss). . . . Fibre Channel required three primary extensions to deliver the capabilities of Fibre Channel over Ethernet networks: -Encapsulation of native Fibre Channel frames into Ethernet Frames. -Extensions to the Ethernet protocol itself to enable an Ethernet fabric in which frames are not routinely lost during periods of congestion. -Mapping between Fibre Channel N_port IDs (aka FCIDs) and Ethernet MAC addresses." --Wikipedia

It's all about what the swich is capable of

[sirwired]

Your plain-vanilla 10GbE switch does not have the flow-control bits required to make Ethernet lossless; without essentially lossless traffic, SCSI/FC perf goes in the dumpster. (0.03% packet loss == approx. 50% performance cut.)

In addition, there must be at least one switch in the VLAN that can provide FC services, such as zoning, address assignment, name services, etc.

nobody buys 10GbE either...

[SuperBanana]

10GbE ethernet is really very nice.

Too bad you can't really buy it, and it's insanely expensive, with per-port costs in the hundreds of dollars range. Lots of choices for adapters (which are also insanely expensive)....but I went looking for a 10GbE switch for our small-ish server room for some of our higher bandwidth systems that easily saturate gigabit ethernet...and came up very short in terms of selection. The vast majority of the market consists of switches with 1-2 10GbE uplink ports. That's slightly useful for some situations (for, say, a backup server with a lot of bandwidth, or linking to a main backbone), but not so useful if you want to link up a whole bunch of systems.

I will admit though, performance is impressive....

[Desmoden]

With no tuning (other than Jumbo frames for FCoE) I was able to get 9.7Gb/s using FCoE over 10Gb ethernet.

While 16Gb FCP/FC is around the corner, you will be able to run FCoE over 40Gb and 100Gb ethernet in 2-3 yrs. (at MUCH $$)

Keep in mind however, iSCSI has been around for over 10yrs now. These things take time to grow, mature, attach.

So lets wait a few more years before declaring anything dead or alive =)

And keep in mind, FCoE is not meant to replace FCP/FC, its meant to fix what is keeping iSCSI from doing better.

GBIC's Still To Expensive

[__aajwxe560]

Beyond this, the physical costs versus 8gig are just not justified yet. With the overhead of FCoE, you can roughly say 10gig FCoE is the same speed as more traditional 8gig FC. If you believe that to be roughly true, then price is the next factor to consider, as what are you really getting?

8gig Fibre Channel GBIC for a SAN fabric averages around $150-$200.
10gig network (CNA) GBIC for a more traditional network averages around $1100.

I am building out a new virtual farm now, and much as we tried to go the converged route with 10gig network, the price point simply isn't there yet (technology is still maturing this year as well). You can work around this with copper for very short runs, but the expense comes in per-rack network gear.

This should start to settle in the fall as the standards fall together better.