Intel Announces Open Fibre Channel Over Ethernet

sofar writes "Intel has just announced and released source code for their Open-FCoE project, which creates a transport allowing native Fibre Channel frames to travel over ordinary ethernet cables to any Linux system. This extremely interesting development will mean that data centers can lower costs and maintenance by reducing the amount of Fibre Channel equipment and cabling while still enjoying its benefits and performance. The new standard is backed by Cisco, Sun, IBM, EMC, Emulex, and a variety of others working in the storage field. The timing of this announcement comes as no surprise given the uptake of 10-Gb Ethernet in the data center."

Comments

Fiber channel

[smittyoneeach]

Fiber channel
In ye olde patch panel
Beats fiber thin
On your chinny-chin-chin
Burma Shave

Speed?

[Chrisq]

As far as I can see this is a way of bridging fibre channels over Ethernet. This does not necessarily mean that you will get fibre-like speed (throughput or latency). I am sure that this will have some use, but it does not mean that high performance data-centres will just be able to use Ethernet instead of fibre.

Re:Speed?

[farkus888]

I am not too sure about the latency, but I don't know of any storage solution that can saturate 10 Gb sustained speeds. except maybe something like gigabytes array of ram as a hd system. simply reducing the number of drives on a daisy chain should keep you running happily as far as throughput goes.

Re:Speed?

[afidel]

According to this netapp paper even NFS over 10GbE is lower latency than 4Gb FC. I imagine if the processing overhead isn't too high or offload cards become available then this would be significantly faster than 4Gb FC. As far as bandwidth 10>4 even with the overhead of ethernet framing, especially if you can stand the latency of packing two or more FC frames into an ethernet jumbo frames.

Re:Speed?

[afidel]

Huh? Our piddly 150 spindle SAN could keep a 10Gb link saturated no problem if we had a workload that needed it. In fact that's less than 7MB/s per drive, about one tenth of what these drives are capable of for bulk reads or about one fifth for bulk writes. Even for totally random I/O 150 spindles could probably keep that sized pipe filled.

Re:Speed?

[shaggy43]

I have an account that I support that has completely saturated 4 4G ISL's in-between 2 Brocade 48k's, and had to re-balance their fibre. Granted, and individual HBA doesn't hit a sustained 2G/sec, but 16G/sec saturated to a pair of HDS Thunders is impressive.

Mostly, I think this technology will compete against iSCSI, not dedicated fibre, with all the drawbacks -- plus an added drawback of currently being single-platform.

Re:Speed?

[jsailor]

For that type of project, look to the hedge fund community. I know of 2 hedge funds that have built their own storage systems that way - Ethernet, Linux, direct attached disk, and a lot of custom code. My world doesn't allow me to get into the details, so I can't elaborate. My only point is that their are folks doing this and it tends to be the guys with large storage needs, moderate budgets, and a great deal of freedom from corporate standards and vendor influence.

Re:Speed?

[canuck57]

My only point is that their are folks doing this and it tends to be the guys with large storage needs, moderate budgets, and a great deal of freedom from corporate standards and vendor influence.

Stay with them, these are good environments. BTW, I am not anti-standards, but at the end of the day they need to make sense. That is, not a standard for pure political posturing.

Re:Speed?

[Intron]

Umm. The paper says that the test load was sequential reads entirely cached in memory, so not exactly an unbiased test.

10GE is a heck of a lot cheaper

[afidel]

As long as a server is within the distance limit of copper, 10GE is about 3-4x cheaper then even 2Gb FC. We've also had a heck of a lot more stability out of our 6500 series switches then we have out of our 9140's and the 9500's are extremely expensive if you have a need for under 3 cards worth of ports.

High End customers will not go to this.

[BrianHursey]

As we have seen with iSCSI the bandwidth capability over Ethernet just is not there. I with the EMC this will probably be great for the low end company that needs a mid tier and low tier environment. However large corporations with large database and high number of systems still need to stay with fibre frabrics. This probably will be only on the mid tier platforms like clariion.

Re:High End customers will not go to this.

[totally bogus dude]

I expect you're right, but it's interesting to note they're referring to this as Fibre Channel over Ethernet, and not over IP. The reduction in overhead there (not just packet size, but avoiding the whole IP stack) might be enough to really help; and if you're running separate 10 Gigabit Ethernet for the storage subsystem (i.e. not piggy backing on an existing IP network) it might be really nice. Or at least, comparable in performance and a heck of a lot cheaper.

On the other hand, really decent switches that can cope with heavy usage of 10-GigE without delaying packets at all aren't going to be massively cheap, and you'd need very high quality NICs in all the servers as well. Even then, fibre's still probably going to be faster than copper... but that's just something I made up. Maybe someone who knows more about the intricacies of transmitting data over each can enlighten us all?

There was recently an article about "storing" data within fibre as sound rather than converting it to for storage in electrical components, since the latter is kind of slow; how does this compare to transmission via current over copper?

Re:High End customers will not go to this.

[afidel]

Latency and bandwidth are comparable for copper and fiber ethernet solutions today, the drawback to copper is you need to be within 15m of the switch. This isn't so bad in a small datacenter but in a larger facility you would either need switches all over the place (preferably in 2's for redundant path) or you'd need to go fiber which eliminates a good percentage of the cost savings. FiberChannel used to have copper as a low cost option but it's not there in the 4Gb world and even in the 2Gb space it was so exotic that there was almost no cost savings due to lack of economies of scale.

Re:High End customers will not go to this.

[Chris Snook]

Bullshit.

The bandwidth is there. I can get 960 Mb/s sustained application-layer throughput out of a gigabit ethernet connection. When you have pause frame support and managed layer 3 switches, you can strip away the protocol overhead of iSCSI, and keep the reliability and flexibility in a typical data center.

The goal of this project is not to replace fibre channel fabrics, but rather to extend them. For every large database server at your High End customer, there are dozens of smaller boxes that would greatly benefit from centralized disk storage, but for which the cost of conventional FC would negate the benefit. As you've noted, iSCSI isn't always a suitable option.

You're probably right that people won't use this a whole lot to connect to super-high-end disk arrays, but once you hook up an FCoE bridge to your network, you have the flexibility to do whatever you want with it. In some cases, the cost benefit of 10Gb ethernet vs. 2x 4Gb FC alone will be enough motivation to use it even for very high-end work.

Re:High End customers will not go to this.

[myz24]

I think you could follow up with some info about your setup. I mean, there is no way you're getting those speeds without tuning some network parameters or with some serious CPU and RAID setup. It's not that I don't believe you, I have a buddy that has done the same but with NFS but he's using an opensolaris system with TCP offloading cards and a heck of a RAID array.

Re:High End customers will not go to this.

[Znork]

"those speeds without tuning some network parameters or with some serious CPU and RAID setup."

Basic setup is approximately this; CPU's for both servers and clients range between AMD XP 3500+ to AMD X2 4800+. Motherboards are Asus (Nvidia 550 and AMD690) cards, with 2-4GB memory plus an extra SATA card on the iSCSI servers, and extra rtl8168/9 gigabit cards (the forcedeth driver has some issues). Disks on the iSCSI servers are striped with LVM, but not to more than 3 spindles (I dont care that much about maxing speed, I just want close-to-local disk performance). Tuning's been mainly on the iSCSI side with InitialR2T and ImmediateData. I've played around with network buffers, but basically come to the conclusion that it's more efficient in my case to throw RAM on the problem.

The peak rates (90-97MB per sec) have been obtained on completely unloaded systems, and the standard 40-60MB/s read is with disk mirrored against both iSCSI systems (and then striped on the iSCSI systems).

"I have a buddy that has done the same but with NFS"

Ah. Ehm. NFS. Yes.

Well, to tell the truth, I started out this interesting journey towards a home SAN using diskless systems booted over PXE and mounting NFS roots (mainly to silence my mythtv systems, and simplify backups). Lets just say that after testing and switching the first system over to iSCSI I could barely believe my eyes.

NFS is much, much, much harder to get decent performance out of. No matter how much tuning I've done I've rarely managed to get more than 20-40MB/s peaks, and for many small file accesses the performance is horrible. I'd thought that was what I could expect out of a gigabit Lan, after all, NFS saturated a 100Mbps network. I was quite surprised when my first iSCSI tests got 60-70MB/sec.

If your friends setup is such that block devices would work instead of NFS (at least for some parts), I'd really suggest he try running an iSCSI target. I cant vouch for the Solaris version, but I know there is one and that it sounds similar to the Linux one. Linux ietd (iscsi enterprise target) has been very stable and highly performing for me (more than a year running it now with no lost data :) ). It lacks some features like some forms of SCSI-3 reservation, but I can live with that.

Re:Bumper cars.

[morgan_greywolf]

Oh this should be interesting. Fibre over a collision-and hold-off architecture. They have this newfanged technology. It's called switched Ethernet! It's amazing! With switched Ethernet, you get no collisions!

eth0 Link encap:Ethernet HWaddr 00:00:0D:03:01:04
                    inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
                    inet6 addr: fe80::000:00f0:0043:0084/64 Scope:Link
                    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
                    RX packets:1781638 errors:0 dropped:0 overruns:0 frame:0
                    TX packets:1651683 errors:0 dropped:0 overruns:0 carrier:0
                    collisions:0 txqueuelen:1000
                    RX bytes:803882935 (766.6 MiB) TX bytes:333706343 (318.2 MiB)
                    Interrupt:18 Base address:0xd800


(address details fudged only)

Re:I'm more interested in AoE

[Hal_Porter]

You should give it a snappier name like Serial ATA Networking, or SATAN. Lots of interesting logo possibilities in that, and it'll be funny watching 'technology evangelist' types stutter, sweat and mumble when they give PowerPoint presentations to born again potential customers.

Re:Bumper cars.

[cait56]

FCOE really does rely on "new fangled technology". More than switched ethernet is required, it has to be an enhanced Ethernet that prevents virtually all congestion related drops.

Work on such features is indeed in progress in both IEEE 802.1 and the IETF. The comparison of FCOE vs. iSCSI in those environments will be a lot more even than the comparisons presented by FCOE champions currently. Those compare storage traffic that requires neither routing or security, and tests FCOE over forthcoming Ethernet vs. iSCSI over current Ethernet.

Those comparisons involve a lot more than wire transport protocols. For example, open-fcoe is a good start, but open-iscsi is a much more mature project.

Re:I'm more interested in AoE

[Intron]

How is ATA over ethernet offtopic in a discussion about a way of migrating SAN technology to ethernet?

ATA, SATA and SAS all have severe connectivity limits. They don't have a way of addressing a large number of devices, running long distances or supporting multiple initiators. While they might be fine for your home they are worthless for the SAN/LAN environment where fibre channel and FCoE are targeted.

Re:Srsly, FC or iSCSI?

[Joe_NoOne]

Some important limitations of iSCSI :

1) TCP/IP doesn't guarantee in-order delivery of packets (think of stuttering with streaming media, etc...)

2) Frame sizes are smaller and have more overhead than Fibre Channel packets.

3) Most NICs rely on the system to encapsulate & process packets - a smart NIC [TCP Ofload Engine card] costs almost as much as a Fibre Channel card.

Re:I'm more interested in AoE

[psydeshow]

FreeBSD is sometimes a tough sell to religious groups because of the devil mascot.

"You want to put... a demon? On our server?"
"Daemon, it's a daemon."
"..."

Re:Srsly, FC or iSCSI?

[onemorechip]

It doesn't *deliver* packets in order (at least, not unless the underlying network does). It provides the capability to reconstruct the original order. GP was talking about *delivery* of packets.

Re:Bumper cars.

[Gothmolly]

Dude, thats MY IP address !!!!