Slashdot Mirror
iSCSI vs. Fibre Channel vs. Direct Attached Disks?
mrscott asks: "Does anyone have any good, simple benchmarks about iSCSI performance in a SAN as it relates to fibre channel and direct attached storage? There's a lot of information out there about iSCSI TCP offload adapters that improve performance, but it's still hard to get a handle on even those stats without the original numbers for comparison. We're considering an iSCSI solution from Lefthand networks, but finding independent (and reasonably simple) numbers has proven somewhat difficult, even with the awesome power of Google behind me."
in terms of speed, iSCSI can't touch 2GB Fibre-Channel...
"The world only exists in your eyes. You can make it as big or as small as you want." - F Scott Fitzgerald
You want manageability. You want the ability to take "some disk" and add it to a server, anywhere, at any time. You want the ability to grow/shrink the filesystems on those servers. You want redundancy, and you want top notch vendor support. Direct disk might be faster, with local processing handling the FS buffering in local RAM, but what happens when ServerA needs 20G of the 100G disk you installed in ServerB?
I want to delete my account but Slashdot doesn't allow it.
... avoid it at all costs. Everyone I've talked with about iSCSI, from driver writers to end users does not like it. ATA over ethernet shows more promise, as it's much more simple than iSCSI. We'll see what future will bring ... but for now I'd stick with fibrechannel.
Both iSCSI and FC are networked version of SCSI, and all 3 technologies are much faster than their respective disks, thereby not being the bottleneck at all. After Ultra160, the standard PCI channel is saturated, and 64-bit PCI like PCIX is needed for Ultra320, all the while usually even in the burst mode ( from cache) disks cant saturate this available bandwidth, say 6x RAID5 15K RPM disks in read mode.
FC and iSCSI are much more expensive than SCSI Ultra320, which is commodity hardware now. FC just sends the data in optic to outside the system, where larger datawarehouses can be managed instead of getting bigger and bigger Unisys boxen.
So if you need terabytes of data all in one place (I mean at least 10 terabytes), consider iSCSI and FC and putting the disks outside the system for better management. We are getting a NAS solution to replace our backup tapes, requirement was 1.2TB. We will get 4x 300GB Maxtor Maxline II SATA disks... the slow cheap ones, and put them in an IBM xSeries 206 which are going at $500 CDN, with an Adaptec RAID card.
Upto 16 SATA 400GB disks can be managed by a simple adaptec raid card, beyond that, think FC arrays.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
Seriously, its not so much a speed issue but an issue of how you manage your environment. You can get enough performance out of all three solutions and there are other, more important things to look at.
As you see, I would recommend worrying less about the performance and more about what you really need and what your environment looks like.
If you just want performance for cheap, then local disks are unbeatable. Instead of spending money on expensive fibre or iscsi offload controllers buy tons of cheaper scsi cards, instead of running fibre and buying a fibre switch buy tons of disk drives.
Most people make the mistake to worry about capacity - in reality, its the number of spindles you need to look at and the capacity or your storage is a result of that. Figure, each disk attached gives you about 100 i/o ops per second. If you need to do 5000 ops, you need 50 disks - no matter how they are connected. Figure a disk can get you 20 MB/sec - if you need 200MB/sec throughput, thats 10 disks.
In that example, I need 50 disks then to satisfy my requirements. Next, take the max throughput and divide it by 3 - that's 66MB/sec for fibre then, 100MB for scsi and 33MB for iscsi over gigabit. So to run my 200MB/sec I'd need only 2 scsi channels, 3 fibre or 6 iscsi connections.
Next of course can't have 50 drives on 2 channels - more than 3 disks drop the transfer rate dramatically... Since fibre and iscsi mask the physical spindles they don't care but I need to have 16 scsi controllers to really run the disks.
Peter.
FC-AL is the "gold standard" for performance and reliability, but has limitations when you want to expand clustering (FC switches still cost gobs of $$$). Fibre Channel cabling also has distance limitations that go away with iSCSI.
... provided the network will drive it. 10Gbit Ethernet and more will definitely fuel this migration, sooner than you think!!
The acid test -- for me anyway -- is seeing LARGE customers (banks, airlines, government agencies [pick a 3-letter acronym], pharmaceuticals, major industries such as oil and energy, entertainment companies and movie studios, etc.) implementing iSCSI on an equally LARGE scale, and quite successfully.
With few exceptions, if the underlying Ethernet network is functioning properly, iSCSI performs remarkably fast. No, you won't get the 2 Gbit FCAL rates -- *yet* -- but we have customers running dozens of large (>TB) databases off a single appliance over one or two GigE ports and iSCSI.
Generally, it's recommended to segment off the iSCSI traffic so it's not routed or mixed with public traffic anyway, but even those (small) customers that pipe all of their storage appliance data through a single 10/100 Ethernet interface only have problems if they put too many users on there as well. (A direct crossover cable is *ideal* for iSCSI.)
In addition, the Microsoft iSCSI initiator has finally outgrown its initial bugs/problems (with our help in some cases), and is darn solid with plenty of different targets.
I'd love to drop a bunch of example company names, but I'm sure those companies consider that information to be competitive, and it's not my place to divulge it. Any large company you can think of already has an investment in FC-AL, and all but a very small percentage have iSCSI infrastructures as well. Medium-size and small (50 employees) companies are also seeing HUGE benefits from iSCSI their own implementations.
The 0-day acid test (which works amazingly well in our labs with the right HBAs) is SAN booting over iSCSI. Imagine having an nnn-Terabyte set of storage, from which ALL your servers boot EVERYTHING. Not a single magnetic disk is required in the servers themselves. Makes server clustering and blade/grid computing so very attainable
(As an engineer for a major storage vendor (FC/iSCSI/near-line IDE storage/archiving), I work with all of this stuff on a daily basis. Not saying I'm an expert, just that I kinda know what I'm spouting here...)
Mike
"Not an actor, but he plays one on TV."
The original question was regarding real world performance of iSCSI in particular, and since frew of the posts seem to touch on that I may as well tell what I've learned from hard experience regarding the other technologies: SCSI and FCAL.
My experience is with very high transaction volume OLTP databases (oracle) backing a financial website. I've found that neither SCSI nor FCAL adapters limit performance significantly. This was with qlogic qla3200 adapters, or with highend adaptec Ultra320, on Solaris 9 and the last few versions if RedHat enterprise. Only the older versions of redhat had some kind of problem with the qlogic driver, plus bounce buffer IO, which drove down performance. But then to be nit picky, that was the driver not the HBA. Solaris was always fine, and now redhat is too.
The main performance challenge was *always* tuning the database and spreading out on lots of spindles. The HBAs at over 200M/sec each never posed a problem on larger sun boxes (8 or more procs) with 7 or 8 way parallel sequential reads going. On smaller hosts or smaller disk arrays,k the problem was always on the host itself or the disk seek times respectively, not the hbas themselves.
A 10k rpm drive will do about 70 mbytes/sec off the outer platter (near block 0) and as a rule of thumb, a 2Gbit fcal adapter will do 200mbytes a second (at least on solaris or newer redhat EL). So my dual qlogics would do 400Mbytes a second under absolute optimal disk access, but typically it's not that perfect 8 way parallel *serial* scan off the outer platter, its usually farily random.
So in the high end database applications (datawareyouse or OLTP) least the usual tuning challenge (and $$ for that matter) are with getting a fat spread across a lot of spindles, and making sure the application is either caching well (OLTP) or doing orderly, sequential scans (datawarehouse)
"Hi. I'm thinking about buying a car. Which is the best one?"
Ummm What are you doing with it? What's your budget, and what are your expectations? Some people think a TB is big, some consider 20+ TB a building block standard (I do).
1. Look on the net and in mags. - read read read
Is this a big implementation, or a small one. I'd bet a small one, or you probably wouldn't be asking here - that probably means you don't want iscsi. Direct FCAL on multi-port RAIDs might be your speed - if you aren't big enough for that, maybe some low-end storage from Dell or Apple might be your thing.
2. Bring in some vendors, tell them what you want - ask them how to do it. - talk to a few.
3. Ask about install costs to implement what you want to do. (A lot of people don't realize the work involved for large scale implementations)
If you don't have the staff to know if iscsi is for you, you may not have the staff to implement and troubleshoot it. Direct attached fibre - or even via a switch will be far easier to deal with.
If you can afford it, stay away from SATA. The problem is that it's sooo much cheaper vs FC disks on larger systems - but the seeks are slower, the mechanicals are poorer, and the command set less complete. - It's $ vs reliability.
Cheers,
a storage guy.
Throughput is limited by the number of spindles and the network speed (eg, 120MB/s on GigE). It doesn't matter if its iSCSI or FC or DA. Caveat: I've found some of the iSCSI HBAs can't quite reach linespeed, but you'll be fine with a software initiator.
Latency is probably going to be "a little" higher on iSCSI (over ethernet) than FC or DA.
Host CPU cost is higher on iSCSI if you don't use a HBA.
If you're serious about storage, manageability is shit on DA.
Summary:
Expensive, fast, manageable: FC
Cheap, possibly a little slower, manageable: iSCSI
Cheap, fast, impractical: DA
Is really apples-to-oranges.
It would be theoretically possible to run iSCSI over IP-over-FC, for example.
Now, if you want low-latency speed, get yourself an InfiniBand SAN, or 10GigE for roughly the same speed at somewhat higher latency (but open standards-based... yay)
The only reason I don't like it is there are very few server platforms (apple XServe being the exception) that boot from fibre-channel storage systems.
I have an HP MSA 1000 fibre channel SAN that runs Linux, Netware, Windows 2000 and Windows 2003. The servers connect to the SAN via a fibre channel switch and HP(Emulex) HBAs in the servers. All systems boot from the SAN! There are no disks in any of the servers. This makes replacing failed server (hardware) a 3 minute plug-and-play operation. The hardest part is lifting the server in and out of the rack.
I set it up about a year and a half ago and it wasn't easy at the time. Initially, I couldn't get it to work for anything but the first server. After a very long time fighting with it, HP came up with a firmware upgrade for the fibre channel HBA. After that it worked like a charm and was stupidly easy. Any OS should be able to do it as the "boot from SAN functionality" is actually in the HBA not the server hardware or OS. The HBA simply presents the 'physical' LUN, what ever it is, as a virtual LUN 0 to the server's BIOS. Presto, it boots!
SATA scares me a little, simply because Native Command Queuing doesn't allow the OS to impose any kind of ordering constraints on the commands, the way that Tagged Command Queuing does.
/. FS holy wars).
So, as I see it, in a power-failure, all that hard work done on ensuring that a modern file-system is always consistent just goes out of the window: meta-data updates happening before the data itself is updated. In fact, that's likely to happen quite a bit with many writes to separate files, since the metadata's slightly more likely to be grouped together.
However, I don't know how this works out in practice, as opposed to theory from reading what the features provide. Do any OS/FS combinations actually support enabling even TCQ in a safe manner, instead of just a potentially unstable performance boost? If so, recommendations on an FS choice gratefully received (please, specific to this issue, not the usual
Disable NCQ and get a major performance hit? Or pick a platform (SCSI, ATA4) which supports TCQ and an OS/FS which then uses this intelligently for meta-data handling?
Sure, there's UPS power and decent SAN storage arrays take battery-backup to decent quality levels, but with each network link being a point of failure the next time some monkey is let loose on "unrelated" cabling issues, and the general flakiness of data-center UPS kit not quite managing to perform as advertised, having SATA with NCQ in a reliable environment leaves me apprehensive and I really don't want to be responsible for a Supported System relying upon SATA/NCQ in a separate box from the box which has the file-system drivers in it.
Or am I smoking crack?
The "all your eggs in one basket" metaphor only works if all the baskets are equal. And if you are comparing iSCSI/FC to internal storage, you're talking about completely different 'baskets'.
I could have a lot of fun with that metaphor but i'm sure you get the point without it.
After doing consulting for an ISP installing a mail cluster for 70,000 users, we found that the interconnect speed has little to do with performance. In this instance, the filesystem was the biggest bottlekneck and it normally wouldn't have been if it wasn't for file locking race conditions. There is no way to tell if a given system will be better or worse for your application, and it is true... you can't find accurate benchmarks. I would suspect that OSDL would be in a good position to do reviews because they have lots of "big iron" hardware.
My gut feeling is that iSCSI is kind of in its infancy and that 2GB fiber channel is pretty much standardized. As the previous posters said, the interface speed less important than the speed of the storage array controller and drives.
For a different project, we used a 2GB fiberchannel disk array of 16 250gb hard drives. These gave a single machine r/w time of about 75 megabytes a second (on par with SCSI). There are LOTS of people who resell intel controller based raid boxes with either fiber channel or SCSI 3 interconnect.
Has anyone seen a reverse-iSCSI adapter? Sorry, this isn't the best possible name, I'm sure.
I've seen plenty of adapters that you can hook to a SCSI device and call it an iSCSI device but that assumes you have a proper iSCSI HBA.
What I need is something I can attach to a SCSI HBA that has an ethernet connector on the other side, so I can connect it to a network and do iSCSI over it.
This particular instance is for an old machine that noone's ever going to make an iSCSI HBA for to connect it to a tape drive on the other end of a Cat 6 cable run a hundred feet or so away.
It seems obvious but I haven't been able to dig one up on Google or Processor.com. I've even written a SCSI/iSCSI bridge manufacturer who was intrigued but unable to help.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)