Slashdot Mirror
Fibre Channel Storage?
Dave Robertson asks: "Fibre channel storage has been filtering down from the rarefied heights of big business and is now beginning to be a sensible option for smaller enterprises and institutions. An illuminating example of this is Apple's Xserve Raid which has set a new low price point for this type of storage - with some compromises, naturally. Fibre channel switches and host bus adapters have also fallen in price but generally, storage arrays such as those from Infortrend or EMC are still aimed at the medium to high-end enterprise market and are priced accordingly. These units are expensive in part because they aim to have very high availability and are therefore well-engineered and provide dual redundant everything." This brings us to the question: Is it possible to build your own Fibre Channnel storage array?
"In some alternative markets - education for example - I see a need for server storage systems with very high transaction rates (I/Os per second) and the flexibility of FC, but without the need for very high availability and without the ability to pay enterprise prices. The Xserve Raid comes close to meeting the need but its major design compromise is to use ATA drives, thus losing the high I/O rate of FC drives.
I'm considering building my own experimental fibre channel storage unit. Disks are available from Seagate, and SCA to FC T-card adapters are also available. A hardware raid controller would also be nice.
Before launching into the project, I'd like to cast the net out and solicit the experiences and advice of anyone who has tried this. It should be relatively easy to create a single-drive unit similar to the Apcon TestDrive or a JBOD, but a RAID array may be more difficult. The design goals are to achieve a high I/O rate (we'll use postmark to measure this) in a fibre channel environment at the lowest possible price. We're prepared to compromise on availability and 'enterprise management features'. We'd like to use off the shelf components as far as possible.
Seagate has a good fibre channel primer, if you need to refresh your memory."
I'm considering building my own experimental fibre channel storage unit. Disks are available from Seagate, and SCA to FC T-card adapters are also available. A hardware raid controller would also be nice.
Before launching into the project, I'd like to cast the net out and solicit the experiences and advice of anyone who has tried this. It should be relatively easy to create a single-drive unit similar to the Apcon TestDrive or a JBOD, but a RAID array may be more difficult. The design goals are to achieve a high I/O rate (we'll use postmark to measure this) in a fibre channel environment at the lowest possible price. We're prepared to compromise on availability and 'enterprise management features'. We'd like to use off the shelf components as far as possible.
Seagate has a good fibre channel primer, if you need to refresh your memory."
Sun's A5200s are cheap on eBay, and you can pick up something like a 420r or a 250 to drive the thing. Put a qfe card in with the free sun trunking now for Solaris 10 and it'll serve up your files super speedy, all for very reasonable.
My friend, recursive green, has three A5200s in his basement right now, one stores his *ahem* photo collection and is web accessible.
I think new(er) fibre things are getting cheaper, but what was often high-end data-center-only big-$$ of a few years ago hits the price point of "at home" now.
Wheeeee
Speaking as the owner of two XServe RAID devices (5TB and 7TB models) as well as several other Fibre Channel devices I can say that the Apple Fibre Channel is by no means slow. Each SATA drive has pretty much equal performance to the SCSI drives we use in our Dell head node. Combined together there are times where we can pull several hundred megs a second off the XRAID's. Plus our XRAID has been fairly immune to failures thusfar. I have yanked drives out of it and it just keeps right on going.
Another little hint, if you are really worried about speed you can just install large high RPM sata drives yourself. Its not that hard to do at all.
Check out Alien Raid for more information.
I can toss in a bit of advice as I've been working with fibre channel from the low to the high end for several years now. Currently I'm managing a lab with equipment from EMC, HDS, McData, Brocade, Qlogic, Emulex, 3par, HP, Sun, etc. from the top to bottom of the food chain. I'm personnaly running a small FCAL setup at home for my fileserver.
0. Get everything off of ebay.
1. Stick with 1GB speed equipment. It's older, but an order of magnitude less expensive.
2. Avoid optical connections if you can -- for a small configuration, copper is just fine and often a lot less expensive. Fibre is good for long distance hauls and >1gb speed.
3. Pick up a server board with 64bit pci slots, preferably at 66mhz.
4. Buy a couple of qlogic 2200/66's. These are solid cards, and are trivial to flash to sun's native leadville fcode if you desire to use a sparc system and the native fibre tools. They also work well on linux/x86 and linux/sparc64. These should run about $25 each.
5. Don't buy a raid enclosure. Get a fibre jbod. You can always reconstruct a software raid set if your host explodes if you write down the information. If you blow a raid controller, you're screwed. Besides, you won't want to pay for a good hardware raid solution, and I have yet to see a card-based raid; they're always integrated into the array. I recommend a Clariion DAE/R. Make sure you get one with the sleds. These have db-9 copper connections and empty should run about $200. Buy 2 or 4 of these, depending on how many hbas you have. They'll often come with some old barracuda 9's. Trash those; they're pretty worthless.
6. Fill the enclosures with seagate fc disks. If you're not after maximum size, the 9gb and 18gb cheetahs are cheap, usually like $10 a pop on ebay and are often offered in large lots. They are so inexpensive it's hard to pass them up. Try to get ST3xxxFC series, but do NOT buy ST3xxxFCV. The V indicates a larger cache, but also a different block format for some EMC controllers. They are a bitch to get normal disk firmware on.
7. Run a link from each enclosure to each hba. Say you have 2 enclosures with 10 disks each. Simple enough; and 1gb second up and down on each link.
8. Use linux software raid to make a bigass stripe across all the disks in one enclosure, repeat on the second enclosure, and make a raid10 out of the two. Tuning the stripe size will depend on the application; 32k is a good starting point.
With that setup, you should pretty much max out the bandwidth of a single 1gb link on each enclosure, and enjoy both performance and reduncancy with the software raid, and not have to worry about any raid controllers crapping out on you.
You should be able to get two enclosures, 20 disks, a couple of copper interconnects and some older hbas for about $750 to $1,000 depending on ebay and shipping costs.
This should net you some pretty damn reasonable disk performance for random access type io. This is NOT the right approach if you're lookign for large amounts of storage. You'll get the raid10 redundancy in my example, but if you want real redundancy (and i mean performance-wise, not just availability -- you can drive a fucking truck over a symmetrix, then saw it in half, and you probably won't notice a blip in the PERFORMANCE of the array--something fidelity and whatnot tends to like) you have to pay big money for it. The huge arrays are more about not ever quivvering in their performance no matter what fails.
Hope this was of some use.
man tunefs | grep fish