Slashdot Mirror
Best Server Storage Setup?
new-black-hand asks: "We are in the process of setting up a very large storage array and we are working toward having the most cost-effective setup. Until now, we have tried a variety of different architectures, using 1U servers or 6U servers packed with drives. Our main aims are to get the best price per GB of storage that we can, while having a reliable and scalable setup at the same time. The storage array will eventually become very large (in the PB range) so saving just a few dollars on each server means a lot. What do people out there find is the most effective hardware setup? Which drives and of what size? Which motherboards, etc? I am familiar with the Petabox solution which is what the Internet Archive uses — they have made good use of Open Source software. So what are some of the architectures out there that, together with Open Source, can give us a storage array that is much better than the $3 per GB plus that the commercial vendors ask for?"
I love shoeboxes for storage. Cheap and modular.
If $/GB is a dominant factor, I would suggest Coraid's products. They have a pretty niffy technology which is dead simple and extensively leverages OSS. From my personal experience as a customer, I think they are a bunch of good folks as well. They also seem to constantly be wringing more and more performance our of their systems. Anyway, something to explore if I were you.
If price is your only requirement, go with tape. Otherwise, let's hear about redunancy, speed, reliablity, availablitlty, etc.
It's just fibre attached storage, so you can use whatever servers you want as the head units. List is 7 TB for $13K... if you're going to scale up a lot you can certainly ask Apple for a discount.
Not *the* cheapest out there, but fast, reliable, and works well with Linux or whatever server heads you want.
It all depends on what you are trying to do
For some workloads, many servers with four drives each may work. This is the Petabox/Google model. This works if you have a parallelisable problem and can push most of your computation out to the storage servers.
Remember, you don't have a 300Tb drive, you have 300 servers, each with 1Tb of local storage.
For other workloads, you need a big disk aray and SAN, probably from Hitachi, Sun, or HP. This is the traditional model. Use this if you need a really big central storage pool or really high throughput.
Many SAN arrays can scale into the PB range without too much trouble.
Nowadays, a PB is enough to arch your eyebrows, but otherwise not that amazing. It seemes that commercial storage leads home storage by 1000. When home users had 1GB drives, 1TB was amazing. Now that some home users have 1TB, many companies have 1PB.
Rendundancy, aviability, and reliability is best served in the most cost effective way by using the 'lots of copies' method of storage.
You simply get the cheapest PCs you can get. The Petabox uses Mini-itx systems due to low power and thermal requirements. Stuff them with the biggest ATA drives you can get for a good price.
Then fill up racks and racks and racks of stuff like that. As long as the hardware is fairly uniform then you have some spares and you have maintainance taken care of.
Then you copy your information on it, use a database (redundant also) to keep track of your files and such. Scripts and programs that keep files in sync and spread out on multiple machines so there is no single point of failure. I don't know of any sort of OSS software that does this that is out there, but I would bet that some place like Archive.org will share what they use.
Think about it like you have a huge library of tapes for backup. Then you have a automated method to use a robot to fetch, copy, and return various tapes as you need them.. Except that instead of automated hardware and tapes your using software and harddrives.
This sort of system will scale much higher then any sort of currently aviable SANS solution.
NOW... If you want all relaibility, aviability, redundancy with SPEED then that is going to fucking cost you huge amounts of money.
This is purely for STORAGE, not for work scratch space.
Personally were I work we have a 3 terrabytes worth of disk space for 2 copies of our major production database and scratch work space. The information is completely refreshed every 2-3 months. No data is kept older then that. In the back we have about 22 thousand tapes for longer term storage and backups for weekly jobs...
Personally I'd rather have a room full of mini-itx machines to go along with the SAN rather then a bunch of fucking tapes.
So having thought about this a lot, here's what I would do:
1) Run a FC SAN as the backend. This allows you to connect anything you want without wondering what future technology will allow for - ATAoE, iSCSI, ???
2) Love thy Apple. XServe RAID's are 3u, 7tb (raw) and $13,000 - get a bunch - each controller see's 7 disks, set them up as a RAID 0 and uplink the thing to a FC switch.
3) Use DNFStorage.com's SANGear 4002 / 6002 devices to RAID 5 across the XServe RAID 0 LUN's. Your data security then can tolerate half of an XServe RAID going offline. RAID 6 allows for an entire unit to become DOA. Make sure to have an online spare or two.
4) Repeat - but remember, just because you can create it, doesn't mean you can reasonibly back it up.
Now the stupid question - what are you trying to do that would require this much space when you don't have the budget to get a "tested, supported, enterprise" solution? Building things is fun, but at some point you need to back up and say, "Am I willing to risk my company on my solution". EMC, HDS, IBM, HP and other big vendors are willing to step up and make sure your solution works, runs and will not fail (see that video with the SAN array getting shot?)
With the right filenames, you could store all your stuff on P2P for eternity!
"I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)
The best option here is Coraid.
:)
15-drive array = $4000
750GB Seagate Drive = $420
Full Array (14-drive RAID5, one hot spare) = $10,300 for 9.75 Terabytes
That's $1.06 per gigabyte RAID5 with hotspare. It doesn't get any better than this. Even with labor to assemble and set it up, and shipping, it's hard to get above $1.50 a gigabyte.
I suggest CLVM and Xen on the servers. Xen makes it really easy to turn up a new box. The space is available everywhere. CLVM is flexible enough to allow you to migrate stuff across arrays (or span arrays) very easily. I actually boot off of a flash chip and pivot_root my Linux systems onto a filesystem running off of these.
These numbers are roughly my cost. E-mail me if you'd like to buy one and we can talk about it.
I think Mauve has the most RAM. --PHB (Dilbert Comic)
Ok... so if you are planning on driving down the cost of the setup, you will have to sacrifice other features.
/. to 'roll your own', but I don't see any financial institutions 'rolling their own'. Educate yourself, put out RFPs to various storage vendors. Educate yourself.
How important is performance? Reliability? Scalability? If you are building PB of storage, how do you plan to back it up? What are the uptime requirements? Hitless codeload? Do you need multipathing? Snapshots? They all do raid, but do you need SATA/FC disk? How much throughput (MB/s) and IO/sec do you require? What management tools do you need? Callhome? SNMP/pager/email notification? Can you save $$$ by putting in a high end array but use iSCSI in the hosts to save $800/host?
In the end it comes down to. How valuable is your data? What is the impact to your business if it is down.
It sounds sexy on
...is a HP EVA 8000 SAN... or any other SAN with a virtualizaion layer. IBM TSM (aka ADSM) for backup.
It is not cheap.
If you require high levels of performance (=comparable to local direct-attached-disk) or reliability (=must be online "all the time") then stop right now and go out talking to commercial vendors. You will not save enough money doing it yourself to make up for the stress, people-power overheads and losses the first time the whole house of cards falls down.
However, if your performance or reliability requirements are not so high (ie: it's basically being used to archive data and you can tolerate it going down occasionally and unexpectedly) then doing it yourself may be a worthwhile task. I get the impression this is the kind of solution you're after, so you'll be looking at standard 7200rpm SATA drives.
Firstly, decide on a decent motherboard and disk controller combo. CPU speed is basically irrelevant, however, you should pack each node with a good 2G+ of RAM. Make sure your motherboards have at least two 64bit/100Mhz PCI-X buses. I recommend (and use) Intel's single-CPU P4 "server" motherboards and 3ware disk controllers. I believe the Areca controllers are also quite good. You will have trouble on the AMD64 side finding decent "low end" motherboards to use (ie: single CPU boards with lots of I/O bandwidth). Do not skimp on the motherboards and controllers, as they are the single most important building blocks of your arrays.
Secondly, pick some disks. Price out the various available drives and compare their $/GB rates. There will be a sweet spot were you get the best ratio, probably around the 400G or 500G size these days (it's been several months since I last put one of these together).
Thirdly, find a suitable case. I personally don't like to go over 16 drives per chassis, but there are certainly rackmount cases out there with 24 (and probably more) hotswap SATA trays.
Now, put it all together and run some benchmarks. In particular, benchmark hardware RAID vs Linux software RAID and see which is faster for you (it will probably be software RAID, assuming your motherboard is any good). Bear in mind that some hardware RAID controllers do not support RAID6, but only RAID5. Prefer a RAID6 array to a RAID5 + hotspare array.
You now have the first component of your Honkin' Big Array. Install a suitable Linux distribution onto it (either use a dedicated OS hard disk, get some sort of solid-state device or roll a suitable CD-ROM based distro for your needs). Setup iSCSI Enterprise Target.
Finally, you need a front-end device to make it all usable. Get yourself a 1U machine with gobs of network and/or bus bandwidth. I recommend one of Sun's x4100 servers (4xGigE onboard + 2 PCI-X). Throw some version of Linux on it with an iSCSI initiator. Connect to your back-end array node and set it up as an LVM PV, in an LVM VG. Allocate space from this VG to different purposes as you require.
When you start to run out of disk, build another array node, connect to it from the front-end machine and then expand the LVM VG. As you expand, investigate bonding multiple NICs together and additional dual- or quad-port NICs to supplement the onboard ones. I also recommend keeping at least one spare disk in the cupboard at all times for each of your storage nodes, and also a spare motherboard+CPU+RAM combo, to rebuild most of a machine quickly if required. Ideally you'd keep a spare disk controller on hand as well, but these tend to be expensive, and if you're using software RAID, any controller with enough ports will be a suitable replacement for any failures.
We do this where I work and have taken our "array" from a single 1.6T node (12x200G drives) to 10T split amongst 3 nodes. We are planning to add another ~6T node before the end of the year. *If* this is the kind of solution that would meet your needs, I can offer a lot of help, advice and experience to you.
However, our "array" has neither high perfo
You're providing some really interesting information, but with all due respect, your approach is backward. Why would you first design the solution, and then ask what the guy has in mind for it?? It's like delivering a shiny new Porsche to someone, just to find he wants to move tonnes of dirt around.
First establish what he wants to use the storage for. Video streams? Archive? Massive web farm? Database systems?
Always start with establishing purpose. Designing the storage to match will be a lot easier.
To Terminate, or not to Terminate, that's the question - SCSIROB
Before I throw my two cents in a disclaimer; you focus pretty much exclusively on cost. No performance, no management, no reliability. With that in mind;
Dell MD1000 SAS/SATA JBOD + 15x 500GB SATA disks: ~$10500
Three of those, daisy chained to some head end server through a single SAS RAID controller. Guessing ~$4000 for the box.
That's 22.5TB of raw storage for ~$35k, or $1.57/GB, less if you work a deal.
You'll need about 45 of these for a petabyte (raw); $1.6M
Fabric (~5x 24 port layer-2 gigabit switches): ~$12k
600u of racks (5) + power: ~$25k
etc...
Tons of single points of failure, limited performance and zero management. Don't even suspect it's possible to get good IOPS from this; too few controllers, major throughput constraints on the 1Gb/s uplinks, etc... It's good for light to moderate streaming archival type use. Use Linux with lots of md and lvm for at least some measure of manageability.
Frankly, $3/GB for something with management tools, better reliability and/or better performance and looks pretty good, but if you're all about cost...
Lurking at the bottom of the gravity well, getting old
2) Love thy Apple. XServe RAID's are 3u, 7tb (raw) and $13,000 - get a bunch - each controller see's 7 disks, set them up as a RAID 0 and uplink the thing to a FC switch.
3) Use DNFStorage.com's SANGear 4002 / 6002 devices to RAID 5 across the XServe RAID 0 LUN's. Your data security then can tolerate half of an XServe RAID going offline. RAID 6 allows for an entire unit to become DOA. Make sure to have an online spare or two.
Lets think about this - what happens when you loose a single disk in one of your Xserve's ? Your LUN fails and you then have to rebuild from parity 3.5TB of data.
How long will that take ? This is a horrible solution.
You won't be able to grow your LUNs either.
Alex
It's been almost exactly two years since we put together the first petabox rack, and both the technology and our understanding of the problem have progressed since then. We've been working towards agreement on what the next generation of petabox hardware should look like, but in the meantime there are a few differing opinions making the rounds. All of them come from very competent professionals who have been immersed in the current system, so IMO all of them are worthy of serious consideration. Even though we'll only go with one of the options, a different option might be better suited to your specific needs.
One that is a no-brainer is: SATA. The smaller cables used by SATA are a big win. Their smaller size means (1) better air flow for cooling the disks, and (2) fewer cable-related hardware failures (fewer wires to break, and more flexibility). Very often when Nagios flags a disk as having gone bad, it's not the disk, but rather the cable that needs reseating or replacing.
Choosing the right CPU is important. The VIA C3's we started with are great for low power consumption and heat dissipation, but we underestimated the amount of processing power we needed in our "dumb storage bricks". The two most likely successors are the Geode and the (as-yet unreleased, but RSN) new low-power version of AMD's dual-core Athlon 3800+. But depending on your specific needs you might want to just stick with the C3's (which, incidentally, cannot keep gig-e filled, so if you wanted full gig-e bandwidth on each host, you'll want something beefier than the C3).
It has been pointed out that we could get better CPU/U, disks/U, $$/U, and power/U by going with either a 16-disks-in-3U or 24-disks-in-4U solution (both of which are available off-the-shelf), compared to 4-disks-in-1U (our current hardware). This would also make for fewer hosts to maintain, and to monitor with the ever-crabby Nagios, and require us to run fewer interconnects. Right now it looks like we'll probably stick with 4-in-1U, though, for various reasons which are pretty much peculiar to our situation.
I heartily recommend Capricorn's petabox hardware for anyone's low-cost storage cluster project, if for no other reason than because a lot of time, effort, brain-cycles, and experimentation was devoted to designing the case for optimizing air flow over the disks (and figuring out which parts of the disks are most important to keep cool). Keeping the disks happy will save you a lot of grief and money. When you're keeping enough disks spinning, having a certain number of them blow up is a statistical certainty. Cooler disks blow up less frequently. Most cases do a really bad job of assuring good air flow across the disks -- the emphasis is commonly on keeping the CPU and memory cool. But in a storage cluster it's almost never the CPU or memory that fails, it's the drives.
Even though the 750GB Seagates appear to provide less bang-for-buck than smaller solutions (400GB, 300GB), the higher data storage density pays off in a big way. Cramming more data into a single box means amortizing the power/heat cost of the non-disk components better, and also allows you better utilization of your floorspace (which is going to become very important, if you really are looking to scale this into the multi-petabyte range).
When dealing with sufficiently large datasets, silent corruption of data during a transfer becomes inevitable, even using transport protocols which attempt to detect and correct such corruptions (since the corruption could have occurred "further upstream" in the hardware than the protocol is capable of seeing). We have found it necessary to keep a record of the MD5 checksums of the contents of all our data, and add a "doublecheck" step to transfers: perform the transfer (usually via rsync), make sure the data is no longer being cached in memory (usually by performing additional transfers), and then recompute the MD5 checksums on the destination host and c
Do not use XServe RAID. It's the worst possible pseudo-enterprise SAN product. This is not to rag on Apple in general - the company is full of smart people, many of whom are friends. This is just a lame product in an otherwise excellent product line. There are plenty of SATA based SAN storage devices out there which are cheap. I'm partial to Nexsan, having worked with them, and if you need slightly higher quality the Sun Storagetek, EMC/Dell boxes, etc. Software RAID (Veritas or open source) striping on top of large HW RAID (RAID 5, or RAID 10) SAN storage array stacks works just fine.