Slashdot Mirror
Terabyte Storage Solutions?
DeMechman asks: "As many on Slashdot may know, storage is one thing which you can never have enough of. Given the current situation with CD/DVD rot (Personally I can attest to a 10% attrition rate) hard drives in a RAID configuration seem to be a better and more economical solution. If you own more than fifty CD/DVDs, it can be a daunting task to find a file. I am wondering if anyone has found a hardware solution that can inexpensively be set up to handle 10 or more 250GB HDDs in a RAID configuration. Primarily, has any case manufacturer tackled this niche market yet?"
I bought a case from http://www.servercase.com/, a 3Ware RAID Controller and 8 200GB IDE drives. I've got 1400GB of usable space in RAID5. It runs Linux with Samba and NFS. I also use it for a MythTV Backend.
Unfortunatly, once you have all this space, you WILL find a way to use it all and need more. I put this system together about 10 months ago, and it's at 85% capacity now. I'm preparing to build a new server with 12 250GB drives, to have just over 4TB between the 2 systems.
I'm sure you were expecting this, but their disks are not technically raid as they are not redundant. lose one disk, lose the lot. this means that your data is less safe than it would be just spread across 2 or 4 seperate discs as if ou lose a drive then, you only lose some data, not all.
;)
also, I beleive they don't even qualify as the badly named, raid0 as I under the impression that the disks are concatted together, not striped.
what I'd love to see is an Xraid mini as it were. something with much of the managability of the full size xraid, but not as much redundancy. so perhaps a nice desktop case (to match the g5 *of course*:) that could take 4 or 5 sata disks in hot swap caddies (maybe the same caddies as in the xraid) with a hardware raid controller on board for striping, mirroring and raid 5. a single gig ethernet on the back and then fw400 and 800 ports.
if it had the same cross platform compatibility as the the big xraid, same type of management tools etc, then it could be a big hit, and be an official filling for the big hole that is g5 storage.
sure, the xraid is great and cheap, but it's price of enhtry i still high when all you want is a terabyte or so of fast storage for one of two machines at home, ie no rack to place, no need for redundant psu's and fibre channel connectivity, that kinda thing.
HD video editors esp need something as for the data speeds they need for uncompressed hd (180MBps) thats 4 striped disks which you can't place in a g5 without using third party solutions.
just a thought, come on apple. and when you make one, I just ask for a fully loaded one for myself
dave
I have two systems, with about 1.3 and 2.5 TB respectively for archiving DVD quality video and MP3s. I looked at RAID but found it was not necessary. I prefer to manage the disks (some are removable) and do not need high performance even when streaming the video over my in home LAN.
I use DVArchive with DVD or satellite to ReplayTV for video capture and play back, DVA is great for managing multiple volumes and dynamically discovers vidoes if I want to move them to another drive. It also supports copy/move between the two systems (I use a 1Gb switch between systems). CPU performance is not key for play back though it is critical for transcoding (I use a dual processor system for transcoding and it smokes my single CPU system).
I have a LARGE MP3 collection (forgive me for not publically admitting to its size) and I find the same systems/drives are ample for supporitng a digital audio library. I switched to iTunes for managing music (MusicMatch melts down when the number of files gets large) and stream it with SlimServer to squeezebox devices for high quality playback on home theater and other receivers.
My recommendation is to go with generic disk drives - brand names, 7200 RPM with 1-3 year warranties --I get them locally on sale for under $150, sometimes $130/250GB, thats 52 cents per GB, a little more per GB than a DVD-R disk but more reliable and infinitely more flexible. I can recreate a DVD off of the disk image if needed.
I am more concerned with heat and power consumption (it adds up) than disk performance, someone will need to explain to me why I'd need to mess with RAID for this...
The cheapest way I know is still a large PC tower case, a 3ware SATA raid controller a big PSU and a couple of large fans withs attitude. On the PCI side you don't need much unless you want to do gigabit (or of course just shove your server in the same case and dont do the I/O networked)
I don't understand why people automatically assume that DVD+-R/RWs are prone to the same problem. The recordable layer in these disks is buried in plastic. It's NOT on the surface. There's no oxygen coming to it, so in theory DVD+-R/RWs should be a heck of a lot less prone to "rot".
Yeah--oddly though I'm getting better Bonnie results by using Linux RAID 5 than their hardware RAID 5. But it is possible (just) to stream full-size, full-framerate PAL video to 'em over NFS! (sustained 40MB/s). Anyway in software you can now do RAID 6 :)
The reason why you're getting better RAID 5 results from software RAID vs. hardware RAID is because of the parity calculations involved with writing to a RAID 5 volume. On a hardware RAID setup, these are calculated on the RAID card itself, which probably has a 200 or 400 mhz. chip that does these calculations. Back when CPUs were only 400 mhz, this was great, because there was no load put on the CPU, and the RAID controller worked just as fast or faster than a software RAID setup. Now that CPUs are 3 ghz. +, there's no way a dedicated hardware RAID card can keep up, and unless you're running a huge load on the server, youv'e probably got 1 ghz. or so of free CPU bandwidth to burn for software RAID...
Want to see the performance really increase? Give up RAID 5 and go with a real RAID solution like RAID 1 or RAID 1+0.
"When the president does it, that means it's not illegal." - Richard M. Nixon
The reason they gave is that the even a fraction of modern CPU performance still far outclasses the chips on hardware RAID cards. Also, data cached on the card still has to go over the PCI bus, but data cached in RAM... well, it's already available.
A RedHat employee who was there confirmed that RedHat has seen the same thing in their own testing. For performance go with software RAID. With anything over about a 800Mhz CPU, you would be hard pressed to notice the CPU use.
In fact, unless you are doing something that is virtually entirely computational like SETI@Home, you are going to be generating a fair amount of output. Enough that the faster disk IO actually increases your speed more than what would be gained by moving the RAID load to seperate hardware. It also lets you spread disks over a couple SATA controllers and potentially multiple PCI buses (if your MB supports it.)