Slashdot Mirror
Terabyte Storage Solutions?
DeMechman asks: "As many on Slashdot may know, storage is one thing which you can never have enough of. Given the current situation with CD/DVD rot (Personally I can attest to a 10% attrition rate) hard drives in a RAID configuration seem to be a better and more economical solution. If you own more than fifty CD/DVDs, it can be a daunting task to find a file. I am wondering if anyone has found a hardware solution that can inexpensively be set up to handle 10 or more 250GB HDDs in a RAID configuration. Primarily, has any case manufacturer tackled this niche market yet?"
It's not RAID, but you could buy a 1-terabyte drive from LaCie.
In soviet russia, You ask not what country do for you, but what you do for country!
Oh wait...
you can "cheaply" buy 3U rack mount cases that hold 15 drives in hotswappable SATA or SCSI cages up front. Combined with a 3ware 9500-12, and leave 3 cages empty(or spare drives just not cabled up), this will give you 2.75 TB in each unit of raid5 storage. If you were really hard up for space, you could use a pair of 9500-8's and this would give you 3.25 TB per unit. Some 4U units hold 16 drives, which gives you the full 3.5TB in 2 x raid5 arrays.
What a rip off!!!
Go buy a Lian Li case, 8 x 200gb maxtor harddrives and a 3ware raid controller.
Controller $500
Drives $150 each
Case $150
Total for 1.4TB = $1850
With 400gb drives maybe $3000 for 2.8TB
I bought a case from http://www.servercase.com/, a 3Ware RAID Controller and 8 200GB IDE drives. I've got 1400GB of usable space in RAID5. It runs Linux with Samba and NFS. I also use it for a MythTV Backend.
Unfortunatly, once you have all this space, you WILL find a way to use it all and need more. I put this system together about 10 months ago, and it's at 85% capacity now. I'm preparing to build a new server with 12 250GB drives, to have just over 4TB between the 2 systems.
> Anyone have any ideas on how to back up 1TB in a home environment? i.e., not $3000 tape drives & $200 tapes
Ummm, yeah, it'll cost you ~$600. make another one and make a copy occasionally...
Sorry, couldn't resist...
- Find any tall beige-box case. ($150)
- Find 9 good 250g Serial ATA drives. ($100 each = $900)
- Get an 8-port serial ATA hardware RAID controller like these ($300)
- Get a good 400-500W power supply ($200)
- Any motherboard and CPU will do ($200)
- Spend a few extra bucks on gigabit ethernet ($50)
Put 8 of the hard drives into a RAID-5 array. (1 for your O.S/system use). That makes about 1.4 TB for only $1800 total. The 3Ware IDE raid thing works great with FreeBSD, which is what we use for everything.Rip all your CDs as FLAC so that (1) you never have to rip them again (it's lossless), but (2) it's half the size of saving WAV files
At least that's what we've done with our 68,000 CDs we have here.
A way of doing it (Which I did)
0 = 8* $170 250GB ATA drives.
8 Firewire drive enclosure: (i have the 4 drives version).
$600. http://www.cooldrives.com/fi80013oc5fi.html
$136
$700 = Hardware for a Linux machine as correct file server
= $2930 for 2TB of raw space, 1.5TB Of raid 5 with an hot spare, or 1.75TB of raid five with no hot spare.
You got yourself a nice fileserver for home usage... install that with mythtv and you're up for hours of video....
Some people have had a surprising level of success uing the software raid potential of Linux to do this for some time, getting prices as low as $0.60US per GB.
Some slashdot articles on some previous attempts:
Bulk Data Storage For The Common Man?
Home-brewing a 1.2TB IDE to Firewire Monster
Books on it:
Managing RAID on Linux
Even applicable controller hardware:
LSI Megariad 150-6
3Ware 9000 series
And soon to be applicable storage hardware:
Hitachi Announces 400GB Hard Drive
One key thing to add, when building a mass storage system *always* buy drives from different lots. Drives in the same lot will often fail very close to the same time, so spreading your your expected drive failures by buying different lots is a very good idea. Buy drives from multiple vendors and even manufacturers if at all possible.
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
I can attest to this:
/home volume on a 3ware controlled array. Sometimes, we get those users that decide they need to write out their incremental data sets across the NFS mount... from 48 nodes. Sure, a parallel file system would be great, but from what we've seen, only GFS was close to production quality (and they just recently gpl'd it).
Our 48 Node beowulf has a
Anyway, that kind of load brought that head node (dual proc 1700+ MP) to its knees until we decided to rebuild it. Moving from the hardware controlled raid to linux's software raid completely resolved that problem.
Controller $500
Drives $150 each
Case $150
Redundant Power Supply for RAID Array.....priceless
cat
Unfortunately, the earlier 3ware cards won't allow you to build an array unless all the drive IDs match EXACTLY, meaning that this is not possible.
Hopefully, they've changed this for the newer 7 series cards, but the 5 series are 'broken' this way.
Oh, bullshit.
Linux software RAID1 is just as fast as several of the hardware RAID1 setups I've tested using Bonnie++ -- These are fucking fileservers, not renderfarms. The processor's sitting there doing jack shit anyway, and you're more than likely putting a P4 in there since you can't buy anything else with decent reliability. Throw in a decent GigE network card and your processor is STILL at 0% utilization. Make that a RAID5 with hot-standby drive and I would be very surprised if you noticed any difference in the apparent "feel" of the server compared to a hardware RAID solution.
Hardware RAID's okay but now you've got a proprietary format array with a SPOF (the RAID card(s)) -- sure you can keep spare RAID cards around but honestly unless you need every last bps on your network transfer and you've got your server so overloaded that SW RAID is impacting your performance you're just incurring extra expense. I am very happy that I can take any RAID array I have and throw it in another system should a motherboard or controller fail and I need the system up immediately. I'm very happy that LVM Just Works and works happily on top of software RAID. There's no issues and no extra question marks like there are with any hardware RAID "solution".
Want beeping? Write a script. Want email/phone/paging when something goes wrong? Write a script. Or use any of the monitoring and alerting systems you can find on Freshmeat (mon, nagios, etc.). Jesus H Christ, give your head a shake.
Oh wait, you're trying to build a performance system using an OS built for pushing pixels. Perhaps that is your biggest problem. Windows has its place, but high performance data transfer just isn't one of them. I guess if you've decided to spend a couple hundred on an OS license that gets you nothing you may as well blow another couple hundred to get hardware to go with it.
Apple has claimed that they do pick their drives from different lots. Atleast, that is what I've heard from insiders.