Slashdot Mirror
Best Solutions For Massive Home Hard Drive Storage?
i_ate_god writes "I download a lot of 720/1080p videos, and I also produce a lot of raw uncompressed video. I have run out of slots to put in hard drives across two computers. I need (read: want) access to my files at all times (over a network is fine), especially since I maintain a library of what I've got on the TV computer. I don't want to have swappable USB drives, I want all hard drives available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be moderately quiet and/or hidden away somewhere and still keep somewhat cool. So Slashdot, what have you done?"
Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
My personal storage solution consists of a 4U rack case with a computer with a c2d CPU, gig-E NIC, a few gigs of ram, a bunch of 7200 RPM disks and FreeBSD on the system disk (I also have the system disk mirrored just in case). All the storage disks are then pooled using RAIDZ. Pretty simple yet powerful. Just don't expect too much in the way of performance.
Greylisting is to SMTP as NAT is to IPv4
I personally set up a downloadserver that also functions as a media server to stream the content to other devices. I put in a couple 5400 RPM 1,5 TB drives, they use less power, generate less noise and heat than a regular 7200RPM drive but since you're not running any applications of them, you won't really notice the difference in performance. Prices have gone down a bit so the sweet spot for $/GB might be at the 2TB mark now. If you don't want to go for an entire computer, maybe a NAS solution would be best for you, with the same 5400RPM drives. A NAS will have less room for the disks if you really want *massive* amounts of storage, and also you usually must purchase one + the disks. The PC you can build from spare parts lying around. I personally put gentoo linux on mine, but you also don't exactly need top of the line equipment for a nice windows XP install. The NAS however will have outputs directly for your TV and will take up less room and power.
Still, the key is 5400 RPM + 1,5/2 TB.
Good fucking god, $700 for the Drobo FS?
You could build a capable home server box AND buy some of the drives for that much.
Mod me down, my New Earth Global Warmingist friends!
I've tried a variety of approaches, but overall I've been happiest with just buying a NAS box.
I have a Synology DS209, and I've been very satisfied. It's a relatively cheap way to get 2 TB RAID 1 storage with really simple backup to an external USB drive. If you need more storage, you can buy NAS devices with more than just two bays.
This. Even ready-made resellers have pretty small devices built on Win Home Server that can take a LOT of drives. Mine supports 4, but there's many models that can take 8, 12, or more drives. The OS is rock solid and has a lot of neat features, like being able to access your network from an SSL secured web app (built in) from anywhere with indexed search, and its easy to develop plugins for (though there's a ton available already) to extend it.
You made no mention of a budget. I'd go with a Drobo - probably the DroboFS. http://www.drobo.com/products/drobo-fs.php
Informatus Technologicus
I have two of these servers now. Each server can hold as many as 16 disks (possibly more actually as the programmer keeps bumping that up) with one disk reserved for parity. Data is NOT striped and parity is ONLY stored on the one drive. If a disk fails I lose no data, if two fail I lose two disks of data but nothing else. No hot spares or any other crap. If a disk isn't being used it goes to sleep and saves me heat and power. Disks can be ANY size but the parity disk must be as big or bigger than any of the data disks. Runs on a pretty decent selection of hardware although keeping the list of what works and what doesn't up to date is apparently tough since hardware changes so fast. It's Linux based but pay for play, yes he's followed the GPL. It's not super expensive and it boots from a USB drive to be web administered. I use full tower cases with SuperMicro 5n1 trays, 2gig of memory, Celeron CPU, power saving PSU, and supported mobo that have onboard video and GigE which you WILL need.
Their forums are a big help and active, users are working to expand the capabilities of these NAS and the programmer is working on making that easier too. Check it out, I've not found anything better yet and with some of the newer versions of SAMBA in the code it's pretty fast too! Perfect for a HTPC but not so great for a big transactional database
http://www.lime-technology.com/
Build it, Drive it, Improve it! Hybridz.org
http://en.wikipedia.org/wiki/ATA_over_Ethernet
This is something I've always wanted to play with. It's a little expensive (for a home user) to get into, but it's extremely scalable. If I moved all my DVDs and such to on-line storage, I think this is what I would opt for. It can be run in all sorts of RAID configurations, doesn't require matched sized hard drives, and it can all be racked up very nicely.
4 drive bay, USB, FW400/FW800 and eSATA. Will take 2tb drives, RAID 0, 1, 5 and 10. Comes pre-populated or unpopulated, the latter is what I got and added my own drives. http://www.macsales.com/ No financial connection, just a satisfied customer (they have great tech support!)
This is obviously not a build-it-yourself storage array, but is a good option if you want a commercial out of the box solution.
Even if you "need' a 16000 RPM drive, just make it for your local drive that you play your videos directly off of. Use 5400 for all the other ones. Just move your file before watching it. Sure, if you're an impatient baby and want to watch something within 5 seconds of it entering your mind, then you might have to wait 5 minutes if the file is 4.5G. Then again, it's the type of waiting you can go pee or make your snack during.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
If you want cheap, affordable storage get:
A decent full tower case with a modular PSU
A motherboard with 8+ SATA ports (cheap)
A 4-port SATA expansion card
=
12 SATA slots + 12x SATA power for cheap
Get a cheap bunch of 1.5 TB drives for up to 18TB total. If you say home I assume you don't mean 99.9% redundancy. You can buy a new PSU or motherboard or whatever and have it delivered and that's okay. Softraid two drives in RAID1 for 1.5 TB less storage. If you need more protection then upload it to some offsite backup - any external disk or second machine is still vunerable to theft, fire and whatever. It works for me, though I only have ~10 TB due to due of old low-capacity disks.
Live today, because you never know what tomorrow brings
I've got room for 30TB of data storage in two machines for a total of 60TB. However I have only populated them to around 12TB right now, I don't add drives till I'm out of space! :-) Not what I would call massive yet but getting there!
Build it, Drive it, Improve it! Hybridz.org
There are large capacity "green line" drives from some manufacturers, 5400 RPM, that might be perfectly enough.
Do they work in RAID? or do they randomly stop responding to the raid controller and then get dropped from the raid, triggering a rebuild, to show up a few minutes later, to trigger another one? http://en.wikipedia.org/wiki/TLER
All of the above was encrypted with a Quad ROT-13 method. Unauthorized decryption is in violation of the DMCA.
you forgot to mention that it can take any sized hdds, you don't have to match them in size or brand, and if the whole hardware fails the files are just stored in a regular ntfs partition. So you can just mount it on a normal windows/linux/etc box to grab files off the drives. Very cool. That and its easy to migrate drives between different hardware since its a software based solution.
Check again: you're almost certainly comparing 1TB 7200RPM drives to 2TB 5900RPM drives. And Hitachi drives don't count, being the cheap pieces of garbage they are.
When it's going to be used by only a handful of people, nearly always in a sequential access pattern, on the other end of a 1GbE link, why would you want hotter, noisier, 7200rpm drives ?
Check again: you're almost certainly comparing 1TB 7200RPM drives to 2TB 5900RPM drives. And Hitachi drives don't count, being the cheap pieces of garbage they are.
When it's going to be used by only a handful of people, nearly always in a sequential access pattern, on the other end of a 1GbE link, why would you want hotter, noisier, 7200rpm drives ?
Because sadly, that is probably speculative on your part. Certain file systems, even with tons of free space, will fragment files that are in the low megabyte range. I suspect fragmentation gets even worse on the large files the OP is asking about.
StarTrekPhase2 - The Five Year Mission Continues!
I'm in the process of building a 5-bay SATA port-multiplier solution right now. What I've learned thus far is:
* Most commodity motherboard chipsets don't support port multipliers. You'll need an expansion card.
* If you have this much data, look into ZFS and RAIDZ2 for reliability. Avoid RAID5.
* The bigger the disk, the longer it takes to rebuild a degraded array
* FreeNAS is at an inflection point. If you're not scared, use PCBSD directly instead to serve your data.
* You don't need "enterprise-class" storage speeds to serve up movies and media. Slow, green drives are fine.
* Don't buy all of your drives from the same lot, all at once.
Cheers, and have fun in the process.
I've posted this idea before, but I would like to see a harddrive jukebox, so you don't have 20 drives running 24/7, just a couple.
Basically your could plug 20+ hard drives into what amounts to plastic holders for the harddrives.
You then have a jukebox/cataloguing software like DVD jukeboxes or some sort of virtual drive that keeps track of all the files on all the drives. When you need to access a file, it powers on and spins up the correct drive. Ideally two or three drives could be activated at once. The Jukebox software would automatically handle all of that.
As far as how the connections were made: You could do it a couple of ways. Have each drive with its own SATA connectors that all fed into an electronic switching hub that handled activating the drives. Or you could have it be a phsyical/motorized scenario, where each drive plugged into a custom SATA header that then interfaced with a motorized SATA connector to attach to a specific drive.
This has already been done with tape drives & DVDs obviously, but I am talking about something CHEEEP (the extra E is for extra cheap).
http://blog.slaingod.com
Mapping drives to drive letters is so 1995. You should be mounting them to a single folder (each drive is a subfolder) called c:\mnt, and then just sharing your mnt folder on the other computers. You can do that in disk manager on windows (since linux you would already be doing that presumably) instead off mapping the drive to a drive letter.
There are some quirks in certain copy programs if you are moving files from one mapped drive to another and the copy program isn't 'mounted disk' aware. Nothing to worry about, but you might not get progress bars for moves, for instance.
http://blog.slaingod.com
I've got a $20 case, $50 500W power supply and $40 motherboard with SVGA and ethernet, its 5 PCI slots each stuffed with $25 4x SATA cards, 20 $100 1TB HDs. Running Linux, network mountable drives and ssh login.
That's $210 PC + $2000 HDs for 20TB. That's a lot of porn storage for you.
--
make install -not war
Thecus is really an awful suggestion. I own the N5200 Pro and can with certainty say that the device sucks. The web UI looks like an old version of what you find in a Linksys router, and its not pretty. It keeps kicking disks out of the RAID even though they're on the supported hardware lists. Emailing customer support never gives you an answer. Opening a ticket in their issue tracker never gives you a response. Forum is full of complaints. I'll never buy anything from them again.
I've since purchased a Synology and it is much faster, has a modern and feature-rich web UI, plus it actually works.
Yes, but using NTFS would be a bad idea for this, far from "best solution" territory as mentioned by the submitter. For a massive home storage system I wouldn't recommend using Windows for the server. Set up a gigabit LAN and a samba file server with the multi-TBs of locally attached storage drives. If you want RAID, use software RAID. Add a UPS, configure NUTS, configure hardware monitoring, smartmontools and RAID monitoring (if you have RAID).
:).
Yes it's a fair bit of work that seems unnecessary when you could just buy a NAS device, but AFAIK most don't do a full self-test at 3 am in the morning and send you an email when one of your drives fails with a CRC prob. Nor would they do an orderly shutdown when the UPS runs low on juice.
And this is Slashdot of course. If it were some other site, I'd suggest a NAS device
I'd consider 1TB small today. And this guy probably as well; You can't put a lot of 1080p movies on one 1TB disk.
My own setup is a box that has two thecus N5200Pro NASes NFS mounted. One has 5x1TB, the other one has 5x2TB. Both are RAID-6 arrays. I know I throw away 6TB of storage but I'd rather spend a couple extra bucks than loose my episodes of Dharma & Greg.
If something goes wrong on the 5x2TB array I'm up for a 2 day array rebuild though, praying no other disk fails as well. The newer Thecus NASes have zfs and shouldn't have this problem.
Please login to access my lawn
This was a solution created for a customer. It works well, it scales well and it is as cheap as the hardware ou need to buy.
You need to get a PC with a mobo that supports VMDirectPath and has several PCi-e available. Then, install VMWARE ESXi 4 to a first small hard disk (and old one, for instance).
You can then buy an external enclosure or just create something yourself and you just keep on adding HDs and SAS or SATA controllers (wichever you like).
(Take care to avoid HD vibration, tough - It's bad for a HD's performance.)
Make your first VM Open Solaris and give direct access to the controllers (and consequently the disks) to that VM, and make sure you give it CPU and RAM priority.
You can then setup a ZFS filesystem using RAID-Z or RAID-Z2 (much better than traditional RAID 5, etc). Solaris and ZFS make it easy to make it available as NFS or SMB or iSCSI.
You should worry about speed and the bus speed limitations as you add controllers and disks, but on the other hand if you need more availability than speed this should not be a problem.
Basic hardware to build this should be no different than your average home computer, and you can use any mb, if you'd like, instead of one that supports VMdirectPath, by using raw device mapping, but you should take into account that it is not supported by VMWARE.
Also, because Open Solaris is given direct access to the HDs and ZFS is thus created directly on the hds, if you plug those disks into another PC or VM running OpenSolaris, you access all data on your array with a single command line - Hardware Agnostic. (Usefull if your HD dies).
This has the advantage of letting you raise whatever other VMs you may want. Of course, you can do this, installing open solaris directly. ZFS is the way to go.
Note: You can use Nextenta Store free version if you plan to use store than 12TB of Data. Easier to set-up, but same principles.
I have a similar setup to previous post. Server running Linux with a 3ware RAID card running RAID 5. 8 500GB disks with ~3TB of usable disk space. This has been running flawlessly for over 5 years. I have a a movie collection, music and network share for 3 HTPC in the house. Works very well but I wish my RAID card supported the ability to power down the disks and save of power/heat when not in use.
I am almost at capacity on the RAID volume, so to expand I have another RAID card that I can put in the server and create a new volume or replace the 500GB drives with 1TB or now 2TB disks. Replacing the disks would save power and heat, but I would need to backup and restore 3TB of data. Adding another RAID card is easy, but crates more dives that I can't turn off and eat up power.
I am actually thinking about building a new server with the thought of being able to add an many SATA ports as possible (via SATA cards) and then us port multipliers. The use a software based file system or RAID that allow me to add drives of different sizes to the volume. Similar to ZFS but more open. This would make growing the system much easier and allows me to power down drives when not in use. I would still get plenty of performance for my needs.
The other thing that I am doing that most people don't think about, is I backup my entire NAS to another server. I took another old PC that I had and put a 4 port SATA card in it and four 1 TB disks and run Linux and software RAID on it. Each night it powers up and runs a script to back up the primary NAS. I do this just in case something catastrophic happen to my primary NAS and I also use it when I moved to larger disks on the NAS previously. I use rsnapshot to look for changes on the primary NAS' file system and only back up data that has changed. It also keeps the lat 3 months of files that have been changed or deleted, if I need to recover something. When the script is finished, it powers down the backup NAS and wait until the next night to run again.
I've used "real" hardware raid twice. Both times the hardware died (after about 5 years though). Both times I could not get a compatible controller. All data lost.
I currently run 3 separate Linux software raids. I've never had an issue with it. I've performed at least 4 rebuilds due to hardware failure, each time it was perfect. It runs faster than the hardware raid. It consumes a whopping 1-2% of CPU time on a 2Ghz AthlonXP. I've lost motherboards that the systems have been on. I've changed system architecture. I've replaced single IDE disks with SATA. And it cost me 30 minutes of tutorials.
Hardware raid is for idiots with too much money.
This thread has lots of good suggestions for storage. I have a distant business relationship with Drobo, and think they're an interesting choice. I have a Windows Home Server as well, and find it to be a step-up from my previous Buffalo Terastation NAS box from a reliability and performance standpoint. I happen to also have a full-size tower and appreciate the simplicity of throwing lots of hard drives at the problem.
However, as formats switched from DVD to Blu-Ray and equivalent HD content, the economics shifted IMO as well. Given the both my Sony TV's, as well as both Tivo's can stream on-demand HD directly (Sony does it without annoying buffering by the way), and considering that I only rarely watch a movie more than once or twice, it's actually become more economically feasible to simply rent HD on demand for $4.99 a shot. There are still going to be a few Blu-Ray discs worth buying to own the content, and more than half of those seem to ship with a free digital copy for import into Window Media Player or iTunes. Even if you own all your HD content on disc now, it's probably worth your while to look into a hybrid model where you rent what you have a passing interest in, and buy/store those few things that either aren't available on demand, or that you have a more long-term interest in retaining.
Oh, and for the porn, a 2TB drive in a large tower should be more than sufficient. Windows 7 and Bit-locker full drive encryption doesn't impede HD playback on a reasonably speedy system. Although it is debatable whether or not 1080p is actually a good thing in some cases. Anyone want to go into business with me creating a unique line of porn-star body make-up to deal with pimples, waxing irritation, and razor-burn?