Best Solutions For Massive Home Hard Drive Storage?

i_ate_god writes "I download a lot of 720/1080p videos, and I also produce a lot of raw uncompressed video. I have run out of slots to put in hard drives across two computers. I need (read: want) access to my files at all times (over a network is fine), especially since I maintain a library of what I've got on the TV computer. I don't want to have swappable USB drives, I want all hard drives available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be moderately quiet and/or hidden away somewhere and still keep somewhat cool. So Slashdot, what have you done?"

Comments

Define "massive"

[jschen]

How much data constitutes "massive"?

Re:Define "massive"

[svirre]

My pricing indicate 2TB disks are slightly cheaper/GB than 1TB

Re:Define "massive"

[Mr_Insightful]

How much data constitutes "massive"?

640K of memory should be enough for anybody.

Re:Define "massive"

[digitalchinky]

For most people 1TB is pretty big.

To answer the question, NAS boxes. They are moderately cheap, come with numerous drive bays, they are usually small, quiet, and unobtrusive when you stick tape over the blue LED's.

Re:Define "massive"

[Anonymous Coward]

Except that this is completely untrue. Even Windows 95 scanned the table of free blocks for a reasonable area of consecutive free blocks.

  HKEY_LOCAL_MACHINE \System \CurrentControlSet \Control \FileSystem
DWORD ContigFileAllocSize

http://technet.microsoft.com/en-us/library/cc768196.aspx

Re:Define "massive"

[BikeHelmet]

When I hear a question like this, I usually recommend heading over to the NCIX forums. There's some crazy guy over there - death_hawk - building a 100TB array.

What I did was a bit less ambitious. A regular old NAS running off a cheap non-RAID SATA card in a case with lots of HDD bays.

For interest, I'll throw up a build that easily scales to 12TB. Since you mentioned noise, I'll prioritize that instead of capacity. I'll use a case geared for silence, a fanless mobo/cpu, a quiet PSU, WD Green HDDs, and a ridiculously cheap SATA card.

Case - 8 bays: http://www.ncix.com/products/?sku=51277&vpn=6900654&manufacture=Fractal%20Design *1
Motherboard/CPU - Silent: http://www.ncix.com/products/?sku=50891&vpn=AT5NM10-I&manufacture=ASUS *2
DDR2 - 1GB: http://www.ncix.com/products/index.php?sku=18584&vpn=VS1GB667D2&manufacture=Corsair&promoid=1114 *3
PSU: http://ncix.com/products/?sku=33357&vpn=CMPSU-400CX&manufacture=Corsair&promoid=1114 *4
SATA Card: http://ncix.com/products/?sku=19892&vpn=SY-SA3114-4R&manufacture=Syba *5

HDD - 2TB 4KB http://ncix.com/products/index.php?sku=49591&vpn=WD20EARS&manufacture=Western%20Digital%20WD&promoid=1114 *6
HDD - 2TB 512b: http://ncix.com/products/index.php?sku=36130&vpn=WD20EADS&manufacture=Western%20Digital%20WD&promoid=1114 *7

OS: FreeNAS, Ubuntu, Win7, Other *8

*1 Only six will be filled. 6 SATA ports.
*2 Case still requires fans/airflow.
*3 A NAS probably only needs 512MB, but 1GB is cheap. A Win7 NAS may benefit from 2GB.
*4 Must be capable of spinning up 6-8 HDDs at once.
*5 Must be flashed with new non-RAID BIOS to avoid silent data corruption for > 1.0TB HDDs; disk read/write speeds around 30MB/sec, in my experience, on ext2. (but running with a VIA CPU - not dual-core Atom)
*6 Must be specially formatted under Windows and Linux. (Most distros only support 4KB sectors when the drive reports 4KB - these report 512b to maintain XP compatibility)
*7 May have longevity issues. (too early to say right now - lots of complainers, which reminds me of the 7200.10 days. A heck of a lot of those chirping barracudas perished early)
*8 Please verify SATA card support first. Ubuntu and FreeNAS work fine with this card, but I've never checked if Win7 has drivers. Do note that you'll have to flash it. *9 If that's a problem, buy a more expensive card. (which may give better performance, and SATA2 support) Promise makes nice non-RAID SATA cards.

*9 Flashing the PCI SATA card requires making a DOS boot CD: http://www.hiren.info/pages/bootablecd

Please note: A solution like this will take 12+ hours to set up. It's highly likely you'll blow a whole weekend, even if you know what you're doing. You may have to try multiple distros to get proper Atom D510 support, unless you go with Windows. When I put mine together, atoms weren't available affordably, so I went with a cheap VIA board. Ironically, Ubu

Re:Define "massive"

[drinkypoo]

Does using RAID controllers actually provide superior price:performance to using software RAID? Last I checked, the processors on most cheap RAID controllers were slower than dogshit and using md under Linux would give you better performance than basically any of them, at the cost of some CPU. But since CPU is cheaper than RAID, it probably makes sense. For example, going from a Phenom II X3 720 to a Phenom II X6 chip of the same clock rate takes the CPU from $100 to $200. How much would it cost to go from four crappy RAID controllers to four good ones? It would probably cost you at least $400.

The answer is probably to just go ahead and install Debian on a machine with as many CPU cores as you want to blow money on, and to use software raid. Put lots of system RAM in it, which the OS will automatically use for disk buffers. Current versions of grub work fine with USB keys, because they can use UUID for the groot, and the UUID never changes. If you want it to boot quickly, find a motherboard with coreboot support. If you want external disks you can use firewire cheaper than eSATA, if you get the external disks or just some enclosures at a good price. It makes maintenance a lot easier, but involves substantial power waste due to all those inefficient wall warts.

P.S. OpenSolaris is circling the drain, please don't suggest it to anyone for anything.

Something like this

[fotbr]

Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Re:Something like this

[Anonymous Coward]

Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

If possible use something like ZFS (or btrfs if you feel confident about it) so that you get checksumming data protection.

If you're going to put all your eggs in one basket, you better watch that basket very carefully.

The creators of that kit don't use any kind of redundancy with-in the box because their custom software stack handles replication (kind of like Google FS / Hadoop FS).

Paranoia

[Anonymous Coward]

So Slashdot, what have you done?

Why? What have you heard??

Cheap solution

[tomhudson]

1. create a million gmail accounts (or buy them from spammers in bulk at 3 for a penny
2. link the accounts as one big drive
3. 640k petabytes ought to be enough ...

Why do you need them available at all times?

[VinylRecords]

I used to work for ABC news and we never kept archive footage always accessible like you want. If we wanted something that was really old we'd have to dig it off a tape, an unplugged hard drive or powered off computer, or we'd have to find another news agency that had the footage and grab it off of a satellite feed. And this was a 24/7 TV news station responsible for national news programming where we would be tracking stories for years. If we didn't need a system where everything was instantly accessible then you needing it on an individual level might be overkill in my opinion.

I have over 30TB of music, movies, and raw video footage on my home computers and I just keep everything on separate external hard drives. I label the drives, back them up twice each, and then keep an index in a .txt file that is easy to search through. So if I want a 1080p backup copy of Blade Runner I search 'Blade Runner' in the .txt file and I see it's on drive 'A' and then I plug in drive 'A' and dump the movie on my computer. I also keep an external drive that has backups of every TV show I own on DVD. So if I want to watch The Wire then I plug in the external drive labeled 'TV' and have at it.

Re:Why do you need them available at all times?

[KahabutDieDrake]

First off, judging personal wants and needs by the way a giant corporation acts is hardly reasonable. ABC has cost/benefit to consider when trying to keep data available, and it's probably easier/cheaper to do it the way you say they do, rather than implement a fully digital, fully available storage system.

That being said, the solution is SIMPLE. If you have a bunch of hard drives with data you want, you put together a low end PC, install it into a server case, and fill it with hard drives and SATA controllers. When it's full, you build another one. You have 30tb of data, mostly not accessible. I have 10tb of data accessible from any internet connected computer on earth, and it's twice as much storage as I actually use. It cost me about 500$ to build and deploy a personal storage server, and it doubles as an HTPC. ( I already had most of the drives, and some parts) It's likely most people here have enough hardware laying around to implement a basic storage server. There really isn't any reason not to do it. As a bonus, since it's not a machine you need to access directly most of the time, you can hide it in a closet and forget all about it.

Sure, you could buy a premade NAS/SAN or stand alone data box. However, they are costly and not any more suited for the job than an old machine, or low end new system. At least, not in a personal environment. If you actually require robust data storage, I'd suggest a NAS, from any number of sources. But now we are talking about 4k worth of hardware, and requiring proper power systems to be added if you really want longevity out of it. However, that's overkill for a home storage solution, no matter how much data you have. Simply because you don't need enterprise class data serving, when only one or two computers are accessing the data.

If you don't know how to build and deploy a system with lots of drives accessible over a network, then you probably started at the wrong website for help. You want DELL/HP/IBM small office sales line.

Re:Why do you need them available at all times?

[clarkn0va]

I see you've never tried to maintain a ratio on a private tracker.

Solution

[Anonymous Coward]

Don't worry, we'll be right over and take care of everything. You'll never have to worry about it again.

MPAA

P.S. My sister, Riaa wants to know if you're into MP3s

SATA port multipliers

[the eric conspiracy]

SATA port multipliers - 5 to 1 for about $50 + 5 2 TB gives you 10 TB off 1 SATA port.

Re:SATA port multipliers

[EdIII]

They work, but will slow the system down considerably. If you connect 5 drives up to 1 multiplier, the total speed you will get is the same as 1 drive hooked in directly. In otherwords, a bottle neck.

So technically it is possible to hook up 250 SATA drives into a single SATA RAID card, but you are not going to be that impressed with the performance.

Re:SATA port multipliers

[Kjella]

Well, from what I understood there are two modes - one which will give you only time slots so 5 drives each get 1/5th of the time. That's the cheap variety. The other variety is traffic based, you can't exceed 3 Gbps but you can get the cumulative read/write speed up to that point. The SATA spec site has more.

Sounds like one hell of a porn collection

[Trailer Trash]

/. is definitely the place to ask...

WHS

[Barny]

Windows home server, 1TB 7200rpm main drive with seagate LP 5900rpm drives, lock it away and never have to think about it till you need to drop another drive in.

The reason for the fast main drive is that with WHS when you copy data to it, it stores it on the main drive first, then schedules it to be distributed out to the storage drives the next time a "storage balance" is done.

Works fairly well, its based off windows server 2003 at the moment, but if you can wait till the end of the year they have a server 2008r2 version coming out soonish.

pervert

[Cyko_01]

I download a lot porn, and I also record a lot of masturbating videos. I have filled two computers with porn already. I want access to my porn at all times, especially since I maintain a porn site. I don't want to have swappable USB drives, I want all my porn available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be disguised or stashed away and not overheat. So Slashdot, what have you done?

Re:Look at the DroboPro

[Firehed]

Having done that in the past, I'll say that buying a Drobo was worth the cost. Granted, I hunted around a bit to get a good sale price (it's not too difficult... though the FS is brand new so maybe not on that model yet), but unless you really enjoy tinkering with getting samba shares set up and working properly, sometimes it's just easier to buy your sanity.

Don't get me wrong - I wish they were cheaper. But their system worked better and more reliably than anything I ever put together, and I'm by no means incompetent. And their BeyondRaid tech, while proprietary, is pretty damn cool and works incredibly well. Being able to mix drives and not waste tons of storage space is a huge advantage that (as far as I know) I'm not going to get anywhere else.

Just a happy customer, not an employee or anything like that.

Re:ZFS

[0100010001010011]

ZFS + Solaris.

I have a standard ATX case with 4-in-3 adapter from Newegg. I didn't get the more expensive ones with trays because I didn't need to hot swap.

I have 2x1TB drives in ZFS mirror for boot. 5x1.5TB drives in RaidZ as a tank and 2x200GB drives in mirror with a virtual block device for Xen Debian and Windows 7.

OpenSolaris is amazingly simple to use, if you're just doing your home network.

At the most basic level:
zpool create tank c5t0d0s0 c5t1d0s0
zfs sharenfs=on tank
zfs sharesmb=on tank
zfs shareiscsi=on tank

Now your new drives are all shared over NFS, SMB and iSCSI.

I keep looking for the old school full height 'desktops' at a bargain store or so. Search newegg. 3.5" external works just as well as internal.

This has 11 3.5" bays and 3 x 5.25" bays. With a 4 in 3 linked above you could have 15 hard drives in a case for $100. Or if you care about hot swappability This one has 20 hot swap bays (at 3x the cost).

If you want more performance, get some SSDs to work as the ZFS "cache".

Re:This should be modded up

[InsurgentGeek]

A contrary opinion. I have had a Drobo since the original release and it has been nothing but a disappointment. Drive incompatibilities, an extraordinarily high drive failure rate (at least 1/quarter)and a very confused partitioning scheme. Not something I'll repeat in the future. Oh, and data loss that had to be corrected via a firmware update. In short if I'm spending the money for Raid - I don't want to lose data. Period.

Re:Look at the DroboPro

[AmaDaden]

I have a Drobo and a DroboShare. The DroboShare runs a slimmed down version of Linux so a network attached Drobo uses typically uses samba. The benefit of having one is NOT the ease of software set up. The reason I love it is the ease of drive management and small hardware size.

Due to the small size and slick style I keep mine in my TV cabinet. I've done the measurements and no PC case on newegg can fit in this same space, never mind something that can house 4 harddrives.

The other thing that is so valuable about a Drobo is how well it manages it's RAID array. They call it BeyondRaid but I hear it's just a as many normal RAID arrays as it needs to organize the drives to both optimize space and maintain redundancy. Also you can pop harddrives in or out while it's on and it will automatically restructure the RAID on the remaining drives to still be redundant with out any need to shutdown or stop sharing data. I recently needed to test this out for my self. I popped out my 4th drive, plugged it in to my PC, formatted it and started moving data from my Drobo to the harddrive I just removed from it while the Drobo was still restructuring. I expected a huge mess, but everything worked exactly like the advertised. I was kinda shocked.

FYI the reason I did that swap out was because I foolishly formatted my Drobo as NTFS. This worked ok but I had one to many problems talking to it from my Linux PC. The permissions were all messed up over samba. New folders and files I created on the Drobo were root access only for some weird reason. So I decided to format it as ext3. Since the DroboShare runs Linux this is the best option for a shared drive and works fine while talking to mac and windows as long as you do so over the network.

Backup Your NAS

[s.whiplash]

I have a similar setup to previous post. Server running Linux with a 3ware RAID card running RAID 5. 8 500GB disks with ~3TB of usable disk space. This has been running flawlessly for over 5 years. I have a a movie collection, music and network share for 3 HTPC in the house. Works very well but I wish my RAID card supported the ability to power down the disks and save of power/heat when not in use.

I am almost at capacity on the RAID volume, so to expand I have another RAID card that I can put in the server and create a new volume or replace the 500GB drives with 1TB or now 2TB disks. Replacing the disks would save power and heat, but I would need to backup and restore 3TB of data. Adding another RAID card is easy, but crates more dives that I can't turn off and eat up power.

I am actually thinking about building a new server with the thought of being able to add an many SATA ports as possible (via SATA cards) and then us port multipliers. The use a software based file system or RAID that allow me to add drives of different sizes to the volume. Similar to ZFS but more open. This would make growing the system much easier and allows me to power down drives when not in use. I would still get plenty of performance for my needs.

The other thing that I am doing that most people don't think about, is I backup my entire NAS to another server. I took another old PC that I had and put a 4 port SATA card in it and four 1 TB disks and run Linux and software RAID on it. Each night it powers up and runs a script to back up the primary NAS. I do this just in case something catastrophic happen to my primary NAS and I also use it when I moved to larger disks on the NAS previously. I use rsnapshot to look for changes on the primary NAS' file system and only back up data that has changed. It also keeps the lat 3 months of files that have been changed or deleted, if I need to recover something. When the script is finished, it powers down the backup NAS and wait until the next night to run again.