Best Solutions For Massive Home Hard Drive Storage?

i_ate_god writes "I download a lot of 720/1080p videos, and I also produce a lot of raw uncompressed video. I have run out of slots to put in hard drives across two computers. I need (read: want) access to my files at all times (over a network is fine), especially since I maintain a library of what I've got on the TV computer. I don't want to have swappable USB drives, I want all hard drives available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be moderately quiet and/or hidden away somewhere and still keep somewhat cool. So Slashdot, what have you done?"

Comments

Define "massive"

[jschen]

How much data constitutes "massive"?

Re:Define "massive"

[svirre]

My pricing indicate 2TB disks are slightly cheaper/GB than 1TB

Re:Define "massive"

[BLKMGK]

I've got room for 30TB of data storage in two machines for a total of 60TB. However I have only populated them to around 12TB right now, I don't add drives till I'm out of space! :-) Not what I would call massive yet but getting there!

Re:Define "massive"

[Mr. DOS]

Check again: you're almost certainly comparing 1TB 7200RPM drives to 2TB 5900RPM drives. And Hitachi drives don't count, being the cheap pieces of garbage they are.

Re:Define "massive"

[drsmithy]

Check again: you're almost certainly comparing 1TB 7200RPM drives to 2TB 5900RPM drives. And Hitachi drives don't count, being the cheap pieces of garbage they are.

When it's going to be used by only a handful of people, nearly always in a sequential access pattern, on the other end of a 1GbE link, why would you want hotter, noisier, 7200rpm drives ?

Re:Define "massive"

[Mr_Insightful]

How much data constitutes "massive"?

640K of memory should be enough for anybody.

Re:Define "massive"

[digitalchinky]

For most people 1TB is pretty big.

To answer the question, NAS boxes. They are moderately cheap, come with numerous drive bays, they are usually small, quiet, and unobtrusive when you stick tape over the blue LED's.

Re:Define "massive"

[ls671]

NTFS algorithm is kind of silly and will pick the first free block it encounters to begin to write a big file *without regards* for the amount of consecutive free blocks available there.

On the Windows install you are talking about, all it takes is a little temporary file getting deleted. NTFS encounter a couple free blocks freed by that file and starts writing there only to realize there is not enough space for the whole new file it needs to write once it uses the last consecutive free block available there.

In contrast, even ext2 FS keeps some knowledge of the global state of the FS so it will pick the best free available block to start to write the file *with regards* for the amount of consecutive blocks available.

I have never defragmented an ext2 FS in my life, the write algorithm takes care of keeping fragmentation low.

Re:Define "massive"

[inKubus]

What you want is cheap 5U rack servers with either OpenFiler or FreeNAS. Personally, I like openfiler better. iSCSI is going to be the way to go unless you want a thick OS on the server and all the other admin issues that come with that. Plus, with openfiler you can still do block level snapshotting and change replication. Also, I've heard good things about Open-e as well. And if you want to mess with ZFS, there's OpenSolaris.

What you do is get yourself a huge (4 or 5U) barebones server from newegg or a cheaper place. Make sure to get a couple of good SATA RAID controllers. Not FakeRAID! SAS would be better, but the drives are a lot more, even for the nearline drives that are basically SATA drives with a SAS interface. Adaptec makes some real SATA raid cards, and there's 3Ware as well. You don't have to worry a lot about the cache, but if it isn't battery backed, you're going to write though it anyway. Who cares, you have 16 spindles! Load it with a bunch of drives. They don't have to be the biggest, anyway more spindles means more performance. 16 500GB drives is going to be fine, for instance, because then you can take 1/3 of that for RAID 6, have some hot spares, etc. Get the slowest drives you can, maybe get a little SSD to use as a boot drive (there are small ones for around $100). You could even boot from a USB key if you feel like the hassle. You don't need a ton of processor. A celeron would probably work, but you probably do want something 64 bit so you can put a bunch of RAM in it as you get more advanced.

Also check out Storage Search. Not a very well designed site but tons of goof info under iSCSI and SAN and NAS.. If you're rich, you might try out an EqualLogic, they are around $28,000 for 8TB but pretty slick.

Re:Define "massive"

[Anonymous Coward]

Except that this is completely untrue. Even Windows 95 scanned the table of free blocks for a reasonable area of consecutive free blocks.

  HKEY_LOCAL_MACHINE \System \CurrentControlSet \Control \FileSystem
DWORD ContigFileAllocSize

http://technet.microsoft.com/en-us/library/cc768196.aspx

Re:Define "massive"

[TheLink]

Yes, but using NTFS would be a bad idea for this, far from "best solution" territory as mentioned by the submitter. For a massive home storage system I wouldn't recommend using Windows for the server. Set up a gigabit LAN and a samba file server with the multi-TBs of locally attached storage drives. If you want RAID, use software RAID. Add a UPS, configure NUTS, configure hardware monitoring, smartmontools and RAID monitoring (if you have RAID).

Yes it's a fair bit of work that seems unnecessary when you could just buy a NAS device, but AFAIK most don't do a full self-test at 3 am in the morning and send you an email when one of your drives fails with a CRC prob. Nor would they do an orderly shutdown when the UPS runs low on juice.

And this is Slashdot of course. If it were some other site, I'd suggest a NAS device :).

Re:Define "massive"

[carnivore302]

I'd consider 1TB small today. And this guy probably as well; You can't put a lot of 1080p movies on one 1TB disk.

My own setup is a box that has two thecus N5200Pro NASes NFS mounted. One has 5x1TB, the other one has 5x2TB. Both are RAID-6 arrays. I know I throw away 6TB of storage but I'd rather spend a couple extra bucks than loose my episodes of Dharma & Greg.

If something goes wrong on the 5x2TB array I'm up for a 2 day array rebuild though, praying no other disk fails as well. The newer Thecus NASes have zfs and shouldn't have this problem.

Re:Define "massive"

[Hadlock]

You'd have to have one hell of a bit torrent hobby/debilitating movie watching problem to need more than 2 TB of video on tap on a hard drive for entertainment purposes.
 
Unless you're doing HD video editing, or you like to keep a copy of every picture ever taken by your 8+ MP DSLR in RAW format, few people actually need that space. You might be able to fill 100GB with installed video games but the average person who is buying a 1TB drive is probably upgrading granny's computer and thinking "well hey, for $30 more I can get ten times the space" and opt for the $100 1TB drive instead of the 100GB drive. I just replaced the primary drive on my file server and said "hey, for $6 more, I can upgrade from an 80gb drive to a 320 gb model".

Re:Define "massive"

[BikeHelmet]

When I hear a question like this, I usually recommend heading over to the NCIX forums. There's some crazy guy over there - death_hawk - building a 100TB array.

What I did was a bit less ambitious. A regular old NAS running off a cheap non-RAID SATA card in a case with lots of HDD bays.

For interest, I'll throw up a build that easily scales to 12TB. Since you mentioned noise, I'll prioritize that instead of capacity. I'll use a case geared for silence, a fanless mobo/cpu, a quiet PSU, WD Green HDDs, and a ridiculously cheap SATA card.

Case - 8 bays: http://www.ncix.com/products/?sku=51277&vpn=6900654&manufacture=Fractal%20Design *1
Motherboard/CPU - Silent: http://www.ncix.com/products/?sku=50891&vpn=AT5NM10-I&manufacture=ASUS *2
DDR2 - 1GB: http://www.ncix.com/products/index.php?sku=18584&vpn=VS1GB667D2&manufacture=Corsair&promoid=1114 *3
PSU: http://ncix.com/products/?sku=33357&vpn=CMPSU-400CX&manufacture=Corsair&promoid=1114 *4
SATA Card: http://ncix.com/products/?sku=19892&vpn=SY-SA3114-4R&manufacture=Syba *5

HDD - 2TB 4KB http://ncix.com/products/index.php?sku=49591&vpn=WD20EARS&manufacture=Western%20Digital%20WD&promoid=1114 *6
HDD - 2TB 512b: http://ncix.com/products/index.php?sku=36130&vpn=WD20EADS&manufacture=Western%20Digital%20WD&promoid=1114 *7

OS: FreeNAS, Ubuntu, Win7, Other *8

*1 Only six will be filled. 6 SATA ports.
*2 Case still requires fans/airflow.
*3 A NAS probably only needs 512MB, but 1GB is cheap. A Win7 NAS may benefit from 2GB.
*4 Must be capable of spinning up 6-8 HDDs at once.
*5 Must be flashed with new non-RAID BIOS to avoid silent data corruption for > 1.0TB HDDs; disk read/write speeds around 30MB/sec, in my experience, on ext2. (but running with a VIA CPU - not dual-core Atom)
*6 Must be specially formatted under Windows and Linux. (Most distros only support 4KB sectors when the drive reports 4KB - these report 512b to maintain XP compatibility)
*7 May have longevity issues. (too early to say right now - lots of complainers, which reminds me of the 7200.10 days. A heck of a lot of those chirping barracudas perished early)
*8 Please verify SATA card support first. Ubuntu and FreeNAS work fine with this card, but I've never checked if Win7 has drivers. Do note that you'll have to flash it. *9 If that's a problem, buy a more expensive card. (which may give better performance, and SATA2 support) Promise makes nice non-RAID SATA cards.

*9 Flashing the PCI SATA card requires making a DOS boot CD: http://www.hiren.info/pages/bootablecd

Please note: A solution like this will take 12+ hours to set up. It's highly likely you'll blow a whole weekend, even if you know what you're doing. You may have to try multiple distros to get proper Atom D510 support, unless you go with Windows. When I put mine together, atoms weren't available affordably, so I went with a cheap VIA board. Ironically, Ubu

Re:Define "massive"

[jridley]

I have a lot of storage for the same reason that the OP does, and I PREFER 5400 RPM drives. They run cooler and are still faster than what I need.
I prefer WD Greens, but Samsung EcoGreen works well too. I buy the green ones because, again, they run cooler.

1.5TB drives have been cheapest $/GB for a while now, though I suspect 2TB will take its place, especially after the 3TB drives hit the shelves.

Re:Define "massive"

[AmiMoJo]

Actually NTFS is pretty good at keeping files unfragmented.

If a program opens a new file and them immediately seeks to the end of it to fix it's size then NTFS will look for a continuous block of free space to save it in. NTFS caches all writes so it can wait to see what the program actually does with a file before committing it to disk.

It also has a system designed to reduce the fragmenting effects of small files by being able to store their data in the same block as their metadata.

The only major fragmentation problem Windows XP has is when a machine has very little RAM and it allocates a rather small page file. It then ends up needing to expand the page file repeatedly and it gets highly fragmented causing severe slow down. I think they fixed it in Vista/7 by simply specifying a sensible minimum size and expanding it in larger chunks.

Re:Define "massive"

[drinkypoo]

Does using RAID controllers actually provide superior price:performance to using software RAID? Last I checked, the processors on most cheap RAID controllers were slower than dogshit and using md under Linux would give you better performance than basically any of them, at the cost of some CPU. But since CPU is cheaper than RAID, it probably makes sense. For example, going from a Phenom II X3 720 to a Phenom II X6 chip of the same clock rate takes the CPU from $100 to $200. How much would it cost to go from four crappy RAID controllers to four good ones? It would probably cost you at least $400.

The answer is probably to just go ahead and install Debian on a machine with as many CPU cores as you want to blow money on, and to use software raid. Put lots of system RAM in it, which the OS will automatically use for disk buffers. Current versions of grub work fine with USB keys, because they can use UUID for the groot, and the UUID never changes. If you want it to boot quickly, find a motherboard with coreboot support. If you want external disks you can use firewire cheaper than eSATA, if you get the external disks or just some enclosures at a good price. It makes maintenance a lot easier, but involves substantial power waste due to all those inefficient wall warts.

P.S. OpenSolaris is circling the drain, please don't suggest it to anyone for anything.

Re:Define "massive"

[tgd]

Your analysis completely ignores the cost of the electricity to run a setup like that.

I went from an older similar setup with about 1TB of storage to a dedicated NAS box with 2TB of storage with similar performance characteristics -- and saved $40 a month in electricity.

A 500GB drive draws as much power as a 2TB drive, and server motherboards and power supplies devour power.

Re:Define "massive"

[BitZtream]

Yea, in 1992 maybe ... not sure what version of Windows you're comparing too but that hasn't been true for years.

Re:Define "massive"

[Hadlock]

250 movies that you watch every year, in addition to the ones you rent, or go see with friends, or simply non-movie stuff you watch like sitcoms and/or live events like the news/sports? You must only work 2 hours a day to keep up with your busy viewing schedule and still have time to sleep, shower and spend time with other humans (they exist outside of movies, you know). 10 movies that you re-watch year after year I can understand, but 250 just blows my mind. Do you schedule that a year in advance? What happens if you miss a day?
 
I mean, you already have it in another (optical) format. If you already have a physical backup, what's the point of archiving it on a hard drive? It's going to take you just as long to find that movie in explorer as it is going to take you to pull it off the shelf and stick it in the drive. I can understand the need for photo storage, since there's no other physical media they come on and memory cards are relatively expensive. But unless you're accessing the same data you already have in optical format at least once a month it seems that you're backing it up to hard drive just to be able to say you've got 250 movies on your hard drive. It's unrealistic to watch all those movies every year just to justify having them on a $100-200 hard drive + the time (how long does it take to rip a movie, 30 minutes?). Maybe I just value my time better.

Re:Define "massive"

[sjames]

I've heard one too many sad stories about old on-disk RAID structures not being compatible with the new version of the old failed RAID card. I prefer the md device since it has been consistent for quite a while and the on-disk format is well documented.

Re:Define "massive"

[schon]

When a drive fails, SW RAID doesn't allow one to boot the system unless one has (1) Used RAID-1

True enough, but (honestly) how hard is it to use RAID-1 with hot spare(s) for your boot partition, and RAID5/6 for everything else? (answer: not very, I'm doing this at home.)

Setup the BIOS to use the two mirrored disks successively while booting.

Most modern motherboards already do this. Contraiwise, even if you have to go out of your way, it's *still* much easier than screwing around with driver disks for HW cards when installing.

SW RAID often exhibits very poor performance when a drive fails as the underlying drivers want to try to make the operations against the disks work and will often expend a fair amount of effort to retry operations and wait for extended periods of time to force the disk operation to work.

Never seen this happen. "bad" drives on SW RAID mark out just as quickly as those on cheap HW controllers.

Standard disk controllers do not support hot swap. So when a drive fails, replacing the drive involves shutting down the server, swapping the drives, and then bringing it back up.

Bull-fucking-shit.

I regularly attach and detach drives while my home server is running (I back up to external HDs, connected to the internal disk controller with an e-SATA cable.) Motherboard is an ASUS P5EVM-DO with a "standard disk controller".

Re:Define "massive"

[PipsqueakOnAP133]

I'm not sure even more than 1 core is necessary for software RAID.

I just (last week) built a OpenSolaris based ZFS NAS with 2 drives on a old Core Duo. Its never gone past 20% CPU utilization even when I'm copying from 3 disks over gigE to it with on-the-fly gzip compression enabled as part of ZFS. I think it sustained 48MB/sec.

CPU would have been a Core Solo instead had the ebay seller not mailed me the wrong chip.

I'd say anything P3 class or better would be fine with one core.

Also, I'd object to saying that OpenSolaris is a bad choice. We're taking file servers here. ZFS works well and OpenSolaris is open source. It's not like you want bleeding edge commits from a tree to run the thing that stores your precious data anyways.

Finally, if this is dedicated to being a file server, might as well boot off the USB flash drive or liveCD. Your distro ought to be small enough to fit and you have less chance of OS corruption from a read-only image.

Something like this

[fotbr]

Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Re:Something like this

[Anonymous Coward]

Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

If possible use something like ZFS (or btrfs if you feel confident about it) so that you get checksumming data protection.

If you're going to put all your eggs in one basket, you better watch that basket very carefully.

The creators of that kit don't use any kind of redundancy with-in the box because their custom software stack handles replication (kind of like Google FS / Hadoop FS).

Re:Something like this

[EdIII]

The only caveat about that particular solution is the lack of redundant power, poor serviceability in the rack (may not apply like you said), and slow speed.

Their solution achieves the density it does because they are using SATA multiplexers, but that effectively creates bottlenecks and lowers overall speed. It works for BlackBlaze's application requirements, but YMMV.

Protocase.com makes the enclosure and will sell it to you for a pretty reasonable price. Getting all the parts is not such a big issue. I think we estimated we could build one without drives for less than $3k.

If you don't have it in a rack, then serviceability will be a lot better for sure. Rackmount solutions require cable management and heavy duty slide rails, and wide aisles, in order to gain access to the drives. The backplanes are parallel to the ground, facing up, and require taking the top off to access. Not exactly IT friendly.

Since the person in the article is not using this in a datacenter, cooling is going to be an issue. I suspect BackBlaze survives due to hot-cold aisles and plenty of airflow. Sticking one of those enclosures in a closet without ventilation/cooling is a recipe for disaster.

Re:Something like this

[mindstrm]

Unless they changed something since they published their howto - your BZZT has it's facts wrong....

Backblaze pods use 45 drives in each pod.

Each pod is made up of 3 Raid-6 volumes.

Each Raid-6 volume is made up of 15 drives (13 drives + 2 parity)

There are two parity disks for every 13 data disks. That's higher than you might want to go in a normal enterprise setting, but as they also handle redundancy between pods, it's an acceptable tradeoff in their case that I'm sure they've calcluated out quite well given drive sizes and rebuild times and all that......

Cheap NAS

[Bodhammer]

DLink - DNS-323 with two WD 1 TB Green Drives. Quiet, works out of the box and is also Linux hackable if you feel the need.
Enjoy!

My 2 cents

[ars vitae]

Personally I'm using a Synology solution at the moment, for my NAS. They offer a relatively low cost feature rich hardware, with low power and depending on the HDD's you use, lower power consumption than that of a always on PC.I've been thinking about later on upgrading, since a general rule of thumb, you can never have to much storage. For HD BluRay images I would recommend making sure the network isn't the bottleneck and use gigabit ethernet, as I'm finding on my aging 10/100 switches it's not cutting it. xvid's and MP3 streaming it seems to be fine.

Paranoia

[Anonymous Coward]

So Slashdot, what have you done?

Why? What have you heard??

ZFS

[mikael_j]

My personal storage solution consists of a 4U rack case with a computer with a c2d CPU, gig-E NIC, a few gigs of ram, a bunch of 7200 RPM disks and FreeBSD on the system disk (I also have the system disk mirrored just in case). All the storage disks are then pooled using RAIDZ. Pretty simple yet powerful. Just don't expect too much in the way of performance.

Re:ZFS

[0100010001010011]

ZFS + Solaris.

I have a standard ATX case with 4-in-3 adapter from Newegg. I didn't get the more expensive ones with trays because I didn't need to hot swap.

I have 2x1TB drives in ZFS mirror for boot. 5x1.5TB drives in RaidZ as a tank and 2x200GB drives in mirror with a virtual block device for Xen Debian and Windows 7.

OpenSolaris is amazingly simple to use, if you're just doing your home network.

At the most basic level:
zpool create tank c5t0d0s0 c5t1d0s0
zfs sharenfs=on tank
zfs sharesmb=on tank
zfs shareiscsi=on tank

Now your new drives are all shared over NFS, SMB and iSCSI.

I keep looking for the old school full height 'desktops' at a bargain store or so. Search newegg. 3.5" external works just as well as internal.

This has 11 3.5" bays and 3 x 5.25" bays. With a 4 in 3 linked above you could have 15 hard drives in a case for $100. Or if you care about hot swappability This one has 20 hot swap bays (at 3x the cost).

If you want more performance, get some SSDs to work as the ZFS "cache".

How much is a lot?

[jimmyfrank]

..., are the movies you download compressed at all? You say you run out of slots, how big are the drives you're putting in the slots? Personally, I let Netflix do the storing for me. I have a few TB's but never come close to filling it up.

Cheap solution

[tomhudson]

1. create a million gmail accounts (or buy them from spammers in bulk at 3 for a penny
2. link the accounts as one big drive
3. 640k petabytes ought to be enough ...

Why do you need them available at all times?

[VinylRecords]

I used to work for ABC news and we never kept archive footage always accessible like you want. If we wanted something that was really old we'd have to dig it off a tape, an unplugged hard drive or powered off computer, or we'd have to find another news agency that had the footage and grab it off of a satellite feed. And this was a 24/7 TV news station responsible for national news programming where we would be tracking stories for years. If we didn't need a system where everything was instantly accessible then you needing it on an individual level might be overkill in my opinion.

I have over 30TB of music, movies, and raw video footage on my home computers and I just keep everything on separate external hard drives. I label the drives, back them up twice each, and then keep an index in a .txt file that is easy to search through. So if I want a 1080p backup copy of Blade Runner I search 'Blade Runner' in the .txt file and I see it's on drive 'A' and then I plug in drive 'A' and dump the movie on my computer. I also keep an external drive that has backups of every TV show I own on DVD. So if I want to watch The Wire then I plug in the external drive labeled 'TV' and have at it.

Re:Why do you need them available at all times?

[KahabutDieDrake]

First off, judging personal wants and needs by the way a giant corporation acts is hardly reasonable. ABC has cost/benefit to consider when trying to keep data available, and it's probably easier/cheaper to do it the way you say they do, rather than implement a fully digital, fully available storage system.

That being said, the solution is SIMPLE. If you have a bunch of hard drives with data you want, you put together a low end PC, install it into a server case, and fill it with hard drives and SATA controllers. When it's full, you build another one. You have 30tb of data, mostly not accessible. I have 10tb of data accessible from any internet connected computer on earth, and it's twice as much storage as I actually use. It cost me about 500$ to build and deploy a personal storage server, and it doubles as an HTPC. ( I already had most of the drives, and some parts) It's likely most people here have enough hardware laying around to implement a basic storage server. There really isn't any reason not to do it. As a bonus, since it's not a machine you need to access directly most of the time, you can hide it in a closet and forget all about it.

Sure, you could buy a premade NAS/SAN or stand alone data box. However, they are costly and not any more suited for the job than an old machine, or low end new system. At least, not in a personal environment. If you actually require robust data storage, I'd suggest a NAS, from any number of sources. But now we are talking about 4k worth of hardware, and requiring proper power systems to be added if you really want longevity out of it. However, that's overkill for a home storage solution, no matter how much data you have. Simply because you don't need enterprise class data serving, when only one or two computers are accessing the data.

If you don't know how to build and deploy a system with lots of drives accessible over a network, then you probably started at the wrong website for help. You want DELL/HP/IBM small office sales line.

Re:Why do you need them available at all times?

[hawguy]

Maybe that's because ABC could not afford to keep years worth of (uncompressed?) video spinning on disk - they probably had hundreds of hours of programming each month (week? day?) -- and they have digital librarians to take care of storing and retrieving tapes. They'd pay at least an order of magnitude (probably 2 orders of magnitude) more for disks than the original poster -- EMC, Netapp, Hitachi, etc are not cheap and a major news organization is not going to build their own Linux box to store video. A home user has different needs (I haven't watched the Little Mermaid in 2 years and I want to watch it NOW DAMMIT!), much less video to deal with both in number of hours and in raw size since most of his stuff already came from a compressed format), and the economics are different for home gear -- a home built Linux fileserver is a fraction of the cost of enterprise storage from a major vendor. Plus he's only talking around 10TB of disk space, that's not much for a home hobbyist to build.

Re:Why do you need them available at all times?

[clarkn0va]

I see you've never tried to maintain a ratio on a private tracker.

Re:Why do you need them available at all times?

[gravis777]

For his "downloaded" 720p/1080p movies, its reasonable to assume that these are most likely reencoded to mp4 / mkv files or ts streams, probably between 2-15gig each. An external USB2 Harddrive should be able to keep up with the transfer rates. As such, you could probably go with something such as a USB hub and tons of external harddrives.

But I agree with you. I have DishNetwork DVR with the external Harddrive option. I currently have three external Hard Drives filled with movies. I keep a spreedsheet on the computer that tells me what I have on each drive. If you want to go fancy, CollectorZ has a great databasing program that you could probably use.

Now, as for his raw uncompressed footage, I have a couple of questions here - how many projects are you working on at once that you need to have tons of uncompressed footage, and do you really understand what Uncompressed means? Whether HDV, MP4, or MPEG2, every HD capture technology I am aware of uses some form of compression. The exception may be if you are scanning film in, taking several seconds to several minutes a frame, but it certainly does not sound like that is what you are doing.

I have been doing video editing for about 10 years, and always had space issues. I have handled this in two ways over the years. The first is to work on your project in 2-5 minute segments. Export those to an end format, such as MP4s, then stitch them together later.

Oh, but that's double recompression, you might say. Well, not necessaraly - you can easily use something LIKE VirtualDub (I dont think it currently supports MP4) to stitch video together without reencoding. This takes care of a LOT of harddrive issues when working on a large project.

The second is, and you won't like this, but hear me out, is to go ahead and compress your videos. I am talking about something along the lines of, oh, 20-40Mbps, depending on content, using a good codec and compression software. Saves a buttload of harddrive space, the drop in quality will hardly be noticable. Even then you can still edit, and export again in a fairly high quality. My experience is that you can probably go a couple of generations - IF you use a good codec, good software and high bandwidth - before you start really noticing degregation in video. Sure, if you compare the original to your finished work in front of a 30-40 inch monitor 18 inches from your face, you will probably notice a difference, but anyone else you show the finished product to - from a normal viewing distance, probably won't notice.

So to sum up, use external harddrives to put your "downloaded" content to, should be plenty of bandwidth on USB2 (or firewire800 if you are a stickler) to do it with, stop keeping around terrabytes of uncompressed video (any project exceeding, oh, lets use a magical number of 500 gig of raw source material, is insane), and, shoot, if all else fails, pickup a Blu-Ray burner. Discs are certainly coming down in price, although it is probably still cheaper to buy external harddrives.

Once again, as the parent pointed out, there is no reason AT ALL to keep all your videos online at once. Not even video production houses do that.

Solution

[Anonymous Coward]

Don't worry, we'll be right over and take care of everything. You'll never have to worry about it again.

MPAA

P.S. My sister, Riaa wants to know if you're into MP3s

SATA port multipliers

[the eric conspiracy]

SATA port multipliers - 5 to 1 for about $50 + 5 2 TB gives you 10 TB off 1 SATA port.

Re:SATA port multipliers

[EdIII]

They work, but will slow the system down considerably. If you connect 5 drives up to 1 multiplier, the total speed you will get is the same as 1 drive hooked in directly. In otherwords, a bottle neck.

So technically it is possible to hook up 250 SATA drives into a single SATA RAID card, but you are not going to be that impressed with the performance.

Re:SATA port multipliers

[Kjella]

Well, from what I understood there are two modes - one which will give you only time slots so 5 drives each get 1/5th of the time. That's the cheap variety. The other variety is traffic based, you can't exceed 3 Gbps but you can get the cumulative read/write speed up to that point. The SATA spec site has more.

Re:SATA port multipliers

[whyde]

I'm in the process of building a 5-bay SATA port-multiplier solution right now. What I've learned thus far is:
  * Most commodity motherboard chipsets don't support port multipliers. You'll need an expansion card.
  * If you have this much data, look into ZFS and RAIDZ2 for reliability. Avoid RAID5.
  * The bigger the disk, the longer it takes to rebuild a degraded array
  * FreeNAS is at an inflection point. If you're not scared, use PCBSD directly instead to serve your data.
  * You don't need "enterprise-class" storage speeds to serve up movies and media. Slow, green drives are fine.
  * Don't buy all of your drives from the same lot, all at once.
Cheers, and have fun in the process.

Re:SATA port multipliers

[gasgesgos]

That's only accurate under the assumption that a single drive can max out a 3 Gbps line. I'd like to see a reasonably-priced consumer grade HD that can pull THAT off. It doesn't really matter anyways as the ultimate bottleneck here will be the network at 1Gbps. Five drives evenly using a 3 Gbps channel would still be allowed 62.5 MB/s each, and that's still pretty good for network transfer.

5400 RPM

[Anonymous Coward]

I personally set up a downloadserver that also functions as a media server to stream the content to other devices. I put in a couple 5400 RPM 1,5 TB drives, they use less power, generate less noise and heat than a regular 7200RPM drive but since you're not running any applications of them, you won't really notice the difference in performance. Prices have gone down a bit so the sweet spot for $/GB might be at the 2TB mark now. If you don't want to go for an entire computer, maybe a NAS solution would be best for you, with the same 5400RPM drives. A NAS will have less room for the disks if you really want *massive* amounts of storage, and also you usually must purchase one + the disks. The PC you can build from spare parts lying around. I personally put gentoo linux on mine, but you also don't exactly need top of the line equipment for a nice windows XP install. The NAS however will have outputs directly for your TV and will take up less room and power.

Still, the key is 5400 RPM + 1,5/2 TB.

Sounds like one hell of a porn collection

[Trailer Trash]

/. is definitely the place to ask...

WHS

[Barny]

Windows home server, 1TB 7200rpm main drive with seagate LP 5900rpm drives, lock it away and never have to think about it till you need to drop another drive in.

The reason for the fast main drive is that with WHS when you copy data to it, it stores it on the main drive first, then schedules it to be distributed out to the storage drives the next time a "storage balance" is done.

Works fairly well, its based off windows server 2003 at the moment, but if you can wait till the end of the year they have a server 2008r2 version coming out soonish.

Re:WHS

[Shados]

This. Even ready-made resellers have pretty small devices built on Win Home Server that can take a LOT of drives. Mine supports 4, but there's many models that can take 8, 12, or more drives. The OS is rock solid and has a lot of neat features, like being able to access your network from an SSL secured web app (built in) from anywhere with indexed search, and its easy to develop plugins for (though there's a ton available already) to extend it.

Re:WHS

[Sabriel]

Another happy WHS owner here. I do recall reading that one of the service packs (there have been three) fixed the requirement for a big first drive - files now copy directly to the storage drives.

That said, I still use a fast system drive, and the rest are a mix of 7200 and 5400 rpm drives (depending on what was cheapest at the time).

Bought the original Coolermaster Stacker case. The front of the chassis is solely 5.25" drive bays - eleven of them - technically twelve if you mod the case to move the power+usb front panel elsewhere. :)

Oh, and despite being based on Server 2003, one of the nice things about WHS is that unlike the former it doesn't cost an arm and a leg.

Re:WHS

[Barny]

Yeah, because a bug (running vista on your client and using the server without the latest updates) over a year old and fixed is a problem...

I will also point out that the very first linux release wouldn't run on my 8088 cpu... ... ... ...

Please sir, if you are going to google for bugs, check your dates :)

Re:WHS

[gmurray]

you forgot to mention that it can take any sized hdds, you don't have to match them in size or brand, and if the whole hardware fails the files are just stored in a regular ntfs partition. So you can just mount it on a normal windows/linux/etc box to grab files off the drives. Very cool. That and its easy to migrate drives between different hardware since its a software based solution.

pervert

[Cyko_01]

I download a lot porn, and I also record a lot of masturbating videos. I have filled two computers with porn already. I want access to my porn at all times, especially since I maintain a porn site. I don't want to have swappable USB drives, I want all my porn available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be disguised or stashed away and not overheat. So Slashdot, what have you done?

Re:Look at the DroboPro

[binarylarry]

Good fucking god, $700 for the Drobo FS?

You could build a capable home server box AND buy some of the drives for that much.

NAS

[DoofusOfDeath]

I've tried a variety of approaches, but overall I've been happiest with just buying a NAS box.

I have a Synology DS209, and I've been very satisfied. It's a relatively cheap way to get 2 TB RAID 1 storage with really simple backup to an external USB drive. If you need more storage, you can buy NAS devices with more than just two bays.

Budget?

[HaloZero]

You made no mention of a budget. I'd go with a Drobo - probably the DroboFS. http://www.drobo.com/products/drobo-fs.php

unRAID from Lime Technology!!

[BLKMGK]

I have two of these servers now. Each server can hold as many as 16 disks (possibly more actually as the programmer keeps bumping that up) with one disk reserved for parity. Data is NOT striped and parity is ONLY stored on the one drive. If a disk fails I lose no data, if two fail I lose two disks of data but nothing else. No hot spares or any other crap. If a disk isn't being used it goes to sleep and saves me heat and power. Disks can be ANY size but the parity disk must be as big or bigger than any of the data disks. Runs on a pretty decent selection of hardware although keeping the list of what works and what doesn't up to date is apparently tough since hardware changes so fast. It's Linux based but pay for play, yes he's followed the GPL. It's not super expensive and it boots from a USB drive to be web administered. I use full tower cases with SuperMicro 5n1 trays, 2gig of memory, Celeron CPU, power saving PSU, and supported mobo that have onboard video and GigE which you WILL need.

Their forums are a big help and active, users are working to expand the capabilities of these NAS and the programmer is working on making that easier too. Check it out, I've not found anything better yet and with some of the newer versions of SAMBA in the code it's pretty fast too! Perfect for a HTPC but not so great for a big transactional database

http://www.lime-technology.com/

Re:Look at the DroboPro

[Firehed]

Having done that in the past, I'll say that buying a Drobo was worth the cost. Granted, I hunted around a bit to get a good sale price (it's not too difficult... though the FS is brand new so maybe not on that model yet), but unless you really enjoy tinkering with getting samba shares set up and working properly, sometimes it's just easier to buy your sanity.

Don't get me wrong - I wish they were cheaper. But their system worked better and more reliably than anything I ever put together, and I'm by no means incompetent. And their BeyondRaid tech, while proprietary, is pretty damn cool and works incredibly well. Being able to mix drives and not waste tons of storage space is a huge advantage that (as far as I know) I'm not going to get anywhere else.

Just a happy customer, not an employee or anything like that.

Have you considered ATA Over Ethernet (AOE)?

[jcwren]

http://en.wikipedia.org/wiki/ATA_over_Ethernet

This is something I've always wanted to play with. It's a little expensive (for a home user) to get into, but it's extremely scalable. If I moved all my DVDs and such to on-line storage, I think this is what I would opt for. It can be run in all sorts of RAID configurations, doesn't require matched sized hard drives, and it can all be racked up very nicely.

OWC Qx2 4-drive RAID array

[david.emery]

4 drive bay, USB, FW400/FW800 and eSATA. Will take 2tb drives, RAID 0, 1, 5 and 10. Comes pre-populated or unpopulated, the latter is what I got and added my own drives. http://www.macsales.com/ No financial connection, just a satisfied customer (they have great tech support!)

This is obviously not a build-it-yourself storage array, but is a good option if you want a commercial out of the box solution.

You are correct.

[ClioCJS]

Even if you "need' a 16000 RPM drive, just make it for your local drive that you play your videos directly off of. Use 5400 for all the other ones. Just move your file before watching it. Sure, if you're an impatient baby and want to watch something within 5 seconds of it entering your mind, then you might have to wait 5 minutes if the file is 4.5G. Then again, it's the type of waiting you can go pee or make your snack during.

Two Options

[Zarjazz]

1) Cheap tower server + your favourite unix distro + software RAID + many, many cheap 2TB drives.

2) Standalone NAS device. Everyone so far seems to recommend different makes so I'll carry on the trend and suggest Thecus. Just slot in the drives and you're ready. Install the SSH module and you also have a Linux server too.

Re:Two Options

[E-Prime]

Thecus is really an awful suggestion. I own the N5200 Pro and can with certainty say that the device sucks. The web UI looks like an old version of what you find in a Linksys router, and its not pretty. It keeps kicking disks out of the RAID even though they're on the supported hardware lists. Emailing customer support never gives you an answer. Opening a ticket in their issue tracker never gives you a response. Forum is full of complaints. I'll never buy anything from them again.
I've since purchased a Synology and it is much faster, has a modern and feature-rich web UI, plus it actually works.

Forget NAS

[Kjella]

If you want cheap, affordable storage get:

A decent full tower case with a modular PSU
A motherboard with 8+ SATA ports (cheap)
A 4-port SATA expansion card
=
12 SATA slots + 12x SATA power for cheap

Get a cheap bunch of 1.5 TB drives for up to 18TB total. If you say home I assume you don't mean 99.9% redundancy. You can buy a new PSU or motherboard or whatever and have it delivered and that's okay. Softraid two drives in RAID1 for 1.5 TB less storage. If you need more protection then upload it to some offsite backup - any external disk or second machine is still vunerable to theft, fire and whatever. It works for me, though I only have ~10 TB due to due of old low-capacity disks.

Oracle Sun Fire X4540

[xavi62028]

48 SATA drives onboard a 4U rack, dual six-core opterons, redundant power supply. This thing won't let you down. Go big and heavy with this and it'll cost $1/GB. http://www.oracle.com/us/products/servers-storage/servers/x86/031210.htm Make sure to ask for a discount, only suckers pay full price for Oracle gear.

Re:This should be modded up

[InsurgentGeek]

A contrary opinion. I have had a Drobo since the original release and it has been nothing but a disappointment. Drive incompatibilities, an extraordinarily high drive failure rate (at least 1/quarter)and a very confused partitioning scheme. Not something I'll repeat in the future. Oh, and data loss that had to be corrected via a firmware update. In short if I'm spending the money for Raid - I don't want to lose data. Period.

Re:"I won't need 16,000 RPM drives"

[cynyr]

There are large capacity "green line" drives from some manufacturers, 5400 RPM, that might be perfectly enough.

Do they work in RAID? or do they randomly stop responding to the raid controller and then get dropped from the raid, triggering a rebuild, to show up a few minutes later, to trigger another one? http://en.wikipedia.org/wiki/TLER

Cheap COTS NAS

[straponego]

I've been running off an Acer EasyStore H340 for about a month. I'm very happy with it; it's very cool, quiet (if anything else is making noise, you won't hear it; in a closet, you definitely won't hear it) and plenty fast for most households. 4 hotswap SATA bays, eSATA, USB, and GigE. I can push 75MB/s via NFS to and from it (reading and writing from RAM), which is plenty for streaming video. It comes with Windows Home Server and it's headless, but I popped the drive into another box and installed Ubuntu with an SSH server. Worked like a charm. I'm also running a 2TB software RAID1, mt-daapd (iTunes) and squeezebox servers. I'll probably put Samba on it too.

The only thing I'd change is that a dual core Atom would be better. I actually haven't run into a bottleneck yet, but I wouldn't try reencoding videos on it. I believe the dual core model will be out this month. No affiliation with Acer; I'm just geeked because this is just the quiet, cheap server I've wanted for years. Sounds like sharing your other computers via NFS (automount) or CIFS plus one of these would address your needs; if not, maybe the info will help somebody else.

Re:This should be modded up

[Sabriel]

I looked at a Drobo - but being on a budget, I kept on looking elsewhere. I don't doubt they deserve those reviews, but they are not cheap. And if the Drobo itself dies... good luck getting the data off those drives without another Drobo handy.

My solution - Raid 6

[Zen]

You don't specify what constitutes lots of data. In my case, 2 years ago I went for 6 750GB SATA drives in a Raid6 configuration. There's some very good posts here about some lesser known data reliability options, but personally I wanted to go with a worldwide standard that had been around for a long time and wasn't reliant on a couple guys hacking code in their spare time to make disk redundancy and file access work.

I bought a standard full size tower case, got a very large power supply, and spent a good deal of money on a mid-tier Raid controller. My primary requirement was Raid 6 so I could lose two drives without losing all my data, and my secondary requirement was having true hardware raid support. Most Raid controllers that are not enterprise business class are not true hardware raid - meaning that they use software and the CPU for some of the operations. This slows down file read/write. I did the research and read reviews and got a decent Promise card - if you have the money, go for LSI, Areca, or 3ware. Next, I got a Promise hot swappable 4 drive SATA bay. Not really sure why, it doesn't serve any purpose since in 2 years I haven't had a failure and thus have not had to hot-swap a drive. A very important thing is that I also purchased 7 drives for my 6 drive setup. So I already have a spare if I need it, and I don't have to worry about having the spare cash when a drive fails, or waiting on an RMA if it was still under support, etc. The one thing I wish I had done, and still might, is buy a spare raid controller with the exact same chipset. If your raid controller fries, ALL of your data is gone unless you can get the array up and running on an identical controller. That's a freaky thought!

6 drives in raid 6 at 750GB gives me a little under 3TB of disk. I wanted that in a single partition for ease of use, so I messed around with some 64 bit Linux distributions and did not have any luck. I finally settled on Vista of all things, but only after I got fed up with fighting with Linux - I didn't give it a fair shot, I should have been able to make it work. The only thing I can think is that it didn't like my controller or motherboard.

So, 6 drives of 750GB in Raid6 gives me 3TB. At the time I had less than 1TB of stuff, and wanted to make sure I had room to grow. I didn't grow anywhere near as quick as I expected, and I'm still at less than 2TB today. 2TB drives in a raid6 would give you 8TB, and that's if you only used 6 drives - you could easily add more into that same Raid6 array (depending on how good your Raid controller is). Even if all of your movies are dual layer quality, say 6GB each, that's over 1300 movies. That'd certainly last me a long time!

Harddrive Jukebox

[slaingod]

I've posted this idea before, but I would like to see a harddrive jukebox, so you don't have 20 drives running 24/7, just a couple.

Basically your could plug 20+ hard drives into what amounts to plastic holders for the harddrives.
You then have a jukebox/cataloguing software like DVD jukeboxes or some sort of virtual drive that keeps track of all the files on all the drives. When you need to access a file, it powers on and spins up the correct drive. Ideally two or three drives could be activated at once. The Jukebox software would automatically handle all of that.

As far as how the connections were made: You could do it a couple of ways. Have each drive with its own SATA connectors that all fed into an electronic switching hub that handled activating the drives. Or you could have it be a phsyical/motorized scenario, where each drive plugged into a custom SATA header that then interfaced with a motorized SATA connector to attach to a specific drive.

This has already been done with tape drives & DVDs obviously, but I am talking about something CHEEEP (the extra E is for extra cheap).

Re:DIY. Map-Drives, Dir, Grep

[slaingod]

Mapping drives to drive letters is so 1995. You should be mounting them to a single folder (each drive is a subfolder) called c:\mnt, and then just sharing your mnt folder on the other computers. You can do that in disk manager on windows (since linux you would already be doing that presumably) instead off mapping the drive to a drive letter.

There are some quirks in certain copy programs if you are moving files from one mapped drive to another and the copy program isn't 'mounted disk' aware. Nothing to worry about, but you might not get progress bars for moves, for instance.

Re:Look at the DroboPro

[AmaDaden]

I have a Drobo and a DroboShare. The DroboShare runs a slimmed down version of Linux so a network attached Drobo uses typically uses samba. The benefit of having one is NOT the ease of software set up. The reason I love it is the ease of drive management and small hardware size.

Due to the small size and slick style I keep mine in my TV cabinet. I've done the measurements and no PC case on newegg can fit in this same space, never mind something that can house 4 harddrives.

The other thing that is so valuable about a Drobo is how well it manages it's RAID array. They call it BeyondRaid but I hear it's just a as many normal RAID arrays as it needs to organize the drives to both optimize space and maintain redundancy. Also you can pop harddrives in or out while it's on and it will automatically restructure the RAID on the remaining drives to still be redundant with out any need to shutdown or stop sharing data. I recently needed to test this out for my self. I popped out my 4th drive, plugged it in to my PC, formatted it and started moving data from my Drobo to the harddrive I just removed from it while the Drobo was still restructuring. I expected a huge mess, but everything worked exactly like the advertised. I was kinda shocked.

FYI the reason I did that swap out was because I foolishly formatted my Drobo as NTFS. This worked ok but I had one to many problems talking to it from my Linux PC. The permissions were all messed up over samba. New folders and files I created on the Drobo were root access only for some weird reason. So I decided to format it as ext3. Since the DroboShare runs Linux this is the best option for a shared drive and works fine while talking to mac and windows as long as you do so over the network.

Cheap PCI Mobo + Multi SATA Cards

[Doc Ruby]

I've got a $20 case, $50 500W power supply and $40 motherboard with SVGA and ethernet, its 5 PCI slots each stuffed with $25 4x SATA cards, 20 $100 1TB HDs. Running Linux, network mountable drives and ssh login.

That's $210 PC + $2000 HDs for 20TB. That's a lot of porn storage for you.

Re:Bzzzt. Still Wrong.

[twistedcubic]

You gave a bunch of links which require wading through hundreds of lame arguments, including 100 different people proudly proclaiming "RAID is not a backup solution". Nobody wants to read that shit.

RAID 5 or 6

[Hamsterdan]

Creating a RAID 5 array on OS X, is really not that hard. On Windows, pretty trivial too, never got around playing with it on Linux though, must be pretty straightforward.

Personnally, I prefer SOFTWARE RAID, because it's easier to recover from a failed array compared to a hardware card that is not manufactured anymore.

(example, take the drives to another MAC or Windows machine, they will be recognised automagically if it's done with the OS, not so if the controller went belly up...)

I'm running a raid 1 on an old G4, and RAID-5 on an old Win2k Server machine, and they are easier to manage than the one in my Proliant (Hardware)

Do you really need -massive- storage?

[Edis Krad]

I have been where you've been, and let me tell you, what you are considering is a waste of money.

Moore's law applies particularly well to harddrives. Every two years you can buy a new hard drive that is twice as big, for the same price. Or pay half as much for the same storage capacity. If you stock up now, you'll spend a lot of money for something you can buy a lot cheaper in two years' time.

My advice: Buy new hard drives and replace them as you run short on space. If you run out of space inside your rack, move the contents from your oldest disc into the next, and you can sell, discard or get an external enclosure for your oldest disc.
Allocating massive storage without immediate need for it, is going to cost you a lot of money.

OpenSolaris (within ESXi4 or not) is a solution...

[Anonymous Coward]

This was a solution created for a customer. It works well, it scales well and it is as cheap as the hardware ou need to buy.
You need to get a PC with a mobo that supports VMDirectPath and has several PCi-e available. Then, install VMWARE ESXi 4 to a first small hard disk (and old one, for instance).
You can then buy an external enclosure or just create something yourself and you just keep on adding HDs and SAS or SATA controllers (wichever you like).
(Take care to avoid HD vibration, tough - It's bad for a HD's performance.)
Make your first VM Open Solaris and give direct access to the controllers (and consequently the disks) to that VM, and make sure you give it CPU and RAM priority.
You can then setup a ZFS filesystem using RAID-Z or RAID-Z2 (much better than traditional RAID 5, etc). Solaris and ZFS make it easy to make it available as NFS or SMB or iSCSI.
You should worry about speed and the bus speed limitations as you add controllers and disks, but on the other hand if you need more availability than speed this should not be a problem.
Basic hardware to build this should be no different than your average home computer, and you can use any mb, if you'd like, instead of one that supports VMdirectPath, by using raw device mapping, but you should take into account that it is not supported by VMWARE.
Also, because Open Solaris is given direct access to the HDs and ZFS is thus created directly on the hds, if you plug those disks into another PC or VM running OpenSolaris, you access all data on your array with a single command line - Hardware Agnostic. (Usefull if your HD dies).
This has the advantage of letting you raise whatever other VMs you may want. Of course, you can do this, installing open solaris directly. ZFS is the way to go.
Note: You can use Nextenta Store free version if you plan to use store than 12TB of Data. Easier to set-up, but same principles.

42TB for under $4k

[DedTV]

Here's my setup that's currently capable of holding 44TB of storage (I have 18TB so far). Nothing fancy, Just something to hold all my media that isn't horribly noisy or hot and that was still relatively cheap. I have it sitting on a coffee table in my home office so you could put it pretty much anywhere.

A $320 Norco 4020 case that has 20 hot swappable drive bays plus 2 more fixed drive bays inside. http://www.newegg.com/Product/Product.aspx?Item=N82E16811219021
A $250 server motherboard with at least 2 PCI-X slots. I chose http://www.newegg.com/Product/Product.aspx?Item=N82E16813182142 because it takes a Core 2 CPU and DDR2 and I already had plenty of those laying around so I saved a few bucks in parts. I also had a CORSAIR CMPSU-850HX laying around and used it for a power supply which runs about $190.
$99 SUPERMICRO AOC-SAT2-MV8 SATA card. 2 of them will fill the server if you pull the DVD drive (I use an external anyway). 6 sata on the Mobo + 16 more from the 2 cards. Some people complain they're slow but I can pull 60+ Mb/s over the network from them. My guess is they're putting them in regular PCI slots on regular MoBos and not PCI-X slots on a server board.
For an OS, I simply use Windows Home Server. It's $99, windows simple, and is perfect for just storing video files. Reinstalling the OS can be a massive pain though as WHS reinstall script thing never works when there's a controller that WHS doesn't support out of the box (ie. the Supermicro cards). And the new version of WHS based on WS 2008R2 is on the way and there won't be an easy way to migrate.
I also use Flex Raid (Software Raid 4) and sacrifice one disk as a parity drive because duplication isn't much safer but eats a lot more of my space. I just have it do the rSync when no one is likely to be doing anything with the server so it's never a hassle.

So the base cost is within a few hundred bucks on either side of a grand. Less, if you have parts that can be cannibalized from old machines

From there, I just add Western Digital Caviar Green 2TB drives as needed. They're cheap, quiet and run cool. The EARS ones need a jumper (and none are included with the drive) to run under WHS but are $15 cheaper than the EADS ones on Newegg. And of course you can use any old drives you have laying around too.

I have Acer Aspire Revos ($330 http://www.newegg.com/Product/Product.aspx?Item=N82E16883103235) running XBMC Live installed on a USB Drive (xbmc.org) hooked up to each of the TVs in the house and use wired Gigabit (provided by whatever the cheapest 5 port Gigabit switch was on Newegg at the time, I think it was about $25) to stream DVD ISOs from the server to the TV. I don't have much HD stuff and the Revo can get bogged down when you try to play really high def video. 10Mbps works fine with XBMC using vdpau, but an 18Mbps MKV was a bit too much and went slideshow in places.

Backup Your NAS

[s.whiplash]

I have a similar setup to previous post. Server running Linux with a 3ware RAID card running RAID 5. 8 500GB disks with ~3TB of usable disk space. This has been running flawlessly for over 5 years. I have a a movie collection, music and network share for 3 HTPC in the house. Works very well but I wish my RAID card supported the ability to power down the disks and save of power/heat when not in use.

I am almost at capacity on the RAID volume, so to expand I have another RAID card that I can put in the server and create a new volume or replace the 500GB drives with 1TB or now 2TB disks. Replacing the disks would save power and heat, but I would need to backup and restore 3TB of data. Adding another RAID card is easy, but crates more dives that I can't turn off and eat up power.

I am actually thinking about building a new server with the thought of being able to add an many SATA ports as possible (via SATA cards) and then us port multipliers. The use a software based file system or RAID that allow me to add drives of different sizes to the volume. Similar to ZFS but more open. This would make growing the system much easier and allows me to power down drives when not in use. I would still get plenty of performance for my needs.

The other thing that I am doing that most people don't think about, is I backup my entire NAS to another server. I took another old PC that I had and put a 4 port SATA card in it and four 1 TB disks and run Linux and software RAID on it. Each night it powers up and runs a script to back up the primary NAS. I do this just in case something catastrophic happen to my primary NAS and I also use it when I moved to larger disks on the NAS previously. I use rsnapshot to look for changes on the primary NAS' file system and only back up data that has changed. It also keeps the lat 3 months of files that have been changed or deleted, if I need to recover something. When the script is finished, it powers down the backup NAS and wait until the next night to run again.

Best cost/benefit, if your desktop runs Linux...

[durval]

... is to simply put the extra disks on it.

My main desktop at home has been running Ubuntu Linux for the last 6+ years, and for the last 3 years I migrated the functions of my NAS (which used to run on a separate linux box) to it, so I have one box less to administer, to consume power, or to break down.

I have currently 10TB of total disk space using 5 2TB disks (8TB usable, when you account for the RAID5 redundancy overhead), but could easily migrate to 20TB on 10 2TB disks (16TB usable, and gaining extra redundancy by moving from RAID5 to RAID6). The result is:
- Very upgradeable (starting with 250GB disks back in 2004, I've migrated all the way to 2TB disks, and will continue doing so; the old disks are simply replaced with the newer/bigger ones and re-purposed as off-line storage, being plugged on a eSATA dock when I need them
- very fast (as the disks are on the machine itself, it's much faster than acessing the files on a NAS over the network);
- very usable (it's all mapped on a couple of ReiserFS filesystems, created on top of LVM volumes, directly accessible without needing to mount anything or configure anything over the network);
- very reliable (I'm protected against any one of the five disks failing, thanks to RAID5 configured on top of Linux MD, through for more disks RAID6 is really recommended, and ReiserFS in my experience is very reliable against crashes and power outages, at least *much* more reliable than EXT3 and XFS).
- very cheap: I've used the SATA controller already available on my motherboard, providing for 6 SATA disks, and apart from the disks I only had to spend money on a multi-disk internal rack: these are great, they fit 5 SATA disks on 3 x 5.25" bays on the front of your desktop, are very cheap (around $75) and give you hot-swapping and great ventilation (via a large, low-noise fan in the back) to boot. Just be sure to use a computer case that has no "rails" or other protuberances between each 5.25" bay, or else you won't be able to insert the rack as it spans 3 bays.

You could use a SATA RAID controller (or even SAS disks and a SAS controller), but I found that it's quite expensive, and unnecessary as the above setup gave me all the speed I needed, and them some.

In short, I'm very satisfied with my setup, and I recommend it to anyone who has large disk space requirement at home.

Some pointers to the hardware I'm using:
- Motherboard: Asus M4A78-EM which is reasonably cheap, very stable and has 6 SATA ports (5 internal and 1 external), fitting the bill perfectly;
- Disks: Seagate SATA 2TB 5900RPM Retail kits : The retail kit (instead of the bare OEM drive) gets you a disk with FIVE years of warranty (instead of just 3 years) and comes much better packaged (so reducing the chances of early death due to shocks during transportation).
- 5-disks-on-3-bays internal SATA enclosure: NORCO SS-500 : great little bay, as described above. - External eSata dock: Startech SATADOCKU2E: with it, when I replace my old (smaller) disks with new big ones, I can re-purpose the old ones immediately as off-line storage,very efficiently (my motherboard already has an eSata connector) and very cheaply (I store the disks on plastic storage cases when they are not docked, very cheap and compact.

Hope the above is of help.

Re:This should be modded up

[gbr]

I got a Drobo as well, the 4 drive USB/Firewire version.

I had a RAID array die, and I needed something fast to save my data before the weekend was over. The Drobo worked.

I'm now back on a Linux RAID system, and the Drobo is relegated to backups. Why? It's slow. Way slow. I wouldn't recommend for daily use.

Use D-Link Drive Enclosures - we do :-)

[code-dweller]

We have a similar problem at the Mad Lab - we generate a lot of data (from our digital studio and other projects), we need access all the time, and we need reliable storage.

At first we were putting lots of drives into a PC -- but that led to problems. For one thing there was a single point of failure (main board, power supply, take your pick). Another problem was that the system was loud and power hungry. Then there was the backup problem -- there was no efficient way to do it without building another system just like it --- you can't ship TB of data off-site via the 'Net for backups, it just isn't practical. Then to make matters worse we decided we couldn't do anything else with the server without putting our data at risk... that was the last straw for me -- The server was overkill for the task and couldn't be used for anything else. I was stuck in a paradigm - I knew better - but I'd forgotten that temporarily...

Then I hit upon the solution of using D-Link 2-Bay network storage devices. http://www.dlink.com/products/?pid=509 These little guys are reliable, solid, efficient, and affordable. They're also pretty green because they will spin down when you're not using them thus saving power and POH time on the drives.

We use them in pairs: NS0 for storage, and BS0 for backup. With a firmware upgrade they will do NFS - so we have one of our servers map the two devices and then rsync NS0 to BS0 once per day. Newer versions may have the NFS capability built in (it was on it's way, it was beta firmware when we did it).

Now we have the key features of the high-end NAS solutions we use in data centers, but we have it on the cheap. The solution is scalable (more storage, more enclosures), reliable (mirrored drives all around, fast and easy to access (NFS or Samba - take your pick), provides redundancy (outboard power supply for each enclosure - easy to swap, separate controller for each pair of drives), and easy to manage (what's not to love about a scheduled rsync task via nfs for automated backups?).

We can easily access the data from either windows or *nix boxen on the internal network without any trouble. When we need to access the data from outside the Mad Lab we shell into a server and sftp what we need from there.

Here is a pic of the two of them in the rack: http://www.lifeatwarp9.com/wp-content/uploads/2010/01/MadRackBefore-225x300.jpg We've been using this setup for quite a while now and it's been dead solid. When we need to expand we just plug in another unit and map it. We've also installed this kind of configuration in customer facilities to manage backups and solve other storage problems on the cheap.

_M

This was my Holy Grail - about two years ago

[rcolbert]

This thread has lots of good suggestions for storage. I have a distant business relationship with Drobo, and think they're an interesting choice. I have a Windows Home Server as well, and find it to be a step-up from my previous Buffalo Terastation NAS box from a reliability and performance standpoint. I happen to also have a full-size tower and appreciate the simplicity of throwing lots of hard drives at the problem.

However, as formats switched from DVD to Blu-Ray and equivalent HD content, the economics shifted IMO as well. Given the both my Sony TV's, as well as both Tivo's can stream on-demand HD directly (Sony does it without annoying buffering by the way), and considering that I only rarely watch a movie more than once or twice, it's actually become more economically feasible to simply rent HD on demand for $4.99 a shot. There are still going to be a few Blu-Ray discs worth buying to own the content, and more than half of those seem to ship with a free digital copy for import into Window Media Player or iTunes. Even if you own all your HD content on disc now, it's probably worth your while to look into a hybrid model where you rent what you have a passing interest in, and buy/store those few things that either aren't available on demand, or that you have a more long-term interest in retaining.

Oh, and for the porn, a 2TB drive in a large tower should be more than sufficient. Windows 7 and Bit-locker full drive encryption doesn't impede HD playback on a reasonably speedy system. Although it is debatable whether or not 1080p is actually a good thing in some cases. Anyone want to go into business with me creating a unique line of porn-star body make-up to deal with pimples, waxing irritation, and razor-burn?