Ask Slashdot: DIY NAS For a Variety of Legacy Drives?

An anonymous reader writes "I have at least 10 assorted hard drives ranging from 100 GB to 3 TB, including external drives, IDE desktop drives, laptop drives, etc. What's the best way to setup a home NAS to utilize all this 'excess' space? And could it be set up with redundancy built-in so a single drive failure would cause no data loss? I don't need anything fancy. Visibility to networked Windows PCs is great; ability to streak to Roku / iPad / Toshiba etc would be great but not necessary. What's the best way to accomplish this goal?"

Comments

Not worth it.

[Anonymous Coward]

Those older drives are probably failures just waiting to happen. With the cost of the hard drive space continually dropping, just use new drives. It's not worth screwing around with old ones for anything other than salvaging old data off them, even though the urge to do so is strong in the more frugal among us.

Re:Not worth it.

[ZeroSumHappiness]

I agree to an extent. Take anything SAS or SATA that's 1TB or greater and re-think the project with just those. Sell or recycle the rest of the drives. Depending on your needs the remainder should be RAID-1, 5 or 6'd (using software RAID if speed isn't an issue) and then put on an OpenFiler or FreeNAS box. Anything non-replaceable should then be backed up to a respectable backup provider in addition to your home-grown solution.

We need more information though -- what are your actual drive sizes and what do you want to put on this NAS?

Re:Not worth it.

[PaladinAlpha]

This.

With such a wide range of storage sizes, you're going to have serious trouble setting up any kind of redundant encoding. To mirror a segment of data (or the moral equivalent with RAID-5 or RAID-6) you need segments of the same size; those segments are going to have to be no larger than the smallest drive. That means larger drives have to store multiple segments, but that the segments have to be arranged in a way such that a drive failure on one of the large drives doesn't take the RAID down. If the drives can't be bisected -- that is, divided into two piles of the same total size -- this is impossible, and the fact that your range is from .1TB to 3TB implies this might be the case.

Think about it -- it's probably going to take most-to-all of those smaller drives to "mirror" the larger drive to make it redundant (and mirroring is the best you can do with just two drives). But having one side of the mirror spread across 9 drives makes failure laughably likely, to the point where you're paying performance penalties for nothing.

Your alternative is to use a JBOD setup and have just contiguous space across all of the disks. This is the same problem, except when a drive goes you lose some random segment of data. That's acceptable for two or three drives in scratch storage, but you don't want to actually store things on that.

Make no mistake -- those drives are going to die.

Trust me on this; don't go down this road. Your actual options are to either pair up the disks as best you can, supplimenting with strategic purchases, and make 2-3 independent raids (and maybe even RAIDing those, but it'll be painful), or just write the whole thing off, put disks in if you have obvious candidates in your hardware, and donate the rest.

Re:Not worth it.

[cpu6502]

Agreed.

Just take the sub 1/2 TB drives and mail them to me. ;-) I need some small USB drives for work to hold my music.

Re:Not worth it.

[King_TJ]

Yep.. I agree. "Not worth it." sums it up nicely.

Seriously, I completely understand the desire to re-use unused equipment you've got lying around. Seems like the smart thing to do, reclaiming as much of that unused storage space as possible and pooling it together so even the smaller drives add up to something worthwhile. But as a FreeNAS user myself, trust me on this one. It's NOT really a good idea.

As other already pointed out, most RAID configurations are limited by the size of the smallest drive in the array, so that would create major problems for you right there. But even assuming you skip RAID (or set up multiple RAID pools, each consisting only of very similar sized drives -- and then join all of them into a virtual master storage "device"), you're still in a situation where the lower capacity drives probably have slower data xfer rates than the newer, larger ones. That will drag the overall performance of the server down, whenever something gets loaded or saved to the slower/older disks.

Even if all of THAT doesn't discourage you? I have to ask what your time is worth, and to a lesser extent, what your data itself is worth? Old drives as small as 100GB capacity have got to be at least 4 -6 years old by now. Unless you bought them new and just stored them in a box this whole time, chances are, they've seen a lot of hours of operation already. They don't have a resale value more than $20 or so these days, so you're simply not out much money to throw them away or give them to a recycler. Meanwhile, you'll probably get into a much more complex and time consuming NAS configuration, trying to best utilize them in your drive pool. Even if you only make $10/hr. at your job, that means 2 hours of time spent messing around with this is worth the entire value of one of those old drives!

I'm kind of a pack-rat for computer hardware (since I have an on-site repair business besides a day job in I.T. and computers as a spare time interest too). But even I started throwing away IDE or SATA drives under 250GB a while ago. I keep a *couple* small ones around, but only for odd situations (like someone who wants to revive a really OLD PC with a BIOS that can't recognize larger drives properly). Otherwise, everyone who wants to go to the trouble of swapping an old/dead drive out for a replacement may as well spend the relatively small extra amount of money for a current model of much larger capacity, AND a full warranty still on it. Your data is usually worth it!

Re:Not worth it.

[hoggoth]

"This little drive is not worth the effort. Come, let me get you something..."
"The frugal is strong in this one."

Re:Not worth it.

[nschubach]

Depending on your needs the remainder should be RAID-1, 5 or 6'd

Wouldn't btrfs supposedly resolve this? It's supposed to put your redundant data on multiple devices and I would assume if the device had no more space it could use another device on the array as long as it wasn't the same as the redundant data. From everything I'm reading on the FS so far it looks like it's perfectly usable now if you can schedule a regular data scrub (like a midnight cron job) to check integrity, which wouldn't be bad for a personal server (enterprise is another story.)

Re:Not worth it.

[Stoutlimb]

Agreed with a caveat. My NAS has 500gb and up drives... anything else gets consolidated. I tried using those old drives, but I soon realized it just wasn't worth it. The 500gb drive is going byebye soon too. After a certain point, the empty slot and power draw becomes valuable real estate that could be populated by a larger drive. Slow speed becomes a factor for obsolescence in some cases as well.

What smaller drives, even the 80-120gb types, are good for, is boot drives for crappy refurbished computers for grandma, or the kid down the street who has nothing. They can always upgrade, but it's great for starter and old systems for people who won't be downloading very much.

Re:Not worth it.

[the_B0fh]

And what do you think RAID 5 is?

Re:Not worth it.

[bodangly]

I've had nothing but bad luck with btrfs, including irrecoverable data. (well, data not as valuable as time it would take to restore) It is my opinion that the push to make btrfs the new standard is happening way too quickly and for the wrong reasons. It has been my experience that it simply isn't as reliable as the more established file systems. I would highly recommend XFS over btrfs.

Re:Not worth it.

[hairyfeet]

I probably shouldn't answer an AC but since the AC has been put as "insightful" I will answer with why I think he/she is wrong: If the data has already been backed up (which if you don't back up your data you're a dumbass who should be posting to yahoo Answers and not here) then frankly there is no "risk" to using drives you already have as its only a question of how long it would take to restore.

Now of course he can mirror the data on half the drives to add some redundancy but your answer of using new drives? Until WD is back up to full speed frankly that isn't possible without a LOT of extra cost. Sure you can find cheap Seagate drives but you know what? they're shit. Not badmouthing Seagate but read the reviews and you'll see that Seagate drives over 640Gb are having a crazy high failure rate. Some say they are using ARM controllers made by the Maxtor guys and those are shit, some saying its a firmware issue, I personally don't know but what I DO know is that I've had to shitcan a crazy number of Seagate drives, especially the 1Tb and 1.5Tb drives which are the only really affordable ones ATM.

Now for how to do this, I'm gonna stay out of the software side since I don't want to jump into a Windows VS Linux flamewar, I'm sure the guy has an OS he is comfortable with and will probably go that way anyway so I'll deal with the hardware. This is how we cooked up something similar at my previous shop with a shitload of SCSI drives the boss got at an auction...Buy a couple of matching cheap full size computer cases, geeks has several for pretty cheap. We then tore the cases apart leaving a couple of skeletons, how far you take them apart is up to you, one can just as easily take the side of one and the opposite side off the other and cut the bracing, the reason we did it this way will become obvious in a minute. Pick up a cheap old server board, you want one that will fit the case and has as many PCI slots as possible, you will of course fill the PCI slots with SATA adapters just like we did with SCSI. Then in our case for the final touch we wired up a $10 Walmart box fan to the side to cool all the drives we piled into that sucker. In our case we used a copy of Win2K Server since we had drivers for the SCSI cards in Win2K Server, but again software is your choice.

And there you have it! While drives were topping out at 400Gb and cost an arm and a leg we had nearly 2Tb of SCSI goodness containing every single driver for every single part for every single version of Windows from 3.1-WinXP. I don't see why someone couldn't do the same with SATA, sure PCI won't give you as much bandwidth as PCIe but if you have a decent sized amount of RAM (say 2Gb) to buffer I don't see why it wouldn't work. in the end this is about using something you already have which won't cost anything VS spending hundreds of dollars to acquire a more compact solution. Personally I'm all for using what you already have, that is why I have a drawer filled with drives from 80Gb to 400Gb that I then throw into computers that are short on space, certainly cheaper than buying a new WD drive and as I said I don't trust Seagate ATM. Personally I'm just glad I loaded up on Samsung EcoDrives right before the flood when Tiger had them cheap so I can afford to wait until the prices drop before i have to look at drives. But if he already has them, why not use them?

Re:Not worth it.

[isorox]

This.

With such a wide range of storage sizes, you're going to have serious trouble setting up any kind of redundant encoding

Yes. But wouldn't it be awesome if there was some magical filing system daemon that ran on multiple machines, automatically meshed together, and presented a single contiguous file system, which kept at least n copies of data on m machines, auto-replicated (reducing available free space) if a disk died or a machine was offline for > p hours, and offered some simple admin interfaces. Throw in an LTO interface (perhaps LTFS based, perhaps not) for good measure too.

This is slashdot, we all know how to use raid, and we're capable of looking up where things like drbd are, but something like unraid at a network level?

Re:Not worth it.

[nschubach]

RAID5 requires that all the disks be the same size and uses one drive as a parity bit drive.

btrfs works on the block level and should be able to write redundant block data on multiple drives no matter what size they are.

Re:Not worth it.

[tlhIngan]

Old drives as small as 100GB capacity have got to be at least 4 -6 years old by now. Unless you bought them new and just stored them in a box this whole time, chances are, they've seen a lot of hours of operation already. They don't have a resale value more than $20 or so these days, so you're simply not out much money to throw them away or give them to a recycler.

Exactly. Though, 100GB drives are plenty of storage for people. If you want to earn some brownie points at the next family gathering, you can stuff them in your parent's PC or something or get a small drive-less NAS appliance and use it as a backup server.

Doesn't cost much, but lots of brownie points and you ge tto fulfill that need to reuse old hardware. Especially the backup side - most likely they don't have a backup routine or schedule or something.

Or hell, maybe they've been complaining that their hard drive is full. Use it to upgrade their PC on the cheap. Or install a new version of Windows without wiping out the ability to use XP or something. Or stick on Ubuntu on it.

The common user will be thrilled to get a "big" hard drive for free. Use it to do best practices (backup), evangelize (install Linux), or just make them happy by having more space for their photos, whatever.

Re:Not worth it.

[19thNervousBreakdown]

It depends--is there a total of 6 TB of drives that doesn't include the 3 TB drive?

Take each disk, make an LVM physical volume from it. From those physical volumes, logical volumes. You don't have to make all of them the size of your smallest drive, you just have to be careful. Say you have the following:

1: 3 TB
2: 2 TB
3: 1 TB
4: 1 TB
5: 750 GB
6: 750 GB
7: 150 GB
8: 150 GB
9: 100 GB
10: 100 GB

On your 2 TB drive, make partitions matching the drives under 1 TB.

On your 3 TB drive, make the following partitions:

1 TB: RAID-5 with #3 and #4
750 GB: RAID-5 with #2 and #5
750 GB: RAID-5 with #2 and #6
150 GB: RAID-5 with #2 and #7
150 GB: RAID-5 with #2 and #8
100 GB: RAID-5 with #2 and #9
100 GB: RAID-5 with #2 and #10

You'll end up with the following volumes:

1: 2 TB
2: 1.5 TB
3: 1.5 TB
4: 300 GB
5: 300 GB
6: 200 GB
7: 200 GB

Then take those, LVM the RAIDed LVM volumes (fairly certain you can stack [traditional meaning of "stack"] as a contiguous disk, just use an easy FS like ext3, I've run into problems with stack size [programming meaning of "stack"] using XFS on LVM). You end up with 6 TB total space, and, just like normal RAID-5, you don't lose anything unless two disks from one of those groups die. That is, if a disk in 200 GB #6 dies, and a disk in the 1.5 TB #3 dies, you still haven't lost anything. Even if your 3 TB drive dies, which is clearly the worst case since it has data for every array, or the 2 TB which is nearly as bad, you'd still need to lose a second disk to lose any data, so for failure rates it should be the same as a 10-drive RAID-5 array, which isn't quite advisable although it's not murderously bad, but this isn't work and the primary motivation is probably maximizing space with a decent reliability increase, not making next to certain it never goes down. I'm sure it feels really weird, but I don't think you're actually increasing your odds of failure at all over the 10-disk-all-same-size RAID we're used to, other than not trusting older drives--and I'm not so sure those are much more likely to fail than new ones. After all, they've lasted this long, and I've had brand new drives die within weeks. But in point of fact, there's some 2-drive failures that don't take anything down, so I think overall you're doing slightly better than the 10-same-size disk case.

Now, your disks probably won't divide up as nicely, and you might end up having to either leave some space on the floor or subdivide in weirder ways or both, but with very careful partitioning (never put two stripes of the same array on the same disk), you can do this. Set all the arrays to verify weekly (mdadm can do this) and e-mail you on a failure. Don't set up an audible alarm, you're not going to lose a second disk at 3 AM (but you will wake up to fix it, and be worthless at work the next day for probably nothing) and even if you did lose another disk, you're not using RAID as a replacement for backups, right? Right?

ZFS would be really nice if it did all this complex stuff for you, but do you have enough control/is it smart enough to allow you to ensure that you get as good or better reliability? It'd be ridiculously easy to make a bad mistake in layout with the above scenario. Because overall, I agree with the title: It's just not worth all this effort so you can use that crappy 100 GB disk. Once it goes down, now you have to replace it.

Re:Not worth it.

[geekoid]

"Even if you only make $10/hr. at your job, that means 2 hours of time spent messing around with this is worth the entire value of one of those old drives!"

The idea that the wage you make at work applies to all your time is a fallacy. The ONLY exception is someone who can just say "Hey, I'm going to work a couple extra hours on the fly".

And do you seriously believe the time learning and building technical stuff has the same value as some min. wage job?

Great, you throw out perfectly good drive. Bully for you.

Re:Not worth it.

[nschubach]

Regarding empty slot usage... I imagine the plan is to replaced failed drives with larger drives over time and have the system adapt. I've given old drives to people in kind only to have them come back a few months later because there was a problem with their machine (mostly bad clusters) and since they didn't build the machine they didn't know how to do regular disk checks or have the machine running during the schedule I set up.

Re:Not worth it.

[ZeroSumHappiness]

RAID 5 stripes the parity across all the drives. RAID 3, I believe, has a dedicated parity drive. Otherwise correct though, I think.

Re:Not worth it.

[spongman]

And what do you think RAID 5 is?

inflexible?

Re:Not worth it.

[JesseMcDonald]

... larger drives have to store multiple segments, but that the segments have to be arranged in a way such that a drive failure on one of the large drives doesn't take the RAID down. If the drives can't be bisected -- that is, divided into two piles of the same total size -- this is impossible, and the fact that your range is from .1TB to 3TB implies this might be the case.

There is a simple formula to determine the available space in these circumstances: If the largest drive is larger than all the other combined, the available space (after mirroring) is the sum of the smaller drives. In this case the largest drive mirrors all the others. Otherwise, the available space is half of the total of all the drives, and no space is wasted. A filesystem like BTRFS (or, presumably, ZFS) can work out the details automatically if you set it up in RAID-1 mode.

But having one side of the mirror spread across 9 drives makes failure laughably likely, to the point where you're paying performance penalties for nothing.

Agreed. I would probably forget about using any drive under 1TB. The more drives you throw into this sort of mixed-size RAID-1 array, the more likely you are to have multiple drive failures and a risk of data loss, particularly since we're talking about older hard drives.

Re:Not worth it.

[Xeno man]

Learning is one thing but the question was "what is the best way" The answer is not at all. Those perfectly good drives are far from perfect and probably nearly no good. Regardless how much you value your time, don't bother wasting your time putting together a bad system that will cause you many more hours of headaches in the near future when it does fail.

Re:Not worth it.

[Anonymous Coward]

If you want to earn some brownie points at the next family gathering, you can stuff them in your parent's PC or something or get a small drive-less NAS appliance and use it as a backup server.

And then you can enjoy not being *invited* to the next gathering, after a backup appliance full of out-of-warranty disks sends all their files to the Great Beyond.

As others have pointed out, there's not a huge number of uses for drives that are that prone to failure - scratch disk comes to mind, but anybody who actually needs 100GB of scratch probably needs it to be seriously fast, making old drives less likely to help.

Re:Not worth it.

[arth1]

Weeell, true, but...
A bunch of small drives plus a fast one means you can have a reasonably fast raid of the smaller drives, and do nightly backups of it to the large drive (or nightly reassemble and restore if more drives have failed than you have resilience for).

I has a setup like that once, with six 40 GB drives and one 80 GB drive (this was a while ago).
Two RAID1s striped to make a RAID 10, plus two standby drives and a backup drive. Worked like a charm, and survived three drive replacements over the years. Once the second 40 GB drive died, it was replaced with an 80 GB drive, and then the 80 GB backup drive died and was replaced with a 120 GB drive.

Of course, then SATA came out and I started over.

Re:Not worth it.

[AmiMoJo]

There are plenty of non-RAID and non-critical ways you can make use of smaller drives. For example most online storage accounts require sync with a local drive (Dropbox, Google Drive, Skydrive) to work and that can add up quickly. Doesn't matter too much if the disk dies because you have an online backup. You can use something like SecretSync to encrypt the log (which also doubles the storage requirements, another reason why a spare HDD is useful).

You could give some drives to friends and set up your own mutual offsite backups. Crashplan lets you do that for free with full encryption. Again, if the drives fail it isn't the end of the world.

Offline backups are not a bad idea either. Storing your home videos on optical discs alone is not such a good idea and these days you can shoot hundreds of gigs over a weekend. Use HDDs as a secondary storage medium. If they fail all is not lost because you have the optical media backups.

Consider getting/upgrading your PVR. Many TVs have a PVR feature built in, you just need to add a HDD.

The key is to find uses that involve data you can afford to lose.

Re:Not worth it.

[AmiMoJo]

The OP probably isn't too worried about security because most of the data will have been downloaded. There are a few jobs where people produce terrabytes of data, but those people will spend money backing it up properly.

Forget RAID and just use JBOD. If one drive dies get downloading again.

Re:Not worth it.

[cayenne8]

But I'd have a use for it...over the years, I've gathered a bunch of disks when on sale...many barely used...some still in boxes....

I'd like a way to throw them all together and use them for backup storage.

From what I've gathered...use FreeNAS, with ZFS...and it will let you set this up, and allow for up to 2x drives to fail at the same time....

I think in my case...this would be reasonable. Heck, if I set up two FreeNAS boxes...had one mirror the other one...that would indeed be a decent backup system, no?

I have a lot of friends like me....often buying stuff on sale for "to use someday on something"...but they just sit and gather dust....I think this would be a good reason to use them, and keep buying new drives, here and there when they go on sale, to replace on the FreeNAS as drive on it do start to fail....

Heck, thinking of keeping one FN server here..and maybe put a 2nd at friend or parents house out of state...to mirror it...

Re:Not worth it.

[cayenne8]

I think the assumption is that all the drives are old and have been run though the ringers.

If they're like me, and I mentioned this on a previous post....maybe they have a ton of drives, but most of them are little used or even unused in a box.

I have a habit of buying stuff, like drives when I see them on sale...and just set them back for use some day...and kinda forget about them.

I've got computers I was going to use for something....new-ish drive in it...and that box had RAM or MB problems...just sitting there. I could toss those boxes out....use the drives and make use of them with something like this?

My trouble is...not knowing the best configuration for a computer to use...to hook up all these old drives...to be able to hook the max drives up to it.

I have some SATA...I have old, almost unused IDE drives....I suppose set up a couple boxes...but how old? What kind of CPU? For the IDE ones..I guess I could only max that out with 3 maybe 4 drives....but for one with SATA...how many drives could I hook up to that?

Re:Not worth it.

[undeadbill]

You mean like GlusterFS? www.gluster.org

It supports access via CIFs, NFS, and Fuzelibs. Of course, his smallest disk is going to limit assignment of one filespace brick for replication on another drive to a file of the same size, but he could conceivably jigger around his assignments to use all of the disk space he has.

Well, I hope he has a *really* fast switch if he does that; and there is that issue with power for all of those hosts, if he wants that kind of redundancy...

Re:Not worth it.

[segin]

I have a number of old and "low" capacity drives that I wouldn't mind utilizing. I know they have well over two or more years working life in them on average, as their actual usage until now was very sparse. If "attic time" didn't take from a drive's operational lifespan, I'd say they've got 6-7 years left (given that MTBF is 8 years, on average) - in this case, it's not too bad of an idea to try to utilize them. Also note that MTBF is Mean Time Before Failure, not Maximum Time Before Failure. Drives are known to fail much later (and much earlier) than the standard 8 years. I've drives that have 10+ years of operation to date, and they are checked regularly just in case (and all important data is redundantly stored elsewhere - just the systems are old and I haven't extra drives or the give-a-fuck to worry about it. They work, so let'm keep on truckin'.)

Re:Not worth it.

[segin]

You can always buy PCI or PCI-Express IDE controllers for trinary and quandary IDE channels.

Re:Not worth it.

[deAtog]

Generally, yes it's not worth the time or the effort. However, if you're serious about taking advantage of your old drives, I'd suggest using RAID on top of LVM. LVM will allow you to group drives of different sizes together to form a logical volume. You can then use software RAID to ensure data integrity. Over time, as drives fail, you can replace failed drives with new ones and rebuild the failed logical volume. A simple Samba server should suffice for your file sharing needs.

Re:Not worth it.

[skids]

...if you keep them spinned up when you aren't using them.

I've got 4 or so drives crammed into a nice old, quiet, low-wattage G4 powermac, running Debian headless. (3 onboard controllers + a 2x pci card so I could fit 10 drives as log as I don't care much about speed). They stay unspun pretty reliably once you tweak a few small things. Unfortunately WoL is broken on this model, so if I ever moved all the non-FS services off it, I couldn't power the mainboard off remotely.

Re:Not worth it.

[Immerman]

What's the real-world difference? Assuming you perform verify-on-read then if you have N total disks you always need to read/write N total blocks at a time to store N-1 blocks of real data a time so performance should be the same and, if one drive fails you can recreate it's contents from the missing data from the data on the other N-1 disks, regardless of whether the drive was holding "real" or parity data, or a mixture of both. It doesn't seem like it should matter what particular arrangement of real and parity data is used. What am I missing?

Re:Not worth it.

[wampus]

Or you could buy a new disk the size of everything you're going to connect to this IDE controller for $10 more, use less juice, and be less prone to failure.

Re:Not worth it.

[asdf7890]

RAID 4 is the dedicated parity counterpart to RAID 5.

RAID 3 is striping with parity (and all parity information held on one device) though so you are not wrong in that fact. RAID 2 is also. R2 and R3 are the same as R4 but they work on individual bits or bytes where R4 works on blocks of bytes. Because of the performance characteristics of all modern drives (and old ones for that matter) R2 and R3 are not generally ever used.

Re:Not worth it.

[Pentium100]

Even if you only make $10/hr. at your job, that means 2 hours of time spent messing around with this is worth the entire value of one of those old drives!

Too bad that if even if I work additional two hours at my job, they don't pay me more, just a fixed salary. So, the time when I am not working is worth zero.
Also, while somebody would not pay a lot for my car if I was selling it, the car is still worth to me more than that (that's why I'm not selling it). Same thing with the drives. If I sold all the old drives, I would not be able to buy a new drive of equivalent capacity, so it means that the drives are worth to me more than what the buyer would pay and that's why I'm not selling them.

I have a few old drives ranging from 120GB to 500GB (mostly IDE) that are still in use, but just set up as separate drives. Some are even in separate PCs (ran out of case space).

Still, new drives are expensive, I bought one recently, but for use in addition to all the old ones. Buying a 2TB drive just to replace all the old drives (with no gain in total capacity) is really not worth it and puts most of the data on a single drive (at least now if one drive fails I don't lose all the data, so I can just make multiple copies of the really important data and put them on different drives).

Re:Not worth it.

[Anonymous Coward]

Doesn't matter if they have never been used. a 100Gb drive is 1/30th of a 3tb drive but uses the same electricity. A better idea is not to buy things just because they are on sale, but because you have a need for them. Buying things coz they are on sale is a stupid womans habit.

Re:Not worth it.

[Anonymous Coward]

+5 insightful and it doesn't answer his question at all. Typical of most forum answers unfortunately. He didn't ask 'is it worth it to setup a NAS with these old drives I have?', he asked 'how do I setup these old drives with NAS?'. Some people enjoy doing this kind of thing. Some people fix up old cars when it is surely easier and you can get better performance just buying a new one.

I have an entire network at home with outdated hardware and outdated routers. I've put some routers behind other routers. Why? Because I learn how to setup varying firewalls and test them all within my home. I better understand how things work. Does the end result give me anything that newer hardware wouldn't? Does putting routers behind each other give me any benefit at home? Only in my knowledge. I know more about subnets, port forwarding, and static vs dynamic ips than 90% of certified techs do.

I enjoy this kind of tinkering. It's why I'm a geek. So while I'm not an expert I'll at least share my thoughts on answering the question. I would suggest starting with the interfaces. If speed is not issue (and presumably power consumption) you might start with unifying them in a way in which you can add more. You'll either need to start adding cards to connect to all of those drives or consider something like this to be able to keep adding drives.

Once you get them all connected the next part of the question is making them backup each other up. You might consider combining them with software partitions (ie striping them) and then using software to back them up. In other words, take you're total drive space and split it in two. Create two software partitions adding up to the total. Then make one partition your primary and the other the backup. More than likely the split won't divide evenly between your drives and you won't want to split one physical drive between your two software partitions so take that into consideration (ie in other words your smallest drive will only use part of it's space to pick up the slack on one of your partitions, the remainder will either be wasted or create a 3rd partition that isn't backed up).

You might consider looking at FreeNAS based on BSD.

I haven't ever done anything like this so my thoughts are just speculation. You are likely to run into a lot of road bumps but you are likely to learn a lot in the process. I'd love to know how it turns out.

Re:Not worth it.

[similar_name]

I'm curious. Would there be any benefit to putting the NAS OS and swap drive on two of the smaller drives? They might be a little slower than the larger drives but is there any benefit to keeping the drive heads from having to switch between OS, swap, and data reads? Of course cheap RAM may solve that and I'm probably showing my age.

Re:Not worth it.

[Immerman]

Hmm, yeah, I don't see it protecting against catastrophic drive failure, but then I don't think I've never actually experienced an immediately catastrophic drive failure, what actually happens in virtually every drive failure I've experienced is that blocks start going going bad at an ever-increasing rate. When it becomes obvious there's a problem I get a replacement drive, cross my fingers, and start copying stuff over starting with the most important. When the process eventually chokes on an unrecoverably bad block it chugs away for a while before throwing it's hands up in despair, by which point even more of the blocks that used to be good have gone bad. Usually it takes several attempts to recover most of the files from a failing section of the disk, and it's not that uncommon for the directory listing itself to go bad before everything potentially recoverable has been copied (What do you mean directory Foo doesn't exist, you were just copying files from it!)

Now assuming that's a common failure mode and not just my personal gremlin's sense of humor then I could see a very good use for that old small drive as a parity backup. Say it's 1/5 the size of your primary drive, then for every 5 blocks on the primary you store a parity block on the secondary. If the primary starts to fail you get a new drive and start the copying process, only this time any time it encounters a bad or failing block there's a fair chance it can recreate the data from still good blocks and parity data rather than repeatedly retrying the same block and losing more data all the time. It won't be 100%, but it could dramatically increase the amount of data you could save from a failing drive, especially if you reserved additional space for complete backups of partition and file system information so that those don't immediately lose you all the data they contain.

I suppose actually you could get much the same effect by simply having an error-repairing parity system between your file system and the hardware, though that would cost you capacity on the primary drive instead of making use of an older low-capacity drive.

Okay, probably not huge demand for this, but if anyone decides to build such a thing as a fun project, let me know about it, would you?

Hmm, now I've got myself thinking - a good data-recovery technique could be to immediately copy the entire file system block by block, without any attempt to get a good read on the problem blocks. Then go back for a second try on all the problem blocks, then a third try on all the ones that still couldn't be read, and so on until the drive completely chokes. Ideally you'd probably want to just read the directory information to start with and let the user indicate high and low priority areas and order the recovery process entirely. And you'd probably want to focus recovery first on those files that only contains a few bad blocks rather than wasting precious time trying to recover blocks in a file that only contains a few good ones. Does anyone know if there are tools that do something like this? Most everything I've seen has seemed to focus more on getting the most perfect copy possible of file X rather than recovering as much as possible from an actively degrading filesystem.

Re:Not worth it.

[whoever57]

The best value ($/GB) for hard drives at the moment is 2GB. 3TB hard drives are still relatively expensive and if you go for 3TB WD hard drives, you have to deal with the "Advanced Format" issues.

Re:Not worth it.

[whoever57]

Whoops: s/2GB/2TB/

Re:Not worth it.

[cayenne8]

I don't really have anything as small at 100GB drives.......and none of the drives I have bought, were small or out dated when I bought them...just was on sale and a good deal at the time.

I have 500GB ones laying around, in boxes....would like to use them....especially if I could put a box together to use like 8 or more of them....

Re:Not worth it.

[cayenne8]

Or you could buy a new disk the size of everything you're going to connect to this IDE controller for $10 more, use less juice, and be less prone to failure.

Wow...I've seen other posts like yours about the electricity usage.

Are there that many people out there these days, having problems paying the power bill? I mean, if you've got money to have computers laying around wanting to load them with hard drives laying around....I'd have to think you could afford to not be terribly concerned about your power bill.

And many of my drives I have laying around....many are still new in box....or in computers I put together and rarely fired up for one reason or another...not a lot of usage on the used ones, so not worried about failure on them really.

Re:Not worth it.

[v1]

BASIC raid5 requires identical capacity drives. Intelligent raid5 does not. Imagine a four drive array, containing three 40gb drives and one 30gb drive. So you can treat the first 30gb of the 40's, and the 30gb as one segment of the storage, (3 data and a parity, total of 90gb of protected storage) and the remaining 10gb of space on the three 40's run as 2 data with a parity, for an additional 20gb of protected storage. (if you tried to just ignore the oversize on the 40's you'd lose out on almost 20% of your potential capacity) Those sizes aren't realistic but you can see the application easier.

You just organize the data groups to each span as many drives as possible, to make the most efficient use of the storage. It's not a requirement to use the same grouping strategy on all groups within the raid. Some of the groups may even end up as mirrors. (as would be the case with two 40's and a 30, the extra 10 on the 40's would be mirrored to recover an additional 10gb of space)

Re:Not worth it.

[ZeroSumHappiness]

If you do not verify-on-read: RAID-4 (dedicated parity) doesn't use the parity disk on any reads. RAID-5 spreads reads among all disks evenly.
If you do verify-on-read: RAID-4 always reads from a particular disk on all reads. RAID-5 spreads reads among all disks evenly.

This would mean that wear levels would be uneven on RAID-4 but not RAID-5 (excepting highly-used files being on particular disks of course) and with verify-on-read RAID-4 can't parallelize reads since it always needs to read the parity disk but RAID-5 can.

Writes are similar with respect to read wear and write speed.

The difference in speed will probably be trivial for a variety of reasons, but the difference in wear level could be meaningful.

Re:Not worth it.

[Mista2]

Generally, I'd agree, however this risk aside:
Get a tower case with enough power to run all the drives you want, and any sort of mainboard/CPU just up to 1GB memory should be fine, and enough SATA/IDE orts for the drives you want.
Then install OwnCloud or OpenFiler. Job Done.

Re:Not worth it.

[AnRkey]

I agree to an extent. Take anything SAS or SATA that's 1TB or greater and re-think the project with just those. Sell or recycle the rest of the drives. Depending on your needs the remainder should be RAID-1, 5 or 6'd (using software RAID if speed isn't an issue) and then put on an OpenFiler or FreeNAS box. Anything non-replaceable should then be backed up to a respectable backup provider in addition to your home-grown solution.

We need more information though -- what are your actual drive sizes and what do you want to put on this NAS?

I use freenas in enterprise situations. It's solid, reliable and it works! For a good, low powered machine that runs as an ISCSI target, use a HP Microserver with a 2GB flash drive slotted into the machine inside. It will take 4 drives and another 1 if you convert the ROM bay. I have 4 running 24/7 and they have not let me down yet. R

Re:Not worth it.

[neyla]

It's not about being able to afford the power-bill.

It's about the fact that it makes no financial sense to spend (say) $30 in electricity keeping a bunch of old drives running when the same additional capacity in a new disk costs $10.

Re:Not worth it.

[Joce640k]

I think the assumption is that all the drives are old and have been run though the ringers.

"wringers", the word is "wringers"

Re:Not worth it.

[Joce640k]

The real problem with C++ for kernel modules is: the language just sucks.
-- Linus Torvalds

That actually says far more about Linus than it does about C++, just sayin'.

Re:Not worth it.

[Anonymous Coward]

Consider power and heat requirements there..

and stop spending limited income on things you do not need ( a bargain on 80 gig drives when you have no use for them turns into far less of a bargain 10 years later when you have a bunch in a box)

Take the advice being offered.. pull out the largest ones and use them in a single box .. even with all the power in the world.. your still gonna face the logical limits of drives in cases.. especially pata.. where your bargain bin raid card that supports more than the 4 standard devices an old mainboard would support..likely is so legacy in terms of support that it will not be supported in a current version of openfiler or freenas..

Before you think i am just hating on putting old hardware to use though... lemme offer a constructive use for all that old outdated hardware you cannot use...

Find a charity that builds computers for schools or repurposes them.. make a lovely donation of all that bargain hardware.. it will actually be put to use.

http://www.computerswithcauses.org/ is just one of many organizations putting older devices past their prime into hands that can actually use them!

Out of what is left as has been suggested grab the largest sata drives and build no more than 1 NAS.. this will help your power bills/heat generation.. if you STILL have extra drives.. consider using them as "near online" storage via one of the clever little sata docks that do not require "a case" to make a sata drive work.. just drop it in and go (another option is spend a few bucks on a case with a hotswap drive bays or buy a hotswap bay but that goes against the idea of making use of slightly older tech.. though again it will save on power consumption in the long run)

Also most 100% definately DO NOT RAID any ide drives you have in your possession, there is no chance in hell that even a brand new in the box 80 gig drive from 1990s is going to be reliable "just because its new" .. they will fail due to age even though they are new or nearly new (this applies also to early sata) you are better off struggling with multiple mount points than you are trying to deal with .. your entire raid being done for.. because you used older drives and the replacement to get your RAID going again is gonna cost you 400$ new or wasting 99% of your capacity to get the data back..

100% suggest donation strategy, and using your newest drives if they are over say 750gigs each for a sata based openfiler/freenas box

Re:Not worth it.

[GameboyRMH]

I second the "not worth it" sentiment, and as a cheapass with great technical knowledge, trust me, I've tried.

Now here's the secret to cost-efficient hard disk usage: Only replace a disk with one at least twice as large, then you let the space "trickle down" to your other drives (for example, my home server has 2TB torrents, 1TB other storage, 320GB backups, 20GB OS...when I upgrade it will be 4TB torrents, 2TB other storage, 1TB backups, 320GB OS. At one time it had a 320GB disk for the torrents. Each one has a partner of the same size for external backups). By the time a disk is finally retired from use it will be puny by the standards of the time.

Another trick is to pool many smaller disks in a failure-prone way with ZFS/BTRFS*/Hammer/etc and back that fragile pooled storage up to one big disk, but this is bad for power consumption.

*Not ready for production yet...trust me, I've tried.

Re:Not worth it.

[GameboyRMH]

+1 BTFS isn't ready. I was messing around with it, wrote a dedupe script and on the reflink operation it froze the whole PC and ruined one of the filesystems I was testing it on.

Re:Not worth it.

[cayenne8]

It's not about being able to afford the power-bill.

It's about the fact that it makes no financial sense to spend (say) $30 in electricity keeping a bunch of old drives running when the same additional capacity in a new disk costs $10.

I dunno...why waste 500GB - 750GB drives, many that are NIB....? I mean, I'd rather use them, then spend a few $100 for new 1TB drives......I mean, the power difference between running a FreeNAS system with 5-6 drives...isn't going to be much different if I'm loading it with 500GB drives, vs 1-2TB drives....??

Re:Not worth it.

[julesh]

Unless he means they've been modified to look like another drive so that they won't be spotted as stolen. (cf http://ringer.urbanup.com/3183368 )

Re:Not worth it.

[queBurro]

you're right, replace all those old IDE drives with brand new SSD ones! what could go wrong...

Re:Not worth it.

[mcmonkey]

I don't think anyone is saying a new or newish 500 GB SATA drive isn't worth using. They're saying a 80 GB IDE drive, even MiB, isn't new enough to expect reliability.

And if you're in a position where you need to spend money to use the 80 GB drive, you can likely spend a similar amount on a new drive that will be orders of magnitude larger, have a higher expected reliability, and use less power.

Re:Not worth it.

[mcmonkey]

Wow...I've seen other posts like yours about the electricity usage.

1. why spend it if you don't need to? Is another $10 on the electricity bill each month going to cause me to go hungry or lose my home?

No, but that's doesn't mean at the end of every month I take a $10 bill and throw it away because I am so rich.

2. More electricity usage means more heat. This compounds the cost if you pay for cooling and shortens the life of components.

Do you have every light and every electric appliance in your home on and running 24x7? Why not?

Re:Not worth it.

[Artifex]

I agree. About two months ago a 2TB drive I'd set aside for backup purposes grindingly crashed when I attempted to check to see if anything was on it. Turned out it was one of a batch of Seagate drives with bad firmware; they'd made a patch available but didn't make it easy to use back when I'd filled the drive already, and I forgot later. (And since I've now read several batches of different models have had different types of firmware-based failures, Seagate's been booted from my list of vendors.) The sticking point, of course, is that new 2TB drives are more expensive now than they were back when I bought mine.

Re:Not worth it.

[CAIMLAS]

I agree with this 100%. 100GB drives are in the ballpark of 8 years old now: that's almost outside the failure curve. I would not trust them to consistently spin up at this point. (Granted, I've seen entirely too many 'enterprise' 36GB disks in use, still, and those keep going too...)

In terms of the possibility of the project, I would consider it impractical if not impossible. It is certainly not maintainable. In terms of software technology, you are basically looking at using either something like Linux md RAID with lvm, ZFS, or (god forbid) FreeBSD geom. Geom really has nothing going for it, but mdraid is going to probably fail for you due to having frequent disk errors and ZFS will likely fail due to not handling vdevs with differing performance characteristics well and not being able to be dynamically expanded (like mdraid). You'll need at least 2 disks of the same/similar size for each primary component of the system (eg. a 150G and 120G in a single parity 120G array (RAID1)) unless resiliency and total data loss is not a concern for you.

A limitation you may run into is number of disk controllers. Not knowing how many disks or systems you've got, I can't say how feasible this is. I can say that the 8-disk LSI 1068e controllers are well vetted and fairly cheap. IBM and many others have been bundling PCI cards with either these controllers or their crippled siblings (eg. only 4 disks) with their servers for years, so if you have a machine with 4 PCIe slots and know where you might be able to get some of these (and then flash with the latest LSI 1068/1064/etc. TI firmware) you've got something to look at. Once you start spending money, though, I would recommend just buying disks (again, if this is anything more than an experiment and your data is worth anything). My personal price cutoff for 'janky' is about $25 unless there's a ready personal benefit (satisfaction of something convoluted/difficult) in doing so.

It would be interesting as an experiment, however, to see how resilient something like ZFS is with such old and error prone drives. Personally, I would not even attempt to use anything like this for more than just a JBOD backup backup, or the like.

Re:Not worth it.

[mcmonkey]

Still, new drives are expensive, I bought one recently, but for use in addition to all the old ones. Buying a 2TB drive just to replace all the old drives (with no gain in total capacity) is really not worth it and puts most of the data on a single drive (at least now if one drive fails I don't lose all the data, so I can just make multiple copies of the really important data and put them on different drives).

2 TB drives are a little over $100. (I recently purchased 2 for a NAS.)

I did a little web searching and back of the envelope calculations. It seems power consumption of HDDs hasn't changed much over the last 5 years. What has changed, however, is power per GB as storage capacities have changed.

If you have a 2TB drive drawing 8 watts and pay $0.11 kWh, running the drive for 5 years will cost about $38.57. Add the cost of the drive (e.g. $120) and your 5-year cost is $158.57.

If you have 4 500-GB drives with no out-of-pocket cost, your power cost for running those 4 drives for 5 years is about $154.28.

So you saved 4 bucks and change. Plus you get a higher risk of failure with multiple older drives rather than a single new drive.

You mention the redundancy of multiple drives but also say the new drive has no gain in capacity. Well, you only get one or the other. If you're using 4x500 GB as redundant storage, you don't have the same capacity as a single 2 TB drive.

If you want the same storage AND redundancy, you'll need more than 4 of those 500-GB drives, and that pushes up your cost.

So how are new drives expensive?

Re:Not worth it.

[cmdrbuzz]

With the HP Microserver (I bought one yesterday) would you use a RAID card or just standalone drives?

Re:Not worth it.

[Pentium100]

OK, I may have 1.95TB instead of 2.00TB - I don't have that much irreplaceable data - I tend to back it up. The thing is, if a 2TB drive fails, I just lost almost 2TB of data, most of it can be replaced but it's still a pain. If one of the 4x500GB drives fails, I lose 500GB of data, which is less than 2TB and all of it can be replaced.

Also, buying a new drive means I have to have the money right now - I don't really want to turn the drives off and save the money by not paying for the electricity that the drives are not using (since they are off) over a year or so.

Re:Not worth it.

[Azmodan]

I, for one, welcome our new overlord 19thNervousBreakdown, for writing one of the best +5 Informative post in a long time.

Re:Not worth it.

[Reziac]

As someone who runs a bunch of older hardware and still finds those "small" hard drives useful... I am utterly horrified that you're pitching them out, and I wish to know the location of your trash bin. :D

Re:Not worth it.

[hairyfeet]

The magnets are also great to scavenge as they'll put a magnetic charge on your screwdrivers you wouldn't believe. But it is nice to see someone get what I'm talking about because once your data is backed up those drives are "free" so any uses you find for them are just gravy.

One of the things I like to do with mine is after giving them a good testing slap them in a cheap external case and make portadrives out of them. External drive enclosures can be had for around $5 and paired with something you may not have heard of called an Nbox or Nbox HD which paired with those free drives and cheap enclosures make kick ass media tanks for your less skilled friends or relatives. Like for example I loaded one up for dad with his favorite genres of movies, or if you have a relative with kids give them one loaded with kid shows and watch how you are looked upon as a god among men.

In the end as long as your data is backed up spending hundreds on drives when you have a "free" source is frankly stupid. Better to find good uses in them like a home made NAS, media tanks, portadrives, just use your imagination.

Re:Not worth it.

[unitron]

"...why waste 500GB - 750GB drives, many that are NIB....?"

If they're IDE/PATA, find people with Series 1 and 2 TiVos that need more storage space.

 

Re:Not worth it.

[unitron]

Unfortunately "Advanced Format Disease" has spread to 1TB, 1.5TB, and 2Tb drives as well.

And the manufacturers don't make it easy to find out which models are or are not when you're buying retail boxed drives which have different model numbers from the model number of the actual drive inside the box, not to mention that there's no guarantee that the same model number box has the same model drive inside this month that was in boxes with that model number last month.

the 2 main choices:

[gbjbaanb]

FreeNAS or OpenFiler.

I think FreeNAS (the BSD based one) is lighter and easier, as OpenFiler seems to be going in a more "fully featured" direction with less support for older hardware, but they're both good.

Re:the 2 main choices:

[cptdondo]

You can stripe the smaller drives to create a larger one that equals the capacity of a large drive. Then RAID-1 the larger drive and the collection of striped drives.

Me, I'd use the smaller ones for target practice and just get a second 3TB drive.

Re:the 2 main choices:

[gbjbaanb]

yes it does - it uses ZFS that has some fancy replication features, especially z-pools that are like software raid. You can have a 100GB vdev on both the 100GB and 3TB drive as a mirror. Of course if you have just those 2 drives, nothing is ever going to get you full data redundancy (obviously!) but ZFS gives you a lot of flexibility to use what you do have.

Re:the 2 main choices:

[cpu6502]

Can these programs mirror the contents of one USB: drive to the other USB: drive? It's a pain trying to copy-and-paste files in drive 1 to the backup drive 2. THX. :-)

Re:the 2 main choices:

[fuzzyfuzzyfungus]

It wouldn't necessarily be pretty; but I think that, with LVM, you could implement the ugly hack of carving all disks into partitions the same size as the smallest disk, and then creating your volumes on top of those, with redundancy between the chunks.

Having to do reliability calculations when physical disks take out a single logical chunk and others might take out 100 or more, though, would be pretty gross...

Re:the 2 main choices:

[damien_kane]

Can these programs mirror the contents of one USB: drive to the other USB: drive? It's a pain trying to copy-and-paste files in drive 1 to the backup drive 2. THX. :-)

These are BSD/*nix based systems

mkdir /bigdrive_mountpoint/smalldrive_identifier
cp -advf /smalldrive_mountpoint/* /bigdrive_mountpoint/smalldrive_identifier

Re:the 2 main choices:

[McKing]

ZFS does this much more simply with no ugly hacks. You can have mismatched drives when you build a mirror (the mirror is the size of the smallest drive in the mirror set), and then you stripe across the mirrors. As the older, smaller drives fail, replace them with newer, bigger drives and the pool magically gets bigger. 100GB + 500GB mirrored (100GB usable). 100GB dies, swap in a 750GB drive and now this pool is automatically resized to 500GB. Get 2 more drives? Mirror them and add them to the pool and your pool expands with no one the wiser.

Seriously, if you haven't played with ZFS before, download FreeNAS and give it a whirl. When I was a Solaris admin, ZFS was the most fun thing to work with by far.

Re:the 2 main choices:

[aarongadberry]

I don't see the ability to dynamically expand FreeNAS. (Just add a drive and expand the protected space)

I cautiously recommend unRaid. I have not had an ideal experience with it, but most of it was due to my lack of diligence in ordering compatible hardware and fully reading all 10,000 forum threads before logging in. Mainly the hardware thing.

Re:the 2 main choices:

[ratboy666]

FreeNAS can use ZFS as the filesystem. And this is what you want! Now, the actual configuration depends on the drives you have available.

For drives with the same, or very similar capacity -- ZRAID can be used. With 3 drives, ZRAID1, or with more, use ZRAID2 (the number is the number of drives which can be failed). ZRAID offers the capacity of the smallest drive, which may waste space. If all drives are (eventually) increased in size, more storage is added.

For drives with different capacity, ZFS offers the ability to keep a redundent number of copies of the data (eg. specify two copies, or three). Then, ZFS will duplicate the data onto multiple drives.

As well, ZFS continually monitors the drives and redistributes any failed areas, and ensures that no bit errors accumlate in the file system. ZRAID and multiple copies can be combined.

The main point of ZFS is to keep your data clean and safe from corruption.

As well, "fsck" is not needed -- it happens when you "scrub", which slows down the array, but doesn't leave it unusable.

If you have sufficient memory, ZFS can also "dedup" the blocks in your filesystem, merging identical copies of data (but copying/raiding to maintain data integrity). This feature takes a LOT of RAM (2GB per TB of disk, 32GB for 20TB of disk, and possibly more). Also, some ZFS versions offer encryption (not sure about the one in FreeNAS).

ZFS drives can be physically moved to another system, and used (eg. FreeNAS x86 to SPARC). Endian and format issues are correctly handled. Not a feature most people would ever use, but it is nice. ZFS is available on Solaris, BSD, Linux, Mac (well, used to be).

Also, ZFS support snapshots, which can be browsed.

Finally, ZFS has an eight year history in production.

In all, what's not to love?

Re:the 2 main choices:

[jvin248]

Openmediavault is a great choice. I ran with freenas (bsd) for several years but if you have a problem, there are not many ways to recover your data with a liveCD for *bsd. Drives might be fine but what happens when the cat knocks over your server and magic smoke comes out of it? For Debian-based Openmediavault I have all kinds of livecd's and workstations to access and recover from issues. OMV has been solid and plays nice on the LAN.

Re:the 2 main choices:

[cayenne8]

Just been looking at the FreeNAS site, and watching the videos...VERY interesting stuff.

I'm trying to figure out the way to best use hardware to maximize using a bunch of older disks....most in the 1TB range.

I'd get some kind of box...like core2 duo maybe...how would I best hook up the maximum number of hardrives to it?

If a drive goes out....would you just have to shut it down...take out bad drive, plug in new one...turn it on...and FreeNAS would rebuild it? (assuming using ZFS)

Would it work with hooking to it something like what I have sitting around unused...a Sans Digital Towerraid TR5M+(B)? Could I load it up with drives...maybe treat it as a JBOD..and let FreeBSD manage the raid on it...etc?

Re:the 2 main choices:

[gbjbaanb]

it uses ZFS, go read up on it, it is "teh win" in filesystems.

try this post for a quick summary: http://slashdot.org/comments.pl?sid=2827537&cid=39883221

Re:the 2 main choices:

[GrandCow]

I don't see the ability to dynamically expand FreeNAS. (Just add a drive and expand the protected space)

Adding a drive to FreeNAS is incredibly simple. Especially if you are using ZFS (which you should be). It's a single command in the CLI to add a drive to the pool and get it working with the rest.

Re:the 2 main choices:

[epine]

In all, what's not to love?

ZFS does nothing to protect integrity in memory, and especially in the dedup case, your data sits in memory a long time.

I wouldn't run dedup on a non-ECC mainboard. Had an experimental ZFS that suffered a failed memory stick (this array not run in dedup mode after a performance test following the initial build). The next scrub found inconsistencies on disk. Even after copying all the data to a new location on the same storage tank and deleting the old location, there were internal consistency errors. This didn't surprise me, but illustrates that memory-induced corruption will often kill the entire array. Keep plenty of offline backups.

Now if you just happen to have a fresh Opteron 3250 lying around on a mainboard populated with the right type of memory, with full server chip validation and background memory scrubbing, fill your boots with those old 30MB IDE drives.

I'm running my test array on three 500GB drives. Two are enterprise grade and the third was a Seagate warranty replacement (consumer grade refurb). I could have run with the consumer drive as an idle hot spare, but decided to run a three-way mirror, which keeps your hot spare silvered at all times. Note that the consumer drive limits my peak write bandwidth, as the enterprise drives have higher read/write performance. The reads seem to be distributed so that the hot silvered consumer drive works out to a net performance gain.

My scrub on 50GB of data takes just under 15m. Concurrent read traffic is not greatly impacted at home network levels.

Re:the 2 main choices:

[tzanger]

It's not much of a fileserver if it doesn't weigh more than 50 pounds. If a cat were to tangle with a real fileserver, I'd be more worried for the cat. :-)

Re:the 2 main choices:

[bigrockpeltr]

I would also suggest NexentaStor Community edition, It is a hybrid OS using a Solaris 11 kernel and linux userland i beleive so it gets the latest ZFS versions which arent available in *BSD yet. I switched to this about 15 months ago after a couple months on FreeNAS7 mainly for the newer ZFS version. At that time FN7 didnt have snapshots or dedup although FN8 has snapshots now.
I realised dedup wasnt useful for most of the stuff i was storing and was probably affecting performance so I ended up creating a couple pools one for my media (music, video, pictures, graphic design/ photography workflow spaces, some documents) and another for development stuff like source code. this also made it easier to set different snapshot schedules for the entire drives.
Im also taking a look at btrfs as an alternative to ZFS since ideally i would prefer a debian/ubuntu based OS

Re:the 2 main choices:

[jpvlsmv]

Name one filesystem that does protect data integrity in memory.

If bytes can't move from the CPU's ALUs to the spinning rust without damage, there is no way to safely store your data.

--Joe

glusterFS

[Anonymous Coward]

maybe put them in machine, make a glusterFS ?

Stream is the word.

[InvisibleClergy]

Not streak to iPad. Stream. Streaking to iPad would require cleaning supplies at the point of impact.

Re:Stream is the word.

[dbialac]

Not streak to iPad. Stream. Streaking to iPad would require cleaning supplies at the point of impact.

Or full-frontal nudity.

Re:Stream is the word.

[EmagGeek]

You say Stream is the word? I guess that is only because you haven't heard. You know, I was under the impression that everyone had heard.

Re:Stream is the word.

[hendridm]

That would be pretty funny. Ima try it.

Re:Stream is the word.

[Lunix Nutcase]

Yeah that would be funny... if you were 5.

Re:Stream is the word.

[Neil Boekend]

Since the harddisks are doing the streaking I had an image of a harddisk with no hardware encryption. It is streaking to the Roku because it is showing it's bare data to the Roku.

FreeNAS or Unraid.

[detritus.]

Look at FreeNAS or Unraid. Unraid has a 3-drive limit IIRC for the free version, but supports an unlimited amount of drives for the non-free version.

Re:FreeNAS or Unraid.

[jtownatpunk.net]

unRAID does not support unlimited drives in any version. It comes in 3 (free), 6, and 21 drive versions.

I've been using it for a year or two and, while it's got some limitations, it's a good choice for this application. Mostly because the guy's using a random collection of old drives and is likely to have bad sectors across multiple drives at some point. There is no striping with unRAID so the worst thing that can happen is he'll have to mount the drives individually and copy the data to a new array.

Re:FreeNAS or Unraid.

[detritus.]

Thanks for the clarification -- 21 drives ought to be enough for anybody. :)

Re:FreeNAS or Unraid.

[BLKMGK]

I've been using unRAID since the beginning, unless this is new I do not believe you can have redundant data on multiple drives. You CAN spread a single folder across multiple drives and the parity drive protects against crashes and enables recovery. Since parity isn't striped drives spin down when not needed too.

So far I've not lost any data and I've had multiple drives fail over the years - never a dual failure however which unRAID cannot protect against. However if a dual failure occurs you don't lose it all either and I don't need tons of hot spares and crap spinning.

unRAID is perfect for this application IMO...

waste of power

[Anonymous Coward]

What you're suggesting is a colossal waste of power. Just buy a new drive and junk your ancient old drives.

Re:waste of power

[Lunix Nutcase]

And tying an onion to your belt used to be the style at the time.

Windows 8 Storage Spaces

[aaron44126]

If you use Windows, the forthcoming Windows 8 "Storage Spaces" feature appears to be perfect for situations like this. http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing-storage-for-scale-resiliency-and-efficiency.aspx

FreeBSD and ZFS

[Anonymous Coward]

FreeBSD has fast ZFS support which is wonderful file system to fight data loss.

Re:FreeBSD and ZFS

[MightyYar]

Agreed. Do this for fun, not for anything practical - I mean, there are USB thumbdrives larger than your 100GB drive!

Pair the drives up to match them as closely as is possible so that you have 5 redundant mirrors. More realistically, you'll only have space or sata hookups for 4 pairs.

Anyway, use FreeBSD and zfs to pair them up and then combine the 4 or 5 pairs into a single pool. As the drives die or as you acquire bigger drives, you can hook up an additional drive and use the "zpool replace" command to swap out the smaller/dead drives.

Two caveats:
1. The size of each mirror is limited to the smallest drive. This might waste space if your drives don't match up well.
2. You can grow the pool by as many mirrors as you want, but you can never shrink the pool! Or, rather, the support for shrinking pools isn't there yet - maybe never is a strong word.

ZFS is fun to play with, and so is FreeBSD as I have discovered.

You can , but probably without RAID

[sl4shd0rk]

If you just use LVM and group all your disks together into one PV, that would make the array appear as "one big drive" to the system.

Redundancy (RAID) would not work so well because your array would be limited by the smallest disk in the array. Sure, raid the 300GB to the 1TB, but you end up with a RAID-1 array of 300G.

Re:You can , but probably without RAID

[Anonymous Coward]

That sounds awesome. Should have a MTBF of about 20 minutes

Re:You can , but probably without RAID

[julesh]

Who cares, as long you've configured enough spares?

The mega surplus continues!

[digitalsushi]

Ah ha! Who else amongst you has a huge surplus of huge hard drives going unused, now that netflix streaming has displaced 60% of all the crud you had spinning idle in a closet the 3 years before you signed up?

My storage requirements went from about 3 terabytes to about 30 gigabytes over the past 2 years. I believe I am the archetype and that I am doing the same thing as the average geek. I suspect there are piles of huge disks sitting offline because of this streaming displacement.

It cost me about 18 dollars a month to leave my x86 file server online, idle (killawatt meter, nh rates); netflix is cheaper than that.

Come on, who else has a comment related to this.

Re:The mega surplus continues!

[Anonymous Coward]

The OP is looking to build a giant porn vault. All of the other words in his post are just cover.

Notice how he talks about "visibility" and "streaking". He's got the porn on his mind.

Netflix is very light on the porn, so it is N/A here

Come on guys, we need a modern porn vault solution here, iPorn, Porndroid, Porno on Rails, something big

Re:The mega surplus continues!

[Xtifr]

Not me. Before Netflix streaming, I got most of my movies via...Netflix! In fact, I kind of curse Netflx streaming, because I find I'm wasting a lot more time watching movies and shows than I used to, and less time reading and working on my hobbies.

Re:The mega surplus continues!

[cpu6502]

>>>Who else amongst you has a huge surplus of huge hard drives going unused

No.

>>>My storage requirements went to about 30 gigabytes

WOW. I still download a ton of stuff via Utorrent, and I need the space since I acquire movies/shows faster then I watch them. I also need to space to "seed" back the stuff I've taken. My 1 TB drive is quite full.

I don't subscribe to Comcast or Netflix or anything else. It's just entertainment... not really worth paying for it, when I can acquire it for free (hulu, AntennaTV, etc) or dirt cheap ($1/month for F&SF magazine).

Re:The mega surplus continues!

[Hatta]

There's no netflix for classic video games. Until I can hop on the cloud and download an ISO of whatever PC Engine game I happen to want to play today, I'm going to have to keep the TOSEC on my hard disk.

Re:The mega surplus continues!

[pavon]

Not me. My terabytes of data were being used for PVR recordings, and Netflix doesn't have enough current content to replace that function. Hulu was getting close, but not quite because of the random restrictions on what can be watch on a TV set vs in the browser, and the numerous shows which would expire from the queue before you could watch them. With Hulu's proposed cable subscription requirement, it looks like my PVR will be getting even more use in the future, not less.

Re:The mega surplus continues!

[rgbrenner]

Interesting.. I have the exact opposite experience. Lots of extra drives from constant upgrades. I have around 7-8TB (including 3TB of backups).

The file server gets maxed out.. so I upgrade the drives with new HDs.. take the old ones and put them in the backup server (so it has enough space for the new data).. and then remove the oldest drives from the backup.

The "old" drives get placed in any new computer that gets built.

Recently, I've had 2x250GB drives fail... both with about the same amount of time in service (one with bad sectors.. the other constant clicking). both about 6.5 years old.

So I would say anything 250gb or less (and probably even 500gb or less) is probably too old to be used for anything.

Re:The mega surplus continues!

[PCM2]

Nah, it's the kind where you can get it up in a real hurry but it has serious performance problems.

Re:The mega surplus continues!

[hetfield]

You are definitely the archetype... of people who really trust The Cloud (tm). I do not.

1) I used to have Netflix. Then they jacked up their price and lost something like 60% of their already mediocre streaming selection. Their boneheaded CEO is still there. I have not seen a press release from Netflix that has convinced me it's time to go back.

2) All of the Internet providers in my area are media companies that want to sell you TV service and have basically announced they don't believe in net neutrality. My service comes with a yet unenforced 250GB cap per month. If I were to go back to Netflix or a similar service, the majority of my movie viewing will be in high definition. I'm too lazy to do the math, but my streaming, my g/f's streaming, and our other Internet use (like Skype) would probably chew up our allotment pretty quickly. My future children will definitely make this a lot worse. Maybe caps will be raised by that point...

3) I like to own my movies and music. I'm such a luddite that I actually prefer having a physical copy that I can rip to disk, back up, and cuddle as I fall asleep.

I'm actually looking to put together a multi-terabyte FreeNAS server to serve as a backend to an XBMC console in my living room as well as convert my other servers to Xen virtual machines. Based on how popular home theater PC hardware is right now, along with the popularity of software like Windows Media Center, MythTV, and XBMC, I'm pretty sure I'm not alone in this.

Re:The mega surplus continues!

[LanMan04]

Ah ha! Who else amongst you has a huge surplus of huge hard drives going unused, now that netflix streaming has displaced 60% of all the crud you had spinning idle in a closet the 3 years before you signed up?

Three reasons to keep all that on a disk:
(a) Bandwidth. My kids like to watch movies over and over and over. Download it once and you can watch is multiple times without eating inbound bandwidth.

(b) Permanence. Netflix media contracts come and go (e.g. Starz). Hell, Netflix may disappear one day. I want to keep the movies and shows forever.

(c) That extra 40% that Netflix doesn't have.

Re:The mega surplus continues!

[unitron]

In my day all we had were those self-developing photographs someone took with their Pornaroid camera.

FreeNAS, for sure

[fmachado]

FreeNAS can use ZFS for aggregating multiples drives, independent of size, technology etc, all with varying degrees of protection.

It's by far the best solution to your case.

Flavio

Do you care about your electricity bill?

[jimicus]

Do you care about your electricity bill at all? If you do, it'll probably be cheaper over the course of 6-12 months to buy a simple NAS box or a cheap atom board and plug in a couple of 2TB hard drives.

You meant steak, not streak

[Anonymous Coward]

However even if you replace the word "streak" with "steak," the sentence still makes no sense.

WHS V1

[clickclickdrone]

Windows Home Server (V1) - mix and match to your hearts content and all the addins you can eat for adding features.

Re:WHS V1

[dodgerfan]

I second this idea. Very simple to setup. WHS v1 will adopt every drive that you can attach.

Don't do it.

[Jawnn]

If you have pairs of drives with reasonably similar size and performance specs, you could deploy them in RAID 1, or RAID 5 if you have three or more similar drives, and have some redundancy. FreeNAS, OpenFiler, or Nexenta will all work, but you're still rolling the dice in a rigged game, man. Old hard drives are for target practice.

Storage Spaces

[Sensi]

http://www.winsupersite.com/article/windows8/windows-8-feature-focus-storage-spaces-142537

Re:How about throwing it all out?

[cayenne8]

What are you running at home? A particle accelerator? Who needs all this storage? For what?

Well, lots of things.

Media can suck up a lot of drive space...even if it is all legal!! You might want to rip all your CDs to various formats (flac for good stereo in the living room, mp3s for ipod or car).

Ripping your dvds/blurays...to watch conveniently. Then with all this, you might like a few backups so you don't lose all that ripping work too easily.

I'm about to buy a new high end DSLR....storing pictures....HD video for production...archiving the raw video parts as well as the finished products. Still photos....originals...plus processed ones....redundancy copies for these can eat up a lot of space.

There's a ton of stuff you might want that will eat up space in today's digital world....

Ditch your small drives

[Anonymous Coward]

Ditch your small drives. A 3TB is equal to 30 100GB drives, so there is absolutely no point in keeping them.

Group your drives into similar sizes and assign into a few different raid arrays. The array will only use the capacity of the smallest drive connected.

I would use RAID 1 if you only have 2 drives in the array. Raid 5 for 4 drives or more.

Hard drives are very reasonable priced these days. It will be much more cost effective to purchase new sata drives then attempt to use any IDEs at all.

Kevin

Greyhole!

[gregthebunny]

Why am I the only one saying this? Setup Greyhole, throw a bunch of disks at it, and enjoy! And to all those saying "those drives are going to die soon", you can actually tell Greyhole that you consider a drive "broken" and it will still use most of its storage (albeit redundantly) until it does die and have to be removed.

Re:Greyhole!

[Dan667]

with raid 5 you lose a percentage of the disks you use (like ~25% if you use 4 disks). Maybe I am missing something, but Greyhole looks like you lose significantly more than that making multiple copies of files on different disks?

Re:Greyhole!

[Hatta]

That's what I was going to suggest, if I could have remembered the name. Greyhole. heh

Re:Greyhole!

[ZeroSumHappiness]

The two key points I see with Greyhole is that it works with differing drive sizes and you can, for each folder or file (didn't quite care to get the exact configuration setting) set what redundancy you want.

So yes, you'll lose more space than if you use RAID 5 or 6 but it looks really easy to set up. But it looks slightly more likely to catastrophically fail than RAID 1 in the event that a drive fails before Greyhole duplicates the new files on it.

Re:Greyhole!

[Anonymous Coward]

I also came here to say about Greyhole. Looks like it's exactly the solution the OP needs.

Also check Amahi, http://www.amahi.org/, which is a Home Server appliance which uses Greyhole to do HDD pooling. I haven't used it yet, but looks very nice.

Re:Greyhole!

[EdBear69]

with raid 5 you lose a percentage of the disks you use (like ~25% if you use 4 disks).

You're right about the amount for 4 disks, but the general answer to how much you lose in RAID5 is "one disk". The percentages change with the number of disks involved in the array. So in a 3 drive array (the minimum for RAID5), you lose one disk; 1/3 = 33%. For a 10 drive array, you still lose 1 disk, but 1/10 is only 10%, so the efficiency is better the more drives you use.

Re:Greyhole!

[JobyOne]

I don't know that it *is* more likely to catastrophically fail. Just make the landing zone on one of the more reliable drives, or better yet mirror the landing zone with RAID or ZFS. That way your landing zone is redundant, and the only way you could realistically catastrophically lose a fresh file is by having exactly the wrong two drives fail within at most a few minutes of each other.

I've been personally working on switching my own file server over to greyhole (from ZFS) this week. I'm doing it because instead of defining redundancy on a per-device basis, it lets me set redundancy on a per-directory basis. In the end that will let me make a more thorough use of my set of drives and their mismatched sizes. I have a relatively small amount of files that are actually very important and would be genuine problems if lost.

With greyhole I can mirror those truly important files across multiple drives for redundancy (and even send them off site for super safety), while all the MP3s don't need the same kind of redundancy. If I lose them I can just download them from Google/Amazon again, or rip them from CDs again.

I think the main draw of greyhole is that flexibility in how the redundancy is handled. It lets you make the most efficient use of your drive space, as long as you have a similar situation, with files that have drastically different redundancy needs.

Another thing I enjoy about greyhole is that its failures won't be as catastrophic as RAID or ZFS. Since it's dealing well above the file system all your files are still just files. Even with zero redundancy if a drive fails the entire pool doesn't drop dead, you only lose whatever files happened to wind up in that particular spot, and all the others are still safe.

Don't Build.... Buy a Drobo

[InitZero]

1. Throw away everything that isn't a standard-sized SATA drive.
2. Buy a Drobo (http://www.drobo.com/products/professionals/drobo-fs/index.php).
3. Put the five (or eight) largest drives in the Drobo.
4. Throw away the rest of the drives.
5. When you get a drive that is larger than the smallest drive in your Drobo, pull the smaller drive out and insert the larger drive.
6. Find peace in the universe.

When I was young and foolish, I tried to keep every drive spinning, even long after its time had passed. I had *nix boxes stuffed with drives and SCSI-attached arrays. I learned a lot about drive management and system administration but, mostly, I learned that there is a value to my time and my time isn't best utilized playing disk administrator.

Drobo doesn't pay me a dime and I am still more excited about Drobo than any technology product since TiVo.

Cheers,
Matt

Re:Don't Build.... Buy a Drobo

[Anonymous Coward]

I'm a systems and network admin, and I just built up one of those HP N40L's that were $130 from a couple of weeks ago.

4 x 1TB drives, for a RAID 0 stripe. Booting from a 10,000 RPM raptor drive.

I am running OpenMediaVault, an offshoot of FreeNAS. Love it. Works as CIFS, NFS, iSCSI and plays nicely with VMware ESXi, Winders, OS X and Android clients. Quiet, low power use, fast. And 1/3 the cost of the closest network-attached Drobo.

Re:Don't Build.... Buy a Drobo

[zeet]

4 x 1TB drives, for a RAID 0 stripe.

How do you handle backing up the 4TB of data?

Re:Don't Build.... Buy a Drobo

[rdunnell]

4 x 1TB drives, for a RAID 0 stripe.

How do you handle backing up the 4TB of data?

You have the same backup problem with a mishmash of drives that you cobble together on your own...

Re:Don't Build.... Buy a Drobo

[countach]

I have to agree 100%. I bought a Drobo several years back, and its been extremely reliable. When a disk dies, it alerts you, and you slot in another one. It just works as advertised, and my life has just been a ton easier since then. Before that I was using various disks and raids and all sorts of things, but they're a pain in the butt when you run out of space or a disk dies. Get a Drobo and be done with it.

As for backing up the Drobo, unfortunately you pretty much have to get another Drobo. I mean, in theory maybe you can get away without it, because the Drobo has redundancy against disk failures, but I don't fully trust that. So for full peace of mind, you need 2 Drobos. But I don't think you'll be sorry.

Re:Don't Build.... Buy a Drobo

[Solandri]

I haven't owned a Drobo so I can't comment on the quality or functionality. But QNAP and Synology are generally considered the leaders in the NAS market. SmallNetBuilder has pretty thorough coverage and benchmarks of your NAS options.

If you don't need a NAS, just some form of aggregate storage, non-networked alternatives are made by Mediasonic and Sans Digital. In my case I just needed something to throw my old drives in and power it on every couple weeks to backup my ZFS file server. So one of these connected via USB 3.0 or eSATA worked just fine.

Re:Don't Build.... Buy a Drobo

[mckwant]

Seconded, down to the TiVo comment. I'd shake hands, but I'm worried about a quantum singularity.

However, like the TiVo in 2000, it's a tough check to write. I have to admit I was edgy about getting my FS. It's tempting to think that a FreeNAS (or whatever) would be fine. For comparison, though, here's the short version of the FS install procedure:

- Unpack Drobo.
- Plug Drobo into power and network.
- Insert drives.
- Wait a bit while Drobo configures itself.
- Log into Drobo, create the shares you want.
- Hook the shares up on your desktop.

Ball game. And it doesn't break. It's in my un-A/C'd garage in Austin, and if it didn't complain last summer, it's fine.

Mild qualm about available space. To be fair, the set of drives in there is sub-optimal, and I went with double redundancy (which even I think is overkill). I have 7.5TB of drive supplying 3.08TB to the network. I don't really care, as I'm only using 2.13T, and the drives are either leftovers or birthday gifts, but the geek in me shudders just a hair. For the record, single redundancy would be something like 5T usable.

Oh, and, if you're an Apple guy, it'll work as a Time Machine box, too.

Easily one of the better $500 I've ever spent.

Re:Don't Build.... Buy a Drobo

[cbope]

I'll second QNAP. I've had a 4-drive version (TS-419P) set up for a bit more than a year now, with 2.68TB of RAID-5 storage with one hot spare (four 1.5TB drives). It goes to sleep when there is no disk access and spins the drives down, so the average idle power draw is very low. Spinning the drives up takes only about 10 sec or so when there is a disk access. There are frequent firmware updates and it has been flawless so far. It's also very quiet, I have it sitting on my desk behind my monitor and it's not intrusive.

Re:Don't Build.... Buy a Drobo

[Hast]

I own a Drobo FS and would recommend that you get something else. (I've read good things about Synology, but haven't tried it myself nor do I personally know anyone who has tried it.)

It's pretty slow (I rarely get over 20MB/s read or write over gigabit ethernet) and the way they handle "applications" is pretty scetchy and seems a bit problematic. A few times applications have stopped responding and I've had to reboot the Drobo.

You also need a Windows or OSX machine to get started. At least when I got it it was impossible to do the first configuration without the special software. After that you can do most of the work (but not all) over the web interface (which runs as one of the previously mentioned rather scetchy applications).

It has noticed and warned me when a drive failed though. So that was nice. I'm not entirely sure that the "self healing" did the work properly though as some videos have seemed a bit wonky afterwards. (The entire point of self healing is to avoid bit rot.) Really I'd rather use a good established system for that (like ZFS or perhaps Btrfs) than something "home grown" where they keep the details secret.

Re:Don't Build.... Buy a Drobo

[JaimeZX]

Love my Drobo! Filled it with 2TB Seagates I found on sale for $69/ea right after the floods.

Re:Expensive power consumption

[jones_supa]

Maybe the power consumption problem could somehow be worked out by starting and stopping disks based on idle timeouts? I don't know how well that kind of setup would play with a RAID configuration, but perhaps there's some other method too.

I still kinda like the concept of keeping old hardware running for ecological reasons (making new stuff takes a lot of power and resources). And it would be interesting to find some kind of solution for this case even though it's gonna be somewhat hacky. My two cents is to consider putting all the IDE disks behind IDE->SATA converters to make the whole setup SATA-only, to streamline it a bit.

Use WHS v1 (2003)

[Snuusnuu]

Windows Home Server v1 (2003). Drive extender will lump all sizes of hard drives into a JBOD. Turn on folder duplication for redundancy. Full integration with Windows 7. Use Plex media streaming to stream to an iPad. That's my setup and works great.

Re:Windows Server 2012

[McKing]

2006 called, and it's pissed that MS stole all those features from ZFS....

Another vote for unRAID

[sirwired]

I've been using unRAID for years and it's a great solution for a small home NAS box. If you ever change your mind about using it, you simply turn your parity drive into a regular Linux boot disk, and the remaining drives are just regular Reiserfs2 filesystems. Most RAID systems and/or software would require much gymnastics to de-RAID them, if it could be done at all.

In addition, hardware-based striped RAID makes you dependent on the RAID controller; if it dies and you can't find a replacement compatible with the original's striping mechanism, your data just disappeared.

Re:Another vote for unRAID

[RCSInfo]

+1 For unRAID. Works perfectly with a mix of IDE and SATA (USB will mount and share as individual drives but will not be included in array). Mixing of drive sizes is OK, total space will be sum of all drives minus the biggest drive. Plenty of community plug-ins including various streaming apps. Also it powers down idle drives after a configurable timeout and only spins up drives as they are needed.

Re:Another vote for unRAID

[letsief]

I've been an unRAID user for a couple years now and I'm reluctant to to strongly recommend it. Lime Technology, the small company behind unRAID, seems to be a one-man show. And that one man seems to disappear for weeks or months at a time. If customer service and technical support are important to you, and you desire timely updates for new features and bug fixes, then unRAID might not be for you. I've largely lost confidence in the unRAID developer. unRAID v5 has been in beta for almost 2 years. At first it seemed like the developer was struggling to fix some compatibility problems that plagued recent releases, but more recent messages indicated that he wasn't paying attention to all the people complaining about those compatibility problems. Now that's he's finally listening to the bug reports things are looking up. He released V5 RC1 about a week ago, but pulled it down after a day due to the bug reports he received. But, a couple days later he posted a new version that seemed to clear up the major compatibility issues people were having for the last 6 months. That was great, although part of me is even more upset now that I know he could have easily fixed the bugs introduced in the betas 6 months ago if he had just listened to the beta testers.

Anyways, unRAID's features are a pretty good fit for the OP, but as an overall product it might not be great for his needs if he wants good support and updates.

Why?

[DarwinSurvivor]

Why are you combining 100GB and 3TB drives? First of all, the 3TB drive is litterally 30 times the size (giving you a space increase of 3%). Second of all, the 100GB is probably fairly old, so shouldn't even be trusted as stable. You are going to spend more on the ATA adapter for that drive than the value of the space it provides. Currently a 3TB drive costs about $100, that's $0.03/GB which means that 100GB drive is worth ... wait for it ... $3. Sata to IDE adapters run about $9 a piece.

I've been in the same situation, it was only a year ago that I was running on multiple 10GB drives and an old 120GB laptop drive because I only had IDE in my server. So I went to newegg and got a low powered an E350-onboard-cpu motherboard (doesn't even need a fan) for $130, 8GB of ram (I use ZFS) for $50 and a 2TB drive for $70 (drives have gone up since then, but not terribly high) and threw the thing into an old case with a cheap power supply. That's basically an entire system with about 15 times the storage space as my old one for $250 shipped to my front door and the system can take 5 more drives without so much as an expansion card.

the "d" is redundant

[Chirs]

Once you specify "a", the "d" is redundant.

ZFS

[Anonymous Coward]

Don't use FreeNAS. Normally I would recommend it because it's very straightforward and does a good job of what it does, but it's not ideal for your exact situation. It has a fairly old version of the ZFS file system. You'll want a new enough version to use a RAIDZ configuration. This way you can put all the drives in one pool, and it will automatically juggle redundancy and parity to allow a single drive to fail without losing data. That has the end result of presenting some fraction of your total storage as one single mount point.

This will maximize the amount of usable storage you get while keeping things redundant and failure-resistant.

FreeNAS only supports simple striping or mirroring. With striping you'll lose the whole pool if one of your old drives fails, so it's right out. Mirroring works best with matched pairs of drives with the same capacity, and you'll be wasting space to drive pairs of mismatched sizes. Also, even in the best case scenario mirroring leaves you with only half the usable space relative to the total capacity of all your drives.

Some things to keep in mind:
- RAIDZ doesn't allow shrinking pools (I think). So every time one of your old drives dies you need to immediately replace it with *something.* That could get expensive if they're old and you have a lot of them.
- You could probably replace the capacity of all these old drives with $300 of new drives. Is it worth it? Especially knowing that you'll have to replace them with modern drives as they fail anyway, and will likely soon enough end up with a crazy glut of drive space, and an empty bank account from buying all the replacement drives.
- There's a PPA for ZFS on FUSE available for Ubuntu Server, that would probably make setting something like this up pretty trivial.

Synology

[spire3661]

Seriously, buy a Synology NAS, dump all your data on it and call it a day. The cheapest 2 bay model they make is straight up badass for its price point.

Re:Synology

[PhilHibbs]

Will it hot-swap and reconfigure itself like Drobo? Can you just replace a small drive with a larger one and get the space increase? (I know you won't get 100% - I think the formula is "total of all disk sizes minus the largest disk")

Re:Synology

[spire3661]

I dont care about hot swapping. Its a 'whiz-bang' feature that has no REAL use in home. I dont even RAID the drives, i just keep another Synology NAS at my in-laws and sync them once a month. Drobo is also quite a bit more expensive for their cheapest model and I dont think their software suite is as robust. Synology's DSM OS is fucking killer. Does so much more then store files. DHCP, LDAP, torrent, terminal, Print serve, surveillance recording, anti-virus, etc etc. Drobo is nice, dont get me wrong, but the hot-swap is generally sold to dummies who cant figure out how to power cycle the device and make sure the drive is operating normally when it comes back up.

Re:Synology

[PhilHibbs]

Thanks, that makes sense to me.

StableBit DrivePool + WHS 2011

[alexpi]

Full disclosure: I am the developer

Check out: http://stablebit.com/DrivePool

It's a software disk pooling solution that combines any number of disks of any size into one big virtual pool. You can designate certain folders to be duplicated on the pool. Any files placed in duplicated folders will be stored on 2 disks at the same time.

The implementation is a hard core NT kernel driver with a virtual disk. There is a full NT kernel storage stack, no user mode hacks here.

Unlike RAID and similar solutions, all your pooled files are stored as standard NTFS files on each individual disk in the pool. This means that you can simply plug in any pooled disk to any system that can read NTFS to get at your files in case disaster strikes.

It's commercial software, $20 USD per server.

JBOD

[Larry_Dillon]

Put all of the small drives in a JBOD array and use the 3TB as an internal backup because RAID is not a backup solution.

Use FreeNAS or OpenFiler.

Drobo performance sucks (with more than one concurrent user).

Low-end core i3 processor and lots of RAM because RAM is cheap these days.

Re:JBOD

[PhilHibbs]

Sure, RAID is not a backup, but a backup device that uses RAID is still a backup device.

take look at amahi.org

[JonySuede]

look at amahi.org, it is a turn-key Home Server based on fedora and greyhole as it's replication engine.
Dump anything less than a TB except one drive and you are set.
You set the replication level by share and it keeps a full copy on each drive until the replication count is reached for that file on that share.

Example:
you have 4 1TB drives and 1 500Gb drive.
You have the share photo configured to replicate on each drive.
You have replication off on the video share.
You have a replication level of two on the mp3 share.

When you store a photo greyhole write it to your 5 drives.
When you store a video it goes on a random drive.
When you store a mp3 it goes to 2 random drives.

So if you lose a drive you should loose about 25% of your videos, 6.25% of your mp3 and 0% of your pictures.

Re:take look at amahi.org

[JonySuede]

fuck I meant lose 2 drives and I rounded up to 25% to compensate for the fact that one drive is smaller than the others so it has less chance to be used than the other

Re:take look at amahi.org

[JonySuede]

note to myself : do not do stats after work

Re:take look at amahi.org

[Gothmolly]

Use proper RAID and not lose any data.

I like NASLite

[mikeiver1]

Though it's not free it is cheap and works well. It supports allot of hardware and is efficient with resources. I have gotten it to run SCSI, ATA, USB, FC-AL, SATA, and SAS drives all at once. A build usually only takes about 5 minutes including a basic configuration. I have been running it for over 7 years now with no real issues to speak of. Mike

Interesting

[Firethorn]

It looks like l'm going to have to read up on this stuff again.

Given the spread of drive ages I'd definitely want redundancy, and given the variety of capacities, a traditional RAID system isn't going to cut it. I'm actually thinking of cloud computing technology, with it's attendent abilities to duplicate data(and services!) across sites of uneven capability, even optimize resources across different locations to optimize resources.

Basically, you'd be looking a 'cloud' of HDs, with an underlying system that's aware of the different drives and a directive to, say, keep all data on at least 3 different drives. How it does that depends on the alogorithms, but it isn't structured so rigidly like traditional RAID.

Just throw them away

[zmooc]

Powering 10 old harddrives for some time is going to be much more expensive than just getting a new one. A modern drive uses about 5W on average. these oldies probably use much more. 10 drives using 10 watts at $0.10 per kwh will set you back $87 per year. You do the math.

Re:How about throwing it all out?

[MightyYar]

I'll add that having a single machine for backups is very convenient. I have a FreeBSD ZFS machine in the basement, and I run CrashPlan on it as well as netatalk so it can pretend to be a Time Capsule. Whenever I fix a friend/relative's computer I make them install CrashPlan on their own computer and point them to my server. Sure it uses some of my drive space up, but it saves me hours (days?) of time when their machine dies.

As you point out, all that digital crap sure adds up - and I have a fair amount of ill-gotten media as well. It's really nice to just tell SabNZBd to just download each episode of your favorite show and not have to worry about cleaning up the space right away.

www.pogoplug.com

[JazzLad]

Pogoplugs are great, can plug in 4 drives via USB or more with a USB hub. I paid $25 for mine, can't really go wrong.

Use BtrFS or Drobo

[digitaltraveller]

Use drobo if you are time poor and money rich, use btrfs if you are time rich and money poor.

Btrfs's capabilities are nothing short of amazing. Here is a vid about it:
http://www.youtube.com/watch?v=9bQc_z-Cb7E

Re:Use BtrFS or Drobo

[UnknownSoldier]

Thanks for the link !

Please mod parent up -- its not to often you get jazzed about a modern fileystem design and implementation !
 

Re:Windows Server 2012

[Bert64]

Well you could wait 2 months for a release candidate of an os that few people will touch before the first service pack...

Or you could use ZFS, which has had those features for years already and is supported on several stable tried and tested platforms.

Re:How about throwing it all out?

[AngryDeuce]

My music collection alone is in the 100's of GBs. It's not at all inconceivable to need this much storage if you're trying to digitize your physical media collection. I'd probably need 20+ TB's to rip everything I own on CD/DVD/BD, I'm just waiting until you can get a good brand for about $20/TB or so.

Nexenta with Napp-it or Solaris Express with Napp-

[grc]

If you don't care about the electricity costs, Illumian, NexentaCore, OpenIndiana or Solaris 11 with Napp-it on top makes a great ZFS file server. If has native CIFS (Windows) file sharing and can share AFS (Apple) and NFS with no problem. ZFS is by far the best file system around, and lets you combine different size drives with no problems. Napp-it turns the whole thing into a web managed appliance.
You can even stream with the minidlna (lightweight), ushare or mediatomb addons! Rock solid and very fast if you give it ample RAM.

USB/eSATA Drive enclosures

[linebackn]

My advice would be to find some inexpensive USB or eSATA drive enclosures for the smaller drives and just use them as off-line storage.

Take some data you don't need instant access to, put it on one drive, and make an identical copy on a second for backup. Put them in a corner and only power them up when needed.

Or just use the smaller drives as partial backup for a larger NAS. Can be handy if you suddenly need to grab a collection of files and go.

Like everyone else is saying, no sense keeping them spinning and eating up power. Might even think twice about the larger drives unless they are power efficient models.

Re:USB/eSATA Drive enclosures

[marcovje]

That's what I use old HDD's for. Offsite backup. In an enclosure and out of the house.

ZFS + FreeBSD

[utkonos]

The answer to your question is ZFS on FreeBSD.

unRAID from Lime Technology

[Anonymous Coward]

Look into unraid from lime technology. Same basic premise as drobo but not a ridiculously priced rip off, predates drobo too, plus it is a linux-based distro so you can mess with it if you want, but it isn't necessary. best feature, besides not carrying about differences in drive sizes, is that the data isn't really in a raid, it just appears that way and if one drive fails, you can use parity to recover your data, and even if you cannot, you only lose the data that was on the failed disk since the underlying file system is just ReiserFS.

Sell them all on eBay...

[funkboy]

...after you wipe them, and buy a real NAS like a ReadyNas, Synology, etc. smallnetbuilder is a great resource for this.
Alternatively, use FreeNAS and build your own, with recent drives.

Salvage... and backup

[bagofbeans]

Create a Truecrypt file filling each old drive, after a full format. Use for full (not incremental backups) every 6 months, starting with the smallest sizes (to use them up). Then put them in your Mum's garage, suitably labelled.

Last tip for backups. Do "dir /on /s > backup_2012_04_23" for each drive after filling it, and keep the list on your main machine, so you can see if you've got a copy of something (and where) before fishing around.

Cloud is the keyword

[DrYak]

There are several distributed file systems which are able to cope with a varied array of drive and speeds, and will try to keep as much redundancy as possible.
On need to read a little bit and pick the best candidate up.

Re:Why?

[Pentium100]

Let's say a drive uses about 10W of power. Running 10 drives uses 100W or, about 73kWh/month, which to me would cost about 9EUR. The cheapest 1TB drive costs ~80EUR, so buying a new drive instead of using 10 old ones would pay off in 9 months. But if I do not have that 80EUR now, it's either wait 9 months and then buy a hard drive or grab an axe, go outside and ask kind strangers for money. If my drives are 200GB instead of 100GB the payoff time is longer (because a 2TB drive is more expensive).

Been doing this for a while

[StarHeart]

I recently had 20 drives across three machines. I was using a combination of raid5, iscsi, mhddfs, and samba. Machine1 mounts the iscsi devices from the other two machines, and then mhddfs combines them into one virtual filesystem. Samba is then used to share files out with laptops.

What I found is that network card drives in 3.2 kernels are currently in a horrible state. They crash left and right under real load. This is after trying different brands, tweaks, version of drivers, etc. In addition it seems iscsi client in Fedora 16 is also not in a great state. Independent of network issues, I would still get failures. The machine running CentOS 6.2 used to be Fedora 16, but was converted to make things more stable.

My latest plan is to do basically the same thing I was doing before, but on a smaller scale. I am going to retire the 1tb drives in Machine3, and replace them with the 2tb drives from Machine1. I am also going to convert Machine3 to CentOS 6.2 for stablility. Then Machine2 will mount Machine3's iscsi device, and use mhddfs and samba. This reduces the number of machines involved from three to two, and takes Fedora 16 out of the mix. It will also reduce the number of drives involved in mass storage from 20 to 14.

I plan to add two 2tb drives to Machine1, for storage, but it end up being only a desktop.

Machine1
Fedora 16
6x2tb raid5

Machine2
CentOS 6.2
5x1.5tb raid5
3tx2tb raid5

Machine3
Fedora 16
6x1tb raid5

Re:Expensive power consumption

[BLKMGK]

unRAID will do this and in fact I often come home and find all my drives spun down as nothing is accessing them. Can put them all in one array too and the only "waste" will be the single parity drive. Been running it for at LEAST 5-6 years or more and have never lost data...

Do the math ...

[garry_g]

- how much power does the PC use when running
- how much does each drive use
- how likely is it that one of the older drives fail
- how much work is it to replace the drive, including the "cost" of the downtime

I would assume that over - say - three years the power usage of a regular PC (especially on that has enough room and power for 10 drives) will easily outweigh the cost for some low-power NAS and something like 3-4 large hard drives, which will most likely run without a glitch for something like 3-5 years ... at least that's the results I came out at, before picking up a 4-bay QNAP NAS that I put 4 1.5TB drives in ... mind you, I do like to tinker with hard and software, but at some points, it's just not worth it ...

Glad to see NAS appliance mentioned

[LostMyBeaver]

People have mentioned electricity and such. They have mentioned the utility of the smaller drives etc...

Let's be frank... any home built PC with the ability to flood a gigabit Ethernet line or two will actually be quite power hungry. These Atom based NAS boxes aren't actually using straight ITX Atom boards, they have precisely what is needed, nothing more, nothing less. They are extremely power efficient as well.

FreeNAS, OpenFiler and several others out there are awesome tools. I have a FreeNAS server myself... though my home file server is running Windows 7 with 8x2TB with a SAS RAID controller, my iSCSI box for VMWare is running FreeNAS. I love it... but to be honest, it is not power efficient. If I spent 6 months to a year focussed on tuning FreeNAS to this specific system, I might be able to get power usage and the function set up to what I could have gotten from QNAP for example for a few hundred bucks.

For my super important file storage, I will very likely this week, after years of FreeNAS simply buy a two drive box with 3TB drives in RAID1 and put it on a different floor of the house. My network in the house is now made up of multiple Cisco Catalyst gigabit switches with four cable etherchannel between each floor. So, a little NAS with a RAID1 that supports etherchannel would be great for backing up parts of my server :)

No... my house doesn't look like a mess... I have very clean cabling and nice looking racks that blend with the furniture :)

old drives = clutter

[food_hacker]

The best thing I ever did with my old drives was to give them to my kids along with a set of screw drivers and a hammer. They a great time and cleaned up everything when they were done.

Re:Why?

[julesh]

Except drives don't use 10W. Looking at one I pulled from a machine recently to put a bigger one in, it's rated maximum consumption is 8W, but in actual use (1) it'll only use that much occasionally -- while it's both spinning its platters up from stationary and seeking at the same time -- and (2) in an application like this it'll likely be in a power saving mode 75% or more of the time. I'd hazard about 1.5W is a much more reasonable figure to use as an estimated average.

DIY? Why...

[chiantii]

Synology products are so cheap these days, and synology has a proprietary SHR (synology hybrid raid) structure that allows you to mix and match any kind of drive you want, from 100 gigs all the way to 3tb as you mentioned with redundancy built in - consider it a modified raid 5 that allows you to mix and match drives from different sizes and vendors.

THE Solution

[infinit.io]

This summer, Infinit (infinit.io) will be released. The program aggregates local storage across all of your devices and allows you to access the data on them at anytime directly through your file management system and eventually mobile devices. In addition, if your friends or colleagues create Infinit networks with their storage space, you can exchange storage space to ensure that your files are replicated on trusted nodes outside of your own network. All files in the Infinit network are encrypted and distributed in bits across the network so only you have access. You can find out when it's available via twitter @infinitdotio or via FB http://www.facebook.com/infinitdotio

A different question, or the big question...

[quippe]

...is why people is still actually saving everything they download? If you have the bandwidth, storing would be only limited to what you really could need fast; and talking in terabyte terms seems unreasonable. If you have the band but capped downloads, well you should only need some kind of temporary storage, sized accordingly to the cap. If you have small bandwidth, that's not your problem.