Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: DIY NAS For a Variety of Legacy Drives?

Posted by timothy on from the thinking-about-a-giant-USB-hub dept.

An anonymous reader writes "I have at least 10 assorted hard drives ranging from 100 GB to 3 TB, including external drives, IDE desktop drives, laptop drives, etc. What's the best way to setup a home NAS to utilize all this 'excess' space? And could it be set up with redundancy built-in so a single drive failure would cause no data loss? I don't need anything fancy. Visibility to networked Windows PCs is great; ability to streak to Roku / iPad / Toshiba etc would be great but not necessary. What's the best way to accomplish this goal?"

29 of 260 comments (clear)

  1. Not worth it. by Anonymous Coward · · Score: 5, Insightful

    Those older drives are probably failures just waiting to happen. With the cost of the hard drive space continually dropping, just use new drives. It's not worth screwing around with old ones for anything other than salvaging old data off them, even though the urge to do so is strong in the more frugal among us.

    1. Re:Not worth it. by ZeroSumHappiness · · Score: 4, Insightful

      I agree to an extent. Take anything SAS or SATA that's 1TB or greater and re-think the project with just those. Sell or recycle the rest of the drives. Depending on your needs the remainder should be RAID-1, 5 or 6'd (using software RAID if speed isn't an issue) and then put on an OpenFiler or FreeNAS box. Anything non-replaceable should then be backed up to a respectable backup provider in addition to your home-grown solution.

      We need more information though -- what are your actual drive sizes and what do you want to put on this NAS?

    2. Re:Not worth it. by PaladinAlpha · · Score: 3, Informative

      This.

      With such a wide range of storage sizes, you're going to have serious trouble setting up any kind of redundant encoding. To mirror a segment of data (or the moral equivalent with RAID-5 or RAID-6) you need segments of the same size; those segments are going to have to be no larger than the smallest drive. That means larger drives have to store multiple segments, but that the segments have to be arranged in a way such that a drive failure on one of the large drives doesn't take the RAID down. If the drives can't be bisected -- that is, divided into two piles of the same total size -- this is impossible, and the fact that your range is from .1TB to 3TB implies this might be the case.

      Think about it -- it's probably going to take most-to-all of those smaller drives to "mirror" the larger drive to make it redundant (and mirroring is the best you can do with just two drives). But having one side of the mirror spread across 9 drives makes failure laughably likely, to the point where you're paying performance penalties for nothing.

      Your alternative is to use a JBOD setup and have just contiguous space across all of the disks. This is the same problem, except when a drive goes you lose some random segment of data. That's acceptable for two or three drives in scratch storage, but you don't want to actually store things on that.

      Make no mistake -- those drives are going to die.

      Trust me on this; don't go down this road. Your actual options are to either pair up the disks as best you can, supplimenting with strategic purchases, and make 2-3 independent raids (and maybe even RAIDing those, but it'll be painful), or just write the whole thing off, put disks in if you have obvious candidates in your hardware, and donate the rest.

    3. Re:Not worth it. by King_TJ · · Score: 5, Informative

      Yep.. I agree. "Not worth it." sums it up nicely.

      Seriously, I completely understand the desire to re-use unused equipment you've got lying around. Seems like the smart thing to do, reclaiming as much of that unused storage space as possible and pooling it together so even the smaller drives add up to something worthwhile. But as a FreeNAS user myself, trust me on this one. It's NOT really a good idea.

      As other already pointed out, most RAID configurations are limited by the size of the smallest drive in the array, so that would create major problems for you right there. But even assuming you skip RAID (or set up multiple RAID pools, each consisting only of very similar sized drives -- and then join all of them into a virtual master storage "device"), you're still in a situation where the lower capacity drives probably have slower data xfer rates than the newer, larger ones. That will drag the overall performance of the server down, whenever something gets loaded or saved to the slower/older disks.

      Even if all of THAT doesn't discourage you? I have to ask what your time is worth, and to a lesser extent, what your data itself is worth? Old drives as small as 100GB capacity have got to be at least 4 -6 years old by now. Unless you bought them new and just stored them in a box this whole time, chances are, they've seen a lot of hours of operation already. They don't have a resale value more than $20 or so these days, so you're simply not out much money to throw them away or give them to a recycler. Meanwhile, you'll probably get into a much more complex and time consuming NAS configuration, trying to best utilize them in your drive pool. Even if you only make $10/hr. at your job, that means 2 hours of time spent messing around with this is worth the entire value of one of those old drives!

      I'm kind of a pack-rat for computer hardware (since I have an on-site repair business besides a day job in I.T. and computers as a spare time interest too). But even I started throwing away IDE or SATA drives under 250GB a while ago. I keep a *couple* small ones around, but only for odd situations (like someone who wants to revive a really OLD PC with a BIOS that can't recognize larger drives properly). Otherwise, everyone who wants to go to the trouble of swapping an old/dead drive out for a replacement may as well spend the relatively small extra amount of money for a current model of much larger capacity, AND a full warranty still on it. Your data is usually worth it!

    4. Re:Not worth it. by bodangly · · Score: 3, Interesting

      I've had nothing but bad luck with btrfs, including irrecoverable data. (well, data not as valuable as time it would take to restore) It is my opinion that the push to make btrfs the new standard is happening way too quickly and for the wrong reasons. It has been my experience that it simply isn't as reliable as the more established file systems. I would highly recommend XFS over btrfs.

    5. Re:Not worth it. by hairyfeet · · Score: 3, Interesting

      I probably shouldn't answer an AC but since the AC has been put as "insightful" I will answer with why I think he/she is wrong: If the data has already been backed up (which if you don't back up your data you're a dumbass who should be posting to yahoo Answers and not here) then frankly there is no "risk" to using drives you already have as its only a question of how long it would take to restore.

      Now of course he can mirror the data on half the drives to add some redundancy but your answer of using new drives? Until WD is back up to full speed frankly that isn't possible without a LOT of extra cost. Sure you can find cheap Seagate drives but you know what? they're shit. Not badmouthing Seagate but read the reviews and you'll see that Seagate drives over 640Gb are having a crazy high failure rate. Some say they are using ARM controllers made by the Maxtor guys and those are shit, some saying its a firmware issue, I personally don't know but what I DO know is that I've had to shitcan a crazy number of Seagate drives, especially the 1Tb and 1.5Tb drives which are the only really affordable ones ATM.

      Now for how to do this, I'm gonna stay out of the software side since I don't want to jump into a Windows VS Linux flamewar, I'm sure the guy has an OS he is comfortable with and will probably go that way anyway so I'll deal with the hardware. This is how we cooked up something similar at my previous shop with a shitload of SCSI drives the boss got at an auction...Buy a couple of matching cheap full size computer cases, geeks has several for pretty cheap. We then tore the cases apart leaving a couple of skeletons, how far you take them apart is up to you, one can just as easily take the side of one and the opposite side off the other and cut the bracing, the reason we did it this way will become obvious in a minute. Pick up a cheap old server board, you want one that will fit the case and has as many PCI slots as possible, you will of course fill the PCI slots with SATA adapters just like we did with SCSI. Then in our case for the final touch we wired up a $10 Walmart box fan to the side to cool all the drives we piled into that sucker. In our case we used a copy of Win2K Server since we had drivers for the SCSI cards in Win2K Server, but again software is your choice.

      And there you have it! While drives were topping out at 400Gb and cost an arm and a leg we had nearly 2Tb of SCSI goodness containing every single driver for every single part for every single version of Windows from 3.1-WinXP. I don't see why someone couldn't do the same with SATA, sure PCI won't give you as much bandwidth as PCIe but if you have a decent sized amount of RAM (say 2Gb) to buffer I don't see why it wouldn't work. in the end this is about using something you already have which won't cost anything VS spending hundreds of dollars to acquire a more compact solution. Personally I'm all for using what you already have, that is why I have a drawer filled with drives from 80Gb to 400Gb that I then throw into computers that are short on space, certainly cheaper than buying a new WD drive and as I said I don't trust Seagate ATM. Personally I'm just glad I loaded up on Samsung EcoDrives right before the flood when Tiger had them cheap so I can afford to wait until the prices drop before i have to look at drives. But if he already has them, why not use them?

      --
      ACs don't waste your time replying, your posts are never seen by me.
    6. Re:Not worth it. by 19thNervousBreakdown · · Score: 5, Informative

      It depends--is there a total of 6 TB of drives that doesn't include the 3 TB drive?

      Take each disk, make an LVM physical volume from it. From those physical volumes, logical volumes. You don't have to make all of them the size of your smallest drive, you just have to be careful. Say you have the following:

      1: 3 TB
      2: 2 TB
      3: 1 TB
      4: 1 TB
      5: 750 GB
      6: 750 GB
      7: 150 GB
      8: 150 GB
      9: 100 GB
      10: 100 GB

      On your 2 TB drive, make partitions matching the drives under 1 TB.

      On your 3 TB drive, make the following partitions:

      1 TB: RAID-5 with #3 and #4
      750 GB: RAID-5 with #2 and #5
      750 GB: RAID-5 with #2 and #6
      150 GB: RAID-5 with #2 and #7
      150 GB: RAID-5 with #2 and #8
      100 GB: RAID-5 with #2 and #9
      100 GB: RAID-5 with #2 and #10

      You'll end up with the following volumes:

      1: 2 TB
      2: 1.5 TB
      3: 1.5 TB
      4: 300 GB
      5: 300 GB
      6: 200 GB
      7: 200 GB

      Then take those, LVM the RAIDed LVM volumes (fairly certain you can stack [traditional meaning of "stack"] as a contiguous disk, just use an easy FS like ext3, I've run into problems with stack size [programming meaning of "stack"] using XFS on LVM). You end up with 6 TB total space, and, just like normal RAID-5, you don't lose anything unless two disks from one of those groups die. That is, if a disk in 200 GB #6 dies, and a disk in the 1.5 TB #3 dies, you still haven't lost anything. Even if your 3 TB drive dies, which is clearly the worst case since it has data for every array, or the 2 TB which is nearly as bad, you'd still need to lose a second disk to lose any data, so for failure rates it should be the same as a 10-drive RAID-5 array, which isn't quite advisable although it's not murderously bad, but this isn't work and the primary motivation is probably maximizing space with a decent reliability increase, not making next to certain it never goes down. I'm sure it feels really weird, but I don't think you're actually increasing your odds of failure at all over the 10-disk-all-same-size RAID we're used to, other than not trusting older drives--and I'm not so sure those are much more likely to fail than new ones. After all, they've lasted this long, and I've had brand new drives die within weeks. But in point of fact, there's some 2-drive failures that don't take anything down, so I think overall you're doing slightly better than the 10-same-size disk case.

      Now, your disks probably won't divide up as nicely, and you might end up having to either leave some space on the floor or subdivide in weirder ways or both, but with very careful partitioning (never put two stripes of the same array on the same disk), you can do this. Set all the arrays to verify weekly (mdadm can do this) and e-mail you on a failure. Don't set up an audible alarm, you're not going to lose a second disk at 3 AM (but you will wake up to fix it, and be worthless at work the next day for probably nothing) and even if you did lose another disk, you're not using RAID as a replacement for backups, right? Right?

      ZFS would be really nice if it did all this complex stuff for you, but do you have enough control/is it smart enough to allow you to ensure that you get as good or better reliability? It'd be ridiculously easy to make a bad mistake in layout with the above scenario. Because overall, I agree with the title: It's just not worth all this effort so you can use that crappy 100 GB disk. Once it goes down, now you have to replace it.

      --
      <xml><I><am><so><damn>Web 2.0</damn></so></am></I></xml>
    7. Re:Not worth it. by cayenne8 · · Score: 3, Informative
      But I'd have a use for it...over the years, I've gathered a bunch of disks when on sale...many barely used...some still in boxes....

      I'd like a way to throw them all together and use them for backup storage.

      From what I've gathered...use FreeNAS, with ZFS...and it will let you set this up, and allow for up to 2x drives to fail at the same time....

      I think in my case...this would be reasonable. Heck, if I set up two FreeNAS boxes...had one mirror the other one...that would indeed be a decent backup system, no?

      I have a lot of friends like me....often buying stuff on sale for "to use someday on something"...but they just sit and gather dust....I think this would be a good reason to use them, and keep buying new drives, here and there when they go on sale, to replace on the FreeNAS as drive on it do start to fail....

      Heck, thinking of keeping one FN server here..and maybe put a 2nd at friend or parents house out of state...to mirror it...

      --
      Light travels faster than sound. This is why some people appear bright until you hear them speak.........
  2. the 2 main choices: by gbjbaanb · · Score: 4, Informative

    FreeNAS or OpenFiler.

    I think FreeNAS (the BSD based one) is lighter and easier, as OpenFiler seems to be going in a more "fully featured" direction with less support for older hardware, but they're both good.

    1. Re:the 2 main choices: by gbjbaanb · · Score: 3, Informative

      yes it does - it uses ZFS that has some fancy replication features, especially z-pools that are like software raid. You can have a 100GB vdev on both the 100GB and 3TB drive as a mirror. Of course if you have just those 2 drives, nothing is ever going to get you full data redundancy (obviously!) but ZFS gives you a lot of flexibility to use what you do have.

    2. Re:the 2 main choices: by McKing · · Score: 4, Informative

      ZFS does this much more simply with no ugly hacks. You can have mismatched drives when you build a mirror (the mirror is the size of the smallest drive in the mirror set), and then you stripe across the mirrors. As the older, smaller drives fail, replace them with newer, bigger drives and the pool magically gets bigger. 100GB + 500GB mirrored (100GB usable). 100GB dies, swap in a 750GB drive and now this pool is automatically resized to 500GB. Get 2 more drives? Mirror them and add them to the pool and your pool expands with no one the wiser.

      Seriously, if you haven't played with ZFS before, download FreeNAS and give it a whirl. When I was a Solaris admin, ZFS was the most fun thing to work with by far.

      --
      If only "common" sense was actually that common...
    3. Re:the 2 main choices: by ratboy666 · · Score: 3, Informative

      FreeNAS can use ZFS as the filesystem. And this is what you want! Now, the actual configuration depends on the drives you have available.

      For drives with the same, or very similar capacity -- ZRAID can be used. With 3 drives, ZRAID1, or with more, use ZRAID2 (the number is the number of drives which can be failed). ZRAID offers the capacity of the smallest drive, which may waste space. If all drives are (eventually) increased in size, more storage is added.

      For drives with different capacity, ZFS offers the ability to keep a redundent number of copies of the data (eg. specify two copies, or three). Then, ZFS will duplicate the data onto multiple drives.

      As well, ZFS continually monitors the drives and redistributes any failed areas, and ensures that no bit errors accumlate in the file system. ZRAID and multiple copies can be combined.

      The main point of ZFS is to keep your data clean and safe from corruption.

      As well, "fsck" is not needed -- it happens when you "scrub", which slows down the array, but doesn't leave it unusable.

      If you have sufficient memory, ZFS can also "dedup" the blocks in your filesystem, merging identical copies of data (but copying/raiding to maintain data integrity). This feature takes a LOT of RAM (2GB per TB of disk, 32GB for 20TB of disk, and possibly more). Also, some ZFS versions offer encryption (not sure about the one in FreeNAS).

      ZFS drives can be physically moved to another system, and used (eg. FreeNAS x86 to SPARC). Endian and format issues are correctly handled. Not a feature most people would ever use, but it is nice. ZFS is available on Solaris, BSD, Linux, Mac (well, used to be).

      Also, ZFS support snapshots, which can be browsed.

      Finally, ZFS has an eight year history in production.

      In all, what's not to love?

      --
      Just another "Cubible(sic) Joe" 2 17 3061
    4. Re:the 2 main choices: by epine · · Score: 3, Insightful

      In all, what's not to love?

      ZFS does nothing to protect integrity in memory, and especially in the dedup case, your data sits in memory a long time.

      I wouldn't run dedup on a non-ECC mainboard. Had an experimental ZFS that suffered a failed memory stick (this array not run in dedup mode after a performance test following the initial build). The next scrub found inconsistencies on disk. Even after copying all the data to a new location on the same storage tank and deleting the old location, there were internal consistency errors. This didn't surprise me, but illustrates that memory-induced corruption will often kill the entire array. Keep plenty of offline backups.

      Now if you just happen to have a fresh Opteron 3250 lying around on a mainboard populated with the right type of memory, with full server chip validation and background memory scrubbing, fill your boots with those old 30MB IDE drives.

      I'm running my test array on three 500GB drives. Two are enterprise grade and the third was a Seagate warranty replacement (consumer grade refurb). I could have run with the consumer drive as an idle hot spare, but decided to run a three-way mirror, which keeps your hot spare silvered at all times. Note that the consumer drive limits my peak write bandwidth, as the enterprise drives have higher read/write performance. The reads seem to be distributed so that the hot silvered consumer drive works out to a net performance gain.

      My scrub on 50GB of data takes just under 15m. Concurrent read traffic is not greatly impacted at home network levels.

  3. FreeNAS or Unraid. by detritus. · · Score: 3, Insightful

    Look at FreeNAS or Unraid. Unraid has a 3-drive limit IIRC for the free version, but supports an unlimited amount of drives for the non-free version.

    1. Re:FreeNAS or Unraid. by jtownatpunk.net · · Score: 3, Informative

      unRAID does not support unlimited drives in any version. It comes in 3 (free), 6, and 21 drive versions.

      I've been using it for a year or two and, while it's got some limitations, it's a good choice for this application. Mostly because the guy's using a random collection of old drives and is likely to have bad sectors across multiple drives at some point. There is no striping with unRAID so the worst thing that can happen is he'll have to mount the drives individually and copy the data to a new array.

  4. Windows 8 Storage Spaces by aaron44126 · · Score: 3, Interesting

    If you use Windows, the forthcoming Windows 8 "Storage Spaces" feature appears to be perfect for situations like this. http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing-storage-for-scale-resiliency-and-efficiency.aspx

  5. FreeBSD and ZFS by Anonymous Coward · · Score: 3, Interesting

    FreeBSD has fast ZFS support which is wonderful file system to fight data loss.

  6. The mega surplus continues! by digitalsushi · · Score: 5, Interesting

    Ah ha! Who else amongst you has a huge surplus of huge hard drives going unused, now that netflix streaming has displaced 60% of all the crud you had spinning idle in a closet the 3 years before you signed up?

    My storage requirements went from about 3 terabytes to about 30 gigabytes over the past 2 years. I believe I am the archetype and that I am doing the same thing as the average geek. I suspect there are piles of huge disks sitting offline because of this streaming displacement.

    It cost me about 18 dollars a month to leave my x86 file server online, idle (killawatt meter, nh rates); netflix is cheaper than that.

    Come on, who else has a comment related to this.

    --
    slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
  7. FreeNAS, for sure by fmachado · · Score: 3, Informative

    FreeNAS can use ZFS for aggregating multiples drives, independent of size, technology etc, all with varying degrees of protection.

    It's by far the best solution to your case.

    Flavio

  8. Do you care about your electricity bill? by jimicus · · Score: 3, Insightful

    Do you care about your electricity bill at all? If you do, it'll probably be cheaper over the course of 6-12 months to buy a simple NAS box or a cheap atom board and plug in a couple of 2TB hard drives.

  9. WHS V1 by clickclickdrone · · Score: 3, Interesting

    Windows Home Server (V1) - mix and match to your hearts content and all the addins you can eat for adding features.

    --
    I want a list of atrocities done in your name - Recoil
  10. Greyhole! by gregthebunny · · Score: 5, Interesting

    Why am I the only one saying this? Setup Greyhole, throw a bunch of disks at it, and enjoy! And to all those saying "those drives are going to die soon", you can actually tell Greyhole that you consider a drive "broken" and it will still use most of its storage (albeit redundantly) until it does die and have to be removed.

  11. Don't Build.... Buy a Drobo by InitZero · · Score: 3, Insightful

    1. Throw away everything that isn't a standard-sized SATA drive.
    2. Buy a Drobo (http://www.drobo.com/products/professionals/drobo-fs/index.php).
    3. Put the five (or eight) largest drives in the Drobo.
    4. Throw away the rest of the drives.
    5. When you get a drive that is larger than the smallest drive in your Drobo, pull the smaller drive out and insert the larger drive.
    6. Find peace in the universe.

    When I was young and foolish, I tried to keep every drive spinning, even long after its time had passed. I had *nix boxes stuffed with drives and SCSI-attached arrays. I learned a lot about drive management and system administration but, mostly, I learned that there is a value to my time and my time isn't best utilized playing disk administrator.

    Drobo doesn't pay me a dime and I am still more excited about Drobo than any technology product since TiVo.

    Cheers,
    Matt

  12. Another vote for unRAID by sirwired · · Score: 3, Informative

    I've been using unRAID for years and it's a great solution for a small home NAS box. If you ever change your mind about using it, you simply turn your parity drive into a regular Linux boot disk, and the remaining drives are just regular Reiserfs2 filesystems. Most RAID systems and/or software would require much gymnastics to de-RAID them, if it could be done at all.

    In addition, hardware-based striped RAID makes you dependent on the RAID controller; if it dies and you can't find a replacement compatible with the original's striping mechanism, your data just disappeared.

  13. StableBit DrivePool + WHS 2011 by alexpi · · Score: 4, Interesting

    Full disclosure: I am the developer

    Check out: http://stablebit.com/DrivePool

    It's a software disk pooling solution that combines any number of disks of any size into one big virtual pool. You can designate certain folders to be duplicated on the pool. Any files placed in duplicated folders will be stored on 2 disks at the same time.

    The implementation is a hard core NT kernel driver with a virtual disk. There is a full NT kernel storage stack, no user mode hacks here.

    Unlike RAID and similar solutions, all your pooled files are stored as standard NTFS files on each individual disk in the pool. This means that you can simply plug in any pooled disk to any system that can read NTFS to get at your files in case disaster strikes.

    It's commercial software, $20 USD per server.

  14. take look at amahi.org by JonySuede · · Score: 4, Interesting

    look at amahi.org, it is a turn-key Home Server based on fedora and greyhole as it's replication engine.
    Dump anything less than a TB except one drive and you are set.
    You set the replication level by share and it keeps a full copy on each drive until the replication count is reached for that file on that share.

    Example:
    you have 4 1TB drives and 1 500Gb drive.
    You have the share photo configured to replicate on each drive.
    You have replication off on the video share.
    You have a replication level of two on the mp3 share.

    When you store a photo greyhole write it to your 5 drives.
    When you store a video it goes on a random drive.
    When you store a mp3 it goes to 2 random drives.

    So if you lose a drive you should loose about 25% of your videos, 6.25% of your mp3 and 0% of your pictures.

    --
    Jehovah be praised, Oracle was not selected
  15. Just throw them away by zmooc · · Score: 4, Informative

    Powering 10 old harddrives for some time is going to be much more expensive than just getting a new one. A modern drive uses about 5W on average. these oldies probably use much more. 10 drives using 10 watts at $0.10 per kwh will set you back $87 per year. You do the math.

    --
    0x or or snor perron?!
  16. www.pogoplug.com by JazzLad · · Score: 3, Interesting

    Pogoplugs are great, can plug in 4 drives via USB or more with a USB hub. I paid $25 for mine, can't really go wrong.

    --
    "If you have nothing to hide, you have nothing to fear." - Every fascist, ever
  17. Use BtrFS or Drobo by digitaltraveller · · Score: 4, Informative

    Use drobo if you are time poor and money rich, use btrfs if you are time rich and money poor.

    Btrfs's capabilities are nothing short of amazing. Here is a vid about it:
    http://www.youtube.com/watch?v=9bQc_z-Cb7E