Data Storage For Home?

kuom asks: "Every couple of years, I face the same problem: running out of hard drive space. No matter how big of a hard drive I get, I seem to find ways to fill it up within a few months. The size of my hard drive grew from 2.5G to 13G, to 20G, to 40G, 80G, 120G, and most recently, 200G. Today, I have a combined hard drive space of 280G, but I again find that I only have about 2G of free space left. My collection of family photos, web site content, TV episode captures, music files, and my archive of ISO files for various operating systems, they just eat up my hard drive space so fast. I could get a 400G hard drive, but I figure maybe it's time to think about something long term, something like EtherDrive or StorageWare. But the price tags are definitely out of my range. Slashdot readers, what do you suggest for home data storage?"

Comments

DVD

[Planesdragon]

Your solution is to move things off of a hard drive. Correct. But you don't need a long-term professional solution--especially not when there's a long-term (or, if you want to be anal, medium-term) archival technology already tasked for the home.

Spend $100 and get yourself a DVD burner. Don't just use it for backup, but actually move things that you don't reference all the time--ESPECIALLY those ISOs you almost certainly don't need live--off of the live storage. For things that are important / irreplacable, make several copies. Distribute them far and wide to friends and family if you want.

Re:DVD

[rincebrain]

$100!? $40 nets you a nice one off Newegg. Dual layer, too.

Re:DVD

[Reg Nullify]

Just don't use a sharpie pen to write labels on your disks. I did this on some CD-ROMs I used to backup some digital video and 2 years later the disk was badly corrupted. When you write on the silvery label you're actually writing on the back of the recording medium. The solvents from the sharpie leak right through the thin layer and corrupt the data. Printing out labels or using a special pen made for labeling CD's and keeping the Cd's out of the sun will help the data remain secure.

Re:DVD

[fm6]

You're right, of course. But that leads us to our next Ask Slashdot: "I have hundreds of data DVDs witch all kinds of shit on them. I can never find the one I want. How do I keep track?"

Re:DVD

[grondu]

Labels can be bad too.

This is true. I had several CDs I had burned a few years ago that were unreadable. I removed the labels by soaking the disc in isopropyl alcohol and after that, I had no problem reading the discs.

Re:DVD

[Phreakiture]

When you write on the silvery label you're actually writing on the back of the recording medium.

Of CDs: true. Of DVDs: False. DVDs sandwich the recording medium between two layers of substrate, rather than having it sprayed on the one side of the disc. This has several benefits:

• You can't scratch the data off of the disc accidentally (you can scratch the disc, but the data is still there and can be rescued)
• You can create 2 sided discs (can't do that with a CD, because the focal point of the laser will be wrong)
• Solvent properties of labeling materials won't directly affect the data layer. They can dig into the substrate a bit, but have to dig really deep to get to the data. (On a side note, I have at least two commercially-recorded CDs that have gone bad because the black paint used in labelling them cut through to the aluminum reflective layer. Such a thing would be far less likely on a DVD.)

On labels, we have a problem here at my workplace in that we receive data from a certain other department on DVD. Regulations require that the DVD be labelled in a particular fashion (doesn't say use a label, does say $wordy_legal_boilerplate must appear on the disc) and the other department refuses to cease using sticky labels to do this. Our department all use laptop computers (because we all have to work weekends from home occasionally) and the sticky label causes the disc to warp slightly as it heats up in our machines, thus causing it to stop being legible after about 15 minutes unless you keep it 100% spun up the entire time. Labels on optical discs are evil

Prioritize what you absolutely need

[tomlouie]

Replace "hard drive storage" with "space in my home", and you'll see that it's not a matter of getting enough space to store all your stuff, it's a matter of deciding what's important to you that needs to be kept.

Where it's electronic bits or physical items, some things are more important to you than others. Take a long hard look at what things you absolutely need, and toss the rest. Will your life be that must worse if you didn't have "______" within easy reach at any given moment? Probably not. And you'll feel better knowing that the things you do keep are the important ones.

And don't think of it as parting with things you'd rather have kept. Think of it as making room for more new stuff.

Good luck.

Tom

What's your price range?

[dtfinch]

If you're cheap, you can get the cheapest new system you can find, rip out the cd drive to free up an ide channel, find a way to safely mount four 250gb drives in it (I'd recommend the 250gb seagate 7200.8 for longevity, but that's just me) to make a raid 5, install a floppy drive, and use the floppy drive to do a net install of debian. Then set up samba for windows file sharing. That 750gb of redundant raid 5 storage will set you back between $500 and $900, depending on if you already have a system in mind to use for the job. Or you can get a good server with a similar setup for around $1500 or $1800, depending on if it needs to be fast.

Or you can just delete the ISO's and TV episodes, if they're not worth the extra cost and you're never going to use/watch them again anyway.

Whatever drives you get, make sure you research their quality first, especially if you don't care for the extra cost of a redundant raid and/or backups.

Re-evaluate.

[ColaMan]

Re-evaluate what's on your storage. You sound like me - a bit of a hoarder.

Firstly, get a good catalogue system going. Put all your crap in one area. Get a sorted listing of creation time,last access time and categories. Get some hard-and-fast rules going as to when it should be archived offline. Go through once a month (or more often) and work out any new stuff that you need to backup. You do backup, don't you? Of course you do.

Then go through and check the last access times of your categories, and move to offline storage as appropriate.The advantage of getting this sort of regime going is that you've got more chance of having a backup offline somewhere when the inevitable happens and your drive wakes up dead one day.

For example, once you've got categories and last access times sorted:

- Digital Photos, family movies , documents - anything more than 12 months - Offline to DVD. Use par2 for archive copies (sent to distant relatives for storage), but make another set without par2 for normal, semi-random access.

- ISO's of distros? Got a broadband connection? Ditch them.
No broadband? Get them all offline, regardless of age.

- TV Episodes? If it's later than say, 3 months - off to DVD they go.

- Web Site content? 3 months.

And so on. Work the times out for yourself.

You really need a good cataloging system to help find out where the offline files are. Everyone's idea of a good catalog is different - Hell, I just label DVD's and keep them sorted by catagory - so I'll leave it to you.

If you organise your data and find that you still haven't enough drive space to keep all your current data online, then (and only then) go and look at the expensive options.

I don't think you want to know, but...

[slaker]

I'm a believer in segregating storage from the machines I'm likely to use/want to reload etc.
And I've spent thousands of dollars on my home network and attendant PCs, to solve the problems that the original poster will only have if he manages to actually get enough storage for his needs.

Presently I have four identical storage servers, with the following characteristics:
Athlon64/3000, 1GB RAM, Gigabyte K8VM800 motherboards, 4 Hitachi 7k250s (RAID5 on 3ware ), 2 Hitachi 7k400s (soft RAID0, stores a daily snapshot of the RAID5, which is the data that is actually shared), 1 Samsung SP1614 Boot/OS drive, a 3Ware 8506-4LP, Intel Gbit NICs.

These machines run a series of scripts that collect and copy (pictures or MP3s) or move (video) whatever I happened to have dropped on my various workstations (each have between 300GB and ~1.7TB) to appropriate filesystems on the various servers (one for porn, one for ripped DVDs and TV shows, one for music, one for pictures); those filesystems are then exported via NFS to another Linux machine where the whole mess is presented back to all my machines as a single file system.

Getting enough storage is simply a matter of applying money. 250GB drives are quite reasonable nowadays and 160GB drives are downright cheap, but dealing with dinky little disks make getting enough SATA ports problematic. Sub-$100 2- and 4-port SATA controllers from the likes of Adaptec, Promise and Highpoint have their own problems. Most don't do online volume management and REALLY only do RAID through a driver, rather than an actual onboard processor. They're fine for storage expansion, a JBOD or RAID0 (note: RAID0 is normally a VERY stupid thing to do, since most people aren't doing STR-intensive things with their PCs and the chance of losing data is substantially higher than for any single disk), but as with everything else, you get what you pay for, and ports on a proper controller are probably worth more than the disks you're attaching to them.
RAID5 is also kind of a bad deal for write-intensive data - lots of little files that get updated a lot, while I'm at it. Do RAID1 or RAID10, (or maybe RAID3 if you can find a controller that supports it) for data you care about. Spend money. :)

USB2 and Firewire enclosures are NOT a good solution for adding primary storage most of the time. Generic enclosures frequently have difficulty with larger drives, and often have VERY cheap fans that either fail quickly or detriorate to the point that they sound like a penny in a vaccuum cleaner. Additionally, the performance and CPU utilization of USB2 enclosures both tend to be god-awful. Brand-name enclosures have a few different problems: many use 2.5" disks, which in my experience are rather delicate. Others are not properly cooled, and almost all of them are sealed enclosures. Better to put a drive inside a computer if possible. I tend to think of USB2/Firewire drives drives as backup devices only.

Disk-wise I tend to prefer Hitachi 250 and 400GB models, or Samsung 160s or 200s, and SATA over PATA when possible. The Hitachi 500GB models get too damned hot, and it's the only one that's out (available for purchase) at the moment. Seagate and Maxtor ATA products tend toward tepid performance, and in the case of Maxtor, quality hasn't been good since the Quantum merger in 2001. I will not purchase a Western Digital drive for any reason, but specifically I avoid the geek-favorite Raptor 360GD; I was party to the construction of a small 20-drive SAN using Raptors (client's spec, not my idea) a couple years ago where the drive failure rate was approximately one drive every 33 days.

Delete? Never!!

[swillden]

Take a long hard look at what things you absolutely need, and toss the rest.

Not me. For two reasons. First, I've too often found myself wishing I had something that I had deleted and second, it is simply not worth my time to wade through it all and decide what I need and what I don't. Disk space costs well under 50 cents per gigabyte, and even after you add some redundancy (RAID), it's still very cheap.

Simple.

[clambake]

Print a harcopy.

what works for me

[Eil]

It sounds like you may be a digital packrat. If you are, I sympathize as I was one too after I got broadband. I stopped after realizing how much time/money I was wasting on crap that really didn't improve my quality of life. Now I buy new hard drives for my file server once every three years instead of every three months. Following are some of the things that helped me.

1) Download less porn/warez. Or put off downloading more until you've watched/run/played what you have. If you're just one person cranking through that much space that quickly, then you're downloading things just to have them. Stop that.

2) Go through and 'rm -rf' files and directories that you haven't accessed in a year. Don't keep obsolete versions of operating systems around, because you won't use them. As soon as you download a CD image, burn it and rm the ISO.

3) Archive on external media anything that's sentimental but rarely accessed.

4) Make it a routine to burn stuff to CD/DVD at least once a week. Eventually, you'll get tired of wasting time burning crap that you don't use and this will help you realize that you really don't need it at all.

5) If you do a lot of video editing or webmastering that requires huge amounts of data, and you're making money at it, then you need to invest in a proper server to keep all that. Be sure to make backups too. If you do this for a company, have them take responsibility for this.

6) Take a page from Linus's book: Upload it to an FTP server and let the world mirror it.

Re:I took the easy way out

[karnal]

RAID5 will fuck you if you depend on it to be your only failsafe on your data.

Repeated again:

RAID5 will fuck you if you depend on it to be your ONLY failsafe on your data.

Your motherboard/controller could screw you. You could delete some files. 2 drives can fail at the same time (power surge,etc.)

There's just no excuse for not backing things up. I personally have a DDS3 tape drive in my file server for once yearly backups. Every 6 months I do a set of rewritables (DVD+RW). Every year or so I make a permanent copy on DVD+ or -R, and I buy decent burnables for that.

I've had instances where controllers, cables, and my own screw ups have lost data. But the cost to my time is minimal since I have backups in place. The way I figure it, the time I spend safeguarding my data is worth its weight in gold WHEN I have to depend on the backup for critical data.

Besides, Corporations back up their data - why can't we? :)

Re:here's an idea...

[dougmc]

2 TB should be enough space.

For now. Not forever.

MAKE SURE YOUR NETWORK THAT THE SERVER IS CONNECTED TO IS 100 MBIT (preferably the client transfering files to/from as well)

Really, at this point in the game, if you've got 2 TB of disk, you should have every computer in your house using 100 Mbit or better networking. (I'll let your slower 802.11* connected computers slide, but you'll find that wireless to be maddenly slow when moving large files ...)

100 Mbit equipment is so cheap now, every PC that's at least 100 MHz should have it. If you have some older workstations, 100 Mbit cards may be hard to find, but these would have to be pretty old boxes to not come with 100 Mbit ...

Furthermore, if you're using ethernet, remember that usage above 60%, especially with multiple clients (say, two people pulling/pushing lots of the server at once), suffers from rather high latency and packet collision.

That's not true if you've got a switch rather than a dumb hub, and switches are pretty much the norm now. Switched networks have almost zero collisions, no matter how busy they are. (Granted, they should have zero collisions, not almost zero, but switches do occasionally have bugs ...)

Sure, latency does go up when you're pounding on the network, but on a 100 Mbit network it's not that much -- just a few miliseconds at most. And for moving lots of data around, latency doesn't matter much anyways -- it's throughput that matters.

as data moves 10 times as fast down the wire.

Not quite. You won't actually see 120 MB/s with today's hardware, but you might see 20-30 MB/s, maybe a little more on really serious hardware, which is still a lot better than the 12 MB/s you'll get out of 100 Mbit ethernet. And it's relatively cheap now -- I recently got gigabit PCI cards for $5 and a 5 port gigabit hub for $20 at Fry's.

Re:2.8TB RAID 5 for $4100

[Glonoinha]

He is looking for a way to back up porn.
For $4,100 he could just hire people to come to his house and fuck each other while he watched.