Slashdot Mirror
Data Storage For Home?
kuom asks: "Every couple of years, I face the same problem: running out of hard drive space. No matter how big of a hard drive I get, I seem to find ways to fill it up within a few months. The size of my hard drive grew from 2.5G to 13G, to 20G, to 40G, 80G, 120G, and most recently, 200G. Today, I have a combined hard drive space of 280G, but I again find that I only have about 2G of free space left. My collection of family photos, web site content, TV episode captures, music files, and my archive of ISO files for various operating systems, they just eat up my hard drive space so fast. I could get a 400G hard drive, but I figure maybe it's time to think about something long term, something like EtherDrive or StorageWare. But the price tags are definitely out of my range. Slashdot readers, what do you suggest for home data storage?"
(get the 'trial' from microsoft and find tweaknt to remove the timebomb, don't worry about activation it should no problem).
I suspect someone in Redmond might have a problem with this. Using Linux and Samba would be a much better idea, and easy to do with a distribution like Ubuntu, Mephis or Suse.
-- $G
Your solution is to move things off of a hard drive. Correct. But you don't need a long-term professional solution--especially not when there's a long-term (or, if you want to be anal, medium-term) archival technology already tasked for the home.
Spend $100 and get yourself a DVD burner. Don't just use it for backup, but actually move things that you don't reference all the time--ESPECIALLY those ISOs you almost certainly don't need live--off of the live storage. For things that are important / irreplacable, make several copies. Distribute them far and wide to friends and family if you want.
Replace "hard drive storage" with "space in my home", and you'll see that it's not a matter of getting enough space to store all your stuff, it's a matter of deciding what's important to you that needs to be kept.
Where it's electronic bits or physical items, some things are more important to you than others. Take a long hard look at what things you absolutely need, and toss the rest. Will your life be that must worse if you didn't have "______" within easy reach at any given moment? Probably not. And you'll feel better knowing that the things you do keep are the important ones.
And don't think of it as parting with things you'd rather have kept. Think of it as making room for more new stuff.
Good luck.
Tom
If you're cheap, you can get the cheapest new system you can find, rip out the cd drive to free up an ide channel, find a way to safely mount four 250gb drives in it (I'd recommend the 250gb seagate 7200.8 for longevity, but that's just me) to make a raid 5, install a floppy drive, and use the floppy drive to do a net install of debian. Then set up samba for windows file sharing. That 750gb of redundant raid 5 storage will set you back between $500 and $900, depending on if you already have a system in mind to use for the job. Or you can get a good server with a similar setup for around $1500 or $1800, depending on if it needs to be fast.
Or you can just delete the ISO's and TV episodes, if they're not worth the extra cost and you're never going to use/watch them again anyway.
Whatever drives you get, make sure you research their quality first, especially if you don't care for the extra cost of a redundant raid and/or backups.
First off, burn your shows to DVDs and your ISOs to CDs. Why have them on the hard drive if you aren't going to use them?
I'd set up an older computer system (P2 or newer or older..) with a RAID configuration. Something like RAID1 for simplicity and redundancy. Then you just store everything on that central server and your client machines don't matter.
Or get a bunch of chinese kids in your basement to memorize strings of 0's and 1's.
Death by snoo-snoo!
Re-evaluate what's on your storage. You sound like me - a bit of a hoarder.
Firstly, get a good catalogue system going. Put all your crap in one area. Get a sorted listing of creation time,last access time and categories. Get some hard-and-fast rules going as to when it should be archived offline. Go through once a month (or more often) and work out any new stuff that you need to backup. You do backup, don't you? Of course you do.
Then go through and check the last access times of your categories, and move to offline storage as appropriate.The advantage of getting this sort of regime going is that you've got more chance of having a backup offline somewhere when the inevitable happens and your drive wakes up dead one day.
For example, once you've got categories and last access times sorted:
- Digital Photos, family movies , documents - anything more than 12 months - Offline to DVD. Use par2 for archive copies (sent to distant relatives for storage), but make another set without par2 for normal, semi-random access.
- ISO's of distros? Got a broadband connection? Ditch them.
No broadband? Get them all offline, regardless of age.
- TV Episodes? If it's later than say, 3 months - off to DVD they go.
- Web Site content? 3 months.
And so on. Work the times out for yourself.
You really need a good cataloging system to help find out where the offline files are. Everyone's idea of a good catalog is different - Hell, I just label DVD's and keep them sorted by catagory - so I'll leave it to you.
If you organise your data and find that you still haven't enough drive space to keep all your current data online, then (and only then) go and look at the expensive options.
You are in a twisty maze of processor lines, all alike.
There is a lot of hype here.
(the mobo doesn't have to be great, nore the ram)
Noooo, you don't need good hardware for a fileserver. Stability and data integrity don't matter at all. [/sarcasm]
Dimwit.
"The newly born animals are then whisked off for a quick run through a giant baking oven." --heard on Food Network
save up the cash and put together a 2 terabyte SATA storage server (the mobo doesn't have to be great, nore the ram)
That all depends on how important your data is. High quality parts should be of top concern. Also, don't skimp out on the ram -- the more ram you have, the more the server will have to cache recently accessed files.
2 TB should be enough space. MAKE SURE YOUR NETWORK THAT THE SERVER IS CONNECTED TO IS 100 MBIT (preferably the client transfering files to/from as well). It might be a good idea to maybe invest in gigabit.
2 TB will fill up rather quickly, especially if the server is used to store high definition video (like HDTV). Also, 100 Mbit is a huge bottleneck -- that only gives you roughly 12 MB/s, which isn't enough to keep up with highspeed DVD burners or even play full resolution HDTV. Furthermore, if you're using ethernet, remember that usage above 60%, especially with multiple clients (say, two people pulling/pushing lots of the server at once), suffers from rather high latency and packet collision. Moving the same amount of data over a 1000 Mbit connection reduces collisions and congestion by 99% as data moves 10 times as fast down the wire. Using gigabit (at least when ethernet is concerned) is really the only option with decent performance.
Be relentless!
Ya, I don't get the main parent: "I keep filling up disks and buying bigger ones and I am sick of it. Can anyone recomend a slightly bigger disk that will permanently solve my problem?"
I'm a believer in segregating storage from the machines I'm likely to use/want to reload etc.
:)
And I've spent thousands of dollars on my home network and attendant PCs, to solve the problems that the original poster will only have if he manages to actually get enough storage for his needs.
Presently I have four identical storage servers, with the following characteristics:
Athlon64/3000, 1GB RAM, Gigabyte K8VM800 motherboards, 4 Hitachi 7k250s (RAID5 on 3ware ), 2 Hitachi 7k400s (soft RAID0, stores a daily snapshot of the RAID5, which is the data that is actually shared), 1 Samsung SP1614 Boot/OS drive, a 3Ware 8506-4LP, Intel Gbit NICs.
These machines run a series of scripts that collect and copy (pictures or MP3s) or move (video) whatever I happened to have dropped on my various workstations (each have between 300GB and ~1.7TB) to appropriate filesystems on the various servers (one for porn, one for ripped DVDs and TV shows, one for music, one for pictures); those filesystems are then exported via NFS to another Linux machine where the whole mess is presented back to all my machines as a single file system.
Getting enough storage is simply a matter of applying money. 250GB drives are quite reasonable nowadays and 160GB drives are downright cheap, but dealing with dinky little disks make getting enough SATA ports problematic. Sub-$100 2- and 4-port SATA controllers from the likes of Adaptec, Promise and Highpoint have their own problems. Most don't do online volume management and REALLY only do RAID through a driver, rather than an actual onboard processor. They're fine for storage expansion, a JBOD or RAID0 (note: RAID0 is normally a VERY stupid thing to do, since most people aren't doing STR-intensive things with their PCs and the chance of losing data is substantially higher than for any single disk), but as with everything else, you get what you pay for, and ports on a proper controller are probably worth more than the disks you're attaching to them.
RAID5 is also kind of a bad deal for write-intensive data - lots of little files that get updated a lot, while I'm at it. Do RAID1 or RAID10, (or maybe RAID3 if you can find a controller that supports it) for data you care about. Spend money.
USB2 and Firewire enclosures are NOT a good solution for adding primary storage most of the time. Generic enclosures frequently have difficulty with larger drives, and often have VERY cheap fans that either fail quickly or detriorate to the point that they sound like a penny in a vaccuum cleaner. Additionally, the performance and CPU utilization of USB2 enclosures both tend to be god-awful. Brand-name enclosures have a few different problems: many use 2.5" disks, which in my experience are rather delicate. Others are not properly cooled, and almost all of them are sealed enclosures. Better to put a drive inside a computer if possible. I tend to think of USB2/Firewire drives drives as backup devices only.
Disk-wise I tend to prefer Hitachi 250 and 400GB models, or Samsung 160s or 200s, and SATA over PATA when possible. The Hitachi 500GB models get too damned hot, and it's the only one that's out (available for purchase) at the moment. Seagate and Maxtor ATA products tend toward tepid performance, and in the case of Maxtor, quality hasn't been good since the Quantum merger in 2001. I will not purchase a Western Digital drive for any reason, but specifically I avoid the geek-favorite Raptor 360GD; I was party to the construction of a small 20-drive SAN using Raptors (client's spec, not my idea) a couple years ago where the drive failure rate was approximately one drive every 33 days.
-- I wanna decide who lives and who dies - Crow T. Robot, MST3K
Take a long hard look at what things you absolutely need, and toss the rest.
Not me. For two reasons. First, I've too often found myself wishing I had something that I had deleted and second, it is simply not worth my time to wade through it all and decide what I need and what I don't. Disk space costs well under 50 cents per gigabyte, and even after you add some redundancy (RAID), it's still very cheap.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Print a harcopy.
Where'd you learn math? 200 DVD's times 4.5GB (actually more like 4.4) per disc is only 900GB, less when you consider you can rarely fill the disc up completely due to odd files sizes. Plus most of those disc changer robots run to the thousands of dollars, you're better off dropping $6-700 on 4 or 5 300GB hard drives and a cheap PC to serve up files from them, I also suggest a seperate smaller drive (20-40GB is more than enough) to boot your OS off since software raid arrays aren't bootable. Put together a cheap system for $2-300 to strap the drives in and you've got over a TB of live networkable storage for less than $1000, spend a hundred more on a tuner card and install MythTV on the machine and you just got a massive PVR system as a bonus.
It sounds like you may be a digital packrat. If you are, I sympathize as I was one too after I got broadband. I stopped after realizing how much time/money I was wasting on crap that really didn't improve my quality of life. Now I buy new hard drives for my file server once every three years instead of every three months. Following are some of the things that helped me.
1) Download less porn/warez. Or put off downloading more until you've watched/run/played what you have. If you're just one person cranking through that much space that quickly, then you're downloading things just to have them. Stop that.
2) Go through and 'rm -rf' files and directories that you haven't accessed in a year. Don't keep obsolete versions of operating systems around, because you won't use them. As soon as you download a CD image, burn it and rm the ISO.
3) Archive on external media anything that's sentimental but rarely accessed.
4) Make it a routine to burn stuff to CD/DVD at least once a week. Eventually, you'll get tired of wasting time burning crap that you don't use and this will help you realize that you really don't need it at all.
5) If you do a lot of video editing or webmastering that requires huge amounts of data, and you're making money at it, then you need to invest in a proper server to keep all that. Be sure to make backups too. If you do this for a company, have them take responsibility for this.
6) Take a page from Linus's book: Upload it to an FTP server and let the world mirror it.
RAID5 will fuck you if you depend on it to be your only failsafe on your data.
:)
Repeated again:
RAID5 will fuck you if you depend on it to be your ONLY failsafe on your data.
Your motherboard/controller could screw you. You could delete some files. 2 drives can fail at the same time (power surge,etc.)
There's just no excuse for not backing things up. I personally have a DDS3 tape drive in my file server for once yearly backups. Every 6 months I do a set of rewritables (DVD+RW). Every year or so I make a permanent copy on DVD+ or -R, and I buy decent burnables for that.
I've had instances where controllers, cables, and my own screw ups have lost data. But the cost to my time is minimal since I have backups in place. The way I figure it, the time I spend safeguarding my data is worth its weight in gold WHEN I have to depend on the backup for critical data.
Besides, Corporations back up their data - why can't we?
Karnal
100 Mbit equipment is so cheap now, every PC that's at least 100 MHz should have it. If you have some older workstations, 100 Mbit cards may be hard to find, but these would have to be pretty old boxes to not come with 100 Mbit ...
That's not true if you've got a switch rather than a dumb hub, and switches are pretty much the norm now. Switched networks have almost zero collisions, no matter how busy they are. (Granted, they should have zero collisions, not almost zero, but switches do occasionally have bugsSure, latency does go up when you're pounding on the network, but on a 100 Mbit network it's not that much -- just a few miliseconds at most. And for moving lots of data around, latency doesn't matter much anyways -- it's throughput that matters.
Not quite. You won't actually see 120 MB/s with today's hardware, but you might see 20-30 MB/s, maybe a little more on really serious hardware, which is still a lot better than the 12 MB/s you'll get out of 100 Mbit ethernet. And it's relatively cheap now -- I recently got gigabit PCI cards for $5 and a 5 port gigabit hub for $20 at Fry's.I have two OpenBSD systems (Compaq D5S SFF boxes), each with a 400GB hard disk. One sits at home in a segregated part of the network running Samba. This serves up files for a Powerbook (OS X) and my wife's Dell (Win XP). The other system sits at my best friend's house. Both systems allow SSH with public key auth only. I tunnel rsync over SSH from the master box at home to the other system to sync files. This happens once a day at 2am.
The master box at home has a cheap DDS4 drive which I got from eBay for about US$65. I get brand new tapes for about $4 a piece. I have a 4 set cycle on which I do a full backup every week. After completion these tape sets get taken to a different friends house and she keeps them for me. Incrementals during the week I keep hold of.
My Powerbook has a superdrive and I do a set of DVD+RW's every once in a while. I copy my critical files such as accounts etc additionally to a 512MB USB key, which is encrypted and I keep in my bag. That USB key contains the data that I simply could not do without.
As for storage needs, when I begin to outgrow my 400GB allocation and tape & DVD backups/archival becomes too time consuming, then I will move to a RAID 10 solution. However, for me this probably won't be until 2008. At that point I will probably do another home brew solution. I have started looking at what those options are. There are quite a few companies that offer rack mountable chassis with lots of storage bays. My idea was to put a stripped down version of OpenBSD booting from industrial grade CF card, similar in concept to how one might go about making a firewall device. I'd go for a good quality server motherboard and then add a well supported SCSI or SATA RAID Card. Seagate Barracuda drives have a 5-year warranty on them, so I'd probably go for some of those. I'd then add striped and mirrored storage as required as my storage needs expanded. This is probably going to be a fairly costly exercise, but I shall be doing an audit of my filesystem at some point this year to clear out the old files and pornography with which I have become bored.
The most important thing to do is work out what data is actually worth protecting. If like me you have ripped your Futurama and "24" DVD episodes into DivX for easy access and local copies whilst on the road, bear in mind that you can always re-rip them. If the house goes up in a puff of smoke, it would probably be cheaper to re-purchase the DVD set with the insurance money rather than keep archiving copies of stuff onto media which you have to continually protect, maintain and renew. The stuff I'm paranoid about is the data that I *can't* buy again - i.e. pictures of holidays, family and honeymoon, my accounts files and scans of receipts for my business, plus logs of chats and text messages.
Good luck.
delete most of your porn. It mostly looks alike anyway, so just ditch it. You should then have close to 200 GB free.
Some people are like slinkys. They're useless, but it puts a smile on your face to push them down the stairs.
SGI has a neat solution to this problem called DMF. It will automatically migrate infrequently-accessed files onto tape, and 'recover' them when accessed (takes a little longer than normal) automatically when there's a tape library, and I expect it will prompt an operator to load a tape if the tape isn't online.
I've no idea what the cost is, or if there's a low-end solution.
Max.
The bigger the disk, the longer it takes before you get the dreaded 'disk full' message, but it will take exponentially longer to recover from it.
Who cares? The last disk I added took three days to be fully on-line. But my investment in the process was about 30 minutes. It took 20 minutes to write the script to iterate through my RAID-5 volumes, run "pvmove" to migrate the data off of each volume, remove it from the volume group, rebuild it with the new disk and add it back into the group, then another minute or two each time I checked on the progress. In the meantime, my data was always on-line, and always safe from up to two (consecutive) disk failures.
The problem you describe is only a problem if you have lousy disk management tools.
Oh, just a nit: your use of the word "exponentially" is inaccurate. Even without LVM or the like, recovery time increases linearly with total storage, not exponentially.
there is no shops open to get new disks or all your storage enclosures are already filled to capacity with the biggest disks you can get
Yes, you could have a problem if you allowed this situation to arise. So don't do it. I keep a goodly chunk of my available space unallocated to any logical volumes, so when I actually run out of space, I simply run "lvextend" to allocate the space, and then set about getting more storage. The bigger problem is when there's no more room in the box for additional drives. The solution to that is simple: Don't let it get that way. Always make sure you have room for one more. When your box is almost full, and you need more storage, you have to either add another enclosure or, more likely, replace a drive with a bigger one. The scenario where they're already the biggest drives available is pretty unlikely. My file server can accomodate six drives so filling it up with the currently-largest drives would be 3TB of storage. That's a *lot* of home movies.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
See my signature. And that was seven to nine months ago, so it might very well be cheaper now, and/or possible to get more capacity with the newly-available 500GB drives.
Yes, $4100 is a lot of money. But I built it for exactly the reason you mentioned; I've always been of the philosophy of "do it once and do it right," and this way I've taken care of my storage needs for years to come.
1TB ought to be enough for anybody.
... remember that whole 640K thing.
Now, Bill
The higher the technology, the sharper that two-edged sword.