Slashdot Mirror
Data Storage For Home?
kuom asks: "Every couple of years, I face the same problem: running out of hard drive space. No matter how big of a hard drive I get, I seem to find ways to fill it up within a few months. The size of my hard drive grew from 2.5G to 13G, to 20G, to 40G, 80G, 120G, and most recently, 200G. Today, I have a combined hard drive space of 280G, but I again find that I only have about 2G of free space left. My collection of family photos, web site content, TV episode captures, music files, and my archive of ISO files for various operating systems, they just eat up my hard drive space so fast. I could get a 400G hard drive, but I figure maybe it's time to think about something long term, something like EtherDrive or StorageWare. But the price tags are definitely out of my range. Slashdot readers, what do you suggest for home data storage?"
(get the 'trial' from microsoft and find tweaknt to remove the timebomb, don't worry about activation it should no problem).
I suspect someone in Redmond might have a problem with this. Using Linux and Samba would be a much better idea, and easy to do with a distribution like Ubuntu, Mephis or Suse.
-- $G
Your solution is to move things off of a hard drive. Correct. But you don't need a long-term professional solution--especially not when there's a long-term (or, if you want to be anal, medium-term) archival technology already tasked for the home.
Spend $100 and get yourself a DVD burner. Don't just use it for backup, but actually move things that you don't reference all the time--ESPECIALLY those ISOs you almost certainly don't need live--off of the live storage. For things that are important / irreplacable, make several copies. Distribute them far and wide to friends and family if you want.
I know several people personally who have had success with the MacGurus 'Burly Box', you can do up to 1.2TB (RAID 5) for around $1500. See the following URL:
. php
p
http://macgurus.com/productpages/sata/sataguide_1
The 'product page' is at http://macgurus.com/productpages/sata/satakits.ph
Replace "hard drive storage" with "space in my home", and you'll see that it's not a matter of getting enough space to store all your stuff, it's a matter of deciding what's important to you that needs to be kept.
Where it's electronic bits or physical items, some things are more important to you than others. Take a long hard look at what things you absolutely need, and toss the rest. Will your life be that must worse if you didn't have "______" within easy reach at any given moment? Probably not. And you'll feel better knowing that the things you do keep are the important ones.
And don't think of it as parting with things you'd rather have kept. Think of it as making room for more new stuff.
Good luck.
Tom
Software RAID5, I've got one along my 6 40Gb HDs.
Not all that big (only 200Gb total) but if I got 3 of them HDs as payment on an old debt. I'm sure that if I had invested a little bit more I would have bought 4 SATA drives and rigged them up. Software raid isn't quick but it'll store stuff with the added bonus of allowing a drive to fail without losing anything.
If you're into spending more money for better performance, buy a SATA RAID (levels 0,1,5) controler, there are a few on the market that aren't overly expensive, and a few SATA drives - I got a TB solution at work for less than it cost to build a new user-level PC.
Ash nazg durbatuluk, ash nazg gimbatul Ash nazg thrakatuluk agh burzum-ishi krimpatul
If you're cheap, you can get the cheapest new system you can find, rip out the cd drive to free up an ide channel, find a way to safely mount four 250gb drives in it (I'd recommend the 250gb seagate 7200.8 for longevity, but that's just me) to make a raid 5, install a floppy drive, and use the floppy drive to do a net install of debian. Then set up samba for windows file sharing. That 750gb of redundant raid 5 storage will set you back between $500 and $900, depending on if you already have a system in mind to use for the job. Or you can get a good server with a similar setup for around $1500 or $1800, depending on if it needs to be fast.
Or you can just delete the ISO's and TV episodes, if they're not worth the extra cost and you're never going to use/watch them again anyway.
Whatever drives you get, make sure you research their quality first, especially if you don't care for the extra cost of a redundant raid and/or backups.
Your harddrive will be nearly empty.
Or you could try a tape drive, but large DLT tapes can be expensive.
I saw an automatic DVD changer. Its a robotic library that can automatically select one dvd out of 200 and load it. Thats 800GB. Get a DVD burner and one of these babies. Got more money? Get a large chassis and just add 500GB hitachi drives as you need them.
So I guess eventually you're back to just buying the largest harddrives, the least expensive and most convenient thing next to cutting down on fat in your collection.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
First off, burn your shows to DVDs and your ISOs to CDs. Why have them on the hard drive if you aren't going to use them?
I'd set up an older computer system (P2 or newer or older..) with a RAID configuration. Something like RAID1 for simplicity and redundancy. Then you just store everything on that central server and your client machines don't matter.
Or get a bunch of chinese kids in your basement to memorize strings of 0's and 1's.
Death by snoo-snoo!
Seriously, 200GB is peanuts. My MP3 collection alone is 100GB. I also back up all my computers (including the one with the 100GB MP3s) to hard drive, so I have two copies of all this stuff.
I just bought one of these: LaCie Bigger Disk for my Mac, which is 1TB RAID for under $1000.
Just buy a TB or two, and if you fill that up any time soon, then, well, start deleting stuff. My personal files take up megabytes, my MP3s take up gigabytes, so 1TB is fine for me. 1TB ought to be enough for anybody.
I'm even thinking of buying another one of these, syncing the first one up to it, and then putting it in a safe deposit box and repeating every 6 months.
Re-evaluate what's on your storage. You sound like me - a bit of a hoarder.
Firstly, get a good catalogue system going. Put all your crap in one area. Get a sorted listing of creation time,last access time and categories. Get some hard-and-fast rules going as to when it should be archived offline. Go through once a month (or more often) and work out any new stuff that you need to backup. You do backup, don't you? Of course you do.
Then go through and check the last access times of your categories, and move to offline storage as appropriate.The advantage of getting this sort of regime going is that you've got more chance of having a backup offline somewhere when the inevitable happens and your drive wakes up dead one day.
For example, once you've got categories and last access times sorted:
- Digital Photos, family movies , documents - anything more than 12 months - Offline to DVD. Use par2 for archive copies (sent to distant relatives for storage), but make another set without par2 for normal, semi-random access.
- ISO's of distros? Got a broadband connection? Ditch them.
No broadband? Get them all offline, regardless of age.
- TV Episodes? If it's later than say, 3 months - off to DVD they go.
- Web Site content? 3 months.
And so on. Work the times out for yourself.
You really need a good cataloging system to help find out where the offline files are. Everyone's idea of a good catalog is different - Hell, I just label DVD's and keep them sorted by catagory - so I'll leave it to you.
If you organise your data and find that you still haven't enough drive space to keep all your current data online, then (and only then) go and look at the expensive options.
You are in a twisty maze of processor lines, all alike.
There is a lot of hype here.
We recently purchased a NAS from eRacks, which we are happy with & is slightly less expensive than the options you posted.
Ther are MANY tutorials on building your own RAIDed NAS on the net. Some have been slashdotted. Here's one. Google for others.
(the mobo doesn't have to be great, nore the ram)
Noooo, you don't need good hardware for a fileserver. Stability and data integrity don't matter at all. [/sarcasm]
Dimwit.
"The newly born animals are then whisked off for a quick run through a giant baking oven." --heard on Food Network
I buy drives four at a time, along with a 4 channel disk controller. I put them into Coolermaster 4-in-3 drive cages that have a big 12cm fan on the front (to keep the drives cool). I RAID5 the drives, then add them to an LVM VG which I use to divvy the space up into LVs as I need it. I started with 4x40GB drives about 5 years ago, then got another 4x120 and recently 4x250. I'm nearly out of space again, I'm just waiting for 300G drives to hit a nice price point before I grab another 4 of them. One luxury I have afforded myself is a second-hand, high end server motherboard with 4 PCI-X busses and 5 PCI-X (+1 PCI) slots, so the RAID rebuilds if/when the machine crashes don't overwhelm the machine and take half a day to finish.
save up the cash and put together a 2 terabyte SATA storage server (the mobo doesn't have to be great, nore the ram)
That all depends on how important your data is. High quality parts should be of top concern. Also, don't skimp out on the ram -- the more ram you have, the more the server will have to cache recently accessed files.
2 TB should be enough space. MAKE SURE YOUR NETWORK THAT THE SERVER IS CONNECTED TO IS 100 MBIT (preferably the client transfering files to/from as well). It might be a good idea to maybe invest in gigabit.
2 TB will fill up rather quickly, especially if the server is used to store high definition video (like HDTV). Also, 100 Mbit is a huge bottleneck -- that only gives you roughly 12 MB/s, which isn't enough to keep up with highspeed DVD burners or even play full resolution HDTV. Furthermore, if you're using ethernet, remember that usage above 60%, especially with multiple clients (say, two people pulling/pushing lots of the server at once), suffers from rather high latency and packet collision. Moving the same amount of data over a 1000 Mbit connection reduces collisions and congestion by 99% as data moves 10 times as fast down the wire. Using gigabit (at least when ethernet is concerned) is really the only option with decent performance.
Be relentless!
Heh. Kids.
I had a buddy who's dad told him, "You'll never fill up that 10MB hard drive." That is not a typo. Ten megabytes was a cavernous space at that time.
There are always more data.
-Peter
Ya, I don't get the main parent: "I keep filling up disks and buying bigger ones and I am sick of it. Can anyone recomend a slightly bigger disk that will permanently solve my problem?"
I'm a believer in segregating storage from the machines I'm likely to use/want to reload etc.
:)
And I've spent thousands of dollars on my home network and attendant PCs, to solve the problems that the original poster will only have if he manages to actually get enough storage for his needs.
Presently I have four identical storage servers, with the following characteristics:
Athlon64/3000, 1GB RAM, Gigabyte K8VM800 motherboards, 4 Hitachi 7k250s (RAID5 on 3ware ), 2 Hitachi 7k400s (soft RAID0, stores a daily snapshot of the RAID5, which is the data that is actually shared), 1 Samsung SP1614 Boot/OS drive, a 3Ware 8506-4LP, Intel Gbit NICs.
These machines run a series of scripts that collect and copy (pictures or MP3s) or move (video) whatever I happened to have dropped on my various workstations (each have between 300GB and ~1.7TB) to appropriate filesystems on the various servers (one for porn, one for ripped DVDs and TV shows, one for music, one for pictures); those filesystems are then exported via NFS to another Linux machine where the whole mess is presented back to all my machines as a single file system.
Getting enough storage is simply a matter of applying money. 250GB drives are quite reasonable nowadays and 160GB drives are downright cheap, but dealing with dinky little disks make getting enough SATA ports problematic. Sub-$100 2- and 4-port SATA controllers from the likes of Adaptec, Promise and Highpoint have their own problems. Most don't do online volume management and REALLY only do RAID through a driver, rather than an actual onboard processor. They're fine for storage expansion, a JBOD or RAID0 (note: RAID0 is normally a VERY stupid thing to do, since most people aren't doing STR-intensive things with their PCs and the chance of losing data is substantially higher than for any single disk), but as with everything else, you get what you pay for, and ports on a proper controller are probably worth more than the disks you're attaching to them.
RAID5 is also kind of a bad deal for write-intensive data - lots of little files that get updated a lot, while I'm at it. Do RAID1 or RAID10, (or maybe RAID3 if you can find a controller that supports it) for data you care about. Spend money.
USB2 and Firewire enclosures are NOT a good solution for adding primary storage most of the time. Generic enclosures frequently have difficulty with larger drives, and often have VERY cheap fans that either fail quickly or detriorate to the point that they sound like a penny in a vaccuum cleaner. Additionally, the performance and CPU utilization of USB2 enclosures both tend to be god-awful. Brand-name enclosures have a few different problems: many use 2.5" disks, which in my experience are rather delicate. Others are not properly cooled, and almost all of them are sealed enclosures. Better to put a drive inside a computer if possible. I tend to think of USB2/Firewire drives drives as backup devices only.
Disk-wise I tend to prefer Hitachi 250 and 400GB models, or Samsung 160s or 200s, and SATA over PATA when possible. The Hitachi 500GB models get too damned hot, and it's the only one that's out (available for purchase) at the moment. Seagate and Maxtor ATA products tend toward tepid performance, and in the case of Maxtor, quality hasn't been good since the Quantum merger in 2001. I will not purchase a Western Digital drive for any reason, but specifically I avoid the geek-favorite Raptor 360GD; I was party to the construction of a small 20-drive SAN using Raptors (client's spec, not my idea) a couple years ago where the drive failure rate was approximately one drive every 33 days.
-- I wanna decide who lives and who dies - Crow T. Robot, MST3K
Take a long hard look at what things you absolutely need, and toss the rest.
Not me. For two reasons. First, I've too often found myself wishing I had something that I had deleted and second, it is simply not worth my time to wade through it all and decide what I need and what I don't. Disk space costs well under 50 cents per gigabyte, and even after you add some redundancy (RAID), it's still very cheap.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Print a harcopy.
Where'd you learn math? 200 DVD's times 4.5GB (actually more like 4.4) per disc is only 900GB, less when you consider you can rarely fill the disc up completely due to odd files sizes. Plus most of those disc changer robots run to the thousands of dollars, you're better off dropping $6-700 on 4 or 5 300GB hard drives and a cheap PC to serve up files from them, I also suggest a seperate smaller drive (20-40GB is more than enough) to boot your OS off since software raid arrays aren't bootable. Put together a cheap system for $2-300 to strap the drives in and you've got over a TB of live networkable storage for less than $1000, spend a hundred more on a tuner card and install MythTV on the machine and you just got a massive PVR system as a bonus.
This thing, from http://www.infrant.com/, can be had for around $600 US without drives, which is not much more than it would cost to build your own. It lets you start with two mirrored drives in RAID 1, then add another drive or two, and it transparently migrates to RAID 5. Pretty cool.
:)
Note I have no financial interest in Infrant, I just want their products.
It sounds like you may be a digital packrat. If you are, I sympathize as I was one too after I got broadband. I stopped after realizing how much time/money I was wasting on crap that really didn't improve my quality of life. Now I buy new hard drives for my file server once every three years instead of every three months. Following are some of the things that helped me.
1) Download less porn/warez. Or put off downloading more until you've watched/run/played what you have. If you're just one person cranking through that much space that quickly, then you're downloading things just to have them. Stop that.
2) Go through and 'rm -rf' files and directories that you haven't accessed in a year. Don't keep obsolete versions of operating systems around, because you won't use them. As soon as you download a CD image, burn it and rm the ISO.
3) Archive on external media anything that's sentimental but rarely accessed.
4) Make it a routine to burn stuff to CD/DVD at least once a week. Eventually, you'll get tired of wasting time burning crap that you don't use and this will help you realize that you really don't need it at all.
5) If you do a lot of video editing or webmastering that requires huge amounts of data, and you're making money at it, then you need to invest in a proper server to keep all that. Be sure to make backups too. If you do this for a company, have them take responsibility for this.
6) Take a page from Linus's book: Upload it to an FTP server and let the world mirror it.
Cheap PC and RAID-5.
Assuming you already have a cheap PC, throw a SATA replication card in (Cheapest way to get RAID-5) and you have yourself an array.
Using cheap drives (WD 7200RPM 250GB has the lowest cost-per-gig here) you can get 1TB of total storage for for $488 USD. Of course to get 1TB in RAID5 that means an extra drive, so the cost for 1.25TB of total storage is $610 USD.
Of course if you don't have a PC to use, a NAS server ($130 USD), two USB enclosures ($25 each) and of the 250GB drives and you have half a terrabytes in unreliable storage on the cheap.
100 Mbit equipment is so cheap now, every PC that's at least 100 MHz should have it. If you have some older workstations, 100 Mbit cards may be hard to find, but these would have to be pretty old boxes to not come with 100 Mbit ...
That's not true if you've got a switch rather than a dumb hub, and switches are pretty much the norm now. Switched networks have almost zero collisions, no matter how busy they are. (Granted, they should have zero collisions, not almost zero, but switches do occasionally have bugsSure, latency does go up when you're pounding on the network, but on a 100 Mbit network it's not that much -- just a few miliseconds at most. And for moving lots of data around, latency doesn't matter much anyways -- it's throughput that matters.
Not quite. You won't actually see 120 MB/s with today's hardware, but you might see 20-30 MB/s, maybe a little more on really serious hardware, which is still a lot better than the 12 MB/s you'll get out of 100 Mbit ethernet. And it's relatively cheap now -- I recently got gigabit PCI cards for $5 and a 5 port gigabit hub for $20 at Fry's.Indeed, and what is more, a bigger disk only takes a bit longer to fill up, but it takes much longer for you to clean it up for getting some space available.
...if you haven't thought about it in five years, haven't used it in two years, or haven't seen it this past year, you should throw it out.
--
Don't like it? Respond with words, not karma.
As far as long-term, this will buy you quite a while (probably). The CoolerMaster Stacker case is pretty nifty, and employs 3-to-4 drive bay inserts to turn 3x5.25" bays into 4x3.5" bays and has a 120mm fan mounted directly on the insert for better drive cooling. Then, just throw mobo+proc+etc... into it and get a 3ware SATA RAID card (the 12-port 9500 series is a good choice, especially if you want good scalability). Then, I'd say make 1 (or two) 2TB RAID5 setups using 6 x 400gig Seagate SATA drives. I say Seagate mostly because they come with a 5 year manufacturer waranty (and I'm sure at least one drive will fail in that time period). I should also note that you'll probably need at least a (good) 500W power supply per raid, and you might want to get one of the adapters that allows redundancy for the motherboard power. That's another nice feature of the Stacker case: it has ample room for two power supplies. I hope that's of some help.
I'm with you on this issue. I've gone from 20MB way back when in the day to about 2TB of storage on my current network. I have 2 Desktops with RAID1 setups (2x200GB and 2x250GB respectively), a small server with RAID1 (2x120GB) and a few other assorted PCs with multiple different drives of varying capacities.
...the machine I am loving the most these days is the Buffalo Terastation (http://www.buffalotech.com/products/product-detai l.php?productid=97&categoryid=19). This thing rocks. 4x250GB you can configure anyway you want (I went for RAID5, 750GB of NAS storage), with gigabit ethernet, SMB, AFS, NFS and FTP. Runs some embedded Linux I believe. Also has 4 USB2 ports through which you can attach more disks for more storage. I think it can also print through them, but I don't use that feature (stupid old parallel printer) so I'm not 100% sure on that.
:) And before I get my ass roasted, I do pay for those games, I just like to play them with the convenience of a virtual CD and not have to keep shuffling CDs in and out of my computer. If I want to play the CD swap game I'll sit in front of my game consoles.
And I don't want to sound like an ad, but...
Coupled with an SMC 8 port gigabit switch I am flying through files and backups. And the damn thing ran about $900 from buy.com. They also have a 120GBx4 model and a 400GBx4, but the former's disks are too cheap for the expense of the surrounding hardware while the latter's disks are too expensive for their own good. The 250x4 seems to be the sweet spot. You can almost buy 2 for the price of a 400x4.
As for DVD storage like people are suggesting, I'd say screw it. I hate having to reach for something physical to find bits. These days I even try to keep my games as ISOs so I don't have to find a CD case. Of course all the new copy protection mechanisms suck, but hey, gamecopyworld is your friend.
-Jack Ash
First, I've too often found myself wishing I had something that I had deleted
Something that can easily be prevented by a well thought out short/medium/longterm storage policy.
and second, it is simply not worth my time to wade through it all and decide what I need and what I don't. Disk space costs well under 50 cents per gigabyte
The bigger the disk, the longer it takes before you get the dreaded 'disk full' message, but it will take exponentially longer to recover from it. In other words, you can't prevent the problem entirely, and having too much space will just maake for having to spend a lot more time on cleaning some of it when you really need the space and there is no shops open to get new disks or all your storage enclosures are already filled to capacity with the biggest disks you can get.
Take a look at 'man rm'. There all answers to Your question. If this solution does not fit Your needs, simply buy 1 or 2 >400GB HDD and USB/Firewire case for each, then You don't need any servers and can shuffle between hard drives 'live'.
It runs a really weird hacked up version of PPC Linux:
admin@TERASTATION:~$ uname -a Linux TERASTATION 2.4.20_mvl31-ppc_linkstation #15 Tue May 31 10:18:19 JST 2005 ppc unknown
An sshd went on there about 16 minutes after it was in the house. Mine's the low-end 4x160 model (not 4x120, as you state). Got it on sale at Fry's for $525. I would have bought two if I could have. But no worries there: when you run low on space, buy four 1.5 TB drives (after all, it's 2009, right?), put them in one at a time, and slowly rebuild your array. It's all XFS, so there's no real practical limit on disk/partition size.
The print server is nice. I have a Samsung QL 6050, it's parallel only. I spent $30 for the Belkin parallel to USB adapter cable and it works like a champ.
It's a very nice appliance. Now that I have ssh access, I have NFS exports. It replaced my home-built 80GB RAID1 box, and I'm glad that I have a storage "appliance" now. That old PC ran really warm and was loud. The TeraStation is very quiet and doesn't seem to get all that hot.
I wish it did remote logging out of the box, though. Also, I'd have liked to see disk quotas. But that's no big deal for a home user.
I'm definitely sticking with the TeraStation over home-built.
-B
Ash and Hickory, straight-grained and true, make excellent bludgeons, dandy for the cudgeling of vegetarians.
I have two OpenBSD systems (Compaq D5S SFF boxes), each with a 400GB hard disk. One sits at home in a segregated part of the network running Samba. This serves up files for a Powerbook (OS X) and my wife's Dell (Win XP). The other system sits at my best friend's house. Both systems allow SSH with public key auth only. I tunnel rsync over SSH from the master box at home to the other system to sync files. This happens once a day at 2am.
The master box at home has a cheap DDS4 drive which I got from eBay for about US$65. I get brand new tapes for about $4 a piece. I have a 4 set cycle on which I do a full backup every week. After completion these tape sets get taken to a different friends house and she keeps them for me. Incrementals during the week I keep hold of.
My Powerbook has a superdrive and I do a set of DVD+RW's every once in a while. I copy my critical files such as accounts etc additionally to a 512MB USB key, which is encrypted and I keep in my bag. That USB key contains the data that I simply could not do without.
As for storage needs, when I begin to outgrow my 400GB allocation and tape & DVD backups/archival becomes too time consuming, then I will move to a RAID 10 solution. However, for me this probably won't be until 2008. At that point I will probably do another home brew solution. I have started looking at what those options are. There are quite a few companies that offer rack mountable chassis with lots of storage bays. My idea was to put a stripped down version of OpenBSD booting from industrial grade CF card, similar in concept to how one might go about making a firewall device. I'd go for a good quality server motherboard and then add a well supported SCSI or SATA RAID Card. Seagate Barracuda drives have a 5-year warranty on them, so I'd probably go for some of those. I'd then add striped and mirrored storage as required as my storage needs expanded. This is probably going to be a fairly costly exercise, but I shall be doing an audit of my filesystem at some point this year to clear out the old files and pornography with which I have become bored.
The most important thing to do is work out what data is actually worth protecting. If like me you have ripped your Futurama and "24" DVD episodes into DivX for easy access and local copies whilst on the road, bear in mind that you can always re-rip them. If the house goes up in a puff of smoke, it would probably be cheaper to re-purchase the DVD set with the insurance money rather than keep archiving copies of stuff onto media which you have to continually protect, maintain and renew. The stuff I'm paranoid about is the data that I *can't* buy again - i.e. pictures of holidays, family and honeymoon, my accounts files and scans of receipts for my business, plus logs of chats and text messages.
Good luck.
Check out this free utility. You might be surprised how much stuff you DON'T want is lurking on your system. This will help you track it down.
I agree. Last week I set up such a system in my house using Suse and the setup was easy and it has worked flawlessly. I did the FTP install of Suse 9.3, set up the samba server and VNC in YAST with a few clicks and away I go, 240 gigabytes of storage in in time it took to do an FTP install.
delete most of your porn. It mostly looks alike anyway, so just ditch it. You should then have close to 200 GB free.
Some people are like slinkys. They're useless, but it puts a smile on your face to push them down the stairs.
SGI has a neat solution to this problem called DMF. It will automatically migrate infrequently-accessed files onto tape, and 'recover' them when accessed (takes a little longer than normal) automatically when there's a tape library, and I expect it will prompt an operator to load a tape if the tape isn't online.
I've no idea what the cost is, or if there's a low-end solution.
Max.
The bigger the disk, the longer it takes before you get the dreaded 'disk full' message, but it will take exponentially longer to recover from it.
Who cares? The last disk I added took three days to be fully on-line. But my investment in the process was about 30 minutes. It took 20 minutes to write the script to iterate through my RAID-5 volumes, run "pvmove" to migrate the data off of each volume, remove it from the volume group, rebuild it with the new disk and add it back into the group, then another minute or two each time I checked on the progress. In the meantime, my data was always on-line, and always safe from up to two (consecutive) disk failures.
The problem you describe is only a problem if you have lousy disk management tools.
Oh, just a nit: your use of the word "exponentially" is inaccurate. Even without LVM or the like, recovery time increases linearly with total storage, not exponentially.
there is no shops open to get new disks or all your storage enclosures are already filled to capacity with the biggest disks you can get
Yes, you could have a problem if you allowed this situation to arise. So don't do it. I keep a goodly chunk of my available space unallocated to any logical volumes, so when I actually run out of space, I simply run "lvextend" to allocate the space, and then set about getting more storage. The bigger problem is when there's no more room in the box for additional drives. The solution to that is simple: Don't let it get that way. Always make sure you have room for one more. When your box is almost full, and you need more storage, you have to either add another enclosure or, more likely, replace a drive with a bigger one. The scenario where they're already the biggest drives available is pretty unlikely. My file server can accomodate six drives so filling it up with the currently-largest drives would be 3TB of storage. That's a *lot* of home movies.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
"gigabit PCI cards for $5"
To hell with that; GigE can totally hammer the dinky PCI bus most systems have these days, you want something with proper interrupt moderation, 100% working checksum offload and decent DMA support; jumbo frames are nice too. Cheapy Realtek cards need not apply; spend a bit more on an Intel Pro/1000 or so, unless you like having 80% of your CPU dedicated to servicing interrupts and copying data around pointlessly.
PCI-E stuff seems much better, thankfully; makes on-board GigE on consumer level systems actually quite nice.
You did not understand what I said.
I meant recover from a disk full condiion, not recover from a failure or adding a new disk or whattever.
Tthere are momenets when a new/extra disk is not an option, ie, you need the space now, and there is no shop open, your storage enclosures are all filled up already etc.
The more capacity you have, thte longer itt takes to figure out how to make the inmediately needed space available.
See my signature. And that was seven to nine months ago, so it might very well be cheaper now, and/or possible to get more capacity with the newly-available 500GB drives.
Yes, $4100 is a lot of money. But I built it for exactly the reason you mentioned; I've always been of the philosophy of "do it once and do it right," and this way I've taken care of my storage needs for years to come.
I ran `cat /dev/zero | nc other-host 6666' with a `nc -l 6666 > /dev/null' running between a dual p3/700 and an Athlon XP 2600, with both commands run on both systems, and saw about 27 MB/s going in both directions at the same time. CPU usage on the p3/700 was around 40% of one cpu (and it was less on the faster Athlon) -- which is signifigant, but it was certainly still usable, and the data was flowing quickly enough.
Better cards may perform better, but the cheap cards aren't bad at all. They don't seem to support jumbo frames, however. But that's not essential, just nice. For an enterprise file server, you might want to do better, but for home? This is fine.
Oh, I had to recompile the kernel on the Athlon box to disable the interrupt acceleration stuff on that driver (r8169) ... otherwise, the Athlon box would hang under heavy network load until the cable was removed (then it would come back.) Obviously a problem with the driver/kernel somehow -- the p3/700 box (running a 2.4 kernel) had no such problem.
I eventually replaced it with an Athlon64 XP 3000 w/ a sk98lin card built into the motherboard. That one wouldn't work with the FC4 kernel at all (it was seen, but just didn't move any packets) but worked once I built my own kernel. A lot more trouble than it should have been, but ...
Tthere are momenets when a new/extra disk is not an option, ie, you need the space now, and there is no shop open, your storage enclosures are all filled up already etc.
Only if you let that happen. It's not hard to stay ahead of your needs.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Expanding your home costs serious money, and is a real hassle. It makes sense to invest time and effort to get rid of things that might even feel emotionally attached to, just so that you can avoid this.
Adding drive space is cheap and easy (USB storage can be added for well under $1/GB, takes a second to install, and will last for years). It would be quite easy, across multiple hard drive clean ups, to spend more effort organizing and deleting your hard drive content than it costs for you to simply add more storage.
If deleting items on your hard drive bothers you so much that you need to ask others how to proceed, it hardly makes sense not to spend a trivial amount of money buying more space.
Well, presumably they've improved somewhat since the cheapy chipsets I played with last year then; they could hardly have gotten worse ;)
:)
;)
At ~30MB/s I see something like 3% CPU use on my X2 4400+ and an on-board PCI-E Marvell; once I get my fileserver sorted out with PCI-X (surplus rackmount server) I'd hope to see more exciting transfer rates more in tune with modern HD's. Presumably similar could be said with two consumer-level PCI-E systems now; that makes me very happy
For comparison, on a 3200+ I see ~30% CPU use on an E1000 and ~80% on a Realtek using similar transfer rates, so you see where my hate for rl comes from
There is a limit to how much storage you can attach. That limit changes a bit depending on how much you are willing to spend. Staying ahead of your needs is simply not always possible.
Not to mention that noone can see into the future, all you can do is predict when your disks will be full based on current usage. That is never gonna give you an absolutely correct picture.
Hmmm, you are effectively writing from a zero latency unlimited bandwidth device on one machine, sending it over a GigE network to a computer that effectively discards it (another zero latency, unlimited bandwidth device) and only getting 27MB/s?
/. journal - I filled up a machine with memory and made a big ramdrive, copied massive files from that ramdrive on one machine to /dev/null on other machines over my GigE network and ran 60MB/s for a single pull from a single machine. I was able to pretty much peg the GigE switch (110MB/s, theoretical max being 125MB/s) by adding in other machines pulling down the same file from the 'server' and copying it to /dev/null.
Check back a few months in my
Consider putting a few more clients on the network and sending streams of data to each of them, if you are interested in your peak throughput from a single machine - you ought to do way better on your GigE network than that.
Disclaimer - I was using Windows networking, not Linux, in case that makes a difference.
Of course at the end of the day, real world constraints (ie, hard drive speed) limit our true throughput so 27MB/s or 110MB/s - doesn't matter as long as it exceeds the sustained throughput of your hard drive (which it does if you have more than one task happening on your network.)
Glonoinha the MebiByte Slayer
Back when I was your age sonny, I had this problem with disks filling up. My dad did not let me have many disks, and never got me a double density drive. For years I worked hard to keep all my data on my collection of floppies. Over time things grew to over 100 disks, but never as fast as I needed them.
As technology marched on, dad eventually bought a computer with a large 80 megabyte harddrive (Back when 40 was standard), which when combined with dr-dos's disk compression lasted a few months before I filled it. Then it was more more more, but it never happened.
When I got my own job I bought a 1.6gig harddrive (put in the same machine as the 80 meg harddrive), which I promptly filled.
After college I built a computer with a 4 gig drive, and then added a 9 gig drive latter. I never felt the need for more though, but cause not long after I bought that 9 gig drive I grew up and discovered that computers were not the only thing in life (though I still have not figured out how to get the girls).
So my answer to you is just have patience sonny, some day you will grow up, and then you will discover that you don't need more drive space.
...is something like thisa r-SC101NA.html (Use your favourite seller, this was the first link that I clicked on.) ... I can't find anyone who has them on stock. :-( ... could well be ISCSI though. Will know for sure as soon as I can get my hands on one.
http://www.pcsuperstore.com/products/G22301-Netge
It's a true SAN device, hence block access, which means fast. You can just add more devices as needed. If I remember correctly, you can "team" devices for better performance, or larger virtual drives.
The only problem so far
Also, I am not sure what the access from Linux situation is; I think I saw a mention somewhere that you need a driver
"Consistency is contrary to nature, contrary to life. The only completely consistent people are the dead." A. Huxley
And the hub involved is certainly low end as well.
And even /dev/zero isn't unlimited bandwidth -- ultimately, it's limited by cpu and memory bandwidth.
Really, the GigE switch's theoretical maximum should be 125 MB/s per direction per port. If you're benchmarking the switch itself, you'll want to pound on every port in both directions.And even your ramdrive benchmarking isn't terribly realistic. Most transfers will be from drive to drive, so you'll be limited by the speeds of the drives, though caches will certainly help.
But all in all, my points were that 1) gigE is cheap enough that just about anybody can afford it at home now, and 2) without high-end hardware all around, you won't see the theoretical maximums that gigE can offer.
There is a limit to how much storage you can attach. That limit changes a bit depending on how much you are willing to spend. Staying ahead of your needs is simply not always possible.
Unless your needs include very large amounts of video, I disagree. Disks are so large these days that it's pretty easy to stay ahead. When I can start ripping HD video from Blu-Ray disks, then you may have a point. Then again, perhaps fixed disk storage will keep up. Or not lag too far behind.
Not to mention that noone can see into the future, all you can do is predict when your disks will be full based on current usage.
That, plus keep a reserve. In practice, it works very well.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Unless your needs include very large amounts of video, I disagree. Disks are so large these days that it's pretty easy to stay ahead. When I can start ripping HD video from Blu-Ray disks, then you may have a point. Then again, perhaps fixed disk storage will keep up. Or not lag too far behind.
Or when your needs include saydoing multitrack audio recording? or you want a library of your music in a non-lossy format on your disks, and you happen to have a somwhat substantial amount of music?
Or you happen to be among those weird people who actually make backups of their machines to a more centralized place?
There are quite a lot of other siituations that I can think off and that are pretty realistic to occur that can end up requiring many terabytes of storage.
Or when your needs include saydoing multitrack audio recording? or you want a library of your music in a non-lossy format on your disks, and you happen to have a somwhat substantial amount of music?
No, I don't think audio does it. Consider that 1TB will hold over six *thousand* hours of FLAC-compressed CD-quality audio. Unless you're a professional recording studio, there's no way you'll have multiple terabytes of audio.
Video is different, but it still requires a *lot* to push the limits.
Or you happen to be among those weird people who actually make backups of their machines to a more centralized place?
I do that, actually. Although with Gig-E I'm preferring instead to migrate the storage to a central location. With a fast LAN there's no difference between local and remote storage and it's easier to manage when it's all in one place.
Still, for most, even if you're backing up a half-dozen machines, a couple of TB goes a very long way.
There are quite a lot of other siituations that I can think off and that are pretty realistic to occur that can end up requiring many terabytes of storage.
I doubt that. Not for home and even small office use. Professional video, audio and graphics work are a different story. But for those applications it's reasonable to spend thousands on network storage appliance devices which can provide massive amounts of storage.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
No, I don't think audio does it. Consider that 1TB will hold over six *thousand* hours of FLAC-compressed CD-quality audio. Unless you're a professional recording studio, there's no way you'll have multiple terabytes of audio.
First of all, I know quite a few people who do have that much music around, you obviously are not among those.
Second, I don't have to be a professional recording studio for doing things like uncompressed multitrack recording, one of the very cool things about modern computers and sound hardware is that high-end multitrack recording is available to any amateur band, and those just interested in playing with digital audio.
An hour of 16 track 48khz recording makes for a nice 4GB already. When doing a recording, its seldom right at the first take, and usually you end up with some 10 or more takes and then go on mixing them down and combining them. Have more then a single project? well..
Video is different, but it still requires a *lot* to push the limits.
Go try editing a raw video stream someday, you will see how wrong you are.
I do that, actually. Although with Gig-E I'm preferring instead to migrate the storage to a central location. With a fast LAN there's no difference between local and remote storage and it's easier to manage when it's all in one place.
There is a definite difference in responsiveness due to latency. Not important for many uses, very important for others.
On another note, unless you netboot all your machines, using your central storage is not going to help getting your computer running really.
doubt that. Not for home and even small office use. Professional video, audio and graphics work are a different story. But for those applications it's reasonable to spend thousands on network storage appliance devices which can provide massive amounts of storage.
You disregard home studios, people who like fuddling with video as a hobby etc etc.
You seem to be making the mistake that what you do not do at home is also not something others might be doing at home.
And then when I need something that's archived on CD or DVD I copy it to harddisk again. I just want stuff readily available, so instead of spending countless hours archiving back and forth I now just get that bigger harddisk and usually by the time I run out of space, common/affordable harddisks increased in size plenty. It takes very little effort to move data to a bigger harddisk.
I only backup stuff I really don't want to lose on DVD (which in my case is mainly smaller sized files like docs and pics). Just in case my harddisks all decide to die on me at the same time.
Sample this!
In August 2004 I purchased a LaCie Big Disk Extreme 320 GB striped RAID array. Very nice, extremely fast performance on my Power Mac G5, but it failed 3 days outside of the one year warrenty. So what happened? One of the two hard drives in the unit stopped working. LaCie attempted to repair the drive by replacing everything but the two drives inside the unit with out success. DriverSavers.com quoted $1500 - $4500 for data recovery while OnTrack.com quoted $2500 - $3500 for data recovery. Since my thesis is on this drive I must get it back, Ontrack.com is going to get my business.
The fact is with these external drives in close-packed metal enclosures is that they typically don't have adequate cooling, which is the likely reason for the failure. Your best bet is to get your own high capacity hard drive and purchase a FireWire/Serial ATA/USB 2.0 enclosure WITH A FAN and build it yourself. From my experience, off the shelf commodity PC components will work with all modern (1999+) Macintoshes without special software/drivers, etc...
Well if collecting is you're sport you can start again try: del *.* Hmm you might burn the family photo's on CD first before you try this magic trick, otherwise you're family might get a bit angry.
I know you're out there. I can feel you now. I know that you're afraid. You're afraid of us. You're afraid of change.
I think you're a bit of a horder. I mean come on, who keeps TV episodes on his hard drive? Actually I've got CloneWars, but that's different!
Mind you I've just bought 2x200Gb drives for my fileserver, as the old 40+160Gb was not enough for DVD rips and a load of VMWare sessions.
My routine is:
1. download/rip etc. the files and store them on the hard drive.
2. If I've not used a file in a month:
2a. If I think I may need to lay my hands on it quickly (like ghost images and fedora iso's), bung it on the 80Gb Firewire drive.
2b. If I don't think it's essential, burn to DVD/CD (this also prevents people stumbling on your pr0n collection when browsing your machine!)
Compression's not an option these days really as most stuff you download is already compressed anyway.
The only stuff I keep forever on the fileserver or local drives is really important stuff (company docs) or really big stuff (like yum repos) but that's also backed up of course.
#include <sig.h>
My $7 Realtek 8169's from Fry's are a lot more reliable and in fact slightly faster with lower CPU usage than the Intel Pro 1000 CT I have.
Although that might be due to crappy Linux drivers for the Intel (it hard crashes the box if you throw too much data at it).
Where I used to work we had a dual 3.2GHz Xeon with 2Gb RAM and a $4000 PCI-E gigabit card with fibre and that could not make much over 600MBps (the Realteks got about 220MBps on a regular PC/CAT5e).
#include <sig.h>
ascii art
Makin' money, makin' friends, makin' whoopee and wearin' Depends
on a windows box?
windows movie maker is your friend, drop a movie into moviemaker, check the resolution of the movie.. then save your movie in the smallest file size that keeps those dimensions.. chews up a lotta CPU cycles, but can reduce filesizes by half..
I drop most of my videos right off, quality does suffer, but- still very viewable. I just dropped a 1.06 gb download into 602 mb...
every day http://en.wikipedia.org/wiki/Special:Random
I have over 2 terabytes on my LAN, and would never dream of buying an HD less than 400G nowadays. I also have about 5000 CD-Rs and 2000 DVD-Rs full of offline-content. Everything is indexed in greppable text files.
There is no escape. Drives become full, and then you must offload files as fast as you add new ones. For me, this is 5-10G a day.
My point is: There is no escape. You can't win, you can't break even, and you can't quit. Don't futz around with uber-expensive "storage hardware"; everything is just a more expensive harddrive. Backup your data to disc, then you wont need a RAID array. Most of my 2 terabytes on my LAN is "ready to delete" in case of a real space crisis.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
You can think of buying hard drives as an investment, and see it as a never-ending buying process, or think it as a lease:
Don't jump on the great new big drive, stick with the better $/G ratio (250G or 320G combinations these days).
I evaluated some RAID, big drive, external multi-drive systems to "definitively solve the problem" and finally sticked with that "see it as a lease" way of thinking.
After some years with that model, I spent around $300/y for as much space as I need (from 100G in 2002/2003 to 1,5T now). The first 100G cost around $300, as the last 500G!
I know it's difficult to shift to such a mind-state as we think of computers as investments, but just think of those fantastic 10GB SCSI drives that were so big, so fast and so expensive at the end of the 90's.
ClaudeBBG
get two, delete your older p0rn, and move on.
If you can't afford the storage, you don't need to save.... pretty simple if you take a step back and look at the situation.... Like the yocal in Mad Max said, "Speed is just a matter of money, how fast do you wanna go?" You gotta pay to play. sorry for the reality check.
Sig Hansen?
Use PaperDisk! http://www.paperdisk.com/aboutpd6.htm Go to Staples, and get this: http://www.staples.com/webapp/wcs/stores/servlet/S taplesProductDisplay?storeId=10001&catalogId=10051 &langId=-1&productId=119926&cmArea=SEARCH
There you go... 2GB for $3.84!
http://www.rayn.net . Funny. Stuff.
...then you will need to buy a RAID storage solution. Yes, this may be expensive, but it works great, as far as not failing all the time, and also it will be a good investment, as if you was to buy, say a 200GB hard drive every 6 months, it would cost more than RAID. As it sounds, I would suggest RAIDing the storage to about 2TB. That should last you for a while!
OS ISOs? When do you need that OS disk so badly you can't wait an hour or two to have it downloaded? And you probably want to get the latest version off the net anyway, so delete those. The same goes for movies. If you have seen it twice or more delete it.
My system consist of two 100Gig drives mirrored on my home server for documents and photos, and a 300Gig drive for ISOs, moves and music that i listen to regularly.
I download a lot of movies and games from P2P-networks(poor student, but with great 5Mbit broadband :) and have never had any trouble with disk space at all.
Get a Qualstar library. It's fun to watch the arm grab and load tapes for about 10 seconds. :)
The nomadic paradigm, or how I learned to free myself of geography and why this matters more than ever to me. I've been thinking about this issue for a while, and I can't think of a better place to ask the question than on this thread: Why would I want to build out data storage for my home when I want to go virtual? Maybe it's just because I'm a New Yorker. For a long time, I've been trying to go virtual: online mail, online backup, online everything. Since the mainstreaming of cell phones and Web services, very little tethers us to one location. As a New Yorker, I've been thinking a lot about mobility, especially since 9-11, the tsunami, and most recently hurricanes Katrina and Rita. What do I need if I want to pack and need to move to another town on a minute's notice? And it got me thinking about communications, transportation, delivery of goods and services. If any of us had to pack up and go, what would we need? What helps us become nomads, not extend roots deeper into our house? Here are my must-have nomadic goods and services: >>Communication: Cell/Treo/Laptop >>Financial: Cash/Amex card/ATM card >>Work/Collaboration: Webex/salesforce.com/Gmail >>Entertainment/Leisure: Netflix/eBay/Amazon.com With these, I can pretty much live and work anywhere. What am I missing?