Slashdot Mirror

← Back to Stories (view on slashdot.org)

Bulk Data Storage For The Common Man?

Posted by timothy on from the the-common-man-with-a-tb-a-month dept.

Vigyaan writes "Lately, I have been looking into different bulk data storage options available to a common man. My work depends on generating, storing and analyzing a large amount of data -- averaging about 1 TB per month. I would like to have a storage system which is automated, fast, reliable and most importantly does not cost the price of an eye. Right now, I have a 4 node Linux cluster with 10 large hard disks (total capacity 1.6 TB); data storage roughly costs about $0.60/GB (excluding the cost of PC hardware). But long term storage is painful -- DVDs cost about $0.10-$0.15/GB but takes too much human time and leaving data on hard disks makes me nervous because of possible failures. RAID is a possibility, but it increases the cost significantly. I was wondering, if Slashdot readers have any recommendations for a cheap automated way to store and retrieve data."

3 of 483 comments (clear)

  1. age old problem... by Lumpy · · Score: 5, Insightful

    Ahh the large amount of data that has X value versus a storage solution...

    If your data is worth $20,000.00 then a $2000.00 solution is dirt cheap.

    what is your data worth? that is where you need to start and then look at the 10-30% of the data's value to start looking at how must to spend on it's storage.

    If 1 month's data was lost forever, how much money would it cost the company? that is your actual $ amount that you should be shopping at.

    and that is how I got the company to buy a $20,000.00 1000 tape DLT jukebox.

    my data is worth over $100,000 a month and is much lower than yours is size.

    That is where you need to start. Justify your storage costs by figureing out what it is worth to begin with.

    --
    Do not look at laser with remaining good eye.
  2. Do what Google does by glinden · · Score: 5, Insightful

    Build yourself a cluster of cheap boxes with cheap IDE disks and replicate your data across them. Because the data is replicated across your cluster, no need for backups or RAID.

  3. options options, what is your time and data worth? by segfaultcoredump · · Score: 5, Insightful
    Lets see.... hard Drives are running about $0.50 per GB, DVD's are running about $0.06 per GB (100 pack, "house brand", not something I'd put my data on but this is slashdot, and there are idiots out there who think that it is a good idea), and tapes are also running about $0.20 -> 0.50 per GB (for the DLT/AIT/LTO type, the ones that have enough capacity to not drive you nuts)

    So, you can put your data on 4-5 HD's, 10 tapes or 232 DVD's per month. The Cost of doing so will be about $500 per month for the tapes or HD's and $50 for the DVD's (assuming your time cost $0)

    At work, we had a need to keep a few TB of data online permanently, so we purchased a few NexSAN ATABeast's. At $50,000 for 10TB of usable storage ($5/GB), they may be a bit out of your price range. The advantage is that you can hold almost a years worth of data and it is protected by RAID5. It also makes management a lot easier, since it is very difficult to mount 42 300G drives in a single chassis (and it takes only 4U of rack space).

    On the low end, NexSAN has the ATABoy2 or ATABaby (2TB or 1TB) for the $8-$15K range. This will let you hold a months worth of data

    On the high end, You have EMC disk arrays (Think upwards or $20+/GB for the 'cheap' stuff from them.

    Overall, if you have 1TB per month, you need to either a) get a grant to fund your work, b) hire somebody to swap DVD's for you or b) seriously rethink your data generation.

    Any of the "cheap" storage methods have serious drawbacks, and the low cost ones are, well, not so low cost if $15,000 sounds like a lot of money to you.

    otherwise, good luck