Slashdot Mirror
Long-Term Storage of Moderately Large Datasets?
hawkeyeMI writes "I have a small scientific services company, and we end up generating fairly large datasets (2-3 TB) for each customer. We don't have to ship all of that, but we do need to keep some compressed archives. The best I can come up with right now is to buy some large hard drives, use software RAID in linux to make a RAID5 set out of them, and store them in a safe deposit box. I feel like there must be a better way for a small business, but despite some research into Blu-ray, I've not been able to find a good, cost-effective alternative. A tape library would be impractical at the present time. What do you recommend?"
I don't think you can beat a bunch of conventional hard disks in a RAID5 for both cost-per-TB and backup/restore performance, not to mention medium-term data integrity. Might be able to make hooking up the drives more convenient with an eSATA mult-bay enclosure, but those are kinda expensive. But I bet your backup box already has some sort of hot-swap on it already, like: http://www.amazon.com/Thermaltake-BlacX-eSATA-Docking-Station/dp/B001A4HAFS
I assume you already compress your data, since scientific datasets tend to compress well. You might consider compressing to squashfs, since it will let you do transparent decompression later on so you can skip the restore step if you just need a handful of files.
LTO tape, properly stored, will outlast burned optical media and hard drives. Great stuff and designed specifically for what you're talking about.
http://en.wikipedia.org/wiki/Linear_Tape-Open
Learning HOW to think is more important than learning WHAT to think.
It can get a little pricey for huge datasets, but Amazon S3 now has an option where you can ship your data on a big set of disks directly to them, they will import everything into S3, and it will live there forever. The nice thing about S3 is unlike physical disks, it can grow essentially forever, and comes with retention and redundancy guarantees. And once your stuff is in S3, you can recycle the same disks to mail them more data.
Because Amazon can be *expensive* compared to doing it yourself ($$$ for data in, $$$ for data out, $$$ for monthly storage). But heh, what do I know. I just manage the storage for one of the LHC detectors (5PB spinning disk, 17PB tape). Amazon is good when you've got VC money or have no IT folks.
You misunderstood the post. He needs 2-3TB PER CLIENT, not 2-3TB total.
you make the assertion that a tape archive would be impractical, but really it is the most practical solution. the drive will set you back a couple thousand, but 800 gig tapes are only around 40 bucks each, and they are engineered for data storage unlike hard drives. this will only cost $160 per 3 gig dataset, or 200 if you use par2 files and an extra tape to make it recoverable in case a tape does fail.
Snowden and Manning are heroes.
Tape is probably your best option. You can buy at DAT-5 (or even a DAT-4) tape drive for not very much. The tapes cost about $10 to $30 each (depending on what tape option you choose). Make 3 copies of the data set, store one onsite, store another offsite in a secure/climate controlled facility and send the 3rd to the client. Buy a spare tape drive and use both to make writing across tapes easier. There is a wide variety of software to write to the tape; we use the aging Retrospect.
The disk options is just way too complex; if anything, skip the RAID option and just store 2 copies. Putting the RAID sets back together and finding the RAID software will be nearly impossible in a couple of years. Use some standard formatting on the drives (FAT, NTFS, etc.) and you'll be good to go for the next 15 years.
I work as a contractor for the USGS and the projects I've been involved with host, archive and provide means for customers to access all our different satellite data products. We've got a Long-term archive method for tons of data products (digitally and tangible) and I can honestly tell you the first thing that always comes up is: how often will the data need to be accessed?
For the longest time (almost a decade) we used 3 big, STK tape silos for data archive and retrieval for custom orders. The problem behind that type of design is we used a archive in a completely wrong manner in the fact that we tried to use it as a archive and a quasi-online retrieval system into a caching filesystem. We had tape mount counts in the hundreds and thousands, constant mechanical tape issues because of the excessive use, ect. We actually decided to move it all to online storage using enterprise RAID (EMC Clarion) and moved to a small LTO-4 tape unit for almost permanent, maybe-once-in-a-great-while storage and the rest we leave completely on spinning disk and control the access to it via application layer network protocols as needed.
IMHO, I really think it's going to depend on the access frequency of your data. If that custom needs their data once, and maybe never again in case they lose it, put it on tape. If it's a requirement they can get the data from you any time they want and you've got the hardware and administrative resources, power and bandwidth, put it some RAID.