Slashdot Mirror
Long-Term Storage of Moderately Large Datasets?
hawkeyeMI writes "I have a small scientific services company, and we end up generating fairly large datasets (2-3 TB) for each customer. We don't have to ship all of that, but we do need to keep some compressed archives. The best I can come up with right now is to buy some large hard drives, use software RAID in linux to make a RAID5 set out of them, and store them in a safe deposit box. I feel like there must be a better way for a small business, but despite some research into Blu-ray, I've not been able to find a good, cost-effective alternative. A tape library would be impractical at the present time. What do you recommend?"
I don't think you can beat a bunch of conventional hard disks in a RAID5 for both cost-per-TB and backup/restore performance, not to mention medium-term data integrity. Might be able to make hooking up the drives more convenient with an eSATA mult-bay enclosure, but those are kinda expensive. But I bet your backup box already has some sort of hot-swap on it already, like: http://www.amazon.com/Thermaltake-BlacX-eSATA-Docking-Station/dp/B001A4HAFS
I assume you already compress your data, since scientific datasets tend to compress well. You might consider compressing to squashfs, since it will let you do transparent decompression later on so you can skip the restore step if you just need a handful of files.
And optar:
http://ronja.twibright.com/optar/
You know it makes sense.
Deleted
Unlimited space with several accounts.
It might not be the cheapest option, but with Amazon's AWS, you can snail mail them a copy of the drive with the data and they're store it in S3 storage buckets.
Hard drives are ridiculously cheap these days, especially for how much data you are storing. You may wish to consider buying drives from different manufacturers but of the same size to put in a single mirrored set. This way if there is a problem with a particular batch of drives it won't ruin everything.
LTO tape, properly stored, will outlast burned optical media and hard drives. Great stuff and designed specifically for what you're talking about.
http://en.wikipedia.org/wiki/Linear_Tape-Open
Learning HOW to think is more important than learning WHAT to think.
It can get a little pricey for huge datasets, but Amazon S3 now has an option where you can ship your data on a big set of disks directly to them, they will import everything into S3, and it will live there forever. The nice thing about S3 is unlike physical disks, it can grow essentially forever, and comes with retention and redundancy guarantees. And once your stuff is in S3, you can recycle the same disks to mail them more data.
With the advent of 2TB drives, you could easily combine 3 of these with software RAID 5 as you suggested. Depending on how long you need to keep the data, recording them to dual-layer blu-ray disks might be a better solution. Ya, it's a lot of disks (you can buy 100GB discs now), but they'll last longer and you don't have to worry so much about mechanical failure or needing a certain OS when you want to restore them.
Sabre
You might look at a www.drobo.com as a set of 4,5 and 8 drive enclosures. 1 TB disks gives you 3 TB usable space with a 2 drive failure tolerance. I have the older 4 bay drobo (2 for myself, and 2 at separate clients offices). It is much simpler to use, and will scale to your 2-3tb use and allow mismatched drives that normal raid will not use. Get a enclosure to start with, and then financing permitting, get a 2nd for Drobo redundancy. Not the fastest or cheapest, but reasonably good by both accounts, and simple to use.
Because Amazon can be *expensive* compared to doing it yourself ($$$ for data in, $$$ for data out, $$$ for monthly storage). But heh, what do I know. I just manage the storage for one of the LHC detectors (5PB spinning disk, 17PB tape). Amazon is good when you've got VC money or have no IT folks.
Magnetic disk are not a permanent storage, AFAIK not even for medium, they break, when they mustn't. Optical such as blueray kept in a really safe place; hidden from light, humidity and temperature variations can last much longer. Hard Disk failure may always occour, i had a raid 1 at home.. both disk failed at the same time... You may even do something good such as compressing your data and adding some disks for additional safety with some kind of error correcting code spreaded in them.
Label it something like complete american idol blueray collection and upload it on p2p to piratebay. every couple years rename it to some other horrible popular tv series. It will be self sustaining form of storage with infinite number of redundant hosts.
I would use RAID6 not RAID5, since 2 drive failures means data loss with RAID5, while it takes 3 drive failures to loose data on RAID6.
Linux MDADM has supported RAID6 for years, it's stable.
I would mix and match drives, not buying all the same model from one maker. One Samsung, One WD, One Hitachi, One Seagate.
That gets you 4TB in 4 drives, and unlike a RAID1, any 2 drives can fail with no dataloss.
You can further ensure no dataloss by making a second copy using different brand drives for each clone.
Eight 2TB drives is around $1500. Not bad for a very safe 4TB backup.
you make the assertion that a tape archive would be impractical, but really it is the most practical solution. the drive will set you back a couple thousand, but 800 gig tapes are only around 40 bucks each, and they are engineered for data storage unlike hard drives. this will only cost $160 per 3 gig dataset, or 200 if you use par2 files and an extra tape to make it recoverable in case a tape does fail.
Snowden and Manning are heroes.
Ok, yes, we see you know a lot about this.
So what's your recommendation?
Who wants to burn over 100 discs per client? I guess they make automated disc burners, but that's a little nutso. Plus, with that many discs you have a high chance of failure, so you'll need some kind of ecc scheme (parity discs?). I'd also have to vote for hard drives, although I agree with the shortcomings of that solution.
Cheaper how exactly? Even if you could get BR discs at $2 each, it would cost $80/TB, and I havn't seen BR discs even close to that cheap. That doesn't include the writer which I belive are still $400. For the cost of the writer alone, you could purchase 5TB of HDD.
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
Huh, don't you see has has Too Much to Do?
Tape is probably your best option. You can buy at DAT-5 (or even a DAT-4) tape drive for not very much. The tapes cost about $10 to $30 each (depending on what tape option you choose). Make 3 copies of the data set, store one onsite, store another offsite in a secure/climate controlled facility and send the 3rd to the client. Buy a spare tape drive and use both to make writing across tapes easier. There is a wide variety of software to write to the tape; we use the aging Retrospect.
The disk options is just way too complex; if anything, skip the RAID option and just store 2 copies. Putting the RAID sets back together and finding the RAID software will be nearly impossible in a couple of years. Use some standard formatting on the drives (FAT, NTFS, etc.) and you'll be good to go for the next 15 years.
I didn't see you mention LTO or DLT in your little rant against tape, both are superior to any other method for the storage of bulk offline data.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Repeat never use DVDs as long term storage. I have seen them go unreadable anywhere from 2-5 years. I have fired up disk drives 10 years later with no problems. They are cheap reliable and fast. Don't try and get fancy just compress and store data sets over multiple volumes. Don't use RAID.
I already use S3 for some things. Unfortunately it would be about $500/month/customer's data on S3 right now, so that's out. I could buy a whole computer with hard drives every month for that.
Error 404 - Sig Not Found
I think his/her recommendation was: 17PB Tape.
Probably the giant rare earth magnets I stored in the next box over.
You're welcome. :-D
Check out my sci-fi/humor trilogy at PatriotsBooks.
how about a cluster file system like http://www.moosefs.com/ , where it works well on commodity or even end of life by current standards of hardware. Redundancy is achieved through several chunk server nodes in the system. Performance and size can be dynamically scaled over time by adding more nodes or disks to the system.
Their project is GPL licensed, requires very little effort to set up, and likely much less effort over time to maintain and administer than a RAID or Tape system likely would, with the benefit of being able to choose what data is online or offline (by its presence on chunk servers being connected at the time).
I work as a contractor for the USGS and the projects I've been involved with host, archive and provide means for customers to access all our different satellite data products. We've got a Long-term archive method for tons of data products (digitally and tangible) and I can honestly tell you the first thing that always comes up is: how often will the data need to be accessed?
For the longest time (almost a decade) we used 3 big, STK tape silos for data archive and retrieval for custom orders. The problem behind that type of design is we used a archive in a completely wrong manner in the fact that we tried to use it as a archive and a quasi-online retrieval system into a caching filesystem. We had tape mount counts in the hundreds and thousands, constant mechanical tape issues because of the excessive use, ect. We actually decided to move it all to online storage using enterprise RAID (EMC Clarion) and moved to a small LTO-4 tape unit for almost permanent, maybe-once-in-a-great-while storage and the rest we leave completely on spinning disk and control the access to it via application layer network protocols as needed.
IMHO, I really think it's going to depend on the access frequency of your data. If that custom needs their data once, and maybe never again in case they lose it, put it on tape. If it's a requirement they can get the data from you any time they want and you've got the hardware and administrative resources, power and bandwidth, put it some RAID.
I've been doing backups by copying everything to my unlimited drive for years now. It's amazing - it never fills up!
/dev/null if using Unix).
Just type
copy Edit.* NUL
at your command prompt (or cp *
One day I'm gonna look to see how much data I have in that damn thing!
Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
Feel free to ask more questions.
Almost forgot to add. Never pay for expensive disk systems. Put the intelligence into your application instead. It'll scale faster and much cheaper. You also aren't locked into a technology (and instead, can enjoy the falling costs of storage, both spinning and SSD).
If the setup is raw files on tape (the stuff I've read in from reels recorded in the 1980s) or tar (stuff from the 1990s) it's not a big deal at all. It's only two-bit closed source single developer efforts on MSDOS that are difficult for anyone to read if they have the hardware.
With recent backup systems like AMANDA you can just dump a file with dd or similar and the instructions on how to deal with the data are there in the header in ASCII! It couldn't possibly be easier.
Also the "proper environment" is a lot easier to achieve than with storing hard disks, putting them in boxes in a warehouse without a leaky roof is most of the way there. Those tapes from the 1980s I'm talking about were exposed to 30C temperatures and high humidity every year, which is not ideal. Theoretically they were spare copies that were only made for transport and only had worth after the originals and copies were mistakenly thrown away by the clients.
The "people" problem is very easy to deal with. You make a lot of people responsible for backups of the department they are in, and make it VERY easy for them to do the backups. Somebody in small department X knows to put a tape in a few times a week, and so on, so you get a lot of people that know for certain that there are backups and how old they are. When the inevitable happens and things get lost or deleted there is no longer the panic of tracking down the IT guy and asking lots of questions whether something is lost or not. Instead of spreading panic you just get mild annoyance that it will take so long to get it back.
Also when one person leaves there are still many that know the system.
Of course using rsync to a machine elsewhere to get a copy of everything is useful, but that is not a backup, it's a copy, so it shouldn't be the only option. With things like that you eventually purge old deleted files or write over old versions of files, while the tape from two years ago has what was on there two years ago.
A problem I have here is the definition of 'long term'. To each of us it means something different.
In my job I have to archive 1.6 terabytes of data per day, and keep it around for 45 days (which, BTW, is not my definition of LONG TERM). For this task I utilize Data Domain storage, which utilizes data deduplication techniques for massive compression.
What you find is that at the block level your data may in fact be incredibly deduplicatable. In my case it very much the situation. I am currently storing 86 terabytes of rolling archives within 2.5 terabytes of physical disk space.
The problem with any technology you use for 'long term' storage is the ability to read those archives later. Assuming the media doesn't self degrade inside of the time frame you call 'long term', you must have the tools to read that media again. If you use BluRay, then you must store a compatible drive with it. (Nothing says Sony will not change the standard in two years and make all current drives obsolete, so no one makes them any more). Tape is worse, in that in two major model revisions, drives wont be able to read your media because its density is to low for the new drive head technology. Hardware based disk raid has the issue that the controller the raid was built with needs to stay with that raid. Another controller from the same manufacture, with the same model number, but a different firmware revision may not be able to figure out the raid, and declare the drives empty. Software raid is a little easier to deal with as long as you keep a copy of the OS you used to create it with in the same box. But then, during your defined 'long term' period, will you still have access to a system you can even plug these drives into, or run the OS on?
What you end up dealing with in reality is that as an archivist, you either ignore these facts, or you invest in a constant media / technology refresh and spend large amounts of time keeping your archives on the latest storage available.
Of course, all this falls apart if your definition of 'long term' isn't as long as some will project. In my case, my archives roll over every 45 days. I could easily keep that data alive for years on a live piece of hardware with a service contract. If I do not trust that hardware enough, I can buy two and replicate between them. (which, actually I am, for disaster recovery purposes)
With deduplication my (acknowledged) high initial investment quickly outweighs the cost of single purpose drives holding one copy, and wasting unused space. My purchase cost was less then $60k, but if I had to store all of that data in its raw form, my costs would be in the millions. However, if the data is not deduplicatable, then of course it is a moot point.
Each answer has it flaws. You decide which risks are acceptable, plan your best to deal with obsolesce, and define your definition of 'long term'. You also have to be ready to change your solution, when the one you choose today, fails to be the right solution for your needs in 5 years.
Why distributed FS and not something like live mirroring/shadow copy? I wonder also... what do you consider an "expensive disk system"?
We played around with DIY JBOD a bit (i.e., moving the complexity up into software) because it seemed a lot cheaper, but we have yet to get the thing to operate as reliably and simply as our fibre channel RAID units. The main problem we're running into is that for SATA to be practical, you need to multiplex several SATA disks onto single SATA ports, but that software support for doing this (at least in Linux and Solaris) is terrible. SAS interconnects are obviously better in this regard, but the hardware doesn't end up being any cheaper than standard fibre channel RAID, so you end up spending the same amount of money but doing more work because the spanning/striping/whatever is not managed for you. Also, under Linux, we've been somewhat dissatisfied with mdadm compared with something like OpenBSD's bioctl (which is easy to use and AWESOME). Sadly, OpenBSD is severely limited in some other areas that make it inappropriate to manage large disks (very large filesystems are not possible... fsck is prohibitively expensive with large volumes... etc...).
The other thing I've been wondering, and it is unclear to me whether distributed filesystems solve this is: it seems like they're designed to spread data around on many nodes. Are they also good for allowing access FROM many hosts? E.g., our StorNext system allows us to plug any client into a fibre channel switch and simply use the disk without really having to think through what that means. But StorNext limits us because we can't plug, say, an OpenBSD machine into it (proprietary drivers, no client software) and that licensing can get expensive fast. StorNext's documentation also leaves something to be desired, and their salespeople do not inspire confidence (none of them could tell us how to actually install a license on a machine... I had to figure it out myself).
Why distributed FS and not something like live mirroring/shadow copy? I wonder also... what do you consider an "expensive disk system"?
Distributed file systems are great because your limitations are (almost) always going to be hardware. Want 1000 boxes serving up your content? Get 1000 commodity boxes with disk. Need 10000? Also, not a problem. A box filled with raw disk is WAY cheaper than an EMC, Nexsan, etc (i.e. expensive disk system).
Serving over Ethernet should be fine, as you can always bond network connections together to increase throughput from your storage boxes to whatever boxes are processing the data (or even process the data *on* the storage boxes). I've bonded 8 gigabit ports together on some of our storage boxes without problems (Linux Kernel 2.6). I'm not a fan of fiber solutions. They're expensive, and I can get gigabit networking gear cheaper.
In short, stick to commodity hardware and open source distributed file systems. I know you're a little far along for that, since we had the opportunity to build from the ground up.
But your storage only has to last until December 2012.
Table-ized A.I.