Slashdot Mirror
Long-Term Storage of Moderately Large Datasets?
hawkeyeMI writes "I have a small scientific services company, and we end up generating fairly large datasets (2-3 TB) for each customer. We don't have to ship all of that, but we do need to keep some compressed archives. The best I can come up with right now is to buy some large hard drives, use software RAID in linux to make a RAID5 set out of them, and store them in a safe deposit box. I feel like there must be a better way for a small business, but despite some research into Blu-ray, I've not been able to find a good, cost-effective alternative. A tape library would be impractical at the present time. What do you recommend?"
I don't think you can beat a bunch of conventional hard disks in a RAID5 for both cost-per-TB and backup/restore performance, not to mention medium-term data integrity. Might be able to make hooking up the drives more convenient with an eSATA mult-bay enclosure, but those are kinda expensive. But I bet your backup box already has some sort of hot-swap on it already, like: http://www.amazon.com/Thermaltake-BlacX-eSATA-Docking-Station/dp/B001A4HAFS
I assume you already compress your data, since scientific datasets tend to compress well. You might consider compressing to squashfs, since it will let you do transparent decompression later on so you can skip the restore step if you just need a handful of files.
And optar:
http://ronja.twibright.com/optar/
You know it makes sense.
Deleted
Unlimited space with several accounts.
It might not be the cheapest option, but with Amazon's AWS, you can snail mail them a copy of the drive with the data and they're store it in S3 storage buckets.
Hard drives are ridiculously cheap these days, especially for how much data you are storing. You may wish to consider buying drives from different manufacturers but of the same size to put in a single mirrored set. This way if there is a problem with a particular batch of drives it won't ruin everything.
LTO tape, properly stored, will outlast burned optical media and hard drives. Great stuff and designed specifically for what you're talking about.
http://en.wikipedia.org/wiki/Linear_Tape-Open
Learning HOW to think is more important than learning WHAT to think.
It can get a little pricey for huge datasets, but Amazon S3 now has an option where you can ship your data on a big set of disks directly to them, they will import everything into S3, and it will live there forever. The nice thing about S3 is unlike physical disks, it can grow essentially forever, and comes with retention and redundancy guarantees. And once your stuff is in S3, you can recycle the same disks to mail them more data.
With the advent of 2TB drives, you could easily combine 3 of these with software RAID 5 as you suggested. Depending on how long you need to keep the data, recording them to dual-layer blu-ray disks might be a better solution. Ya, it's a lot of disks (you can buy 100GB discs now), but they'll last longer and you don't have to worry so much about mechanical failure or needing a certain OS when you want to restore them.
Sabre
If you go RAID5, have a known method for recovering if a drive fails. Actually perform a recovery before pushing it into service. I say this because some RAID5 cards use nonstandard methods making recovery very difficult and expensive. I'd also consider a process to transfer your datasets to new drives periodically so as not to lose your data.
Depending on what you're doing, you could consider using a basic version of IBM Information Archive: http://www.ibm.com/systems/storage/disk/archive/
It scales up to 304 TB (Raw Capacity)
Exactly. Let someone else do it. I don't know if Amazon is the right place, but the answer is still the same: Let someone else do it.
Why do we see questions like this so often? Why aren't people going to existing services with guaranteed availability that let you store a generic blob? Pass the buck -- they're probably going to do it better anyway.
You might look at a www.drobo.com as a set of 4,5 and 8 drive enclosures. 1 TB disks gives you 3 TB usable space with a 2 drive failure tolerance. I have the older 4 bay drobo (2 for myself, and 2 at separate clients offices). It is much simpler to use, and will scale to your 2-3tb use and allow mismatched drives that normal raid will not use. Get a enclosure to start with, and then financing permitting, get a 2nd for Drobo redundancy. Not the fastest or cheapest, but reasonably good by both accounts, and simple to use.
Have you already ruled out blu-ray? 25GB per disc, make two copies per customer. Much cheaper than RAID5.
Label it something like complete american idol blueray collection and upload it on p2p to piratebay. every couple years rename it to some other horrible popular tv series. It will be self sustaining form of storage with infinite number of redundant hosts.
Hard drives are by far the best route. The only thing I might change would be to use a pre-packaged external disk array system. Depending on what you get out of your compression, a two or four drive external system with 1TB drives would suffice. You can get them in USB, Firewire, esata, and SCSI. A nice USB2 (or upcoming USB3) system would seem to make lots of sense and be acceptable by future systems. They tend to be fairly compact and have decent read / write speeds.
My main caveot with these things is that just because it's external does not mean it's particularly portable. The larger capacity devices use "desktop" hard drives which are more susceptible to movement than the smaller drives used in laptops. Give it time to power down before moving and move it as little as possible. I've seen too many of these things die because the user would move it while it was on or use it as a transport device. That is NOT what they are for.
No sig for you. YOU GET NO SIG!
I would use RAID6 not RAID5, since 2 drive failures means data loss with RAID5, while it takes 3 drive failures to loose data on RAID6.
Linux MDADM has supported RAID6 for years, it's stable.
I would mix and match drives, not buying all the same model from one maker. One Samsung, One WD, One Hitachi, One Seagate.
That gets you 4TB in 4 drives, and unlike a RAID1, any 2 drives can fail with no dataloss.
You can further ensure no dataloss by making a second copy using different brand drives for each clone.
Eight 2TB drives is around $1500. Not bad for a very safe 4TB backup.
There are going to be quite a few storage service names thrown out as well as compression schemes.
1. Storage vendors you run real risk of having the data go away. There's a huge liability balancing act going this route.
2. Compression schemes. As someone who has lost data to compression errors, the consequences of 'just' compressing a file can be huge. http://www.linuxquestions.org/questions/linux-software-2/recovering-files-from-corrupt-tar-archive...-326716/ (not my post, but similar story)
I would suggest building tape archives, but as I mentioned above, this can be more hazardous than it should be. (ANY backup exec admin who have blindly relied on Symantec's solution without testing, testing, testing have horror stories)
Finally, I'd probably go with WORM optical media as the final storage media with a tape backup. There are lots of process decisions after just recommending hardware, so you are hardly done.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
you make the assertion that a tape archive would be impractical, but really it is the most practical solution. the drive will set you back a couple thousand, but 800 gig tapes are only around 40 bucks each, and they are engineered for data storage unlike hard drives. this will only cost $160 per 3 gig dataset, or 200 if you use par2 files and an extra tape to make it recoverable in case a tape does fail.
Snowden and Manning are heroes.
What you need is Database Archiving Solution (HP offers such tool). With this you can export your DBA using archiving tool into xml files and without having the need to acutally keep copies of the DB's itself. This tool will also allow you to recreate your db using the xml file.
every hard drive (7 or 8) that I've put in my safe deposit box has come out with file system errors. It looks like random bit-flips to me, and all of them were recoverable to some extent (minor data loss).
I have no idea why this happens, and nobody I've talked with can come up with a good explanation. There were probably 3 different brands and not all from the same series from each brand. Some had been used reliably for years prior to storage and others were brand new.
Bottom line: You probably shouldn't assume that safety from physical access is the same as data safety. For all you know, someone is storing their magnet collection above you.
The only answer here is LTO tape stored at a contracted record archival facility. Optical media degrades and is easily damaged, hard drives fail ALL THE TIME and will have obsolete interfaces in a few years. Tape has very long shelf life when stored properly -- it is time tested and trusted. It is not that expensive to get one tape drive and a few carts for each customer.
Get a couple LTO-4 tape drives (one to use, one to keep in the safety deposit box with the archived tapes...just in case) and do it right.
You can get 1TB+ on a tape. Each dataset will take you 2-3 tapes. Make 2 complete copies for your archive. If the data is truly important to you, its worth the expense.
Magnetic tape has been proven to be the most reliable long term storage, short of carving your data into stone monoliths.
Who wants to burn over 100 discs per client? I guess they make automated disc burners, but that's a little nutso. Plus, with that many discs you have a high chance of failure, so you'll need some kind of ecc scheme (parity discs?). I'd also have to vote for hard drives, although I agree with the shortcomings of that solution.
http://www.drobo.com/products/droboelite.php
Each Drobo has 8 bays of 2TB drives for a total of 16TB. And 255 Drobos can be linked together over Ethernet to create a single virtual volume of 4 Petabytes.
To back it all up, just buy more Drobos and store those in a separate location. They're too big for a safe deposit box, so your home is just as good as anywhere else (assuming your home is different than your office) -- or a temperature-controlled storage unit if you don't like that idea.
If you can afford it, I recommend the backup strategy I use, which involves four complete sets of the data. The main one is online. The second is always connected to the network and receives a backup nightly via xxcopy (yes, I manage a file server in Windows -- shoot me). The third is on-site but not connected to the network and gets rotated with the second weekly. The fourth is off-site and gets rotated to on-site quarterly.
If you're really wed to the idea of off-line archival storage, get these for each customer -- get two so you can have two sets of data in case one goes bad:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822154428
I've never had good experience with tape, from DC6150 SCSI linear tape at home all the way through an Exabyte library with stacks and stacks of 8mm tapes. Two decades of tape has been two decades of heartache and frustration for me and the companies I've worked with. These days I'm no longer in tech or IT (thank god) but for my personal needs I use RAID-1 for live and DVD-RAM (as cumbersome, slow, and small as it is) for offline.
Tapes just bleed data at an alarming rate, and they are about as reliable as a drunk gabling addict living under the subway next to the OTB shop.
Do the hard drive thing, and store them well, with redundancy, under good conditions, and replace them often.
STOP . AMERICA . NOW
You don't need a tape library. Just get a single tape drive, and you will be able to store everything on 3-6 tapes. Yes, you will have to swap tapes by hand, but it is a lot cheaper.
LTO-4 stores 800 gig per tape, uncompressed. If you let the tape drive do the compression, you might even be able to get away with one or two tapes. Tapes are inexpensive, and are designed for long term storage.
With easily compressible data (e.g. genomics data), I've gotten as much as 5TB onto a single LTO-4 tape using the regular drive compression.
An LTO-4 tape costs me ~$50. It's smaller than a 3.5" SATA drive and easier to handle. It can probably even survive a drop to the floor from chest height.
You'll need to spend some money on a drive or tape library. So it depends on how many datasets like this you need to write.
Heh. That's what she said! Heh ;-)
You have to keep rotating onto newer media, and newer media technologies. This sounds horrible, "oh no! I'm generating ten full drives per year". But realize in a couple years, all those drives will fit on a USB 4.0 stick, or on a card in your cellphone.
If you haven't read it (and recopied it) in a couple years, its probably gone.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Why?
I've worked in Visual Effects production and every time a new project came along we'd have to clear the servers of terabytes and terabytes of data. We used tapes. How are they impractical exactly? Inexperience?
I'd suggest using something Apache Hadoop's DFS. You get a couple of nice things for free. First, you can use really cheap hardware, without raid. Second, you can use really cheap hardware, without raid (this is big enough to say twice). Third, you can a distributed map-reduce system which may or may not be useful for your problem sets.
Another good option is a robotic blueray or MO jukebox. Given that you maintain the environment properly (temp/humidity range) these types of solutions will give you data lifetimes in excess of any spinning media or tape (without periodic re-tensioning). There's some challenges with this type of solution as you'll likely need to get some software to manage the jukebox and files but there's numerous options available. As much as I don't care for Windows there used to be something for controlling jukeboxes or changers built into the server versions of the OS. As for the jukeboxes themselves can be commonly found on ebay and then upgraded to higher capacity media by replacing the drives. I'd also suggest you look into setting things up such that you can have at least physical copies of your files. Lastly, like tape this type of media isn't terribly fast, so don't count on hard disk access times.
Depending on the openness of the data you could ask Google if their Palimpsest project is still operational. Basically they wheel large storage systems around for scientific research. But I believe they want to keep a copy by themselves, Alexandria style.
Tape is probably your best option. You can buy at DAT-5 (or even a DAT-4) tape drive for not very much. The tapes cost about $10 to $30 each (depending on what tape option you choose). Make 3 copies of the data set, store one onsite, store another offsite in a secure/climate controlled facility and send the 3rd to the client. Buy a spare tape drive and use both to make writing across tapes easier. There is a wide variety of software to write to the tape; we use the aging Retrospect.
The disk options is just way too complex; if anything, skip the RAID option and just store 2 copies. Putting the RAID sets back together and finding the RAID software will be nearly impossible in a couple of years. Use some standard formatting on the drives (FAT, NTFS, etc.) and you'll be good to go for the next 15 years.
I would suggest something like this:
http://www.netapp.com/us/products/storage-systems/fas2000/fas2000-tech-specs.html
that is, if you have a datacenter and ability to get 20A (NEMA L6-20) or 30A (NEMA L6-30) power.
Are the datasets generated using similar chunks of data? If so, deduplication could be very very helpful (in addition to compression)
Buy a netapp. Yay, RAID-DP.
That was hard!
* wipes brow *
The problem with storing things is that they tend to degrade over time, and you never know when they'll fail.
Without being ridiculous, four sets in two locations is the best bet. Two sets are on line, and a regular parity check should be made between the two, with full data verification on a longer scale basis. One backup set gets made of each online set (an external drive which is sync'd once a week/month is likely good enough) and stored unpowered. This prevents local disaster from destroying your data, electrical damage from destroying your data, and (hopefully) bit rot from corrupting your data (two online sets provide a cross check, offline sets allow polling if the parity is off).
Mechanisms usually last longer when they are in service than when they are left unattended, though it takes power and - more importantly - a human to keep tabs on the system. You should also have a 5-8 year migration plan so that the data is updated to current interface standards on a regular schedule. The biggest fear, short of actual data loss, is that your storage medium will be unreadable at the indeterminate point in the future when it becomes necessary to retrieve the data.
This is, no doubt, more money than you've budgeted for the storage. Whatever you do, don't use RAID5. Two failures = zero data. Better to use a RAID4 (JBOD with Parity). If you lose 2 drives, you lose data, but at least you only use 1 drive of data for each additional drive failure.
Is it just my observation, or are there way too many stupid people in the world?
http://www.cddimensions.com/Blu-ray-Libraries/products/192/
I think one could DIY a par + WORM jukebox with a waaay off-site tape storage and rest easy.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
I don't know why people would even suggest using raid for backing up. Harddisks when they sit around for long periods of time tend to fail out-right because of the bearings lubricant. 2-3TB isn't much. Why have the hassle of raid? You want it future proof. You don't have to have to be tied to a specific raid card and plugged in the write way etc. just to get data off, you want to be able to slap the drive in anywhere and get it. I would just make 3 copies on 3 different brands of harddisks and store at 3 different places.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
I'm sure the price has come down some since this article was published...
For those too lazy or paranoid to read the link... It describes how backblaze builds "cheap" 67 TB storage boxes for use in their online backup service. All the hardware specs are open sourced and freely available. They also talk a little bit about the software for managing all of the spce they have, but not in any real detail...
I have a friend who works here, they might be 1 possibility? http://www.ironmountain.com/digital-archiving/digital-archiving-services.html
As far as I know the 2TB Raid problem hasn't been fixed. http://blogs.zdnet.com/storage/?p=162 If anyone knows differently, please let me know.
I've been using a drive docking station and splitting my backups for large databases.
"The mind works quicker than you think!"
I have a bunch of old data backups on CDR that were great for years but they've started to degrade. I'd be willing to bet that any magnetic disk would be even more vulnerable to data corruption over time. I don't think your RAID 5 storage technique is a good long-term option.
This could be a ridiculous suggestion, but have you considered something like cloud storage for this? You could encrypt the data and store it in somebody's cloud and let them worry about backing everything up.
The question I have is what happens when you run your data sets through a compression program like RAR? One of our standard Database backups weighs in at 60gb, but compresses nicely down to about 5gb. If you can get your compressed data set under 2tb, I suggest backing up the compressed data set on to a mirror set and store it. The trick is getting it small enough to fit on a single drive. If it doesn't compress nicely, and still ends up requiring some kind of disk spanning, just stick with what you're doing. It's a solid solution.
Repeat never use DVDs as long term storage. I have seen them go unreadable anywhere from 2-5 years. I have fired up disk drives 10 years later with no problems. They are cheap reliable and fast. Don't try and get fancy just compress and store data sets over multiple volumes. Don't use RAID.
As you generate the datasets, you should seriously consider archiving the method (scripts, software, etc.) you used to generate them rather than the output.
Namgge
the good about compression, is less storage and better chance of detecting errors. the bad is at a minimum every bad bit becomes at least a bad byte, and if it is in the header, all data in that archive.
for example:
About corrupted compressed archives: gzip'ed files have no redundancy, for maximum compression. The adaptive nature of the compression scheme means that the compression tables are implicitly spread all over the archive. If you lose a few blocks, the dynamic construction of the compression tables becomes unsynchronized, and there is little chance that you could recover later in the archive.
So if the nature of the data is, "one bad bit means all data is garbage anyway" then compress away. If the nature of the backup, is "I may need a couple files out of a backup someday, who cares about the rest, then don't make a single compressed archive.
My method for filesystem backups was to do a file by file gzip in place on the backup server, this way I could lose a file, but not all files in a single archive. .gz files with uncompress in place gzip on windows machines, so users would just have to click, wait a second, then click again if we had to jump to the backup file server...
I associated all
Honestly, I don't know of any offline storage that I trust with that much data for any serious length of time. Have you considered building your own online/nearline storage? The backblaze storage pod is an excellent example of online storage on the cheap.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
with DLT. My posting is purely my opinion, and is anecdotal. But it is my opinion.
STOP . AMERICA . NOW
Print it out on a tractor feed dot-matrix printer to make sure the data set stays collated. Maybe even use carbon copy paper to have a backup around. Store it in an Iron Mountain (or under your desk). Seriously, whose going to know 100 years from know what a Blu-Ray is?
Why store a RAID array? Your data set isn't too big (2-3TB should fit nicely into a couple of today's 2TB hard drives), so it seems somewhat roundabout to store the array itself. Tarball it up and if the compression can squeeze it onto a 2TB hard disk, then you're golden. If not, use split to break it up. Then handle the tarball like any other file. Restoring is just as easy and if you use split, the opposite is our friend cat.
Store the array for instant access, but use the hard drives (you'll make copies) as medium term backups. If you need long term storage, then go with tape.
And I'm sure by the end of the year, you can just use one hard disk for your entire dataset backup. Which means you can duplicate it multiple times for safety. And just plug it into any Linux PC and access it immediately without having to reinitialize the array.
---
Data Integrity Feed @ Feed Distiller
Seriously, multi-petabyte disk array on RAIDZ w/ ZFS. Use OpenSolaris instead of Solaris and its as cheap as and technically superior to a Linux RAID solution. This is a no brainer. Wasn't there a slashdot story recently about how rackspace builds their disk arrays? (hardware)
Or are you keeping it just in case. If your customers are paying you to keep this data in archive form for them for some set period of time, then you need a solution that will meet that. Taking hard drives offline and hoping for the best is not a reliable long term way to store data. If you leave them online then at least you can detect disk failures, but then you are at risk from accidental or malicious deletion. If you are serious about protecting your customer's data, then it needs to be written to a real offline media (tape, optical, cloud), and preferably 2 copies one on your site, one at a managed 3rd party site that the customer can access if your business model doesn't pan out. If the data contains any sensitive information then it should be written in an encrypted format, and you should provide your customer a way to unencrypt that data if you are unable. The cloud providers have some great options for these services, but the data sizes you are talking about should be much cheaper to manage with 1-2 LTO4 drives direct attached to a system of your choice, and bonus is that LTO4 supports hardware encryption and is widely adopted. You may be a small business but multi-TB data archives are normally a mid to large size business problem. So, the small business online solutions that I have seen are not priced to solve your problem (but there are new ones all the time). Finally, LTO-5 will be out soon, so LTO4 prices should be coming down some over the next few months.
If you are not storing this data for your customers benefit, but for your own - to reuse in future work etc - then just decide what its worth to you. I still think tape or maybe bluray will be a more reliable / dependable / repeatable than hard drives, but if it’s your data do what you want with it.
You obviously haven't been in business very long, and I have seen no reasonable answers in this thread at all. Nuke the data as soon as you ship and offer to "recover" it (that is, regenerate it) for an additional fee should they require it.
Noobs.
I the same problem when it comes to backing up my moderately large dataset of study material for human anatomy. Granted, all of the data exists in the cloud, but I would hate to have to spend another 10,000 hours browsing porn sites to download it all again
They're completely self contained and hot pluggable RAID arrays. They have a SATA connector on the back, they look just like a hard drive to the host, no driver or controller card needed.
http://www.high-rely.com/HR3/includes/RAIDFrame/RAIDFrame5Bay.php
Just set up some torrents on the Pirate Bay and let the entire internet do your backup for you!
how about a cluster file system like http://www.moosefs.com/ , where it works well on commodity or even end of life by current standards of hardware. Redundancy is achieved through several chunk server nodes in the system. Performance and size can be dynamically scaled over time by adding more nodes or disks to the system.
Their project is GPL licensed, requires very little effort to set up, and likely much less effort over time to maintain and administer than a RAID or Tape system likely would, with the benefit of being able to choose what data is online or offline (by its presence on chunk servers being connected at the time).
Just translate it to binary and print out the data.
Rent some decent off-site storage at an established data centre and get a leased data line.
Don't bother messing about with tapes, it will be a full time job maintaining the library and space will be an issue after a while - I presume this is why you think it is impractical. With a proper data centre, you shouldn't have to worry about drives failing or the storage medium degrading due to age; most offer multiple site redundancy as well.
Seriously, don't get clever; save yourself the hassle and your business' reputation if something goes wrong with your 'lockbox' method.
I work as a contractor for the USGS and the projects I've been involved with host, archive and provide means for customers to access all our different satellite data products. We've got a Long-term archive method for tons of data products (digitally and tangible) and I can honestly tell you the first thing that always comes up is: how often will the data need to be accessed?
For the longest time (almost a decade) we used 3 big, STK tape silos for data archive and retrieval for custom orders. The problem behind that type of design is we used a archive in a completely wrong manner in the fact that we tried to use it as a archive and a quasi-online retrieval system into a caching filesystem. We had tape mount counts in the hundreds and thousands, constant mechanical tape issues because of the excessive use, ect. We actually decided to move it all to online storage using enterprise RAID (EMC Clarion) and moved to a small LTO-4 tape unit for almost permanent, maybe-once-in-a-great-while storage and the rest we leave completely on spinning disk and control the access to it via application layer network protocols as needed.
IMHO, I really think it's going to depend on the access frequency of your data. If that custom needs their data once, and maybe never again in case they lose it, put it on tape. If it's a requirement they can get the data from you any time they want and you've got the hardware and administrative resources, power and bandwidth, put it some RAID.
As usual, the "Ask Slashdot" doesn't have enough information to make a proper recommendation. There are a number of factors that need to be considered.
1. How valuable is the data?
2. How far into the future must you be able to produce the data for your customer?
3. What resources do you have at your disposal to solve this problem?
4. How secure do you need this to be?
5. How quickly do you need to be able to produce the data?
Once you answer those questions, you'll have a lot more insight into how to proceed. For example, if you suspect that a customer might want you to do something with their data in the next year or two, then your current solution seems reasonable.
On the other hand, if you are contractually obligated to provide copies of the data upon request for the next 10 or 20 years, you would need to invest in proper future-proof archiving technologies and duplicate environmentally controlled storage in geographically disparate locations. You may even have to go so far as to include schemas or transform the data to more universal formats.
As a related aside, if you incur big expenses to secure a client's data, you should be charging a premium for this service. At the very least, you should be using this as a selling point.
I don't know exactly what you are looking for but what about using RAR files to split up the data and using parity files to ensure the data can be restored properly? This way you would avoid having to create a raid array to have enough space to store the files. plus it will limit your chances of accidentally losing the data when trying to recreate the raid array. You could then store these files on whatever media you want (SSD, Blu-ray, HDD, Tapes etc.) Plus this would make it easier to use multiple media sets for redundancy, say store your main copy on a couple of HDD and then have a Blu-ray backup set JIC. Being a Network Engineer myself, I still strongly believe you should be using a tape drive/library for backing up these files as the media will last a long time. Even opting for the cheaper LTO2 drive to save some money and just requiring a few more tapes. Depending on how well your dataset compresses an LTO2 tape will hold up to 400GB(Compressed) and 200GB(Uncompressed). I personally like EMC's Networker product for running backups but since price is an issue you might want to look at a product like Arcserve or an Open Source product instead.
While working from home several years back got a safe deposit box to store hard-drives in in this manner.
I never had a problem, but about six months in read the Terms & Conditions again -there was a very strongly worded statement (in the fine print of course) along the lines that they did not recommend storing magnetic data in the facility.
When I thought about it this made sense -this was a general safe deposit facility; drug money laundered, secret agent stuff, jewels, rare-earth magnet collections...
Unless the facility you use (for hard disk or tape) explicitly sets out to protect you from it, you can't be sure what other people put in the box next to yours.
If the backup is very important, make sure you check the facilities policy and enforcement on this.
These guys offer distributed, 96-way encrypted RAID (32 parity slices, 64 data): http://www.symform.com/. Check them out. You will have to pony up the same amount of disk space as you consume, though, but to me this does seem a heck of a lot more reliable than RAID 5 or 6, and the drives can be super cheap.
I've been doing backups by copying everything to my unlimited drive for years now. It's amazing - it never fills up!
/dev/null if using Unix).
Just type
copy Edit.* NUL
at your command prompt (or cp *
One day I'm gonna look to see how much data I have in that damn thing!
Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
If you are only going to use one or two tapes per task then an autoloader is a bit of a waste of space.
They are not even that useful for scheduled backups since somebody already had to take the tape out of the machine and put it somewhere safe anyway. There's not much point having all of a weeks backups still in the drive when something happens to the server room.
I've got a small storage shed full of reels of tape from the 1980s, and every year one or two clients that have lost their originals wants to read the data in - tapes last for a long time. I'm not sure if a current drive stored in low humidity for thirty years would even spin up because all the lubricated parts may have dried out, but thirty year old tape can be read at a lot of places.
Never goes bad, never has a bad bit.
Non impediti ratione cogitationus.
I often hear on slashdot that tapes are much better for long term storage. However I have never used one myself.
What kind of experiences do slashdot readers have with tapes? How long do they last in storage? At what size of storage do they become cost effective?
I know some manufacture somewhere will have data sheets that could answer all my questions, but I would prefer to hear first hand accounts rather then marketing ads.
And under what conditions do they need to be archived? Everyone else is busy spewing technical and product solutions. However archival is more dependent upon issues like "Is this historical data collected for a scientific study?" If so, you want to chat with groups like photo astronomers as to what they do for data archival. Whatever archival media you choose will be obsolete within a few years and you'll need to regularly migrate the data as new technology is adopted. If it's generated through computation and simulation and is likely to be requested again in the near future, invest in the infrastructure to keep it spinning until it's cheaper to recompute it. For cases where it is needed infrequently, tape with an offsite storage company is the better bet. Remember that with tape you should be doing random pulls monthly to make sure you can actually retrieve the data.
And it's probably worth punting to your comptroller and legal department. If the liability for having just discarded the data is less than the cost to archive it properly, they're going to tell you to delete it after the customer says its good. Plus if archival is important, they have to figure out how to drop the cost onto your customers anyways.
Now when it actually comes time to decide technology... pick the vendor that does the best job of taking you and your peers out to lunch regularly. Their product, like everyone else's, will fail to meet expectations, but you'll get lunch.
Try cloud storage at www.zetta.net
I don't recommend using hard drives for archival purposes, they're not rated for that. Have you considered LTO-4 tapes: the price per unit GB comparable and they are intended for archival storage. You could ship them to an offsite location or make two copies and send to two locations if you're paranoid. Setup is relatively cheap if you just buy a tape drive, somewhat more if you buy fancy software and a tape robot.
LTOs (and the DLTs that preceded them) really are designed for reliability in a way that exabytes absolutely weren't. Exabytes were fast and cheap, and insanely unreliable. We have thousands of LTO tapes in our libraries, we might have one tape a year that has problems. Really, this is a solved problem.
I love Mondays. On a Monday, anything is possible.
Basically it is a NAS that breaks your data into sub-block sized chunks at ingestion, checksums it and single instances the chunks. Most users get 20:1 average compression. Obviously different data types de-dupe at different efficiencies. Our VM images typically hit 40:1, SQL Server DBs about 30:1, medical images 5:1. DDOS now includes some archive specific features, which it sounds like you might find useful. It also has a very robust replication feature. Since it was originally designed for backup, it is optimized for data integrity. DD is dead simple to configure and operate and the support is great. http://www.datadomain.com/products/appliances.html
It is cowardly, and a betrayal of whatever it means to be a Jew, to act as a white man
-James Baldwin
1. How are you generating this 2-3 TB of data?
2. Are you compressing it to its absolute minimum size?
3. Actually I don't know way more than a couple of things, but I digress.
Now, having spent most of the last year trying to determine the best method for storing huge amounts of data with limited space while retaining every important detail in a complex system that changes every few seconds --(deep breath)-- I can say that in OUR case, we found some very efficient ways to reduce the storage requirement at the front end instead of throwing money at the backend. Before I start rambling I will repeat the disclaimer regarding my total ignorance of your data situation, and your environment in general.
1. When you generate 2-3 TB of data per customer, what creates this data? Is there some type of progression or versioning in play (like a new value for a particular datapoint every millisecond or some such thing?) If so then whatever is creating the data may benefit from some type of delta checking before it saves data. Less data generated means less to compress. If you were to post a sample of what your uncompressed data values look like without revealing any sensitive information you may see replies with meaningful suggestions on how to change the data itself.
2. How exactly are you compressing the data? We've found that doing everything possible in step 1 above results in much less data meaning smaller source size and a smaller need for compression. But if whatever is compressing is dumb then you can turn a 500 GB archive into 1 TB which ends up costing you way more than rethinking step 1.
Again, I may be completely missing your point here and you may have already thought of these things many times over. Or you may be at the mercy of coders that don't care how much data they generate. But I think, lacking the answers to the above questions, no one here could give you the solutions that have the best bang for the buck.
50 year archive life, and up to 1.6TB capacity. No idea how much it costs.
http://www.inphase-technologies.com/products/default.asp?tnn=3
I work for a small business where we needed to store ~15TB on the cheap. We decided to go with a distributed filesystem called HDFS. It's vastly cheaper than large HA raid arrays b/c the filesystem itself handles the redundancy by creating 3 copies of each file across the cluster. SATA works fine for HDFS so you don't need to pay the premium for SCSI. Since most of our data happens to be log data, we use Facebook's Scribe to aggregate the logs and throw them onto HDFS for long term storage. It's not the most traditional or easy to work with storage, but it's definitely easy to scale out b/c all you have to do is add more nodes to the cluster. Hope that helps
If the setup is raw files on tape (the stuff I've read in from reels recorded in the 1980s) or tar (stuff from the 1990s) it's not a big deal at all. It's only two-bit closed source single developer efforts on MSDOS that are difficult for anyone to read if they have the hardware.
With recent backup systems like AMANDA you can just dump a file with dd or similar and the instructions on how to deal with the data are there in the header in ASCII! It couldn't possibly be easier.
Also the "proper environment" is a lot easier to achieve than with storing hard disks, putting them in boxes in a warehouse without a leaky roof is most of the way there. Those tapes from the 1980s I'm talking about were exposed to 30C temperatures and high humidity every year, which is not ideal. Theoretically they were spare copies that were only made for transport and only had worth after the originals and copies were mistakenly thrown away by the clients.
The "people" problem is very easy to deal with. You make a lot of people responsible for backups of the department they are in, and make it VERY easy for them to do the backups. Somebody in small department X knows to put a tape in a few times a week, and so on, so you get a lot of people that know for certain that there are backups and how old they are. When the inevitable happens and things get lost or deleted there is no longer the panic of tracking down the IT guy and asking lots of questions whether something is lost or not. Instead of spreading panic you just get mild annoyance that it will take so long to get it back.
Also when one person leaves there are still many that know the system.
Of course using rsync to a machine elsewhere to get a copy of everything is useful, but that is not a backup, it's a copy, so it shouldn't be the only option. With things like that you eventually purge old deleted files or write over old versions of files, while the tape from two years ago has what was on there two years ago.
LTO tape is $50/400GB ($125/TB) - PAINFULLY SLOW, SEMI CHEAP
Disk is approx $150/2TB ($75/TB) - CHEAPEST AND PRETTY FAST
S3 is $153/TB/mo + xfer fees - SLOW, NOT CHEAP
Buying your own harddrives and storing them yourself is the cheapest option and probably will have very good retention. Since you're doing this frequently, it seems that it might be worth it to buy a SATA SAN that you can mount several drives in and a *bunch* of SATA drives. Put in 3-4 drives, raid them, copy your data (might take a while). Put the drives somewhere safe. If this is customer data, you can charge them a fee for data retention, so you don't have to eat the whole cost, but you'll have to put some money into the platform to begin with. If you roll your own, ZFS might be a better option instead of Linux's software raid because you can turn on compression and move data around if you need to. Getting something going with hot-plug drives (a PC chassis or SAN or whatnot) might also be a good investment. You may be able to re-use the drives after a while. Drive costs will also drop over time and you'll be able to buy 4TB and 8TB drives for the same price in a year or less too. After a year or two, we're talking a few bucks to store this amount of data on your own, fast media.
As far as storage, a safety deposit box will only work for so many drives. Might look into http://www.ironmountain.com/ for secure off-site storage, or just encrypt the data and take it home (off-site) with your or one of your employees so it's physically in more than one location.
RE: and we end up generating fairly large datasets (2-3 TB) for each customer
how many customers do you have? 1? 2? 10? 100? 1000? 5000?
how many IT staff do u have managing this?
From the description it sounds like you need long term storage and infrequent (if ever) access. Optical is obviously bad for that. Hard drives are too. Storing hard drives for long periods of time without spinning them up can result in the platters seizing. Tape really is your only reasonable option. You can pile it above some poor post doc's desk like they do at my lab or you can store it properly, your choice, but either way it's better than the other options.
It'll cost you anywhere between $100K-1 Million, you can look at vendors like EMC, and bluearc, or Netapp, or Sun, and buy a nice SAN. Use SATA drives for 2nd teer storage You can mount everything via NFS, or do it ISCSI via a fiber network. Dead easy, and the best thing is you expand it as needed with no downtime. These vendora are pretty hands on about support, for example you'll have a tech at your door to replace a failed disk you didn't even know about because they monitor this stuff, or have them come in to update frimware.
A problem I have here is the definition of 'long term'. To each of us it means something different.
In my job I have to archive 1.6 terabytes of data per day, and keep it around for 45 days (which, BTW, is not my definition of LONG TERM). For this task I utilize Data Domain storage, which utilizes data deduplication techniques for massive compression.
What you find is that at the block level your data may in fact be incredibly deduplicatable. In my case it very much the situation. I am currently storing 86 terabytes of rolling archives within 2.5 terabytes of physical disk space.
The problem with any technology you use for 'long term' storage is the ability to read those archives later. Assuming the media doesn't self degrade inside of the time frame you call 'long term', you must have the tools to read that media again. If you use BluRay, then you must store a compatible drive with it. (Nothing says Sony will not change the standard in two years and make all current drives obsolete, so no one makes them any more). Tape is worse, in that in two major model revisions, drives wont be able to read your media because its density is to low for the new drive head technology. Hardware based disk raid has the issue that the controller the raid was built with needs to stay with that raid. Another controller from the same manufacture, with the same model number, but a different firmware revision may not be able to figure out the raid, and declare the drives empty. Software raid is a little easier to deal with as long as you keep a copy of the OS you used to create it with in the same box. But then, during your defined 'long term' period, will you still have access to a system you can even plug these drives into, or run the OS on?
What you end up dealing with in reality is that as an archivist, you either ignore these facts, or you invest in a constant media / technology refresh and spend large amounts of time keeping your archives on the latest storage available.
Of course, all this falls apart if your definition of 'long term' isn't as long as some will project. In my case, my archives roll over every 45 days. I could easily keep that data alive for years on a live piece of hardware with a service contract. If I do not trust that hardware enough, I can buy two and replicate between them. (which, actually I am, for disaster recovery purposes)
With deduplication my (acknowledged) high initial investment quickly outweighs the cost of single purpose drives holding one copy, and wasting unused space. My purchase cost was less then $60k, but if I had to store all of that data in its raw form, my costs would be in the millions. However, if the data is not deduplicatable, then of course it is a moot point.
Each answer has it flaws. You decide which risks are acceptable, plan your best to deal with obsolesce, and define your definition of 'long term'. You also have to be ready to change your solution, when the one you choose today, fails to be the right solution for your needs in 5 years.
Loads of comments, alternating between disk and tape, and a few ridiculous, impractical options (Amazon, etc) but nobody here's found the best, cheapest, and most highly redundant method!
You take your 2 TB file, zip compress it, Encrypt it, rename it "Britney Spears Gang Bang kdiaKiS93kDw.mpeg" and stick it on Bit Torrent! It's highly redundant, very secure, costs basically nothing, and your chances of finding it again in 10 years are at least as good as finding a tape 10 years in the future!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Sorry, but disk is free and RAID 5 has some really nasty rebuild times for large drives. It's also important to remember that RAID is for availability, not backup.
Our largish datasets (2-10TB) run on RAID 1 mirrors for availability and are fully rsync'd to a separate system (also mirrored) for backup. The stuff we really care about is further rsync/rsnapshot'ed offsite.
Disk. Is. Free.
Let's see, where'd I put that stack of 12,000 AOL disks?
Table-ized A.I.
Don't use raid 5. With Linux RAID 5, you will be at the mercy of finding a Linux box with the same exact software format as when you wrote the tapes. At the very least, you must put a liveCD that works with that cluster, and configuration notes.
The only reason to use Raid 5 is if you have a single file that needs to span multiple drives (> 1-ish TB). Don't do that. Change your dataset to have multiple files, and write a script that will segment your dataset out into multiple disks, and pull them back again. Write down those rules in English, not in code, and put those rules with the disks. If you do have a single large file, write a script to slice it up and put it back together again.
Then, pick a file format. I think the only two reasonable choices today are FAT32 and EXT3, with NTFS as a dark horse, but this is crystal ball territory - your judgement may vary. It'll be hard to argue against the chances of finding FAT32 support 10 years from now. If you think file format risk is greater than disk failure rate risk, make one copy in each format.
Then run your script to pack each drive full. Then make a second set. Then, if you want, make a third set.
This is ghetto RAID 10, which is the best combination of failure resistance and price, without getting in bed with a RAID format. And, you get partial restores in case of catastrophic failures.
And - realize that 2T drives have a huge price premium over 1.5T drives (2T at $300, 1.5T at $100 - 3x more price for 33% more storage). We also know that stability goes down at the edges of the performance envelope in every hardware domain. So pick the second-best.
If one was to assess risk, buggy drives is a strong contender. I would strongly suggest making a stripeset of one manufacturer, and a stripeset with another. Seagate had all those firmware bugs, maybe the next round of bugs will be WD. No way to know.
I know you said a tape library is impractical, but depending on how many customers you have, it may still be your best solution. An LTO-4 tape drive with a small multi-tape bay and robotic picker really isn't that expensive (in the scheme of things) anymore. You can get a small sized LTO-4 vault/autoloader (24 slots) from Oracle/Sun for about $5k. Tapes are around $40-50 a piece and store 1.6TB each, which is a heck of a lot cheaper than hard drives for the size. So for your 2-3TB you would need 2 $50 tapes, vs., 7 $70 500GB hard drives, for a savings of almost $400 just for one customer's data sets. If you have 10 data sets, you just paid for your tape autoloader and are using a proven long term storage safe solution.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
I'd be tempted by a SATA toaster http://bit.ly/bXDpcx, but worried about it being on disk frankly. What would be nice isis some one could produce an authoratative summary for everyone. Bit like lifehacker's hive.
If you have a general preference for disk vs. tape, you might want to check out a ruggedized removable disk solution such as RDX. It can be safely handled/dropped like tape, is quickly replacing the low end tape technologies like DAT, and is sold by all the major server manufacturers (Dell (as RD1000), HP, IBM, and others). You can find more info at http://www.rdxstorage.com/. If you are looking for more of a managed solution, the InfiniVault product which also uses RDX basically shows up as an infinite capacity NAS device and is designed for specifically this purpose. You can find more at http://www.prostorsystems.com/.
the very high data rates (80+ MB/sec) required to keep the drives streaming without "shoe-shining" (starting and stopping a lot, causing the tape and mechanism to wear out). You really can't just plug an LTO drive into your white-box PC and dump the hard drive file system to it, especially if you're doing other stuff with the PC at the same time. Enterprise backup systems generally have a RAID box dedicated to backup. You transfer the data from the PC to the RAID over your local network, then dump from the RAID to tape. Often you use a robot to change the tapes. Even after all that, it's still a huge pain. The place I worked at gave up on tape after a while and uses redundant disks for everything (about 2000 computers backing each other up).
I would use a tiered approach simply use a cheap S-ATA RAID 5/6 for things you still need access for. And once you need to archive them clone it to tape and preferably keep 2 copies of everything stored at different locations, doesn't add much too the cost. But can save you alot if one copy gets lost or doesn't work.
If they don't want you to, problem solved.
If they do, ask them how badly they want to (ie $$$)
If they won't pay the costs, problem solved
If they will, again problem solved
My advice would be to cost out the price of a proper tape library - I know you said you don't want to do that, but honestly if you don't want to do it properly you are taking a huge risk by cutting corners. Tape is one of the best archival forms for storing (you've seen how many people are doing it) and to do it on disk takes more work and effort that you can probably manage. If the customers want multiple copies kept, then charge it at a sensible price and use a commercial storage firm.
It may sound strange, but if you do it properly while you are a small business, then you will have fewer problems when you become a large business. If you are trying to grow your own solution then you are effectively expanding your menu of services to scientific and storage services, or potentially risking becoming a small business that failed.
A tape library would be impractical at the present time. What do you recommend?"
If it's your business then you need to make it practical. There's a reason why tape is still the industry standard for backup and offsite archiving. If you can't find a way to fit secure backup and archiving into your business model then perhaps there is an issue with your business model?
Hi, If you are really talking about multiple terrabytes of archiving and you are allowed to keep it online I would look at something like Isilon IQ 36NL or IQ 72NL Series. This gives you a redundant storage environment where you can store depending of the size of the cluster starting at 100TB up to Peta Bytes of storage. And it is affordable. These amounts of data are just to big to consider things like tape or blueray. JHP
Tape sucks, the reality is that it doesn't last, the drives themselves are fragile and prone to failure. Firmware changes in what appears to be the same product may make tapes unreadable and they are expensive. Whilst the tape companies survive they're not thriving and living on pure inertia. This may change if a new technology appears but...
Hard disk size will simplify your problem a couple of years. A 5 TB drive will give you appropriate storage and you can mirror this and offer the other one to the client. That way they are complicit in any data loss scenario.
If you're moving those amounts of data around your network, I assume that you've got at least Gig ethernet. You can leverage this by using ATAoE as the basis of network storage and ether buy coraid's disk enclosures or build one and export it using Qaoed or vblade http://aoetools.sourceforge.net/. In terms of bang for buck this is one of the cheapest network storage options.
Then let bittorrent take care of your distributed storage for you.
If you were blocking sigs, you wouldn't have to read this.
Tape is designed for this and it's how the LHC stores its petabytes of data (with a HDD based "cache" in front of it).
Disks are not designed to be used as a long term (by that I mean decades) storage platform. You also need to consider the stability and longevity of the formats you store on whatever media you use. Everything from the file format to the partition format to the volume manager format. These days even the magic the disks do to map to the physical sectors is important (e.g. 512byte to 4k block mapping in new drives).
We manage a moderate system of 5000 TB, growing to 10,000 TB in the next two years. Lots of sensor data. We get about 5 TB a day and that is growing to 15 later this year. Next year looks worse.
We have your problem and found that how you store your data is dependent on many things. We use a combination of disk and tape -- about 1600 TB of online disk (for fast cache) and the rest in 1 TB tapes (ok, 930 GB after fuzzy math).
We split data sets to make them fit and then duplicate the medium. So for a routine data set of our (3-5 TB per day), we break the data into chunks that fit within a tape and then make sure we have two copies of the tape. Disk access is all RAID (Hardware based 5 and 6, depending on vendor hardware). At any time we have at least two copies in the system, and more if you count copies at other sites.
In your case I would split your data set down into manageable chunks (each less than 1 disk in size) and then stuff it in.
Do not use RAID. We had an experiment where data sources would send us RAID data just like you suggest (Linux RIAD via hard drive in big padded cases from remote locations). Not optimal. Issues with firmware, drivers and the like are surprisingly frequent and take time to rectify.
Use several sets of them, from different manufacturers (to exclude the possibility of nasty firmware bugs or manufacturing problems killing your data).
Do not use any kind of RAID just for storing the data. That's an invitation for problems, and hard drives are _cheap_. The main purpose of RAID isn't extra safety, it's to shorten or eliminate the downtime if one disk fails. Since you're merely storing the data and therefore don't need to hot-swap, you don't need that.
My $0.02.
There was a story on Slashdot 10 years ago about a guy that developed a way of putting 50gb of data on one sheet of regular paper based on an algorithm using shapes and colors...I am uncertain about the degradation of the paper over time, but you may today be able to buy into that technology...although I am uncertain as to where or how....
We have a very similar issue.
The solution we've adopted is to build the cost of storage devices into the quote.
When the job (analysis) is done the client gets the TBs to store how they want.
We don't end up paying for or storing anything.
Cuneiform Tablets.
They last forever.
take a look at S3 or jungledisk and see if you can somehow have that make sense. Mozy might not be a bad idea either. The big tradeoff is
1) limited SLAs (privacy / latency / bandwidth) for getting to your data; once you host with a provider and accept their physical / logical storage footprint - you are constrained to living in their hosting model
vs
2) providing quick access to your dataset b/c you have special sauce
The timing / requirements to get back to the data are the things that should drive your behavior. It might make sense to turn the data over to the customer - so when they need you to work on it- they provide it back.
I think you might be looking at this all wrong - why not redo the analysis and charge for the whole thing again? (According to the RIAA/MPAA isn't that what should happen when we scratch our movies or music?)
--Adrian
Other than the bare physical layer, you will want to think about restoring and reusing the data at some point in the future when the software used to generate the data is no longer available. What good is a RAID 5 of a couple Gigs of data when you can't use it? Or don't know what the data actually meant?
You mentioned putting them in a safety deposit box, so you are talking off line storage. You probably dont want to raid them. It is just another point of failure weather you are talking a battery in a raid controller a software raid. I would just store them in a regular filesystem on large capacity drives. Simple, easy to plug in and start again. Having said that Tape is still king and the LTO 5 drives will be out this month (march 2010) according to both HP and dell reps that I have heard from. I would put one copy on Hard drive and one on tape. good redundancy; store them in different locations.
You should consider some new and novel ways to de-duplicate your data at the blocklet level. Say you generate a few terabytes per customer. I betcha there's a fair amount of that data that overlaps, but not at the file level. Let a well managed de-duplication engine reduce it, then save the greatly reduced data on disk, or even tape. Here's one such product, backed by a company that has dedicated itself to this new market:
http://www.quantum.com/Products/Disk-BasedBackup/DXi6500/Index.aspx
How about using blue-ray juke box or some sort of WORM technology? Google Powerfile (I have no vested interest in this company) and you might find their technology useful. It's proprietary, but I'm assuming long-term data storage cost should go down significantly. Obviously, there are other WORM technologies out there that might do the job just fine.
It would be much easier to suggest something reasonable if we knew what the impact was if the data couldn't be recovered, and also why the data needs to be kept. Are you keeping it for yourself? If you're just keeping it around in case the customer comes back for something, why not give it all to them and let THEM worry about it?
http://www.permabit.com/products/roi-calculator.asp
Not sure of your budget, but I used to work at permabit.
Archive storage, always online, with in-line de-duplication, RAIN technology (250X more reliable than RAID 6) All at less than $1 per GB, that's right, less than the cost of a medium coffee per day. Not happy that I lost my job, but hey, their stuff works, and is quite impressive (MIT startup, in Cambridge, MA)
The technology is hardware agnostic, new 1rmu units can be addded/removed as technology improves, or storage needs increase. A new node is automatically added to the system capacity once added in! Data remains accessible throughout..
Tape is impractical but you're going to store RAID5 disk sets in safe deposit boxes? How is tape impractical? Speed? Upfront cost?
One thing you didn't mention in your post - how critical is the data? If you lost it what would happen: Nothing? Would you lose a few hours time recreating it? Would you go out of business? Would you get sued for breach of contract? Would Knees and Knuckles be paying your family a final visit? Knowing that would make a difference in how I would store the data.
As several posters have stated and you've noticed yourself - nothing beats a hard drive for cost/byte.
But then you need to determine what to do with that - do you keep it online at all times (power, space, cooling may become issues to consider).
How many copies of the data do you keep? Hard drives fail. Just because it's raid doesn't mean it's safe.
What's your bigger plan for dealing testing for failure of the backup media and determining when to retire them? Periodic testing has to be one of the most important parts of your plan. You build something test it once and later find out that your last 2 months of backups were worthless. Testing can help you avoid that.
Do you need to keep copies off-site? Having 2 copies in your data-center located in your basement is no good if it floods.
How much total storage do you need, need 6 months from now, 2 years, etc.? There's an interesting article from the online backup folks at backblaze.com. They put together 4U enclosures that store 67 Terabytes for about $8,000 USD. Complete instructions are on their site for how to do it (they don't sell them but use them for their business). However, it's not exactly portable. While not physically huge, it's gotta weigh a bit. Perhaps 2 at separate locations with a network connection between them to keep them mirrored?
There's a number of firms that offer various online backup solutions where your data is automatically uploaded to their datacenter automatically, however I suspect that you're going to exceed their usual offerings unless they have some "poweruser" or business option. For individuals, "$5/mo unlimited storage" seems to be the norm. However, their are 2 limiting factors to that "unlimited" - your/their available bandwidth. If it takes 2 months to push out your dataset - is that acceptable? Also, many firms delete your data 30 days after you delete it. So if you move those hds to your safety deposit box, does the backup co see them as deleted and then delete their copy? Comparing the cost of doing it yourself vs them may be attractive, esp if they have some appropriate business plan that's not much more then their individual plan.
Does the data need to be encrypted? If you loose those hds on the way to your offsite location will it be merely inconvenient or life altering when someone finds it and reads it?
Finally do you need to somehow need to index the data so you can find your backups?
Making backups is easy. Doing it right so you can actually get your data back takes a little more work.
Paul
This way if there is a problem with a particular batch of drives it won't ruin everything.
Data that can be digitally copied is pretty much immortal.
Mechanical hard-drive are not.
I would go for :
- RAID, too. Preferably RAID6 (better data survival and check-ability in case of failure)
- Some way to check sum that data themselves (in check summed file systems or archives).
- Periodically get the harddrives out of storage and do I full check on them (check drive's SMART status, check driver's surface readability, check's array's parity, check files checksums).
- Every now and then, replace the drives (Either copy and swap one drive at a time inside the array, or migrate the whole content to a bigger array, while checking everything at the same time).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Thank you all for your replies, for allowing me to benefit from your collective experience. Both the straight-up answers to my question, and questions requesting further clarification were quite helpful.
Error 404 - Sig Not Found
Barracuda networks offers a backup service that scales very well with large databases. They also use a deduplication technology that examines each file part by part. If two parts are the same then only one is kept and the other has a marker. In most cases the deduplication ratio is at least 5x (meaning 1tb turns into 200gb) and sometimes it can go as high as 50x deduplication (this is mostly the case with images). After dedup, the data is encrypted and compressed (compression far out weighs the encryption in terms of size) and THEN it's sent off to the cloud.
You can get an LTO4 Internal drive for $2.5k (HP 1760 Ultrium) hooked to a cheap desktop PC if you don't mind swapping the tapes yourself about $3-3.5k total expenditure. The "enterprise-y" route would be $7-$9k if you want a robot to do the swapping for you (look at HP MLS2024 for example). And that will get you 2U with robot and I believe two 12 slot magazines all LTO4. You can hook this up to a very inexpensive single socket server for around $2k. Then you've got media. LTO4 tapes are running around $40 for 800/1.6 tapes. Total solution that way figure $9k-$11k.
Now compared to disk I'd go with $180 2TB Seagate Barracuda drives for a reasonable option. Three of them give you 3.5+ TB of usable space per customer which meets your requirements with some breathing room. Now the important part - you need a server to put these drives in. So when you're comparing the cost of the infrastructure for tape, don't forget you need infrastructure for the RAID5-array-builder-machine-guy-thing.
There are numerous other advantages for tape as well. Easy encryption, easy restores, less storage space, less likely to fail, etc etc etc. For me tape is a no brainer.
Is LTO-5 the last hurrah for tape?
What about the problem of having a working LTO-4 drive ten years from now, if the tape industry begins to wither as other solutions continue to eat away at the tape market?
I think the creativity here is negotiating the nature of the SLA contract with the clients. My preference is just to set up a local disk array with enough spinning capacity, not promising to survive a site disaster, charging for the service only so long as the data remains live.
To complement this, send each client a master LTO-4 tape (or a disk drive) and tell them "it's up to you if you need to recover this, but I'll help you out if I'm able".
Otherwise, you get into this horrible risk calculus where the client is not thinking through the cost benefit with rational comprehension.
I would try to find some way to unpack convenience from autonomy from ultimate responsibility, because if you don't, your clients shouldn't be balking at the price of Amazon S3. If they are balking, it's because they don't really want all three of these packed together, but the timid bureaucrats don't wish to admit this, in case the day comes when data is lost.
In economics, it is common to do net present value calculations. It would be interesting to do a backward discount on prudence if the day comes when the shit hits the fan. There's a lot of weird asymmetry in human psychology associated with risk.
From a business perspective, it's sometimes good to give your customer's options priced at levels where you expect not to get many takers. See the story about The Economist subscription model in Ariely's lecture:
Dan Ariely on our buggy moral code
It's amazingly hard to find residential fire statistics on a per annum risk basis, if we're looking at personal acts of god rather than communal acts of god (hurricane, earthquake, etc.) The firefighters meticulously count the number of times they respond, but seem not to talk to the fire insurance people about the number of structures insured. Not one report I looked at from the UK, the US, or Canada denominated the statistics per residence.
A loose estimate for Canada in 2002 is a residential risk rate of 1 in 300 per annum. Older building stock with plush curtains and deep fat friers will have higher rates, recent building stock with working fire detectors and no children will have lower risk.
Another table shows me that the risk of a 45 year old male being diagnosed with cancer by age 50 is 1.5%, or about 300:1 against per annum. Radon gas causes 15% of lung cancer, and 15% of American homes exceed recommended action levels. How many of the stripes+parity+fail_over+hot_spare+IronMountain crowd here have bothered to purchase the $50 home radon test (excluding smokers who smoke indoors, who are in a different risk category altogether)?
The human mind seems to incorporate an instinctive Bayesian prior that if you are actively discussing a risk, the risk is immediately ten or a hundred times greater than it was five minutes ago, before the risk entered the conversation. Likewise, any hill you are standing at the bottom of seems ten times higher or steeper than any comparable hill on the other side of the valley.
I'd go further and say that in my opinion, and especially at these dataset sizes that are only small multiples of physical drives these days, RAID5 is a hinderance not a benefit to price-performant backup, because it requires validity of all-but-one of the drives in your array... typically, in the CORRECT array, so swapping mirrors in/out may be quite a headache.
Don't use any data level striping; break your data into a couple chunks drive-sized in the filesystem. Keep mirrors of each chunk on drives, both onsite and (one or more) offsite. Bit-compare the drives occasionally to look for loss.
I recommend at least 3 drives for any dataset; at least one onsite and at least one in your lockbox; that leaves one to be in transit at a time.
Replace the drives with newer versions every few years. Use a variety of brands/models.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
I don't know much about Linux RAID, but as far as I see, if you must split across multiple disks, use the simplest approach possible so that you aren't left in some years time with a jungle of problems trying to assemble them again. Something like a simple split of the file if it is one large archive, or if it is a bunch of individual files, just distributes them between the disks somehow.