Slashdot Mirror
Backing Up 100 Gigs in an Hour?
cybrthng asks: "I am faced with finding a backup solution capable of archiving to tape about 200 gigs of a financials data in a 2 hour window. I originally looked into DLT8000 Jukeboxes with 2-4 drives but have recently discovered the new LTO drives. I am interested in knowing real world experiences with these drives as there has to be a catch. I mean there is a 3 fold performance increase in data transfers, two fold increase in tape capacity and a minimal price increase overall. With these drastic differences is there something I'm giving up with LTO over DLT or vice versa? Which backup applications are more geared to handling volume and integrate with Oracle RDBMS? Restoring speed is even more critical then backup speed so i'm curious about how these two drives compare and which applications are best geared for this much data on a nightly bases. Mind you there will also be about 500 gigs of data in an end-of-week backup as well."
Backing to a stagging array and then moving to tape is a great idea. Backing up to HD means you only get to keep the last couple of back ups. The reason to use tape is to go way back in time. With a good grand father-father-son rotation, you can keep a lot of history with a reasonable amount of tapes. I have backups dating to my first week at this company several years ago. I can raise the dead from almost any 2 week checkpoint since then. I have had requests for e-mail that has been gone for 6 months and documents that have gone for more than 18 months. It makes you look really good to just smile a friendly smile when somebody asks sheepishly, "I need a file from an employee who quit last Easter. I'm not really sure what it was called, but I think I know where it was on the network. Can you help me?"
Floods, tornadoes, fires, etc happen. Sometimes people fly planes into buildings. When that happens, tapes are the only thing that keeps your business in business.
I know this is completely off topic, but sometimes tape just isn't cost effective, particularly when you figure in the costs of manually storing and maintaining a library of data tapes in a vault somewhere. (Most of that cost is in head count: you have to pay somebody to do that work, and that's not a $19,000 a year job.)
We're presently doing the cost analysis on a kind of radical idea. We're storing many terabytes of data in a data center in San Jose, California. The data center is as good as it can be, but there's still the danger (however unlikely) of earthquake or some other drastic event.
Rather than trying to back everything up to data tape, we've gotten pricing from a telco on a dark fiber link between the San Jose data center and another data center somewhere in Colorado-- can't remember where. Since we're already putting an HDS 9960 in the San Jose data center, we can put an identical one in Colorado and use the 9960's internal "NanoCopy" software to keep them in sync.
Believe it or not, it's working out to be more cost effective. One of the big reasons is that keeping that much tape on-line in a data center would require a StorageTek PowderHorn silo, and data center floor space is expensive. The difference in cost between the floor space and the dark fiber is so small that they cancel out.
Like I said, I realize this is light-years away from what the poster was originally asking about, but it's kinda neat nonetheless.
Yes, medium has to be tape. We have to follow standards set upon us by governement and other standards bodies.
:)
The safety deposit box we use for month end tapes is just the right size for 3 rows of 12 tapes as well
i get around 120 gigs/hr *sustained* (that includes the time to change AIT2 tapes using the robotic arm) using 4 sony AIT-2 tape drives (i.e. saturating all four drive slots on my AIT-2 library) while writing 2.75TB in one go with hardware compression on using amanda for the backup software. peak writing capacity is well over 160Gigs per hour or more with hardware compression switched off.
each ait-2 tape is the size of a small pack of cigarettes and really cheap. the tape library itself with 30 tapes loaded can be carried fairly easily under your arm if you want to go lugging it around as a portable device. its about as heavy as a computer monitor and about as bulky.
get ait-3/4 which will double or quadriple the capacity per tape while retaining the same tape size if you have the budget for it. AIT3/4 should also give you around 200 gigs per hour or more.
BTW, i use an sun E420R to whack data across to the library.
Here's a copy of the email I sent to cybrthg:
:-).
Storage is all that I do for a living. Here's a quick summary of how you could do it:
1) First you have to make sure that the drives you have your data on are going to be able to give you the read rates that you need. I highly suggest that you go with a raid array of some sort. Preferrably one with substantial cache in front of it. The raid controller should be smart enough to do what's called "read ahead" caching. That reduces the read miss ratio and speeds up sequential (i.e. backup) applications tremendously.
2) Get whatever tape drive you choose. Base your decision on the speed of the device - use only the native performance. LTO is either 15mb/s or 16mb/s depending on whos drive you get. It is safe to assume that you will get about 1.2x compression. So for LTO that would get you either 18mb/s or 19.2mb/s. Assuming you get the 15mb/s drive you can realistically expect to get ~64GB/hr per drive.
3) You have to get some sort of advanced backup package to support those rates. I would suggest that you go with either Veritas NetBackup, Legato Networker, SyncSort Backup Express, or any of the enterprise class products. Don't go with cheap software - in general they do not have the performance coding necessary to move data at very high rates efficiently. This is a hard choice, but if you stick with the three I told you, you should do fine.
4) Get a library that can handle several drives, so that you can use them in parallel.
5) Put each tape drive on a separate scsi bus, or if you go with fibre channel put at most 3 drives on the channel. There's a ton of way of architecting this side of the house, but in general if you stick with those numbers, you should be fine.
6) Try not to send data over the network, even with GigE, the effective rates are going to be drastically slower than those of direct attached devices. GigE also severely impacts the server - tcp/ip overhead is a bear for high throughput environments. There are ways around this, but that ouside the scope of this email
That should do it for you for the traditional backup methodology. There are other ways of doing backups - making mirrors that you can split of. Taking snapshots..... There are a ton of products that can help you on this. Some of them are software based packages that sit on the server with the data. Some of them are hardware/software devices that sit on a SAN or a NAS. Again, going into this is quite lengthy, but it can be done. I have a customer for which we are doing over 1TB/hr backups using a combination of lots of tools. You problem is quite a bit simpler.
-- thoughts on one of those things: http://amuyu.com/