Automated CD/DVD Archival?

An anonymous reader asks: "Our department used to use a Cedar Technologies Desktop CD-R Publisher for fully automated backup of data (~2 CDs per day) controlled by a Linux PC. The publisher just broke and we are looking into a new backup solution to automatically burn and print CDs or DVDs. Solutions for CD/DVD duplication are available for Windows and Mac (for example: Primera and Rimage [which acquired Cedar in 2000]) but not for Linux. While a Mac would be OK, none of the manufacturers seems to offer scriptability or a command line interface which is essential for our task. Tape and HD backup are not an option - the data is already mirrored on RAIDs. Has anyone set up a similar archival system using Linux?"

Comments

Use an ipod

[biophysics]

You can take it to home/beach in the weekends.

One word: SCSI.

[yppasswd]

Just go with a SCSI jukebox; it should work fine on Linux
with "mt", "mtx" and some shell woodoo; ours did.
You might perhaps check if your vendor supports standard
SCSI commands, though.

And?

[bconway]

Tape and HD backup are not an option - the data is already mirrored on RAIDs.

What does one have to do with the other? Why are you insisting on CD backups when there are superior solutions available? It sounds like you're intent on duplicating a poor solution instead of examining the problem as a whole and finding the best solution for your needs.

Re:And?

[MrCool80s]

CD/DVDs sure seem like an ideal solution for making cheap (due to good frequency) backups for off-site storage. Excellent if the backups must be accrued and stored for longer than a week or two.

Re:And?

[j-turkey]

CD/DVDs sure seem like an ideal solution for making cheap (due to good frequency) backups for off-site storage. Excellent if the backups must be accrued and stored for longer than a week or two.

But pretty unreliable for long-term storage. If he's keeping backups for over 6 months, backing up to CD is usually a bad idea.

It would have been nice if the submitter had qualified why tape backup was not a possibility. At this point, we can only speculate why. (Does he/she need a read-only backup solution? Can he not afford tape? Is he ignorant?)

Re:And?

[Kz]

As pointed by another poster, there are situations where CD/DVD are superior to either Tape/HD.

but the existance of RAID mirrors have nothing to do with that.

a mirror protects you from hard disk failure, not against data corruption, for that you need a different thing.

also, the original poster uses 'backup' and 'archive' words. these are _totally_ different things! I think he means: i need archive, i had archive with CD/DVSs, i don't want tape/HD backup. but it seems he hadn't checked up the possibility of an HD archive.

I've recently replaced a wall full of DVDs (around 6000 discs) with a near-line HD archive system, at just 1.56$/GB total cost!

of course, for offsite backup of the archive, it still burns DVDs

Re:And?

[alienw]

I think DVDs are a hell of a lot more reliable than HDs (and I doubt you got good-quality HDs for that price). IDE hard drives in my experience have a lifetime of about 1 year before they develop bad sectors (and corrupt your data). Recordable media lasts for many years when properly stored (no large temperature/humidity swings, no light exposure, etc).

Re:And?

[DustMagnet]

I think DVDs are a hell of a lot more reliable than HDs

I find DVDs are much more reliable, once written. I have yet to find a long-term reliable DVD or CD writing solution. Yes, I can buy high quality drives and high quality disks, but I still get spindles with write problems. At home I just use cheap disks and find I'm saving money by tossing out spindles that don't work. It bothers me that sometimes I get no errors on writing, but get errors on reading.

Anyway, I have yet to find a reliable way to write CDs and DVDs. I'd love to use them more, but still use removable hard drives and tape instead.

BTW, IDE hard drives last a lot longer stored in proper conditions than they last during regular use. I have a set of removables that are close to 10 years old and not a single one has failed, while I have a RAID that's a few years old and half the disks have been replaced. Still, I never archive data on hard disks.

Re:And?

[Kz]

optical discs are really realliable when properly stored, unfortunately that means not being used. Here in Lima we get more than 90% humidity most of the year, so even in well ventilated, A/C indoors, optical discs degrade quickly.

the main advantage of HDs is that it's easy to check data integrity with checksums. of course, HDs do die, that's why now all those DVDs are the backup of the archive. And now that they're not used except when accidents happen, they can be stored properly.

Re:And?

[Anonymous Coward]

So how do you read that old MFM drive you used for archival?

Re:And?

[DustMagnet]

So how do you read that old MFM drive you used for archival?

I think I still have a computer that can read an MFM drive. A quick search shows, "Compaq began selling PCs with integrated hard disks using Western Digital controllers starting with their IBM-compatible Deskpro 386 in 1986." Ok, I guess I didn't make it clear. I don't use hard disks for archive, just data transfer. Final archives are normally on tape.

I have to admit you did catch me. My 10 year old removable drives are SCSI. We added removable IDE five years ago. I haven't had one of those fail either.

I still stand behind my claim that properly stored IDE drives last a lot longer than those used 24/7 inside computers. I never claimed they were a good way to archive data.

Perl!

[madaxe42]

Get a DVD burner, write a perl script which will once a week do a complete tarball of directories to be backed up, and burn it to DVD (tarred up, preferably, for space saving), and daily write a tarred diff of the directorys to be backed up to the DVD. You could also set it up to send you an email, or whatever, with a manifest.

Next silly question?

Re:Perl!

[Anonymous Coward]

Based on the links provided in the question, this guy is looking fox a box where he can stack in 100 CDs which the unit will automatically load/unload them as needed. If that's accurate, your answer is useless since it doesn't offer that functionality (which may be important to him for whatever reason). For example, he says he's backing up about 2 CDs a day, and it's probably not practical to send someone into the office four times over the weekend just to swap discs. Using DVDs instead of CDs would certainly mitigate the problems since he could go half a week without swapping discs, but that may not be workable either (week-long vacations by the sysadmin). Multiple drives would help more, but then you're adding both cost and complexity as well as probably breaking his current system.

I tend to agree that for a mere 2 discs a day an expensive CD library might not be the best solution for him, but maybe he has some weird needs he hasn't told us about that rule out plain old CD/DVD burners.

Re:Perl!

[Nos.]

okay, that will automatically burn, but then how does it automatically print/label the DVD?

Oh, and by the way, tarring a directory does not save space

Its not such a silly question if you actually try and answer it.

Re:Perl!

[Anonymous Coward]

"try to answer".

Re:Perl!

[Gudlyf]

"Next silly question?"

The questions is not nearly as silly as your hack at a solution. Since when does tarring files save space? I also would think that he'd need to make sure whatever he's burning to DVD didn't exceed the capacity of one disc, or else write something smart enough to write only what can fit on one DVD, then make some sense out of what goes on the next one, etc.

Re:Perl!

[swv3752]

A bit of script hack-fu will handle any of that. He probably also meant to zip the tarballs. Though GNU tar supports gzip and bzip2 compression as an extra flag.

Re:Perl!

[Anonymous Coward]

Tarring saves spaces, if you have tens of thousands small files. (remember allocation units?)

Google

[Pan T. Hose]

You need some simple script driving mkisofs and cdrecord. When you Google for cdrecord mkisofs backup, the first result is scdbackup:

Simplified Backup on CD or DVD for Linux
Purpose:
backup large amounts of data on one or more CDs or DVDs,
simple (therefore no excuse not to do the backup),
no special tool needed for reading the backup

Sounds exactly like what you need. There are many more tools like that. Good luck.

Let me tell you about the Rimage systems

[everyplace]

Recently at work I had to recommend an automated cd printing / burning system, and I went with the Rimage 2000i. We're a mac-only design shop in nyc, and needed the machine not for backup, but more for one-offs with automated labels, in a machine that was networkable.

One of the things on the Rimage website that's kind-of misleading (at least it was to me) was that it NEEDS a windows pc in order to share the rimage machine with other machines, like a mac. But once it's setup, the machine works wonders.

What's interesting about this machine though, is that despite the ridiculous setup hurdles, after it all works they provide a fairly decent way of writing your own scripts to control the machine.

The entire device uses xml files in order to handle job requests, and the client they ship it with is actually just a beast of a java app. But the xml files are used for the imaging orders, the production reports, everything. They also have a fairly extensive sdk that allows you to do pretty much anything.

I had an unfortunately difficult time setting this thing up, but the tech support (while their english was a little lacking) were actually incredibly knowledgeable. One of the things they told me was that almost no-one who buys this device uses the provided client. It is designed to be integrated into custom work solutions, so for you this might actually be appropriate.

If you're looking for a solid dvd archival device / printer that has an autoloading function and is fully scriptable, the Rimage 2000i (or any of their devices higher end than that one) could work.

Re:Let me tell you about the Rimage systems

[everyplace]

Sorry, I forgot one point. The fact that the entire production ordering system operates solely on xml file handling (to me) means that this is really OS independent if you write the front-end for it. The only hurdle really is a) writing your own frontend for linux and b) accepting that you need a windows box to serve the device from. I'm currently experimenting with making a web-based front end for this, but its too early to say whether or not I'll have success.

Xor?

[yppasswd]

If he says "Tape and HD backup are not an option", well,
I'd think he has considered all the other solutions.
There are cases where CD archiving is the only solution.
For example:
- Legal requirements of read-only media. My case.
- If archives must be guaranteed readable by common
hardware (and I mean COMMON, try buying a tape reader
in your favorite supermarket...)

Sure, backup on CD is a pain, but this was not
his question.

Re:Xor?

[karnal]

I don't know about you, but I usually am not sent to the supermarket by my company to buy a dds4 drive. We've got vendors for our equipment. :)

Re:Xor?

[bluGill]

Several companies make write-once tape drives. We use Sony AIT drives at work, but there is no reason to assume the others won't work. (I know DLT has a write-once mode)

Tape drives are common hardware for enterprize backup, more common that CD/DVDs. CD/DVD is consumer media, I it easy to show in court that they are not up to the other standards that anyone who must save data must maintain.

Re:Xor?

[seigniory]

HP has a line of WORM Ultrium tapes that satisfy your read-only requirement and provide all the performance and durability of standard tape backup.

http://h18006.www1.hp.com/products/storageworks/ wo rmdps/related.html

Re:Xor?

[superpulpsicle]

The only reason why tape drives are still common for enterprise backups, is because disk backup solutions were not affordable before.

Writing 40GB at 5MB a second is how tape manufacturers market to you. What they never tell you is how long it takes you to restore.

Re:Xor?

[Anonymous Coward]

I think that was the point. People who don't have vedors or the budget for expensive equipment may wish to read the media.

Re:Xor?

[Lord Kano]

Sure, backup on CD is a pain, but this was not
his question.

I don't think it's unreasonable to ask for clarification. The stated reason for Tape and HD not being options didn't make any sense.

If you offer someone a cup of coffee and they respond "No thank you, I'm allergic to tea", it's only normal to ask for more information.

It's most likely that the submitter has valid reasons for not being able to use HD or Tape backup, but s/he was extremely vague about why.

LK

Needlessly elaborate solution

[AndrossUT]

take an old linux box with a burner. Set up an automated backup burn... and use a LEGO Mindstorm setup to pull out the disc, slap an autmatically printed label on, and put a new, blank disc in. Sure it's not the most cost effective or efficient way, but it would be damn cool looking.

Re:Needlessly elaborate solution

[91degrees]

Not mindstorms, but you reminded me of this home made autochanger

Links on that page link to a lego version

amanda

[martin]

will backup to CD's no problem. www.amanda.org

Re:amanda

[gl4ss]

however, they're looking for an automated setup - not just a burning program.

they want the cd to pop out, and then some label machine label it and put it away.

Re:amanda

[martin]

all possible with the tape changer program..

Re:amanda

[tyen]

Notwithstanding the OP's implied reliance on their RAID array as their backup strategy being a fatally flawed approach, and tape backups for that RAID a potentially recommended course of action because archival grade discs cost more per gigabyte than tape for starters, recommending they use tape when they have obviously thought of it and rejected the approach might not be fulfilling certain business requirements they didn't share with us. For all we know, they already have an overwhelming, substantial investment into their current approach that eclipses adopting other media. Or they might be in an industry where regulatory requirements require sending off data on disc to some government agency. Or they really might be using archival grade discs and want the data protection assurance that comes with those versus tape; still cheaper than magneto-optical media for example, especially if they have a requirement to take daily backups off site and their data just happens to fit pretty nicely on about 2 discs a day.

For your amanda suggestion to be complete, you need to offer a solution that also automatically labels the tape cartridges. Generally this requires getting a tape library that has a built-in labeler, though I don't know how Linux-friendly these units are (I'm looking into Overland Storage offerings myself).

There are also other companies like the poster who wrote about his Rimage who need automated production of one-off discs and disc sets. The nirvana for these companies is something that can automatically burn, print, place in Tyvek sleeves, stuff into an appropriate sized envelope, weigh the envelope, print postage or express package delivery label, apply the label (or perhaps print directly onto the envelope, then stuff it), seal the envelope, and dump it into an appropriate bin for pickup by a postal carrier or express package delivery staff. Then customized discs can literally be produced as part of a normal workflow system at the touch of a button.

Re:amanda

[gl4ss]

maybe yes.

but a program doesn't become magically hardware that takes the cd out and writes on it.

(and he states that he doesn't want tapes, doesn't state the reason though)

Re:amanda

[Noksagt]

recommending they use tape when they have obviously thought of it and rejected the approach might not be fulfilling certain business requirements they didn't share with us.

He didn't recommend tape. He suggested AMANDA with optical media.

For your amanda suggestion to be complete, you need to offer a solution that also automatically labels the tape cartridges.

AMANDA supports barcode readers. It can also automatically print labels. If you have the hardware to automatically apply the labels to the media, you're more than set.

Outsource It...

[Anonymous Coward]

Probably end up costing you $0.50 an hour and your data will have the added benefit of being held off-site.

Rimage?

[Anonymous Coward]

I would not name my company Rimage -- I wouldn't be able to stomach the illicit alternative connotation.

For 2 CDs per day,

[popeyethesailor]

Hire a monkey. Or a college student if you want cheaper.

Re:For 2 CDs per day,

[R2.0]

Yeah, but monkeys require care and feeding. At least college students an wipe their own...oh, never mind.

Linux/Primera

[wonkavader]

I must say I have been horrified by my experience with a Primera Bravo box. Not because it's bad -- it's really great -- but because there are no linux drivers, and the Windows methods are absurdly awful.

To burn a disk, you go into a GUI and mockup (or just load) an image to print on the disk.

Then, you print it to file -- something.prn

Then you go into another GUI and set up the task, picking an ISO image, and the image file you just made, click here, click there, then burn.

That works just fine for 40 of the same disk, but if you want a different image on each (different date or different text, or the ISO filesize) you need to make each change manually (or with tags) and then print to file and then set up each task.

Man.

In unix/linux, or with command-line tools for windows, even, that would be:

create_postscript_with_substitutions [inputs] > printerfile.prn
burn_image_and_print isofile printerfile.prn

Done. You'd be able to do everything this guy wants and more with 10 seconds of typing. You'd be able to automate processes. And it's not hard. Primera's been selling this stuff for years, and yet, no Linux support, and no command-line support.

If this had Linux support, or even DOS command-lines, I'd recommend it to everyone I meet. As is, it's an anchor.

Splitting Discs

[Wiwi Jumbo]

I'd like to do (somewhat) automated backup's as well... tho more on a homesize scale...

I'm trying to back up my pictures and my mp3's on a regular basis, but at 15 Gig's for one and 56 Gig's for the other even putting them on DVD can be a pain.

Every app I've used (at least on XP) can automatically "split" the files over multiple discs but they all use their own format for it making recovering the file difficult if not impossible with out the original program.

Is there anything that will split on the file structure so I can just read the files like they were burned normally?

Right now I keep adding and removing files from the "to burn" list and try and get as close to each disc limit and then do the same for the next disc... makes me put off my backups for longer then I should....

Re:Splitting Discs

[biglig2]

Wouldn't that be easy to write e.g. a perl script to do that? Get a list of the file sizes, work out where to make the split, and shell out to the burner software?

Re:Splitting Discs

[Wiwi Jumbo]

Sorry, I don't write perl scripts. /me = "Windows Weenie" ;)

Ack Thpt

[binaryspiral]

Burned DVDs and CDrs have piss poor shelf life for archival material. RAID arrays can disappear at the snap of a raid card's whim (yes - it happened to me last week).

Please - re-evaluate your solution before you actually need to recover data or get (gasp) audited and need every single file to work 100% for the auditors.

Recordable optical media sold retail are not up to the standards of archival requirements of most governing bodies (like the SEC). There are optical WORM drives that are used by medical data centers and hold only 30GB on a huge platter. And most of those are getting retired for other methods.

(this is not a plug, it's just what works best for us)
There are many tape solutions like Exabyte's VXA-3 with 160GB native storage space on an $80 tape. Granted $/GB is higher than a DVD-R - the tape will not let you down. The tape is equal to ~35 DVDs and writes at 500MB+/minute.

We have an Exabyte autoloader with 10 tapes on a weekly rotation - and it was as close to heaven when we needed to restore a server. We also backup >400GB of data weekly from a few of our database servers - and need it to be there when something fails.

The entire rig will set you back about $3000 including tapes. This will give you over a TB of backup space. And the tapes are archival ready.

For your use - 1-2CDs/Day each tape will last you about 3-4 months. But it will also allow you to rotate your backups off site and give you much better utilization of space and much higher chances of recovering this data years down the road.

Please reconsider your backup solutions... if it's worth saving at all - it's worth being able to get it back later.

Re:Ack Thpt

[Noksagt]

Granted $/GB is higher than a DVD-R - the tape will not let you down.

And the tape can be ERASED and REWRITTEN, which may reduce media costs considerably. Most tape media is cheaper per gig than most DVD-RWs.

Re:Ack Thpt

[Trixter]

writes at 500MB+/minute

Sorry, I think you meant 500mbit/s, not 500MB/s.

Re:Ack Thpt

[binaryspiral]

No, I meant 500 Megabytes per minute. Some research may have helped you respond.

VXA says you'll get 43GB per hour, but in a real life you'll see ~500MB per minute which works out to about 30GB/hour. With that speed, we have backups dumped on to a 1TB NAS then written to tape. It allows the backups to flow as fast and smooth as possible to tape without having to worry about network congestion.

Re:Ack Thpt

[Grandmaster Mort]

If you're referring to those Super DLT tapes that hold 160 GB uncompressed and 320 GB compressed, then I know which type of data tapes you're referring to. I once worked in a data center that stored some of its backups onto those Super DLT tapes, and those tapes were kind of flimsy with regards to the latch on the ends of the tape media itself that was used by the tape robots to pull the tape out of the spindle. Those plastic latches would break off a lot. The data medium itself may be more reliable than CD-R and DVD-R, but the tape is useless after those plastic latches break off.

Re:Ack Thpt

[binaryspiral]

Similar to SDLT, but not quite.

SDLT Tape vs VXA-2.

VXA-3 will exceed the capacity of the SDLT and will be the exact same size, using the same tapes and autoloaders. All I have to do is swap out the drive and gain double my storage.

Microtech

[bill_mcgonigle]

Check out Microtech's ImageAutomator line.

It's windows-based but I've set up a few of them to pull their data from a Samba share. Think of it as an appliance. I wrote some software on the linux side to control it - they have specs available for their file formats if you want to explore writing your own. That software has probably done about >30K CD's so it's definitely workable.

Pioneer DRM 3000

[Ayanami Rei]

Product Brochure

This is probably overkill, but it is a really cool piece of equipment, and it doesn't rely on shitty windows software to do it's job. Unfortunately it costs $10000 fully loaded with 4 DVD-RW drives.

Re:Pioneer DRM 3000

[Brento]

We've got one of the 7000 units, bought during the heady dot-com days.

The Windows-based management tool sucks, lemme just say that. But the cool part is that it can share a virtual drive on the network, and then when people make folders off the root directory, those folders are archived as DVD's. So you can let your users set up their own folders and archives without having to manage anything. The system will automatically burn them when they reach DVD capacity, or when you specify to do it in the management tool. If you use these virtual folders in combination with your own apps, you can do some slick stuff, like making your own backup utility that copies folders to the virtual drive.

The hardware itself has a few shortcomings. Don't plan on swapping a lot of disks in and out of the changer. Whenever the 50-disk cartridges are changed, and new DVD's are put in the cartridges, the changer has to read every single DVD to build the table of contents to know which files are in which drives. It takes an agonizingly long time to read 50 DVD's.

While the cabinet can indeed house a lot of DVD readers, there's only one robotic arm to actually change the DVD's. In our company, somebody originally planned to use it to serve images over the internet - problem was, the robotic arm couldn't swap DVD's fast enough to keep up with demand, so we switched to a hard-drive based system instead. The system works well if you don't have a lot of simultaneous reads from different DVD's, but if your usage scenario involves each user changing DVD's more than once every 2-3 minutes, look elsewhere.

Let's say each user plans to look at a series of files on a DVD, and then change DVD's and look at other files every 60 seconds. And let's say you're planning for ten simultaneous users. That means the system will need to switch DVD's an average of once every 6 seconds. But the robot's "maximum disc change time" is 9 seconds. You might be able to skate by in this scenario, but we weren't able to - especially when users wouldn't adjust their work habits to looking at the same DVD - instead, they would jump around. Your client applications will start timing out because Mr. Roboto can't change DVD's in time to service their requests.

Otherwise, the hardware is nice and reliable. Looks impressive in a data center.

RAID != A Backup Method

[seigniory]

RAID isn't going to help you if your building goes up in flames.

RAID isn't going to help you if a file is deleted accidentally.

RAID isn't going to help you when someone comes in and steals your boxen.

In 15 years I have never, ever, EVER recommended that someone back up to optical media as their only recovery method. DAT drives can be had in the sub-$200 range, and the $/MB cost is cheaper than DVD media - and much more reliable.

I realize that this doesn't really answer your question, but it's an important point that shouldn't be overlooked.

Re:RAID != A Backup Method

[ErikZ]

I don't know about you, but all my data will be fine when my building goes up in flames.

Rsynced to a remote RAID device. Accidentally delete a file? I suppose setting up an undelete function would work.

And as for stealing my boxes. Get a guard dog already.

Re:RAID != A Backup Method

[seigniory]

Find an "undelete function" that will recover your data when a user "accidentally" runs a SQL package against the wrong data. Or one that will roll back to a version from 3 days ago when a (l)user in your company realizes that someone else got rid of a chapter that took 3 weeks to write.

Recovering a deleted file != recovering altered data (which is far more common).

As for your diaster recovery / business continuity preparedness (and your assumed desire for continued employment), you might want to take a DR class.

I used to fall into the same trap I see you heading towards - cool technology is very cool - I'm a huge geek for that kind of thing - but there's a time and a place for everything, and it's rare that one solution is the best or perfect one. Usually I find that it's a conglomaration of some very cool shiznit, and some older time-tested methods. Old and new guard working together to satisfy the suits and the geeks. :-)

Re:RAID != A Backup Method

[ErikZ]

Find an "undelete function" that will recover your data when a user "accidentally" runs a SQL package against the wrong data.

Shrug. I keep a nightly SQL dump on my RAID. That's not what the undelete function is for. You said "When a user accidentally deletes a file." Not "When a user totally screws up and pillages the system."

Or one that will roll back to a version from 3 days ago when a (l)user in your company realizes that someone else got rid of a chapter that took 3 weeks to write.

And they don't have a local copy ANYWHERE? I find that scenerio to be improbable.

As for your diaster recovery / business continuity preparedness (and your assumed desire for continued employment), you might want to take a DR class.

Although I appreciate that you think I'm employeed as an admin, these are just my home machines and my commercial website.

I work as a systems operator. And I have to deal with admins who think the tape backup is actually reliable. As the time windows come and go, it usually takes several tries (weeks) to get things back to the way they are.

A backup is not a replacement for users handling their own data in a good manner. I'd hate to be the guy running around every day trying to fix someone else's screwup.

Are there DR classes that don't require you to have expensive backup solutions?

Re:RAID != A Backup Method

[seigniory]

re there DR classes that don't require you to have expensive backup solutions?

DR (Disaster Recovery / Readiness) is really more a state of planning and procedure and is (or shoudl be) independent from technology. Sure, tech is (obviously) involved, but should never be the driving force.

From a higher level, take a good look at what you'd need to do if, in a worst case scenario, you lost everything - pretend that "Poof!" it doesn't exist anymore. If you can figure what it takes to get 100% of your shiznit back, you can easy figure out what subset of that you need for a less serious incident.

A warning, though - don't downgrade the technology just because the incident is less serious. You'll end up burning yourself in the end.

And they don't have a local copy ANYWHERE? I find that scenerio to be improbable.

I find it unlikely that users keep local copies of 2TB databases on their own. :-) Since this doesn't seem to be the case with you, and you trust your users to be responsible for their own data, that's up to you. However, it's always good PR to have the backups anyway - CTA (cover their

Shrug. I keep a nightly SQL dump on my RAID. That's not what the undelete function is for. You said "When a user accidentally deletes a file." Not "When a user totally screws up and pillages the system."

Well, kinda - a user should never be able to pillage a system, but I have seen instances where a botched script caused errors that didn't get picked up for a few days and only a small subset was "broken".

Anyway, it's always better to be safe than sorry. Proper backups & DR can be more time consuming and expensive than a cursory review may imply. But if you stand to lose $500,000 over the next 5 years if you had a major issue, and you didn't spend $15k to make sure that $500,000 was "in the bank" hindsight won't be plesant to look at. :-)

Re:RAID != A Backup Method

[ErikZ]

(blink)

You work in enviroments that I've only heard of in legends.

I work in places where "The tapes are going bad, and they're expensive to replace, and the tape drive is years out of warranty, fire the operator!"

Man, a proper DR budget. If they could afford that, they could afford to hire people instead of hacking their budget and getting temps/contractors.

Ah well. I've got to get back to working on my local RAID array. The array went corrupt, and I'm hoping upgrading to the 2.6 kernel will help.

rtfm

[SHEENmaster]

http://unixhelp.ed.ac.uk/CGI/man-cgi?split
http://unixhelp.ed.ac.uk/CGI/man-cgi?cat

You can get them in XP through Cygwin.

I've had something lke this running

[wakejagr]

on linux for a few years now. It's actually a fairly quick hack that I've never gotten around to fixing up. Each of my computers runs a backup program at 4am on monday morning, and saves a tar.gz of the dirs. The tar.gz is owned by an account that is not used by anything else. Then, at 5am, my main computer (the one with the burner) uses scp (with dsa id keys) to download the tar.gz files. It then mkisofs's and cdrecord's, so that I have a CDRW backup of my data.

The programs are pretty crap (only very basic error checking and file size checking are implemented), and I often forget to put a CDRW in the drive, but it works pretty well. The other main problem for your situation is that you need to burn multiple CD's per backup. It seems to be that a second burner (or a DVD burner) would work better than trying to get a system that can deal with multiple discs.

From my point of view, you want a dedicated machine to do this. A network writable SAMBA drive (limited to the maximum size of the burnable media?) would be my choice. Then, you can use generic backup software to backup neccessary files on whatever OS's you are running.

Tiger Automater

[Anonymous Coward]

This is exactly what Tiger's (the new MacOS out next month supposedly) new Automater feature is for. The original demo of Automater involved someone setting up an automated DVD burn of a slideshow movie every so often.

Granted, this doesn't solve your problem TODAY, but it makes it ridiculously easy within a month or so.

http://www.apple.com/macosx/tiger/automator.html

Re:Tiger Automater

[Anonymous Coward]

"...we are looking into a new backup solution to automatically burn and print CDs or DVDs."

Please tell us how Automator (it's with an 'o') is going to print/label the DVDs. And please tell us how Automator is going to load and eject DVDs automatically. I can't find anything about this on Apple's web site.

cost of drives?

[beegle]

Depending on your needs, you might be able to get away with a handful of DVD burners. If, for example, you know that a backup will always take less than 3 DVDs, buy 3 or 4 burners. Then, make your backup, split it, and write a piece to each burner.

Spending a few hundred dollars on burners may be better than spending a few thousand on a robot.

RAID1 Array as backup

[stefanlasiewski]

There are many comments about how RAID isn't a backup scheme. While that's mostly true, I've known some shops that actually do use a RAID1 array as a backup scheme.

Not sure of all the details, but it works like this:

1. Setup 2 disks with RAID1 mirroring. Designate one as the backup disk.
2. Copy files to the mirror
3. Once a week, unmount the backup disk, take it out and insert a new disk.
4. Turn on the machine. When the array comes back online, the disks sync.

I'm still trying to figure out if this is a wise idea, but it's low-cost and effective. 1 Device and 4 decent-quality 200GB disks will cost you what-- $1200? How much do 200GB tape solutions cost now?

Anyone hear of some similar solutions?

Re:RAID1 Array as backup

[binaryspiral]

That's a gheto backup! But man, it would work.

Sure puts your drives and raid controller (or CPU if you're running software raid) through hell for a few hours.

I hope nobody plans on using it for anything :)

Re:RAID1 Array as backup

[hackstraw]

That is technically not RAID. Its kinda of a hack using RAID software. I do the same thing with a script that I call 'raid1.pl', but its not a RAID system at all. I simply use rsync to sync up two drives at night.

The difference between this and real RAID1 is that with RAID1 you do 'rm -rf /something/important' and its gone from both disks (or an overwrite, or anything). In my experience, if a backup is done nightly, even if a whole day's work is lost, it is preferable to having potentially YEARS of data lost, and it is easy to do with a typo in a script or something. Its great to have something like:

rm -rf /$SOME_DIRECTORY

and $SOME_DIRECTORY is not defined. Granted I would never do this, but some inexperienced scripters do.

Redundancy vs. Archival

[powdered toast dude]

Please don't confuse redundancy and archival needs. To simplify: Redundancy (e.g. RAID) protects you against hardware failures. Archival protects you against "oops I deleted it" failures. Both are important.

$0.02,
ptd

This device claims is supports Linux

[LWATCDR]

http://www.akfentertainment.com/vcic/encore.htm

How well it supports is anyones guess. We have an old Cedar CD duplicator that I would love find the protocal for. The strange thing is the robotics are controled via SCSI.
Anyone know how to capture SCSI data?

rimage 2x burning rant

[tweedlebait]

2 years ago-
Worst tech support experience I've consistantly had was with rimage and it lasted over 1 month!! All I was trying to do was replace a broken drive they had assured me was standard way back in the presale days. They had no replacements, wanted me to buy a new unit at $6k. My old software which was great in its earlier days had gone through some 'improvements' post veritas buyout and wasn't working anymore. Rimage had a new $$product that was supposed to replace all these features and more. I demoed it all and it was horridly inefficient- IIRC- 4 cd-r's took 3 hrs to make 4 cds, nevermind the crashes. The old unit with 4x drives was much faster. Then they too told me nobody actually uses this out of the box, they just buy the dev kit and customize it. Then they recommended I do the same for my team of developers. Sounds like a handy tech center stock answer. Guess what. I dont have a team of developers! I suggested they use their team of developers to make their own stuff work as advertised.

After much strife they finally provided me with someone at another company who could provide me with a copy of their customized firmware so i could replace my drives with identical ones. The robotics finally gave up the ghost a year later.

My replacement for it was from a less known company handled through discmakers. Discjuggler runs it with a special autoloader product called imagejuggler which works very nicely (you can also use DJ.NET and they have a web interface too. They could improve a few things but all in all robotics are top notch and simple, no weird firmware, replacable (upgradable!) drives (as long as your burning app supports it, which dj handles many) and I can easily burn different images simultaneously. Furthermore both discmakers and DJ support was great.

http://www.discmakers.com/duplicators/products/eli te_series.asp

and yup its only for windows. but at least you get browser ui.

want script for mythTV dvd output

[tod_miller]

Copy shows I tape onto a DVD with interface done with script. Any possibility? That'd be a good ask /.

Rimage DiscLab with SDK API

[ericrolph]

Rimage Developers API

The Rimage Client API :
Tight integration with your application
Order and system status in real time - no polling
Fault tolerance
APIs in C, C++, VB, Java, .NET
Industry standard XML support
Multi-platform support

Rimage Network Publishing Multi-Platform :
Monitors a network folder for text file orders
Generates XML from file orders and submits to Rimage system
Multi-platform support

Based on poking around in (our company owns the DiskLab) the SDK, it'll do almost anything you could possibly imagine. I'd call Rimage up and talk to a developer -- they're friendly and quick on advice.

Informative? Not.

[cjsnell]

If you read the friggin' article sumary, you would see that what is needed is an automatic CD-R loader, not just the software to burn CDs.

7.5 Terabytes archived on DVD

[dhempy]

We're using DVDs to back up 5 2TB NAS servers of video files. Our hardware is a 4-burner Flexwriter, similar to this one: http://www.amtren.com/products/sa4.shtml

The drivers come from Padus. We don't use their GUI much (DiskJuggler and ImageJuggler)...instead we primarily build jobs and submit them from a another machine via a perl script and their command-line tool, pfcnet. The level of automation we needed for this project simply required scripting. I haven't looked into their DJ.net product...it may have good potential.

If you like stats, we have the capacity of burning about 400 DVD's per day, but rarely hit it. The input spindle holds 200 disks. We don't often make it through the night without a failure that halts the burning. Don't get me wrong...the machine is a workhorse, but it should not be considered a lights-out unit. After we got our system built and pretty well tuned, we could pretty reliably back up one NAS (1200 files, 2 TB) in about two weeks. This includes a fair amount of idle time, especially on the weekends.

Also note that we're burning one file per disk, 60% are 1GB and 35% are 2GB. Project specs indicate that we only want one video (file) per disk. These files will never change (many are encoded videos 30 years old!), and must be accessible both by set-top DVD players as video and as ready-to-edit .mpg files on a computer. Fortunately, most modern set-tops will play .mpg files from a non-vide DVD, so we don't have to author these disks.

So far, we've burned about 5000 disks holding about 6TB of data. Disk failure rate is running around 5%. The Padus drivers do a good job of reburning failed disks, but with this many disks and the automation behind it, some still fall through the cracks. As such, we're printing a barcode on each disk, and all disks will be scanned at least twice so we can catch the missing ones.

Our biggest bottleneck initially was network speed. Over 100 Mbit ethernet, we could not keep four burners busy at 4x. We attempted to upgrade the machine to gigabit, but design limitations forced us to choose between fast network or a fast hard drive, and we decided the hard drive was more critical. Part of this was due to a high failure rate burning at 4x, and anecdotal advice that 2x disks have better shelf life than 4x.

After backing down to 2x burn, 100Mbit is adequate to keep the burners working full-time. The bottleneck now seems to be CPU utilization while burning and network transfers are both occurring.

All of the machines (other than the NAS's) are Windows boxes.

We recieved good support from our vendor (Amtren) and outstanding support from Padus. Both worked closely with us to resolve several software and hardware issues. Both admitted that we were pushing the system harder than any of their other clients and were eager to see us succeed. Extra special kudos go to Fred P. at Padus! :-D

...

In a related project, we will be producing video DVD's on demand. Customers can choose one or more episodes. Those files are authored into a DVD, complete with an onscreen menu with titles, semi-custom graphics, a custom printed label on the disk and a mailable jacket sleeve. This process is up and running now, with no hands-on intervention, but hasn't been launched yet. I've written a perl module that facilitates this, weaving the Padus software in with dvdauthor and ImageMagick to provide an end-to-end solution.

...

That's all the trivia I can think of. I'm happy to discuss the project with anyone who's interested.

-dave

Oh my god.

[Ayanami Rei]

Who the hell thought that using a DVD array would be better than a disk array/SAN? Just the thought of a mechanical anything moving constantly to keep up with random access makes my head hurt.
If you don't have at least 16 DVD readers it's pointless. You'd need at least that just to saturate SCSI bandwidth. And all those moving parts. GAASAAAH.

And $10000+ when 1TB RAID would cost the same at the time of purchase... double GAAAH.

*Cough cough* *wheeze*

I sure hope they fired the guy who thought of that brilliant idea. ;-)