This incident still gives me nightmares...
I used to be a 'cocky' sys admin for a UK bank, at a site where all the prog development went on (and that bank spent a fortune on these kind of projects).
Most of the code and development was then stored on Compaq servers running Netware - which also had 2Gb Archive Python DAT backup units attached to them for what I presumed was backup.
The time came to re-organised the partitions, and increase the amount of available space to the project - easy!! Just:
1 backup for 9 hrs
2 nuke the RAID5 array,
3 add disk and create a larger array,
4 restore!
Ha -if only.
Steps 1-3: piece of cake (feeling smug)
Only hiccup - Half way into a restore using ArcServe (Noooooooo!)... It starts going on about bad blocks on the tape.
Remember - I've nuked the array. The only data available now, is on tape...
Damn DAT tapes... feeling panicky now... - not so smug
Put a cleaning tape in the DAT unit,
Clean drive
Restore
Should be simple eh? Not quite... All alone - on a Saturday morning - and time is ticking...
So after about 10 attempted drive cleanings - the tape is bad - it must be...
So I go back to the previous days tape...
Same problem, double the panic!
So I go back 14 days worth of *backups*. Nothing. Nada zip.
I grab another DAT unit from another server
Same problem - time is ticking... and it looks like I have lost the data and code data for the whole project...
12 hours later and combos of 14 tapes with 5 drives, and I'm a snivelling mess...
I finally make the call to my boss... He listens carefully and comes in to assist (aka *supervise* and more panic to the whole situation).
Sunday night, and we make the call. The tapes have been fragged by the faulty backup unit. We make a call to OnTrack data recovery. The guys drive up on Monday - grab the tapes, read them into their recovery unit, and overnight spit out all the data except for 100Mb out of 2Gb...
From that day onwards, I am paranoid about DAT units, and I verify EVERY backup I make (as well as copy the data to HD)...
Thank goodness you can now expand an array just by adding a disk, and telling the HD to re-stripe.:'-(
Ed
Slashdot Mirror
This incident still gives me nightmares... I used to be a 'cocky' sys admin for a UK bank, at a site where all the prog development went on (and that bank spent a fortune on these kind of projects). Most of the code and development was then stored on Compaq servers running Netware - which also had 2Gb Archive Python DAT backup units attached to them for what I presumed was backup. The time came to re-organised the partitions, and increase the amount of available space to the project - easy!! Just: 1 backup for 9 hrs 2 nuke the RAID5 array, 3 add disk and create a larger array, 4 restore! Ha -if only. Steps 1-3: piece of cake (feeling smug) Only hiccup - Half way into a restore using ArcServe (Noooooooo!)... It starts going on about bad blocks on the tape. Remember - I've nuked the array. The only data available now, is on tape... Damn DAT tapes... feeling panicky now... - not so smug Put a cleaning tape in the DAT unit, Clean drive Restore Should be simple eh? Not quite... All alone - on a Saturday morning - and time is ticking... So after about 10 attempted drive cleanings - the tape is bad - it must be... So I go back to the previous days tape... Same problem, double the panic! So I go back 14 days worth of *backups*. Nothing. Nada zip. I grab another DAT unit from another server Same problem - time is ticking... and it looks like I have lost the data and code data for the whole project... 12 hours later and combos of 14 tapes with 5 drives, and I'm a snivelling mess... I finally make the call to my boss ... He listens carefully and comes in to assist (aka *supervise* and more panic to the whole situation).
Sunday night, and we make the call. The tapes have been fragged by the faulty backup unit. We make a call to OnTrack data recovery. The guys drive up on Monday - grab the tapes, read them into their recovery unit, and overnight spit out all the data except for 100Mb out of 2Gb...
From that day onwards, I am paranoid about DAT units, and I verify EVERY backup I make (as well as copy the data to HD)...
Thank goodness you can now expand an array just by adding a disk, and telling the HD to re-stripe. :'-(
Ed