Slashdot Mirror
Ask Slashdot: Networked Back-Up/Wipe Process?
An anonymous reader writes "I am required to back up and wipe several hundred computers. Currently, this involves booting up each machine, running a backup script, turning the machine off, booting off a pendrive, and running some software that writes 0s to the drive several times. I was wondering if there was a faster solution. Like a server on an isolated network with a switch where I could just connect the computers up, turn them on and get the server to back up the data and wipe the drives." How would you go about automating this process?
Don’t know of any off the shelf software that does this, but should be easy to homebrew if you have the available skill set.
At the very simplest, you could probably build a custom livecd linux distro to automate the process after plugging in the machine and inserting the CD/pendrive. It’s not as complicated as it sounds if you base it off an existing livecd distro!
More complex, you could do PXE if the boxes are capable/configured for it (if not, probably more effort to change the bios settings than it would be to plug in the CD).
You’re probably content just with the backed up files, but I’ll also throw out there that I’ve found a very effective way to back up old machines/drives is to convert them into virtual disk files. Lets you boot up the old machine in a VM and poke around should the need arise. (disclaimer: I’m a dev not a sysadmin, so this is purely from “at home” experience).
Then don't automate it.
Nobody has demonstrated the ability to recover data after that outside of a carefully controlled lab.
Microsoft User State Migration Tool + Microsoft Deployment ToolKit + Sdelete http://technet.microsoft.com/en-us/sysinternals/bb897443
You should be able to backup the profile, load the OS and run a zeroing delete on all "empty space" on the drive.
EA David Gardner -"... but the consumers have proven that actually what they want is fun."
There are two commonly used techniques to the wipe process.
In Europe the preferred method is to fold the paper in half before wiping. In the US the preferred method is to scrunch up the paper in a ball before wiping.
Check whether the PCs you are wiping did a number one or a number two. Male PCs do not need wiping for a number 1.
"That's the way to do it" - Punch
As for a whole problem solution, I think you will need to do a bit of DIY. But just a note on the wipe process. Just writing 0 to the drive repeatedly will not ensure all the possibly sensitive data is non-recoverable, you really need to write random 1's and 0's at least 3 times to each bit of the drive. For that there is no better program than Derek's Boot And Nuke (DBAN) that I think is available as a liveCD and is available to several distros, including The Ultimate Boot CD (UBcd) and that may be a good place to start for a single boot backup, wipe solution. if you can write a shell script that can run from a pen drive while UBcd is in the CDbay.
Take the hard drives out of them, label them, and stick them in a closet.
I would look at FAI or kickstart. For FAI a pretty early hook to backup and wipe.... for kickstart a %pre script.
Of course, if you are working alone, and don't know how to configure DHCP/NFS etc.... it may take you a couple of days just to get the basic setup going, as they can be very finicky, but the quickstart guides out there should generally be able to get you going. If all goes well, you could be working on your scripting in a couple of hours, if not..... well....I hate troubleshooting NFS.. (and don't forget to check your IPTables setup if you are having trouble getting it working...amazing how much better NFS works when its packets are not being dropped.
Overall, I like FAI better than kickstart, but thats probably because I have used it less and those early stages (DHCP/NFS mount) are hard to troubleshoot with kickstart since stage2 (and thus a shell with which to troubleshoot) isn't available until that works.... though.... you probably don't have the same constraints I do and can just switch USB keys and boot off a fully functional system to test poke around.
"I opened my eyes, and everything went dark again"
There isn't a whole lot to optimize in your process. Backups and wipes take time. One thing that could save you a step is to run the backup from the pen drive. That would allow you to script the entire process, such that you only need to boot off the pen drive, preferably have it cache itself into a ramdisk and start the script automatically, then move on to the next box. That would bring the whole process down to maybe 2 minutes per box.
Having ghosted a bazillion machines this way, it's monotonous but if you create 4-5 of those pen drives, you can do a bunch in parallel.
-Billco, Fnarg.com
Acronis or Ghost Enterprise can do this with every PC on a single network segment.
[RIAA] says its concern is artists. That's true, in just the sense that a cattle rancher is concerned about its cattle.
Everyone else (anyone else?) will answer the automation question, but if you're ever done a PXE based linux install, you're about 99% of the way there.
The mystery I have, is where are you going to store "several hundred" drives worth of backups? And who or what is going to back up and maintain and store and recover the backups?
I'm guessing the best answer is open all the boxes, remove the drives, install new blank drives, all done? Given the cost of storage and admin time, this might even be the cheapest solution.
If this is a forensics issue, its a heck of a lot simpler legally to stuff THE drive in a evidence bag and buy a new one, rather than try to explain how your image is a true image crypto signed so it wasn't altered after it was signed, except how do you prove it wasn't altered before it was signed, blah blah blah.
Are you talking about backups where you only store relevant user "my documents" type data which might be practically nothing, or merely all files on a stereotypically mostly empty drive which would be at most a couple gigs, or a full bit for bit forensics dump of hundreds of 1 TB drives?
There's a big difference between "it all fits on a single USB attached consumer grade 1 TB drive" and "We're gonna need multiple racks of multimillion dollar NAS to hold all the images".
How valuable is the data? If it leaked would you lose PCI / CC / HIPPA / SOX stuff and its the end of the world or at least your corporation and job, or is it just a university computer lab and the most valuable/sensitive thing is a couple rickroll videos and some lolcats?
What do you intend to do, if anything, with the backups? The simplest / cheapest / most efficient way to store backups might involve just throwing the machines in a rented storage room. Climate controlled if possible. You can rent a heck of a lot of storage space for a long time for the cost of a couple hundred hours of admin time.
Finally whats your liability? If for example, one doesn't boot due to hard drive failure or whatever, are you shipping it to one of those $10K data recovery places, in other words you actually care, or if you lose some, eh, whatever, it was just a "nice to have"? If you can lose one, can you lose all of them with the same "eh" attitude? If your liability is significantly lower than your costs, your best plan might be to skip the backup and destroy the drives.
In summary the problem isn't how to "transfer" a couple hundred terabytes, that is a long solved question, no big deal. The unsolved problem is how to store / collate / search / backup / distribute / secure a couple hundred terabytes.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Looks like you forgot the verify step. md5 the hardware drive, md5 the image, they better match (bet they occasionally don't!). Also if you're taking a bare image, you don't mount the drive, you just copy the raw partition. If you mount the partition, then you have "issues" if the filesystem is semi-corrupt, was powered down while active, etc.
Also you forgot your exception process/monitor/procedure/whatever. At least some of these "hundreds" are not gonna spin up, are gonna barf out read errors on obscure corners of the FS... How much is management willing to spend to "recover" the data? There are places out there willing to take 5 figures to recover bad drives, and their success rate is not 100% so you need a plan for that too.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
That's perfect for the wipe, but he also needs backup.
PXE boot to pxelinux for a nice menu, then choose the backup (e.g. Clonezilla), throw a sticky note on the computer to indicate state, then reboot after the backup and choose DBAN.
http://www.linuxjournal.com/magazine/pxe-magic-flexible-network-booting-menus
A clever setup would keep track of which machine is in which state and hand out DHCP options accordingly, making the menu unnecessary.
With that many computers, backing up the drives to a ZFS volume with deduplication enabled is probably worthwhile (but make sure you have at least a GB of RAM per TB of disk).
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
That will make the backup a lot easier.
Unless you have a Lot Of Time to Test this BEFORE HAND, you could easily end up with an automated screw-up-the-back-up and nuke-everything system
This might be the best (only?) justification for buying non-free beer non-free freedom software I've ever seen, because you can intentionally buy the cheapest cruddiest non-working commercial software out there, then when all the data is lost, you don't have to maintain, backup, search, restore and otherwise admin the images for eternity minus a day, and you can blame the commercial software provider instead of yourself... Everyone, especially in management, knows commercial software just doesn't work sometimes and its no ones fault. Heck you could just skip actually imaging and wiping the drives, blame the software, and spend the new spare time playing skyrim.... Hmmm.... I think we're on to something here.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
FOG is a PXE cloning solution. http://www.fogproject.org/ Install FOG and storage where you want backups, setup PXE IP on network, and input all MAC addresses you want backed up. Through web interface to clone all. When done backing everything up, put a .img file of DBAN on the FOG server. http://www.dban.org/ Configure it in the FOG PXE boot menu, and make it an option but NOT default. Add appropriate start up flags for the level of wiping you want. Restart all computers you want to wipe, and select wipe option after PXE boot menu comes up.
I suggest you set that option with a password, since it will be available on all computers, not just the one's with the MAC address since only the FOG boot authenticates to MAC, not DBAN.
I8-D