Slashdot Mirror
Ask Slashdot: It's World Backup Day; How Do You Back Up?
MrSeb writes "Today is World Backup Day! The premise is that you back up your computers on March 31, so that you're not an April Fool if your hard drive crashes tomorrow. How do Slashdot users back up? RAID? Multiple RAIDs? If you're in LA, on a fault line, do you keep a redundant copy of your data in another geographic region?"
Simple. Redundancy backup.
It's a raid.
The backup gem( https://github.com/meskyanichi/backup ) + an dedicated server + some cron processes.
I back up my important questions by submitting them to Slashdot. If I ever need to find my question about backups again, it's right here on the front page EVERY DAMNED DAY.
Apple hate aside, time machine is an amazingly excellent backup system.
It backs up to a Netgear Readynas configured in RAID 5. Hourly, daily, weekly backups. I've never lost anything thanks to this great system.
In linux I try to approximate this with BackupPC.
http://backuppc.sourceforge.net/
It is really an excellent piece of software, though no where near as refined of course. You pretty much only get daily backups though since the kernel in linux does not track filesystem changes so hourly backups would be very prohibitive.
It's easier to fight for one's principles than to live up to them.
With a loud beeping noise.
William of Ockham had no beard. The most likely explanation is that it was chewed off by squirrels every morning.
No, really, this begs the question. Do you back up? (Captcha is "paranoia". Are you trying to tell me something, Slashdot?)
Can we moderate this article flamebait?
Change is certain; progress is not obligatory.
I currently sync my files across three computers, each of which does a time machine backup. The files are also backed up via Jungledisk to Amazon S3. Occasionally I do full-disk images of things.
Files that would be inconvenient to lose, but which are not irreplaceable, are stored on a Drobo (redundant drive enclosure). This includes, for instance, my music library which could be reripped from CD.
rdiff-backup over ssh to a trusted server in a different country.
duplicity (rdiff-backup's encrypted brother) if the server is not trusted.
Both are network friendly (use rsync algorithm) and support incremental backup by default.
Between 3 active computers I use, there's enough redundancy since they're rarely in the same place. SpiderOak manages absolutely, completely vital stuff (currently my thesis drafts).
But there's no real, constructive and useful pattern to it yet. The problem is less backups and more change management. Keeping copy-on-write sane on Windows is difficult, and migrating my servers XFS partition to ZFS is problematic since I need just tons of storage to do it which I presently can't afford.
The issue is far less "backups" and more "making them meaningful". Backing up is useless if I overwrite the media with the important changes, or it takes forever to dissect a working copy of the data.
My weekly backups: something like:
0 0 * * 0 /home/me/backup.sh
#### backup.sh ##### /dev/null
cp -r home/me/*
I haven't missed a backup yet :-)
All in the subject!
I have a local repository and also subscribe to crashplan central.
It's brainless and easy.
Any data that's actually valuable is in my Dropbox - so it's "backed up" on 2 of my computers, a work(university) computer and on their "cloud" systems as well.
It's free and protects in almost all disaster scenarios, such as the house burning down, all my computers getting stolen, etc.
All that doesn't fit in Dropbox is just something produced by others. If a disaster strikes and some a music album or computer game that I bought is not recoverable anymore, I'll have no moral regrets in getting a backup copy that some pirate has put online.
For our personal computers, we use Time Machine - but manually triggered.
For the media server, I've got a second disk and a once-a-day cron job.
Offsite... well, okay we're not really there yet. So we're covered in case of hard drive failure, but not a catastrophic fire.
(I am assuming this question is about home, not about work)
#DeleteChrome
I basically use this shell script once a week:
drive=/backup/drive
bpaths=/some/paths
for d in $bpaths ; do
dout=`echo $d|sed -e "s/^.*\///"`
echo Backing up $d as $dout
ionice -c3 rm -f "$drive/bkup/$dout.*z"
ionice -c3 tar -c "$d" | gzip -c | ionice -c3 openssl aes-256-cbc -salt -out "$drive/bkup/$dout.tgz.aes" -pass pass:"WouldntYouLikeToKnow"
done
I then copy the data to my USB drive on my keychain if it's plugged in. (Hence the encryption.) I also have a scheduled task on my laptop to copy the data from my desktop the next day.
(T>t && O(n)--) == sqrt(666)
My house is full of Macs, so I use Time Machine for on-site backup - each machine has its own Time Machine drive dedicated to it. Each machine also runs nightly image backups using SuperDuper onto yet other drives dedicated to that purpose.
All info is also backed up offsite. I use CrashPlan Pro, which backs up over the net to their servers somewhere in the American Midwest (Milwaukee?) - in the event of a fire or a giant sinkhole opening up under my house, I can get the full contents of all my computers shipped to me within a few days on external hard drives.
Mudge
In theory, theory and practice are the same.
In practice, they're not.
With two copies for redundancy. Don't use cloud solutions because I don't trust them. Well, unless you count mailing shit to myself and using Gmail as a cloud backup solution, anyway.
Encrypt any important data that you don't want to lose, and keep it close to crazy terrorist rants in plain text files. With all the government snooping going on, in the interests of our security, the government will secretly make backup copies of it for you. Their experts will try to decrypt it, and be baffled by the message hidden in your wedding video.
In case of a disk crash, just ask the government politely to give you a backup copy of your data. They will kindly oblige.
Probably.
Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
Live on the edge guys...
/does/ boot or you /don't/ get toasted by bolts of electricity then the sense of relief is wonderful !
;-)
When you boot up in the morning and it takes a little longer than usual, the heart beats a little faster and you think "OMG is the machine going to fail? My data will be gone". Or perhaps there's an electrical storm to liven your day up - "If that thunder gets any closer I might have to shut down the PC, but if lightning hits then everything's toast !".
These scenarios, and many others, all get the blood pumping in fear. If the computer
Try it - it's fun
while (true != false) process_more_stupid_code();
For personal machines, imaging to an external USB/Firewire HDD is the only way to go. Drives are so large and cheap these days, I've got several of them in a rotation. Using blank optical disks, tape, etc.these days for personal use is just silly. Occasionally use some free online storage or other removable media for small amounts of important things I may be working on at any given point in time.
At the office (small business) someone setup the server for the cloud using Carbonite before I got there. Probably a good idea to have off site storage for business. I may get them an external enclosure to backup occasionally for more redundancy and quicker recovery time if they loose everything.
Great offsite backup:
http://www.crashplan.com/
For a couple dollars a month it's a great deal.
You can always download new and better porn from the internet. Everything else of importance can fit on a CDROM.
I use a combination of BackupPC and CrashPlan+ Family Unlimited to keep the data on all my systems safe. Works like a charm.
I use a secure distributed grid. The software is an open source tool, Tahoe LAFS (http://tahoe-lafs.org). The grid is composed of ~15 servers contributed by different people all over the world. There are a half dozen servers in various locations in the US, about the same number in Europe, and the remainder in Russia and the Ukraine.
My files are AES256-encrypted on my machine, split into 13 pieces using Solomon-Reed coding, any five of which are sufficient to reconstruct my files, and then those 13 pieces are distributed to the servers in the grid. I run daily backups, but since uploads to the grid are idempotent, only the changed or new files are stored. I also run a bi-weekly "repair" operation which checks all of my files (all versions, from all backup runs) to see if any of their pieces are lost. If so, it reconstructs the missing pieces and deploys them to servers in the grid. The individual servers in the grid are fairly reliable, but problems do happen, so repair is important.
I get about 100 KBps net upload rate, so this isn't a good solution for backing up terabytes, and the occasional "surge" in my data generation (usually caused by a day of heavy photo-taking) often causes my "daily" backup to take a few days to run, but all in all it works very well.
Should my server ever die, I only need two pieces of information to get all of my data back: The grid "introducer" URL, which will allow me to set up a new node connected to the grid, and my root "dircap", which is a ~100-byte string containing the identifier and decryption key for the root directory of my archive. That directory contains the decryption keys for the files and directories it references.
Since this grid is all volunteer-based, the only cost to me for this backup solution is the hardware and bandwidth I provide to my grid (I provide 1 TB of disk and grid usage consumes a fairly small fraction of my Comcast connection), plus the time I spend administering my server and checking to see that my backup and repair processes are running. Oh, and I also contribute (a little) to the Tahoe LAFS project, but that's due to interest, not a requirement.
I'm very, very happy with this solution.
BTW, the grid could use another 20 nodes or so, if anyone is interested. There's a fair amount of trust required of new members to the grid, though, so it might take us a while to vet new members. The trust is required not because other members of the grid might have access to files that are not their own, but we need to verify that new members will behave appropriately -- providing their fair share of storage and bandwidth, and not consuming too much.
Anyone interested should check out the grid's policies and philosophy at: http://bigpig.org/twiki/bin/view/Main/WebHome. If all of that looks good, join the mailing list, introduce yourself and we'll consider allowing you to join the grid.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Ugh.. Apple's Time Machine is complete crap, totally inefficient.
ZFS is where it's at.
Hasn't this been covered ad nauseam? Not to troll or anything but it seems like a third of all Ask Slashdot questions are backup questions, and the responses to these cover close to all the special cases.
I am far more interested in World IPv6 day. We need to get moving over to that.
I prefer the "u" in honour as it seems to be missing these days.
http://www.youtube.com/watch?v=2g76DUy7jKY (had to do that)
Each of my computers has a dedicated external hard drive where I push backups on a regular basis. I have one external hard drive stashed away at my parents where I make an backup of my laptop every time I visit them. My laptop pretty much has all of my important data so it serves its purpose. All the hard drives involved in this are encrypted of course.
On top of that my most important text documents(not necessarily important in the way of having personal information, but a lot of work put into) are backed up to Ubuntu One as well as Wuala.
Given how infrequently a lot of my data changes, I find the backups adequate.
Think through what you're backing up and why. For most people a thumbdrive should be sufficient for personal data; software can be reinstalled as needed. If you have more data than will fit on a thumbdrive you need to look at what's important.
Really large volumes of data almost always are static; usually music, eBooks, or video which can just be backed up once on a DVD and put away. No need to keep copying that stuff over and over.
Backing up software projects is another issue. A remote versioning site is best. Working in Java you'll need all the space you can afford; for a language like Python an old floppy drive is sufficient.
You joke, but photography can be used to back up anything where the data can be seen under normal light.
If you get really up-close, phonographic records and optical media fit the bill. Paper records (think: Microfilm copies), film, and on-film movie soundtracks also fit the bill.
I've even heard of a way to make the bits on 1960s-era magnetic tape visible under certain wavelengths of light.
As for making a "photographic image backup" of my magnetic disks or solid-state disks, well, if it can be done at all, it's probably destructive and definitely inefficient.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
All of our important files (even the kids' files) are on the server. It backs itself up automatically 3 times per week to external USB drives. I rotate the USB backup drives every few weeks. So we need do nothing special today, as the backup works fine.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
I use just a three-level hierarchy:
1. Photos and documents are on my RAID-5 array (4 × 1 TB Hitachi enterprise drives) in my desktop, backed up occasionally (every month or so) to a Toshiba 1 TB eSATA external drive sitting on my desk
2. Music, movies, TV shows, are on the RAID-5 array, not backed up
3. Windows and programs are on my 80 GB SSD, not backed up.
So I'm not protected at all against my house burning down, but this has worked for me for the past 10 years. (For my old system, which ran 2003–2010, it was a WD Raptor, not an SSD. And the RAID 5 was 4 × 200 GB.)
Karma: pi (Mostly due to circular reasoning in posts).
I really like Areca Backup. It has a fairly straightforward GUI and you can easily back up groups of files to different backup locations or media. If you run a differential or incremental backup, the GUI presents a "logical view" of that backup against the last full backup or series of backups. Now, if only I could find some easy way to tag and organize 20,000 mp3s...
World backup day? How about world test your restore day? All the backups in the world don't mean anything unless you test your restores and know your data.
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
Killthre... I mean The Tao of Backup
Mon, wed Fri: Ghost Backup of OS and C: drive to a separate drive
Tue, Thu: filecopy of all required files (Pics, mp3s, important data to same separate drive
Daily scheduled task that runs and will mirror said separate drive to another drive that is mounted as Y:. This drive is stored at work and is fully encrypted using truecrypt. Usually bring it home on Mondays but if I forget, it will automatically run any other day
rsnapshot seems to work pretty well for incremental rsync'd backups for me. It uses symlinks to maintain the older snapshots, to save on total filesystem usage. It can do rsync over ssh for backing up remote servers/pushing local vital data to a safe remote location.
Local backup server uses Linux software RAID for good measure (5x1TB RAID 5 + 10x2TB RAID 6).
I do rdiff-backup of plain files and cp --sparse=always of iSCSI shares and VM images to internal SATA drives in an eSATA cradle. Those drives are stored in a fire and waterproof lockbox in our detached workshop. Given our high ground location (flooding is very unlikely, so is landslides, no underground mines and low sinkhole probability ), anything bad enough to destroy the computers in the house and render the contents of the lockbox unuseable probably is bad enough that I don't care about the data.
At work most of our customers have their heads in the sand... It's like pulling teeth trying to convince them to at least use a USB drive.
Oh yeah, one of those iSCSI shares is a time machine backup... which can be copied to physical disk if a baremetal recovery is required.
Backup is only half the problem. Restore is the other half. And indeed that's where I've usually had the most problems. The third problem is validating the restore. You always worry that you are either going to overwrite something on the restore target or miss something on the restore source and end up in an inconsistent state.
Time machine is revolutionary because it is so simple and seems to be almost flawless. I've had lots of backup systems over the years including dump 0 but everyone has been plagued with issues that arose when things were off normal. I've cobbled all sorts of things like rsync and cpio but the only thing that comes close to working as flawlessly as time machine is a NetApp.
At work where I can control the remote servers securley on a closed network I am able to use time machine for a remote backup. But at home I don't have a remote server I can target for the remote backup.
TO do a remote bakcup at home I use Crashplan. I looked a lot of competitors like Mosy but settled on crashplan for two killer reasons. The giant problem with all these commercial backups is that while the incremental backups are simple over the net, the restore of a whole hard disk cannot be done over the net. You have to pay them to burn DVDs and send them to you. ANd that assumes you know what time period you want to recover.
UNlike all the other methods crashplan lets you pick a buddy who runs crash plan and then you can back up your disks to each others computer. If you need to to a massive restore you just drive over to your buddy's house and pick up the drive, bring it home, and restore locally. This also solves the problem of the first dump being too large to send over the net as well. You do it locally then drop the drive off to your buddy.
Brilliant!! plus with crash plan you pay for the app once not monthly.
I've used it for years now and it works very well and it very easy to set up. All your files are encrypted so buddies can't read each other's drives.
The only flaw with crashplan is that it runs in java so you have this instance of java running 24/7 and not to put to fine a point on it: java sucks. I don't know if it is crashplan or other things that run in the JAVA VM but over the week it bloats up to 600MB to 800MB. THe workaround solution is to kill the java VM every few days. Empirically crashplan is robust enough to survive this and restart. But that's a really awful solution.
Some drink at the fountain of knowledge. Others just gargle.
It's also the day of Earth Hour - a day on which, for one hour, people around the world turn off the lights.
http://en.wikipedia.org/wiki/Earth_Hour
I wonder how many here personally partake.
-----
As for backing up goes, I'll just re-paste here what I said in a recent other Ask Slashdot question:
It's not really 'managing' my data.. it's a storage/backup solution. The difference is that if I 'managed' my data, I wouldn't have tens of thousands of digital camera photos in a bunch of folders with meaningless names, but just a few that are actually worth saving to me. It's not that I'm saving all the others for future generations either, I just don't have the energy to go through so many photos and delete all but the best (the very best I've already shared anyway).
But if it's just storage/backup...
1. Every write made to the main HDD is mirrored via a mirroring RAID setup. Pure mirroring, I don't want to deal with RAID levels that use parity/etc. that may save some space but are a PITA to rebuild (and must be rebuilt - a simple mirrored HDD mounts just fine when taken out of the RAID).
2. Files are written to a versioning filesystem, so that if I delete something that I later regret, I can get an older version back (presuming things didn't run out of space and it had to be overwritten with new data).
3. Files saved to a specific area are further synced with a cloud storage solution. These are basically files that I need to be able to access from any location at any time (short of the cloud hoster folding/etc.) asap in case of an emergency. There's very few files that qualify, so bandwidth and monthly caps aren't an issue. I did upload about half a GiB worth initially, though.
4. Every night the computer does a differential backup to an external, also mirrored, HDD, over the network. This is a set that is in a completely different area of the house, so if I manage to trip and splash water all over everything here, the others are fine.
5. Every 2 weeks (used to be weekly) I bring one of the HDDs in the mirroring set in the other room to an off-site location (basically a storage locker). From that off-site location I bring back another HDD and put that into the RAID, and force an update of that HDD from the other one.
So -if- one of my main HDDs dies, there's always the other one. If they both manage to die at the same time, I've still got a daily backup in another room. If that dies, that has another one. If those both die, I still have a 2-weekly backup in an offsite location. If that one's dead as well (what are the odds??), then all my most important stuff is also in the cloud. If that cloud storage solution goes belly-up at the same time and data can't be retrieved? Well, I'm screwed. But life does go on - people whose houses burn down often don't have such a rigorous backup method in place, and they pick up again as well.
That said...
6. Of very important photos, I've got prints (a Kodak booth does better than your home inkjet) or even negatives (the better photography stores can point you in the right direction for that). Of very important documents, I've got print-outs (laserjet). Of very important video? Nothing. Of very important (music) recordings? Also nothing. I have no such 'very important' of the latter two - but I think you get the idea: I would have gotten those transferred to film and/or tape. The reason is that those can easily be seen by human eyes or played back for human interpretation - digital data not so much.
My solution is mirroring data from the computer to a NAS with RAID, then a harddrive I take offsite with all the impossible/hard to replace data.
Ironically a Harddrive in my NAS died two days ago, so I have had to do a rebuild and as a result bought a new external hard drive that is big enough for all the data, even the easy to replace data. So currently my NAS, which rebuilt the array successfully is copying all of its data to a new external harddrive.
Chinese espionage hackers do it all for us free. They copy our stuff over to their side. It's as off-site as you can get.
Table-ized A.I.
For laptops, I use a scheduled rsync to a central server mounted using sshfs. For offsite, an rsync to an EncFS filesystem on a portable drive. If bandwidth limits ever get reasonable, I'll switch to using DropBox or SpiderOak, but the bandwidth limitations remove that as a solution for all but important data.
Datasette, compact cassette .. works great, never lost a program.
I just put all my stuff there. Rest, like music, movies, e-book and adult films I can just re-download
I use vitalEsafe's shadowSafe (http://vitalesafe.com/new/backups) product. Local AND offsite block-level (not file level) complete machine point-in-time backups. Uses Microsoft Volume Shadowing Services, so you can make "hot" backups (backups while running) of MS SQL, MS Exchage, MS SharePoint, Oracle, Pervasive databases. If I have a catastrophic failure, they will ship my backups to me on external drive. Backups can be restored to bare metal (no operating system required) and can be restored to different hardware. 200GB restore to empty machine takes approximately 1.5 hours. vitalEsafe supplies external backup hardware, all software, remote/onsite installation, secure offsite storage, and daily monitoring of my backups. It isn't cheap, but it is a very comprehensive product that covers all the bases.
Full disclosure: I'm the CTO for vitalEsafe so I designed and very much believe in our product.
I put the car in reverse, check my mirrors, then floor it.
I have two systems I use.
For my servers, I use AMANDA with encrypted virtual tapes to do nightly backups. Shortly after the backups run, cron calls a shell script in order to copy the virtual tapes to an offsite location via rsync.
For my desktop PC, I don't need to back up as often, so I do a weekly backup via Windows Backup to a TrueCrypt volume on an external hard drive. When it's not being used to back up my PC, I keep the external hard drive at my office. I figure if something happens where both my office and home are destroyed, then at that moment I've got bigger problems to worry about than my data. :-)
Just my $.02...
I use Duplicity to do incremental encrypted backups to an old computer that I only keep around for this purpose once a week, or more often if I have data that I really don't want to lose before next saturday. I'm planning to get a second backup disk to keep at a friends house and sync with my local backup once a month or so, but I think I'll wait with that until disk prices are sane again.
I just wanted to throw out a decent rotation system I used for making backups when I had some pretty important data to keep track of.. it works with 11 sets and goes like this.. you get 11 sets of whatever..tapes, drives, thumbs..your media... you label them mon, tue, wed, thu, fri1, fri2, fri3, month1, month2, month3, month4 (we didnt do weekends.) The rotation is pretty obvious..and allows you to go back any day this week, any friday of the month and any month for 4 months. We did this so we could roll back far enough to be sure if we started copying corrupt files or whatever we could have a good clean point to recover from. You can of course add sat and sun and plop months on the end for longer times.. but we never needed to roll back farther than 4 months.
very little is worth backing up. That which is gets uploaded.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
I cannot praise this versatile little free* program more. I have given up on personal RAID. I can't afford RAID 5, which is the only RAID I might feel assurance from. I started playing with the original consumer Promise cards and dealt with the headaches. Even the Medea RAID 5 units I used at work became a hassle. I learned the hard way not to trust these plug-and-play RAID 1 units offered by LaCie and the like. I had one where the controller circuitry failed, but both internal drives (and ostensibly the data on them) were intact... But they were housed in an proprietary format, so I had no way to get the data off even if i pulled the drives out. I sent it back to LaCie and even though they noted that it was just the controller they replaced, they still nicely formatted the volume, destroying all data. I have since taken too keeping two physical drives of the same size in separate enclosures and have them incrementally update one to the other nightly. The app I use to do this is GoodSync (Win/Mac). It's really low-profile/low-overhead and very configurable. My nightly updates are just a simple A->B incremental job. But I also have a job that connects to my colo server via SFTP as a midpoint, effectively allowing me to run my own Dropbox-style file locker. For this it has the ability to handshake with the server whenever any file changes in the specified local folder. I then run the client on multiple machines. Additionally it has ability to work with commercial cloud services, such as Azure and Amazon, but I haven't tried those. *There is a pay-version, but I haven't found anything yet that I couldn't do with the free version.
zfs or hardware raid on all servers. bacula for all MS windows systems and all servers which are switched on most of the time. Apple Time Capsules for all the macs. regular backups of all phones. auto-sync of all address-book and calendar info over icloud.
as you ask, why yes, i have lost data in the past due to lack of backups. ain't gonna happen again.
-bh
I use Crashplan, my home computers backup to a external hooked to a netbook in the the basement, the netbook backups music and media to my desktop.. The critical data, like photos and client work also gets backed up to a bare sata drive in a dock that I take to my office on a weekly basis. So even if my house gets toasted or flooded at least I am no missing much. I wanted to use crashplan's cloud service, but it would take forever to get all my data offsite.
I have an old (Athlon 900MHz) spare computer which boots each night at 3pm and grabs the data to a mirrored LVM volume. Once it's done it waits another 15min then shuts down. Munin tracks the SMART values and alerts me if a disc is failing. The system itself supports 6 SATA-drives that are easily swappable.
I've been using rsync to back up 1/2 dozen machines here for some time now. Great for both local and remote (internet) backups. My mom's imac is 300 miles away. She has a little 500gb firewire drive locally backing up with time machine, which gives her instant recovery as well as application-spicific recalls (get that email back or that address book card back easily for example) as well as versions. Then a custom made cronjob to rsync to my server here runs nightly for offsite in case of fire/theft/etc. That's really the best of both worlds there, highly recommended. No recurring costs.
I work for the Department of Redundancy Department.
Seriously, who cares about what happened 5 years ago?
When did you last look at that porn clip you downloaded in 2002? How many emails do you really need to keep for more than a month? Would Battlezone 2 even run anymore if you tried it?
When did you last get audited by the IRS? Screw that, my accountant should have copies anyway, and most info was electronically reported anyway.
Between the cloud, my iPhone, 4 synced home boxes, a drop box for some important stuff, Google, I can recover anything non-trivial.
I back up by putting my car in reverse. Or by walking backwards. Duh.
Time Machine to a 4 TB drive on my main computer. For my wife's machine and my laptops, Time Machine to a 2TB Time Capsule. Haters can hate, it just works.
Rsync to a local Drobo with Drobo RAID, a Local fiber channel array with RAID 5, and to a remote NAS with RAID 5, every night.
For my employers: There's a DFS off site copy. There's a nightly Rsync copy to a standby system. Then the nightly tape job runs. And, I'm in the process of shoehorning a nightly Acronis imaging into the mix.
At home: Three PCs no backups. And, I just suffered a hard drive failure and total loss on one of them... Perhaps I should setup the NAS I've been contemplating for the past few years.
It's like the Linux version of Flyback or Timevault.
http://backintime.le-web.org/
For a single computer only, it can be fairly inexpensive with 2-3 TB HDs.
1) Clone the entire working drive once every one - two weeks, so I can go back to a working OS if the OS is corrupted as that takes a day to reinstall all OS, Apps & Utilities and migrate data back. Hence, this is a recovery point for the entire HD on the time period one picks.
With cheap 2-3 TB hard drives, one could actually clone the HD every day for a lot of people & then overwrite such that you had a complete 2 weeks of cloned HD contents that continually gets overwritten.
2) Then, daily backups of changed files to a 2-3TB HD that gets swapped with a HD kept in a different location.
I back up everything about once every 2 month to a RAID external drive. I also keep extra copies of really important stuff on a flash drive, REALLY important stuff (as in I couldn't function without (so mostly video game save files...)) are backed up AGAIN in a hidden folder on a computer at a family members house. Also about every 2 years or so I back up everything again, wipe my computers built in drive, and replace it, and then put everything back.
The backup drive has another bonus, I never delete my backups from it (I've heard that can cause drives to go bad faster) so if I ever want to check an old version of a file, I can just find the month I want and open it.
The chances of me losing anything are almost zero.
Earlier I used backup2l to first make local backup and then rsync to server. The only problem was it was wasting disk space on each host, specially laptops.
Recently I moved to bup, provides more efficient backups with very small local storage. Now I have in every laptop, desktop and my email server (all running either Debian or Ubuntu) in /etc/cron.daily/bup-backup:
#!/bin/sh /var/mail /home /var/lib/mysql /var/mail /home /var/lib/mysql
echo Backup starting at $(hostname) $(date)
bup index -u
bup save -r backups.example.com: -n $(hostname)
echo Backup ending at $(hostname) $(date)
Because I do not want remote root logins on my file server at home, there is a need for small tweak in /root/.ssh/config:
Host backup.example.com
User bupups
Compression yes
HostName filesrv.example.com
Now as each system backups to the same bup archive, deduplication is taken care automagically.
Each month I rsync /home/bupups to external USB drive that is stored in different place in case of fire or other mishap it will be protected. At some point I've considered installing a low-power server at my father-on-law place for automated off-site backups; will keep the off-line copy in any case.
Of course, I do not much care about taking backups from movies, music - expect maybe some kids love; but they are already on some USB drive to entertain them while travelling...
The machines needing backup do so wirelessly to a Time Capsule whenver they're connected. I also do a full image to an external hard drive every once in a while (I lie to myself and say it's biweekly but it's really more like bi-whenever-I-remember-it). Even with the recent increase in HD prices there's no real excuse to not have backups.
bah.
My backup for a multi-boot laptop that other solutions (e.g.: running from one OS) don't seem to work for:
1) Buy a second copy of your main hard-drive + USB Interface (SATA enclosure)
2) Boot Linux on computer using CD
3) Use dd to mirror entire HD to external HD. Run before you go to bed, setup to shutdown when done. Save stdout/stderr somewhere like a USB flash drive.
4) Wakeup to a backup.
The advantage of this is when your hard drive fails, recovery is about 60 seconds away. Swap out one hard drive and you are done. Or you can recover specific files by just using the backup HD like a normal external HD, since everything is just under normal filesystems. If you'll be on business for a while take your second hard-drive with you (try to store somewhere it won't get stolen with laptop).
I actually keep two mirrors, partially because of travel and wanting to have one backup with me. This also makes sure that if your computer fails half-way through doing the mirror due to a power surge it doesn't fry your original + mirror. Keep one at a friends house or similar.
Into the Microwave in case of an EMP from the Sun? The Faraday cage will protect the drives.... :)
I BACK UP and start over.
Easy, and doesn't need much maintenance.
My Ubuntu 12.04 server is btrfs using the built-in RAID 10 features (I have 7.2 TB usable on four 2TB drives thanks to compression too!) and I use time machine for the macs. My windows machine is Pro so I'm able to backup to a network drive, with a samba share running on the ubuntu server that I mount as a drive letter in windows. It's been fantastic overall for all machines including for the odd restore.
I'd like to get time machine going to the ubuntu server as well but so far I'm just using external usb drives for the macs.
========
77 77 77 2e 6d 65 6c 76 69 6e 73 2e 63 6f 6d
I use a system of rsync over ssh to do both onsite and offsite backups daily with a rotating system of hardlinked trees that give me seven days of daily backups, 5 weeks of weekly backups, 3 months of monthly backups, 6 months of quarterly backups and 1 year of semi-annual backups with each location doing both local backups and remote backups for the other site (which are 40 miles apart).
The premise is that you back up your computers on March 31, so that you're not an April Fool if your hard drive crashes tomorrow.
Thanks for reminding me to stay away tomorrow.
Sheesh, evil *and* a jerk. -- Jade
I love it. Never have to worry about backups. It's saved me so much pain when hard drives go belly up.
It's the ideal medium for long-term storage. Durable (hundreds of years), reading technology is easy to develop from scratch if needed.
The only problem is storing the 2100 km of tape a full backup generates.
Everything worth being backed up I am working on is pushed to a git repository. The git repositories are synchronized manually (but pretty much after each important update) across about 5 machines in 3 physical locations (home, work, computing center). Though they are in the same city.
I don't think I will lose anything important anytime soon. Or if I do, I think I'll have more important concerns...
That's In-Person backup.
Ok, in is a bit of an exageration, but I back up my (important) data on a USB stick I carry around all the time. Two, redundant, in fact. If anything happens to them (both), the I probably won't care since that same thing will have happened to me.
I use windows live sync to synchronize one main folder (to rule them all) across all my machines. Then, from one of the machines, I periodically dump that folder onto a NAS. On the NAS I have a script running that does snapshot backups (like time machine) of that folder. Losing data from an accidental delete is far more likely than a hard drive crashing.
The only disadvantage is that the machine that dumps to my NAS has to be at home on my local LAN.
I am thinking of trying out Synology's DSM 4.0: http://www.synology.com/dsm/index.php?lang=enu
If it claims to do what they say it does, it should be like my own dropbox in the cloud. Then I can sync my one main folder across all my machines and my NAS. This eliminates the need for windows live sync - and I don't have to be at home to dump data to the NAS. I can also setup a script to make periodic snapshots of the folder. The only issue is currently the synology only has windows clients - and all my machines are macs... so I am still waiting to pull the trigger.
use dropbox
Then I turn the key and crank the engine. Since the transmission is already in reverse, I simply apply gas, and suddenly release the clutch. That said, I try to avoid back ups as much as possible, because the hard drive seems to cause crashes more often than not. (Mod me+1 Flamebait)
Correct Horse Battery Staple: 72 bits of entropy. Enter "Correct H" into google. When it generates the phrase, that's
So I put a special unique string in every document. If I ever lose a document, I can Google it back from the hackers.
I have some years ago emigrated and live abroad, which has given me an opportunity to use servers in three different locations in two countries. At home, at my brother's and at a friend's.
My Linux servers back up automatically with a homebrew Time Machine-like functionality based on rsync. It consists of a script and a configuration file with information of what files and directories to backup/not backup. The structure is fairly simple and has worked well for a very long time now.
The mail servers back up any mail in the mail directories to each other, but overwrites/deletes as needed instead of the TM-type functionality.
The family Windows machine is backed up manually with an rsync-based BAT-file whenever I feel like it (which is rather often) to one of the local servers which propagates the backup through the TM script to one of the overseas servers, which again propagates it to the other overseas server.
All in all it seems to be sufficiently redundant (three copies of all relevant data in less than three hours) and sufficiently dispersed more than 600 miles between the farthest points and no sites closer than about 200 miles.
Obviously, I reciprocate the service for my brother and my friend.
For years now, I've been making all of my backups with a script I wrote that uses rsync and faubackup, the latter being a disk-based backup solution. All important data is backed up on a daily basis, locally and to other servers across the Internet. All partitions involved (source and target) are on RAID1 or RAID5 arrays (by itself, RAID is not a backup solution, but does increase the reliability of the storage medium). The only way I've ever lost any data with this system is when I forget to add a source to the backup script.
A combination of Dropbox and Time Machine. I rotate the Time Machine drive about once a quarter keeping the second out of the house, so even if the house burns down I lose at most 3 months of heavy assets (movies, photos, etc.). Dropbox provides the nice side benefit of meaning I won't lose access to current working files needed for day-to-day things.
Most of the heavy assets can be easily replaced with the exception of family photos/video, but those will probably still be available on the original device or in iCloud's photostream, so those 3 months are relatively low risk.
... plus raid.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
#1: rsync -aihxv --inplace --delete root@some-host:/ /backups/@some-host /backups/@some-host /backups/@some-host-`date %F--%T` /backups/@some-host-(date and time from some cycles ago)
btrfs subvolume snapshot
btrfs subvolume delete
(repeat for all hosts)
#2: Crashplan.com
I quit using rdiff-backup when I discovered that all my backups had been failing for a while because rdiff-backup did not like a filename (I think it was very long) and, instead of skipping the file and continuing (or even handling the odd filename), it had been hanging at that point. I consider that a giant FAIL for a backup system.
rdiff-backup wasn't doing anything that I could not do with rsync and some scripting anyway, so, what's the point?
The real "Libtards" are the Libertarians!
Two drives in the computer. Several times a day, copy changed files from one to the other using a .bat file and xcopy.
Mozy online backup, twice a day.
One removable drive. Several times a month, copy changed files to it using a .bat file and xcopy..then take it out and put it on the shelf.
I use the built in Windows "Robocopy.exe" feature via a CMD file to backup the files on my Windows computers. Robocopy used this way can be scheduled, and is extraordinarily fast. It pulls files seamlessly through my local area network, so I backup my desktop, my laptop and my partner's laptop all onto one large external hard drive via the network. I have it set to skip files that have the same timedate stamp, so after the first copy process, it runs in under a minute. The thing I like most about using robocopy is that I end up with the same exact set of files and folders on my external hard drive as on my working computers. This makes the backup or restore process very transparent and easy to use for even beginners, though I am fairly expert. I do have it set so that if I delete a file on my working computer, the backup remains on the external hard drive. I've used this since XP days, and was thrilled to find robocopy built into Vista and Windows 7. To learn how to use it, just search for "robocopy".
At home, I use RAIC: "Redundant Array of Inexpensive Computers" :)
/backups/`
/backups/${CLIENT}/yesterday
/backups/${CLIENT}/today /backups/${CLIENT}/yesterday
/proc etc., etc.,etc"
Okay, okay...that's really only *part* of the backup strategy. I also use a Time Machine with mine and my wife's Macs; my Linux machines used to backup to a separate hard drive using tar and gzip, but I haven't set that up again since we moved into our new house a year ago.
At work, we use a combination of rsync and ln, like so:
Backup Server Side:
#!/bin/bash
for CLIENT in `ls
do
rm -Rf
ln
done
Backup Client Side:
#!/bin/bash
EXCLUDES="/dev
HOSTNAME=`uname -n`
rsync -av --exclude ${EXCLUDES} / rsync://${BACKUP_SERVER}/${HOSTNAME}/today/
DIsclaimer: this is more pseudocode than actual code. We actually keep 90 days of backups at work, and dump the last day of the month to an external hard drive which is stored off-site, so it's a bit more complicated than I've suggested above. However, you should be able to get the general idea from what I've posted.
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?
luckily there isnt really THAT much that needs to be archived, everything else just gets shuffled to the NAS
It must be time to get a new calendar! :)
$ calendar
Mar 31 Rene Descartes born, 1596, mathematician & philosopher
Mar 31 Eckert-Mauchly Computer Corp. founded, Phila, 1946
Mar 31 Automated Retroactive Minimal Moderation (ARMM) released, 1993
Mar 31 National Day in Malta
Mar 31 Joseph Haydn is born in Rohrau, Austria, 1732
Mar 31 N'oubliez pas les Benjamin !
Mar 31 Bonne fÃte aux Benjamine !
Mar 31 Ã la saint Benjamin,
Le mauvais temps prend fin.
Mar 31 Erstes Gesetz zur Gleichschaltung der LÃnder, 1933
Mar 31 Joseph Haydn in Rohrau geboren, 1732
Mar 31 ÃrpÃd
Mar 31 Ð"ÐнÑOE ÐоÑÑÐÐÑÐоРÐÐÐоÐÐÑÑfÑÑ
Mar 31 King Nangklao Memorial Day (àà±à(TM)à--ààà£àà¥àààzà£ààsàà--àààà"àààzà£àà(TM)à±ààààà¥àààààààààààà±à) in Thailand
Like the beaver, it's just Dam one thing after another
I start the car. I move the shift lever to "R". I scream "Beep, beep, beep!" and stomp on the accelerator until the tires smoke....
I do not fail; I succeed at finding out what does not work.
Timemachine to drive on Airport Extreme.
And all important documents, pictures and home videos I have on my qnap NAS which backs up everything to elephantdrive where I have a 250gb account.
I only have a 20/2 adsl line so the first backup took a month or something like that at 100KB/s.
Pretty simple, in a retail environment each night we run a script to copy our database to 3-4 other computers on site. Each morning the first person in the store copies the previous night's backup to usb and drops it in a fireproof safe in the storeroom. The usb drives get recycled twice a week at which point I take a copy off site on my usb. We have multiple stores which all run the same server software and have at least one spare machine that can act as server at each site if their server goes down, one spare machine at my home and one at the owner's home. The owner also takes an external backup whenever he's in any of the stores. Isn't redundancy nice? :)
> How do Slashdot users back up? RAID?
I work at a data-recovery company. RAID does not replace backups!
There are a bunch of things that could happen.
1) disk drive dies.
2) Power supply dies and fries all drives in the machine.
3) building goes up in flames.
4) City is wiped from the face of the earth.
A raid protects against a type-1 mishap. The broken powersupply fries all disks in the raid at once. You could use an external USB drive to protect against that. You need to store the USB drive or other backups off-site to protect against number 3. You need the off-site backups far away to protect against the big world-shattering disasters.
I'm protected up to level 3: I have off-site, same-city backups (on raid systems of my main data which lives on a raid system).
My backup setup is pretty simple. An Apple Time Capsule for hourly local backups that I can access right away, and Backblaze for off-site backups in the event of theft, fire, etc. I never have to worry about backups anymore.
For local backup I run Macrium Reflect to create daily snapshots that are stored on an external hard drive. Older images are deleted as space is required. Great if some files should accidentally get deleted to be able to go back 60 days or so to retrieve them.
For my "fire proof" offsite backup up of important files such as family photos and documents I use BackBlaze which allows unlimited storage (although I only use about 20 GB). Their service also allows you to recover files that have been deleted up to 30 days ago and your backups can be encrypted with a private key for privacy.
Storage is much more affordable than recovery.
http://backuppc.sourceforge.net/
Fry's (and other places, I'm sure) has shoebox PCs running low-power CPUs for about what you pay for a good external USB drive. Pick up a big drive and some RAM for it, install LAMP and BackupPC and then have it backup your other computers on your home LAN. Nice thing is that it has deduplication and uses rsync, so there are clients that work on Win, Mac and Linux.
Simple, I push the clutch in, shift into reverse, let the clutch out whilst simultaneously and judiciously applying pressure to the accelerator. If I,m not in the car just stepping backwards seems to do the trick. ...or did I miss something?
What I need to keep can fit on a decent sized USB stick or Micro SD card. For added redundancy, I can keep a copy of the most important items online, on another PC in my house, or in someone else's house. The vast majority of files on my computer can be replaced, should the worst happen.
ThinkPad T61p here.
Main OS drive is a 60GB SSD.
CD/DVD Bay has a 500GB SATA drive in it. Automated daily backups to that from the main drive.
Any 'data I can lose' such as music purchases and Steam game installs (some stuff I special-case back to the SSD, along with the main Steam install itself) is stored on the 500GB drive along with scratch-data and my paging file to avoid wearing out the SSD as much.
Swap the 500GB drive periodically if you want 'offsite' backups to boot.
I join the republican party and then I am backed up to the 1950s. LOL
I go to work and then take one step forward and two steps back, then I'm backed up, lol.
All my data is in the clouds and I pray I can remember my passwords :)
I'm surprised rdiff-backup hasn't been mentioned yet. It's a very nice piece of software, does incremental backups, and is easy to automate.
-- B.
This sig does in fact not have the property it claims not to have.
There's nothing easier. And if you remember that the goal isn't to make backups, but to have successful restores, there's nothing that works better and easier in restoring a desktop or laptop.
Net App snap mirror plus tape backup for servers at work. Time Machine useless there.
I don`t have a huge amount of data, so I just grabbed a spare 320GB HDD, USB-to-IDE/SATA adaptor, and power supply, and copied everything on my laptop to it. On a more frequent basis, I also copy important files to CF cards. I still have files from the `90s from my first PC and from floppy disks prior to that. Whenever I replace a drive (or the whole computer) I copy everything to the new unit and usually keep the old one around also. Only once I had a drive hardware failure, but was still able to get most of the files before it died completely (and had another machine with redundant stuff on it). And there was one time I had file system corruption in a desktop machine (I think it was related to a failing power supply) and had to restore the whole drive from the backup.
I have been using RAID for many years — RAID-1 at work as I only have two drives and don't need much storage space, and RAID-5 at home. A couple of years ago when I upgraded my computer at work, I downloaded at least three different backup systems to try out. The goals were simplicity of use, keeping historical versions of files, and relatively low storage space.
After setting up bacula, I never bothered with the other backup applications.
I found bacula to be highly flexible, adapted very well to the set of many virtual machines I use, and is the easiest to maintain. I just set it up once (or after any major re-partitioning) with a specific list of files and directories to back up or exclude, then practically forget about it. It's saved my files a number of times already from accidental deletion or overwriting, and I used it once for a full restore at home after upgrading my computer including a new RAID array.
At work my excess hard drive space is enough to store all my full and incremental backups locally, but I also have it back up critical files to a corporate NFS server. At home I use LTO-4 tapes, which provide plenty of backup storage for over 2 terabytes of data; and whenever it runs a full backup I take the used tapes off-site for extra security.
I have a cronjob running rsync to my server every night. It runs at 20h (8pm) every day which is when I expect the pc to be on and online the most often.
Since it runs every day I don't really care if a give day's run fails because the PC was off, or I had no internet connection.
It runs off of a shell script, so I can run it manually whenever I want to make sure that new data is backed up properly (like when I make significant changes to my thesis, or when I take the laptop somewhere with me).
The problem is pure rsync isn't really backup, it's more like mirrorring.
I like it because this means that the files on the server are exactly like the files on my PC, so I can access them directly. But if for some reason my files become corrupted and I don't notice it, the backup won't help me.
The other problem is that the server lives in my apartment. In case of a fire, or more likey, a commet hitting my home, it's possible to lose my pc and server simultaneously.
The backups are actually done to a small external hard drive. At least in some of those cases, it might be possible to grab the drive and run for it, but I'm still looking at a possibility to start doing the backups remotely.
... with 2 extra hard drives for each one to be backed up. And if you get the external type and one of those drive bay devices that lets you plug externals in, your backup is ready to use immediately (no need to wait days for your stack of DVDs or DATs to be restored, or hours to copy from USB).
now we need to go OSS in diesel cars
I am almost afraid to put this, but
cp -r /home/$USER /media/$EXT_hard_drive/backup$date
It takes a while, but I haven't got round to figuring out how to use rsync yet.
I'm using this in an enterprise level environment, over 100 servers in 24 different locations backing up just over 2500 Winderz desktops. The only feature I find missing in this environment is the lack of a 'single pane of glass' to manage it all. Granted I don't think it was intended to be an enterprise solution, however for the cost and what it does it's simply amazing.
You might want to google the raid 5 hole. Unless you have an NVRAM card, your raid 5 system is just asking for trouble.
So my understanding is that hard drives use magnetism and they gradually lose their data due to the weakening of the field if stored for a long time. So I figure I can just boost the storage time by attaching all of my backup drives to a large electromagnet. So in a few years when everybody's drives are failing mine will in pristine condition. I'd advise anybody else who wants to save their data do the same thing, it is the only way to be sure.
Base classes, properties, functions, & procedures I used were from the TZipFile class from the RunTime library functions in System.zip.pas unit:
---
class function IsValid(ZipFileName: string): Boolean;
procedure Open(ZipFileName: string {or TStream}; OpenMode: TZipMode);
class procedure ZipDirectoryContents(ZipFileName: string; Path: string);
procedure Add(has Compression params here to vary level that are overloaded);
class procedure ExtractZipFile(ZipFileName: string; Path: string);
procedure Extract({has overloads that are doable here});
---
* They all work to do the job for me here for a streaming backup, for about a year now, in a fairly simple program I wrote up for it, which runs every Sunday here on its own scheduler (simple clocktick = 1 week has gone by test on timer)...
However - the nicest part is, when I need to make it different? Since I wrote the code up, I can change it, as needed, whenever I wish...
(That's the best part of "growing your own code" really, AND, sticking to std. file formats (like .zip, which I use in "maximum compression" settings, bit slower, but resulting files are tinier too)).
APK
P.S.=> It makes data copies from my data disk & then a full volume (of my main OS + Programs drive), across 4 other disks (each has a redundant copy).
All disks used are Western Digital Velociraptor SATA II drives (150gb-300gb) @ 10,000 rpm 16mb buffers on each, & all run from a Promise Ex-8350 SATA II 128mb ECC RAM Caching Controller running on the PCI-Express x16 bus (very fast)... apk
For actual backup, I use an old freebie Mozy account. I don't recommend them these days-- apparently there's better for the money if you look. I've tried Spideroak, but christ almighty their client runs like puke under Windows.
I have a small wallet sized external drive that I keep in my safety deposit at the bank. Every couple of months, I go to the bank and switch it with my other external drive which has a more recent backup. Then I take the other external drive home. By having two external drives, I can avoid having to make an extra trip back to the bank, each time.
I also have a slightly larger sized external drive that I keep at home for doing more frequent backups. The other smaller wallet sized external drive in my safety deposit box, is my off site backup. There is always the slight chance that the external drive at home could be lost through a burglary or fire. I use a total of three different external drives for doing the regular backups on my home computer.
On my Linux desktop computer at home, I use the rsync command to regularly update the backups on each of my three external drives.
Another off site option that I have considered, is keeping an encrypted backup on an external drive at a nearby older relatives house.
I'm very excited (about backup software?) about this new backup program from an old buddy of Linus Torvalds':
http://liw.fi/obnam/
It seems like it will be the most featureful, forward-thinking backup software, ever: deduplication across multiple clients, compression, and strong encryption for untrusted storage. He's very keen on unit tests, too, and it has good verify and restore functions. I'm already using it for some things. GPL, of course, so no proprietary lock-in, ever.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I have a dedicated remote server which (among other things) exports via iSCSI a partition on a RAID 5.
On each of the backup-ed PCs/servers a cron script connects to the iSCSI target, open the LUKS volume on the target, and mounts the FS. Then the backup-ed PC/server runs rsnapshot to create a snapshot of its current status. So I can retrieve anything from any of the backuped servers/PCs as it was at anytime. Plus, I don't have to trust anyone on the remote server site because everything leaves the backuped servers/PCs encrypted. Works like a charm!
The specifics of the tools and such I use aren't important to me. What is important is that weekly I push a duplicate full backup to my VPS instance, which lives in a hosting facility in another state. Should my home, or my entire city, fall to some disaster that wipes out everything in my study (and I survive) I will still be able to get to my backup using standard tools that have been around since the dawn of the internet. It might not be terribly convenient, certainly nothing like an actual restore, but the contents will be there, accessible.
I back up by putting my car in reverse. Or by walking backwards.
Thanks for that.
A word to the fellow wise(-asses): it's easier to do a quick search and confirm that you're not the only one with a fast comeback than type the whole thing, submit, and THEN find out.
How many fucking times is this question going to be posted?
I'll just leave the computer switched off until Monday and use Sunday to do something useful.
Problem solved!
Apart from that, I have my normal backup routines anyway, why do something special for one day in the year? Granted there's the usual socially irresponsible bastards who think effing about is so funny, but as I said, I'll be switched off on Sunday.
Have a nice day!
I haven't bothered with offsite backups. I don't need to because I live in Florida and it's not like we ever get hurricanes or anything like that.
I have a 3ware raid card in my 10.04 box with 4 drives in raid 5, as well as an eSATA drive. I export a TB of the RAID array and a TB from the iSCSI drive via iSCSI to two 2k8 servers running in Virtualbox VMs. In the Windows VMs, DFS mirrors the data to the two mountpoints. I export those shares to a Z: drive which maps on login. I set up the free MicrosoftSyncToys powertool to mirror the local My Documents directories to the Z: drive. When SyncToy is run, and the data is backed up in two places.
I have another esata drive which mirrors my home partition every night. This is slightly complicated because I have a couple dozen virtual machines that could be running (it's usually less than 10), so what I wanted was a way to pause any VMs that might be running, back everything up, then unpause. Here's the script I wrote to do that.
#!/bin/bash /usr/local/bin/nightly_backup &>>/var/log/nightly_backup.log /etc/logrotate.d/nightly_backup file /var/log/nightly_backup.log { .gvfs
#
# nightly_backup: Script to pause any virtual machines that are running,
# do an rsync backup, then unpause the virtual machines. Set the SRCE
# and DEST variables below, as well as the USER variable. Script assumes
# that $DEST is a separate partition. If this is not the case for you,
# comment out the line _mount_check below.
#
# Sample cron entry:
# 30 04 * * *
#
# Sample
#
# monthly
# missingok
# rotate 4
# compress
# }
#
# --exclude-from file syntax:
# Copy directory but not its contents:
# + Cache/
# - **/Cache/**
#
# Do not copy (file or directory)
# -
#
# $Id: nightly_backup,v 1.1 2011/12/03 19:23:15 doodleboy Exp kevin $
PATH=/bin:/usr/bin
USER=doodleboy
SRCE=/home
DEST=/archive
ARGS="-aHS --delete --stats --exclude-from=/usr/local/bin/rsync_exclude"
# Function to pause or resume running virtual machines
_pause-resume() {
ARG=$1
VMS=$(su - $USER -c "vboxmanage --nologo list runningvms")
if [ -n "$VMS" ]; then
printf "$VMS\n" | while read VM; do
VM=${VM%% \{*}
printf "Running $ARG on $VM...\n"
su - $USER -c "vboxmanage --nologo controlvm $VM $ARG"
done
else
printf "No VMs are running.\n"
fi
}
# Abort backup if $DEST partition is not mounted
_mount_check() {
if mount | grep -w "$DEST" &>/dev/null; then
printf "$DEST is mounted. Proceeding with backup.\n"
else
printf "$DEST is not mounted. Aborting backup.\n"
printf "*** $(date): Aborting nightly backup ***\n\n"
exit 1
fi
}
# Start banner
printf "*** $(date): Starting nightly backup ***\n"
# Make sure $DEST is mounted
# Comment out _mount_check if $DEST is not a partition
_mount_check
# Pause virtual machines
_pause-resume pause
# Flush pending writes
sync
sleep 3
#
WHS works a lot like Time Machine (I suppose). All our machines are backed up automatically every day to a server (in another building). And most importantly, the restore works and is surprisingly fast! I've had two machines go belly up and didn't loose any data.
Wanted: witty unique signature. Must be willing to relocate.
SVN of course! (Or not.)
In our home of 4 notebook computers, one HTPC and one general-use server, we back them all up using BackupPC. All Linux here -- BackupPC takes extra work for Windows. BackupPC initiates backups with the 'clients' on its own, according to the backup schedule configured in the web-based interface. It uses rsync for efficient network transfer and disk de-duplication for efficient storage (1.4TB of history in 123GB, currently). Backups on the notebooks run in the background while they are in use, and no one seems to notice when backups are running (at least I never hear any complaints).
The server running BackupPC uses linux md RAID 10 with 7.2k RPM SATA drives. Once per week the BackupPC server image is copied to one of several eSATA external HDs. Each HD is stored in its own fire resistant case, but a major shortcoming right now is that the external drives don't rotate to an off-site location. eSATA is a big win -- I regularly see transfer rates in excess of 70 MBps. And BackupPC has a simple web interface so the home users can recover their own files whenever they want.
We've used this setup for about 7 years or so. Its fast, reliable, mostly invisible, and the admin effort is hardly more than plugging and unplugging eSATA drives on a weekly basis. I don't know how many times we've recovered files -- it's so easy no one has to ask me how to do it. I used it to do a 'forklift' upgrade of the HTPC hardware a while back and the recovery went quickly and smoothly.
Me (casual user: hobby coding; amateur art; web; email; games; home finances; TB's of digital media): I backup weekly using Time Machine to a NAS disk I keep on our wireless network. Very Important files (like my personal finances and other important data) I encrypt and backup onto two thumbdrives: one on my keychain and one I keep at the office. (This data amounts to only a few hundred kB.) The digital media goes on to an external USB RAID array and really I don't care about losing most of it. The utmost favorite stuff is replicated on several drives, but can still probably be downloaded again if somehow it all gets lost.
My Wife (business user: business finances; customer communications; business docs; contact lists): Backs up only when I nag her into it, or do it for her, amounting to maybe twice a year and inevitably to a standalone USB portable hard drive using the built-in WinXP backup utility. When her PC crashes, or fails to boot, my first question is "When did you backup last?" -- this is always answered with "Whenever you did it last," and guess who gets stuck with trying to recover the data from her failed HD? The only restores that have ever worked for her have been me mounting the failed hard drive to an OS X machine and copying the pertinent data.
It really peeves me that the default backup options for WinXP SP2 are so fucking completely useless for restore as to be a total fucking waste of time.
Yes she needs a new computer, but no she claims she cannot afford one (despite me telling her that the cost of replacing her laptop before it fails is miniscule compared to the cost of replacing it after the old one fails and loses her data). I'm not buying her one because I need a new one myself but at least mine degrades gracefully. She's not a dummy, just a typical business user who has no interest in or time for thinking through the details and intricacies of data management, not to mention failure modes and recovery scenarios. For these kinds of people it needs to be made automatic, idiotproof, and restore robust. Not an easy task.
I can see the fnords!
I use Boar, it's a version control system for large binary files, like photos and videos. I then maintain a number of clones of my boar repository on external HDDs which I keep outside my home. When people lose data, it's usually due to user error, and that's why version control is essential. Check out Boar at http://code.google.com/p/boar/
RAIDZ3 you insensitive clod! My pr0n is important!
...Cause my daddy taught me good.
I have a 500 GB USB HDD plugged to a pogoplug at home, and have put another one pogoplug+hdd at a friend's house. I backup my macbook pro using arsync over aftp, my linux boxes using rsync over ssh, and my wife copies files from her win7 manually (she only has punctually a few stuff she wants to save) using windows shares because i added samba to my pogoplug (it's a linux and they give you root access). Then, the pogoplug software automagically mirrors my home's pogoplug to the one at my friend's. So there you have local daily backups and automatic off-site backups for 300$ (100$ for 2 pogoplugs, 200$ for these HDDs when i bought them; you'd get one TB each now i guess). And I can access my files using a web interface from anywhere, and give web links to any file i want for friends to download stuff. Cheap, convenient, multiplatform, reliable, you have total control. What else?
My
Well, the point is that it stores diffs of files for incremental backups, rather than backing up the entire file whenever part of it changes. And it keeps old diffs. rsync is fine if you only care about the latest version of the file--but then you aren't protected against human error like truncation or deletion that goes unnoticed for a while. I almost lost my GPG key that way--thankfully I had an old DVD backup using a 7zip file.
I hope you reported the bug. Ones like that may be rarely encountered, and so may go unnoticed until they hurt someone.
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
I set up cron on my laptop to run rsync hourly to sync files from /home, /root, and /etc to an external 1 TB USB drive (when my laptop is docked, since the drive is connected to the docking station). When I finally get the backups on the USB drive to be encrypted then I'll buy another USB drive, and occasionally swap them and throw the alternate in the trunk of my car for a cheap offsite-storage solution.
The company I work for, which is a leader in information security based products, has a lot of road warriors; me being one of them. We use a service called Crash Plan Pro - http://www.crashplan.com/business/ Crash Plan Pro essentially provides you with onsite and off-site backup with a secure cloud storage feature. Basically, for us road warriors, we have a client that is configured to backup certain directories on a certain frequency. It's pretty awesome; we never have to worry about losing data. My laptop does a backup of my C:\users\USERNAME directory about every 15 minutes; only the files that have changed, not a full backup every 15 minutes. At home, I have a Buffalo Linkstation 1TB NAS configured with RAID 1 (disk mirroring). It's an awesome home solution. I've had the NAS since 2008 and I've only had to replace 1 hard drive, which was a cost of 50 bucks off Amazon. Replacing the Serial ATA drive took about 5 minutes and then the RAID rebuild took about an hour. It has FTP and web services that allow you to access your data from anywhere.
I have successfully used time machine backups on my external 2Tb FW800 drive to migrate over from a MBP to my iMac.
Time machine gives me backup and redundancy in case something in a file becomes corrupt or I accidentally delete something. Versions in Lion gives me even greater granularity allowing me to revert to a previous revision of a file in programs that support that feature.
Jesus was a compassionate social conservative who called individuals to sin no more.
Carved in stone, like the Egyptians did.
Don't be apathetic. Procrastinate!
A shell script calling rsync with the --backup option is excellent for having a full up-to-date backup with daily historical changes in separate folders. Then a second script to backup to tape once a month.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
... hit F8 in MC while moving data to a machine with automatic backup.
Thankfully, I used ext4so it was impossible, for me, to recover any of it.
Linux machines rsync to an encrypted & normally unmounted backup drive on the home server, that one backs up to an encrypted external drive.
Windows gaming machine backs up to an external drive using vshadow and robocopy.
"When information is power, privacy is freedom" - Jah-Wren Ryel
Am I the only one who bought an LTO2 drive from ebay to use? I have an external USB drive as my data drive, and just tar it to tape. I keep the tapes at work. The main problems are that I still need more than one tape to backup, and need a scsi card.
If i die, my data goes too, but it wont matter then, will it?
---- Booth was a patriot ----
http://www.youtube.com/watch?v=rpdiXspBALg
SuperDuper gives you a bootable clone - no need to reauthorize applications -- it's better for doing an entire system restore than TimeMachine. Actually, they complement each other. http://www.shirt-pocket.com/
A lot of folks who say they are using rsync are probably doing what I do in a scripted form:
rsync --linkdest /dest/prev/. /srcdir/. server:/dest/next/.
In other words, we're creating whole snapshots which hard-link share all the common files between multiple snapshots. This has the advantages of full backups (being able to read from any snapshot as a normal file tree), incremental backups (having less space usage for common cases of files not changing very often), and rsync differential transfer (being able to send only parts of files over the network when they do change). It doesn't, however, store an incremental representation of the files, so there is more space usage if you have very big files that change a little at a time.
What's interesting is that you can use almost the identical scripted solution but using a copy-on-write filesystem on the backup server, and then you get incremental storage too. You use "--inplace" instead of "--linkdest", but you also issue a snapshot command on the server to remember the previous state of the tree before rsync starts modifying the files again.
I store my media collection and documents on a local Raid 0 (Mirrored Hard Drives) and also backup my documents in the cloud using Mozy Pro. I've also duplicated the bulk of my media collection to an offsite location (Family Member's House/External Hard Drive). I keep this updated incrementally with an FTP process.
The strategy is that the Raid provides convenience against a single drive failure that I can manage at my computer's location. Mozy provides "set it and forget it" daily backups of my crucial documents and the offsite backup is a disaster recovery plan.
To me it all comes down to categorising the data. Some things need backing up daily, others weekly, monthly, annually and others once and never again. Also some things never need backing up. My first step with backup is to categorise the data when it is stored, then use the right backup strategy. If you don't sort the data properly you end up wasting money backing up stuff that isn't important, and may even miss things that are important. My servers are based at home, close to the desktop/laptop machines, so though I have RAID for important data, everything is vulnerable to fire/flood/theft etc... However, I do have a co-located server based 300 miles away to run several websites I look after. When I looked for the hosting provider I factored in the price for extra storage, and then ordered the server with a fair chunk of extra space than I will ever need for the web servers. I now use the colocated server as a rsync destination - no private data (in case it is compromised), but it makes me feel better that me and my family survive a house fire, the family pictures will too.
We run hand-me-down SDLT tape drives from work (new media from ebay), full backups monthly, differentials weekly, incremental daily. Heck of a lot more robust than USB drives; the tapes are exchanged with a family member across town every week (we take their USB drives, they take our tapes, and they also hold a spare SDLT drive, controller, cable, and emergency restore media kit for us)
However so far in 20+ years of operation (starting out with DAT backup drives) we have not had a single failure requiring restore (only drives in RAID or mirror sets) so outside of testing, we haven't needed them.