Slashdot Mirror
How Do You Sync & Manage Your Home Directories?
digitalderbs writes "A problem plaguing most people with multiple computers is the arduous task of synchronizing files between them: documents, pictures, code, or data. Everyone seems to have their own strategies, whether they involve USB drives, emailed attachments, rsync, or a distributed management system, all of which have varying degrees of success in implementing fast synchronization, interoperability, redundancy and versioning, and encryption. Myself, I've used unison for file synchronization and rsnapshot for backups between two Linux servers and a Mac OS X laptop. I've recently considered adding some sophistication by implementing a version control system like subversion, git, or bazaar, but have found some shortcomings in automating commits and pushing updates to all systems. What system do you use to manage your home directories, and how have they worked for you for managing small files (e.g. dot configs) and large (gigabyte binaries of data) together?"
On the windows side there is a great utility called Beyond Compare, around $30, that I have used to do this. I even had a small client once that could not afford a real backup software, so we faked the backup using portable USB hard drives and the Beyond Compare utility to sync her server and desktop to the drives. Worked quite great and the while thing was done for under $200.
If i'd elaborated, i wouldn't have made first post!
However, i use subversion for two things - backup and syncing my development system with my remotely hosted web server. Neither of which is really how i "sync and manage home directories", but if i needed to do that subversion is what i would use.
Some months back, i foolishly pointed to my web hosting service that there was a serious security hole in the way their system (cpanel) was configured for subversion - and they killed the subversion service and haven't reinstated it. So i have to do 'svn update' over an sshfs virtual file system, which is mildly irritating.
Anyway, i've got a single repository set up on my system and i check in all new web sites i'm working on. Then i check them out onto the server - and update the files on the server with 'svn update'. It's easy, reliable, and reasonably fast. It also makes backup nice and easy, as i just sync the repository with a mirror on an external hdd.
Have a look at Jake. Official website: Jake
It is aimed for the average user (no server setup needed) and provides a syncing solution across the Internet with a nice UI. Free and open source, available for all operating systems.
Check it out!
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
for backups I used to swear by rsync plus hardlinks. But since time machine came out it's oh so much much better. For one thing rsync is still a bit unstable on huge directory trees that contain lots of hard links. And it also boofs on some extended type attributes, forks and file types, though it keeps getting better (perhaps it's perfect now). Rsync + hardlinks also does not retain the ownership and privledges and ACL faithfully either.
But even if Rsync + hardlinks didn't have those troubles, time machine is so flawless it's just the thing to use. What is especially nice about time machine is the recovery and inspection process. it's not too hard to figure out what files chaged (there's even a 3rd party gui application for this) and because this info is stored in meta data it's faster and more relaible to retreive than a massive FIND command looking at time stamps. The time machine interface for partial recoveries is intuitive and easy to drill down. In many cases it's even application aware so you can drill not on the file system itself but on say your mail folders in the mail application. this is actually a pretty stunning achievement that needs to be seen to be believed how paradigm shifting it is.
And full recoveries could not be easier. you just boot off the CD and within ten clicks you have picked the souece and destination and it has done a series of idiot checks. While that might not seem too amazing, it sure is comforting. It's a mildly nerve wracking process of trying to recover from a back-up cause there's lots of ways to goof and maybe even wreck your original ( like oops, I didn't do a -delete, or I didn't tell it to reassign links, or worse I copied the wrong direction).
Here's a super nice tip: you can have two disks operating with time machine that you rotate. Actually the best way i've found is to have one constantly attached then on fridays attach the other one, redirect time machine to it, let it back up all the changes since last friday, then detatch it and let time machine go back to the main disk.
You can even use this as a way to sync your two computers though it's better as a backup than as a synch. have time machine back up just your home directory to a thumb drive, take this from home to work. plug it to the drive at work, back it up. then revert this to the backup from home. now home and work are synced plus, if there was one special file or two that was newer at work, well you have that in the backup you made! ( by the way to do this kind of thing requires fiddling with the backup cookie so two computers can share the same repository. google this if you want o know how)
Some drink at the fountain of knowledge. Others just gargle.
Windows Live Mesh
https://www.mesh.com/
"With Live Mesh, you can synchronize files with all of your devices, so you always have the latest versions handy. Access your files from any device or from the web, easily share them with others, and get notified whenever someone changes a file.
Working on one computer, but need a program from another? No problem. Use Live Mesh to connect to your other computer and access its desktop as if you were sitting right in front of it. "
Yes, dropbox is very nice. I'll second the recommendation. Dropbox can also automatically keep previous versions of files around. Works on PC, Mac, and linux.
If you need security, truecrypt with dropbox is nice. Dropbox supports incremental (delta) change file uploads/downloads, which makes large-ish truecrypt containers useful on dropbox. The only real limitations are that (1) you have to unmount the truecrypt container before synchronization can occur, and (2) you have to insure, manually, that only one PC/Mac/linux box is accessing the truecrypt container at any one time.
An alternative to dropbox is syncplicity, but I haven't tried it. The feature set looks similar, though.
Another alternative is jungledisk, which uses Amazon S3 to store your data. The advantages here are that everything is encrypted with a key (stored only at your end, unless you enable the web interface), that you pay only for what you use, and that there's no limit on storage capacity (as long as you have money). Disadvantages include:
The biggest shortcoming of CVS that I know is the lack of ability to rename a file. Yes, you can copy it then delete the original but CVS sees this as a new file with no revision history. If I understand correctly subversion was created by former CVS users to overcome a few shortcomings of CVS with this being the biggest one. Thus SVN has a similar "feel" though not identical commands to CVS and a superior feature set.
I have two main computers, desktop and server.
File layout:
desktop:Documents -- everything I want backed up regularly
desktop:Server -- symlink to latest backup from server
server:Documents -- a few server-specific files, and stuff I always want accessible (I turn my desktop off if I'm not using it).
server:Desktop -- symlink to latest backup from desktop
There is an @reboot cronjob on the desktop PC to backup the server, and tell the server to backup the desktop. I use the rsync --link-dest thing so I can have incremental backups (using hard links for files that haven't changed). There are a few other additions -- automatically deleting old backups (except keep a backup from every 10th day) and updating the symlink to the latest successful backup.
The script is written in ZSH, to take advantage of the fantastic globbing that's available.
The most important lines in the script are:
older=($backups/$user/*(/om))
($older is now an array of directories, ordered newest-to-oldest).
rsync --verbose -8 --archive --recursive --link-dest=${^older[1,20]} \
--files-from=$scripts/${from}2$HOST-I-$user \
--exclude-from=$scripts/${from}2$HOST-X-$user \
$user@$from:/ $backups/$user/$date/
(The variables like $from and $HOST are because I use the same script to copy some stuff to my laptop, but that has a small drive so I don't copy everything. I think the strange syntax after --link-dest expands the array like --link-dest=/dir/one --link-dest=/dir/two ... --link-dest=/dir/twenty)
over2weeks=( $backups/*/???????[012346789]-????(/Omm+14) )
end=$(( $#over2weeks - 5 ))
rm -rf $over2weeks[1,$end]
($over2weeks is an array of directories, being backups not taken on the 5th, 15th or 25th day of the month, and at least 14 days old.
$end is the length of the array minus 5)
I have found that using Subversion (svn) with the aid of a bash script that is run manually actually works really well and provides a number of special advantages. Here's how I have it constructed:
First, I don't actually make my whole home directory a svn checkout. I have a subdirectory in it that is the checkout, and my bash script ensures there are symlinks into it for the things I want sync'd. This makes it easy to have some differences between locations. In particular, I can have a different .bashrc for one machine than another, but keep them both in svn as separate files; it is just a matter of making the symlink point to the one I want to use in each location. My bash script will make the symlink if the file doesn't exist, and warn if the file does exist but isn't a symlink. It does this for a number of files.
Another benefit of this method is that I don't put all my files in one checkout. The core files I'll want in all my home directories (e.g. .bashrc, .vimrc, ssh .config and public keys, etc.) go in a checkout called "homedir". But my documents go elsewhere. And my sensitive files (e.g. private keys) go somewhere else still. I choose what is appropriate to install at each location (usually just the "homedir" checkout on boxes I don't own). My bash script detects which checkouts I have and does the appropriate steps.
The bash script not only sets up the symlinks but it also does an "svn status" on each checkout so I'll know if there are any files I've created that I haven't added, or any files I've modified that I haven't committed. I prefer not to automate adds and commits. I'll definitely see any pending things when I run my sync script, and can simply do an "svn add" or "svn commit" as necessary.
I also prefer not to automate the running of the sync script. I like being in control of my bandwidth usage, especially when connected via slow links (e.g. Verizon EV-DO, AT&T GPRS). Plus dealing with conflicts is much easier when it is interactive (although I can usually avoid that scenario). It also simplifies authentication to run it from my shell, as it can just use my ssh agent (which I forward, which is setup in my sync'd ssh config).
The sync bash script takes care of a few other edge-case issues, like dealing with files in ~/.ssh that have to have certain permissions and whatnot. And I've taken care to ensure that the script doesn't just blow away files; it will warn if things don't look right, and leaves it to me to fix it.
Using Subversion has another big advantage: it is likely to be installed already in many places. So when I'm given an account on someone's computer, I can usually get my environment just the way I like it in a few short steps:
svn co svn+ssh://my.server.tld/my/path/to/svn/trunk/homedir ~/homedir .bashrc not being a symlink and whatnot .bashrc
~/homedir/bin/mysync # This is my bash script to do the syncing
# Correct any complains about
~/homedir/bin/mysync
# Log out and back in, or source
No fuss, no muss. No downloading some sync package and building it just to get your .bashrc or .vimrc on a random box, or asking the admin to install something. Subversion is usually there, and even if it isn't, most admins are happy to install it. Subversion deals well with binary files, and even large files. For bulk things (like a music library), I'm more likely to rsync it, partly because it is bulk, partly because it doesn't benefit from versioning, and partly because it only needs to be a unidirectional sync. I could easily add that to my sync script.
I am simply in the habit of typing "mysync" from time to time (my .bashrc puts ~/bin/ in my $PATH). This works for me very nicely. Some people may prefer a little more automation, and of course my script could automatically do adds and commits, and even skip the log messages. But I prefer a bit more process; after all, this is my data we're talking about!
If there is interest, I may post my sync script.
I had problems with earlier versions of LiveMesh but I haven't had any problems for a while. For a few months from November 2008 to February 2009 I'd been using both DropBox and LiveMesh. Now I'm using just LiveMesh. For me what swung it for LiveMesh was:
- Being able to sync any folder on my PC, e.g I sync my favorites folder between 4 PCs
- Being able to easily control which PCs get updated with what as I don't want everything synced between all my PCs
- Being able to configure folders to be transferred just PC to PC, e.g. I have 30GB of family photos that I sync between my wife, myself, and my parents accounts on their respective PCs. I don't need these photos in online storage
One tip though when using such sync tools - keep a backup as if one person trashes the folder it trashes everyones folder
Australian running a company that does C# / C++ / Java / SQL / Python / Mathematica