Slashdot Mirror
How Do You Sync & Manage Your Home Directories?
digitalderbs writes "A problem plaguing most people with multiple computers is the arduous task of synchronizing files between them: documents, pictures, code, or data. Everyone seems to have their own strategies, whether they involve USB drives, emailed attachments, rsync, or a distributed management system, all of which have varying degrees of success in implementing fast synchronization, interoperability, redundancy and versioning, and encryption. Myself, I've used unison for file synchronization and rsnapshot for backups between two Linux servers and a Mac OS X laptop. I've recently considered adding some sophistication by implementing a version control system like subversion, git, or bazaar, but have found some shortcomings in automating commits and pushing updates to all systems. What system do you use to manage your home directories, and how have they worked for you for managing small files (e.g. dot configs) and large (gigabyte binaries of data) together?"
Subversion.
I recently started playing around with Dropbox for some smaller folders than my entire home directory and haven't yet run into any major problems. And the versioning it provides is nice as well, and as a plus they don't consider the deleted files that they still retain versions of as part of the quota.
rsync + hardlinks
Since I don't have a lot of content to sync, Dropbox meets my needs perfectly (and it's free!).
I'm always positive; it's my nature.
Myself I simply store contact info "in the cloud", use my MP3 player to hold all my music and occasionally sync it to all computers to have a copy, any needed documents are either somewhere on my e-mail or on a USB drive, same with code. I have different computers for different purposes and have different data on each one. I never really liked the idea of having the same everything on all computers, most of my computers have different OSes, distros and desktop environments.
Taxation is legalized theft, no more, no less.
I'd be interested in answers for this from the "I want the same homedir contents everywhere, but NFS makes me vomit" standpoint :)
For managing the default profiles around the office we use git - the dotfiles & such are managed, and the rest is left as an exercise for the user.
It's not ideal (I hate it), but it's what we've got...
/~mikeg
I have been considering this more of late as I utilize my new netbook. It's my first portable computer ever; I've clutched to my desktop since the beginning of time.
My comfort with this is directly related to who I can trust to store my data with. Right now, it's not a big enough need, so I'm reluctant. However, I can see my needs growing to a point where I'm going to take the plunge.
Any recommendations or horror stories anyone wants to share are welcome here.
I've used a program called SmartSync since the early part of this decade. I think it was around $30, but it does a great job of synchronizing, including deletions. I do believe it's a Windows-only app, though, so while it may not work for your situation, some other slashdotters may find it of some use...
Do Windows users even have anything like this, aside from Active Directory?
Strictly speaking this isn't about my "home directories", but I use a combination of things to sync the information I need between my desktop and laptop:
1. Windows Live Sync (formerly known as Foldershare). Yeah, I know, it's from Microsoft, and only available for Windows and OS X, but it works pretty seamlessly and without much intervention on my part. I sync specific directories of projects I am working on using this; it's a set it and forget it tool.
2. Foxmarks for bookmarks.
3. Zotero, version 1.5 of which has the ability to sync resources between multiple computers. This is a godsend.
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
Replication with DRBD.
I use multiple OS X, Linux, and FreeBSD machines daily. One cannot sync all home directory files, as all the config stuff differs between Gentoo, Debian, FreeBSD, Tiger, and Leopard. So it's mostly down to documents, graphics, and a few audio and video files. For the larger ones, I use a usb stick, the smaller ones I email to myself so they're always available via IMAP servers. But most of all I have a bootable, customized version of systemrescuecd installed on a 16GB usb stick, which at any given moment has all the currently important stuff I need. It works well enough for me.
Caveat Utilitor
I wouldn't recommend using one tool for every purpose. I wouldn't want to store multi-GB files in SVN, and I wouldn't want to store all my code on an external hard drive. Maybe using DropBox, or rsyncing with a server somewhere would work.
Live Sync does a fair job of syncing folders in Windows (sorry) hosts. Whenever both computers have internet access, the folders sync. Live ID required (again, sorry.) Few limits on data (no files larger than 4GB, no more than 20,000 or so folders?)
Live Mesh is a similar Microsoft offering, but also give you web access to your files and stores a copy on the cloud. Sadly, there is a 5GB limit on total data.
-- Halfabee
I just store anything saveworthy on the server at home. That way the "client" computers (my main "work"station/gamestation and laptops) contain only relatively expendable data. And if I need to access the stuff from some other location I just SSH into the box.
And backups are handled from the server manually occasionally to an external USB drive. I know, I need to improve on that part.
while true; do eject; eject -t; done
Not sure how much you use bookmarks, but I maintain a good archive of organized chaos myself. At one time, bookmarking a site was a pretty pointless endeavor, since I couldn't keep track of whether I had bookmarked it at work, on the bedroom computer, downstairs computer, laptop, etc.
So a while ago, I looked up a plug-in for Firefox called Xmarks (formerly FoxMarks) that was pretty much exactly what I was looking for. Very easy to set up and use, cross-platform, etc.
Just thought it was useful enough to mention here.
For small backups, every ten minutes, I use backintime (based on rsync). For larger, nightly or more rare backups, I use rdiff-backup. Both work over the LAN, or to locally-mounted hard drives.
I carry a 16 Gig USB flash drive with my working files on it. I've using this method since the days of 100 Meg Zip drives and just keep upgrading the media. My flash drive is automatically backed up to my backup server at home in the middle of the night so, if I forget it at the office, I'm only a few hours behind. Besides, I can use free Logmein to log into the office computer and transfer a file if it's got new and important information on it. It works the same way in reverse if I forget it at home. Since my working files are on the USB drive which is also compatible with my Linux machines, it really doesn't make much difference which machine I plug it into. Did I mention encryption? That's a good idea in case you lose the drive if you've got any sensitive information on it.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
FAST 2009 has a paper on semantic data management using a file system built on top of an object store powered by MySQL. Performance isn't great, but it uses a distributed file system solution to solve the synchronization issue in a very nice way (e.g., synchronize all albums with my iPod, all photos with my laptop and computer, etc...). You can specify rules and I liked it when I heard about it. However performance is actually important, despite their claim :).
Perspective: Semantic Data Management for the Home
Brandon Salmon, Carnegie Mellon University; Steven W. Schlosser, Intel Research Pittsburgh; Lorrie Faith Cranor and Gregory R. Ganger, Carnegie Mellon University
HTML Paper
http://www.usenix.org/events/fast09/tech/full_papers/salmon/salmon_html/index.html
PDF Paper
http://www.usenix.org/events/fast09/tech/full_papers/salmon/salmon.pdf
Slides
http://www.usenix.org/events/fast09/tech/slides/salmon.pdf
--"You are your own God"--
http://www.getdropbox.com/ Works perfectly for me. I use the free option, which has a 2 GB limit, which is more than enough for me to keep all my important stuff in sync. It has a client that integrates nicely with the Nautilus file browser on Ubuntu, which is what I use at home and office. Whatever you put (or symlink) into a designated folder (which you can choose) gets mirrored to their server, from where it gets synced to every other system where you have installed the client, the next time you connect that system to the internet. They also give web-based access to the stored files. There is the issue of privacy for the really paranoid, but I am not very concerned about that with the files I currently choose to mirror. I am more worried about the chance that their client develops a bug that wipes out my files, but I guess I'll take that risk.
freeware @ http://allwaysync.com/
I was playing with this for the first time last night, it gets the job done. Sync software with a nice GUI, and I was easily able to backup my systems to a 1tb backup drive in a reasonable time period. It has the usual features and can sync in multiple directions (one to many, bidirectional or one way).
Be patient with large files
.svn/text-base/ area. This means that your working copy takes up at least twice as much disk space as the original dataset. Beyond that, the Subversion client follows a (currently unadjustable) algorithm for committing files:
.svn/tmp/ (can take a while, and temporarily uses extra disk space)) .svn/text-base/
A nice feature of Subversion is that by design, there is no limit to the size of files it can handle. Files are sent "streamily" in both directions between Subversion client and server, using a small, constant amount of memory on each side of the network.
Of course, there are a number of practical issues to consider. While there's no need to worry about files in the kilobyte-sized range (e.g. typical source-code files), committing larger files can take a tremendous amount of both time and space (e.g. files that are dozens or hundreds of megabytes large.)
To begin with, remember that your Subversion working copy stores pristine copies of all version-controlled files in the
* Copies the file to
* Performs a binary diff between the tmpfile and the pristine copy, or between the tmpfile and an empty-file if newly added. (can take a very long time to compute, even though only a small amount of data might ultimately be sent over the network)
* Sends the diff to the server, then moves the tmpfile into
So while there's no theoretical limit to the size of your files, you'll need to be aware that very large files may require quite a bit of patient waiting while your client chugs away. You can rest assured, however, that unlike CVS, your large files won't incapacitate the server or affect other users.
Really, I think he's asking for one tool to do both small files and large files when (in my mind) it makes more sense to back up ISOs and MP3s over longer periods of time than my source code or documents that I may edit and change daily.
Subversion for source control. A simple script that pushes large files to an external drive. That's all I do. Bulletproof? No way. But it sounds like he's devoting a lot of time to this. I guess he must have a lot more computers than I do.
My work here is dung.
There are numerous ways to do this, but I would use Samba (best used with VPN), because it can pretty much be used everywhere, even on computers with crappy OSes like Vista. There are numerous other solutions for the same thing; FTP, AFS, maybe iSCSI.
All my data goes on a Samba file server in the garage (which is separated from the house so hopefully in the event of a fire I don't lose both) and my various machines access this, with rights according to the user used to mount the Samba shares. This runs a nightly incremental tar to an external eSATA drive. Every time I copy any data I really care about (like new photos) onto the server I also run rsync to sync it to my machine in the house. Future plans include FreeNAS with ZFS (and hopefully snapshot support), and a number of external hard drives that I can keep at parents houses.
I have found that a portable hard drive with an encrypted partition works best. Put DSL in the bootable partition in case I have to work at an untrusted PC. FEBE and gmarks cover the worst of my needs on Firefox. If someone swipes my drive, they don't get my data.
On the windows side there is a great utility called Beyond Compare, around $30, that I have used to do this. I even had a small client once that could not afford a real backup software, so we faked the backup using portable USB hard drives and the Beyond Compare utility to sync her server and desktop to the drives. Worked quite great and the while thing was done for under $200.
I save any data I might need shared to my server at home, and just map a drive on all my pc's I use. I only use Windows at home, so its pretty simple and works just fine.
I spent a long time tackling this, as I am situated at different locations on different days.
I have 2 desktops and a laptop which must remain sync'd and encrypted. I use TrueCrypt for the encryption.
On my Windows boxes - SyncBack handles it. It can be triggered on write or on insertion, or just periodically. Has version control support. Will sync over FTP (poorly) and can create zip files or burn Cds etc. It's a swiss army knife of sync tools.
The key for getting the most out of a sync program is granularity. Inevitably, you'll have exceptions, and you don't want a PASS/FAIL result for your entire backup set. It works much better to sort files into categories and sync the individual groups than to try to make one profile that does your entire disk array. My 2 cents.
If you keep everything really organized a usb stick and an external hd suffice. Occasionally overwriting old files on the hd to keep "backups" of the usb stick. :) I'm looking forward to replacing my usb stick for a 32+ GB usb stick so that I don't have to worry about extra hd's.
The problem is with my music collection is 30GB and my usb stick is 2GB
At home, I've got a Linux server hosting an LDAP structure to mimic MacOS X Server's config. It is sharing my home directory via NFS. My Macs sync this home directory on login and logout, so all my personnal data is centalized for easy backup and available on any Mac I happen to add to my home network.
Menzoberranzan Networks
Whenever I'm on my workplace network I have cron job that uses rsync to sync my Documents directory between my linux desktop & mac laptop. This way the latest file is always on both machines. The Desktop is also rsync'd to a backup server daily, and weekly for off site storage at bank vault. And the Laptop uses Time Machine at my home. This allows me the flexibility of grabbing older files if I need them. For mail I use imap with SSL. So short of a nuclear holocaust, it would be pretty hard to lose a file.
And when the server hosting your NFS share dies, so does your entire home directory on every PC. Check and mate.
Right now, actual synchronization tends to be entirely manual, with scp of subdirectories and possibly a tar -c | tar -x combo to not overwrite newer copies.
I decided to work on data integrity first--but then I have client info to consider.
I have a 3x1.5T zraid array using full disks on the main machine, and an external 1.5T for backup (I'll grab another, so I can have two alternating backups). These will stay disconnected when not backing up, and in two other rooms of the house. I'll probably copy zfs snapshots of /home, probably filtered for any /cache/ and so forth. I also have offsite backup that i haven't gotten around to enabling ):
I'm planning on actually figuring out rsync, and from there specifying the parts of /home/* that get synchronized (possibly to an nfs mount off the main machine?)
hawk
FTP back and forth, select the root and overwrite whatever's newer. Unless the time on the files gets screwed up, it works fine. Worst case scenario, which is the dates/times getting messed up, the FTP client downloads everything. No big deal. I do it daily for all kinds of files.
As older, wiser programmers than myself have always told me: KISS: Keep It Simple, Stupid.
I use DropBox to keep (small) files synced across several machines -- I know some people object to keeping data 'in the cloud', but that's what backups are for. DropBox means no forgetting to svn update or commit and has never really gotten in my way. I keep things like my Firefox profile (bookmarks, addons, etc) in sync by symlinking relevant files in my Firefox profile folders to files in my DropBox. It's a bit of a pain to set up initially but only needs to be done once and then my browsing session is seamless from one machine to the next. I also use this method to sync stuff like my calendar, address book (no $99 per year to Apple for MobileMe, thanks), and SSH and bash config files (always symlinked so that I never need to actually move things).
Primitive, but it works for me.
What?
If you have enough files and move between systems frequently enough that this is an issue, consider a USB flash drive.
Store your files there.
Keep backups on each machine.
Want versioning? Seems to me that files typically have a datestamp for when they were last modified. That's all the versioning people need 99.9% of the time.
If you're in a situation where this is a big problem (many users many files many machines), you want your damned files stored in a more permanent location, such as on, I don't know, a file server?
Save your fucking files to the server. If you're away from the server, upload your file when you get back. When you get to another machine, grab the files you intend to work on.
Hell, Windows has supported this shit for ages with offline files and the whole "Briefcase" bullshit.
http://www.microsoft.com/windowsxp/using/mobility/learnmore/offlinefiles.mspx
http://support.microsoft.com/kb/307885
And yes, Windows does a simple versioning and backup with shadow copies.
In your case, 2 Linux servers and OS X, just sync shit on a schedule when all machines are on the network if you want. The 2 servers should always be in sync with each other (since they're servers and should always be up and networked).
The laptop is the only issue, and should sync when connecting to and before disconnecting from the network.
Any way you want to go about syncing files is fine. Might I suggest a simple GUI drag and drop to/from the server? Seems to me most users can handle that, as long as you beat into them how to know which direction to do it. You could simplify this by making a simple script users could run. This script could include making backups on the server so we don't have issues of people going the wrong way, and so you can timestamp each old version (useful for keeping files for various projects grouped together, so people can grab old versions of a project if that's what they're working on).
You don't need services to handle this for you.
You have 2 servers and 1 laptop. I would say you don't need anything to handle this for you. I wouldn't even go as far as to keep a flash drive laying around. Just, you know, remember to grab files you're going to work on before leaving with your laptop, and remember to reupload those you've changed when you get back to the network.
If you've got a more complicated setup (multiple users accessing and modifying the same files at the same time) THEN you need a version control / checkout service running, and even then, none of them are intuitive, and users WILL get confused and break shit. Especially when you're dealing with mobile users who will be away from the network for unspecified periods of time.
I use Mac (a MacBook Pro and a Mac Pro), so I just do rsync -Cavz ~/Documents/SomeDir/ /Volumes/myusername/Documents/SomeDir/ and then usually an rsync the other direction, and I'm done.
I embed all my documents in porn and post them on various web forums. The recovery procedure involves spidering my spam folder. I recently found my high school history term paper in a jpg of Marylin Chambers.
Insightful and funny are really the same thing, except one has a punch line.
Unison
I use the following tools to stay synced in various ways: Plaxo offers a fairly good Outlook sync utility for free which keeps my Address Book, Calendar and Tasks synced on my work and home desktop computers. Dropbox offers 2-3 GB of free storage that automatically syncs to any computer logged into that account. I keep all sorts of stuff in there. Photoshop prefs and tools, automation scripts, encrypted password database, etc. I set up a cheap home server with FTP. It goes to sleep if unused and I can wake it up with a Magic Packet remotely before doing file transfers. For large libraries of software, music, etc. I have a portable hard drive that I sync on either end using Directory Toolkit about every week or two depending. And Foxmarks for Firefox.
I've used Subversion for years to sync my various systems. I have four different machines (2 Vista, 2 Linux) and 20GB of data that must be kept in sync.
Of course, there could be somethign much better out there. I'm just very comfortable with Subversion, and it works.
All about me
iFolder seems to work best for me. http://www.kablink.org/ifolder I use the Novell/Suse version though.
For items that are larger, or that do not change so often, I use iDisk. This is just a fancy Webdav server that I do not have to manage.
I keep programs on an external hard disk. This is where I also keep my photo library and music and videos. I use one machine for Photos, so I do not really have anything to sync there. My music is not synced either, but I have used some third party software to hel with that.
It is getting to the point where if something goes wrong with a machine, I can have new one set up will all my data in a day. In normal circumstances, I can use any one of three machines and prety much have up to date information.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
I have 1 server computer that stores all videos, photos, music, etc. My other computers and Xbox 360 connect to that one. For work related documents and source code the same server also runs Source Safe.
I've been using CVS for over a decade now. Generally with small files like .cshrc, .emacs, etc.
And when the server hosting your NFS share dies, so does your entire home directory on every PC. Check and mate.
It's not necessarily as world-ending as you make it sound. I use NFS to mount home directories as well as a couple others. I also have redundant hard drives on the Linux server, and each night rsync makes the backup drive a mirror image of the main drive. I could schedule that to run more often, but I don't change things often enough to make more than once a day necessary. If the main hard drive dies, I can very quickly and easily switch over to the backup drive temporarily until I get a new drive with all the data on it. If the server itself dies, it wouldn't take significant time to get it set back up. The only way I'm really stuck is if the main OS hard drive (separate drive altogether) dies, and even that only keeps me without my home directory for a week tops. For some applications, that may be too long, but in those cases you could simply have the rsync nightly backup also copy everything to a backup directory on a second server that could be your backup NFS host. The main article said that the user has two Linux servers at his disposal right now, so that could easily work. It would take all of about 5 minutes to get the backup server to point /home to the backup directory and get NFS shares on the laptop to point to the new NFS host. If all you're looking for is syncing between hosts, NFS is the way to go. If you're looking for versioning too, then it may not fit the bill quite as well.
Isn't that what everyone uses Microsoft's desktop Briefcase for? I thought that's the ultimate in synchronization tools.
I pay $100 a year for a Panix shell account and keep my data there. My own little server in the sky :-)
I setup Novell Open Enterprise Server 2 on a mini Linux server and use iFolder for syncing folders and files across several Windows and Linux desktops and laptops for myself and my wife.
Between my Desktop, Server, Laptop, and iPhone I just keep whatever data on them I want and move what I want when I want. I can always access any other machine from any machine (mobile internet) so I always have access to the files I want or need.
Allows me to share with others as well.
Doubletake software makes an enterprise ready, real time replication suite.
It does block level replication, so only the changed bits of, say, a 10Gb databse gets changed.. It uses on the fly en/decryption so that the data streams are somewhat smaller than they would be otherwise..
I work for a Fortune 10 company, and when we have a need for real-time data replication, this is what we use.
for backups I used to swear by rsync plus hardlinks. But since time machine came out it's oh so much much better. For one thing rsync is still a bit unstable on huge directory trees that contain lots of hard links. And it also boofs on some extended type attributes, forks and file types, though it keeps getting better (perhaps it's perfect now). Rsync + hardlinks also does not retain the ownership and privledges and ACL faithfully either.
But even if Rsync + hardlinks didn't have those troubles, time machine is so flawless it's just the thing to use. What is especially nice about time machine is the recovery and inspection process. it's not too hard to figure out what files chaged (there's even a 3rd party gui application for this) and because this info is stored in meta data it's faster and more relaible to retreive than a massive FIND command looking at time stamps. The time machine interface for partial recoveries is intuitive and easy to drill down. In many cases it's even application aware so you can drill not on the file system itself but on say your mail folders in the mail application. this is actually a pretty stunning achievement that needs to be seen to be believed how paradigm shifting it is.
And full recoveries could not be easier. you just boot off the CD and within ten clicks you have picked the souece and destination and it has done a series of idiot checks. While that might not seem too amazing, it sure is comforting. It's a mildly nerve wracking process of trying to recover from a back-up cause there's lots of ways to goof and maybe even wreck your original ( like oops, I didn't do a -delete, or I didn't tell it to reassign links, or worse I copied the wrong direction).
Here's a super nice tip: you can have two disks operating with time machine that you rotate. Actually the best way i've found is to have one constantly attached then on fridays attach the other one, redirect time machine to it, let it back up all the changes since last friday, then detatch it and let time machine go back to the main disk.
You can even use this as a way to sync your two computers though it's better as a backup than as a synch. have time machine back up just your home directory to a thumb drive, take this from home to work. plug it to the drive at work, back it up. then revert this to the backup from home. now home and work are synced plus, if there was one special file or two that was newer at work, well you have that in the backup you made! ( by the way to do this kind of thing requires fiddling with the backup cookie so two computers can share the same repository. google this if you want o know how)
Some drink at the fountain of knowledge. Others just gargle.
No - he was trying for 5, but only got -1!
When dealing with source code for the software projects I work on, I store everything on SVN repos hosted on my quasi-server box (SVN, CUPS, assorted DB's, Trac, etc). Whatever machine I need to be on (dev machine, XP build machine, etc), I just sync the checkout.
For regular files, I keep most of my stuff on a 1TB drive that is NFS'ed to all my machines. If I need to do a lot of work on a big file, I pull down a copy, edit it locally, then push it back to the share. This handles all of my needs for the most part. My Linux, BSD and OSX machines are good to go, my two XP machines aren't. One is only for playing games that choke in Wine, the other is an old box that just runs a build script for my apps, so I've never bothered to try to get them configured to handle NFS shares.
Both methods are cheap, simple, and fairly pain-free if you keep Windows out of the mixed-mode environment.
del.icio.us has plugins for IE, Firefox, and there is a less useful third-party plugin for Safari which make it easy to "sync" your bookmarks across computers, so I use that for my browser synchronization.
TrueCrypt keeps my really important data (passwords, resume, other sensitive personally identifiable info) safe and secure on my USB drive.
My USB stick on my keychain holds a copy of TrueCrypt to boot from directly when you plug it into a USB port (you need admin authority on the computer you're using to use this feature though), and then some other miscellaneous documents I wouldn't want to lose but aren't sensitive sit on my USB stick in generic folders.
And lastly, my iPod holds a copy of all of the music I care to not lose. (My wife and I also have a 750GB backup drive attached to our iMac at home to keep all of our media files, like photos and video, backed up)
Everything else is either done "in the cloud" online for us, or is proprietary or sensitive data that shouldn't be getting moved off of the primary computer it is on anyway.
All my Macs in the house (5) mount their home directories from a central server. This server was originally a Linux box and the mounting was done via NFS. About a year ago I got fed up enough with Linux to convert that server to a Mac as well and it now shares all home directories via AFP.
On the notebooks I have two accounts: one that mounts the home direcotory when I am home and in wireless range and another account that is a separate desktop for when I am not home. I'll occasionally rsync files from the non-central home directory to my server. I make sure that I stay on top of it so that files do not get forgotten (set a specific desktop folder for content you want to keep/sync).
HTH
Look into sshfs. Keep your home machine ssh-reachable (isn't it already?) and you'll be able to ssh into it, mounting the filesystem on your desktop. It's convenient, secure, and effective. Works anywhere ssh does. Good stuff!
--- For a good time mail uce@ftc.gov
I agree wholeheartedly. I was just knocking the person before me down a notch or 2. Apparently using NFS shares is 'hacking'. I would prefer rsync by FAR over a simple NFS share. It provides redundancy as well as maintains deletions, etc.
http://www.gbridge.com/ very underrated
-- if you mod me down, I will become more powerful than you can possibly imagine
tar cvf - /home | bzip2 | aespipe -w 10 -K /etc/homekey.gpg | bittorrent archive.`date +%s`.foo
boycott slashdot February 10th - 17th check out: altSlashdot.org
DirSyncPro is FOSS, I just started using it a few months ago, and I love it. My company now uses it to keep our two programmers synced(sp?) to the server. Works great, I hope they package it better for MS soon, there is no installer yet, just an executable.
My current setup is a combo of unison/sshfs/rsync.. I've been using it for quite a while at this point, and it works so well that I don't even give it any thought anymore.
I have a Media Center/Fileserver box at my house that is always on, and acts as the 'master' copy of the home directory, on all my workstations my .xinitrc/.xsession calls unison to sync my home dir with the server root as the preferred copy, then calls my WM, after my WM exits, unison is called again with the local homedir as the authoritative copy. A well-crafted ignore list is crucial, ignoring things like temporary file patterns, mozilla cache, machine-specific data, obscenely large files, etc.
I use sshfs-fuse for any of the above mentioned ignored files, sshfs allws you to mount remote filesystems locally, tunneling through ssh (or something like that).
Finally, rsync can be used to give time machine like backups. In my case, I backup to USB drive connected to the fileserver, which I only turn on when running backups.
This is for the most part Linux-only, but unison has a windows binary available, and could probably be used to sync to windows automagically.
"Obscenity is the crutch of the inarticulate motherfucker." - cloak42
I have AD on my file server (and DNS, exchange, etc.) but setup through group policy folder re-direction of the "my documents" to a \\server\user\mydocs directory. Every user that has a domain gets these files created automatically. I also created the local settings folders as well (thus outlook and things like that) in the \\server\user\settings directories. For my linux box I create a folder /mnt/user then in the /etc/fstab create a mount there with full rights. Then finally in my linux box home directory, I have a documents folder, which I create a link to the mount point.
For MP3 and other files they are stored in a separate directory on the same NAS on the windows box, create mounts for them in linux, and can access them via windows shares published in AD. However I use Jiznora as the front end to access all my music and files.
It is not a lot of work to setup, and took maybe 30 minutes once I had the servers up and an AD setup. All pcs can connect in my house (wireless, etc.) and also my xbox 360.
I tried to sync my homedirs between machines, because it's annoying not having my settings and saved passwords etc on every machine..
But, the homedir on my desktop is 300gb in size, neither of my laptops even have that much space on them, especially the netbook.
On the desktop i can keep everything i need, but the netbook needs to keep as small working set of whatever i'm working on at the time.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
I absolutely love allway sync. Very cheap ( I normally never buy software, so low price helped ) but completely simplifies syncing between two home and one work PC.
It's never screwed me up once (fingers crossed!)
Open Source Drum Kit, LPLC deve board - mjhdesigns.com
I don't know that managing them *together* is all that useful. What I have been doing (and what I think is a more flexible way to manage stuff), is to divide the stuff in your home directory into independent 'projects' (e.g. financial documents, stuff for work, source code of my website, project X, project Y, my photo collection...) and manage each project separately in a way that lends itself well to the kind of file being stored. For a directory of small files that are frequently updated, Git is a great way to go. For synchronizing and backing up large collections of large files (like an MP3 or photo collection) you might try something like ContentCouch (disclaimer: I wrote this tool).
Duct tape, XML, democracy: Not doing the job? Use more.
i've tried many approaches - none worked. solution seems to be online. i've already moved my bookmarks to faviki, music to grooveshark, pics and vids to facebook, docs to to google documents. ... i only haven't figured out what to do with my films.
Oh I definitely wouldn't use NFS by itself as there are no backups then. In that I agree completely. The only problem with using rsync by itself is that you either have to run it almost continuously or live with the fact that there will be periodic differences. Using NFS ensures that all home directories are always identical, and then having an rsync backup at an interval adequate to your usage provides the redundancy you're talking about in case of disaster.
As someone else pointed out somewhere else in these comments, it's probably best to not try to use one tool to accomplish everything. It's both a blessing and a curse that most tools for Linux are very specialized. Each does something very well, but you don't typically find the "one size fits all" solutions either. Set up a combination of tools that provides you with all the functionality you need and you'll be completely satisfied.
If NFS is hacking, then cool. I'm a hacker! Sorta... :)
And I hadn't seen the post you were replying to, hence my failure to understand the full context. Sorry about that. :)
I'd like to give a shout-out to the Samba team and creators of RSync. Version 3 and later is what you want. It has many goodies that make backing up a joy, incleding an off-line incremental backup feature that I find really handy.
Still, I'm looking for a good graphical rsync that does not use cygwin on windows though.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
I set up another machine with 6x 1TB HDDs, mounted them, enabled them in samba, connect to them on any of my other machines. If you need redundancy, set up RAID or svn/cvs for code revisioning.
Maybe I misunderstand the question but this seems pretty straightforward for those that might frequent this site. Did the option, "file and repository server" seem too obvious? Why use an online tool or some special software to share between systems? Why store the same data on multiple machines? Just mount a shared drive and read/write to/from it.
-SaNo
I am looking at Unison, and it looks interesting ... I may end up using it.
However, I'm also considering setting up FileVault on my laptop, because I don't want client data compromised if my laptop gets stolen.
What's the chance of getting a sync tool like unison working while one (or both) of the computers in questions uses FileVault?
I stole this sig from someone cleverer than me.
I used to have two computers networked to each other and I would name or label certain folders as shared folders and copy the folders' contents over. With two Macs, you can just use Appleshare over TCP/IP, with a Mac and a Windows machine, you can use SMB, or use a flash drive or some such. Drag only new files if you like, or Select All and make sure you're just keeping the newer versions.
It's not an elegant one-click solution, but it is a simple Command-A and drag solution.
I've got a home Subversion server with separate repositories for documents, settings, contacts, and projects. Been like that for five years now, and it's remarkably stable and nice. For anything data intensive, there's Flickr, del.icio.us, Gmail, WordPress, etc., with a private backup just in case.
For large amounts of files (100s of GBs), the cheapest way to back them up is to use a couple sets of external hard drives, always keeping at least one set off-site.
I would like to know why this is the case. Why is there no service out there that can provide backups for large amounts of data at a price that is competitive with using external hard drives? Such a service should be able to take advantage of scale by storing the bulk of the data onto industrial tape drives, and only retrieving it as needed.
It would require a lot on bandwidth to do the initial backup, but once that is complete, only incremental backups are needed. A database could store the file names, dates, sizes, and hashes, in order to determine what needs to be updated.
This would not be for data that needs to be accessed repeatedly - it is for backups. So they might charge a modest fee to recover a few files which you accidentally deleted. If you lose an entire drive, you could select a new drive model, and they'd ship it to you with your data on the drive, for the cost of the drive plus a service fee.
Is this not feasible, or has it just not been done?
With well over 2TB of RAW files from my digital cameras, a bunch of music, and really all kinds of stuff that I've saved or generated over the years, there's no really good way to manage a huge file library over multiple computers - nor is there any point in doing so.
For years a Windows Media Center Edition PC that my wife won at the state fair (in a truth telling contest) did double duty as a file server. I recently let the smoke out of it with some horrifically bad RAM, and am building its replacement using FreeBSD with MythTV, and using the ZFS as the underlying fabric of what I hope to be a long-term stable and expandible storage server. That way there is one master copy of everything - no merging two divergent variations of a music library that were once identical (for example). I have saved a great deal of space, too, with the on the fly compression that ZFS offers.
Doing this and then sharing the resulting system over SMB and NFS is a hell of a lot less painful than anything else.
The bottom line is that no solution is going to do what you want perfectly. If you really want to tweak your backup/sync process (and learn a lot while doing so) then your best bet is to develop your own scripts in which rsync (and ln) are the real workhorses.
Personally I like doing synthetic backups which are like incremental backups, but the unchanged files are hardlinks (NTFS actually supports hardlinks on MS systems). I don't like storing meta-information about my backups, for some directories like .mozilla I like to perform deletions, however for others like Videos, I never want to delete anything, just free up space on particular drives.
Then when you have a nice script, just through it in your crontab and you're good to go.
My script has matured a lot over the years from a simple static procedure to a full-fledge program with config file parsing and very nice command line operations with getopts.
for example if I use online syncing utilities for my home drives, how do I know someone else won't have access to them? If I am writing a book or software project, how can I know someone won't steal it from me and claim it as their own?
I'd like to set up my own file server and have the synch software store the files there or to a USB drive or something. My old method is to burn CD-R and DVD-R disks, but they can get lost or scratched and then the data is gone. I have, for example, a DVD-R full of my old documents before I reformatted one of my systems. It is lost somewhere in my house, or maybe someone visiting stole it. I need a better solution than that. but I cannot afford to set up my own file server. So I set up an XP box with peer-peer networking for my Network drive and Printer server. I'd much rather use Linux but my Linux box is an old 700Mhz Celeron system and when I ran Fedora or Ubuntu Linux on it, it gets data corruption after a while as it only uses IDE hard drives and I cannot afford a UPS system. Our power goes out a few times a month in our area due to storms and trees hitting power lines. Which contributed to my data being unrecoverable on the Linux system and it forced me to reformat it again and again, until finally I gave up on it. Plus trying to stay up to the latest and greatest upgrades I think trying to install Fedora Core 9 over Core 8 caused the system to have problems with libraries and even running the X-Window GUI with GNOME which forced it into shell prompt mode.
My XP system is a 1.4Ghz AMD Athlon system, not even dual core, but it should be able to make a good Linux server if I can replace it with a new system for running my Windows programs on. Since I am on disability since 2002 I cannot afford new systems and make due with what I have.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Every file I download, generate, work with, etc. stays with me on a small portable hard drive. I also use the portable apps versions of abiword, firefox, etc. I only attempt to do CAD work on the machine in my office, where everything is zen for drafting. I set up an Rsync to copy the contents of the drive to the raid 5 in my office machine at 8:00 every morning. This is way to use my paranoia to the advantage of making sure I come to work on time, satisfying my boss as well.
Spending Resources on Defense leaves Less to defend.
I use the MobileMe service from Apple. The 20 Gb iDisk is enough for syncing files between my several computers. It's mounted on the desktop of my various Macs and with the iDisk's cache, copying files is instantaneous and I can work on my files while offline. When I get back to an internet connection, it syncs. I can also access my iDisks using the webinterface or mount the iDisk in WebDAV on Windows. Also MobileMe Sync my iPhone and my computers Calendars, Address Book, Keychains, Bookmarks, Dashboard Widgets, Dock Items, Mail Accounts, rules, signatures, smart mailboxes, Notes, Preferences (including serial numbers) and FTP favorites. I can use the me.com website to access my adress book, calendars, files and emails from any web browser. Configuration is very simple (login, password, thick a checkbox) and require no software install.
Brice Le Blevennec, Digerati
Call me a paranoid, but I don't trust sending my entire home directory somewhere over the network with a dubious encryption protocol. I use unison with my usb stick as the root of a "star" sync topology, and if I need versioning for some project I create a git repo inside the unison'd directory. Works fine for me.
My first program:
Hell Segmentation fault
Maybe this it to simplistic, but for non critical data I email myself with the attachment. Has search and can be accessed anywhere with web access.
File sharing. Independent backups. I don't understand why the average nerd would need anything fancy at home. Keep your movies on your media box, personal files on whatever box you access the the most, and don't worry about it.
- Windows Live Sync (I know, its Microsoft -- it wasn't when I started using it! It was called FolderShare, and after some hiccups, the re-christened product works just as well) keeps my docs in sync
- Mozy keeps my archives backed-up
- IMAP on Google Apps for your domain keeps my mail in sync
- Google Calendar with various, platform-specific syncing solutions
- Plaxo (blech, but it works) keeps my contacts in sync
- Foxmarks keeps my bookmarks on each computer
- iTunes sharing + AppleTV + AirTunes + two iPhones (one for me, one for my wife) works well enough for sharing media seamlessly
I have two Windows boxes (desktop for work, and a mini laptop) and two Mac's (Mac Mini media "server" and a MacBook Pro) all are in-sync with each other all the time...
Unison to keep directories in sync on multiple machines rdiff-backup for backups and keeping old versions
- Donny was a good bowler, and a good man.
Is it just me or has that site gone down already?
WFM
Reply to That ||
I have a single fileserver (soon to be replaced with a plug computer whenever mine ships).
Everything pulls from there.
Every once in a while I plug in my 1TB external USB driver and sync from the main server, then unplug and put back in the safe.
When I retire my fileserver and move to the plug computer, I will use my 1TB external for the server and buy another one for backups. It will be formatted with a different filesystem.
I use an NSLU2 with an attached 160GB drive as a network backup server that does nightly (actually eveningly) rsyncs of the home directories on my wife's Mac and my Jaunty box. The NSLU2 gets backed up onto DVDR every couple of weeks. I also have rather large collections of MP3s and photos, which are similarly backed up and exported read-only via NFS for all to enjoy. Since these collections only grow, they go on DVDR every time there is enough new material to fill a new one.
Having more then one PC does not mean I have more then one home directory. I just use NFS.
Don't fight for your country, if your country does not fight for you.
Currently I'm doing my masters and I have two types of files that I have to move between two locations. 1) Students Grades 2) Tex Files (for thesis work) So, my solution was that I ssh into my FreeBSD system at home and then use xforwarding to make my spreadsheet program or tex editor show up on my office client. This lets me keep one record of my documents in my home location and all traffic is strongly encrypted. (just in case oliver is a student).
It works on Macs, Windows PCs, and mobile devices, (apparently, but I do use it on the first two happily). Could you not share your desktop folders on your devices, and get MS to sync it all for you? Alas, Linux is excluded. 5Gb space online last time I looked.
Has anybody run into SyncToy's unnecessarily wanting to overwrite files that differ only in 'date modified'? It would be a minor issue, hadn't MS not implement multiple/Shift (un)check.
How did you solve the problem?
MS put SyncToy 3.0 towards Q4 2009 (ie. 2010), so I either have to find a solution to the above or get a suggestion for an alternative?
So, any suggestions for (preferably free) alternatives?
Multiple computers does not mean they all need to be identical. If I need something I get it from the home server otherwise each computer is for it's task.
I dont need all the family photos on my acer aspire one when we travel. Same as my Mac editing system does not need my resume or my letters to relatives on it.
I find a home server is far more useful than syncing machines.
Do not look at laser with remaining good eye.
Fileserver and an NFS export.
My home directories generally don't contain anything that I need distributed. If I need something in a home directory that is not there, then I will copy it there from wherever it does exist. I only really have two computers that I actively use though, so I just do document versioning by hand if there happens to be something I was working with on my laptop and desktop. This is really a problem for people? I would think that anybody who regularly uses more than two computers (desktop + laptop) would be sufficiently capable of setting up something that works for them....otherwise, they probably really don't need to be using so many different computers.
A bash script that runs unison on certain directories if that directory exists. About ten for different directories (like ~/music, ~/movies, etc) .kde as what works on my 7 inch battery powered "netbook" is probably not applicable to my giant monitor high powered desktop. /usr/local/bin all by itself somewhat virally.
I have no interest in backing up dot files like
My tiny laptop for space reasons does not have a ~/audiobooks directory thus the script does not sync ~/audiobooks.
The script of course distributes itself and updates itself into
For system backup (closely related) I only back up config files and have my unisonsync script back it up to all my machines. /bin/bash since there are about 300 world wide debian mirrors that back that up much better than I could ever dream of doing, and an amd64 /bin/bash would be of little use on a replacement i386 machine, etc. /etc/network/interfaces or /etc/ntp.conf etc.
There is no point in backing up
All my machines share a unison'ed ~/backup directory... structure like ~/backup/server/stuff ~/backup/media/stuff ~/backup-mythtv-upstairs/stuff
So, all my machines have a backup copy of all important config files or changed files on all the machines I control. For the eight or so machines it consumes only a couple megs.
I backup things like
mysqldump makes an appearance or two.
When I recently set up "mythtv-downstairs" frontend, I pretty much worked thru each file I backup for "mythtv-upstairs" and it worked, so its of much more use than disaster recovery.
My script also runs certain monitoring scripts and dumps their output into the backup system. run cpuid and dump the output into ~/backup/machinename/cpuid. So in the event of total and utter system failure I know the exact specs of the dead box without any memorization or googling. (did it have one gig or two? easy, read the file ~/backup/machinename/free) I save a copy of the output of lsmod, cpuid, free, df, cfdisk -P s /dev/whatever, cat /proc/mdstat as appropriate, lots of other stuff.
On the big server, a cron job weekly tars up ~/backup and stashes it. Occasionally I burn this vast collection of backup files to a CD or copy to a flash and then store it offsite. I also make offsite copies of my relevant ~/whatever directories as I see fit.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I've been a very happy Unison for many many years. But after recently switching to Ubuntu for my laptop and desktop I finally discovered the joy of Unison over SSH and also discovered the auto=true and terse=true command switches which makes the whole thing automatic, just like Dropbox (I think there's one more switch and I won't even have to press 'g' to run the sync).
I've tried other solutions - DropBox is too much in the 'cloud' for me; rsync is only one way (what's with that?), Robocopy is Windows only, svn/hg/bzr/git store versions and all I want is two way sync.
Unison is the peak of perfection in two way sync; there is nothing else...
The heart of my network is a xen based server running fileserver, DNS, DHCP, web proxy and LDAP.
Physically, the system is an older Athlon 2200 with 3G RAM, mirrored 100G disks for the base OS and disk files. The software stack is CentOS 5.3 based, running xen.
The fileserver virtual machine has its own 500G single disk assigned to it one which I keep media files, pictures etc. I run a rsync backup script to another physical machine. The really critical stuff (family pictures, irreplaceable documents) also get backed up to an external hard drive every week or so (no set schedule). I don't have an automated process to back up my OS images though.
I run CentOS directory server (now Port389). Windows and Linux clients can authenticate, but most systems have local authentication. I also use autodir/autofs on Linux/Solaris systems. This allows me to login on any Unix/Linux system and have my entire work environment ready. I have the following .profile that lets me keep separate profiles for each login:
## Profile Script
LDAP_HOME: Contains the NFS automount directories for a subset of LDAP users.
LDAP_CURRENT_HOST=`hostname -s`
if [ -f ~/.hostconfig/${LDAP_CURRENT_HOST}.profile ]; then
. ~/.hostconfig/${LDAP_CURRENT_HOST}.profile
fi
For my Linux clients, I automount the fileshare to /mnt/fileserver. Each home directory has a symlink to that mount point. You can put this in the skel startup so that each new user gets the link. This allows anyone on my network to play music, watch movies, view pictures, from the share. I set up the directory structure on the fileshare as follows:
Media: Contains subdirs for Audio, Video, and Images.
Documentation: Contains subdirs for Computer, Household, Film, etc..
CVS: The main CVS repository for my work
Software: Contains subdirs for Windows, Linux, MacOS, Java, Solaris
For the Windows clients, I define a network drive on each system pointing to the fileserver. I can also access the CVS server via Eclipse when I need to do something with Java or Windows Perl.
There are some downsides to my setup. I tend to upload images from my Windows machines and for the most part, these are laptops that connect wirelessly so synchronization can take hours to upload an 8G CF card. I'm happy with the setup, however. Placing something in a "critical" folder means it gets a backup rotation so I can retrieve earlier versions. Other stuff is backed up, but not at a high priority and no versioning.
Yea, it's called RAID, or backups.
Tool.
Hello,
I have built a java program. You can find it here: http://www.os10000.net/fs/java/app_dsync/index.html
Features:
* it is GPL
* it is used for exporting and importing
* it creates a digital certificate for the machine it's run on
* it creates a 1-1 relationship with a machine that it's synching with
* it creates an export file on the source machine & imports it on the target machine (you have to move it)
* the export file is a zip file
* you build a ruleset on the export machine (files, directories, regexes) what you wish to export
* you build a ruleset on the import machine (same) what you wish to import
* these two rulesets give you total control even when you're exchanging with someone else
* you have rules for "soft master", "hard master", "soft slave", "hard slave", "progress", etc.
If you can use "unison", use that. If you wish to automate, use "app_dsync".
Have a good day,
Oliver
I don't share EVERYTHING, but I share some things:
Stating on Slashdot that I like cheese since 1997.
My Linux home directory is pretty tiny - only the dot files for my Linux environment (.gnome etc). I keep all my work documents and files on my OpenSolaris fileserver where ZFS provides resilience using RAID1 and point in time restores using the snapshot capabilities. I NFS mount the ZFS filesystem to my Linux box and CIFS share the same filesystem to my Windows PC and Mac.
My MP3 collection and photo albums are handled by iTunes and iPhoto respectively, syncing with my iPhone. The Mac backs these up to a Time Machine disk which in reality is a ZVol on my OpenSolaris server published as a LUN using iSCSI.
The ZFS filesystems and volumes are backed up to external USB drive using the "zfs send" command.
Blatant plug: I've documented most of the above experience on my blog.
For bookmarks, I use Xmarks to synchronise with the cloud, and take notes using Evernote.
With OpenOffice using the OpenDocument format for files, which is XML (plain text) in a zip file, or even MS Office's non-standard XML formats, it would be nice if version control systems could efficiently track and store changes of these files. I have found plugins for SVN and GIT that can reach inside the zip container and do simplistic diffs on the text, but each revision is still stored as a complete binary file.
I guess I really just need to learn LaTeX and do everything in plain text files with a good editor like vim. But in the meantime, having a XML-in-zip-file- aware version control system would be nice.
Seriously, data management problem has caused me to become paranoid to create files and documents in anywhere other than my main computer. I have over a terabyte of files to share over several computers with different OS: Ubuntu, Windows XP, Windows XP x64, and Mac OS X. Currently the only effective way I have is to share the files through samba/windows file sharing over gigabit local network. Gigabit ethernet has give me decent performance and the speed is almost like local access, I can even stream HD movies over the network.
Initially I used a dedicated Windows XP machine to host the files. Because it is always painful when Windows restart or crashed after opening too many applications and my files would be temporarily unavailable. The previous release of Ubuntu had some mysterious bug that made the speed of samba sharing slow, and I wanted to host my files using cross-platform filesystem that have read/write access over different OS, and NTFS seem to be the only option.
Recently I have migrated my files to be hosted on Ubuntu using ext4. The samba bug no longer appeared in Interpid, and the performance has been satisfactory. There is still some other bug that cause frequent disconnect/reconnect over the network, but I still can bear it for now. The other draw back of hosting in Ubuntu is that I can no longer hot plug my hard drives to XP and Mac OS X.
Currently the only annoyance problem I have is on my Macbook, where I won't have access to the files local network when I'm not at home. I am especially paranoid to create any file on the laptop, fearing management overhead to backup or move around the data to appropriate network folders. I still have no good solution to manage the photos I store in the macbook and the desktop. Currently the photos are stored in separate Lightroom catalogs. Even though I do backup the photos but I cannot do any write operation on the backup copy on my desktop, as that would destroy the integrity with the original catalog on my Macbook. My previous photo management tool, Aperture, was even worse that I cannot do any backup of the catalog to the network or NTFS partition as that destroys the chmod permissions and render the whole catalog unusable.
I am currently developing a web application that can also be used as a home/local web server that can manage and synchronize specific file formats as efficient as the web server. For example, imagine a Flickr-like web app, that is installed on both your laptop and desktop and is not only able smart-synchronize all photos, but also automatically publish selected photos to Flickr, Facebook etc, and is powered by the very same program that powers these websites.
Please tell me if anyone have the same experience or have a better solution for terabyte file management.
Yes, Linux is way primitive when it comes to synchronization technology, as in other areas. Windows has a history of providing more-or-less effective solutions for this out of the box, going back at least to the My Briefcase feature of Windows 95. Now there are Offline Files, Live Sync, Sync Center, and even a more consumer-level gadget called Sync Toy. Another example of how much time you can waste trying to use Linux for serious business use.
This is not a self-referential sig.
I use Mercurial, and I don't know why it wasn't mentioned along with the other three.
Need a Python, C++, Unix, Linux develop
We have an environment that includes various versions of Linux, FreeBSD, IRIX, Solaris, and Windows. We use NIS and NFS for all the UNIX systems and SAMBA for Windows.
It works very well for us.
My sig is wonderful. I love my sig.
Currently? Just unison -quiet, running from cron. (I have it wrapped in a script that does locking, since Unison doesn't seem to lock against itself reliably, for reasons I don't understand.) I've had two problems worth watching out for:
1) Try to avoid running it against NFS. It walks the entire synced area every time you sync. Local disk will be two orders of magnitude faster.
2) Be careful syncing between case-sensitive and case-insensitive filesystems. Unison will start failing out if you ever create two files differing only in case.
Beyond that, I'm looking to start using git to version both my code and my textual data. I'm not intending to use git itself to sync the repositories; I'm going to use it for versioning only, and keep syncing using Unison. The reason is because I'm the only user, and for my own convenience I'd like the working copy to be synced. All I really need out of git is versioning anyway; I already have a workable solution for syncing.
I just try to avoid duplication of data.
I have backups here and backups there. One full 500 GB drive full of prized photo's.
A few restores and un-deletes and presto I can find 3+ files of the same thing.
I need a NAS that can do backups and help me avoid data duplications. Now only if the 5+ GB NAS can come down in price.
My Sig indicates the end of the comment I posted.
Nighly (or more frequently, if you like) rsync to an OpenSolaris server running ZFS w/ Time Slider.
Quality versioned backups with little effort, plus data integrity (checksums built into the filesystem), compression, and (if desired) RAID-Z(2) goodness! In addition, the provided time slider interface allows easy browsing of versions.
Just my 2c...
PathSync (GPL)
an interactive directory (path) synchronizer for windows
PathSync can analyze two directories and show the user a list of differences between the directories.
The user can select what actions should occur (which files to overwrite, which to delete, which to ignore), and allow PathSync to synchronize.
I use this in a jiffy, it works by comparing the dates and file sizes of the files; and what's there or missing; and just copy/delete to make a clone of the source.
Furthermore, it's portable (just copy the directory) although I am not sure if it leaves anything behind in the registry
Bonus is that it can save/load the settings that you use before.
Dropbox looks pretty awesome. I just spent some money on a home NAS array so I'd prefer to do that. What's the closest thing to dropbox that I can put on my local NAS array instead?
I use shfs mounts by ( to make sure it stays mounted even if connection is interrupted) and ssh tunnels for everything else, with preshared keys to a central server / proxy, and rsnapshots for backup on the central server with hot swap drives.
This works on desktops, remote office, and for notebooks. I essentially don't trust my employees or myself to remember to encrypt everything or use "secure" protocals all the time, and so I remove the need to remember from the whole process. I can then focus on securing one system. Great if everything else is secure, but just in case. Very good for notebooks jumping from open wireless to open wireless systems, and also keeping track of employees activity in one location. I can log fairly easily everything they do or don't do (yea, the 2 hour coffee break sticks out like sore thumb in the logs).
Among other things this also has the nice side effect that should say a notebook or desktop be stolen, it will phone home as soon as it is connected to the internet and send detailed information about what it is doing.
Living in Chile
I would highly recommend Dropbox. I've been using them for close to a year now, and have never run into any major problems. It "just works", which is important for something that you don't want to think about. Having the files in the cloud means you don't need to keep both PCs on at the same time (older apps such as FolderShare required this). The delta sync and the "instant upload" if the file already exists on their server (even if it's not in your own dropbox...for example, if you put a Linux ISO in your Dropbox that someone else has already uploaded to them, it identifies it based on hash and you then don't need to upload it -- it sounds like a privacy issue, but it's not [no hash collisions], read about it on their site) really speed things up. The backup and versioning features are nice, too.
Dropbox is also cross-platform, so you can use it to send files to/from your server if you want to (takes a bit of fiddling to make it work under CLI-only, but their Wiki has instructions on that)
I use Mozilla Weave (hosting my own server since the public one is long since full) for syncing Firefox settings, it works fairly well, but is a pain to configure and troubleshoot sometimes.
I manage all my home directories with Unison, including 80Gig of pictures across Linux and Windows. The only issue I've noticed is that for single files over 1G it gets slow, so I split up my thunderbird folders to get under this. Everything else just works.
-- Anthony Staines
http://kimbriggs.com/computers/computer-notes/linux-notes/samba-setup-ubuntu704-guide.file
I use .MAC (now mobileme) to sync my two macs, this, combined with iphone sync of contacts, calendars and notes makes all my environment transporable. My first mac is a Macbook Pro, the second is a Dell Mini9 to carry everywhere and keep all that vital information with me at all times.
The mobileme web interface makes a lot of sense to me also so in cases of not carrying my laptop i can always access my contacts, files, etc...
In a word - NFS. When you only have 1 home directory there's nothing to sync!
Multiple?
I use automount, with nis and nfs. Linux and Solaris. New machine comes on, it just has the login/home directory. Naturally, it doesn't work on any of the inferior OSs. Neither Windows nor Mac OS X work with this setup "out of the box". Windows can mount my home directory with CIFS. And I just haven't bothered with Macs, though I suspect it would be the same.
On the road, I use unison to sync my home directory into work files, and then resync on return, if I know I will have limited connectivity. I use fuse sshfs if I know I will have reasonable connectivity, mounting my real home directory onto a subdirectory.
All this to keep a SINGLE home directory, under my control, with minimal effort (zero) needed to bring up additional machines.
Just another "Cubible(sic) Joe" 2 17 3061
My way is very simple. I'll connect to the other machine using Filezilla over sftp. Then drag my folders over to my local PC. When prompted, I allow newer files to over-write older ones. It's fairly fast and very simple. Just make sure both computers have a ssh server.
I store users' roaming profiles and home directories on a server running Windows SBS 2003. The server's storage is a SATA RAID-5 (3ware rocks!). SBS backs itself up to disk weekly, which I occasionally transfer to an external hard drive for DR purposes. The profile and home directories are separate SMB shares because the share containing the roaming profiles is configured to disallow client-side caching (which causes problems with the user profile loader on older versions of Windows and maybe even Vista). The shares are accessed via MSDfs because some day I'd like to replicate them to a second server and want any accesses or fail-over to be somewhat automatic (again, for DR purposes). I use Group Policy to move each user's "AppData", "Contacts", "Desktop", "Documents", "Downloads", "Favorites", "Links", "Saved Games", and "Searches" folders to their home directory. In my scheme, "Music", "Pictures", and "Videos" are sub-folders of "Documents", for backwards compatibility with Windows XP. I've also configured Volume Shadow Copy, which allows users to retrieve older versions of their files without needing to bother me about restoring them from archival backups, and deployed Certificate Services on SBS. Each user's enrolled in the domain PKI, so they can encrypt their caches as well as any of their files.
From the users' perspective, everything is automatic: They log in, work with their files, and log out. If they are out of the office, they'll get a warning about working with a cached copy of their profile, but that's about it. When they return, they'll get prompted to sync any conflicting changes made while offline. Windows has featured CSC (also known as "Offline Files") for some time, but it's only gotten really stable in Windows Vista. A few programs don't really play well with CSC but nothing that's a deal-breaker (like Firefox or Skype storing database stuff in the roaming version of the AppData folder when it really should be in local version instead, but I kind of brought that on myself when I redirected it to the network share to start with).
I'm proud of my Northern Tibetian Heritage
Disclosure: SpiderOak is my primary contractor, I do stuff to help their infrastructure. That said, we do versioned, encrypted, zero-knowledge backup of Linux, Mac, and Windows machines.
I use cygwin rsync (via vixie's cron) to sync .git repos and large files. Though /etc (again to an external repo via a
files from my various Windows clients to my
Linux backup host. On my Linux backup host I
use rdiff-backup (again as a cron job) to
backup / sync to an external USB drive. I
tried gibak to externally version my $HOMEs
(.git as a symlink to an external USB drive
filesystem) but had problems (git was not
built to manage 20GB of content)
Anecdotally, git had problems processing
nested
git was a poor choice in versioning my $HOMEs,
it proved to be a great choice for versioning
my
symlink). For this, I used metastore and
etckeeper (a git wrapper utility).
Whatever happened to iFolder ? iFolder
leather-dog muksihs
Blog: @muksihs
I have found that using Subversion (svn) with the aid of a bash script that is run manually actually works really well and provides a number of special advantages. Here's how I have it constructed:
First, I don't actually make my whole home directory a svn checkout. I have a subdirectory in it that is the checkout, and my bash script ensures there are symlinks into it for the things I want sync'd. This makes it easy to have some differences between locations. In particular, I can have a different .bashrc for one machine than another, but keep them both in svn as separate files; it is just a matter of making the symlink point to the one I want to use in each location. My bash script will make the symlink if the file doesn't exist, and warn if the file does exist but isn't a symlink. It does this for a number of files.
Another benefit of this method is that I don't put all my files in one checkout. The core files I'll want in all my home directories (e.g. .bashrc, .vimrc, ssh .config and public keys, etc.) go in a checkout called "homedir". But my documents go elsewhere. And my sensitive files (e.g. private keys) go somewhere else still. I choose what is appropriate to install at each location (usually just the "homedir" checkout on boxes I don't own). My bash script detects which checkouts I have and does the appropriate steps.
The bash script not only sets up the symlinks but it also does an "svn status" on each checkout so I'll know if there are any files I've created that I haven't added, or any files I've modified that I haven't committed. I prefer not to automate adds and commits. I'll definitely see any pending things when I run my sync script, and can simply do an "svn add" or "svn commit" as necessary.
I also prefer not to automate the running of the sync script. I like being in control of my bandwidth usage, especially when connected via slow links (e.g. Verizon EV-DO, AT&T GPRS). Plus dealing with conflicts is much easier when it is interactive (although I can usually avoid that scenario). It also simplifies authentication to run it from my shell, as it can just use my ssh agent (which I forward, which is setup in my sync'd ssh config).
The sync bash script takes care of a few other edge-case issues, like dealing with files in ~/.ssh that have to have certain permissions and whatnot. And I've taken care to ensure that the script doesn't just blow away files; it will warn if things don't look right, and leaves it to me to fix it.
Using Subversion has another big advantage: it is likely to be installed already in many places. So when I'm given an account on someone's computer, I can usually get my environment just the way I like it in a few short steps:
svn co svn+ssh://my.server.tld/my/path/to/svn/trunk/homedir ~/homedir .bashrc not being a symlink and whatnot .bashrc
~/homedir/bin/mysync # This is my bash script to do the syncing
# Correct any complains about
~/homedir/bin/mysync
# Log out and back in, or source
No fuss, no muss. No downloading some sync package and building it just to get your .bashrc or .vimrc on a random box, or asking the admin to install something. Subversion is usually there, and even if it isn't, most admins are happy to install it. Subversion deals well with binary files, and even large files. For bulk things (like a music library), I'm more likely to rsync it, partly because it is bulk, partly because it doesn't benefit from versioning, and partly because it only needs to be a unidirectional sync. I could easily add that to my sync script.
I am simply in the habit of typing "mysync" from time to time (my .bashrc puts ~/bin/ in my $PATH). This works for me very nicely. Some people may prefer a little more automation, and of course my script could automatically do adds and commits, and even skip the log messages. But I prefer a bit more process; after all, this is my data we're talking about!
If there is interest, I may post my sync script.
the idont bother.
My ism, it's full of beliefs.
...to print documents off my main computer, then a scanner to get those documents onto my second computer.
I then repeat the process in the other direction to create a seamless 'bidirectional' sync solution.
For video, I simply record the footage off the monitor using a camcorder, which I carry with me. My video, on the move.
"We live in a global world" - Harvey Pitt, former Securities and Exchange Commission Chairman
I use a constellation of git repositories, and Joey Hess' mr tool to synchronize all of them. I have no automated commits -- I just remember to commit and update manually daily.
I've done exactly the same as you, used every single tool under the sun, eventually settling on Unison until I realised I was being silly...
Let's put it this way - just set up each computer how you want it, and sync the *data*, not the whole home directory.
For instance, my Documents are synced with Dropbox (though tempted to move them to UbuntuOne), my development directories are generally stored in some kind of revision control (svn/bzr/git) and either not synced or at worst, unison-ed, and everything else just stays on the machine it was created on, and backed up with duplicity to a central fileserver hosted in France.
When you realise that syncing home is *not* good, it suddenly becomes clear what you need, and what you want are completely different.
You can read more about it here:
http://blog.gnu-designs.com/snapshot-backups-of-everything-using-rsync-including-windows
Basically I'm using rsnapshot to back up everything, Linux, BSD and yes.. even Windows, with relatively pain-free results. The added benefit (for the Windows users) is that they can browse the snapshot hierarchy (exposed via Samba), to get back any files they want from the hourly/weekly/monthly snapshots on the array.
It works beautifully here.
On Windows the free MS SyncToy works OK for me. On *nix I use rsync. Backup media is a slew of USB drives ranging in size from 120 to 750GB.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Our company has had good luck with two products.... Novell Storage Manager and Novell iFolder. We don't use other Novell products but these two are rock solid. Once we get past the "Novell" stigma our customers won't go back to the old manual way of doing things in microsoft active directory (MAD) or to old home directories in whatever other system they have. On-th-go employees like iFolder because it keeps everything up to date regardless of location and is fast. Larger customers love NSM because of policy-based storage management which is really quite amazing. The support for the products, when there is the need, is great too.
I first got into syncing when I bought my now-departed Thinkpad. I looked at unison but didn't like the way it seemed to dump large amounts of configuration stuff in my home directory. So I wrote scripts using rsync - one set to sync stuff from the pc to the laptop, and one set to sync the other way.
/etc/hosts) and carried on as usual. Then I discovered grsync which is a graphical front-end to rsync.
When I got my EeePC then my Advent (MSI Wind clone) netbook I simply edited my sync scripts (and
Rather than having a script for each directory branch, I just fill in the source and targets, then select which options I want, e.g. which perms to preserve, whether to compress the data (useful when syncing while away from home), whether to delete files that are missing on the target, etc. So instead of a set of scripts, I just have one config file in ~/.grsync which is, of course, pure text and can be hand-edited if necessary.
What's even better about grsync is that between the source and target directory fields is a double-headed arrow; click this and the source becomes the target and vice versa. So I sync to the netbook, do stuff while I'm out of the house, then click the double-headed arrow and sync back to the pc. Couldn't be simpler.
Grsync also has fields for commands to be executed before and after rsync is called, so you can copy, move, zip, write logfiles, etc, etc. It also has a field in which you can enter additional options to rsync, so you could, for example, enter --exclude '*~' if you wanted rsync to ignore certain backup files.
I suppose I should add the usual disclaimer that I'm no relation to the author or his dog. You can find grsync here: http://www.opbyte.it/grsync/
Garry Knight
I keep all of my important files and media files on the server, it also backs up my computers.
I can access it if I am not at home via SSH.
I'm also going to add a cron job and an external HDD for offsite backup.
No one mentioned this yet? Simple, low bandwith, easy to remigrate to a new comp/after a reformat... works great for me (mostly only use for docs, but large pdfs included).
Works over any internet connection I have ever used.
Its MS, get over it, it just works. :)
I think Unison is probably still the best tool for bidirectional sync of directory trees. Unfortunately, it's written in OCAML. In principle, OCAML is a nice language, but Unison is written in a pretty awful style, and, more importantly, the whole thing is kind of hard to port.
15 cents per gig per month, amazon s3; jungledisk for win, linux,mac, $20.00. maybe not for video, but docs, pics, etc...cheap and automatic.
I have a server on the Inet with Debian and root access for websites and CMSes of critical long-term customers and my own project versioning and testing. I use a seperate usergroup for each project and also have a usergroup for my Project directory that basically has around 10 years worth of my projects in it. I sync using the CLI client for ssh+svn on OS X and Linux. If I had Windows in there somewhere - which I don't since about 7 years ago - I'd have TortoiseSVN to cover syncing there.
I find this a practical solution as I can access my current stuff from *anywhere* at any time and I can use the same skillkit I use for my daily developement work at the job. At work we also use SVN for project specific docs and media. The tools are there, svn is tried and true and you can also fix unreversably borked versioning with a little XML editing inside the .svn directory if things should go completely haywire.
For archiving I'd actually archive off the repository itself. Haven't seen the need to do that yet though. Backups I still to with alternating/overturning drag and drop copies of entire dirtrees via the Finder or Nautilus/Konqueror on to external HDDs - which is a leftover from the before-svn-sync times which I intend to change someday and integrate into the pipeline.
We suffer more in our imagination than in reality. - Seneca
I'm happy.
~/art (stuff that I create and work on): git :-)
~/compsci (for the uni course, mostly code and latex): git
~/fetches (random crap that gets downloaded to look at): not synced
~/music (guess): rsync
~/personal (various mostly text documents): git
~/photos (photos organised into $date-$eventtitle/$photonum.jpg then left alone): rsync
~/src (code I write): gets uploaded to FTP and the world backs it up for me
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
I have a simple Windows Small Business Server 2003 running in my home on a very old Pentium III with 1 GB of RAM. It's the primary storage for all my stuff, and my email/calendar on Exchange.
My desktop accesses it directly, but my laptop (with My Documents redirected to the server) is set to use Offline Files.
I don't have *everything* set to go offline -- my laptop HD isn't big enough for that. But, key folders are synced by right-clicking and choosing "Available Offline". It's as simple as that, and works flawlessly.
If I'm away and need a file that I don't have, I can connect back to the server via VPN and there it is. Or, I can mark it for Offline sync and I'm done.
Everything is very transparent, and works great.
And for email/calendar, I do the same with Exchange and Remote Mail (plus my phone uses Windows Mobile).
-David
I put my important files on ftp://ftp.kernel.org/ and expect the rest of the world to mirror it.
I like to sync my files with PowerFolder. I can directly sync my computers in LAN or online and don't need to upload them to any service provider. ...And it works perfectly between my Linux and Windows box. I think a Mac client is also available.
I have a server on the lan, accessible via the web that does automatic backup every night? What else do you need?
I'm not anti-social, I'm anti-idiot.
JungleDisk.com
I've been using CVS for my personal files for years, and I'm happy with the choice. If I had it to do over again I would choose SVN or git. I keep my important files synchronized on 6 machines I use regularly.
Since JungleDisk was purchased by RackSpace, they've added support for RackSpace Cloud Files. Storage pricing is $0.15/GB the same as S3 in the US, but there's no charge for data transfer.
On the other hand, Amazon has importing in beta - you ship them a disk or disk pack, they import it into S3 for you. $80/device plus $2.49 per import hour, no bandwidth charges. eSATA or USB, up to 8U, up to 50 pounds, up to 2000 watts.
With exporting coming soon, solutions built around or including S3 will get more appealing as an offsite backup option. I'd have considered it for some customers, but who wants to back up several hundred gig then have to restore it via T1 line? Heck, it's the kind of thing where you go get a cable modem just for the faster restore speed, because they'll have installers out before you make a significant dent in your data to transfer.
Right now Amazon just has my personal files and wedding photos. Bad to lose, but not fatal.
fencepost
just a little off
I use Perforce version control at home to sync documents and pictures between 4 computers. It's free for up to 2 user accounts and 5 client specs (I only use 1 account). It has worked well for several years and handles small and large files easily.
Note: I'm not affiliated with Perforce but I do use it at work.
I use Subversion through Apache for the documents, and webdav for the "stuff I want to stay on the server" such as my music. I like it because I can access my data from anywhere, and I can secure it with client ssl certificates.
For my email, I use IMAP; for calendars, I use CalDAV; and for contacts, I use OpenLDAP, but that was a pain to set up. I keep my bookmarks synced through Delicious.
If you want to make it easy, use GoogleDocs, GoogleCalendar, Google blah blah blah. Most email clients and calendar clients can interact with Google, and it makes life oh so much easier.
Nfs solves the sync problems fine and unrecoverable server deaths are rare. You should have a backup of the home nfs share. Even better if the backup just mirrors the data and can provide nfs services when the original server goes down.
If you want to go the cheap route have a desktop (out of 1 mac and 2 linux boxes you should have the capabilities) mirror the data if he is on and turn the 'spare' into 24/7 on mode to replace the old one until it is fixed.
Or just have a time machine on mac and provide nfs shares to the linux desktops. Mac shops here are fast and even lend you a spare while they work on your broken hardware.
I use Treecomp with my 16GB and 32GB memory sticks every day, to shuttle work files to & from home (as an academic, work never ends...) Works great, and it's free. It has just the right amount of nagging built-in to avoid any problems with sync'ing the wrong direction.
No - he was trying for 5, but only got -1!
Sounds like my last night out hitting on the ladies :-(
Long time I have been promising myself to try it out and recently I did. Basically it's network filesystem with local cache for offline usage. Setup, while not for Joe Average, is not as hard as it looks on the first sight. Syncing is done in background and this offline mode is exactly what I was looking for. Now I'm far from home but I have a snapshot of the shared directory which will update the server repository when I reconnect next time. If you have public IP address and DNS (for publishing some SRV records), you can even access it from everywhere (by design it's global filesystem).
But it's not perfect:
- Small team, slow development
- Lack of good GUI offering all necessary functionality, especially conflict resolution
- CLI could be more consistent
- Lack of explicit cache management
- Poor handling of big volumes and big files (porn archive is out of question)
Nonetheless, as I stated, I use Coda everyday. Not for entire /home but for dedicated shared directory where I put stuff I know I will need to access from other machines.
P.S. Funny thing but when I was digging the subject about half year ago, I found no alternative offering similar semantics: shared filesystem with offline access capability (yeah, Andrew, but as Coda grew out of it, let's call it same family).
Cheers,
Greg
I use a mix of unison, NFS, autofs, and failover scripts. It works fairly well but there really should be something better. I've considered clustered filesystems but they seem overkill for what I need.
There is no way in hell I'm trusting this stuff to a third party.
I use my OSX laptop as my primary desktop, but I work on more then 20 linux servers. For me it's important to have a good vim and screen configuration. For syncing my conf files I use "roaming homedir" (http://roaming.skumleren.net/) together with git and a personal setup script. I takes me 10 seconds (one command and few yes/no answers) to setup of update a virgin home dir.
so I can write an shosts/rhosts file with + + and voila! You're totally pwned. Suckers.
You could not p0wn your own computer with a full set of passwords, hardware access and a set of screwdrivers. None uses IP based security over the internet anymore.
When it comes to hacking, you bitches be playin' checkers and I'm playin' chess.
You sound like a playground kid running his mouth off without any real understanding of what you are talking about. Don't feel bad about it, everyone goes though that stage. And that rap talk just makes you sound lame.
Do you have enough room for all the user data on one system? For example you could share a folder with the Mac using samba (it is a couple clicks on a Mac to setup) and then mount the directory on the other systems. Everything goes there. Of course you'd want to back up the disks of the Mac periodically. A simple solution would be using Time Machine and an USB drive. All told it takes about 5 minutes to setup something like this on a Mac. Time Machine has a fixed backup schedule though if you want more control you could use another tool, or for example use cron and rsync on a linux box or automator/rsync on the mac.
I don't know about you other people, but I have two different computers for two different purposes. One's strictly gaming + video encoding + other strenuous tasks. If I need to something like this from the other (lightweight) computer, I just RDP into the workstation and do it there. What's the point of having multiple computers if they all do the same thing? :)
The best way I can think of is, buy a NAS and put your homedir on it. They aren't all that expensive any more, and it lets the whole family share things.
The other way, which I use to allow me to work on my projects both from home and the office, is to use a revision control system, like svn, cvs or perforce. Check things in every time you finish working.
Rolled and GPL'ed my own: ;)
http://sourceforge.net/projects/nekjs/
It does exactly what I need it to do
git add .
git commit -a -m sync
git pull
git push
will add any new files to the repository and synchronise with the repository. It works pretty well, except for when a binary file has been modified on 2 computers as then you need to drop to the shell to resolve the merge. But I use it to sync between my desktop, laptop, netbook and server and never had any problems.
It's probably been said lotsa times in this thread but I use rsync to back up stuff to an external drive. I back up /home, /etc, /var/cache/apt/archives and the My Documents folder on the spousal unit's Windows machine.
we see things not as as they are, but as we are.
-- anais nin
Open source, got server/client. Works on Linux, Windows and Mac:
http://www.kablink.org/ifolder
Packaging is on the way for Ubuntu:
https://wiki.ubuntu.com/iFolderPackaging
Myself, I'd solve that issue with a somewhat hefty (~10Mb/s) Internet connection, a local cache and a place in "the Cloud" to store my files. I know of the shortcomings of storing your things in some server online (security, privacy ...), but as many of the other things I use are already there, it's only natural that this is there too.
My weblog in spanish
Btw Robocopy is part of the Windows 2003 Server Resource Kit. :: http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx
:: http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx
Robocopy GUI
and appears to of been supplanted by:
RichCopy
Offline: Tried many solutions, but none ever worked as well as my USB Disk! This is from years of habit though.
Online: Personally, and given the resources, I would just host my own file server from home, for anywhere-anytime-access.
For people who wanna sync two large copies of data (e.g. 100GB) over two remote computers, using a smaller medium (e.g. a USB stick), syncoco maybe useful http://syncoco.sourceforge.net/
It only copy those modified files over.... unfortunately its development seems halted
I take my 4.5 pound laptop into work, xrandr -s 1 and Poof! I have all my files on a large 1920x1080 monitor. I type xrandr -s 0, all my windows move back to the laptop where the files remain. I also have an option for dual displays on the laptop and an external monitor for presentations on a projector. Sneaker net gets my files where they need to go by the time I need them no matter how many gigabytes they consume and how bad local network connectivity is.
Keeping everything important on one backed up (clonezilla, need to rerun) laptop avoids software issues, has single copy update semantics, and exceptional performance (25X faster than NFS at my office, with the tree building in 2 minutes not 50).
Modern laptops are large and fast enough for nearly everything.
I have 2 PC (1 windows + 1 linux) and a distant FTP account. I use JFileSync to sync the 2 PC with samba mounting. There are directories where the windows is the master, whereas other ones are mastered by the linux PC. This sync is also based on the file timestamp. I use FullSync to sync (from windows or linux) home and distant FTP server. This sync is filesize-based, since the timestamp is not kept on the FTP server. They are both open source software.
Novell has a product called iFolder that works well. A few years back I believe they open sourced it and made it free. Looking now this is the link I find to it. http://www.kablink.org/ifolder Might be worth looking at, though you will have to run a server for the backend service.
~Petaris "The world is open. Are you?"
If you file server dies, you put the disk on another computer. If the disk dies, you restore your last backup. There is no solution as convenient as a centralized file server, even if you are using Windows (that can't handle file sharing that well) it is still better to keep things centralized on a server. For that NFS is a great choice at a home network.
Now, there is no solution for keeping our files as safe as making backups. Synchronizing several disks isn't even nearly as good. If you don't want off-site backups, I'd recommend using rdif-backup on a different server at the LAN. Schedule the backup routine at your file server, make it turn the backup machine on and off at start and end of the task, and set the backup machine to send email alerts when it get any SMART error. That is as close as a set-and-forget setting as you can get for backups, once in a while you must look if the backed-up data is on the right place, and that is all.
Rethinking email
when it's in my pocket being transported. It's a years long habit going back to slipping a zip drive in my shirt pocket as I headed out the door for the other location.
I used to carry my email client on it, too. First Eudora and later portable Thunderbird. They actually ran from the flash drive rather than the local HD. Nowadays I use gmail and access it via the web with Thunderbird running on my backup server at home so I have a local backup copy.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
custom script that works well for me: http://pastebin.com/f6239e9c9
For *large files* (like Ubuntu images) I use http://www.krusader.org/ to make backups or copy them somewhere on demand, but for *small, hand-written files* like configs, notes and scripts, I use http://bazaar-vcs.org/.
An example of my workflow would look rather ugly in this comment, so have a look here instead: http://blag.felixhummel.de/junk/slashdot_2009-06-24.html
Two servers, DRBD, & Heartbeat. Or, you know... just keep a backup. Even if your server is unrecoverable, which is very rare, you can restore your home directory to the client from the backup.
Why is it so hot? Where am I going? What am I doing in this handbasket?
I use ubuntu server (in-house) - all you need. Learn linux system administration & unix scripting and your life will be much better (or find a geeky dude who knows how). Heres how I do it: 1) Samba for internal network. Now my mac, linux, and windows are all talking. 2) SSH and FTP for when you are away. 3) apache/mysql/php/perl; I suggest Lampp. 4) Get a domain name and run your own dns updater; I suggest dyndns.org 5) Point your apache config to the directory where all your data is located, and put a php auth and an htaccess auth on it. Now I can access everything through HTTP. 6) Keep everything on mac, and have a back-up copy on the server; linux and windows can just ftp/ssh there happy selves to the data when needed. 7) Unix scripts to back-up my mac up at the end of the day. --Life is beautiful with Ubuntu--
Like Homer Simpson once said: "If something seems hard, it's probably not worth doing". Forget about the automagical sync idea, and version control suddenly becomes attractive.
I keep the important parts of my $HOME in CVS, with the repository available over ssh. I commit my changes when I have something ready, and I update when I suspect there is something to update. You *cannot* automate this -- a human needs to be around to resolve conflicts, in the rare cases where there are any.
One interesting aspect of this is that it simplifies backups. A directory under version control doesn't need backups -- you backup the repository instead. You don't have to spend time excluding certain files/file name patterns from the backup to save space; you have implicitly excluded them by not commiting them.
I recommend AJC Directory Synchronizer.
I can sync two directories in either direction and make incremental backups of the deleted files.
It's great for backing up directories to another hard drive, or USB drive.
See http://www.ajcsoft.com/ProductsAJCDirSync.php
Recently open-sourced iFolder may serve your needs. Does need server-side set-up, but you can do that on any old box you got laying around. http://www.kablink.org/ifolder
Drag-and-drop file back-up, and limited versioning. Works on my Mac, XP, Vista and Linux desktops/laptops.
I use backuppc. anything else is a hassle to explain to my wife.
I know this is late and all but wuala might suite someones needs: http://www.wuala.com/ It's a distributed file system. Pros: - Owned by LaCie - Uses distributed P2P technology to store data fragments (your data is encrypted and distributed). - Ability to "trade storage space" (donate your hdd space and bandwidth for more "distributed storage"). - Multiplatform Cons: - Java application on your desktop :-(
Check out their features: http://www.wuala.com/en/learn/features ...and a google tech talk they gave about their technology: http://www.youtube.com/watch?v=3xKZ4KGkQY8
You don't think enough... therefore you better not be!
SpiderOak https://spideroak.com/ Why? Because it's encrypted and even spideroak can't decrypt your data. That makes all my stuff safe from prying eyes. No one else can do that.
Insert_Ending_Here
I've been using git to track and propagate my various linux/osx home directories. I use it because I can maintain different branches (one for each machine), and propagate changes rather than whole files between them. This is very nice for config files where the content may be very similar between machines, but differ in a few areas here and there. I maintain files on 6 different machines, and have a master public server I use to push and pull from. If I change a file and commit it, and want to propagate the change to all my other machines, I use this script which I hacked up to do it for me:
http://pohl.ececs.uc.edu/~jeremy/propagate-commit.sh
It takes as its argument a commit id, and will propagate the commit to all other branches in your repo. I'm sure it can be modified to work differently and/or be more configurable.
Performance wise, git is fast as hell. And it compresses object very well. Plus you get versioning of course ;)
Too bad Unison is no longer developed.
I've found Allway sync (http://allwaysync.com/) to be a bulletproof multi-directional sync tool, which I regularly use to synchronize 100's of GB of photos, documents and music across laptop, desktop and external drive. Windows only, which will be a problem for some (especially on this forum!), though of course a Windows machine can be used to sync to a file share hosted by another OS. Free for moderate use. Small fee for unlimited "pro" version. One of the few such programs I've ever paid for. Regularly updated with new functionality. I agree with others here that having a single approach for different kinds of data is probably not the right answer. For me, photos, documents and music are my three "buckets" of content, which I sync with different jobs and different frequency. OS and program settings are the other thing I backup manually, typically only when upgrading / migrating machines...