Synchronize Data Between Linux, OS X, and Windows?

aaaaaaargh! writes "I'm using a laptop with Ubuntu 8.04 for work, a netbook with Ubuntu 9.10 when I'm outside, Mac OS X 10.5 for hobby projects, and Windows XP for gaming. For backups, I'm currently using Jungle Disk and Apple's Time Machine, and I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another. Options range from putting everything online (but Jungle Disk disks seem to be too slow for anything else than backup), storing my data on external media like USB sticks or SD cards, or working with copies by synchronizing folders over the network. I don't want to give my data away to some server outside without strong encryption (controlled by me, including the source code) and external media like USB sticks are a bit too fragile according to my taste. The solution should be reliable, relatively failsafe, as simple as possible, and allow me to continue to use Jungle Disk for backup. So what would you recommend?"

Comments

Rsync?

[Misanthrope]

Set up one computer as a server and rsync/ssh to it with either a cron job or at your whim.

Re:Rsync?

[nweaver]

Don't worry about the encryption overhead, just make sure AES is the first cypher chosen. AES is VERY fast.

Re:Rsync?

[frodo from middle ea]

Better yet, Setup ZFS on a opensolaris box, export the ZFS volume as NFS/Samba share or as an iSCSI target, access from Linux,Solaris,Windows etc.
ZFS can also do compression, and automatic snapshots, not to mention rock solid data integrity with raidz2

• data integrity - check
• remote access - check
• automated incremental backups - check
• No limitations of software raid neither the need for expensive hardware raid - check
• Dead Simple to Use - check

Seriously what more do you want from a storage/backup solution ?

Re:Rsync?

[Anonymous Coward]

Better yet, Setup ZFS on a opensolaris box, export the ZFS volume as NFS/Samba share or as an iSCSI target, access from Linux,Solaris,Windows etc.

Note that newer releases of OpenSolaris have built-in kernel support for CIFS with ZFS.

  # zfs set sharesmb=on mypool/mydata
  # zfs set sharenfs=on mypool/mydata

Mapping between Unix UIDs and Windows SIDs is done transparently, and you can change things from the Windows GUI if desired and the ACLs will go live on the Unix side as well: SIDs are first class security objects. None of the usual Samba UID/SID kludginess (though (Open)Solaris comes with a Samba package if you want to go that route.)

Case-(in)sensitivity is also a user toggle away if so desired.

De-dupe was also recently committed to the code base. Crypto is being worked on and I think patches are available.

Re:Rsync?

[Anonymous Coward]

Except it didn't exactly answer his question. He appears to favor local storage. Ha already has a backup solution and if he wanted a snetwork share, I'm pretty sure anyone running the 3 major OSes can figure out how to share a drive via NFS/AFP/SMB.

For a home user, ZFS's big achilles is the inability to change raidz geometry. Most home users add another drive to increase capacity, *not* another raidz of drives into a pool. Nor do they typically want to replace all their raidz drives with larger drives. ZFS fails on that particular common home storage scenario. Everyone knows it and it's been talked about to death. And btrfs is looking better every day.

The answer he's looking for is probably some form of rsync or unison solution. Depends on the complexity of his syncing. And if he needs/wants some form of version control. He knows how to use SVN so he's probably not needing version control.

I'm curious too; I have to keep local assets for CGI apps in sync on a few different systems and rsync is the best I've come up with. You can't drop extra files into the asset directory structure and you can't rely on their state to tell you anything about the need to sync.

Unison

[samuraiz]

I like Unison for this sort of thing.

Re:Unison

[nine-times]

I second the suggestion. It's basically like rsync except it's more like real synchronization. What I mean is, AFAIK with rsync you basically have to say, "Sync folder A and folder B by assuming that folder A is the copy I want to keep, and changing only folder B." With Unison, on the other hand, you can say, "Sync folders A and B by keeping track of when files get modified, and keeping the most recent version of each file. If a file has been modified in both folders since my last sync, let me know and ask me what to do."

One minor annoyance is the project page links to an OSX version that includes a Cocoa interface, and that's kind of a pain if you want to script it to run in the background. Even if you call it from the command line, the GUI will still open up. Also, if you install it from MacPorts, it doesn't do a good job of maintaining resource forks. Of course, if you're syncing to Linux or Windows, you might not care about resource forks.

I think the best place to get the OSX version is here: link

It's been a while since I've downloaded it, though.

Re:Unison

[McPint]

I cannot say whether unison is the best solution, but it is one that I have been using for 7 years or more, between two desktops (home and work), a laptop an external hard drive and another offsite backup. These were on a variety of filesystems between Linux and Windows flavours. MacOS X should be no different.

While I have seen graphical interfaces to unison, the command line interface is so much easier to use.

I would definitely recommend unison... while it is not a revision control system it is an effective solution at working on the same set of files wherever you are provided you synchronise twice a day.

Re:Unison

[coaxial]

I agree, unison is great. I used unison for several years to sync machines. It worked great for linux-linux, and seemed to mostly work for linux-mac/mac-linux syncs as well. The main problem was with resource forks causing a bunch of ._foo files, but there's not too much you can do about that if you're copying data from HFS+ to something else. It's been a while since I used unison (I gave up my desktop, and now only use a laptop.), but seems like unison has the ability to actually merge files by fire off something like meld to resolve conflicts. That's a big win.

dropbox?

[kperrier]

www.dropbox.com

Dropbox

[rtobyr]

https://www.dropbox.com/ will give you 2gb of free space. It'll keep all files in your "dropbox folder" synchronized on all computers where it is installed. It works on Linux, Mac, and Windows. A video on installing Dropbox on Linux from The Linux Journal's Shawn Powers is here: http://www.linuxjournal.com/video/dropbox-linux

Chronosync, rsync, Jungle Disk

[aclarke]

Three options that I use, or have used, are rsync, Chronosync and Jungle Disk.

Jungle Disk is the best solution if you have more money than time and can rely on being online a reasonably good percentage of the time. In the Jungle Disk settings, you can specify a certain portion of your disk to be used as an offline cache. Jungle Disk will then keep downloaded files in that disk cache, so you don't have to worry about the download speed so much, assuming you have the disk space available for the cache. If you don't have the disk space you'll need an online solution anyway.

Chronosync is Mac-based, but you can set it up to sync your files with your Windows and Linux computers. I recently bought it and haven't been through all its features yet, but I'm pretty sure it will do everything you need it to for about $40.

Rsync is of course great if you have the time and expertise to set it up. If you want to take the time to learn how it works, it's probably the best solution. OTOH I think Chronosync will do 99% of what rsync will do (from what I've seen) and is easier so I felt like the purchase price of Chronosync was worth it to me.

Distributed version control

[MichaelSmith]

Mercurial for...

Windows

MacOS

For linux it should be in standard repos.

This way you can make changes on any of your systems and move them around as required, merging only when needed.

Proven technology

[eln]

external media like USB sticks are a bit too fragile according to my taste

That's because you're using the wrong external media. You're going for new and exciting and not old and reliable. What you need to do is convert all of your data to human-readable binary (you know, 1100110001111 and so on), and chisel it on to large stone tablets. These are extremely resistant to wear, especially if stored with the written side down, and are virtually theft proof (who's going to steal a bunch of 500-pound rocks?).

You can also easily transfer data from one machine to another either via forklift or just by bringing the machines to the tablets themselves. Backup to Jungle Disk is simple: store a second set of tablets in the Amazon (you can even have a DR site in southeast Asia!), and hire local villagers to update them for you. You should provide a telegraph machine to the villagers to send data back and forth for synchronization purposes. If the telegraph is too unreliable, you can use an elaborate network of smoke signal or semaphore (the thing with the flags) specialists to send data back and forth.

Encryption may be a little more difficult, but easily solved if you happen to have a couple of old Enigma machines sitting around (and really, who doesn't?). If the Enigma machine isn't reliable enough, Little Orphan Annie decoder pins can be used instead.

HTH.

Or dont' throw money

[goombah99]

I;m a little confused if you are syncronizing, mirroring or backing up data. the title says synch, but time machine and jungle disk are more for backup, and at most can be used for mirroring a master rather than synching.

If you want to backup, take a look at Crashplan. It's got two unique features the other plans lack. 1) you can backup to your own physical media, not theirs. This solves the problem of how do you backup and restore say 200GB of data in less than a month to a remote service. if the disk is yours you do the initial backup locally, them move the disk to the remote location for incremental backups. And then you do the reverse when you need to do more than in incremental restore. 2) they will sell you just the software-- a one time cost-- and you don't have to pay for a monthly remote service cost unless you want to. in which case you just backup to a freinds computer that is hosting your disk. Your data is both safe and remote (and encypted) but you can also go get the drive is if you need it using your toyota rather than the internet.

On the other hand if you want syncronization then look into Unison. It appears the source forge project is not highly maintained but also mature enough to work well. it is cross platform and scriptable.

Answers his own question.

[Rantastic]

I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another.

Am I missing something here? What doesn't SVN do that you need? Clients exist for all named platforms and it handles OO, Latex, and source code files very well.