Laptop/Server Data Synchronization?

gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync. The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"

Comments

rsync

[jshriverWVU]

I do this often and rsync is wonderful for such a task.

Re:rsync

[pixel.jonah]

I'd second rsync.

I'd also take a look at Microsoft's SyncToy if you're on win***s.

Re:rsync

[Roarkk]

rsync is part of the answer. If you're looking for a way to have multiple, incremental backups of laptops with unpredicatable patterns of connecting to the network, BackupPC is the way to go.

BackupPC is a high-performance, enterprise-grade system for backing up Linux and WinXX PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain. Given the ever decreasing cost of disks and raid systems, it is now practical and cost effective to backup a large number of machines onto a server's local disk or network storage. This is what BackupPC does. For some sites, this might be the complete backup solution. For other sites, additional permanent archives could be created by periodically backing up the server to tape. A variety of Open Source systems are available for doing backup to tape. BackupPC is written in Perl and extracts backup data via SMB using Samba, tar over ssh/rsh/nfs, or rsync. It is robust, reliable, well documented and freely available as Open Source on SourceForge.
By using pooling and compression, one client of mine is using BackupPC to backup over 1TB of data distributed among over 100 laptops to a 200GB filesystem on a central server. The network is polled every hour, and any system that hasn't been backed up in the last 24 hours is queued. Beautiful system.

Re:rsync

[frenetic3]

Take a look at Dropbox (http://getdropbox.com/; screencast at http://getdropbox.com/u/2/screencast.html) if you want something that's rsync-like but integrated into Windows and OS X. It's in beta (and full disclosure: I co-founded the company) but was designed precisely because there's nothing out there that does this well and is easy to use.

Re:rsync

[syphax]

Unison is 2-way rsync. But as the poster noted, unison/rsync doesn't easily support automatic synching (that I know of)- you have to kick it off and then deal with any conflicts, etc., manually. I think the poster is looking for ideas of at least automating Unison/rsync (BTW does rsync support 2-way updating, as the poster explicitly mentions?).

As someone who relies on running unison manually (too lazy to figure out how to automate on my Windows box), I'd be interested in relevant solutions.

Re:rsync

[dusty123]

I don't see how rsync solves this problem:

AFAIK, rsync is only one-way, meaning that it overwrites and eventually deletes files. Have a try:

mkdir d1 d2 # Create two directories (e.g. one on server, one on laptop)
touch d1/foo.txt # Create an empty file
rsync -r d1/ d2/ # Sync the directories
echo "123" > d2/foo.txt # Now modify the file on d2 (e.g. laptop)
rsync -r d1/ d2/ # Sync again
cat d2/foo.txt # Ooops - foo.txt is empty!

One possible way I experimented with is the following:

- Integrate a rsync server -> laptop in the startup procedure of the laptop
- Never modify a file on the server while working with the laptop
- Integrate a rsync laptop -> server in the shutdown procedure of the laptop

In theory this works, but practically there are cases where you miss the shutdown/startup sync, e.g. when you have no network at startup (e.g. you took your laptop away from home and forgot to sync it), in case you laptop crashes, the network fails during shutdown and numerous other problems. These lead to dangerous situations, e.g. if the rsync laptop->server fails during shutdown, a startup-rsync may overwrite modified files.

After loosing some of my work, I decided to switch to unison, which is a 2-way sync and lets me decide how to resolve syncing problems.

Nevertheless I'm not entirely happy with the situation - if I forget to sync, I have to resolve things manually, moreover the sync takes quite some time.

In my special case, I have a WLAN connection to my server most of the time, so changes could be written immediately. So I'd favour some kind of network file system that has offline capabilities and can handle two-side modifications in some way. I thought about Coda but it seems to be far too complicated and unreliable and I don't know better alternatives.

So I'm still stuck to my Unison solution, which is somehow cumbersome, but works...

iFolder?

[belly69]

That sounds exactly like what Novell's iFolder is made for:

http://www.ifolder.com/index.php/Home

Windows & Make Available Offline

[goofy183]

I'll likely get buried but here it goes:

In Windows you can mark a folder on a network share as "Available Offline". Windows will copy all of the files to the local HD and if the server isn't available just work with the local copies. When the server is detected Windows will automatically sync the files and pop-up asking the user about conflicts (keep local / keep remote). When connected writes automatically go to both the local copy and the server.

One of the few places that Windows has right and I haven't found a Linux or OS X solution for that is nearly as nice.

Re:Windows & Make Available Offline

[Techman83]

A great solution till it breaks... believe me it does break and when it does be prepared for heartache. There are ways to recover it, but I think it assumes to much and the potential to screw up is a big risk. There were several users at my place of employment that found out this the hard way and now we ban the usage of it. It's not so much finding the best tool, but managing the process overall and how to do that.. Well we are still in the process of developing that one!

In OSX, portable home directories

[shamborfosi]

I have OSX laptops using portable home directories to do exactly what you are asking for.. a network home directory that is automagically sync'd to my laptop (thus making it portable). It works both ways, and I'm definitely happy with it. I'm not sure which OS you're using though. I wrote about how to do it in an article: Full Stack: Portable Home Directory over NFS on OSX authenticated via OpenLDAP on Debian Linux if you're interested. I also just got everything to work over AFP to an OSX server running open directory as well.. but haven't had time to write it up yet (btw, a lot fewer steps).

Coda, AFS, InterMezzo

[RAMMS EIN]

There have been some efforts in the area of networked filesystems with disconnected operations. I remember checking out AFS, Coda, and InterMezzo years ago. At the time, I found something wrong with each of them, but they may have improved since then. Of the three, I think Coda is your best bet.

you already solved your problem

[coaxial]

Simply instally unison or rsync or whatever and have the job kick off with whereami for linux (you'll have to find the main page yourself) or marco polo for macs.

Re:Windows might be good for something

[arivanov]

Offline most likely derives its origins from Coda which was designed to work for 100MB at most. It seems to inherit all of its problems when the data volumes become big. I have had to support an environment where people casually offlined 3-4GB documentation trees and it was falling over on regular basis.

Further to this, offline files has a number of fairly fundamental bugs in the actual implementation. It records both the IP and the name of the server somewhere when doing the offlining. As a result if the name (but not the drive) or the IP changes your entire offline tree goes south and stays offline. You can neither delete it nor reconnect it and the only way of dealing with this is either surgery to the network (aliasing IP addresses) until you reconnect. The only alternative is to rebuild the affected laptops from scratch.

SyncBackSE

[__aalmrb3802]

If you're running Windows, I would recommend SyncBackSE (http://www.2brightsparks.com/syncback/sbse.html), which I expect you should be able to setup to do exactly what you asked.

OpenAFS

[sid77]

As said before there're many choices, each ones with its own pros and cons, so I'll throw this one in: OpenAFS.

As read from the main page:

What is AFS?

AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for federated file sharing and replicated read-only content distribution, providing location independence, scalability, security, and transparent migration capabilities. AFS is available for a broad range of heterogeneous systems including UNIX, Linux, MacOS X, and Microsoft Windows
Hope this helps, ciao

Re:common refrain

[oatworm]

Not that it matters, but since you asked...

Photoshop -> GIMP
Avid -> LIVES - Note: I am not a video editor and have no idea if this program is any good.
Quicken -> GNUCash, among others.

I guess what I'm saying is that, based on your definition of "silly", there's quite a bit of silliness going on in the world today. *grin*

Unison, Rsync & NTP

[Colin Smith]

Unison can be scripted, added to a login script. As can rsync on windows. Alternatively you can add a polling batch file which wakes up every so often and checks to see if the server lives. (Yes, even on Windows)

Rsync can sync in both directions, but you decide one of the sides is the master and sync that one first, in the case of conflicts the master rules. It isn't possible to choose on a file by file basis at sync time as you can with Unison.

Oh, and NTP is absolutely vital when doing any synchronisation.

Basically. Either you do it manually and manage conflicts at sync time, or you do it automatically and define one of the sides as a master in the case of conflict. There's really no way round this, software just isn't sophisticated enough to decide what you're thinking.

The truth is that filesystem syncing isn't ideal for a very dynamically updated file system. It is best used on fairly static filesystems or one way syncing. Documentation, backups and the like.

Re:Windows might be good for something

[dos]

It records both the IP and the name of the server somewhere when doing the offlining. As a result if the name (but not the drive) or the IP changes your entire offline tree goes south and stays offline. Go download csccmd 1.1 from Microsoft.

csccmd /moveshare will take care of this.