Slashdot Mirror
Laptop/Server Data Synchronization?
gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync.
The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"
I do this often and rsync is wonderful for such a task.
How about Subversion? You'll have to write some small scripts that would make it autoconnect, etc., but it could work if you set up SSH keys, etc.
That sounds exactly like what Novell's iFolder is made for:
http://www.ifolder.com/index.php/Home
I use unison. Why would you need to run it manually every time? It can be run in batch mode. I am mostly using it for live backups these days rather than true bidirectional synchronization, so I could really use rsync and some scripts, but I've gotten pretty comfortable with unison.
Man... You are late to the party. People have been struggling with this since the beginning of time (or so it seems). Especially database apps, where they need to work in "detached mode".
I can't give you a flat out solution, because all situations are different. But I can pass on a bit of wisdom. The most important thing for you to do is create business rules for your synchronization. If the data on the server has changed and you made changes offline, who gets priority? You will have three categories of which a file can be... Client changes get priority, Server changes get priority, and Merge files. I would stay away from the last one. If you want to keep things simple, Id go for the "Server changes get priority" approach. In short, if you took an "online" file "offline" and came back, and the server copy has changed since, your offline edits are abandoned. This way, it makes it so heavily edited files have a shorter "check out window" (even if you don't use a checkout system), and forces the person taking the file offline to coordinate with everyone else that may edit this file.
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
From the summary:
"The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop."
So let me get this straight.. You have the old version of the file, somewhere. The new laptop version of file, somewhere. And the new server version of the file, somewhere. And you want the software to decide which to use and copy it to both the server and the laptop?
There are even more issues here, but it kinda sounds like you want some artificial intelligence that you can download.
Aero
Please stop hurting America -- Jon Stewart
You'd have to do a diff between the sever and laptop versions of particular files, and the user would have to choose what edits to save or erase.
Or, you could use a versioning system, making a new point-series branch whenever there was a difference between laptop and server versions. But you would still be left with the problem of choosing which edits to save and which to dismiss.
In the end, you need humans to discuss the changes made and assimilate them into a new master document whenever there is a difference between client and sever versions.
The English word fart is one of the oldest words in the English vocabulary.
I'd say rsync is your best bet.
If you want it to ask only on conflicts, you can write a wrapper script that runs in batch mode, greps the output for conflicts, and pops up the graphical one if there are any.
I hereby place the above post in the public domain.
My Briefcase in Windows 95. It even has a cute ickle briefcase icon.
Somewhat seriously, Offline Files in Win2K/XP is something I've yet to see done well on any other OS.
neuro at well dot com (when I post, it's my opinions, no-one elses)
http://www.coda.cs.cmu.edu/
I'll likely get buried but here it goes:
In Windows you can mark a folder on a network share as "Available Offline". Windows will copy all of the files to the local HD and if the server isn't available just work with the local copies. When the server is detected Windows will automatically sync the files and pop-up asking the user about conflicts (keep local / keep remote). When connected writes automatically go to both the local copy and the server.
One of the few places that Windows has right and I haven't found a Linux or OS X solution for that is nearly as nice.
I've been using Foldershare for several months now to synchronize several folders on three different machines. It has worked well so far and it is free. It's available at: https://www.foldershare.com/
Umm, couldn't you write a hal rule to recognize the device when it's plugged in and then execute rsync?
G++
I work as a unix/vmware consultant for a company in the midwest where every client wants to give me an XP notebook with their special-sauce VPN software installed. I accept the laptop and immediately P2V it using VMware's Converter, then import the VMDK files into Parallels and run the clients system on my MacBook. Lately I've been eyeing an iMac... and you guessed it, how can I run the client's VM on the iMac in the morning at home, and then run to a cafe in the afternoon with my MacBook? The only thing I can think of is bastardizing a DR tool such as DoubleTake. Has anyone tried this? How well does it work?
If you're running Windows on the desktop (I know, I know) you right-click the folder and choose "Make available Offline".
/ learnmore/offlinefiles.mspx
It caches all files locally so if the server connection is lost, you can continue working. When you connect back up, it syncs the versions.
http://www.microsoft.com/windowsxp/using/mobility
There's this great thing called Portable Home Directories. Works great. Does exactly what you want, and all in the background. Helpful, no? :P
I'm not exactly sure what Apple uses under the hood to accomplish it. I don't think it's rsync, because I've fooled with the rsync built into OS X and I get errors frequently, but their home syncing works great.
When you have a mobile user account (i.e. a network account with a local copy of the home folder on the workstation), it will sync every so often (frequency and exactly what is synced/skipped can be configured on the server end, and the user can kick it off manually from the client end). To the best of my knowledge, the sync is bidirectional, so if you log into another machine with a mobile account and modify the server copy, the changes will be reflected on the mobile copy at next sync. It makes my life easier because if a laptop user's machine gets lost, stolen, damaged, or destroyed, we've automatically got a backup copy of the data on it up to the last time it was synced.
In the event of conflicts, the user is presented with a dialog asking which version to keep, including file size and modification date.
Note that I'm not suggesting you throw out your existing hardware and buy Macs to get this feature, but maybe look into exactly how it's done on the Mac and see if you can duplicate it on your systems.
~Philly
I have OSX laptops using portable home directories to do exactly what you are asking for.. a network home directory that is automagically sync'd to my laptop (thus making it portable). It works both ways, and I'm definitely happy with it. I'm not sure which OS you're using though. I wrote about how to do it in an article: Full Stack: Portable Home Directory over NFS on OSX authenticated via OpenLDAP on Debian Linux if you're interested. I also just got everything to work over AFP to an OSX server running open directory as well.. but haven't had time to write it up yet (btw, a lot fewer steps).
I'm a huge fan of Microsofts foldershare (http://foldershare.com). It's free and does a great job for my needs. It's not 100% what you want - the basic idea is that all files are copied to all your computers. When you make a change (on or offline) FolderShare detects the change notification. Once it's able to communicate again with the FolderShare server in the sky it syncs your data with all connected computers. do note that even though there is a server in the sky this server doesnt hold copies of your data, its just there for relay - so that you can get to your files even if you've got firewalls on all sides. my family uses this quite a bit - my Mom has a folder on her desktop named "Chris' Computer" we use this folder as a place to share pictures and other files too large for email. sharing your favorites/my documents is also wicked cool - your laptop, server, and desktop all have the same files -CG
Well their once was a really great program called Groove, but now MS ownes it, but it does exactly what you are looking for... It even has a nice interface. Obviously does not work with Linux without some tweaking but I've had success with CrossOver Office and Groove on FC7.
Have a look at http://coda.cs.cmu.edu/ This is a disconnectable file system. It could be what you are looking for. Certainly, that is what I use for doing the same thing.
What about http://allwaysync.com/? I use it for syncing my laptop to desktop to thumbdrive to other computers on my network... runs on Windoze but eh...cant have everything.
There's no mind reader program (yet) so sorry but you're going to have to make up your mind about how to handle it when the server version changed too.
No, he does not need mind reading voodoo, he needs some good network scripts. Unison prompts the user for conflict resolution. All he needs is something that starts Unison when the laptop hooks up to the right network. There has got to be something that can identifies the right network and asks the user if they want to do a sync.
Friends don't help friends install M$ junk.
Rsync would do this very nicely, except that it requires manual initiation. So what you do is hack up a quick python/perl/etc script to do the following:
1) When it regains connection with the server, run a full rsync
2) When inotify tells the script that a file has changed, rsync that one file. Perhaps buffer changes for set amounts of time so fewer rsyncs need to be run
This should be possible to accomplish in less than a day of hacking in a language of one's choice.
Unfortunately I don't know of any sync programs have a "trigger by network" feature so you'd still need to write some sort of script to initiate the sync.
You want fun, go home and buy a monkey!
Maybe a two-way rsync tool made just for this purpose?
You might have to do A-B, A-C, A-B type syncs for more than 2 paths, unless you stick to a hub/spoke or cascading distribution model.
Not all conflicts are automatically resolved, by default.
http://www.cis.upenn.edu/~bcpierce/unison/
Good luck.
Can Collanos be used for arbitrary files?
Offline Folders on a Windows client connected to a Windows server work reasonably well but sometimes get screwed up.
Novell's iFolder is a very interesting alternative... runs on Linux/Apache/Java stack & only transmits changed blocks over an SSL connection.
Other things worth looking into include Microsoft Groove--let's you synchronize an entire workspace with yourself on other computers or other people - and is relatively network & environment-independent (though Windows only)
NetworkManager combined with a custom script should be able to address running Unison whenever connected to the correct network.
sounds like this is exactly what you're looking for: http://getdropbox.com/u/2/screencast.html we're in private beta, but if you shoot me a message I can get you in ;-).
A few people hit this one pretty well. rsync (and probably rsyncd).
:)
The more complex problem has been thrown at me a few times. What if it's not just one person?
Say you have a repository of data that a dozen people may be working in. When they're all network connected, they're all dealing with the same file pool. When they take their off-line copies with them (unplugged laptops on vacation), they all make changes to the same files. Maybe mine is a one line change. Maybe one guy copy&pasted the first 3 chapters from War And Peace into a comment somewhere in the middle. Maybe another developer did some very intellectual looking changes but hosed some major functionality.
When you start putting machines back on the network, who is right? The 6 guys who did real work are obviously right(ish), but they all made different changes. The very last change will end up being someone's 3 year old kid who was pounding on the keyboard right before daddy shut down the laptop, saving the new changes. Probably the last is the most recent, and right by most methods.
It's not a pretty picture, and requires some intelligence to sort out the mess.
The only "good" resolution I've found is to give logical authority to the changes. Bob is in charge of development. Any changes going into the development or production tree must clear him. He should be able to recognize that the 6 guys made changes, and diff them to come up with the common changes. The 3 chapters of war and peace go by the way side. And the guy with the 3 year old "developer" gets reprimanded.
In the end, a good revision system and good backups are needed too. Something will slip through the cracks, and you'll need to roll back to something you hope is good.
I take control over whatever I'm working on, so if I know I'll be working offline, I'll scp the data to my laptop, work on it on the road, and scp my changes up to the server when I'm done. Anyone else who may have worked in my project space in the duration should have known better.
Serious? Seriousness is well above my pay grade.
But if you are stuck on a windows platform, take a look at Groove. It is one of M$ more recent purchases and included in their office suite. http://office.microsoft.com/en-us/groove/FX1004876 41033.aspx
I would also welcome comments on its security (or lack there of).
Vista & Offline folders does this very very well.
Much better than XP did (or at least more easily)
It is really the only reason I don't reformat my laptop's drive an install XP.
There have been some efforts in the area of networked filesystems with disconnected operations. I remember checking out AFS, Coda, and InterMezzo years ago. At the time, I found something wrong with each of them, but they may have improved since then. Of the three, I think Coda is your best bet.
Please correct me if I got my facts wrong.
The WinXP Synctoy is excellent. Otherwise, you need CVS to allow you to merge changes!
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Not exactly what you're after but worth a mention anyway, I reckon:
PathSync by Cockos (Justin Frankel of Winamp fame's new company).
It has some automation features as well; haven't used them so can't vouch for them.
Simply instally unison or rsync or whatever and have the job kick off with whereami for linux (you'll have to find the main page yourself) or marco polo for macs.
Personally I cant recommend rsync highly enough. The linked article is a basic introduction which explains what makes the utility to uncommonly good.
Nick
I use Vice Versa Pro for 2-way syncing and it works very well. http://tgrmnn.com/
An iDisk set to sync does exactly this. Any work you do locally syncs in the background whenever there is a network connection.
I use FolderShare for this, and have been happy with it. https://www.foldershare.com/
This is a common refrain, and I must say I find it puzzling. What about Photoshop for photo editing? Or Avid for video editing? Or Quicken for accounting? Where are the freeware versions of those applications? MS office applications happen to have free alternatives in OpenOffice and NeoOffice and so on, but only as a result of monumental interest in staging a pushback against the M$ monopoly. Taking this fight to all companies producing software for some amount other than 'free' is just silly.
A-Bomb
No, there's another scenario that causes trouble too (and even more so), if just comparing file systems: A file that exists on one place but not the other does so either because it's new, in which case it should be replicated, or because it's been deleted on one of the two, in which case it should be deleted.
The usual way of handling this is by comparing the timestamps on the directory too, and the one with the newer timestamp wins. This helps prevent new files from being deleted, but it isn't safe. Consider this:
12:00 User creates Important_Document on the laptop.
12:15 Another user deletes Silly_Document on the server, in the same directory.
12:30 Synchronization. The sync software sees two documents on the laptop that aren't present on the server. Since the timestamp on the directory on the server is newer than on the laptop, both documents will be deleted from the laptop.
Bzzzt!
This is why a workable synchronization system must keep its own database of changes, and not just rely on comparing the file systems.
Regards,
--
*Art
If you're running Windows, I would recommend SyncBackSE (http://www.2brightsparks.com/syncback/sbse.html), which I expect you should be able to setup to do exactly what you asked.
As read from the main page: What is AFS?
AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for federated file sharing and replicated read-only content distribution, providing location independence, scalability, security, and transparent migration capabilities. AFS is available for a broad range of heterogeneous systems including UNIX, Linux, MacOS X, and Microsoft Windows
Hope this helps, ciao
This is simple. Require your users to operate online for any data that needs to be shared. Internet access is literally everywhere, and accessible via wireless phones (as a modem), WiFi, hard-wired, internet cafes, etc. If your users are working for your company, setup an RDP or Citrix server and make it a business rule that requires them to operate online. Or, use Sharepoint (which is web accessible from anywhere)
Benefits aren't limited to erasing sysadmin headaches, they also include absolutely no conflicts, no synchronization traffic, instant availability of changed data, instant on-line backups, the speed of SQL (for example) on a real server instead of 2005 Express on someone's laptop with a 5400 RPM drive. You also have all kinds of better security for the data as well compared to someone's laptop getting stolen.
I think notebooks should be used for files you and you alone are going to be working with. If it's ever going to be shared or have the possibility of conflict, and especially with databases, make remote access to it mandatory.
First of all, I couldn't agree more that DRBD is not at all suited for the task.
:)
But I thought I'd point out a slightly newer source of information about it. drbd.org is the home of DRBD. You might be able to pick out the age of the linked howto by the mention of support only being for the 2.2 kernel.
At least not in the manner the OP requires. If you're the only person ever editing these files, then there are lots of software solutions that do this just fine, that other people have already suggested, including the creaking Briefcase from Windows 95.
But if you want to allow for the possibility that other users in the office have changed those very same files, which you had "offline" over the weekend, then forget about it.
For a genuinely multi-user, genuinely collaborative approach to a problem like this, you need to shift to a completely different document management paradigm - for example: using a web-based document or spreadsheet application where multi-user editing and online availability has been built-in from the start (Google?). Of course this only works for specific types of data, not all data, and only when you're connected (for now).
WinRSync could do the job if you're on Windows. http://www.hill30.com/home/WebSite/Applications/Wi nRSync/tabid/157/Default.aspx
In great *nix tradition, you should solve the problem by combinding many single purpose tools.
Use something like http://0pointer.de/lennart/projects/ifplugd/ to launch a script that runs Unison file sync when ever you are connected to the network.
...and that is all I have to say about that.
http://jessta.id.au
If you run an OpenSolaris distribution on the laptop (the server can be any NFSv3 compliant server) then CacheFS will do exactly this for you.
You can even prime the cache using cachefspack initially. It works in disconnected and connected mode and is automatic.
An alternative tool on OpenSolaris distributions is filesync, it uses the same config syntax as cachefspack does but works by simply using
cp/mv/rm/chmod/chown and doesn't include its own transport layer (so needs to use NFS or similar).
I downloaded a little program a few years ago called Syncback, I find it does what I need between windows computers. Does the synchronisation thing too. http://www.download.com/SyncBack/3000-2242_4-10548 273.html?tag=lst-0-1
At least on more recent versions of Linux. I've been looking for a slightly different solution to filesystem synchronization (for webservers using iSCSI over ethernet to cache to local disk). Rsync would be a solution with a kernel that supported inotify, but we are using 4u5 systems so I'm left assessing expensive replication solutions or upgrading the whole system to RHEL5 with a tight deadline.
Quack, quack.
it works fine even on different OSs......I believe you can make bi-directional too....
/d"/>
my version is just few lines and it has email notification when it has finished:
script to run it:
set ANT_OPTS="-Xms256m -Xmx256m" This setting is to handle many files
ant -v -logfile log.txt
build.xlm:
<project name="backup" default="start" basedir="/">
<description>
Script to backup my files
</description>
<property name="backupFrom" value="BackupSystem@mydomain.com"/>
<property name="backupAdmin" value="Me@mydomain.com"/>
<property name="BackupDestDir" value="y:"/>
<property name="NetworkDestDir" value="\\10.10.1.167\Backup"/>
<target name="init">
<tstamp/>
<echo message="Backup started: ${TODAY} @ ${TSTAMP}"/>
</target>
<target name="copy" depends="init" description="copy the files" >
<echo message="Removing ${BackupDestDir} drive....."/>
<exec executable="net" failonerror="false" failifexecutionfails="false" >
<arg line="use ${BackupDestDir}
</exec>
<echo message="connecting ${BackupDestDir} drive....."/>
<exec executable="net" failonerror="true">
<arg line="use ${BackupDestDir} ${NetworkDestDir}"/>
</exec>
<!-- All my stuff -->
<copy todir="${BackupDestDir}/SyncDir" verbose="true" failonerror="false" preservelastmodified="true">
<fileset dir="/Documents and Settings/User" >
</fileset>
</copy>
</target>
<target name="start" description="backs up my files" >
<mail mailhost="mail.mydomain.com" mailport="25" from="${backupFrom}"
tolist="me@mydomain.com" subject="backup's log - ${TODAY} ${TSTAMP}"
charset="ISO-8859-1">
<fileset dir="/Documents and Settings/User/Backupscript">
<include name="**/log.txt"/>
</fileset>
</mail>
<echo message="Backup finished: ${TODAY} @ ${TSTAMP}"/>
</target>
</project>
A sync might take some considerable time (even rsync is a few minutes with lots of files). What if the user needs to interrupt it.
You probably should make this manual (at least via a confirmation dialog), so that if the user is only connecting for a few seconds, it does not try to sync. Also, the user can then control priority - he may want to get to the web first, rather than waiting while an Office service pack downloads...
http://www.beinsync.com/ :( OSX and Linux versions are somewhere in the works.
Does 2 way syncing, with versions and conflict management, offline/online modes and web access , AND includes group syncing as well.
Currently windows only
(full disclosure; one of my old companies, but we did create this software for this exact problem for us )
I use http://www.powerfolder.com/ for syncing about 30 gb of audio between myself and my collaborators. Its worked great so far, can be run without an internet connection - LAN only for instance (or via Hamachi). Has worked great for me.
Try this: http://www.mobiliti.com/ * File synchronization and backup solution (includes outlook backup and synchronization solutions) for Laptop and desktop users. * Virtual network (offline) access to network over slow or unavailable networks. You can travel with your network. * Faster Synchronization using band width optimization techniques like file level differencing. * Central Mobiliti Profile control using Deployment and management solution.
Unison can be scripted, added to a login script. As can rsync on windows. Alternatively you can add a polling batch file which wakes up every so often and checks to see if the server lives. (Yes, even on Windows)
Rsync can sync in both directions, but you decide one of the sides is the master and sync that one first, in the case of conflicts the master rules. It isn't possible to choose on a file by file basis at sync time as you can with Unison.
Oh, and NTP is absolutely vital when doing any synchronisation.
Basically. Either you do it manually and manage conflicts at sync time, or you do it automatically and define one of the sides as a master in the case of conflict. There's really no way round this, software just isn't sophisticated enough to decide what you're thinking.
The truth is that filesystem syncing isn't ideal for a very dynamically updated file system. It is best used on fairly static filesystems or one way syncing. Documentation, backups and the like.
Deleted
DRBD is for high-availability clusters with sub-minute failover and synchronous replication. It is not suitable for server-to-desktop file replication, let alone two-way. The DRBD link in the post is horribly outdated; current information about DRBD is available from http://www.linbit.com/. Or from my blog, for that matter. :-)
rsync has some painful deficiencies when it comes to lots of files being synced on a frequent basis. Csync2 (http://oss.linbit.com/csync2) addresses some of those; it may be more useful for the usage scenario described.
Disclosure: I work for LINBIT, the company behind DRBD. Csync2 was written and is being maintained by a former employee of ours.
It says BackupPC is *nix only. Can it be used with Cygwin on Windows?
There seems to be a project for Gnome called Conduit that sort of does what you want. I think. I haven't tried it. .haeger
It does however look like a very ambitious project aimed at syncing both files and other data between both computers and other services.
You are not entitled to your opinion. You are entitled to your informed opinion. -- Harlan Ellison
I agree—subversion isn't really the right tool for syncing data in two places. Having said that, if you want to keep revision history and sync it, then the latest subversion might be right up your alley. Included is a svnsync tool that is intended to sync a "live" repo (one that you use regularly for commits) with a backup repo (exists solely as a copy).
As far as automating backup, assuming you're talking about linux, I would say that the best approach is to simply write a short script that pings for the presence of your server. If found, it mounts the remote drive, rsyncs, and unmounts. If the transfer is interrupted, no worries—it'll resume next time you connect. Assuming the rsync succeeds and exits normally, you can have a clean umount, otherwise you'll have to umount forcefully.
but have you considered the following argument: shut up.
I use two vista PC's connected on a network ones a Laptop the other a desktop. The laptop is setup to sync with the desktop through the sync centre, so whenever I log into the network with it it makes sure that the chosen directory contains the most recent files in both folders. works great for me
anyone tried tsyncd?
I have used unison on both linux and mac os x, and have been very happy - even with a mixed OS environment (ie. mac laptop, linux server). Currently I use unison, initiated by a script which runs everytime the mac wakes (using the command line utility sleepwatcher on mac os) - the script first checks to see if it has run today (I only want a daily sync) and then pings the server (I only want unison to run when I'm at home on my network) - if all passes, it then syncs.
Finally, it scp'ies a copy of the unison session log across to the server, so I can check that both my and my wife's laptops are synchronising without errors. (The whole process happens in the background on my wife's computer, so she's unaffected by it until she needs the server copy should disaster strike!)
Unison also has the ability to keep backups of 'old' versions. Simply periodically run 'find' on the backup directory with appropriate '-mtime +90' (ie. 90 day old files) and '-delete' flags to delete backed-up versions of older files. Ah, a poor-man's version control...
Anyway, the mac os mix of it-just-works-GUI and command-line-fu is sweeeet. Unison: highly recommended. Custom bash-script: priceless.
djol
I've been trying to set up my Linux desktop with Windows-alike roaming profiles for ages now.
:D
I have the kerberos single sign on for a multitude of services. I have the LDAP user metadata. I have NFSv4 file shares secured with kerberos, and autofs homes that enable me to use the same app settings on each machine. I have a groupware server (Kolab) serving all of my users (all two of them, ha!). But can I find any way of accessing my home drive (reliably) on my laptop when it's not plugged into the network?
If I ask on any forums, I'm either told to just use NFS homes (which of course doesn't work when your laptop might be a) on the other side of the planet and b) not plugged into the internet). So I need a way of syncing my server:/home/user with laptop:/home/user bi-directionally. Since there doesn't seem to be any easy way of doing it built into any Linux distro, I created a bunch of shellscripts (basically just a bunch of rsync commands that don't sync certain directories, e.g. ~/.maildir) that are location aware (i.e. they can tell if I'm on my local LAN or not), and sync to/from the file server at startup, and then just to the fileserver at shutdown (or of course I can inititate them manually). It's not perfect though, and I've had to restore some items that have been clobbered from backup (granted, windows often does the same thing with roaming profiles). It also means keeping a fairly disciplined file hierarchy - otherwise you can get broken symlinks galore, or config files that are repeatedly overwritten.
However, I believe the Linux desktop is crying out for a simple and automated way of doing it (perhaps a modified version of unison that, instead of querying the user for resolving conflicts just dumps conflicting files into a seperate directory?) - for most deployments, shared NFS homes are perfect but for laptop users outside of the network they're useless.
I've tried Coda and AFS and, apart from being hard to set up and containing (to me) severe limitations that doesn't really suit them for online/offline use, they seem to be fairly unstable. Perhaps I suck at adminning Coda and AFS, but I personally think using a different, and rather complex, filesystem for roaming homes is unnecessary and I wasn't really keen to put too much time into it.
Disclaimer: I'm not attempting to replicate MS' roaming profiles, because I think it can be done better - I just don't see any way of doing it other than shell script hackery at the moment, and I definitely need a way of doing something akin to roaming profiles. Should have filed my own Ask Slashdot ages ago
Moderation Total: -1 Troll, +3 Goat
I use CVS to synchronize between several working places (home desktop, work desktop, notebook). It's not 100% transparent, but as you'll always have to push a "sync up/down!" button a "cvs commit/up" is as close as you can go.
I guess any other version management tool can be used too, I just tend to know and like CVS.
- Hubert
As previous posters have said, this is a tricky problem to solve. iFolder probably handles it the best out of most of the products I've tried. You can create folders easily, the background task is very lightweight to transmit changes, and best of all is the SSL web interface to the folders, which was simple, polished, and easy to use. It's multi platform too. When Novell released the code, I figured there would be significant development of it because this was such a common problem, especially as Novell has put a small group of developers onto the project as well. But that never materialized. Community support was non-existent and Novell retasked the iFolder team and development hit a wall. It's sad because of how well laid out it was. The backend was pretty scary in terms of what it ran on, but v3.6 was trying to fix that. I've been running v3.4 to sync around a dozen different folders across probably 25 users, with granular permissions. It's been great, but I really was hoping 3.6 would see the light of day... SVN shows some churn, but not a whole lot. Looks mostly like bug fixes, but the build system is completely broken - builds used to post daily.
Top Most Bizarre/Disturbing Error Messages
This is a classic version control/configuration management scenario.
Look at Subversion. It's good enough for most people. Easy to use clients and servers for Linux/unix/mac/windows.
Deleted
Subversion is for stupid and ugly people, just ask Linus. http://developers.slashdot.org/article.pl?sid=07/0 6/03/004214
http://www.youtube.com/watch?v=4XpnKHJAok8
If you must over think this problem for non-computer literate users, at least use GIT.
Personally, I use the built-in `copy` that 4NT provides with an update switch. http://www.jpsoft.com/4ntdes.htm I suppose xcopy or xxcopy are also options.
Easiest way? Just create a RAID1 array between the laptop's internal HDD and an identical-sized file mounted as an NFS share. (You'll need the relevant modules compiled hard into the kernel, or in your initrd.) If the share isn't found at bootup, the array will run degraded (i.e. with just the HDD). If the share is detected at bootup, the array will be re-syncronised in the background. Metadata on the internal HDD will indicate that it was correctly updated, so you needn't worry about causing an unintentional rollback!
Je fume. Tu fumes. Nous fûmes!
HomeSync works great - although if you're not on OpenDirectory you have to play some hackery to use it. Just a word of warning though - if your server blows up that hosted your network home directory, and you rebuild DO NOT HOMESYNC.
:( I thought "oh, it'll just copy my home back off of my laptop..."
It will think the server copy is newer, and happily blow away the home directory on your laptop!
NOPE.
Anyhoo, absolutely love it. Paired with OpenVPN, my laptop syncs my home directory every 15 minutes so long as I'm connected to the internet or the local network. If you're not afraid of xml files, it can be hacked to sync absolutely everything, otherwise by default it skips ~/Library (as there are some things in there that do not make sense to sync across the network), thus for exampe, Firefox bookmarks don't get synced by default, nor do your extensions. I've modified it to go ahead with those. I'm hoping to move my home directory out of NFS into AFS soon, but that's been slow going. So far those LDAP+Kerberos (definitely worth it!)+HomeSync rocks. Add AFS in and I can ditch HomeSync.
Karma: Chameleon (mostly due to the fact that you come and go).
mobile home directory between os x server and client automagically syncs the machine for you, in the background
dreemkill.
Rsync and many of the others require to be installed on both the server and the client. However, if you mount the remote folder with either NFS or CIFS on boot you can use any program of your choice (even your own script) and it only has to be installed locally. Next you could write a small script to check if the mount is active (check for the existence of any files/folders in the local /mnt location perhaps?) and if so, invoke the program of choice. (I like Unison or ViceVersa under Crossover.)
Lastly set up cron to run the script on boot and then again every X minutes afterwards. Make a shortcut to your script on the desktop and click it before shutdown (if anything has been saved/changed since the last cron increment).
There are other ways I can think of that are easier to implement, but they get horrendously complex from a user point of view.
I've personally been using SyncBack for years with tremendous success, both as a general use and as a data manager. The basic features are usually enough, however the expert mode really shows the usefulness of the utility. http://www.2brightsparks.com/syncback/
OS X uses Apple's MirrorAgent. It works well and exists exclusively on the client. OS X Server is *not* a requirement to make this work. I'm running a FreeBSD server at home to handle this. OpenLDAP and NFS get the job done. My Macs don't care that the server isn't OS X.
How about using redirection? I'm not sure if it is the same as offline folders, but you can redirect the folder to a place on the network. Here's microsoft's tactical explanation
w indows/xp/all/proddocs/en-us/gpx_redirecthome.mspx
http://www.microsoft.com/resources/documentation/
Even if you are not running active directory, on a windows pc, you can run gpedit.msc and enable redirection on an individual basis.
Although this document discusses using the "home directory" you can redirect to any network share and path.
use a mac and set up a mobile account - it handles everything neatly for you . . .
I use Robocopy on Windows to backup my stuff on USB drive. It has an option to run on background. Robocopy synchronizes continuously. I run it as a scheduled task with separate user account (the tasks is set up to start on boot). It continues to work even if I take off my USB drive and attach it back later. It takes about 2 MB of memory and doesn't need much cpu. Robocopy does not have versioning capabilities. On *nixes I prefer rsync or rdiff-backup.
..on linux boxes, and I know there is a win32 version as well. Its command line driven, and uses a database hash list on both sides to determine what has changed. It has conflict resolution capabilities and plenty of options.
It works very much like Lotus Notes replication, but for file systems.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
subject says it all.
You don't think enough... therefore you better not be!
unison -batch performs excellent automated synchronization. incron can be used to run unison -batch whenever a file is modified. Together they'll do the job.
With disconnected operation it isn't possible to transmit locks on a file or bits of file between machines so simultaneous edits will not work unless you have a versioning system like CVS or Subversion which can show you the differences between the files and allow you to merge them manually. And that generally only works for plain text files.
Rsync, Unison and this Dropbox tool are sync tools. No locking and no versioning. Simultaneous edits on a file are a very bad idea in that environment.
For disconnected operation try something like Subversion, though that's not much use if it's something like a Word document. Alternatively, if you want to use a sync tool, avoid editing the same document on different machines.
Deleted
After you've set up your share on the server, map a drive to it in My Computer. Now right-click My Documents and go to properties. Click the move button, point it to the mapped drive, and click Ok. You'll be prompted whether or not you want to move existing files to the new location, say yes.
Once the move is finished, you're ready to set up synchronization. Right-click on My Documents again and click Synchronize. In the lower right hand corner of the window you'll see a button marked Setup, click that. Now check the boxes next to "When I log on to my computer" and "When I log off of my computer" and click ok.
I do this at work and it has worked flawlessly for me thus far. Good luck.
Hi:
;-)
Our shop has a large scale client/server system with about 525 RDBMS tables in the backend. The data is by nature complex, and the table structures are as simple as the data allows. We were asked by senior mgt to provide portability for the system, on laptops used by field staff. Since the RDBMS vendor touted their DB synchronization product and their portable DB server capabilities, we thought it would be, well, not simple, but not really difficult in concept.
Wrong!
The OS on the laptop changed, rendering the ability to run version X of the DB Lite impossible, which impacted other cross-dependencies... A total nightmare!
Then, once all those dependencies were resolved, the number of tables needed for field staff (between 125 and 150) proved totally beyond the ability of the DB Sync tools provided by the DB vendor. After many many help tickets showed us that the vendor (think large yacht) had no real intention or ability to fix their sync tool set, we decided to take another tack.
We assigned a young coder with a degree but not too much experience, and made him a full time sync tool coder with the senior DBA (also a crack coder) and they've been working on writing a tool to sync laptop data with server data. Now about 98% complete, they tell me.
We're using flags to control which data set is current, and the logs will allow us to use a utility to work out conflicts that the custom sync tool has problems updating.
Wish us luck, as I wish everyone confronted by this problem the best of luck! I don't think commercial or open source tools are flexible enough to deal with anything more complex than the most simple set of information and a very static environment. Build your own tool, no one else understands you requirements well enough to help you, really.
Just my $.02 worth of hard experience on synchronizing complex data for field operations.
JR
Think of the Irony!
I would writ eup a small bash script to look for the specific IP of your server, say every 15 minutes. If the server IP (private address) is there, perform an rsync, if not, end program.
Done
Subversion/TortoiseSVN
It sounds like you are doing multiple edits on files on a filesystem from multiple locations simultaneously. None of the sync tools are designed for that, including Unison. Take a look at configuration management tools like Subversion, CVS, ClearCase, Perforce. Then there's git which I've never played with.
For simplicity and cross platform support with integration into just about anything: Subversion. Add Tortoise for ease of Windows users use.
Conflict resolution depends on the data. Anything more than plain text and you may have to look at a commercial system, something like Visual SourceSafe.
Deleted
Have you tried fsync? Perhaps it could be adapted?o w_to_synchronize_your_files_between.html
I do know it is very fast when there are little or no changes.
http://www.associatedcontent.com/article/298977/h
This is meant for WAFS, but if money is no object its about $2500/license and does most of the things it should.
http://www.availl.com/
Only transfers deltas, compression, SSL, version history, etc etc...
Groove is great for this if you are on Windows. In fact you can have as many "nodes" for replication as you like. I use this all the time for syncronising data between my Laptop, Desktop & Home PC. 60 trial and the trial continues to work (without the ability to create new workspaces) after it expires.(See: http://office.microsoft.com/en-us/groove/FX1004876 41033.aspx)
Unison works great for me. I use it to synchronize filesystems for two servers in remote locations over an OpenVPN connection. I also use it to sync my laptop up with my desktop (and then use rsync to backup the desktop copy to a remote server). It works well and can be automated although making the automation work securely can be a hassle.
For the two remote servers I have unison running every half hour via a cron script kicking off unison on server A to an instance of unison listening via sockets on server B. This works well until something kills the connection during a unison session and the unison listening on the socket dies and has to be restarted either manually or by a watchdog. For the laptop/desktop systems I kick off unison manually since I don't need to sync up often.
If you went the unison socket route on the server it would be fairly easy to set up unison on the laptop to run automatically via a script. The script could trigger when the laptop is able to ping the server do a test rsync for a specific file and if it passes follow up with unison).
The best part about unison is that in addition to doing two-way syncs is that it works with both Windows and Unix systems for those rare times that I need to work on a Windows box.
rsync is great for just this purpose, but the frequency with which this question comes up makes me wonder why so few people use terminal services for working on the road. It's efficient, very functional, and you never have to worry about resynching files, because they're all stored in the same place.
Menus: Linux=function, Windows=vendor, OS X=as little as possible. Makes a statement, don't you think?
I have a computer with multiple partitions and operating systems, 1 is windows xp the rest are *nix. XP has been patched so it has ext2 read/write support although it ignores *nix permissions. I want to sync my bookmarks between all the partitions so that if I add bm on one partition the change is imported to the other OSs. But I only want to sync positive changes (adding bookmarks, not removing them). Is this doable without using a spare partition for data?
There are 11 types of people, those who know unary and those who don't.
I use this religiously to sync between my desktop and laptop - could this work for your scenario? It is free (for now). All you need to do is install the client on any machines that need to sync, then set up your directories. If both machines are online, changes (i.e. changed documents, added or removed) made to one synced directory propagate automatically, but if one is offline it stores the changes until the machine goes back online. It even has a handy web interface where you could access your synced files if you are not on one of your synced machines (like a public kiosk).
rsync has a major failure. If you delete a file on your laptop, that file is placed back on the FS at the next sync. Sometimes this is nice. Sometimes it drives you crazy.
synchronex does a nice job of seeing that a file was deleted on one FS and will delete it on the sync'ing file system as well.
Generally it has a terrific set of options and control parameters as well.
Only use the DNS naming specification for the share when you connect, i.e. \\foobar.example.com\sharename ... it will use CIFS and not use netbios at all.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
It will become a nightmare sooner or later. The real solution is to simply connect remotely to the server and work off of it, keeping your data in one place.
You don't want a sync tool: You want an online bookmark manager. Head to sitebar.org. You can keep your bookmarks on their server, or run your own server if you prefer.
(just a mostly happy user)
I'm not a Windows network administrator, but I play one on TV. One way we used to do this is to use Windows Active Directory with domain logons, map a share for their profile (e.g., map H:\ to \\server\username), and then do profile redirection. This basically set up their my documents and desktop to be offline files and folders. They'd connect to network in the morning, and sync. While they were connected to the network, the syncs propagated to the windows server just fine. At the end of the day, they logged off, and it synced again. When they were out of the office, they could log into their laptop and do work "offline", then when they came in later, they'd sync up. The alternative was to set up a VPN connection back to the windows server (Hamachi worked fine), and map the drive to *always* connect. This meant that they were always technically "online", though it would make for slower performance so wasn't really recommended.
Try PowerFolder (sf.net page)
It has active development.
I dont know why you say Unison needs to be started manually. It has a batch mode that can certainly be used, and it even has a "-repeat NNN" option where it repeats endlessly, pausing every NNN seconds in between.
[Actually the "-repeat" option can do more, but for your purposes this should do pretty well...]
If you are in a Windows Environment check out ViceVersa Pro.
I use SyncBack on my XP laptop to keep backups and on occasion to synchronize shared data between more than one machine. Usually it works well and isn't too hard to set up.
http://www.2brightsparks.com/syncback/syncback-hub .html
Incidently, BackupPC was written by the now Atheros CEO.
I used powerfolder successfully in the past.
http://www.powerfolder.com/