Slashdot Mirror
Synchronize Data Between Linux, OS X, and Windows?
aaaaaaargh! writes "I'm using a laptop with Ubuntu 8.04 for work, a netbook with Ubuntu 9.10 when I'm outside, Mac OS X 10.5 for hobby projects, and Windows XP for gaming. For backups, I'm currently using Jungle Disk and Apple's Time Machine, and I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another. Options range from putting everything online (but Jungle Disk disks seem to be too slow for anything else than backup), storing my data on external media like USB sticks or SD cards, or working with copies by synchronizing folders over the network. I don't want to give my data away to some server outside without strong encryption (controlled by me, including the source code) and external media like USB sticks are a bit too fragile according to my taste. The solution should be reliable, relatively failsafe, as simple as possible, and allow me to continue to use Jungle Disk for backup. So what would you recommend?"
Set up one computer as a server and rsync/ssh to it with either a cron job or at your whim.
I like Unison for this sort of thing.
www.dropbox.com
RTFL. Jungle Disk has that covered. Throwing money included.
USB sticks are a bit too fragile
damn straight. that is the number one problem with USB anything. i've seen more broken jump drives, and more broken usb ports from someone tripping over usb cable, than i care to fix. yes, they ARE handy as can be, but to WHOMEVER is designing USB 4 or whatever it will be called, PLEASE make the damn connection more sturdy.
https://www.dropbox.com/ will give you 2gb of free space. It'll keep all files in your "dropbox folder" synchronized on all computers where it is installed. It works on Linux, Mac, and Windows. A video on installing Dropbox on Linux from The Linux Journal's Shawn Powers is here: http://www.linuxjournal.com/video/dropbox-linux
Three options that I use, or have used, are rsync, Chronosync and Jungle Disk.
Jungle Disk is the best solution if you have more money than time and can rely on being online a reasonably good percentage of the time. In the Jungle Disk settings, you can specify a certain portion of your disk to be used as an offline cache. Jungle Disk will then keep downloaded files in that disk cache, so you don't have to worry about the download speed so much, assuming you have the disk space available for the cache. If you don't have the disk space you'll need an online solution anyway.
Chronosync is Mac-based, but you can set it up to sync your files with your Windows and Linux computers. I recently bought it and haven't been through all its features yet, but I'm pretty sure it will do everything you need it to for about $40.
Rsync is of course great if you have the time and expertise to set it up. If you want to take the time to learn how it works, it's probably the best solution. OTOH I think Chronosync will do 99% of what rsync will do (from what I've seen) and is easier so I felt like the purchase price of Chronosync was worth it to me.
www.clarke.ca
End of discussion.
Unison from UPenn http://www.cis.upenn.edu/~bcpierce/unison/
Works on all the platforms you mentioned... It can synchronize 2 disparate directory trees (you made updates to files A, B, C on one system and D, E, F on another system and want to merge them) and when it can't figure out what to do it asks you.
Set up one computer as a master repository and do everything over SVN+SSH.
Its what I do for the complete working set I have, passing between 3+ systems, is everything is through subversion over SSH to a backed-up system.
Test your net with Netalyzr
Mercurial for...
Windows
MacOS
For linux it should be in standard repos.
This way you can make changes on any of your systems and move them around as required, merging only when needed.
http://michaelsmith.id.au
external media like USB sticks are a bit too fragile according to my taste
That's because you're using the wrong external media. You're going for new and exciting and not old and reliable. What you need to do is convert all of your data to human-readable binary (you know, 1100110001111 and so on), and chisel it on to large stone tablets. These are extremely resistant to wear, especially if stored with the written side down, and are virtually theft proof (who's going to steal a bunch of 500-pound rocks?).
You can also easily transfer data from one machine to another either via forklift or just by bringing the machines to the tablets themselves. Backup to Jungle Disk is simple: store a second set of tablets in the Amazon (you can even have a DR site in southeast Asia!), and hire local villagers to update them for you. You should provide a telegraph machine to the villagers to send data back and forth for synchronization purposes. If the telegraph is too unreliable, you can use an elaborate network of smoke signal or semaphore (the thing with the flags) specialists to send data back and forth.
Encryption may be a little more difficult, but easily solved if you happen to have a couple of old Enigma machines sitting around (and really, who doesn't?). If the Enigma machine isn't reliable enough, Little Orphan Annie decoder pins can be used instead.
HTH.
I had my account suspended for this , bluehost is only for hosting files for websites
Half of writing history is hiding the truth.
Been using it for awhile, works great windows, Linux and Mac. Also you don't have to place files in the dropbox folder, you can choose what folders to sync and backup. The also have zero knowledge of your files, which I find reassuring.
Half of writing history is hiding the truth.
I;m a little confused if you are syncronizing, mirroring or backing up data. the title says synch, but time machine and jungle disk are more for backup, and at most can be used for mirroring a master rather than synching.
If you want to backup, take a look at Crashplan. It's got two unique features the other plans lack. 1) you can backup to your own physical media, not theirs. This solves the problem of how do you backup and restore say 200GB of data in less than a month to a remote service. if the disk is yours you do the initial backup locally, them move the disk to the remote location for incremental backups. And then you do the reverse when you need to do more than in incremental restore. 2) they will sell you just the software-- a one time cost-- and you don't have to pay for a monthly remote service cost unless you want to. in which case you just backup to a freinds computer that is hosting your disk. Your data is both safe and remote (and encypted) but you can also go get the drive is if you need it using your toyota rather than the internet.
On the other hand if you want syncronization then look into Unison. It appears the source forge project is not highly maintained but also mature enough to work well. it is cross platform and scriptable.
Some drink at the fountain of knowledge. Others just gargle.
https://www.dropbox.com/
"Free for Windows, Mac, and Linux"
Creates a folder that is kept in sync between different computers. You can share files with other dropbox users too via shared folders. 2GB or so of space is free. It also keeps multiple versions of files so you can go back to a previous version of a file if you need to.
I don't have any affiliation with the company, other than being a satisfied user.
Specialist Mac support for creative pros, Melbourne
Syncing data is a hard problem. It doesn't look like it could be a problem at first, but there is only a surprisingly small number of products which come close to solving it. Different meta data across platforms (time resolution and time zone, different types of time records, permissions, naming etc.), extended file structures (streams, resource forks, ...), comparatively slow and intermittent network connectivity, concurrent changes in more than one place, huge binary files with only small changes, renamed or moved files, open file handles, the list goes on and on. Finding a sync tool which satisfies your requirements is just as hard as finding a reliable backup solution: They are similar problems and there are heaps of tools for both, but only very few which are even worth considering. The rest are half-assed marketing-driven data-eating annoyances. (BTW, I use rsync, but it has deficiencies. For example, it does not deal with open files.)
I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another.
Am I missing something here? What doesn't SVN do that you need? Clients exist for all named platforms and it handles OO, Latex, and source code files very well.
Ask Slashdot: Where bad ideas meet poor googling skills.
On a somewhat related note, my wife recently gave up on backing up to a different medium all together and now just builds backup into her work flow. She is a photographer and takes 50,000+ digital photos a year (in raw TIFF format). Each file is ~70MB so she requires several terabytes of storage. Loosing these images is unacceptable (ie: would result in a financial loss) and ripping to DVD for backup is simply impossible (would require a full time job in and of itself).
So she builds backup into her work flow such that at any one point in time she has at least one copy of the image spread across multiple drives. In the work flow, photos go from raw TIFF -> PSD pre-proccess -> PSD final -> JPEG for online viewing and she uses different drives for each stage in the process. That way, if a drive goes out, she only looses time, not data. For data retention requirements (~2-3 years), she just has ~10 TB of storage and rotates files from older jobs out onto other drives (spreading across 3 - drives as per the work flow).
Of course if something catastrophic were to happen to cause all of the drives to die (since they are all in the same location) then files would be lost, but that's what insurance is for.
Faith is a willingness to accept something w/o complete proof and to act on it. Reason allows you to correct that faith.
That problem is mentioned all over the net, what's not mentioned is that the problem stems from transfers across different OSs.
A suggestion that might be a work-around : Sync Windows to a smb share, and use unison locally on that directory.
I am not using windows, so I'm fine. But, I also can't guarantee that my suggestion works.
I have done what was described: rsyncing a TrueCrypt volume file that changes over time. Yes, there is an "expansion factor", but I would peg it at 2 - 5x the size of the diff within the encrypted volume. To restate: this scheme is entirely tenable with TrueCrypt volumes and rsync. A good binary diff capability is required, but that is the raison d'etre of rsync. I have never used Dropbox, so I have no idea how their deltas work for binary files.
I was originally of the same opinion as you. I anticipated some sort of avalanche effect from small changes in the ciphertext. However, this isn't the case for several reasons -- first, you can read about TrueCrypt's crypto modes of operation and how the blocks work. Second, you can reason that TrueCrypt would be completely unusable if writing a single byte near the beginning of an encrypted hard drive/volume caused a chain reaction that would require the entire partition to be re-encrypted due to cipher block chaining cascading to the whole drive.
I don't claim to be a cryptography expert (perhaps someone can further enlighten me), but it seems that CBC would break down in a disk encryption scheme if there weren't a new IV rather frequently (for the same reason as I mentioned above). Of course, TrueCrypt uses XTS mode, so it seems that I am just idly musing at this point and should click Submit.
Time Capsule works well.
Time capsules appear to have a MTBF of ~18 mos; the power supply dies. Apple will replace it via an APP-covered computer that uses it, but not if you crack the case to get the hard drive out.
So, either eat the $$ after 18 mos to save your 18 mos of backups, or give Apple your data for a refurbed unit, or find a backup solution for your backup solution.
I'm not buyin' another one.
They are way beyond dicks and would go to unbelievable lengths not to provide you with the service you bought. They are proactively scanning your account and will shut you down (without ANY warning whatsoever) for any media file that look suspicious (that includes mp3's you recorded yourself, as in you're the artist).
At first when they started cracking down on people using disk space they just shut down everybody that was using "too much" in their opinion (they were specifically offering 1500 GBytes space and 15000 GB/month transfer at the time - now they are "unlimited"). No questions, nothing. When customers asked they said they modified the TOS and say something like "you're not allowed to store files (!)". They also modified the TOS to say they don't have to notify you when they modify the TOS (which they had accordingly to the old TOS). And they didn't notify anybody, just shut down accounts randomly.
If you use their whois privacy feature (default on and unmanageable via web interface) they will spam your whois with advertising. They also don't seem to get what this feature actually means and insist that you own the domain which can't be true in any legal and meaningful way (yes, you are the owner in a field in their database but legally and as far as ICANN is concerned they are the owner).
The last drop in the bucket is the hidden 50000 files limit. The limit is absolutely secret and you can find about it only when you are kicked out or warned that you are above. However they would still insist the space is unlimited but the number of files is limited. And that it is for your own protection because fsck would take far too long if you have too many files. And that if you don't like it you should go away.
The idea here is only to pay, not to use what you paid for.
To put things into perspective a default installation of gallery with default plugins will eat about half of those 50000 files you are allowed. And use probably a couple tens of megs of space (keep in mind they were offering 1500GB before going unlimited which should mean for any reasonable person more than 1500GB). Oh, and people hosting linux distributions are abusers as well.
Of course they would work more flexibly than a random free provider (you get shell access but you need to fax your ID!). But between random crashes and this "anybody using is abusing" policy you won't get too far.
Stay away from bluehost.
I looked into this problem myself, and it isn't pretty. I use a Windows box for everyday stuff, a Linux based server, and my Sister uses a Mac.
Basically I had the idea that I wanted to share some Anime videos with my Sister who also enjoys the TV shows. Anyway originally my idea was to buy an external USB HD, copy the files and then give her the HD, this way she also gets a new external HD out of the deal. However when I looked into it, not as simple as I thought. While most external drives are compatible with Mac or PC, the emphasis is on "OR". Usually they are formatted differently and are not compatible with the other once formatted.
The only other solutions I could find were to use a NAS (Networked Attached Storage), and this would work, because you would be using the network protocol to translate the data between OS, SMB I think it is called. Anyway at the time, they were very expensive and still are to a degree however they have come down in price more recently and I have seen solutions out there for 200-300$ range. Which is still more than I wanted to spend as an external HD only runs you about 100-200$.
I settled for the getto version and just spent the time and burned like 10 8GB DVD-R's as it was easiest. However for a continual solution, this would not work. I would say your only choice is some sort of NAS with SMB. I am sure given the time and know how you could also build your own linux box to do the same thing, but for price it likely depends if you have an extra kicking around someplace. Either could likely be set up for remote FTP. I would recommend just buying a NAS, likely simpler that way.
If you're willing to pay, you could use Dropbox. If you keep it under 2 GB, it's free. Anything you drag into your dropbox gets synced to their servers and then synced back down to your other PCs you have linked to your dropbox account.
If you would prefer a roll your own solution, and are willing to build a server, then go look at Novell's iFolder http://www.ifolder.com./
Andy