Ask Slashdot: Syncing Files With Remote Server While On the Road?

An anonymous reader writes "Here's a scenario: you are on a vacation trip for a couple of weeks — on the road. Lots of pictures — 2-300 per day. Maybe some text files with short notes etc. You have a camera with Eye-Fi, a PC, and a phone with WiFi and 3G. Files ends up on the PC (mobile storage), phone provides Internet connectivity. Now, if you wanted to upload all files pretty much as you go — given spotty access to Internet over G3 and WiFi — what would be the best way to do that automatically; set-it-and-forget-it style? I would like them to end up on my own server. rsync script? ownCloud? Some BitTorrent setup? Other? I'm thinking of interrupted file transfers due to no network, re-starts etc. And I would not want to lose any files; including scenarios where files gets deleted locally — that should not result in files getting automatically deleted on the server as well. Sure; I could perhaps use something like Dropbox but that would take the fun out of it."

Comments

Fun?

[Anonymous Coward]

You say you don't want to take all the fun out of it, but you're trying to foist this idiocy off on a public forum? Save the fun for yourself, and make a blog post about your solution.

Re:Fun?

[Zaelath]

I'm thinking this guy is a senior manager who would normally ask his IT drones to attempt to solve this problem. Like the one I had in my youth that wanted backup of his laptop to happen automatically for the random 4 hours a day it was connected to the corporate network, without impacting the performance of the laptop by doing anything too heavy like, you know, syncing files across the network.

I'm at a loss as to why people answer these kinda questions, if it was your own family you'd tell them to stop being such a lazy ass and remember to hit dropbox or whatever whenever they have a link.

Re:Fun?

[grcumb]

You say you don't want to take all the fun out of it, but you're trying to foist this idiocy off on a public forum? Save the fun for yourself, and make a blog post about your solution.

Allow me....

Hello from my vacation in Indonesia. There is no automated solution. What you're ignoring is that networks in the developing world are not only patchy, they're flaky too. So, whatever worked for you yesterday might not work tomorrow or even in half an hour's time. Counting on 3G is a bad idea, because of its unreliability, but also because of its cost. Use wi-fi wherever you can. Most hotels these days provide it free of charge. Use rsync (with the zip option if you like), and keep it simple. My update script looks like this:

rsync -av ${SRC}/* ${USERNAME}@${DEST}:${PATH}

Yep, just a single folder in which I dump everything of value and a corresponding folder on my home machine. I just pop open a command line whenever ity's convenient (and possible) and run it. It doesn't always complete in time (the one I'm running as we speak won't be finished before I leave to go scuba diving), but I can always complete the sync later in the day.

Also, bring one or more external disks. Use them for quick and dirty backup while you're on the move. It only takes one rain storm (or fall in a river) to be glad you did. And don't count on buying new SD cards when you're on the road. Most of the ones for sale in the developing world are convincing knock-offs that last about two weeks. That's my experience anyway.

Anyway, simplicity is the single most important step for you when you're backing up data in the developing world. You can't rely on any other factor, so you should at least be able to rely on your own scripts. Which leads to my maxim: "In the absence of robustness, choose simplicity"

Re:Fun?

[thereitis]

I'd recommend the --inplace option so that partially transferred files (due to a network drop, etc.) can be resumed easily. You might also want the --update option so that newer files on the receiving end don't get overwritten (thus you can modify files while on vacation and rsync them back to your home later).

So having us piece something together for you

[Anonymous Coward]

Leaves the fun in it? Either take the suggestions that work, ie Dropbox, or figure it out yourself.

Re:So having us piece something together for you

[icebike]

Dropbox probably isn't going to work.

Yes it will. Perfectly, actually, in my experience.

And it won't delete photos when you delete them from the camera upload directory.

Check it out here: https://www.dropbox.com/help/288

Further, it will chew on each image file till it gets a successful upload.

Re:So having us piece something together for you

[fa2k]

rsync makes no sense to me, because of the requirement that file deletions on the mobile device should not be repeated remotely, which is exactly what rsync normally does.

It doesn't delete files unless you specify --delete

Re:So having us piece something together for you

[Grishnakh]

Calling "the cloud" nonsense is exactly the same as calling the Internet nonsense. "The cloud" is just a stupid marketing term for it, but it's still the Internet. I saw pictures depicting the Internet as a cloud long before "the cloud" came into popular usage.

And calling the Internet nonsense on an Internet forum is the height of hypocrisy.

Let me be the first one to say

[Dot.Com.CEO]

use something like Dropbox. It works fine, does exactly what you want, what's the point in reinventing the wheel?

Re:Let me be the first one to say

[TrekkieGod]

use something like Dropbox. It works fine, does exactly what you want, what's the point in reinventing the wheel?

I think, "that would take the fun out of it" pretty much covers it. Sounds like he's interested in the process of rolling out his own solution and putting it to the text. There's a lot of pride that goes with using something you've built yourself.

Your answer, given that he already says he's aware such solutions exist, is a bit like telling a guy rebuilding a car in his garage to just buy one new, because it'd be simpler and cheaper. Yes, it would, but that's not the point.

Re:Let me be the first one to say

[icebike]

What a useless comment. The original poster is already aware of DropBox, and discarded it from consideration

He discarded it for no valid reason. That is what the GP is pointing out.

It is by far the cheapest most reliable solution to this problem, and it even allows leaving the laptop at home and simply uploading camera pictures to your smartphone where they can automatically be loaded to dropbox.

If the OP is unwilling to consider dropbox, where the solution is handed to him on a silver platter, then why should we waste our time to spec out his system for him?

Re:Let me be the first one to say

[Dot.Com.CEO]

I don't think the guy was actually interested in finding a solution. If he had, he'd have some kind of a proposal set up and would have had us evaluate it, or give him a better suggestion. The whole post sounds like a damn troll to be honest, but let's give him the benefit of the doubt. Using dropbox, or one of the tens of other services (google drive, ubuntu one, microsoft's whatevertheycallit) is pretty much a solution explicitely designed to solve this problem. I have no idea why finding a more difficult way of doing things is better. Sometimes it's fun, sometimes it's just stupid.

git-annex - "The Nomad" use case

[Anonymous Coward]

http://git-annex.branchable.com/

use case: The Nomad

Alice is always on the move, often with her trusty netbook and a small handheld terabyte USB drive, or a smaller USB keydrive. She has a server out there on the net. She stores data, encrypted in the Cloud.

All these things can have different files on them, but Alice no longer has to deal with the tedious process of keeping them manually in sync, or remembering where she put a file. git-annex manages all these data sources as if they were git remotes.

When she has 1 bar on her cell, Alice queues up interesting files on her server for later. At a coffee shop, she has git-annex download them to her USB drive. High in the sky or in a remote cabin, she catches up on podcasts, videos, and games, first letting git-annex copy them from her USB drive to the netbook (this saves battery power).

When she's done, she tells git-annex which to keep and which to remove. They're all removed from her netbook to save space, and Alice knows that next time she syncs up to the net, her changes will be synced back to her server.

Re:git-annex - "The Nomad" use case

[palmer.dabbelt]

I've been using git-annex for this kind of synchronization for a while and have found it works quite well. It supports interrupted file transfers (via rsync), automatically keeps hashes of all your content, and lets you work offline effecively. git-annex is particularly good at synchronizing file renames between two machines: it actually tracks the renames so you won't have to re-upload anything like you would with rsync. It also supports more than one machine, if you want to make backups.

I would definately recommend trying it.

Take fewer pictures

[Tr3vin]

2-300 pictures a day is a lot. I don't know about everybody else, but I actually try to enjoy myself on vacations. I'd rather not consume my time taking pictures every couple of minutes. Once you scale it back a bit, I think you will find that you don't need some complex setup.

Re:Take fewer pictures

[As_I_Please]

Not true. This from an interview with Adams' son:

Ansel Adams frequently made duplicate photographs of his images when taking them. One thing that I tell people constantly is that it is always a good idea to take more than one shot of an image if you can in the camera. According to Michael, Ansel frequently took multiple exposures of the same shots. Many of his negatives are duplicate images of which he'd select the best image to use for printing.

10 Interesting Things I Learned About Ansel Adams

Unison

[Midnight Thunder]

I am a recent convert to Unison. I discovered this because I was trying out Google Drive, which I found worked well, but I don't want to keep more of my data in someone else's cloud if I have to. So far I have been using the Mac OS UI. With a solution with Unison you will need your own server with ssh access to the Internet. The downside is that you have to worry about backups or uptime, though you don't have to worry about some government taking the service offline permanently (or while they spends years trying to establish possible guilt).

The advantages with solutions like Drop Box or Google Drive is that you not have to worry about the server side. Depending on the amount of data you want to store you will have to choose between the basic paid access or the paid access, which gives you more storage.

BitTorrent is probably the worst solution here, since it only works well when the data is massively distributed. If you only ever have one peer, then you are better off with one of the solutions mentioned above.

Three hundred?!

[Bieeanda]

Dude! I think I just heard everyone you know cry out in horror at the merest thought of being invited to your post-holiday slideshow.

Try CrashPlan

[AaronW]

I've been running CrashPlan as an online backup solution for my Linux system to back up all of my photos. It also has a feature allowing you to back up to another PC over the Internet. It's easy to set up so you can back up to your home PC and it's free (unless you buy the cloud backup service). See http://www.crashplan.com/

-Aaron