Slashdot Mirror
Ask Slashdot: Syncing Files With Remote Server While On the Road?
An anonymous reader writes "Here's a scenario: you are on a vacation trip for a couple of weeks — on the road. Lots of pictures — 2-300 per day. Maybe some text files with short notes etc. You have a camera with Eye-Fi, a PC, and a phone with WiFi and 3G. Files ends up on the PC (mobile storage), phone provides Internet connectivity. Now, if you wanted to upload all files pretty much as you go — given spotty access to Internet over G3 and WiFi — what would be the best way to do that automatically; set-it-and-forget-it style? I would like them to end up on my own server. rsync script? ownCloud? Some BitTorrent setup? Other? I'm thinking of interrupted file transfers due to no network, re-starts etc. And I would not want to lose any files; including scenarios where files gets deleted locally — that should not result in files getting automatically deleted on the server as well. Sure; I could perhaps use something like Dropbox but that would take the fun out of it."
You say you don't want to take all the fun out of it, but you're trying to foist this idiocy off on a public forum? Save the fun for yourself, and make a blog post about your solution.
Leaves the fun in it? Either take the suggestions that work, ie Dropbox, or figure it out yourself.
use something like Dropbox. It works fine, does exactly what you want, what's the point in reinventing the wheel?
Mother is the best bet and don't let Satan draw you too fast.
http://git-annex.branchable.com/
use case: The Nomad
Alice is always on the move, often with her trusty netbook and a small handheld terabyte USB drive, or a smaller USB keydrive. She has a server out there on the net. She stores data, encrypted in the Cloud.
All these things can have different files on them, but Alice no longer has to deal with the tedious process of keeping them manually in sync, or remembering where she put a file. git-annex manages all these data sources as if they were git remotes.
When she has 1 bar on her cell, Alice queues up interesting files on her server for later. At a coffee shop, she has git-annex download them to her USB drive. High in the sky or in a remote cabin, she catches up on podcasts, videos, and games, first letting git-annex copy them from her USB drive to the netbook (this saves battery power).
When she's done, she tells git-annex which to keep and which to remove. They're all removed from her netbook to save space, and Alice knows that next time she syncs up to the net, her changes will be synced back to her server.
2-300 pictures a day is a lot. I don't know about everybody else, but I actually try to enjoy myself on vacations. I'd rather not consume my time taking pictures every couple of minutes. Once you scale it back a bit, I think you will find that you don't need some complex setup.
One drawback of BitTorrent is that it is meant for static, large files. RSS integration into bittorrent clients can help, but it's still not a good publishing mechanism. Plus you need quite a few people (or a dedicated seed box) to get it going. For your case it wouldn't be better than pushing to a web server.
The answer also depends on who you want it to view it, and how the access should be. rsync script is probably easiest.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
There is an open source tool for just this kind of situation. It is called Fish-Sync and can be found here: http://fishsync.sourceforge.net/
It basically works like Dropbox or other sync services, but it syncs files between computers you have access to, rather than a third-party server. Fish-Sync is basically a combination of rsync + OpenSSH + Dropbox's LAN sync with an optional pretty graphical interface. (It also works on headless serves if you don't want the GUI.
I am a recent convert to Unison. I discovered this because I was trying out Google Drive, which I found worked well, but I don't want to keep more of my data in someone else's cloud if I have to. So far I have been using the Mac OS UI. With a solution with Unison you will need your own server with ssh access to the Internet. The downside is that you have to worry about backups or uptime, though you don't have to worry about some government taking the service offline permanently (or while they spends years trying to establish possible guilt).
The advantages with solutions like Drop Box or Google Drive is that you not have to worry about the server side. Depending on the amount of data you want to store you will have to choose between the basic paid access or the paid access, which gives you more storage.
BitTorrent is probably the worst solution here, since it only works well when the data is massively distributed. If you only ever have one peer, then you are better off with one of the solutions mentioned above.
Jumpstart the tartan drive.
I suggest you take a look at Unison File Synchronizer, which can be configured as a one way rather than two-way sync. http://www.cis.upenn.edu/~bcpierce/unison/ I've found it to be very good under poor & interrupted communication conditions.
Slashdot: Everything in Moderation, including Moderation itself.
... to ask yourself "Am I a journalist, or a blogger where I need to report every day and send my photos to the mothership???" If the answer is 'no' why go through this elaborate setup? Have some consideration for the people whose open wifi hotspots you'll be leeching off of in order to send hundreds of megabytes over; think of the outrageous charges you'll be incurring for sending that much volume over 3G (and overloading the system for all the other users while you're at it). What's your big rush to send in the photos? Just keep a copy on your laptop, and if you're that paranoid, bring a big enough external harddrive. If you absolutely must, upload to facebook select few photos.
Was that so difficult?
Dude! I think I just heard everyone you know cry out in horror at the merest thought of being invited to your post-holiday slideshow.
1. Call up your phone provider, and shell out $500 or so for a static IP address
2. Hack your phone to run an FTP server.
There,it's done. Your files are not on your own file server, available from anywhere.
or just go use dropbox and stop looking for convoluted solutions.
-- obligatory (but true) caveat: my comments my own, and don't reflect my employer or colleagues' positions.
Can anyone who has used Sparkleshare say if it would work in this situation? I'm looking at building a Sparkleshare server, which is described as an open source version of dropbox, but where you control the server. On some level, it doesn't look that hard to set up, but there are parts of it that still aren't explained well at the website.
But if you search for open source dropbox alternative, Sparkleshare shows up on a lot of lists.
http://sparkleshare.org/
Doctors destroy health, lawyers destroy justice, universities destroy knowledge, religion destroys spirituality
I've been running CrashPlan as an online backup solution for my Linux system to back up all of my photos. It also has a feature allowing you to back up to another PC over the Internet. It's easy to set up so you can back up to your home PC and it's free (unless you buy the cloud backup service). See http://www.crashplan.com/
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I think rsync pretty much provides all you need in one tiny command-line to get data from A to B.
But if you want to increase your resilience against failing network connectivity, and make sure you don't delete anything that hasn't been properly copied to your server, I suggest you take a look at datamover: http://www.cisd.ethz.ch/software/Data_Mover
Essentially, it's a daemon written in Java that monitors an outgoing directory. Everythings that is written in there gets safely copied over to a central storage drive. Behind the scenes, they use rsync to do the copying, but it's wrapped in tons of features that improve the reliability of the moving process, like a quiet period before a file gets moved (good for applications that write their output incrementally and sporadically into files), multiple retries on network time-outs, high-water marks, data transformation (e.g. compression) during the move process, etc. It also is very anal about sending you emails for anything that could possibly be a data integrity problem.
We rely on it to store the raw data from scientific experiments. With the proper configuration, your holiday pictures should be just fine.
Sigh ... this is really, really sad. The original poster is willing to "miss out" on the true vacation by trying to save a digital copy! Am I the only one here thinking ... relax! ... enjoy your loved ones! ... live in the now! What, exactly, is the point of "saving" something that you were never really with 100% in reality?
My experience is that hotel/internet-cafe access is too slow and/or flaky and/or expensive for the purpose you describe. Pay-as-you-go HSPA cell access is very expensive (in Europe and even more-so in North America). I guess if you're only taking 300 small jpegs per day, you might be able to afford the Internet access charges but my experience, even in Europe, is that your best bet is to make your own local backups as you go. My strategy is to travel with a small netbook and a USB drive. Each evening, I offload my SD cards onto both devices and then keep the netbook in the hotel safe or car and the USB drive with my camera. For example, I just returned from two weeks in Tuscany and am currently importing 34GB of photos into a new Lightroom catalog. There's no way that I could have transferred that data over the Internet while on the road without wasting a lot of valuable travel time. Heck, it's taking 20 minutes just to copy the photos off the USB drive at 30MB/s! How much time can you spend drinking espresso waiting for uploads?
I am on holiday now and am currently uploading the day's photos with SugarSync. Its cheaper per byte than DropBox and has more options. I think some people might find the many options confusing but not Slashdotters. eg you can sync multiple folders. When I was in a remote place with a slow connection (Laos) .... the smallest transaction was a file. So it might labor away for an 30 minutes to almost send a 3M file then the connection would drop and resume and it would start again with the same file - ouch.
I found a problem
I'm on the road since more than a year in a round-the-world trip and I think my setup is fit.
Two notes,
- when travelling, internet connection level changes a lot and sometimes you can be not connected for days, so an external disk/memory card rotation should stay your primary backup, especially for photos
- for pictures, I'm taking both JPG+RAW and doing Internet backup only for JPG. If I have a full internet connection at my accomodation and can let transfer during night, if not day, it's working pretty well. RAW (10M per file or more, 5-10 times the JPG) is clearly not possible.
For small files (txt, id/passport copy, ...), using Dropbox as primary (want to stay free), and lsyncd with my home server
For pictures, nothing more powerful than rsync (at least, in one-way; if you make change at both place, could be interesting to check unisson) to my home server or NAS.
rsync is particularly good, as for now, my main changes on pictures are metadata and from what I tested, both lftp and Dropbox would transfer all the files if there is a minor change.
CD/DVD are way too small for JPG+RAW. maybe JPG only. ...
Of course, no pictures transfer if connection is poor or limited (3G or filtered). For China, need to pay for a VPN provider
This is what I use. Folder redirection can be a nightmare when it goes awry, but for the most part SBS keeps my files, documents and photos synced across 4 different machines (two desktops, a laptop and a netbook).
If I need to get at files that haven't yet synced because I forgot to turn on my laptop before walking out the door, I just VPN in to my network (SBS does all the setup and heavy lifting, you basically just turn it on and it works) - and either run a sync, or if I don't have time, I'll just access the files on the redirected folder on my server.
Easy peezy.
Your first idea will do exactly what you want. It'll resume interrupted uploads, verify upload integrity, be secure in whatever context you're in... it's perfect. As long as you don't specify the --delete option, it won't delete remote files just because local files are gone, either.
A crontab line with: "killall rsync && rsync -a ~/local/photo/dir hostname.foo.net:remote/photo/dir" will do the job admirably. Set it to run every 10 minutes or so. You could obviously polish this solution in various ways, but it's quick, it's easy and it will work.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Unless you are comfortable with the idea of items you are storing ending up on the front page of a newspaper the "dropbox" front end to Amazon storage is a bad idea. Those clowns have so many epic failures that plain FTP from 20 years ago is more secure. They still haven't closed the exploit where you can change your password but it doesn't revoke access to anyone you've let in previously. Then there was the day when their authentication failed entirely and they just let anyone that could guess a username on. There were other little exploits based around their deduplication which were used to obtain other people files (making it a bittorrent replacement for popular video files) that could have been put to sinister use but mostly showed that those clowns really were way out of their depth on anything other than an MSDOS box with no network.
There's google drive, spideroak, a really long list of others including rolling your own SFTP that are vastly superior to dropbox. Then there's plain old FTP at any ISP on the planet - still in some ways superior to dropbox despite drawbacks.
Dropbox is an example of marketing hype selling a polished turd.