Slashdot Mirror
Ask Slashdot: Syncing Files With Remote Server While On the Road?
An anonymous reader writes "Here's a scenario: you are on a vacation trip for a couple of weeks — on the road. Lots of pictures — 2-300 per day. Maybe some text files with short notes etc. You have a camera with Eye-Fi, a PC, and a phone with WiFi and 3G. Files ends up on the PC (mobile storage), phone provides Internet connectivity. Now, if you wanted to upload all files pretty much as you go — given spotty access to Internet over G3 and WiFi — what would be the best way to do that automatically; set-it-and-forget-it style? I would like them to end up on my own server. rsync script? ownCloud? Some BitTorrent setup? Other? I'm thinking of interrupted file transfers due to no network, re-starts etc. And I would not want to lose any files; including scenarios where files gets deleted locally — that should not result in files getting automatically deleted on the server as well. Sure; I could perhaps use something like Dropbox but that would take the fun out of it."
You say you don't want to take all the fun out of it, but you're trying to foist this idiocy off on a public forum? Save the fun for yourself, and make a blog post about your solution.
Leaves the fun in it? Either take the suggestions that work, ie Dropbox, or figure it out yourself.
use something like Dropbox. It works fine, does exactly what you want, what's the point in reinventing the wheel?
Mother is the best bet and don't let Satan draw you too fast.
http://git-annex.branchable.com/
use case: The Nomad
Alice is always on the move, often with her trusty netbook and a small handheld terabyte USB drive, or a smaller USB keydrive. She has a server out there on the net. She stores data, encrypted in the Cloud.
All these things can have different files on them, but Alice no longer has to deal with the tedious process of keeping them manually in sync, or remembering where she put a file. git-annex manages all these data sources as if they were git remotes.
When she has 1 bar on her cell, Alice queues up interesting files on her server for later. At a coffee shop, she has git-annex download them to her USB drive. High in the sky or in a remote cabin, she catches up on podcasts, videos, and games, first letting git-annex copy them from her USB drive to the netbook (this saves battery power).
When she's done, she tells git-annex which to keep and which to remove. They're all removed from her netbook to save space, and Alice knows that next time she syncs up to the net, her changes will be synced back to her server.
...a "cloud" service for more expensive options offering less control if you don't.
This is the answer to so many questions.
2-300 pictures a day is a lot. I don't know about everybody else, but I actually try to enjoy myself on vacations. I'd rather not consume my time taking pictures every couple of minutes. Once you scale it back a bit, I think you will find that you don't need some complex setup.
One drawback of BitTorrent is that it is meant for static, large files. RSS integration into bittorrent clients can help, but it's still not a good publishing mechanism. Plus you need quite a few people (or a dedicated seed box) to get it going. For your case it wouldn't be better than pushing to a web server.
The answer also depends on who you want it to view it, and how the access should be. rsync script is probably easiest.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
Unless you are rich, forget the 3G connection, especially as an automatic upload. One, you can go over your data plan limit and get whacked, two, you can be in the wrong area and be roaming and really get whacked! Stick to wi-fi only and only manually delete files after you verify they have been saved on your server, or why even delete them till you get home, backups are a good thing!
Dropbox alternative, all you have to have is git and ssh, cross-platform and open source.
http://sparkleshare.org/
I'm not sure but you maybe want to make this a challenge for yourself? I would personally go for the easiest route which you just set up and takes care of itself without complex problems. Dropbox (if you have enough storage) is the ideal answer as it will sync away in the background so freeing you to do things for yourself. Certainly the last dropbox update seemed to ask me if I want dropbox when I plug in a camera rather than using iphoto.
However I suggest getting a good NAS and my suggestion is a Synology Diskstation of some type (no financial interest, just very satisfied customer). You have your own server without the power overheads. Plus you can set it up for remote access and they have even released their "cloudstation" solution which is like having your own personal dropbox syncing, so would satisfy having pictures on your own server. Would go to http://www.synology.com/ and check it out. I'm sure you could set it up to backup things if you accidentally deleted locally. By the by if you're travelling abroad please do not data roam, it will beexpensive and very regrettable. Either switch off data roaming or get a local sim.
There is an open source tool for just this kind of situation. It is called Fish-Sync and can be found here: http://fishsync.sourceforge.net/
It basically works like Dropbox or other sync services, but it syncs files between computers you have access to, rather than a third-party server. Fish-Sync is basically a combination of rsync + OpenSSH + Dropbox's LAN sync with an optional pretty graphical interface. (It also works on headless serves if you don't want the GUI.
Here is what I would do...
Set up dropbox on the laptop
Copy your files to your dropbox directory
On your server, rsync the dropbox dir to another file store.
Done.
you could mail it
I am a recent convert to Unison. I discovered this because I was trying out Google Drive, which I found worked well, but I don't want to keep more of my data in someone else's cloud if I have to. So far I have been using the Mac OS UI. With a solution with Unison you will need your own server with ssh access to the Internet. The downside is that you have to worry about backups or uptime, though you don't have to worry about some government taking the service offline permanently (or while they spends years trying to establish possible guilt).
The advantages with solutions like Drop Box or Google Drive is that you not have to worry about the server side. Depending on the amount of data you want to store you will have to choose between the basic paid access or the paid access, which gives you more storage.
BitTorrent is probably the worst solution here, since it only works well when the data is massively distributed. If you only ever have one peer, then you are better off with one of the solutions mentioned above.
Jumpstart the tartan drive.
I suggest you take a look at Unison File Synchronizer, which can be configured as a one way rather than two-way sync. http://www.cis.upenn.edu/~bcpierce/unison/ I've found it to be very good under poor & interrupted communication conditions.
Slashdot: Everything in Moderation, including Moderation itself.
You mentioned you have a eye-fi card already. You're describing the eye-fi premium upgrade:
http://www.eye.fi/how-it-works/eyefiview#premium
"Eye-Fi Premium: no limits.
With Eye-Fi Premium, photos & videos sent directly from your camera to your Eye-Fi View are available for as long as you like. Enjoy unlimited storage and the flexibility to access, share and download your media in full resolution anytime. Whether you’re at home on a second computer, on your iPhone on the go or on an iPad on vacation, your photo & video history is always just a few clicks away. Get unlimited access with Eye-Fi Premium for only $4.99/month or $49.99/year. Buy Eye-Fi Premium (Monthly) or save $10 with Eye-Fi Premium (Annual)"
To get the pics on your server, install the eye-fi app, which you already did to use the card, and turn on the computer, it'll then sync.
In summary:
1. Upgrade here: http://www.eye.fi/how-it-works/eyefiview#premium
2. Turn on your computer.
rsync is actually made this this. Just add a launchd, (cron job for linux users,) running every 5 minutes or so, which checks to make sure itself is not already running, then syncs the whole directory one way, (to the remote server,) and is set to NEVER remove files, only add and modify them.
... to ask yourself "Am I a journalist, or a blogger where I need to report every day and send my photos to the mothership???" If the answer is 'no' why go through this elaborate setup? Have some consideration for the people whose open wifi hotspots you'll be leeching off of in order to send hundreds of megabytes over; think of the outrageous charges you'll be incurring for sending that much volume over 3G (and overloading the system for all the other users while you're at it). What's your big rush to send in the photos? Just keep a copy on your laptop, and if you're that paranoid, bring a big enough external harddrive. If you absolutely must, upload to facebook select few photos.
Was that so difficult?
Dude! I think I just heard everyone you know cry out in horror at the merest thought of being invited to your post-holiday slideshow.
Why would you even consider trying to do this? You know up front that your data connectivity will be poor, and that you'll have a lot of files to upload. Save yourself the frustration. Bring along an external hard drive and make backups as necessary.
1. Call up your phone provider, and shell out $500 or so for a static IP address
2. Hack your phone to run an FTP server.
There,it's done. Your files are not on your own file server, available from anywhere.
or just go use dropbox and stop looking for convoluted solutions.
-- obligatory (but true) caveat: my comments my own, and don't reflect my employer or colleagues' positions.
You're on vacation. Just get a couple 16 gig SD cards (or whatever your camera takes). Unless you are shooting in RAW you are not going to fill those cards. Then just dump them to the laptop you have with you. Worry about transferring everything when you get back. You are on vacation, enjoy it.
For $150 bucks you can easily get an external 2-3 terabyte USB hard drive. Dump everything to that and deal with it when you get back home. Unless there's some reason why you want/need those photos and files at home before you get there...
Don't come here and dismiss the obvious solution, (Dropbox) for which which many newer phones support automatic uploading of both phone pictures and photos transferred to the phone from an external camera. (And deletes are not mirrored). You can transfer picture so the phone via Samba server on your phone (at least with Android you can) or bluetooth or cable. From then on you forget it. It will take care of it. It will re-try till it succeeds without you having to do anything.
And when you get home, Dropbox will have all your pictures on your desktop machine waiting for you.
There is also Picasa if you transfer to your pictures either from computer or your Android phone. Flicker, Photobucket, and about 20 other such services are also available. Anything you lash up with your home computer is bound to be flaky, more trouble than its worth, and bound to fail the minute you drive away from the house.
If you insist on rolling your own, and If you carry a computer and transfer all photos to that, you can of course use any number of sync solutions that work in "contribution" mode (local deletes are not replicated on the server), I've used Unison in this mode, or SyncCenter (windows) SyncToy (windows), rsync, etc.
But all of these require leaving a computer on at home, hoping it won't reboot due to any number of reasons, and assuming you can get wifi access from anywhere.
There is little point in building your own these days.
Sig Battery depleted. Reverting to safe mode.
One word: Crashplan.
Here is a new site that launched that offers what you need. https://portal.bitcasa.com/invited/586bcbc373f34afbbb9ea73ad7d413ef/ I think you would need a computer to first upload the files but it syncs it between all the devices.
It will continue when your connection is lost.
The issue might not be so much how, but rather how much.
If they are high-quality images, 2-300 can be pretty large.
Also the time you will be on the road and where you go to will make perhaps going for just one option less interesting.
e.g. if you go on a trip around the world for a year, then other options should be included as well. e.g. making a copy on blueray and fedexing it once a week. Some countries do not have affordable connections to send home that amount of data.
Don't fight for your country, if your country does not fight for you.
simple
Can anyone who has used Sparkleshare say if it would work in this situation? I'm looking at building a Sparkleshare server, which is described as an open source version of dropbox, but where you control the server. On some level, it doesn't look that hard to set up, but there are parts of it that still aren't explained well at the website.
But if you search for open source dropbox alternative, Sparkleshare shows up on a lot of lists.
http://sparkleshare.org/
Doctors destroy health, lawyers destroy justice, universities destroy knowledge, religion destroys spirituality
some hotel / free wifi block ports some times the 3g is nated.
Don't count on having all the ports that you do have at home.
The primary question is: why are you trying to do this? Is it to make sure you have an off-site backup in case all of your electronics gear gets stolen? Redundancy can be best covered with extra hard drives.
Another consideration is what kind of photos you are taking. If you're shooting RAW with a modern DSLR, you're going to have images of 20-30MB each. At 300 pictures per day, you could be looking at a data footprint of up to 9GB per day. I don't know what kind of coverage or data plan you have, but in my opinion that's a lot of upload data for a mobile connection. In this case, you may want to consider batch processing the images to a lower resolution before uploading, just to have some record of the images online.
Even if you're on the right track, you'll get run over if you just sit there. - Will Rogers
If you have the phone already with internet connection, move/take your photos to/with the smart phone, create Ubuntu One account for free with 5GB of space,install Ubuntu One app on phone and let it autoupload/sync to the server.
Ah, thanks!
I've been running CrashPlan as an online backup solution for my Linux system to back up all of my photos. It also has a feature allowing you to back up to another PC over the Internet. It's easy to set up so you can back up to your home PC and it's free (unless you buy the cloud backup service). See http://www.crashplan.com/
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
I think rsync pretty much provides all you need in one tiny command-line to get data from A to B.
But if you want to increase your resilience against failing network connectivity, and make sure you don't delete anything that hasn't been properly copied to your server, I suggest you take a look at datamover: http://www.cisd.ethz.ch/software/Data_Mover
Essentially, it's a daemon written in Java that monitors an outgoing directory. Everythings that is written in there gets safely copied over to a central storage drive. Behind the scenes, they use rsync to do the copying, but it's wrapped in tons of features that improve the reliability of the moving process, like a quiet period before a file gets moved (good for applications that write their output incrementally and sporadically into files), multiple retries on network time-outs, high-water marks, data transformation (e.g. compression) during the move process, etc. It also is very anal about sending you emails for anything that could possibly be a data integrity problem.
We rely on it to store the raw data from scientific experiments. With the proper configuration, your holiday pictures should be just fine.
Sigh ... this is really, really sad. The original poster is willing to "miss out" on the true vacation by trying to save a digital copy! Am I the only one here thinking ... relax! ... enjoy your loved ones! ... live in the now! What, exactly, is the point of "saving" something that you were never really with 100% in reality?
that's what i was thinking. then for a 'backup', copy the pictures to extra blank cards or usb stick and ship home (or to office, parents, whatever).
there is no need to upload up to 20 gigs (or more.. an estimate: our own camera would be about 20 gigs for 4000 photos) of photos over the two weeks over the internet... none at all: would take forever and a day on mobile (if your plan could even survive that volume without insane surcharges); you'd be lucky to get necessary bandwidth at hotels (every one we've been at has had horrendously slow net access) to do it in a reasonable time; and you probably won't ever be at any one free hotspot (restaurant, coffee shop, truck stop, etc) long enough to put a dent in your transfer queue (free hotspots are not normally known for their blazing speed either.. most are on cable or dsl which usually means asymmetrical speeds, much slower upstream than down. here, they're about 512k up at best, which would take about 24 hours to upload 5 gigs)...
just because there are geeky ways to do things (which often become more expensive and/or complicated than when first dreamt up), doesn't mean they are necessarily the best ways.. sometimes, simple is better.
Is there a circle of hell reserved for distracted drivers who kill/injure others?
(Maybe in Nivens Inferno rather than Dante's
If you are taking 300 pictures a day, every day, you're not having "fun" anyway. Put the fucking camera away, dude.
Seven puppies were harmed during the making of this post.
Each time you want to "upload" more files, burn them onto a cheap CD (they're so cheap they're practically free these days) and mail them to your home address. Each update is only the cost of a postage stamp and if you want unattended operation, have your wife or kids do it. :P
9/11 Eyewitnesses to Explosive WTC Demolition 1 of 2
I'm sure that whoever you're going with would appreciate you paying them a little more attention and not spending quite *all* of your time fiddling with gadgets.
Or is it not allowed to let the real world intrude into Planet Slashdot?
Get off my lawn.
When i was a kid we put the spent film in our pocket ( and then suitcase ) and waited until we got back to see the pictures.
---- Booth was a patriot ----
My experience is that hotel/internet-cafe access is too slow and/or flaky and/or expensive for the purpose you describe. Pay-as-you-go HSPA cell access is very expensive (in Europe and even more-so in North America). I guess if you're only taking 300 small jpegs per day, you might be able to afford the Internet access charges but my experience, even in Europe, is that your best bet is to make your own local backups as you go. My strategy is to travel with a small netbook and a USB drive. Each evening, I offload my SD cards onto both devices and then keep the netbook in the hotel safe or car and the USB drive with my camera. For example, I just returned from two weeks in Tuscany and am currently importing 34GB of photos into a new Lightroom catalog. There's no way that I could have transferred that data over the Internet while on the road without wasting a lot of valuable travel time. Heck, it's taking 20 minutes just to copy the photos off the USB drive at 30MB/s! How much time can you spend drinking espresso waiting for uploads?
I used to use cloud sync, but I found that none of them were robust enough for my abusive ways. Most required periodic manual merges.
I switched to Synkron syncing to a network drive (VPNed when I'm on the road) and I've been happy ever since.
Err.. wouldn't Apple's iCloud be the best, since that's what it was designed to do? Sync photos, set and forget style?
Not to threadjack, but I have an eerily similar situation:
I'm going on vacation around the US next month. I need a vehicle that has 4 wheels, and can seat 4 people. It needs to be completely enclosed and be capable of being driven in the rain. Also, it must run on gasoline, preferably on unleaded gasoline. Oh, and this is critical - it must have multiple mirrors so that I can see what's behind me as I operate it. I thought about getting a "car", but that would take all the fun out of it. Does anyone have any suggestions for an IT hipster like myself?
Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
I am on holiday now and am currently uploading the day's photos with SugarSync. Its cheaper per byte than DropBox and has more options. I think some people might find the many options confusing but not Slashdotters. eg you can sync multiple folders. When I was in a remote place with a slow connection (Laos) .... the smallest transaction was a file. So it might labor away for an 30 minutes to almost send a 3M file then the connection would drop and resume and it would start again with the same file - ouch.
I found a problem
I recommend:
while [ 1==1 ]; do rsync -avP local/path host:remote/path; sleep 300; done
Fun - Competent - Vacation
Choose two.
I'm doing something similar to this using rsync over ssh and a rsa key to auto-login to my server. I have the script set run once a day starting at midnight that way it pushes the days work online while I'm not using the internet. This works well for me, but you have the overhead of maintaining your own server and everything that goes along with that.
I'm on the road since more than a year in a round-the-world trip and I think my setup is fit.
Two notes,
- when travelling, internet connection level changes a lot and sometimes you can be not connected for days, so an external disk/memory card rotation should stay your primary backup, especially for photos
- for pictures, I'm taking both JPG+RAW and doing Internet backup only for JPG. If I have a full internet connection at my accomodation and can let transfer during night, if not day, it's working pretty well. RAW (10M per file or more, 5-10 times the JPG) is clearly not possible.
For small files (txt, id/passport copy, ...), using Dropbox as primary (want to stay free), and lsyncd with my home server
For pictures, nothing more powerful than rsync (at least, in one-way; if you make change at both place, could be interesting to check unisson) to my home server or NAS.
rsync is particularly good, as for now, my main changes on pictures are metadata and from what I tested, both lftp and Dropbox would transfer all the files if there is a minor change.
CD/DVD are way too small for JPG+RAW. maybe JPG only. ...
Of course, no pictures transfer if connection is poor or limited (3G or filtered). For China, need to pay for a VPN provider
This is what I use. Folder redirection can be a nightmare when it goes awry, but for the most part SBS keeps my files, documents and photos synced across 4 different machines (two desktops, a laptop and a netbook).
If I need to get at files that haven't yet synced because I forgot to turn on my laptop before walking out the door, I just VPN in to my network (SBS does all the setup and heavy lifting, you basically just turn it on and it works) - and either run a sync, or if I don't have time, I'll just access the files on the redirected folder on my server.
Easy peezy.
What do you mean by "pload all files pretty much as you go " For backup purposes? Or for a way for others to see them?
For backup, use something like crashplan. It should trickle it up when it can.
For others to see them, it'll depend on the service you use.
I have a dSLR with a 32GB SD card. I have an old Asus Eee 1000HA netbook with a 2GB RAM upgrade and Win7Pro. I upgraded the hard drive to a 750GB because of all the media I like to keep portable. When I go on holiday, I snap all the pics I like with my dSLR, and when I get back to the car/hotel I just run a simple batch file script that mirrors them all locally to the netbook to a datestamped folder with robocopy. That way I have a day-by-day structure to the archive (not that file datestamps arent enough, really). Following this, the script checks to see if my external USB drive is connected, and mirrors to that from the netbook's local drive. If the drive is not connected, it skips this step. After the copy completes, and all the files are verified, it wipes the SD card. Back into the camera it goes. Simple.
If I have internet access available, I run a second script that connects to my VPN box at home and launches a second robocopy to my desktop's fileshare. This way I can dump and wipe my SD card even if I don't have internet. The automatic syncing method is too unsure for me. This way, I'm responsible for my own data. If I wanted, I could chain them into one script very easily, but I prefer to have the dump as a quick option independent of the upload.
Sometimes people just make this too hard. I wrote the scripts in about 5 minutes, and didn't have to add any software (since my netbook runs Win7Pro). I already had VPN set up for secure remote operations, so no additional steps were needed. If you didn't have VPN, I'm sure you could manage with sFTP too, for the long haul home. Filezilla is free, so in all you can have a total investment of $0 and just a bit of time.
You're not paranoid if they really ARE out to get you...
I have one, and it's pretty good. Ad hoc directly to the laptop when you get back to the hotel, sync to Eye-fi, Flikr or wherever when you have a reliable connection. For 3rd level redundancy, copy every other day to a SD card and mail home.
I'm not seeing the complexity here.
They were recently bought by Citrix, but Citrix is keeping all the cool stuff baked in! It rock...
You can solve your particular problem with a 10 line script that runs in your notebook. Just write it!
Your first idea will do exactly what you want. It'll resume interrupted uploads, verify upload integrity, be secure in whatever context you're in... it's perfect. As long as you don't specify the --delete option, it won't delete remote files just because local files are gone, either.
A crontab line with: "killall rsync && rsync -a ~/local/photo/dir hostname.foo.net:remote/photo/dir" will do the job admirably. Set it to run every 10 minutes or so. You could obviously polish this solution in various ways, but it's quick, it's easy and it will work.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Unless you are comfortable with the idea of items you are storing ending up on the front page of a newspaper the "dropbox" front end to Amazon storage is a bad idea. Those clowns have so many epic failures that plain FTP from 20 years ago is more secure. They still haven't closed the exploit where you can change your password but it doesn't revoke access to anyone you've let in previously. Then there was the day when their authentication failed entirely and they just let anyone that could guess a username on. There were other little exploits based around their deduplication which were used to obtain other people files (making it a bittorrent replacement for popular video files) that could have been put to sinister use but mostly showed that those clowns really were way out of their depth on anything other than an MSDOS box with no network.
There's google drive, spideroak, a really long list of others including rolling your own SFTP that are vastly superior to dropbox. Then there's plain old FTP at any ISP on the planet - still in some ways superior to dropbox despite drawbacks.
Dropbox is an example of marketing hype selling a polished turd.
Shanghai Shunky Machinery Co.,ltd is a famous manufacturer of crushing and screening equipments in China. We provide our customers complete crushing plant, including cone crusher, jaw crusher, impact crusher, VSI sand making machine, mobile crusher and vibrating screen. What we provide is not just the high value-added products, but also the first class service team and problems solution suggestions. Our crushers are widely used in the fundamental construction projects. The complete crushing plants are exported to Russia, Mongolia, middle Asia, Africa and other regions around the world.
http://www.sandmaker.biz
http://www.shunkycrusher.com
http://www.jaw-breaker.org
http://www.jawcrusher.hk
http://www.c-crusher.net
http://www.sandmakingplant.net
http://www.vibrating-screen.biz
http://www.mcrushingstation.com
http://www.cnstonecrusher.com
http://www.cnimpactcrusher.com
http://www.Vibrating-screen.cn
http://www.stoneproductionline.com
http://www.hydraulicconecrusher.net
I think there is a free version of SYNCBACK that allows FTP backup (don't synchronize files). Set it up to automatically backup all files once a day (for instance when you have crashed at a hotel). It's very easy to do. I've done it this way on several occasions.
johnnygeneric
I take you treat your pictures, text and whatnot as source code. You may possibly edit these things items.
/. 101.
Make decent projects, use git and get over it. Rsync would work as well but without the SCM capability.
Not to moan and bitch, but this stuff is
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
OK, so you ignore a pile of epic failures on the security front and expect these clowns are going to have their act together in other areas?
Why bother with a workaround when there are literally thousands of other superior options (many ISPs still have personal storage for their users)?
If you're looking for a turnkey little utility to do the job, check out GoodSync. http://www.lifehacker.com.au/2011/07/how-to-set-up-a-file-syncing-dropbox-clone-you-control/ It is configurable with any server that has SFTP or FTPS accesss. It also works with a bunch of other cloud services, but I think this tutorial is more in line with what you want to do. The key is that it detects when new files appear in the source directory and triggers a new upload. I've since chucked Dropbox in favor of this.
Well why not take an artistic approach to your vacation.
Rather than looking at quantity of images and striving for completeness and literalness, give the viewers less... a lot less that leaves viewers wishing they had been there and wishing they had the fun you had.
For instance, Ed Ruscha enjoyed driving from LA to his mother in Oklahoma. He took pictures of gas stations on the drive and made a book titled Twenty Six Gas Stations. Thousands of people have looked at those 26 pictures dozens of times over the past 40 years. The gas stations are all gone, the highway is mostly gone, but you can tell it is all from the hand, eye and wistful exploring mind of one person. At the same time the reader might be puzzling about where the author is going, time begins to stand still and the reader's mind wanders.
Vision or seeing is a very elaborate computational, memory and real time experience that could be called almost a mental illusion. Artists have been exploring this gossamer fabric experience for thousands of years. What is that gossamer fabric you work with at work? Some holey and bug ridden thingie like Python or C? Why not use your freely wandering mind on vacation to go explore using your camera some aspect of your work that you see in the patterns of the world while you play. Put the two gossamers tangent to each other and take a few informative photos.
I'm currently cycling around the world and this is what I do: 1) Google Drive + Chrome for anything that's a document. File sizes are small and I *really* can't afford to lose some of this stuff so it works even on the shittiest wifi connections in the craziest of places. 2) USB sticks. Lots of them, posted back home as and when. Admittedly I don't take the volume of shots you do but an 8GB stick holds plenty of reasonably high resolution shots. YMMV. 3) External USB drive. As above, just posted less frequently. 4) rsync. I pay a commercial provider to let me rsync the contents of my hardrive (well the photos, anyway) to their infrastructure. With compression and a reasonable wifi connection this works just fine - I tend to ensconce myself in the corner of a cheap cafe for an hour or two once every couple of weeks - works well.
I use 3 things.
* SDHC camera
* Android Tablet (ssh/rsync/wifi)
* remote Linux server
My camera is dumb. No wifi.
I pull the micro-SDHC card and put it into a Android tablet. The camera takes a full-size SDHC, but the tablet requires a micro, so I only use those with an adapter.
My android tablet has something called "Terminal IDE" - that comes with ssh and rsync. I wrote a script that pushes (no delete) the files from the memory card to my remote Linux server. That script is not automated, because getting a damn hotel that allows ssh-based connections back home seems to be problematic.
Spent 10 days in Europe 2 months ago - only 1 hotel out of 5 allowed ssh to my server. They all had damn proxies that blocked everything except http/https traffic. If I'd setup my ssh server at home to listen on 443, it may have worked, but that eats up an already used IP/port. Smaller BnB accommodations won't have this issue.
Be certain to take a tiny travel wifi router with you. I was amazed at the hotels that do not support wifi in the rooms still. I have an old WiFi-G with WPA router for this purpose. Best $50 I've spent for travel besides the $2 6ft extension cord with 3 US electric plugs.
The main thing is to get the photos/videos onto 2 different media ASAP. Getting them back home soon is a nice-to-have luxury.
If you take a laptop, you will be able to work around issues that I couldn't due to the crap abilities of Android. This was my first attempt to travel without a netbook - I'll not do that again until a full Linux, native, install on ARM is working. I tested everything from the library AND a friends house before I left - keybaord, android, uploads, even remote RDP through an ssh-tunnel. Everything. It all worked perfectly.
I've been using this setup since 2008 traveling the world (Asia, Central and South America, USA) with a Nokia N800 as the tablet (ssh/rsync), but this was my first trip with Android instead of my N800. Did I mention that Android sucks?
What's worse is that both my and my friends Android phones were stolen in Barcelona, so our backup devices were gone. 1 was pick pocketed and the other was stolen while we were dining in a back room of a nice restaurant by gypsies. Fortunately, I had wiped mine and pulled the GSM-SIM before the trip - not even a gmail account was connected. Don't be too into your gadgets in public overseas. They make you a target.
If you are using Linux or *BSD (including OS X), git-annex is the way to go. Seriously.
http://git-annex.branchable.com/
http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own
Which is exactly what people are doing with it. I'm in the situation of trying to stop people using this piece of junk for getting information out to clients when they also have different information to give to other clients that are rivals. They discover to their horror that both sets of clients can see each others data and then three sets of expensive lawyers get dragged in - just because some clowns built a crappy python front end to Amazon EC3 and marketed it hard to make people think it wasn't a piece of shit and trust it when they should run a mile.
Of course you'll say "nobody would be that stupid", and they are not, just lazy and easily let by slick marketing. They don't read the fine print is all.
Or with personal information it's of value for identity theft. Date of birth, address and various identification numbers are of value to people that can use it for fraud and inconvenience you a lot.
One of the things things i couldn't get over with all the chatter here regarding the question is that the poster has admitted that there is spotty internet and they are looking to sync 300 images per day... If I'm holiday the last thing i want to do is try and manage a net connection. Best advise is to take a portable HD, job done. Get back to your holiday..
I think you are describing those options incorrectly for his case.
--inplace is the opposite of what he wants. As I understand it --inplace will defeat some of the automatic duplicate range detection and save *space on the server* by not duplicating data during transfer. This does not help with network bandwidth but *hurt*. He probably doesn't care about space on the server, he wants his files mirrored quickly.
--update won't hurt him here, but it's probably not necessary as you seem to be describing it backwards. If he just mods files on his laptop and rsyncs the newer files on the laptop will of course get transferred. The only reason to use --update would be if he modded files on the server at home *and* on the laptop and preferred to keep the ones at home.
Pat
I prefer mirroring certain files this way as you do not have to resort to a particular backup software to 'unpack' them. If you need to access the files, just open them up. This is especially handy for photos and other media files.
1: I personally found the Eye-Fi to be too slow and I had drain the battery and keep the laptop and camera on at the same time.
2: 3G data prices on vacation in not my home country are crazy.
3: having to do this in the only cafe that had wifi was nice the first day, then a chore every other day.
4: I didn't like spending my vacation working
so I use a product similar to this one. http://www.bhphotovideo.com/c/product/828315-REG/Digital_Foci_P19_320_PST_251_Photo_Safe_II.html
basically a portable HD with battery power and a card reader. I take the phone, tablet, camera and backup drive in a small hard shell case and leave the computer at home. the tablet is just the luxury item so I can view things in a larger format and only gets used on the plane or for research that the phone's screen is too small for. I really don't need it.
everything is backed up for when I return, and then I put it together .
If I absolutely have to work on the remote event, then it's camera > laptop [edit, select] > phone > 3g to web publish. Full online backup is going to cost significant time and money.
get an android phone and a G+ /picassa account , it then becomes automatic , whenever it has connectivity it will upload tou your G+ gallery , you can set it to wifi only , and i've tested this in vacation in china with very little connectivity to any google services and worked great
Except that my wife's camera doesn't have the Eye-Fi cards, so we're manually downloading the pictures from the camera to the laptop.
I have an Ubuntu server running rsync. I've installed cwrsync and am using a very simple script (below) to sync my laptop to the rsync server. I'm using a windows at command to schedule the script to run hourly when the laptop is on. That last part is pure laziness, I haven't bothered figuring out how to test for network connectivity and then to run when connected.
One very nice advantage to running my own rsync server is that I can sync my photos from my Android phones as well. Micha Kowalczuk has written a terrific rsync backup program for Android that's easy to use and can easily be set up to use ssh public/private keypairs for authentication using the instructions on his website. That, in addition to Crafty Apps' Tasker, enables me to backup all of the pictures (and whatever else I want) from my phone every time I have an established WiFi link. (Note that that is my own restriction. I don't want to pay the extra data fees for uploading my pictures over the cell data link.)
Hope that helps.
Note that and are replacing the actual values to protect my server. ;-) Also, I have created ~/rsync folders for each user for the backups. Finally, the switches in use will NOT delete pictures from the server if they're deleted from the laptop.
@echo off
cd c:\Program Files (x86)\cwRsync\bin
rsync -av --chmod u+rwx -e "ssh -i c:\Users\\Documents\certificates\cwrsync" "/cygdrive/c/Users//Documents/rsync/" @:/home/user/rsync/
You never said why you wanted to upload the photos but I'm assuming its for backup.
Why not go with physical backup. For 2 weeks at ~ 500 MB (2.5 MB/pic 200 pics) a day that a total of 7GB of photos. Easy to fit on a single memory card and just backup to your computer. If you are worried about loosing them memory cards are small enough to carry on person.
If it is to share your photos with people I would say stop and only upload your good shoots for the day.
The first step will be to make git-annex watch for changes to your files, check them into git, and automatically sync them to your other repositories. I have a prototype of this using Linux's inotify. It will be extended to also support Mac OS X.
Then I will build a web app that can be used to control and configure things. Watch files as they upload and download, set priorities, etc. No command line needed. I plan to use Haskell's amazing Yesod web framework.
Finally, I will add configuration assistants to help you get the most out of the system. Easily set up syncing to remote computers, store encrypted copies of your data in Amazon S3 or other cloud services, automate moving old files to archival drives. There are many possibilities like these, and I will prioritize the ones my backers need.
I'll spend around one month on each of these steps. Then I'll spend another month or two on an Android port, and additional time I'm funded for to add more features. I'll be blogging about my progress all along the way, and each new feature will immediately be available in git-annex's own git repository.
If he gets another $3500 in funding in the next 6 days, he even promises to spend a least 1 month trying to create a Windows port.
I use a new directory for incremental backup with rsync, using hard links and references to the last completed back up and the latest incomplete backup.
The major problem I have is that photos can take 12 or more minutes each on slow internet connections but I just leave it running overnight.