Slashdot Mirror

Except that my wife's camera doesn't have the Eye-Fi cards, so we're manually downloading the pictures from the camera to the laptop.

I have an Ubuntu server running rsync. I've installed cwrsync and am using a very simple script (below) to sync my laptop to the rsync server. I'm using a windows at command to schedule the script to run hourly when the laptop is on. That last part is pure laziness, I haven't bothered figuring out how to test for network connectivity and then to run when connected.

One very nice advantage to running my own rsync server is that I can sync my photos from my Android phones as well. Micha Kowalczuk has written a terrific rsync backup program for Android that's easy to use and can easily be set up to use ssh public/private keypairs for authentication using the instructions on his website. That, in addition to Crafty Apps' Tasker, enables me to backup all of the pictures (and whatever else I want) from my phone every time I have an established WiFi link. (Note that that is my own restriction. I don't want to pay the extra data fees for uploading my pictures over the cell data link.)

Hope that helps.

Note that and are replacing the actual values to protect my server. ;-) Also, I have created ~/rsync folders for each user for the backups. Finally, the switches in use will NOT delete pictures from the server if they're deleted from the laptop.

@echo off
cd c:\Program Files (x86)\cwRsync\bin
rsync -av --chmod u+rwx -e "ssh -i c:\Users\\Documents\certificates\cwrsync" "/cygdrive/c/Users//Documents/rsync/" @:/home/user/rsync/

I use a combination of copSSH, an excellent OpenSSH package for Windows, port forwarding and good old RDP (Because I don't really like the idea of publishing my RDP connection out on t'internet when I can use Public Key auth with SSH). Plus using SSH gives me SCP for file transfers, which is usually a bit faster than doing it via redirected drives in Windows.

The Windows 7/2008 R2 version of Remote Desktop (v7) has full support for multiple monitors (finally), Aero and for streaming audio and video via WMP so watching stuff is less of a slideshow (though still not really great with your average home broadband upload speeds) as well as local resource mapping (printers, drives, smartcards, etc).

Um, what about this? Or would you say that cygwin counts as "unix?"

I have been using this for a while. I have not tested it on Win7 yet, but I would expect it to work, as long as you don't install under "C:\Program Files".

I think the original poster intended to ask about a client application protocol. Like their reference to FTP. If my assumption is correct, then the poster from the article might be interested in something like rsync, only for Windows instead of Unix. In which that case would be cwrsync - Rsync for Windows.

I invested more time in making this post than researching file transfer solutions for this article. More research by the interested party would be wise.

"Several GB" in 20 minutes? Rsync might be able to help you speed that up a bit. I can copy 120GB in about 90 minutes using rsync (up to 15MB / second or so on my 2x1GB NIC team), and it doesn't take substantial processor time. If you're on a LAN, don't use the "-v" option so it doesn't waste time trying to compress it, just slam it through.

This particular process runs under Cygwin on the server side (Windows 2003, HP BL460c blades with dual quad-cores, 8 GB of memory -- we're moving to 64-bit in the near future but wanted to go ahead and get the memory when we got the blades.) with a similar blade on the other end (RHEL 5 with another 8GB memory). Now granted I have fiber to the SAN (EVA 4400) for both machines.

Rsync is a mission-critical app for us, I really can't say enough about it. The Windows version is available at http://www.itefix.no/cwrsync/

On a desktop level, I also use my MacBook Pro (OSX 10.4, Dual core 2.4, 4GB memory over Firewire 800) with my 1TB LaCIE BigDisk drive. My XP VM is roughly 28GB and it copies in about 12 minutes, which averages more than 2GB per minute. I think I paid $150 or so for the drive, it was reconditioned. The drive (actually 2 drives in a RAID 0) isn't nearly as fast with multiple read/writes happening, and it does eat some CPU while it's going, but it works for me.
HTH.

Rsync is available (and has been available) for quite some time now on the Windows plaform. It's a cross-platform open-source tool that is significantly faster with less overhead than robocopy. I use it in a production environment daily to synchronize Terminal Server user profiles to backup servers (one on, one not on the SAN), guarantee copies to other servers, etc and have had nothing but positive experience with it. Setup can be done without a reboot and the text config file is simple enough for entry-level IT personnel to implement.

We actually keep a "emergency copy" folder for sending large updates to a specific client over the WAN... just copy the (tiny) folder to the client and run the command-line utility. Remove when done. Mix it with PSEXEC and they don't even know what you did... just that that huge file they needed transferred got there in a hurry.
Linkie http://www.itefix.no/cwrsync/

Hopefully a method for those XP laptops is out there somewhere (apart from the obvious like acronis, partimage or just plain dd that avoids the issue by not letting MS window run and lock the stuff - but you can't get anything else done at the same time). I will also persist in calling poorly documented workarounds hacks although there may be better words.

On a sidenote, I even use rsync to back up my Windows PCs. The nice people at http://www.itefix.no/phpws/index.php?module=pagema ster&PAGE_user_op=view_page&PAGE_id=6&MMN_position =150:150 put together "cwRsyncServer" for Windows for free using rsync and Cygwin, which works like a charm.

It installs itself as a service and allows for really smooth backups using BackupPC.

Here's a pretty OS-nonspecific example of cross-platform storage implementation. Some of it is about backups and may seem off-topic but is valuable as an example of how much you can mix platforms and OS to get what you need in network storage solutions.

We protect 3 Terabytes per night from 250 remote servers with a backup strategy using RSYNC. These include both Windows and Netware servers. Our centralized backup file server is a single Dell PowerEdge 2850 with dual Xeon CPUs which runs OpenSuse 10 and has a combination of both Dell Powervault RAID SCSI enclosures and LaCie Big Disk USB External drives attached. Using a fast server with an OS that we can tune gives us incredible multistream-capable throughput for network storage. Think about the speed required folks, 3 Terabytes in 12 hours from 250 hosts at 75 sites. (Well RSYNC means we don't send all the data, but still! ;-0 )

Then, each day, we back up the Linux box using a Windows server installed on a Dell Optiplex workstation box with a tape jukebox attached and running CA ArcServe. That way we get a daily snapshot to tape allowing us to do a scheduled rotation.

This means we are following the Golden Rule of Backups, which applies no matter how much data you back up, which is this: Always have 2 separate backup copies of important data. And it's better if they are different types of media. And with SANS and NAS solutions redundancy is critical. These acronyms should be called AIOB which stands for 'All In One Basket'

RSYNC has done what no commercial software seemed to be able to do: give us a good working backup system for our enterprise. It uses very efficient synchronization and compression algorithms to move the changes from our distributed servers. If you want this rig to do backups too I recommend considering it. Here's a link to the RSYNC Project:

http://rsync.samba.org/

Here's the Novell RSYNC forum:

http://forge.novell.com/modules/xfmod/newsportal/t hread.php?group_id=1148&group=novell.forge.rsync.h elp

And here's a good resource for RSYNC on Windows:

http://art.wilderness.org.au/software/help_cygwin- rsync.shtml

Here are two more good RSYNC Windows links:

http://www.itefix.no/phpws/index.php?module=pagema ster&PAGE_user_op=view_page&PAGE_id=6&MMN_position =23:23

http://www.nasbackup.com/

The NASBackup Project is a neat Open Source effort to make a gui-based RSYNC client for Windows. It works very well.

More info: RSYNC uses an algorithm that only sends the changes in the file systems. This algorithm is so efficient that i can even get down to only sending the changed blocks in an individual file without having to send the whole file. It works very well for us even over DSL/Cable speed connections. You want to optimize your entire I/O schema including all network layers as well as the way you read, write, and cache file and database operations on all connected hosts.

I hope this little bit of info helps you.

First of all, I think you should just look at keeping the existing system, just improve it. Changeover cost in hardware/software is going to be high, even if it's free software. Here's what I'd do to try to stay with Windows 2k or XP (throw this all out if you're on 98/ME and get a real OS!):

1. Antivirus
First of all, why no antivirus? Any reasonable Win2k/XP system should be able to run one. If you want something with very low cpu impact, try Eset's Nod32. Also exclude the directory that the DVR uses to write the videos from virus checks. The videos are unlikely to get infected, and virus checking on those directories will just muck things up. (I'm assuming that this is why you aren't using antivirus.) But everything else then can be protected.

If you have licenses for *any* antivirus product, try it again with excluding the videos directories. Any antivirus product worth more than a warm bucket of spit should be able to do that.

2. Disable services.
Disable every unneeded service on these machines. A *lot* of them shouldn't be on. These systems should be doing practically nothing but writing video files (ok maybe some backups, or transferring files to another server for backups). A decent guide to this is here: http://www.theeldergeek.com/services_guide.htm.

3. Consider turning off Windows networking.
Disabling SMB/Netbios calls should stop most viruses/worms/etc. If you need to transfer data for backups and such, use SSH and SFTP instead. SFTP is what you'd use on a Linux/Unix system, and is *much* more secure.

Free Win32 SFTP client:
http://winscp.net/eng/index.php

Free Win32 SFTP server:
http://itefix.no/copssh

Nice, and not too expensive pay SFTP client (Tunnelier) and server (WinSSHD):
http://www.bitvise.com/

(And you shouldn't be getting email-borne viruses -- these systems shouldn't be used for email.)

You can also use SSH on this to restrict all kinds of other access as well, while providing VPN-style access. Very, very nice. (e.g. you can only Remote Desktop or VNC through SSH)

4. Block ports and such, and firewall it.
Setup a firewall between these systems and the outside world. Restrict ports to *only* those needed (e.g. SSH on port 22). If possible, restrict outgoing data to *only* those IP addresses that need access. Yeah, IPs can be falsified, but it's an extra layer of defense.

You could do this through a software firewall, or even just some cheap $20 hardware firewall boxes.

The XP firewall is better than nothing, but it's only incoming. Much better incoming/outgoing freebie firewalls are available from these companies:
http://www.wyvernworks.com/firewall.html
http://www.jetico.com/

(I'd probably do the hardware firewall, but if you're cash is tight, or the time/cost of installing all these extra hardware boxes is high, at least deploy a software firewall.)

5. Other Windows hardening options
You can also try these two freebie Windows hardening programs. They probably aren't perfect, but they help:
Harden-it: http://www.sniff-em.com/hardenit.shtml
Secure-it: http://www.sniff-em.com/secureit.shtml

And decent googling should turn up lots of different hardening guides to Windows as well.

After these you should have antivirus, you're blocking ports, you've disabled almost all virus vectors, and should have systems that are reasonably secure and stable.

Yeah, you have Windows and not sexy or politically correct OSS. But it's what you have. If you can make it work, use it. Fixing up your Windows boxes is probably a lot less time and money than swapping over

Check out cwrsync

It is a stand alone package of rsync for windows. It even comes with an installer to make it run as a service. I use to it replicate web content on some faily major websites.

A far more efficient method would be to look at using RSYNC with SSH as a few others have pointed out. There's a pretty good HOWTO for Windows here, which avoids the overhead of a full Cygwin install. You'll probably want to check out the link to cwRSYNC at the top of that page too.