Guaranteed Transmission Protocols For Windows?

Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."

Comments

Robocopy?

[wafath]

Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx

W

Re:Robocopy?

[Malc]

It might be using Windows copy protocols, but it definitely is not like copy/paste. It's restartable for instance. It's way more reliable.

We have to copy large files to our office in China. FTP always fails. Windows copy via Explorer often fails, but it is also incredibly painful to do when latency is high and one is browsing over the network. Robocopy (depending on system setup) will motor through and is very persistent when there's a connection hiccup. You definitely want restartability if you copy large files are a couple of hundred MB an hour.

I'd say make sure to break the files up in to chunks if they're large. Also, run 2-4 robocopies in parallel if the latency is high as this will give better throughput. It can do funny things to Windows though (maybe other things wait on some network handle and seem to freeze until one of the robocopy processes moves on to the next file).

Also, consider doing it over a Cisco VPN. It seems to add some robustness if there is packet loss. I often had trouble access servers in the US when I was living in China due to packet loss, but no such problem over a VPN (zero packet loss, but very slow instead, which is better).

BitTorrent

[Inf0phreak]

I'd say BitTorrent -- with firewall rules or some other measure so random people can't see your microscopic swarm. It uses SHA-1 hashes of chunks, so if a torrent client says a file downloaded successfully it's pretty much guaranteed to be true.

rsync should do the trick

[bacchu_anjan]

hi there,

    why don't you get cygwin on both the systems and then do a rsync ?

    between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).

BR,
~A

Line endings!

[sys.stdout.write]

they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals

Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).

Use binary mode and you'll be fine.

Re:TCP?

[Anonymous Coward]

TCP has timeouts. The FTP client and server probably have timeouts. Eventually, some bit of the system will decide the operation is taking too long and give up. The FTP client is probably reporting an error, but if it's driven by a poor script no-one will know.

Re:TCP?

[AvitarX]

I bet it is file systems with different block sizes rounding slightly differently, and an OP that does not understand.

Re:domyjobforme tag

[Anonymous Coward]

Too easily thrown around if you ask me. He's not looking for anyone to set it up, he just wants some options. Isn't that what community is about?

You're kidding, aren't you??

[ballyhoo]

You are kidding about this, aren't you?

Let me get the facts straight:

- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.

If this is the case, you don't have a technical problem on your hands; you have a political one.

"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.

If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.

If your management doesn't do anything about this, then the files are not "mission critical".

Re:You're kidding, aren't you??

[Dravik]

Mission critical means that you need to get it done even if someone else isn't getting their job done. Standing around in a huff and stomping your feet means that the mission critical information isn't getting moved. What he needs to do is find a way to accomplish his mission despite the difficulties, and then document the problems so they can be addressed.

Re:Any encrypted transmission protocol actually

[link-error]

Wrong. FTP has a binary mode. This is probably the reason his files are missing several k at the destination. Sending a binary file in ascii mode is the ONLY TIME I've ever had a file not transfer entirely/correctly using FTP. Unless of course there is a network error/timeout, etc, but the FTP client always errored out in those cases. Using SFTP over an already secure network will only slow things down greatly.

Z-Modem FTW!

[Cytotoxic]

Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!

Z-modem is your answer.