Guaranteed Transmission Protocols For Windows?

Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."

Comments

TCP?

[causality]

The summary states that with FTP, the downloaded files were of the wrong size. Can anyone explain why TCP's efforts to to deal with unreliable networks, such as the retransmission of unacknowledged packets and their reassembly in proper order, would not already deal with this? I am familiar with the concepts involved but I think I lack the low-level understanding of how you would get the kind of results the story is reporting.

Re:TCP?

[Jimmy_B]

Not if you have an "ASCII" file you are trying to read on Windows that has Unix newline conventions. Try opening a newlined file with notepad, for example.

As far as I can tell, the problem is entirely unique to notepad. Every other text editor I've ever tried handles files with Unix-style newlines correctly. Since it would be trivial to fix Notepad, I can only assume that Microsoft either doesn't care at all about Notepad, or is deliberately leaving the incompatibility in place to discourage use of Unix.

Use .complete files.

[Prof.Phreak]

Even on reliable connections, using .complete files is a great idea.

It works this way: If you're pushing, open ftp, after ftp completes, you check remote filesize, if matches local file size, you also ftp a 0 size .complete file (or a $filename.complete file with md5 checksum, if you want to be extra paranoid).

Any app that reads that file will first check if .complete file is there.

If remote file size is less, you resume upload. If remove filesize is more than local, you wipe out remote file and restart.

Same idea for the reverse side (if you're pulling the file, instead of pushing).

You can also setup scripts to run every 5 minutes, and only stop retrying once .complete file is written (or read).

Note that the above would work even if the connection was interrupted and restarted a dozen times during the transmission. [we use this in $bigcorp to transfer hundreds of gigs of financial data per day... seems to work great; never had to care for maintenance windows, 'cause in the end, the file will get there anyway (scripts won't stop trying until data is there)].