Slashdot Mirror
Guaranteed Transmission Protocols For Windows?
Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."
Clearly you're looking for UDP. Next question.
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx
W
Background Intelligent Transfer Service (BITS) can be used to transfer files between windows servers. It is the technology behind Windows Update. We use it in our company to transfer files across a low bandwidth sattelite connection. Great thing is that it can automatically resume transfer after rebooting both machines. SharpBits offer a nice .NET API. You can find it here: http://www.codeplex.com/sharpbits
hi there,
why don't you get cygwin on both the systems and then do a rsync ?
between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).
BR,
~A
they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals
Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).
Use binary mode and you'll be fine.
TCP has timeouts. The FTP client and server probably have timeouts. Eventually, some bit of the system will decide the operation is taking too long and give up. The FTP client is probably reporting an error, but if it's driven by a poor script no-one will know.
The only times I've seen FTP report a successful file transfer and have a file discrepency is when a binary file has been transferred in ASCII mode and the CR/LF sequences are being swapped for just CRs, or visa versa. Nothing wrong with the protocol, PEBKAC...
UNIX? They're not even circumcised! Savages!
... is what you want. Yes, you can use it with Windows (with or without cygwin bloat). Use -c and a short --timeout and you're good to go. If you're using it over ssh you're looking at three layers of integrity (rsync checksums, ssh and TCP), two of them quite strong even against malicious attacks not only against normal stuff. Put it in a script with a short --timeout; if anything is wrong with the link your ssh session will freeze completely, as soon as your --timeout is reached rsync will die and your script can respawn a new one (which will resume the transfer using whatever chunks with good checksum you have already transfered and will again checksum the whole file when it finishes).
Ritchie's Law - assume you have screwed something up *first*, before blaming the tool...
You are kidding about this, aren't you?
Let me get the facts straight:
- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.
If this is the case, you don't have a technical problem on your hands; you have a political one.
"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.
If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.
If your management doesn't do anything about this, then the files are not "mission critical".
I'm gonna learn you some English: First, I will download my photos to my Facebook page. Then I will borrow you my car but in collateral I demand you borrow me you're lawnmower for a week so I can mow my lawn. Your smart, so you will do good on your next test.
I used to get dropped characters and groups of characters in text files using FTP back in the 1990s and early 21st century. It seemed to be a bug in the FTP client, because it only happened when we used the Windows Explorer interface for the product. When we did command line or used the native GUI there was no problem. If you're seeing this type of a pattern where you can see that characters are missing, switch to a different FTP client or try the Windows command line FTP.
Another possibility is that the target Windows system is mimicking a Unix system, so that an ASCII transfer is stripping the CR characters from CR/LF sequences.
On the other hand, if you really want a "guaranteed delivery" with formal acknowledgment and validation, try using a secured protocol like SSH or SFTP or a messaging system like JMS with a handshaking architecture around it. There are plenty of Open Source architectures you can build around (xBus for example), but I don't know of any ready-built executables. Commercially, vendors like IBM (MQ) and Tibco have products that deal with the messaging at a similar level.
We are the 198 proof..
While others point out, probably correctly, that the problem is probably a binary/ascii conversion, in actuality the error checking on TCP is simply not that good.
TCP uses a 16-bit checksum, so you have 1 in 65536 chance of an error packet being incorrectly validated as being correct. To make matters worse, it uses 1's complement instead of 2's complement, so 0x00 and 0xFF are indistinguishable.
Ethernet has a 32-bit, 2's complement checksum so if you're transmitting over that link-layer you're probably in good shape. But depending on that from a systems point of view seems risky.
Much better to only transfer ZIPs and check them at the other end if you only have control over the endpoints. If you can control the transmission, use a better error-correcting high-level protocol or even a forward-error correction protocol on top of TCP.
Or just use rsync.
E pluribus unum
Because of differences between systems like Unix and Windows, where line ends are a simple newline on Unix but a CR/LF pair on Windows. Also systems like VMS which have (had) about thirteen different file formats all inherent in the file structure itself.
In other words, because all ASCII files are not represented the same way by all different operating systems.
I know that Windows uses CR/LF for line termination and *nix uses just LF. That's a very minor inconvenience at worst,
Not if you have an "ASCII" file you are trying to read on Windows that has Unix newline conventions. Try opening a newlined file with notepad, for example.
"Little standalone utilities" are really handy for small files and small numbers of files. It's really handy when you know the format the file you have is in and what it needs to be. Please tell me how you will identify a VMS fixed record file that you have just ftp'd from a VMS FTP server when it gets to your Windows system. It has NO newlines or CR/LF pairs. You might dump the file somehow and notice that the lines are all 93 characters long and then write yourself a perl script to split it up -- or you could simply tell your FTP client that you are in ASCII mode and let the FTP server/client negotiate some resulting format that your system likes. Now try that with a VMS variable length record file, where the lines are variable length, still without line endings.
FTP wasn't designed just for hobbyists who want a file or two and have the time to deal with file formats by hand. It was designed to move data, and anything that can be automated should be. "Little standalone utilities" are a pain in the ass when trying to automate something, especially when the critical information necessary to know what specific utility to use has been lost, or is completely unknown to the recipient's system. Like VMS fixed length records on Unix or Windows.
It just seems like it's not the job of a file transfer protocol to concern itself with what an independent, unrelated application can or cannot do with the file after it's transferred.
ASCII mode in FTP has nothing to do with anyone trying to tell anyone what they can or cannot do with a file after it's transferred. It's all about knowing how to deal with a hundred different ways of representing ASCII data on dozens of different operating systems and making life EASIER for people who have to do that on a daily basis.
If YOU would rather operate in BIN mode and worry about which file formats you've just downloaded and how to convert them to an ASCII representation that your software knows how to deal with, more power to you. I got tired of dealing with this the first time I had to convert a VMS "ASCII" file to Unix and I'll let FTP do it silently for me. Yes, I've dealt with users who didn't know what ASCII mode was and downloaded a zipped file in ASCII mode and it didn't work, but the time I've saved just myself not having to deal with converting crap has more than made up for the time I've spent telling them to use BIN mode.
Wrong. FTP has a binary mode. This is probably the reason his files are missing several k at the destination. Sending a binary file in ascii mode is the ONLY TIME I've ever had a file not transfer entirely/correctly using FTP. Unless of course there is a network error/timeout, etc, but the FTP client always errored out in those cases. Using SFTP over an already secure network will only slow things down greatly.
-Unresolved symbol? Byte me!
Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!
Z-modem is your answer.