Slashdot Mirror
Guaranteed Transmission Protocols For Windows?
Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."
Clearly you're looking for UDP. Next question.
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
SFTP should do since the communications are encrypted, if something changes along the way it should be rejected by the other end. HTTPS and any other protocol-over-SSL should do.
FTP is a plain-text protocol so if something changes along the way it won't give you any issues.
Custom electronics and digital signage for your business: www.evcircuits.com
The summary states that with FTP, the downloaded files were of the wrong size. Can anyone explain why TCP's efforts to to deal with unreliable networks, such as the retransmission of unacknowledged packets and their reassembly in proper order, would not already deal with this? I am familiar with the concepts involved but I think I lack the low-level understanding of how you would get the kind of results the story is reporting.
It is a miracle that curiosity survives formal education. - Einstein
Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx
W
Background Intelligent Transfer Service (BITS) can be used to transfer files between windows servers. It is the technology behind Windows Update. We use it in our company to transfer files across a low bandwidth sattelite connection. Great thing is that it can automatically resume transfer after rebooting both machines. SharpBits offer a nice .NET API. You can find it here: http://www.codeplex.com/sharpbits
I'd say BitTorrent -- with firewall rules or some other measure so random people can't see your microscopic swarm. It uses SHA-1 hashes of chunks, so if a torrent client says a file downloaded successfully it's pretty much guaranteed to be true.
________
Entranced by anime since late summer 2001 and loving it ^_^
hi there,
why don't you get cygwin on both the systems and then do a rsync ?
between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).
BR,
~A
they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals
Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).
Use binary mode and you'll be fine.
... is what you want. Yes, you can use it with Windows (with or without cygwin bloat). Use -c and a short --timeout and you're good to go. If you're using it over ssh you're looking at three layers of integrity (rsync checksums, ssh and TCP), two of them quite strong even against malicious attacks not only against normal stuff. Put it in a script with a short --timeout; if anything is wrong with the link your ssh session will freeze completely, as soon as your --timeout is reached rsync will die and your script can respawn a new one (which will resume the transfer using whatever chunks with good checksum you have already transfered and will again checksum the whole file when it finishes).
Implementations of TCP in most operating systems fall a bit short of that, killing off stalled connections, etc. Also, some firewall suites, and some routers make a habit of killing off connections after a certain amount of time, sometimes without regard to whether or not they are 'active'.
You might have some luck boosting reliability with the TcpMaxDataRetransmissions registry setting in Windows. But ultimately, the poster is going to need to find a file copy suite which retries when connections die.
Ritchie's Law - assume you have screwed something up *first*, before blaming the tool...
Even on reliable connections, using .complete files is a great idea.
It works this way: If you're pushing, open ftp, after ftp completes, you check remote filesize, if matches local file size, you also ftp a 0 size .complete file (or a $filename.complete file with md5 checksum, if you want to be extra paranoid).
Any app that reads that file will first check if .complete file is there.
If remote file size is less, you resume upload. If remove filesize is more than local, you wipe out remote file and restart.
Same idea for the reverse side (if you're pulling the file, instead of pushing).
You can also setup scripts to run every 5 minutes, and only stop retrying once .complete file is written (or read).
Note that the above would work even if the connection was interrupted and restarted a dozen times during the transmission. [we use this in $bigcorp to transfer hundreds of gigs of financial data per day... seems to work great; never had to care for maintenance windows, 'cause in the end, the file will get there anyway (scripts won't stop trying until data is there)].
"If anything can go wrong, it will." - Murphy
You forgot a few:
Windows at both ends... Used to use FTP... Considering windows file sharing...
Is anyone else a little nervous? I hope by 'government' he means Department of Natural Resources or some equally uninteresting entity. I am picturing someone at the SEC going "You know, I swear this accounting data had a few more rows the last time I looked at it-- Oh well it's not like this Madoff guy is actually up to anything strange anyway"
Too easily thrown around if you ask me. He's not looking for anyone to set it up, he just wants some options. Isn't that what community is about?
You are kidding about this, aren't you?
Let me get the facts straight:
- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.
If this is the case, you don't have a technical problem on your hands; you have a political one.
"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.
If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.
If your management doesn't do anything about this, then the files are not "mission critical".
Not to mention the three day latency on refreshing the entropy pool.
It must be open-source with such an odd name.
Close. It's open sores, especially around the wrists.
Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!
Z-modem is your answer.