Slashdot Mirror
Guaranteed Transmission Protocols For Windows?
Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."
Clearly you're looking for UDP. Next question.
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
SFTP should do since the communications are encrypted, if something changes along the way it should be rejected by the other end. HTTPS and any other protocol-over-SSL should do.
FTP is a plain-text protocol so if something changes along the way it won't give you any issues.
Custom electronics and digital signage for your business: www.evcircuits.com
The summary states that with FTP, the downloaded files were of the wrong size. Can anyone explain why TCP's efforts to to deal with unreliable networks, such as the retransmission of unacknowledged packets and their reassembly in proper order, would not already deal with this? I am familiar with the concepts involved but I think I lack the low-level understanding of how you would get the kind of results the story is reporting.
It is a miracle that curiosity survives formal education. - Einstein
Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx
W
Background Intelligent Transfer Service (BITS) can be used to transfer files between windows servers. It is the technology behind Windows Update. We use it in our company to transfer files across a low bandwidth sattelite connection. Great thing is that it can automatically resume transfer after rebooting both machines. SharpBits offer a nice .NET API. You can find it here: http://www.codeplex.com/sharpbits
I love it! Haha... that's probably one of the better tags I've seen.
I'd say BitTorrent -- with firewall rules or some other measure so random people can't see your microscopic swarm. It uses SHA-1 hashes of chunks, so if a torrent client says a file downloaded successfully it's pretty much guaranteed to be true.
________
Entranced by anime since late summer 2001 and loving it ^_^
hi there,
why don't you get cygwin on both the systems and then do a rsync ?
between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).
BR,
~A
Wasn't TCP designed for just this? Guaranteed transmission?
Similes are like metaphors
they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals
Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).
Use binary mode and you'll be fine.
... is what you want. Yes, you can use it with Windows (with or without cygwin bloat). Use -c and a short --timeout and you're good to go. If you're using it over ssh you're looking at three layers of integrity (rsync checksums, ssh and TCP), two of them quite strong even against malicious attacks not only against normal stuff. Put it in a script with a short --timeout; if anything is wrong with the link your ssh session will freeze completely, as soon as your --timeout is reached rsync will die and your script can respawn a new one (which will resume the transfer using whatever chunks with good checksum you have already transfered and will again checksum the whole file when it finishes).
Ritchie's Law - assume you have screwed something up *first*, before blaming the tool...
You don't need to MD5 if you're using rsync. The rsync algorithm already uses checksums to ensure the files are bit-for-bit identical. In fact, rsync 3.x uses MD5.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Think of this transfer model like a car, the further it goes, the more bytes are burned up. they just need to be added back in with a network filling station. I would look to google for a government approved provider.
I've never heard of that product. Who makes it? Can it do binary transfers also? It must be open-source with such an odd name.
Table-ized A.I.
Or a mine-field...
You should look at the EDIINT AS2 protocol, AKA RFC 4130. This is a widely-used e-commerce protocol built over HTTP/S.
AS2 provides cryptographic signatures for authentification of the file at reception, non-repudiation and message delivery confirmation (if no confirmation is returned, the transfer is considered a failure), and is geared towards files. There is even an open-source implementation avaliable.
More complex than FTP/SFTP but entirely worth it if your data is mission-critical and/or confidential. Plus, passes through most networks because it is based on HTTP.
You're not old until regret takes the place of your dreams.
Even on reliable connections, using .complete files is a great idea.
It works this way: If you're pushing, open ftp, after ftp completes, you check remote filesize, if matches local file size, you also ftp a 0 size .complete file (or a $filename.complete file with md5 checksum, if you want to be extra paranoid).
Any app that reads that file will first check if .complete file is there.
If remote file size is less, you resume upload. If remove filesize is more than local, you wipe out remote file and restart.
Same idea for the reverse side (if you're pulling the file, instead of pushing).
You can also setup scripts to run every 5 minutes, and only stop retrying once .complete file is written (or read).
Note that the above would work even if the connection was interrupted and restarted a dozen times during the transmission. [we use this in $bigcorp to transfer hundreds of gigs of financial data per day... seems to work great; never had to care for maintenance windows, 'cause in the end, the file will get there anyway (scripts won't stop trying until data is there)].
"If anything can go wrong, it will." - Murphy
You forgot a few:
Windows at both ends... Used to use FTP... Considering windows file sharing...
Is anyone else a little nervous? I hope by 'government' he means Department of Natural Resources or some equally uninteresting entity. I am picturing someone at the SEC going "You know, I swear this accounting data had a few more rows the last time I looked at it-- Oh well it's not like this Madoff guy is actually up to anything strange anyway"
You are kidding about this, aren't you?
Let me get the facts straight:
- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.
If this is the case, you don't have a technical problem on your hands; you have a political one.
"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.
If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.
If your management doesn't do anything about this, then the files are not "mission critical".
Not to mention the three day latency on refreshing the entropy pool.
"You don't need to MD5 if you're using rsync. The rsync algorithm already uses checksums to ensure the files are bit-for-bit identical. In fact, rsync 3.x uses MD5."
Rsync, by default, does not necessarily do this. I've seen situations where rsync would happily copy files from a remote host over ssh to a destination host and the resulting files failed an independent MD5 test. Rsync was not causing this trouble - but it did fail to detect it. Forcing a checksum of every file (using "-c") would let rsync detect the failure to copy properly (after the entire file was done) and it would retry.
In the end, a router and one of the hosts were rebooted and the problem went away. The point is that just using rsync and ssh does not guarantee anything.
A.
...bringing you cynical quips since 1998
On some level, there isn't much difference between an application and a protocol. In fact, if you ever take a networking theory course, you'll see that each protocol layer in the network stack is, in fact a "protocol machine" (i.e., an application), which does the little protocol dance that makes functions at that layer happen.
But I digress. What the user is running into here is a fundamental problem with TCP over lossy networks. It really was not designed with really lossy networks in mind. E.g., the congestion control mechanism in TCP ("exponential backoff") makes the assumption that there is a wire sitting there and that certain parameters (like bandwidth) are not going to change. If you need certain QoS guarantees on a wireless link, TCP may be hard-pressed to deliver, because TCP's [limited] QoS mechanisms may make the problem worse. There is a HUGE amount of overhead on 802.11 networks to make sure that TCP doesn't suck.
I don't know how this person's microwave link is configured, but they might be better served by thinking about the QoS guarantees in the various layers in their network stack. I know a previous poster was joking when they said UDP might be a good option, but look, part of the problem on wireless is TCP's retransmission mechanism. With UDP it is up to the user/application to ask for a retransmit. Bittorrent works exactly like this, so something like Bittorrent, where each small file chunk gets its own hash, and those hashes are checked upon receipt, might not be a bad idea. I like rsync as well (because it has a rolling checksum feature), but again, you have TCP in the mix, and if I recall correctly, rsync will not retry automatically on failures, which is what you want.
That depends... http://www.defensetech.org/archives/000085.html
The transmission system is irrelevant. All that matters is that you know you have received whatever was sent.
Just make sure you send a checksum and that the received file matches.
oh wait... Windows scripting...
Deleted
I worked on a system for the Utah DNR once. Data about sensitive species, species of concern, and endangered species have security requirements. If someone finds out how many Woundfin we are down to...the terrorist win.
THL phish sticks
It must be open-source with such an odd name.
Close. It's open sores, especially around the wrists.
Windows reports file sizes exactly, to the byte.
It reports both the true file size and the file size on the disk, which is based on the block size and the number of blocks required to store the file. ..
Set up a BSD lpd queue under Cygwin, something like:
sendit:lp=/spool/null:sd=/spool:if=/spool/sendit.sh:sf:sh:mx#0:
Have the sendit.sh script do whatever it is you want with the file. To send a file: lpr -Psendit filename
Configuration of the network queue left as an exercise for the student. (Hint - queue pathnames locally.)
I use sshfs file mounts for all office document file sharing and such, not just one time transfers. SSH encryption security, with the ability to open and edit files over the network. No goofing around with samba or windows file sharing. Regardless, some sort of ssh or sftp at least.
Not sure about getting it to work on windows, but there should be some options.
Living in Chile
Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!
Z-modem is your answer.