Guaranteed Transmission Protocols For Windows?

Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."

Comments

Robocopy?

[wafath]

Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx

W

Re:Robocopy?

[Malc]

It might be using Windows copy protocols, but it definitely is not like copy/paste. It's restartable for instance. It's way more reliable.

We have to copy large files to our office in China. FTP always fails. Windows copy via Explorer often fails, but it is also incredibly painful to do when latency is high and one is browsing over the network. Robocopy (depending on system setup) will motor through and is very persistent when there's a connection hiccup. You definitely want restartability if you copy large files are a couple of hundred MB an hour.

I'd say make sure to break the files up in to chunks if they're large. Also, run 2-4 robocopies in parallel if the latency is high as this will give better throughput. It can do funny things to Windows though (maybe other things wait on some network handle and seem to freeze until one of the robocopy processes moves on to the next file).

Also, consider doing it over a Cisco VPN. It seems to add some robustness if there is packet loss. I often had trouble access servers in the US when I was living in China due to packet loss, but no such problem over a VPN (zero packet loss, but very slow instead, which is better).

domyjobforme tag

[EmagGeek]

I love it! Haha... that's probably one of the better tags I've seen.

Re:domyjobforme tag

[Anonymous Coward]

Too easily thrown around if you ask me. He's not looking for anyone to set it up, he just wants some options. Isn't that what community is about?

BitTorrent

[Inf0phreak]

I'd say BitTorrent -- with firewall rules or some other measure so random people can't see your microscopic swarm. It uses SHA-1 hashes of chunks, so if a torrent client says a file downloaded successfully it's pretty much guaranteed to be true.

rsync should do the trick

[bacchu_anjan]

hi there,

    why don't you get cygwin on both the systems and then do a rsync ?

    between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).

BR,
~A

Line endings!

[sys.stdout.write]

they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals

Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).

Use binary mode and you'll be fine.

Re:TCP?

[Anonymous Coward]

TCP has timeouts. The FTP client and server probably have timeouts. Eventually, some bit of the system will decide the operation is taking too long and give up. The FTP client is probably reporting an error, but if it's driven by a poor script no-one will know.

Re:TCP?

[AvitarX]

I bet it is file systems with different block sizes rounding slightly differently, and an OP that does not understand.

Re:TCP?

[mini me]

FTP, while in ASCII mode, can try to translate line endings. If the carriage returns were removed, in order to be UNIX compatible, the file size would have been reduced.

Most FTP clients allow the enabling of a binary mode which prevents the conversion from happening.

Re:UDP.

[cguillemette]

UDP is bad if you want to make sure you send/receive every single bit. It is used for streaming of music, video and gaming. TCP should work fine.

You're kidding, aren't you??

[ballyhoo]

You are kidding about this, aren't you?

Let me get the facts straight:

- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.

If this is the case, you don't have a technical problem on your hands; you have a political one.

"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.

If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.

If your management doesn't do anything about this, then the files are not "mission critical".

Re:You're kidding, aren't you??

[Anonymous Coward]

Well, the original post never said whose mission. It could be that the employer with the "mission critical" need does not control the network or the computer choices. So, they charged the original poster with fulfilling the mission in spite of the given handicaps.

Now anything that gets my employer paid is "mission critical." I may be (and often am) at the mercy of my customers equipment. But my job is to work through or around those obstacles. Because removing them properly is not an option.

Re:You're kidding, aren't you??

[Dravik]

Mission critical means that you need to get it done even if someone else isn't getting their job done. Standing around in a huff and stomping your feet means that the mission critical information isn't getting moved. What he needs to do is find a way to accomplish his mission despite the difficulties, and then document the problems so they can be addressed.

Re:You're kidding, aren't you??

[eap]

I think you missed the part about the government being involved

Re:You're kidding, aren't you??

[Anonymous Coward]

I think this is a technical problem unless you are working using COVER YOUR ASS engineering.

You can start working under the assumption that a new line is cost prohibitive or impossible, (more expensive than a week of salary for a engineer, or maybe they are replacing the OS of the voyager 2). Also the problem sounds very simple so i don't understand why to make that much of a hassle.

Remember you are paid to solve problems, not to bitch about them.

Re:You're kidding, aren't you??

[Anonymous Coward]

Hmmm.... Here's a thought? What if the "Mission" is something related to a some sort of military activity and the data that is "Critical" is held in a location with spotty connectivity and severely constrained bandwidth (say a beach position with a little satellite dish). I am sure your hand washing memo will be very well received by all those folks who are put in harms way because of your willingness to make it someone else's problem.

You simply cannot assume away the problem or try and fit the problem to meet your expectations. Microsoft and IBM have made a reputation on trying this approach, but reality has this nasty habit of ultimately dictating the terms of the problem in spite of whatever massive resources are expended to the contrary.

Technical work arounds are the nature of the beast in the "real world". Perhaps a dose of exposure there will give you a little more empathy and perspective from which to approach these types of issues in the future.

I don't feel like registering at the moment, so I guess I will go with Anonymous Coward for now... (Tom for short) :')

Re:TCP?

[rezalas]

With the network being as questionable as stated, I can only wonder what part of the network is causing it to be unreliable. Usually if the entire network as issues then you are probably talking about everything in the office coming back to a switch panel and a faulty switch. If only certain transfers from point to point are commonly failing then you probably have wiring issues. In either a hardware or medium case, you need to be fixing the network instead of finding workarounds. Working with the network Admin for the facility to detect the source of the issue should be a two - three hour task at most. Save yourself time in the future and spend the bulk time now to fix the real problem.

Re:TCP?

[SnarfQuest]

Binary verses text mode?
Lousy windows file system screwing up on one or the other end.
Sparse files.
Windows "fixing" the data during transmission.
Loss of packets, and no error checking.
Windows.

SSHFS

[cenc]

I use sshfs file mounts for all office document file sharing and such, not just one time transfers. SSH encryption security, with the ability to open and edit files over the network. No goofing around with samba or windows file sharing. Regardless, some sort of ssh or sftp at least.

Not sure about getting it to work on windows, but there should be some options.

Re:Any encrypted transmission protocol actually

[link-error]

Wrong. FTP has a binary mode. This is probably the reason his files are missing several k at the destination. Sending a binary file in ascii mode is the ONLY TIME I've ever had a file not transfer entirely/correctly using FTP. Unless of course there is a network error/timeout, etc, but the FTP client always errored out in those cases. Using SFTP over an already secure network will only slow things down greatly.

Z-Modem FTW!

[Cytotoxic]

Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!

Z-modem is your answer.

Re:UDP.

[TheTurtlesMoves]

Well I find it nice to have a tar.bz2 file in the same order, and all of it. So you need to add "sequence" numbers and some form of ACKs in there. All you are doing is moving this function to the application rather than leaving in the stack.

TCP does this pretty well on 99% of the internet *and* the internet is aware of TCP. Its only very "different" connections that things can make a real difference. AKA the microwave link. Though we have wrapped the link with specific hardware/software layer that lets IP work well over it in our cases.

Also when people "roll" their own "superior" UDP transfer protocol, many don't bother to check why TCP does what it does. Flow control is *needed* with ACKs and resends --well any connection, buffers are not infinitely big. There is the 2 generals problem etc. Its not as strait forward as many think.