Slashdot Mirror
Guaranteed Transmission Protocols For Windows?
Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."
Clearly you're looking for UDP. Next question.
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
SFTP should do since the communications are encrypted, if something changes along the way it should be rejected by the other end. HTTPS and any other protocol-over-SSL should do.
FTP is a plain-text protocol so if something changes along the way it won't give you any issues.
Custom electronics and digital signage for your business: www.evcircuits.com
Or I guess that would be WWCP. WWJD?
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
Jesus is awesome.
The summary states that with FTP, the downloaded files were of the wrong size. Can anyone explain why TCP's efforts to to deal with unreliable networks, such as the retransmission of unacknowledged packets and their reassembly in proper order, would not already deal with this? I am familiar with the concepts involved but I think I lack the low-level understanding of how you would get the kind of results the story is reporting.
It is a miracle that curiosity survives formal education. - Einstein
Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx
W
Background Intelligent Transfer Service (BITS) can be used to transfer files between windows servers. It is the technology behind Windows Update. We use it in our company to transfer files across a low bandwidth sattelite connection. Great thing is that it can automatically resume transfer after rebooting both machines. SharpBits offer a nice .NET API. You can find it here: http://www.codeplex.com/sharpbits
I love it! Haha... that's probably one of the better tags I've seen.
I'd say BitTorrent -- with firewall rules or some other measure so random people can't see your microscopic swarm. It uses SHA-1 hashes of chunks, so if a torrent client says a file downloaded successfully it's pretty much guaranteed to be true.
________
Entranced by anime since late summer 2001 and loving it ^_^
hi there,
why don't you get cygwin on both the systems and then do a rsync ?
between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).
BR,
~A
Should work fine across a WAN and then just file-copy.
Wasn't TCP designed for just this? Guaranteed transmission?
Similes are like metaphors
they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals
Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).
Use binary mode and you'll be fine.
it's not just for Linux.
You can't legislate goodness. Let each to his own destiny, by will of his freely made choices.
Rsync over ssh and then a script to md5 at source and destination.
The last part may be tricky and/or slow depending on your filesize, but it will do the job for free.
NO SIG
It's crazy but it just might work. Not very quickly though.
There are no guarantees when it comes to the protocols and the internet .... it is always a "best effort" system. Many forget that it is always a best effort system because the internet has come to the point where for all intents and purposes, there are virtually no failures. I would probably use a tried and true protocol like FTP or maybe even SCP. Both work very well. I would think your best bet is to try to work with the government to improve their "dodgy" internal network. SCP has the advantage of securing the transmission as well as excellent error correction and recovery.
I've used WinSCP before, and it seemed to be a good product.
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
Probably tape drives, or hard drives if you prefer. Encrypt with a shared key. I think microwave is LOS already, so your distances can't be that large. It would certainly solve your "flaky" bandwidth and security considerations. You would "packetize" the data, eg: tapes are brought over in serial succession; if a tape went missing, you delete the key that encrypted it's contents and request a resend of the contents of that tape. That verifies it's receipt.
Not sexy, but it's probably the best solution. Since you're a government contractor, I'll now insult you to suggest that you need a project for which you can charge a lot more money, like a carrier pigeon training program, including pigeon consultants, a pigeon breeding program, and a pigeon habitat designer. But that's what you get for asking Slashdot to do your job for you, especially one with an obvious, non-sexy, non-technical solution.
--
$tar -xvf
"Guaranteed ... mission critical files ... microwave connection ... government-run ... very dodgy internal network"
A transactional store and source integrity verification at the destination point. Or something in between that and what you have now, depending on your requirements. I don't know of a tool that does that out of the box though.
... is what you want. Yes, you can use it with Windows (with or without cygwin bloat). Use -c and a short --timeout and you're good to go. If you're using it over ssh you're looking at three layers of integrity (rsync checksums, ssh and TCP), two of them quite strong even against malicious attacks not only against normal stuff. Put it in a script with a short --timeout; if anything is wrong with the link your ssh session will freeze completely, as soon as your --timeout is reached rsync will die and your script can respawn a new one (which will resume the transfer using whatever chunks with good checksum you have already transfered and will again checksum the whole file when it finishes).
Line ending was my first thought too. I've used FTP scripts in Windows to and from *NIX machines with no trouble at all. I can't vouch for how well it works for Windows-Windows transfers because in that case I've always just used shared folders. That worked fine too. Unless the data is sensitive, there's really no need for scp or anything fancy.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
Take an MD5 hash of the data or something, then send it. If it comes back changed, you've got data loss. If it comes back the same, and the files are still a few kb smaller, then either you're the Wizard of File Hashes or you're reading off on-disk size instead of actual data size.
No kidding!!! What do you say at this point?
Ritchie's Law - assume you have screwed something up *first*, before blaming the tool...
ZMODEM!!!!
But seriously, some of the old file transfer protocols had some pretty nifty abilities, like restarting downloads.
The world's burning. Moped Jesus spotted on I50. Details at 11.
DVD burner, FedEx.
--- Worst tagline ever.
I too, am a bit surprised that FTP is failing. It has been my experience that if there are network problems the transfer may slow to a crawl, but unless the network is dropping 10% of the packets, I would be surprised if it failed. Have you tried FTPing the same file back and see if there is really stuff missing, or if it is just technical differences in storage of the file? As an aside - some of the old modem protocols might work for this. The problem is likely the microwave connection coming and going. I have seen MW drop in and out - it's madening. FTP will definitely fail in that scenario. You could also write your own little protocol that breaks the file into small pieces and transfers a chunk at a time and wait for an ack checksum. If the connection is interupted, automatically stop and try to reconnect - then resume. You're reinventing the wheel, but then you know exactly how the process works.
Why don't you try rsync. That should do the trick nicely.
Think of this transfer model like a car, the further it goes, the more bytes are burned up. they just need to be added back in with a network filling station. I would look to google for a government approved provider.
Also look into Windows DFS.
We use it to sync webfarm filesystems in Server 2008 and it works perfectly. At least in 2008 only file changes are sent across so it is very efficient, even for WAN scenarios.
Best regards
DVD = 1, Burner = 2, FedEx = 3
This is my sig.
The problem with FTP/SFTP/SCP is that if the connection breaks or theres other transfer errors (there will always be) and like he says, FTP does have the tendency to break files. You can solve this with checksums and transferring again, but thats not probably the most efficient solution. Something along the lines of torrent protocol would actually be perfect, as it has multiple good checksum tests and the file is partitioned, so you only need to retransfer one piece instead of whole file if its bad. Now torrent is a bit of pain in the ass as you need to create and transfer the .torrent file, but that could be automated and maybe theres other such more suitable protocols aswell.
Leechget is a cool program for this. I think it does data verification during or at the end of the download and supports pausing and resuming. It is primarily a download accelerator/manager but it also installs a right click context menu of "copy here using leechget" for all local file transfers. So go that over the network and you'll not only get it there correctly but it'll go at max speed because it opens multiple connections at once and sends parts of the file then re-joins them.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
You should look at the EDIINT AS2 protocol, AKA RFC 4130. This is a widely-used e-commerce protocol built over HTTP/S.
AS2 provides cryptographic signatures for authentification of the file at reception, non-repudiation and message delivery confirmation (if no confirmation is returned, the transfer is considered a failure), and is geared towards files. There is even an open-source implementation avaliable.
More complex than FTP/SFTP but entirely worth it if your data is mission-critical and/or confidential. Plus, passes through most networks because it is based on HTTP.
You're not old until regret takes the place of your dreams.
Even on reliable connections, using .complete files is a great idea.
It works this way: If you're pushing, open ftp, after ftp completes, you check remote filesize, if matches local file size, you also ftp a 0 size .complete file (or a $filename.complete file with md5 checksum, if you want to be extra paranoid).
Any app that reads that file will first check if .complete file is there.
If remote file size is less, you resume upload. If remove filesize is more than local, you wipe out remote file and restart.
Same idea for the reverse side (if you're pulling the file, instead of pushing).
You can also setup scripts to run every 5 minutes, and only stop retrying once .complete file is written (or read).
Note that the above would work even if the connection was interrupted and restarted a dozen times during the transmission. [we use this in $bigcorp to transfer hundreds of gigs of financial data per day... seems to work great; never had to care for maintenance windows, 'cause in the end, the file will get there anyway (scripts won't stop trying until data is there)].
"If anything can go wrong, it will." - Murphy
How about creating SHA1 checksum and then transferring data using netcat? You could split files in pieces then run them though sha1 and finally send over netcat using udp and retransmit at will. Or if files don't change too much you could try rsync.
This is all unix-centric solutions, so you'd have to install cygwin, unless there exists a python library that does all that.
Back in the very old days, we had slow modems with noisy lines. We used thinks like Zmodem and other things to handle this problem. It might just be the thing that will work now to solve your problem.
mission critical files ... microwave connection ... government
They use FTP? Hopefully only through ipsec or something.
I used to have a similar problem over another connection, where even more advanced file copy utilities would say the file was copied, but a 2-4k chunk would be missing. What I did to solve the problem was to use an archiving utility that supported adding ECC records and install it on both endpoints. Then, I'd just archive the files I need, send them over the faulty link, and usually the ECC records were able to correct any errors that did crop up during the transfer when extracted on the destination machine.
I did this manually, but I don't think it would be too difficult to make a scheduled task that would check for files, use the command line of an archive utility to generate a temp archive, sling the archive across, then the machine on the other side of the link extract the files and if the corruption was too great for the ECC records in the archive, to give some type of warning or notice to someone.
Of course, this is not fixing anything on the network layer, so maybe running either PPP over SSH or a VPN link directly from one machine to another might help.
'hash' is also a nice feature...
I telecommute and need to reliably get install images from my office down to my desktop. I have used GatherBird's Copy Large Files utility for several years, and it has worked out very well. No problems. http://www.gatherbird.com/GBMain.aspx
If you are also utilizing SQL Server (2005/2008), one option would be Service Broker, as it has guaranteed message delivery. And as long as at least one of the endpoints is using the Standard Edition or better, the rest can use the free express version.
Setup a linux box on the same network next to the windows box that is at the 'remote' end of the transfer (eg, not the end the transfer is initiated from).
Use ssh from the 'local' end to transfer the file to the linux box. Then run something appropriate (ftpd? apache? samba?) on the linux box that makes the files directly available to the windows box.
Alternatively, rip the Windows crap out and replace both ends with a real OS.
Establish a openvpn tunnel over UDP. All network traffic tunneled through it, will be encrypted and with integrity ensured. If your wireless network link is too unstable, you will have plenty of dis/re-connections of openvpn but communications going through the tunnel will be intact.
For example, one day I connected home over openvpn and used nfs to transfer a couple of big files over a slow wireless link. After some time going on, I closed my laptop and put it into suspend and moved into another place and connected to another wireless link. After resuming the laptop, openvpn re-established the connection to home and nfs continued to copy the file.
I would reckon that something based on the Bittorrent protocol (or a subset of it) might be an exceptionally reliable way of, while running in the background, sending files from one machine to another one.
The protocol comes with built-in file splitting/recombination, block validation and you can get several GUIs (and I believe at least one command line implementation) for it. It might be a bit overkill though - pretty much everything in the protocol related to dealing with managing communications with multiple peers (most of the protocol really) is unlikely to be useful in your situation.
That said, your own private tracker listing one and only one peer all the time (the machine where the files are being read from) might be the only tricky bit you need to do to use any bittorrent implementation out there.
Xcopy /v over a network share? using the at command?
Does your file have to be transferred in-synchro?
Otherwise you might want to look into Message Oriented Middleware, things like MQ Series or in worst case scenario, even Microsoft MQ. There are plenty of options.
This would allow you to put policies on the messages, handle routing (in case you need to deliver to different recipients), guarantee delivery at least once, do type conversions/transformations etc.
http://www.columbia.edu/kermit/
It was designed to work under the worst conditions and with any type of machine on the planet.
It is old but still in use so it probably works really well.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
You are kidding about this, aren't you?
Let me get the facts straight:
- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.
If this is the case, you don't have a technical problem on your hands; you have a political one.
"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.
If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.
If your management doesn't do anything about this, then the files are not "mission critical".
If the files are often the wrong size, it's probably something like CR/LF translation, or shifting reported file size due to filesystem block size. What do you see if you compare the files? Do they have the same checksum on both systems? My guess is that you don't have a tools problem (FTP, lame as it is, is probably not having as many transmission errors as you describe).
In terms of file copying tools, the options that come to mind are:
FTP doesn't check data integrity, but it's fairly reliable nonetheless, due to the TCP layer, which does retransmission, etc. But if there are transmission errors you may not be notified (i.e. you won't know if there's corruption, but you will be told if the connection dropped. So don't use it.
SCP does check data integrity, so if there's a problem in transmission it'll detect it and tell you, but it won't correct the error - you'll have to script re-sending if there's an error.
rsync checks the integrity of each delivered file, which is great. That being said, rsync is really designed to duplicate a directory tree from one machine to another, not copy individual files, so you may need to play with it a bit to do what you want.
Pando (http://www.pando.com) gives you the above plus data validation and retransmission, which might help in your situation. I've used it when sending very large files over unreliable connections (e.g. dial, weak wireless) and it'll keep hammering away until the data gets through. It's primarily a GUI app (Windows, Mac, Linux) so it's more of a consumer tool, but it can be scripted, etc., so depending on your application it might be what you want.
Enable 3D printed prosthetics!
that this is an issue with the size of your hash... I think.
Check the message boards for documentation. Others have had this same issue.
It's not FTP that's the problem. It's the implementation. Try others.
Switching to a new protocol, especially one like UDP that has no built in error correction is going to be complicated and costly.
Especially if you can fix the issue with FTP; which, I think you can.
This signature has Super Cow Powers
Create a small set of PAR2 files. At the destination compare it. If it needs repair, then get the associated repair blocks. This solves 2 problems...
1. Don't have to retransmit a corrupted file. What's the good in knowing it's bad if you'll have to redownload it again.
2. Fixes the errors for minimal bandwidth. Due to the fact you mentioned the bandwidth limitation, I assume this is a major obstacle to just resending the file again when it's corrupt.
At SFTP and SCP if there is noise or the connection fails (what will have exactly the same appearence) you'll end with no file at all at the receiving side, not a corrupted file. FTP will create a corrupted file.
Anyway, from what I've read, he's looking for rsync.
Rethinking email
I had a similar issue once, and kicked around the idea of making it database-centric instead of file centric and use something like ODBC. Files are first "parsed" into a database format and un-parsed back into files at the other end. The downside is that it may not work well for open-ended documents, but is primarily designed for CSV-like (delimited) or fixed-column ascii data files. The plan's schema somewhat resembled:
// for internal processing // foreign key // sequence in file
table: lines
-----
line_ID
file_Ref
line_sequence
line_text
line_check_sum
table: files // or name
// first transfer pass
------
file_ID
file_check_sum
line_count
After a transmission session, the missing lines can be known using the line count, sequence number, and check-sums. Check-sum is merely the sum of all ascii character ordinal values. The algorithm is roughly:
save(requestFileInfo());
save(requestFileContent());
while (lineList = determineBadLines()) {
for lineID = each in lineList {
statusMessage("retrieving line " . lineID);
requestResendLine(lineID);
}
}
statusMessage("done.");
You may also need a file info check-sum(s) to make sure the file list is good. You can try it without a database, but it's hard to get "random access" to the problem lines without it.
Table-ized A.I.
Give you reliable connectionless (UDP) or connected (TCP) options respectively.
you had me at #!
If you're stuck on windows, have no control of the link, need to guarantee delivery, need encryption options, are stuck in specific delivery timeframes, and need it scriptable with triggering etc. there's quite a few commercial options.
The 2 I've used the most are the following:
Aspera FASP - http://www.asperasoft.com/
Sterling Connect:Direct http://www.sterlingcommerce.com/products/managed-file-transfer/connect-direct/
Connect direct is geared more towards highly distributed delivery channels, where you've got lots of ingress/egress points to different locations (think multi site, multi customer distribution) so it could be a bit overkill, but aspera is great at utilizing a link to it's peak (it'll udp flood a pipe and guarantee delivery), in fact if you don't watch what you're doing you can pretty much storm everyone else off the link, but your data will get there >;)
01:36AM up 426 days, 2:46, 1 user, load average: 0.14, 0.11, 0.05
I know I'm late on this, but I haven't seen it mentioned yet. Doubletake is fantastic, and I've yet to have it fail me when moving stuff from point a to b on Windows systems. You didn't mention cost as a factor in the summary but I should note that it is most certainly not free, but if you have to make sure it gets there it may be worth it.
First off, I second the recommendation for rsync. This is what it's for. It's also easy to do over a SSH tunnel or similar.
As far as your size discrepancy goes, it's possible that such a small variation could be accounted for by different block sizes on the source and destination volumes. The data may in fact be identical, but the two different volumes may need to allocate a slightly different amount of disc space to store it.
rsync does hash checks as part of its normal operation, but if you really wanted to be methodical, you could have a script rsync to three different directories and then compare hashes on all copies. This may or may not be overkill, depending on your mission.
It sounds like there's a good bit of work to be done.
The first would be to address the problems with the internal network. There's no excuse to have problems on a LAN. Upgrade away from hubs and consumer grade "switches". This is an easy and affordable fix. There are plenty of old Cisco switches available on eBay for very little money.
I upgraded an office with 6 Cisco Catalyst 2924-M-XL-EN and 6 WS-X2924-XL-V 100baseFX fiber modules cost $300 from eBay. I used fiber to connect all the suites, which cost about $150. $450 got rid of all their problems that the entire staff had been complaining about for years.
I'd also recommend having a look at the radio link. Why isn't it working properly? Is it a line of sight problem (a tree grew in the way, maybe?), a signal or interference problem that could be resolved with a better antenna or reconfiguration?
For a reliable protocol, you can use rsync and rsyncd. I use this between Windows clients and Linux backup server. rsync provides a very reliable protocol, ensuring the data was received correctly or retrying. FTP isn't a good protocol for mission critical data.
Myself, depending on the link, what's on each end, etc, you may consider putting a Linux machine at each site, and doing a PPP over SSH link. It's easy, free, and very reliable. :) It doesn't take reinventing the wheel, nor playing with VPN servers and clients. rsync over this connection will work very well, and due to compression and encryption by both rsync and ssh, you'll find that it ends up being faster. That may seem counter-intuitive, but I've found it to be true in reality.
For a while, I was on an unreliable home connection, which apparently did a good bit of traffic shaping and some port blocking (ahh, gotta love residential providers). I frequently saw packet loss. Once setting up a PPP over SSH connection, I routed every machine on my home LAN through it, and all of our traffic left through one of my datacenters. Despite adding this extra route and hops, all Internet traffic from home was faster. Because of the magic of SSH, it actually got rid of all my packet loss. :) It was still there, but SSH retried to keep the tunnel working with no losses. Since the provider couldn't see anything that was going over the line other than encrypted data, they couldn't do any sort of traffic shaping. I used an non-standard port number, so they couldn't even be sure of what the encrypted data was. I had already planned on changing ports, if they should start slowing down my traffic, but they never did. It was unidentifiable, so they left it alone. For several things, I enjoyed 10x the normal speed, just because of the PPP over SSH tunnel. Doing the same transfers via HTTP or FTP over the same route (but not over the SSH connection) resulted in very slow speeds.
Serious? Seriousness is well above my pay grade.
Take a note from the newsgroup file sharers. If it's good enough to share GB+ data on an old school discussion forum, it's good enough for your business.
1) Split the file into tiny rars, 1-10MB in size.
2) PAR2 the files with a decent amount of redundancy.
3) Send. Resume any broken files
4) Fix.
If you really wanted to get fancy. Run a news server on each end of the line. Upload files locally to the newsgroup and then grab them from the other end. The pirate community has a ton of tools for making this as painless as possible. (I heart hellanzb)
It's built in and it works. You can get something that's theoretically more efficient with a lot of work, but this is quick, cheap, and simple. Best of all, it actually works.
The client and server each run a checksum against what's on either end after a "successful" transfer. If the xcrc fails, delete and re-send. It's really that easy. If efficiency is an issue, just enable resuming on the client end.
What a simple question. Your problem is solved with about 20 minutes of work/setup/testing and a budget equal to whatever you get paid for 20 minutes worth of work.
-- "Government is the great fiction through which everybody endeavors to live at the expense of everybody else."
Kermit's fine, if you have all the time in the world on your hands to wait for the transfer to complete.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
I worked on a project that had similar deployment requirements, and we could that using Microsoft Message Queuing (MSMQ) as the transport mechanism took care of all these issues.
MSMQ itself only provides the transport mechanism, and there's no front end interface to send files- you'd have to code something up. However, it's the best "guaranteed delivery" system that I've seen on the Microsoft platform. Persistent across reboots, security controlled, FIFO queuing, very robust.
You may not be looking to code something up, but if you are, have a look at that.
This same problem happened to us when I was doing "service". Let's just say that something had a computer with an access database, and the mdb had to be sent over a microwave connection to something else (that also had a computer) which took mdb files from many somethings and did some other things. Well, anyway, at first they we're using FTP, but I got to script a solution that zipped the file over several small ones (50MB each) then sent them over SCP, made a checksum on each file and then rebuilt the mdb file.
I never trust windows copy for large file transfers from disc to disc, on the same machine. One error and BAM! You're left with an inconsistent file system copy. The only RELIABLE way to resolve this is to break out RSYNC and have it finish the copy. But then one wonders, why not start with rsync in the first place. Unlike windows copy, you can have it ignore & LOG errors. Then when the large batch is done, fix the errors and repeat the same command, and RSYNC won't copy the already-copied information.
It is a total joy, aside from the command line!
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
Use MD5 or some other relatively strong checksum (CRC-32) too, just to be _absolutely_ sure that you didn't get any errors. It's a paranoia indulgence though if you are using TCP and rsync.
Absolute statements are never true
Maybe think outside the box for a second and really look at what you are asking...
You plan to transfer _mission critical files_ over an unreliable link. Maybe you should provide a _mission critical link_? You know, a leased line with a backup? If you can't justify the cost, it probably isn't mission critical.
The right tools for the job people... Jez!
GPLv2: I want my rights, I want my phone call! DRM: What use is a phone call, if you are unable to speak?
If you have access to the file at the remote end I would run it though QuickPar http://www.quickpar.org.uk/ at that point you can split the file into a group of smaller files (less chance of any one file being mangled), plus add ckecksum files to the set, that way if one of the files is mangled in transport it's 1: identifiable and 2. automatically fixed by reassembling the par files on your end of the connection.
... this isn't a file copying problem, but rather a verification problem. If a file sometimes winds a few KB short could it be that someone or some process at the destination grabbed the file before it finished copying? If that is what's happening then any file copying protocol that copies the file to a temp location and then links the file into place at completion would solve this problem.
Are you just looking at the file size on the disk, or have you done a data comparison?
If you are just looking at the size as they appear in windows by right clicking at looking a the properties, you are woefully under qualifed to do anything remotly like this and need training.
There is no reason Windows in and if itself can't do this with FTP.
What are you not telling us?
The Kruger Dunning explains most post on
On some level, there isn't much difference between an application and a protocol. In fact, if you ever take a networking theory course, you'll see that each protocol layer in the network stack is, in fact a "protocol machine" (i.e., an application), which does the little protocol dance that makes functions at that layer happen.
But I digress. What the user is running into here is a fundamental problem with TCP over lossy networks. It really was not designed with really lossy networks in mind. E.g., the congestion control mechanism in TCP ("exponential backoff") makes the assumption that there is a wire sitting there and that certain parameters (like bandwidth) are not going to change. If you need certain QoS guarantees on a wireless link, TCP may be hard-pressed to deliver, because TCP's [limited] QoS mechanisms may make the problem worse. There is a HUGE amount of overhead on 802.11 networks to make sure that TCP doesn't suck.
I don't know how this person's microwave link is configured, but they might be better served by thinking about the QoS guarantees in the various layers in their network stack. I know a previous poster was joking when they said UDP might be a good option, but look, part of the problem on wireless is TCP's retransmission mechanism. With UDP it is up to the user/application to ask for a retransmit. Bittorrent works exactly like this, so something like Bittorrent, where each small file chunk gets its own hash, and those hashes are checked upon receipt, might not be a bad idea. I like rsync as well (because it has a rolling checksum feature), but again, you have TCP in the mix, and if I recall correctly, rsync will not retry automatically on failures, which is what you want.
If you need guaranteed, fault tolerant, one-time message delivery you could use IBM's Websphere MQ. It is expensive bloaty, but it gets the job done and will tolerate intermittent connectivity problems. It runs on every platform and there are client API's for many languages.
Place an MQ server at each end, then have the client enqueue the message at one end and a listener dequeue at the other end. If the link or other host is down, the clients can still send messages which are stored and delivered when the comm. link comes back up. Transactions and rollback are supported, and large files are automatically segmented and reconstructed by the system.
You can configure it to send message receipts, prioritize messages, and report any failures to a dead letter queue.
There may be other brands of middleware that do this just as well for free. WMQ is just the one I'm most familiar with (I don't work for IBM).
FTP and TCP cannot "Drop" packets or bytes. You need to learn-up on TCP and FTP.
FTP _does_ translate DOS end-of-line sequences (carriage-return followed by new-line -- 2 bytes) with Unix end-of-line sequences (just new-line -- 1 byte). So your files may become shorter by as many bytes as they contain lines.
The solution is to tell FTP to not treat the file as text, but as binary image information in which new-line characters are treated with no special processing. Traditionally, FTP called this "file type I" and the command to set it is "bin" as in "binary":
C:\Documents and Settings\fred>ftp abc.net
Connected to abc.net.
220 ProFTPD 1.3.1 Server (ABC Global Enterprise Group) [10.13.131.34]
User (abc.net:(none)): freddy
331 Password required for freddy
Password:
230 User freddy logged in
ftp> bin
200 Type set to I
ftp>
Binary verses text mode?
1-to-the-01101
1101101
1001000
0101101!
Word!
Bow-ties are cool.
The transmission system is irrelevant. All that matters is that you know you have received whatever was sent.
Just make sure you send a checksum and that the received file matches.
oh wait... Windows scripting...
Deleted
With some sort of CRC checking going on...
Tsukasa: All I really want, is to be left alone...
Bittorrent?
Also, 2 mbit microwave wtf.
Run a cable for shit's sake.
Just get a big cannon/artillery gun/icbm and fire tapes of data, or flash drives even. Just stuff them into nerf balls. It'll work great, like a sneakernet but with explosions. :D
Use something like Netmotion, although it was designed for mobile wireless usage, primarily to maintain TCP session persistence while simultaneously providing an encrypted VPN, it works wonders at keeping a packet-loss-free connection over a lossy wireless network link too.
Interestingly, it uses a UDP stream thru which it tunnels it's encrypted protocol.
It's old but highly reliable and can be secured.
FTP rides over TCP so it isn't really possible to "lose" data. However the default FTP transfer mode for Windows is ASCII. This means that if you're transferring binary data that some might not make it through as expected.
Change the FTP transfer mode to binary for the transfers and you won't have a problem. The command is "bin" once you have the FTP client open (assuming you're in interactive mode).
I just handled a similar problem where I work and the solution was a pretty simple combination of using rsync over ssh.
Our bugs are smarter than your test scripts.
because microwave apparently is just as prone to loss.
http://www.rfc-editor.org/rfc/rfc1149.txt
.' Windows 98 crashed I am the blue screen of death no one hears your screams... `.
I would take this as a pretty serious suggestion. This is basically what Bittorrent was made for. Transmitting large files over slow connections with a high level of reliability and a low level of re-transmition should things go wrong.
Check out unison.
Directory synchronization over several protocols, brilliant include/exclude syntax, failure protection & rollback, rsync style 1-way or 2-way block synchronization and intelligent file change detection, Unix and Windows support, open source... It's what you need.
A government is a body of people notably ungoverned - AC
Robocopy is commonly used for this problem. Otherwise Cygwin and rsync.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
My sentiment exactly. It seems that you could use Torrents to share the files with more than one server for fail-over security.
Set up a BSD lpd queue under Cygwin, something like:
sendit:lp=/spool/null:sd=/spool:if=/spool/sendit.sh:sf:sh:mx#0:
Have the sendit.sh script do whatever it is you want with the file. To send a file: lpr -Psendit filename
Configuration of the network queue left as an exercise for the student. (Hint - queue pathnames locally.)
If it's a Windows-Windows, Robocopy is what you want. It will do everything you need. It's free too and made for Windows.
Have you checked that the differences in files sizes aren't due to the differences in cluster size?
You can verify content by simply comparing hash values. If you need more assurance, then Sterling Managed File Transfer, or SFTP.
If you are talking DoD, DoE, or anything above a confidential level... for the love of God, call the NSA and request to use Assured File Transfer (AFT) or some other managed system. They should help you with this.
[RIAA] says its concern is artists. That's true, in just the sense that a cattle rancher is concerned about its cattle.
Mod parent "-1 dull as dishwater and half as useful".
Infuriate left and right
Here are the UDP-based file transfer systems that I prefer:
Kencast has been the leader in multicast IP satellite file transfer (where all kinds of weird things can happen in Ku band), now they have a system called BlazeBand built for point-to-point IP connections that used their FAZZT Forward Error Correction technology, validation algorithm, and missed packet collector algorithms. I've used FAZZT over satellite, but haven't tried BlazeBand yet.
Aspera is also widely used as a point-to-point UDP file transfer system in the entertainment industry. I've seen it used to move large video files for network television programming.
Most modern networks expect a highly reliable data link with congestion as the primary concern, and are tuned accordingly. You have a highly unreliable data link with little concern about congestion (assuming the microwave link rarely if ever has more than one simultaneous user). The problem is that when a packet is corrupted or lost, the transmitter assumes a crowded pipe and starts slowing down, when you really want it to retransmit as soon as possible. Since it's doing the exact opposite of what's needed to fix the problem, it can compound on itself out of control, eventually causing it to just give up the file transfer altogether. Therefore, you want a short retransmission timeout for TCP, a longer timeout for FTP (or whatever better application layer many people have recommended you try), and a smaller MTU so there is a lower probability of any one given packet getting corrupted or lost.
Of course, you can go too far the other way too, and the ideal settings are going to be highly dependent on your individual network.
This space intentionally left blank.
Use any common P2P file-sharing system, preferably one that doesn't require a central server. Gnutella would work fine, I know people do this using Shareaza and Limewire, and probably any other of the clients will work. DC would work. I used WASTE for a while to do this. Probably some other P2P software would work, too. They almost all use hash checks to ensure accurate transmission. Set up a private file sharing network, using one folder on each end, and your files should get transferred 100% error-free between your computers. Should be fairly simple, just don't share C: :) The only downside I can see (outside of the IT people freaking out because you've got a file-sharing program on your computer) is that they're "pull" oriented, rather than "push", so the recipient has to retrieve the files. See also Friend-to-friend and Private P2P
Like others have said, find out why your network is breaking data in transmission and fixing that would be the more direct fix, but I'm guessing OP doesn't have control over that part.
I use sshfs file mounts for all office document file sharing and such, not just one time transfers. SSH encryption security, with the ability to open and edit files over the network. No goofing around with samba or windows file sharing. Regardless, some sort of ssh or sftp at least.
Not sure about getting it to work on windows, but there should be some options.
Living in Chile
The poster didn't say anything about scripting but I have noticed that the pro version of teracopy has a original/destination CRC value listed next to each file copied. I don't actually have the pro version and you'd have to get creative with script host (or whatever) to script it but this could very well fullfill your requirements.
Also, XCOPY has a "verify" flag...better than nothing I suppose...
"UNIX is very simple, it just needs a genius to understand its simplicity." -Dennis Ritchie
You can use Metalinks for downloads without errors. It can use whole file checksums, or partial file checksums to repair errors.
I would think that you would want something more secure that FTP or windows copy, even rsync should be across a secure connection.
I killed da wabbit -Elmer Fudd
Posted earlier AC...
http://www.blast.com/cgi-bin/blast.com/view_services.cgi?request=show_aisle_names&dept_id=1
As I stated earlier - guaranteed file transfers - if it says it's completed, the file is guaranteed to be there and be correct.
Who is general failure, and why is he reading my hard drive?
What you will want to do if you have a Windows machine on each end, is to secure each machine first. That would mean, over-writing the Windows operating system with a Unix/Unix-like system (Linux, BSD, Solaris, Hack-Mac). Then you have (most likely) built-in SFTP servers and clients. That is what I would do. Someone would eventually hack into your Windows system. That is the way Windows is, so that MS or an MS-partner can sell you the antidote to your virus woes (even after your cure your woes, they will come up with something another virus so as to sell you another cure to get your money.) Get security right from the start instead of an add-on. Go with Unix or a Unix clone.
This guy is dissin Microsoft. How dare he? FLAIMBAIT. RELGIOUS NUT! OUTDATED idiotic thinking! Throw chairs at him! Microsoft owns the WORLD. It is THE ONLY solution you should ever use. BAWHAWHAHAHAHAHAHAHAW!
Counter-Anti-Microsoft post paid for by Microsoft
a mission critical application under Windows... That in itself is the start of something going bad. TCP does guarantee transmission. The actual copy software will have to do some sort of MD5 check at both ends, in order to verify that content was copied properly. I would suggest you write your own copy script, any copy command would work. For government installation, I would suggest an encrypted copy process, scp, sftp, etc...
TOP DSLR Cameras Reviews of the top DSLRs
But seriously, some of the old file transfer protocols had some pretty nifty abilities, like restarting downloads.
Indeed.
I used ZModem not long ago on a project transferring data over a dodgy link. ZModem implementations are very small, quite fast, and they handle data integrity well. Also handy is the vast number of ready made tools which you can make use of. In my case, I could use ZModem through Minicom, or straight on the command line via sz/rz.
Finally, ZModem (and its predecessors X and YModem) are not that large or complex. A little bit of study and you'll have a good handle on how it works. When working with dodgy hardware or a flaky link (by which I mean home built, prototype boards, etc.), being able to follow along with your transfer protocol can help you isolate problems.
Elrond, Duke of URL
"This is the most fun I've had without being drenched in the blood of my enemies!"-Sam&Max
I'm sure the dodgy government network would love that....
RTFM is not a radio station.
You raise questions on various levels, so let me traverse the stack in reverse order.
Link: just how reliable is it? I assume quality is unpredictable and varies (for instance during mobile and in-theatre deployment), which suggests you need to check for transmission errors in pretty small windows and force a error retransmit ASAP (if you have that capability on your specific type of link).
Protocol: I won't question using TCP/IP, but I would suggest you may want to ensure you tune the stack to small window sizes, and use UDP as that appears to match your transmission quality. MANET could help as well as that's made for mobile use, but I don't really know anything about it - it just may be an option worth checking.
FTP and TX security: I'm not sure of how sensitive this data is, but a microwave link does have stray signal issues, and FTP transmits UID and password more or less in cleartext. IMHO not quite desirable, but it depends what you do. In addition, FTP defaults in Windows to ASCII mode which makes a mess of data that is not of Windows origin or is binary. You MUST set to "binary" mode first before you start transmission, which others have already mentioned. In addition, do some tests with checksummed data so when you find differences you can work out if it's your own interpretation or a real problem.
Personally, I'd grab the PuTTY set and run a SSH session. You can find a server at FreeSSH. Also brutally easy to automate - I expect you're not that much in need of employment that you must generate your own opportunities to watch paint dry :-).
Good luck...
Insert
Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!
Z-modem is your answer.
We've seen that too. We had a really slow connection to hosts in Singapore, and they were our only windows boxes. They set up cygwin and rsync, and we had crashes and incomplete files.
The answer in our case was to move those services to a real server and the not the crufty old former desktops that they were.
I'd rather boot those machines with LiveCDs and mount the disks read-only before cygwin/rsync on windows. What a pain.
-B
Ash and Hickory, straight-grained and true, make excellent bludgeons, dandy for the cudgeling of vegetarians.
I was once in a startup trying to deliver streaming video to tablets/laptops at sporting events via 802.11 wireless networks. Within 30-50 feet of an access point, it worked great, but any further than that and packet loss was too high to decode the video stream. Before our investors pulled the plug, we looked into technology from a company called Digital Fountain. They use a type of forward error correction called raptor codes to make data transmission high resilient to packet loss on unreliable networks. It's a commercial product, but I assume the OP's employer can afford it if they're transmitting "mission critical" files for the government.
Am I the only one to question whether there is a real problem? Did I read it wrong (ADD so it's possible)
What if the File allocation size is just different on the two drives? Then the exact same file can, and WILL be different sizes on disk, no harm, no foul, but maybe fowl in the way of the transmision route.
For instance, if you write a 6k file to a HDD w/ a 4096b allocation size, it's going to take up 8k(ish worth of space, despite the file being 6k.
Now if you write that same file to a system w/ 512b allocation size, that 6k file will take about 6k.
And it can get smaller by 4-xkb in a situation where a newer system with a larger allocation size is sending a file to a (usually) older system w/ a smaller allocation size.
Then again, maybe I failed math. but I thought that's why there's "Size" then "Size on disk"
How much is your data worth? Back it up now.
You need a Disruption Tolerant Network (DTN) solution, there is no commercial DTN's out there that I know off but the DAKNet people at MIT were working on something and a group a U of Waterloo has an implementation. http://www.firstmilesolutions.com/ http://blizzard.cs.uwaterloo.ca/tetherless/index.php/KioskNet
Now, let's assume that it's not an CR/LF problem but that instead for some unknown reason the ftp transfers get aborted and thus the file size mismatches.
Okay, first of all, if you want to guarantee that a file that departed from one system is the very same file after its arrival on another system it is not wise to use the file size for verification, as the two files could have the same length but different contents. Therefore typically md5sum is used. Or better yet, use both MD5 and SHA-1 hashes so nobody could probably ever produce meaningful collisions for both of them at the same time.
Now, what programs should be used for the transmission itself? Well, that depends on your requirements: Is confidentiality important or is it really just about integrity and availability? Is speed or link saturation a topic? Like, if your current pipe is like 80% full, you probably cannot afford to encrypt your data. Otherwise, of course you should except for like if an IPS/IDS maintainer wants to be able to scan the contents. Let's take a look at both possibilities:
Basically suitable is every tcp data transfer application that does by itself not meddle with the data itself. So this kind of excludes ftp as it can substitute CR/LF (Unix) line brakes with CR (Windows) ASCII text line brakes while transferring data from UNIX to windows and vice versa. But then again, you can use FTP just fine if used in binary mode. However, even the Swiss army knife of network transmissions can be easily used for the purpose of reliably transmitting files from A to B: netcat.
nc or nc.exe is available for both Windows and Unix and is often used in the forensics world in manual combination with md5 and/or sha-1 hashes to transmit forensic evidence from e.g. a suspect drive to the examiners workstation. Here the chain of evidence would be maintained by recording a hash of the data on the suspect drive, recording a hash of the data on the examiners workstation after arrival and recording the date, time and contents of the transmission. Note that it might be vital to have a log of what has been transferred when so that it can be proven that you sent some data the other party claims to never having received it. So, recapping, e.g. netcat, ftp, SMB/CIFS shares, HTTP and any other TCP based file transfer utility could be used. HTTP and FTP could even be easily scanned for viruses/malware during transit. UDP based file transfer utilities could be used as well as long as the implementation does take care of the integrity. As most likely a short script would be used in order to generate logs containing MD5 and SHA-1 hashes on both sides, the time and date of the transfer and the filename, this script could as well easily handle data retransfers in the case of packet loss.
Sorry, this posting by now bores me. So, the recap:
Use SSH (SCP), cryptcat (used among others in forensics for the chain of evidence when confidentiality is an issue), HTTPS, SMIME or any other encrypted transfer tool, really. Hell, you could even generate an encrypted PGP file or whatever with a script and pipe it through whatever data transfer application you want. (Like ftp in binary mode ;) )
So, overall, what are needed here are two small scripts that do something like this:
On the sending side:
10 compute SHA-1 / MD5 hash of a file to be transferred (and optionally compress it) 20 send file 30 receive a SHA-1 / MD5 hash of the transferred file from the receiver 40 compare the hashed 50 complete transaction including logging the date, time, filename and hash, if hashed match 60 else goto 20
On the receiving side: 10 receive
Mod parent up. WE used robocopy. Also, depending on how quick you need to do it, DFS? I've used DFS over some pretty fucking ordinary satellite links on the other side of the world, and whilst it may take its time, stuff eventually does get there, and i'm yet to run into any corruption from it....
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Bit Torrent protocol would be the simplest .
I'm not really a Windows guy, but I seem to remember server 2003 having a Distributed File System (DFS) that might work for this. I think 2k3R2 has an improved version that will only sync the changes to the files as well.
See XTP: http://en.wikipedia.org/wiki/Xpress_Transport_Protocol Devices that use XTP to help transfer files over satellite links exist. For example: Mentat/Packeteer/Blucoat's SkyX product. (Disclaimer: I used to work for Packeteer, though not on the SkyX product.)
Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals.
Are you sure you are using BINARY transfers? FTP allows "text" tranfers which can transform the CR+LF pair into 1 byte (CR I think). On large files, you could end-up with several KB missing. This transformation actually depends on client and server, not OS (although it was created because of Windows - Unix conventions).
Regarding what others have said, I don't see how TCP checksumming could affect the file unless a specific attack is made. Even then, it's hard to block the original TCP packet, and altering it will make the receiver transmit a "retransmission request". (I said hard, not impossible)
Whatever happened to ZModem? It's an amazingly resilient protocol that was used to great effect to transfer files over virtually any phone line, no matter how noisy. It accomplished this by an adaptive algorithm, whereby the block transfer size was either doubled or halved after every few blocks, depending on whether the previous blocks got through the line ok. Where's the equivalent in the modern internet world?? There are millions of people with dodgy or unreliable internet connections -- they would benefit from clients / servers that implemented a modern equivalent.
char*f="char*f=%c%s%c;main(){printf(f,34,f,34);}";main(){printf(f,34,f,34);}
http://www.samba.org/rsync/
Or something like it in Windows.
It will check the delivered files, retry chunks if needed.
Much better than just FTP
If it's truly mission critical (and if it is, it sounds like your mission is in real danger if you keep dropping bytes!), you could do worse than look at Connect:Direct from Sterling Software. It's the standard transmission software for bits of the core financial transaction world in the UK and with good reason.
Sure it's "only" a secure transmission and there's plenty of free alternatives, but this is one time when I would recommend paying out for the certainty you need... Others will no doubt disagree, but having used a variety of things for mission critical file transmission, C:D is a safe choice.
SFTP [...] SSL [...] FTP is a plain-text protocol
Wrong. FTP has a binary mode.
When we're talking in crypto-mode---as we are, made evident by the references to SFTP and SLL---the words "plain-text" refer not to ascii vs. base64 vs. bin2hex encoding, but whether the data is encrypted or not.
AFAIK, FTP doesn't have any provisions for encryption.
Your point is orthogonal to that, so your "Wrong" is wrong: your parent is not wrong. Your point is true, however: FTP does have issues with line endings etc.; rule of thumb: use binary for everything and recode newlines on the client side if need be.
Using SFTP over an already secure network will only slow things down greatly.
Sending data unencrypted through the air is not what I consider "secure". So, while true, this point is not particularly relevant (I think).
try the Windows command line FTP
I vote AYE on this! Everyone knows that BSD code is rock solid ;-)
TCP is so horrible. I wish HTTP used UDP by default so I wouldn't have the pro
Kidnapped by CJ too, I take it?
If only certain transfers from point to point are commonly failing then you probably have wiring issues. In either a hardware or medium case, you need to be fixing the network instead of finding workarounds.
Wiring issues? It's wireless. The summary stated that the physical layer is a microwave radio. Radio will always have noise. How many network admins have the domain knowledge to troubleshoot that?
Try opening a newlined file with notepad, for example.
Gedit on my Ubuntu laptop saves with LF newlines, and Windows Notepad can't read them because it expects CRLF newlines. But I add one to Notepad and all is well. In fact, pretty much every text editor but Windows Notepad can handle the differences between UNIX and Windows.
As for VMS, how much VMS is used in world-facing applications as of June 2009? Even HP, the owner of copyright in VMS after having bought Digital's parent company, uses HP-UX, Linux, or Windows Server on its popular public websites. Even HP's site about VMS was found to use HP-UX. Netcraft confirms it: VMS is dead.
And if FTP translates oddball operating systems' conventions for text/plain files, why doesn't it do so for image files (.ppm vs. .bmp), audio files (.au vs. .wav), or other MIME types?
This transformation actually depends on client and server, not OS (although it was created because of Windows - Unix conventions).
OK, I guess technically FTP isn't *quite* older than Windows, but I really don't think there were many people using Windows 1.0 in 1985. :)
Here's some of the operating systems in use at the time:
OS/360, VM/CMS, etcetera: IBM operating systems stored text as 80 column card images, padded with spaces.
VMS and RSX-11 used a one or two byte record length, an optional binary line number, an optional Fortran carriage control character, followed by the text, with null padding added when a line would overlap a block boundary.
CP/M used carriage return or carriage-return line-feed, depending on the application, with a ^Z character to indicate end of file because files only had block counts, not byte lengths.
They'd have KILLED to only have to worry about CRLF. :)
I'm assuming you want something scriptable, but as a regular GUI replacement for Windows' file copy stuff, TeraCopy (http://www.codesector.com/teracopy.php) is quite nice. It's sort of like a GUI version of Robocopy.
FYI, the 32-bit version integrates perfectly with the Windows shell, but the 64-bit version's integration was a complete pile the last time I tried it (a while ago). It didn't work automatically after the installation, and even the manual integration instructions didn't get it working. TeraCopy is much less useful when you specifically have to open the app and select the source and destination to start the copy.
Damn, I picked the wrong RFC.
FTP dates back to 1980. That's older than MS-DOS.
Terracopy is a great program for copying files inside windows. It does error checking and supports resume if your connection drops.
You can download the home edition at http://www.terracopy.com/ to give it a try.
I had issues with copying about 300gb of backups around different servers until I installed that, works great.
We have similar needs, only we're exchanging files across dodgy Internet connections (e.g., satellite links to sites in the developing world). Our requirements including operation over low-bandwidth connections and the ability to suspend and resume transfers. We settled on Windows Live Sync, since it works on Mac OS X in addition to Windows, and because it required no additional software development effort on our part. Had Live Sync not been available, we would have developed our own wrapper around BITS. Because BITS is an extension to HTTP, it degrades gracefully into something interoperable with non-Windows clients. (BITS would also work over a private network, but that wasn't a feature we required.)
I'm proud of my Northern Tibetian Heritage
... is not the tool you use to remove the creases from your shirts.
IANAL but write like a drunk one.
What do you suggest? That he introduces optical fiber to the middle of Nowhereville?
Some times you have to deal with less than ideal situations for your job, yo will certainly have to explain that you'll find constraints, and yes, sometimes you may have to say that something is not possible, but mission critical does not mean what you think it does....
IANAL but write like a drunk one.
Connect:Direct is what the banks use.
It is cross platform and relatively easy to implement.
IANAL but write like a drunk one.
I recently worked on a government program that had some of the same requirements that you describe. Lockheed-Martin proposed that they create a custom protocol to do this at the cost of about a zillon dollars. Some research turned up SCTP "http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol" which is now available in Windows. We were using an implementation for Solaris.
Dealing with a fixed-record length file structure is OS dependent
I can create fixed-record-length files in UNIX, Windows, Windows Mobile, or any other operating system that supports ANSI C. Writing a record looks something like the following (untested):
But after reading this file I'm starting to see how deep the rabbit hole goes. A "stream" text file under VMS is a familiar '\n'-delimited list of strings, but a "non-stream" text file is a list of 16-bit-aligned Pascal strings. Likewise, classic Mac OS had the '\r'-delimited list of strings (TEXT files and TEXT resources) and the list of Pascal strings (STR# resources), but I never thought of STR# as a "text file" as much as a "list of distinct strings".
It costs money, but buy the file transfer edition. You get reliable and encrypted data transmission.
Simple :)
If you ever drop your keys into a river of molten lava, let'em go, because, man, they're gone.
$ cygrunsrv -I cron -p /usr/sbin/cron -a -D
$ net start cron
Or... install some linux distro and get the real thing
Kermit is your friend. You could also use zmodem with sliding windows :)
Presenting results of an experiment is "flamebait"? Please.