Slashdot Mirror

You you always use a UDP solution such as Aspera. Fast transfer speeds, bandwidth management and they have a specific AWS implimentation.

Other options to look at include Smartjog, whose new Bolt product looks quite interesting, Riverbed's Steelhead product, Filecatalyst and Signiant.

There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.

You you always use a UDP solution such as Aspera. Fast transfer speeds, bandwidth management and they have a specific AWS implimentation.

Other options to look at include Smartjog, whose new Bolt product looks quite interesting, Riverbed's Steelhead product, Filecatalyst and Signiant.

There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.

No, TCP is the protocol to use if you're moving video because you want to do an accurate transmission of the data and adding error checking to UDP is silly when there's a protocol that does it out of the box.

TCP is silly because the TCP congestion avoidance algorithm is very inefficient, and it leads to exceedingly slow file transfers especially over connections with high latency (say LA to London). The TCP bandwidth-delay product limit only depends round trip time, and can slow you down far below the actual amount of bandwidth you have actually purchased.

In Hollywood, everyone moves large video files using UDP file transfer protocols such Aspera and Signiant.

Apparently Aspera is working on a solution for S3

No, TCP is the protocol to use if you're moving video because you want to do an accurate transmission of the data and adding error checking to UDP is silly when there's a protocol that does it out of the box.

TCP is silly because the TCP congestion avoidance algorithm is very inefficient, and it leads to exceedingly slow file transfers especially over connections with high latency (say LA to London). The TCP bandwidth-delay product limit only depends round trip time, and can slow you down far below the actual amount of bandwidth you have actually purchased.

In Hollywood, everyone moves large video files using UDP file transfer protocols such Aspera and Signiant.

Apparently Aspera is working on a solution for S3

Although they offer FTP access to the genomic data--including population, alignment and sequences (traces, calls, etc.)--the NCBI has hosted the files with a README and guide (aspera_transfer_guide.pdf) about Aspera's "fasp technology" that the NCBI claims to incorporate automated checksum verification for both casual downloaders, via a browser plugin, and bulk downloaders, via a cross-platform command-line application. Aspera is new to me; they claim to have some throughput (bandwidth) advantages as well.

Nevertheless, the sequence data files embed MD5 checksums directly, per NCBI documentation, which I would expect bulk downloaders to take advantage of independent of any third-party "technology."

Here are the UDP-based file transfer systems that I prefer:

Kencast has been the leader in multicast IP satellite file transfer (where all kinds of weird things can happen in Ku band), now they have a system called BlazeBand built for point-to-point IP connections that used their FAZZT Forward Error Correction technology, validation algorithm, and missed packet collector algorithms. I've used FAZZT over satellite, but haven't tried BlazeBand yet.

Aspera is also widely used as a point-to-point UDP file transfer system in the entertainment industry. I've seen it used to move large video files for network television programming.

If you're stuck on windows, have no control of the link, need to guarantee delivery, need encryption options, are stuck in specific delivery timeframes, and need it scriptable with triggering etc. there's quite a few commercial options.

The 2 I've used the most are the following:

Aspera FASP - http://www.asperasoft.com/
Sterling Connect:Direct http://www.sterlingcommerce.com/products/managed-file-transfer/connect-direct/

Connect direct is geared more towards highly distributed delivery channels, where you've got lots of ingress/egress points to different locations (think multi site, multi customer distribution) so it could be a bit overkill, but aspera is great at utilizing a link to it's peak (it'll udp flood a pipe and guarantee delivery), in fact if you don't watch what you're doing you can pretty much storm everyone else off the link, but your data will get there >;)

UDP is actually a great basis for accelerated file transfer. Several file transfer utilities / protocols have been built around it. I deal with really large files, but I have been using Aspera on several projects with great success. Worth a look.

http://www.asperasoft.com/

Link.

Like most things, IP acceleration is better done in software. You can get all the benefit claimed for special routers or kernel drivers in user-level software by using UDP and controlling the packet injection rate and retransmission yourself. It's not even very hard to code -- unless you also want things like automatic back-off to allow incidental TCP traffic to share the line, automatic bandwidth measurement, parallel transfers (e.g. a dozen machines sharing the work to fill a 10Gbit link), file streamlining (to send whole directory trees with no delay between files), database-logged fully-acknowledged transfers, secure encryption, reliable checksumming, partial-transfer resumption, cooperative scheduled variable bandwidth limits, portability to whatever the people on the other end run, web-server and web-browser integration, GUI- or automated control and monitoring, and/or support. Then you're probably better off using a commercial product. As it happens, the movie and music studios feel that way, and they have mostly settled on the software linked from the page above.