Multi-Threaded SSH/SCP

neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."

Comments

Alternative solution for a trusted LAN

[Diomidis Spinellis]

If you want to speed up transfers and you're working on a LAN you trust (i.e. you don't worry about the integrity and confidentiality of the data passing through it), you can dramatically increase throughput using socketpipe. Although the initial socketpipe communication setup is performed through client-server intermediaries such as ssh(1), the communication channel that socketpipe establishes is a direct socket connection between the local and the remote commands. This eliminates not only the encryption/description overhead, but also the copying between your processes and ssh or rsh.

To *have* such problems...

[pla]

the crypto can sometimes be the bottleneck instead of the wire speed.

Between two devices on my gigabit home LAN, the CPU barely even registers while SCP'ing a large file (and that with every CPU-expensive protocol option turned on, including compression). What sort of connection do these guys have, that the CPU overhead of en/decryption throttles the transfer???


Coming next week: SSH compromised via a thread injection attack, thanks to a "feature" that only benefits those of us running our own undersea fiber.

Re:To *have* such problems...

[totally bogus dude]

Have you measured your actual throughput on the file transfer? It tends to take a crapload of tuning to get anywhere near saturating gigabit, even if you're not using encrypted transfers.

I wrote the bit below which I'll keep because it might be interesting to someone, but dm(Hannu) already mentioned the claw flaw in the logic behind the PP and article summary: if the CPU is the bottleneck, how could adding more threads possibly help?

Just for a laugh I used scp to copy a 512 MB file from my file server to itself, an Athlon 3700+ running at 2.2ghz. I got about 18 megabytes / second out of it. I took a snapshot of top's output right at the end (97% complete) and the CPU usage was as follows:

ssh: 48.6%
sshd: 44.9%
scp: 3.7%
scp: 1.3%
pdflush: 0.7%

So this system was pretty much pegged by this copy operation, and it achieved less than a fifth the capacity of a gigabit network link. Obviously the system is capable of transferring data much faster than this; the source was a RAID-5 set of 5 new 500 GB drives, and the destination was a stripe across two old 40 GB drives. I'd also repeated the experiment a few times (and this was the fastest transfer I got) so it's likely the source file was cached, too.

I do agree that there's probably more interesting and useful things to optimise (and make easy to optimise) than scp's speed, but I know for sure that scp'ing iso images to our ESX servers in a crapload slower than using Veeam's copy utility or the upload facility in the new version of Infrastructure Client (at least I think it's new, never noticed it before).

Re:Must be why rsync over ssh is much faster

[drinkypoo]

If you just want to copy some files from system to system in an encrypted fashion, then the BEST option by far is to use tar, and pipe it through ssh like so:

tar cvfpz - * | ssh user@host '( cd /destination ; tar xvfpz - )

This example will compress and encrypt your data before sending it; on the other end, the file is streamed to tar. This example requires GNU rar or a close facsimile.

Now, if you want to UPDATE a directory, use rsync:

rsync -av -e ssh * user@host:/destination/

Because rsync will do partial checksums and send parts even of BINARY files if the whole file has not changed, and doesn't re-send unchanged files, rsync makes sense when updating a directory. But it provides no speedup benefit over using tar, and in fact the directory scans it does before the sync mean that it may actually be slower.

Use scp only for copying single files, because you're right, scp chokes between each file.