Multi-Threaded SSH/SCP

neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."

Comments

Re:Alternative solution for a trusted LAN

[upside]

You can also use a cheaper cipher. From the ssh manpage:

-c blowfish|3des|des
                          Selects the cipher to use for encrypting the session. 3des is
                          used by default. It is believed to be secure. 3des (triple-des)
                          is an encrypt-decrypt-encrypt triple with three different keys.
                          blowfish is a fast block cipher, it appears very secure and is
                          much faster than 3des. des is only supported in the ssh client
                          for interoperability with legacy protocol 1 implementations that
                          do not support the 3des cipher. Its use is strongly discouraged
                          due to cryptographic weaknesses.

Re:To *have* such problems...

[dm(Hannu)]

They claim that the first bottleneck is actually flow control of buffers, which prevents utilizing full network bandwidth in normal gigabit connections. The threads will help only after this first bottleneck has been cleared. They have patches to fix both problems. The slashdot summary was therefore a bit inaccurate, and reading TFA certainly helps.

Pretty much totaly incorrect summary

[Eunuchswear]

Almost all the improvements they talk about come from optimising TCP buffer usage. The summary of the fixes:

HPN-13 A la Carte

• Dynamic Windows and None Cipher
  This is a basis of the HPN-SSH patch set. It provides dynamic window in SSH and the ability to switch to a NONE cipher post authentication. Based on the HPN12 v20 patch.
• Threaded CTR cipher mode
  This patch adds threading to the CTR block mode for AES and other supported ciphers. This may allow SSH to make use of multiple cores/cpus during transfers and significantly increase throughput. This patch should be considered experimental at this time.
  
• Peak Throughput
  This patch modifes the progress bar to display the 1 second throughput average. On completion of the transfer it will display the peak throughput through the life of the connection.
• Server Logging
  This patch adds additional logging to the SSHD server including encryption used...

So the main part of the patch set is "It provides dynamic window in SSH and the ability to switch to a NONE cipher post authentication" and the only part that has to do with threading is marked "This patch should be considered experimental at this time".

By the way, does anybody else think "the ability to switch to a NONE cipher post authentication" is pretty dodgy?

Re:Pretty much totaly incorrect summary

[rapier1]

As a note - while the NONE cipher switch turns off data encryption we do *not* disable the message authentication cipher (MAC). So each data packet is still signed and authenticated. If it detects any in transit modification of the packet the connection is immediately dropped.

Re:Alternative solution for a trusted LAN

[KiloByte]

Actually, it appears that (at least on Debian) AES is already the default. Selecting 3des gives tremendous slowdown; blowfish is somewhat slower than AES.

Copying 100MB of data over 100mbit ethernet to a P2 350Mhz box (the slowest I got) gives:
* 3des 1.9MB/s
* AES 4.8MB/s
* blowfish 4.4MB/s

Re:Must be why rsync over ssh is much faster

[AnyoneEB]

Unless your servers are running rsh, rsync is probably going to get routed through ssh, in which case it gets encrypted just like scp. ref:

Secure Shell - The security concious of you out there would like this, and you should all be using it. The stream from rsync is passed through the ssh protocol to encrypt your session instead of rsh, which is also an option (and required if you don't use ssh - enable it in your /etc/inet.d and restart your inet daemon if you disabled it for security).

Re:To *have* such problems...

[egede]

The limitations of transfer rates for scp is often the round trip time that consumes time for confirmation of received packages. This is a serious issue for transfers from the Europe to the US West Coast (around 200 ms) or to Australia (around 400 ms). Having several parallel TCP streams can solve this problem and has been in use for many years for transfer of data in High Energy Physics. An example of such a solution is GridFTP http://www.globus.org/toolkit/docs/4.0/data/gridftp/.

Re:Sweet!

[Per Wigren]

Use NX instead of plain old remote DISPLAY or ssh's X11 forwarding or even VNC! It's silly fast! You get a perfectly usable desktop even on slow, high latency connections. The free edition is free as in GPL.

Re:Must be why rsync over ssh is much faster

[rapier1]

As a note, the changes are actually to SSH itself and not just SCP. So any application that uses SSH as a transport mechanism can see a performance boost. This isn't to say *every* user will. This is mainly geared towards high bandwidth delay product networks (greater than 1MB) or GigE LANs.

Some comments from one of the authors

[rapier1]

First off, thank you for taking the time to read down this far. There have been some very interesting and useful comments so far. Second, I need to point out that both Ben Bennett of PSC and Michael Stevens of CMU were instrumental in getting this patch written. Without them there would be no HPN-SSH patch. I also highly suggest that interested people go to the http://www.psc.edu/networking/projects/hpn-ssh and read about what we've done. There is a lot of good material in the papers and presentations section as well as the FAQ.

A couple notes about the multi-threading: The main goal was to allow SSH to make use of multiple processing cores. The stock OpenSSH is, by design, limited to using one core. As such a user can encounter situations where they have more network capacity and more compute capacity but will be unable to exploit them. The goal of this patch was to allow users to make full use of the resources available too them. The upshot of this is that its best suited for high performance network and compute environments (The HPN in HPN-SSH stands for High Performance Networking). This doesn't mean it won't be useful to home users - only that they might not see the dramatic performance gains someone in a higher capacity environment might see. Its really going to depend on the specifics of their environment.
Based on our research we decided the most effective way to do this would be to make the AES-CTR mode cipher multi-threaded. The CTR mode is well suited to threading because there is no inter block dependency and, even better, the resulting cypher stream is indistinguishable from a single threaded CTR mode cypher stream. As a result, we retain full compatibility with other implementations of SSH - you don't need to have HPN-SSH on both sides of the connection. Of course, you won't see the same improvements unless you do.
We still see this as somewhat experimental because we've not yet implemented a way to allow users to choose between a single threaded AES-CTR and multi-threaded AES-CTR mode. As such users on single core machines - if using AES-CTR may see a decrease in performance. We suggest those users just make use of the AES-CBC mode instead (which is the default anyway). Also, you need to be able to support posix threads.
Future work will involve pipelining the MAC routine and that should provide us with another 30% or so improvement in throughput.

Also, its important to keep in mind that these improvements are *not* just for SCP but for SSH as a whole. People using HPN-SSH as a transport mechanism for rsync, tunnels, pipes, and so forth may also see considerable performance improvements. Additionally, the windowing patches don't necessarily require HPN-SSH to be installed on both ends of the connection. As long as the patch is installed on the receiving side (the data sink) you may (assuming you were previously window limited) see a performance gain.

We welcome any comments, suggests, ideas, or problem reports you might have regarding the HPN-SSH patch. Go the website mentioned above and use the email address there to get in touch with us. This is a work in progress and we are doing what we can to enable line rate easy to use fully encrypted communications. We've a lot more to do but I hope what we've done so far is of use and value to the community.