To be honest the ideal test system we are looking for now are 10Gb/s connected multi-core hosts - preferably over a transoceanic or continental path. We're still trying to see how fast we can go.
Some comments from one of the authors
on
Multi-Threaded SSH/SCP
·
· Score: 5, Informative
First off, thank you for taking the time to read down this far. There have been some very interesting and useful comments so far. Second, I need to point out that both Ben Bennett of PSC and Michael Stevens of CMU were instrumental in getting this patch written. Without them there would be no HPN-SSH patch. I also highly suggest that interested people go to the http://www.psc.edu/networking/projects/hpn-ssh and read about what we've done. There is a lot of good material in the papers and presentations section as well as the FAQ.
A couple notes about the multi-threading: The main goal was to allow SSH to make use of multiple processing cores. The stock OpenSSH is, by design, limited to using one core. As such a user can encounter situations where they have more network capacity and more compute capacity but will be unable to exploit them. The goal of this patch was to allow users to make full use of the resources available too them. The upshot of this is that its best suited for high performance network and compute environments (The HPN in HPN-SSH stands for High Performance Networking). This doesn't mean it won't be useful to home users - only that they might not see the dramatic performance gains someone in a higher capacity environment might see. Its really going to depend on the specifics of their environment.
Based on our research we decided the most effective way to do this would be to make the AES-CTR mode cipher multi-threaded. The CTR mode is well suited to threading because there is no inter block dependency and, even better, the resulting cypher stream is indistinguishable from a single threaded CTR mode cypher stream. As a result, we retain full compatibility with other implementations of SSH - you don't need to have HPN-SSH on both sides of the connection. Of course, you won't see the same improvements unless you do.
We still see this as somewhat experimental because we've not yet implemented a way to allow users to choose between a single threaded AES-CTR and multi-threaded AES-CTR mode. As such users on single core machines - if using AES-CTR may see a decrease in performance. We suggest those users just make use of the AES-CBC mode instead (which is the default anyway). Also, you need to be able to support posix threads.
Future work will involve pipelining the MAC routine and that should provide us with another 30% or so improvement in throughput.
Also, its important to keep in mind that these improvements are *not* just for SCP but for SSH as a whole. People using HPN-SSH as a transport mechanism for rsync, tunnels, pipes, and so forth may also see considerable performance improvements. Additionally, the windowing patches don't necessarily require HPN-SSH to be installed on both ends of the connection. As long as the patch is installed on the receiving side (the data sink) you may (assuming you were previously window limited) see a performance gain.
We welcome any comments, suggests, ideas, or problem reports you might have regarding the HPN-SSH patch. Go the website mentioned above and use the email address there to get in touch with us. This is a work in progress and we are doing what we can to enable line rate easy to use fully encrypted communications. We've a lot more to do but I hope what we've done so far is of use and value to the community.
The earlier versions did *not* include the multi-threaded cipher. If you are using one of the older version of HPN-SSH (HPN12 and below) you're limited to one core.
Re:Pretty much totaly incorrect summary
on
Multi-Threaded SSH/SCP
·
· Score: 4, Informative
As a note - while the NONE cipher switch turns off data encryption we do *not* disable the message authentication cipher (MAC). So each data packet is still signed and authenticated. If it detects any in transit modification of the packet the connection is immediately dropped.
Re:Pretty much totaly incorrect summary
on
Multi-Threaded SSH/SCP
·
· Score: 2, Informative
Where the performance boost comes from is going to depend on a lot on the characteristics of the network path. If its a high bandwidth delay product path then the majority of the performance increase may very well come from the dynamic window sizing (this is application layer windowing by the way). However, if path has a low BDP and you are CPU bound then either the NONE cipher switch or the multi-threading may provide more of a performance increase than the window sizing. Alternatively, in some high BDP paths the windowing patch may speed up the transfer enough so that it becomes CPU bound at which point the threading and/or NONE cipher switch will allow the user to make full use of the network capacity. One of our test paths is a transatlantic GigE pipe. With the window patch and full encryption we were able to get around 300Mb/s and then we were CPU bound. When we used either the NONE cipher switch or the multi-threaded AES-CTR mode cipher our throughput increased to 700Mb/s.
As for the dodgy aspect of the NONE cipher switching. I'll be the first to admit that its not a perfect solution. The authentication remains fully encrypted and you can't use the NONE switch in an interactive session which obviates some of the problems. However, it still, in some ways, is counter to the idea of SSH which is why we came up with the threaded cipher. If you are willing to accept the NONE cipher then you can use that but if you want full encryption then you can use the threaded AES-CTR mode cipher.
> if the CPU is the bottleneck, how could adding more threads possibly help?
This is actually a great question. On single core systems its very unlikely that the multi-threading aspect of our patch will be of much use to you. The stock version of SSH is, because of its design, unable to use more than one core regardless of how many cores you actually happen to have. Which means that you could have one core thats pegged by SSH and have other cores that are essentially running idle (if you look at the presentation we go into that after we address with window issues). What we've done is allow SSH to offload the heavy work (the encryption) onto other cores in order to make full use of the CPU resources available.
You may be looking at the wrong process. SCP is basically a wrapper around SSH - all it really does is the file handling and then passes the data to SSH via a pipe. So there will actually be an instance of SSH running every time you run SCP. As such CPU usage of SCP tends to remain quite small but if look for the associated SSH process you'll see that its actually sucking up a lot of compute resources. So you may want to rerun your tests and just look at total load or just run top. You may also want to take a look at the SSHD on the other machine as well.
If you just recompile the stock SSH with the NONE cipher enabled the entire process happens in clear. This includes the authentication. You can test this by capturing the network activity with TCP dump (use a full snaplen) and looking at the results. You'll find that your key/password/whatever is in plaintext.
The NONE cipher switch available in HPN-SSH is different in that the authentication *is* encrypted and the NONE cipher is only used *after* the user is authenticated.
Re:Must be why rsync over ssh is much faster
on
Multi-Threaded SSH/SCP
·
· Score: 3, Informative
As a note, the changes are actually to SSH itself and not just SCP. So any application that uses SSH as a transport mechanism can see a performance boost. This isn't to say *every* user will. This is mainly geared towards high bandwidth delay product networks (greater than 1MB) or GigE LANs.
Actually, this is one of the main reasons why we did this. If you look at where processor architectures are going they aren't necessarily increasing the pure computational power of each core as much as they are using more and more cores. If you are in a situation where you have a single threaded process that is CPU bound - like SSH can be - you'll find that throughput rates (assuming that you aren't network bound) will remain flat. In order to make full use of the available network capacity we'll have to start doing things like this. There is still a lot of work to be done but we're pleased by our progress so far.
Also, if your servers are more than a 64k BDP away then you probbaly want to install the HPN-SSH patch from PSC. This path overcomes one of the perfromance bottlenecks in SSH (limited window size). It also give syou the option of use the NONE cipher - the neat thing is that it still uses encryption for authentication. Unfortunately, it only allows the use of the NONE cipher switching for transfering files and not shell sessions. http://www.psc.edu/networking/projects/hpn-ssh
I think it simportant to remember something in all of this. The best software is *not* the one that has the keenest features or best this or superior other things. The best software is the one you are personally comfortable with. I can imagine why anyone would use vi when emacs is available. However, lots of people *love* vi. I can't stand it personally so for me emacs is superior.
For mail I use pine for personal mail - its fast, lightweight, and lets me do everything I want to. I use gmail for mailing lists and for websites that insist I give them an email address. For work mail I use thunderbird. Each has its advantages in different contexts and so, for me, none of them is *best*. Each shines in a specific area but there is now overall winner.
One thing I really like about pine that will never be availabe in gmail? The ability to grep/less/cat through a collection of messages or pipe them easily into a perl script or easily gzip and drop them all on a cd for archiving. Its nice but its soemthgin I care about that others don't.
I have more problems with my Powerbook G4 than I do with my PC running XP SP2. On my Windows XP box I have fewer crashes, fewer mysterious hangups, fewer stalled applications, etc etc etc. Now, I know that this might be becasue that I do a lot of development work on my Mac (I basically got it because it was close enough to a unix box with a UI that didn't suck to make it worthwhile) and there is every chance I might have screwed something up somehow. It might also be because I don't see the point in powering things down (a long time mac user told me that was my problem). Still, as of this moment my Mac has been up 15 days since the last reboot and my XP box has been up 32 days. Either is pretty good but I think the 'wind0wz sux0rz' rhetoric has gotten a little stale.
I just wish Apple would do something about their TCP stack. Ya know, the BSD stack they built on was the best stack around 5 years ago. Now its one of the worst. Even Vista has a better TCP stack.
Wait, the need to use Word is why 95% of of the market uses Windows? Except for the whole Office for OS X thing I'm sure you are on to something there. Why don't you explore it a bit more and get back to me when you learn where the shift key is.
Oh, as for the get a life thing... biting riposte there. Have you had call to use that one in your other 153 posts on slashdot?
I really hate the attitude you get from some Apple users and from the company itself.
Just shut the fuck up already. I know you think you are superior, sexier, healthier, and smarter.
You aren't any of those. You are just using a fucking computer. Its a tool, not a way of life.
If it is a way of life for you then you have some serious social issues that need to be addressed.
Yes, I use a Mac (12" powerbook). I even do development on it. Its a great machine.
Its just a machine though.
Slashdot Mirror
To be honest the ideal test system we are looking for now are 10Gb/s connected multi-core hosts - preferably over a transoceanic or continental path. We're still trying to see how fast we can go.
A couple notes about the multi-threading: The main goal was to allow SSH to make use of multiple processing cores. The stock OpenSSH is, by design, limited to using one core. As such a user can encounter situations where they have more network capacity and more compute capacity but will be unable to exploit them. The goal of this patch was to allow users to make full use of the resources available too them. The upshot of this is that its best suited for high performance network and compute environments (The HPN in HPN-SSH stands for High Performance Networking). This doesn't mean it won't be useful to home users - only that they might not see the dramatic performance gains someone in a higher capacity environment might see. Its really going to depend on the specifics of their environment.
Based on our research we decided the most effective way to do this would be to make the AES-CTR mode cipher multi-threaded. The CTR mode is well suited to threading because there is no inter block dependency and, even better, the resulting cypher stream is indistinguishable from a single threaded CTR mode cypher stream. As a result, we retain full compatibility with other implementations of SSH - you don't need to have HPN-SSH on both sides of the connection. Of course, you won't see the same improvements unless you do.
We still see this as somewhat experimental because we've not yet implemented a way to allow users to choose between a single threaded AES-CTR and multi-threaded AES-CTR mode. As such users on single core machines - if using AES-CTR may see a decrease in performance. We suggest those users just make use of the AES-CBC mode instead (which is the default anyway). Also, you need to be able to support posix threads.
Future work will involve pipelining the MAC routine and that should provide us with another 30% or so improvement in throughput.
Also, its important to keep in mind that these improvements are *not* just for SCP but for SSH as a whole. People using HPN-SSH as a transport mechanism for rsync, tunnels, pipes, and so forth may also see considerable performance improvements. Additionally, the windowing patches don't necessarily require HPN-SSH to be installed on both ends of the connection. As long as the patch is installed on the receiving side (the data sink) you may (assuming you were previously window limited) see a performance gain.
We welcome any comments, suggests, ideas, or problem reports you might have regarding the HPN-SSH patch. Go the website mentioned above and use the email address there to get in touch with us. This is a work in progress and we are doing what we can to enable line rate easy to use fully encrypted communications. We've a lot more to do but I hope what we've done so far is of use and value to the community.
The earlier versions did *not* include the multi-threaded cipher. If you are using one of the older version of HPN-SSH (HPN12 and below) you're limited to one core.
As a note - while the NONE cipher switch turns off data encryption we do *not* disable the message authentication cipher (MAC). So each data packet is still signed and authenticated. If it detects any in transit modification of the packet the connection is immediately dropped.
Where the performance boost comes from is going to depend on a lot on the characteristics of the network path. If its a high bandwidth delay product path then the majority of the performance increase may very well come from the dynamic window sizing (this is application layer windowing by the way). However, if path has a low BDP and you are CPU bound then either the NONE cipher switch or the multi-threading may provide more of a performance increase than the window sizing. Alternatively, in some high BDP paths the windowing patch may speed up the transfer enough so that it becomes CPU bound at which point the threading and/or NONE cipher switch will allow the user to make full use of the network capacity. One of our test paths is a transatlantic GigE pipe. With the window patch and full encryption we were able to get around 300Mb/s and then we were CPU bound. When we used either the NONE cipher switch or the multi-threaded AES-CTR mode cipher our throughput increased to 700Mb/s.
As for the dodgy aspect of the NONE cipher switching. I'll be the first to admit that its not a perfect solution. The authentication remains fully encrypted and you can't use the NONE switch in an interactive session which obviates some of the problems. However, it still, in some ways, is counter to the idea of SSH which is why we came up with the threaded cipher. If you are willing to accept the NONE cipher then you can use that but if you want full encryption then you can use the threaded AES-CTR mode cipher.
> if the CPU is the bottleneck, how could adding more threads possibly help? This is actually a great question. On single core systems its very unlikely that the multi-threading aspect of our patch will be of much use to you. The stock version of SSH is, because of its design, unable to use more than one core regardless of how many cores you actually happen to have. Which means that you could have one core thats pegged by SSH and have other cores that are essentially running idle (if you look at the presentation we go into that after we address with window issues). What we've done is allow SSH to offload the heavy work (the encryption) onto other cores in order to make full use of the CPU resources available.
You may be looking at the wrong process. SCP is basically a wrapper around SSH - all it really does is the file handling and then passes the data to SSH via a pipe. So there will actually be an instance of SSH running every time you run SCP. As such CPU usage of SCP tends to remain quite small but if look for the associated SSH process you'll see that its actually sucking up a lot of compute resources. So you may want to rerun your tests and just look at total load or just run top. You may also want to take a look at the SSHD on the other machine as well.
If you just recompile the stock SSH with the NONE cipher enabled the entire process happens in clear. This includes the authentication. You can test this by capturing the network activity with TCP dump (use a full snaplen) and looking at the results. You'll find that your key/password/whatever is in plaintext. The NONE cipher switch available in HPN-SSH is different in that the authentication *is* encrypted and the NONE cipher is only used *after* the user is authenticated.
As a note, the changes are actually to SSH itself and not just SCP. So any application that uses SSH as a transport mechanism can see a performance boost. This isn't to say *every* user will. This is mainly geared towards high bandwidth delay product networks (greater than 1MB) or GigE LANs.
Actually, this is one of the main reasons why we did this. If you look at where processor architectures are going they aren't necessarily increasing the pure computational power of each core as much as they are using more and more cores. If you are in a situation where you have a single threaded process that is CPU bound - like SSH can be - you'll find that throughput rates (assuming that you aren't network bound) will remain flat. In order to make full use of the available network capacity we'll have to start doing things like this. There is still a lot of work to be done but we're pleased by our progress so far.
The amazing innovation is marketing.
Also, if your servers are more than a 64k BDP away then you probbaly want to install the HPN-SSH patch from PSC. This path overcomes one of the perfromance bottlenecks in SSH (limited window size). It also give syou the option of use the NONE cipher - the neat thing is that it still uses encryption for authentication. Unfortunately, it only allows the use of the NONE cipher switching for transfering files and not shell sessions. http://www.psc.edu/networking/projects/hpn-ssh
I think it simportant to remember something in all of this. The best software is *not* the one that has the keenest features or best this or superior other things. The best software is the one you are personally comfortable with. I can imagine why anyone would use vi when emacs is available. However, lots of people *love* vi. I can't stand it personally so for me emacs is superior. For mail I use pine for personal mail - its fast, lightweight, and lets me do everything I want to. I use gmail for mailing lists and for websites that insist I give them an email address. For work mail I use thunderbird. Each has its advantages in different contexts and so, for me, none of them is *best*. Each shines in a specific area but there is now overall winner. One thing I really like about pine that will never be availabe in gmail? The ability to grep/less/cat through a collection of messages or pipe them easily into a perl script or easily gzip and drop them all on a cd for archiving. Its nice but its soemthgin I care about that others don't.
I know this is probably heresy but...
I have more problems with my Powerbook G4 than I do with my PC running XP SP2. On my Windows XP box I have fewer crashes, fewer mysterious hangups, fewer stalled applications, etc etc etc. Now, I know that this might be becasue that I do a lot of development work on my Mac (I basically got it because it was close enough to a unix box with a UI that didn't suck to make it worthwhile) and there is every chance I might have screwed something up somehow. It might also be because I don't see the point in powering things down (a long time mac user told me that was my problem). Still, as of this moment my Mac has been up 15 days since the last reboot and my XP box has been up 32 days. Either is pretty good but I think the 'wind0wz sux0rz' rhetoric has gotten a little stale.
I just wish Apple would do something about their TCP stack. Ya know, the BSD stack they built on was the best stack around 5 years ago. Now its one of the worst. Even Vista has a better TCP stack.
Wait, the need to use Word is why 95% of of the market uses Windows? Except for the whole Office for OS X thing I'm sure you are on to something there. Why don't you explore it a bit more and get back to me when you learn where the shift key is.
Oh, as for the get a life thing... biting riposte there. Have you had call to use that one in your other 153 posts on slashdot?
Yes, I use a Mac (12" powerbook). I even do development on it. Its a great machine. Its just a machine though.