Slashdot Mirror
Multi-Threaded SSH/SCP
neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."
If you want to speed up transfers and you're working on a LAN you trust (i.e. you don't worry about the integrity and confidentiality of the data passing through it), you can dramatically increase throughput using socketpipe. Although the initial socketpipe communication setup is performed through client-server intermediaries such as ssh(1), the communication channel that socketpipe establishes is a direct socket connection between the local and the remote commands. This eliminates not only the encryption/description overhead, but also the copying between your processes and ssh or rsh.
There is definitely something funny(strange) about the way scp does bulk copies. It stops and starts. Other applications happily stream through encrypted ssh connections.
And in my experience rsync is faster.
http://michaelsmith.id.au
the crypto can sometimes be the bottleneck instead of the wire speed.
Between two devices on my gigabit home LAN, the CPU barely even registers while SCP'ing a large file (and that with every CPU-expensive protocol option turned on, including compression). What sort of connection do these guys have, that the CPU overhead of en/decryption throttles the transfer???
Coming next week: SSH compromised via a thread injection attack, thanks to a "feature" that only benefits those of us running our own undersea fiber.
I've been wondering, does there exist hardware accelerators usable by OpenSSL or GnuTLS? I work in embedded systems, and our chip includes a crypto and hash processor. I'm surprised nothing equivalent exists on modern PCs, or have I just not been looking in the right places?
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
Digging around for the best way to apply the patch without screwing up my portage updates, I came across a request for this to be merged into the portage back in 2005, and is apparently usable with the HPN useflag.
Not that I'm that surprised to see this is old news, since they're apparently on major revision 13...
...what's wrong with looking at porn with a real girl?
I may be speaking out of ignorance, but doesn't that defeat the point of SSH?
"Our opponent is an alien starship packed with atomic bombs. We have a protractor."
Okay, it's very simple.
/some/path myhost.com:my/directory
/some/path rsync://myhost.com/my/directory OR /some/path myhost.com::my/directory
Encrypted and tunneled over SSH, rsync is spawned by a login shell at the other side:
rsync
Not encrypted, rsyncs daemon must be running at other end:
rsync
rsync
A cat can't teach a dog to bark.
If you just want to copy some files from system to system in an encrypted fashion, then the BEST option by far is to use tar, and pipe it through ssh like so:
tar cvfpz - * | ssh user@host '( cdThis example will compress and encrypt your data before sending it; on the other end, the file is streamed to tar. This example requires GNU rar or a close facsimile.
Now, if you want to UPDATE a directory, use rsync:
rsync -av -e ssh * user@host:/destination/Because rsync will do partial checksums and send parts even of BINARY files if the whole file has not changed, and doesn't re-send unchanged files, rsync makes sense when updating a directory. But it provides no speedup benefit over using tar, and in fact the directory scans it does before the sync mean that it may actually be slower.
Use scp only for copying single files, because you're right, scp chokes between each file.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
There's also message passing and event driven programming, which can be a much simpler model if done right. Multi threading tends to shared state, and that's bad for programmers.
I can throw myself at the ground, and miss.
SSH is one of those uberutilities that has a surprising amount of usefulness once you dig a bit. Sure, secure telnet functionality is great, and I use it a lot. But, I still use ssh on my own LAN where I don't really care about security. I use sshfs because it is easier and more convenient for me than bothering with Samba. SCP/SFTP to avoid bothering with ftp. I use it for forwarding ports between various machines, and I use it for forwarding X sessions. There are surely ways to accomplish all the same stuff that I use ssh (and the closely related stuff like scp) for without using ssh, but I'd just rather not bother about it.
Defeating the point of ssh is like defeating the point of a morning star. No matter how dull you make the point, the other 99 of them will still clobber you in the face.
Sure, it's amazing during the formative years... but later on down the line, when that nurturing environment that made your child into a healthy, independent, and intelligent adult lets him come to his own conclusions that he should vehemently support that which you equally vehemently oppose. When the loving adoration for daddy turns into a sort of pitiful contempt because of some religious, philosophical, or political variable.
Can't see how anyone could feel like anything but a failure in that spot.
Thank you for your very well-written reply, but I wasn't actually being all that serious. (No, the pregnancy was unplanned, and I am actually opposed to the idea of becoming a parent at my current place in life; however, after talking this over with the only other person to have any say in this (the mother), I've decided to go with it).
Don't mistake my badly crafted joke for being completely ignorant of what's ahead of me; before the final decision came, I had consulted with friends who are also parents (carefully not discussing this with any of my single, singlemindedly free-roaming friends), and I am in no way in doubt that I will make this child a net benefit for the human race. There are simply too many rotten parents, spoilt children, miserable families and bad genes in the world for me to actually fail in that respect.
Plus, living in Denmark*, the baby will have pretty good odds for a good life, my involvement notwithstanding.
I am going to have a lot of fun making tech projects for my little one when that time comes, including audio books with his/her favourite bed time stories, video diaries of how the child evolves, and of course, teaching how to solder before the age of 5. How I survived until 15 without that knowledge eludes me to this day.
*: Studies have shown that there is a tie for Country With Best Quality of Life; Denmark and Iceland. I've been to Iceland, and it smelled like rotten eggs. Denmark takes the lead.