Slashdot Mirror
Multi-Threaded SSH/SCP
neo writes "Chris Rapier has presented a paper describing how to dramatically increase the speed of SCP networks. It appears that because SCP relies on a single thread in SSH, the crypto can sometimes be the bottleneck instead of the wire speed. Their new implementation (HPN-SSH) takes advantage of multi-threaded capable systems dramatically increasing the speed of securely copying files. They are currently looking for potential users with very high bandwidth to test the upper limits of the system."
If you want to speed up transfers and you're working on a LAN you trust (i.e. you don't worry about the integrity and confidentiality of the data passing through it), you can dramatically increase throughput using socketpipe. Although the initial socketpipe communication setup is performed through client-server intermediaries such as ssh(1), the communication channel that socketpipe establishes is a direct socket connection between the local and the remote commands. This eliminates not only the encryption/description overhead, but also the copying between your processes and ssh or rsh.
There is definitely something funny(strange) about the way scp does bulk copies. It stops and starts. Other applications happily stream through encrypted ssh connections.
And in my experience rsync is faster.
http://michaelsmith.id.au
the crypto can sometimes be the bottleneck instead of the wire speed.
Between two devices on my gigabit home LAN, the CPU barely even registers while SCP'ing a large file (and that with every CPU-expensive protocol option turned on, including compression). What sort of connection do these guys have, that the CPU overhead of en/decryption throttles the transfer???
Coming next week: SSH compromised via a thread injection attack, thanks to a "feature" that only benefits those of us running our own undersea fiber.
I've been wondering, does there exist hardware accelerators usable by OpenSSL or GnuTLS? I work in embedded systems, and our chip includes a crypto and hash processor. I'm surprised nothing equivalent exists on modern PCs, or have I just not been looking in the right places?
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
If you just want to copy some files from system to system in an encrypted fashion, then the BEST option by far is to use tar, and pipe it through ssh like so:
tar cvfpz - * | ssh user@host '( cdThis example will compress and encrypt your data before sending it; on the other end, the file is streamed to tar. This example requires GNU rar or a close facsimile.
Now, if you want to UPDATE a directory, use rsync:
rsync -av -e ssh * user@host:/destination/Because rsync will do partial checksums and send parts even of BINARY files if the whole file has not changed, and doesn't re-send unchanged files, rsync makes sense when updating a directory. But it provides no speedup benefit over using tar, and in fact the directory scans it does before the sync mean that it may actually be slower.
Use scp only for copying single files, because you're right, scp chokes between each file.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Thank you for your very well-written reply, but I wasn't actually being all that serious. (No, the pregnancy was unplanned, and I am actually opposed to the idea of becoming a parent at my current place in life; however, after talking this over with the only other person to have any say in this (the mother), I've decided to go with it).
Don't mistake my badly crafted joke for being completely ignorant of what's ahead of me; before the final decision came, I had consulted with friends who are also parents (carefully not discussing this with any of my single, singlemindedly free-roaming friends), and I am in no way in doubt that I will make this child a net benefit for the human race. There are simply too many rotten parents, spoilt children, miserable families and bad genes in the world for me to actually fail in that respect.
Plus, living in Denmark*, the baby will have pretty good odds for a good life, my involvement notwithstanding.
I am going to have a lot of fun making tech projects for my little one when that time comes, including audio books with his/her favourite bed time stories, video diaries of how the child evolves, and of course, teaching how to solder before the age of 5. How I survived until 15 without that knowledge eludes me to this day.
*: Studies have shown that there is a tie for Country With Best Quality of Life; Denmark and Iceland. I've been to Iceland, and it smelled like rotten eggs. Denmark takes the lead.