Slashdot Mirror

You can go read about the SSH issue at PSC, who developed HPN-SSH to deal with the issue.
http://www.psc.edu/index.php/h...

HPN-SSH is starting to become standard on BSD and Linux systems.

There is an unofficial patch set that adds the none cypher:

http://www.psc.edu/index.php/h...

I know the authentication handshaking is part of what causes the slowness, but I figure this is still a good spot to plug HPN-SSH (sourceforge link). That is, a set of patches to OpenSSH that increase throughput in simple ways:

1) Multi-threaded cipher implementations - (seriously, OpenSSH does not do this out of the box! And, it yields probably the greatest improvement in performance)
2) Auto-adjusting TCP buffer window sizes
3) A "none" cipher that still uses secure authentication to establish a connection...although it is generally unnecessary, since I've been able to saturate a 10Gbps link with it running the standard AES ciphers in multithreaded mode...

http://www.psc.edu/science/Wol...

http://www.psc.edu/index.php/h...

explains the issue and the fix in detail.

Late FreeBSD 8 and up have HPN-SSH by default.

HPN-SSH with the "NONE" encryption option is blinding fast, up to 2.5Gbit per stream on bigger servers. Filling even an OC-48 is possible with very big servers.

SSH has severe performance issues and hardly anyone uses the high-performance patches. (Hell, hardly anyone knows the high-performance patches exist!)

You'll notice the patches are not being funded any more and that none of the SSH enthusiasts are, well, enthused enough to have volunteered to help the maintainer. Don't get me wrong, I like SSH, but it doesn't write itself and I have very little sympathy for those who complain about under-utilization when doing nothing about helping to address the issues that would improve the situation.

Actually, in the realm of biomedical supercomputing, "none of the above" has already been done. Check out the Anton supercomputer designed and built by D.E. Shaw Research. The entire supercomputer, right down to all of the processor cores themselves, were specially designed and built specifically for molecular dynamics research. The system has no operating system and, as such, no overhead. Every processor cycle goes straight into the calculations. It is capable of churning out simulations of 150,000+ atom protein complexes on the order of several microseconds long, using wallclock CPU time of a few days.

Of course, it's related to a similar fine-grained parallelism idea for crypto that I wish would be widely implemented, and that's offloading most of AES CTR mode onto a separate thread, or several separate processes since each block has a computation step that can be computed in advance in parallel with all the other blocks. I might start doing multi-gigabyte transfers over ssh if that were implemented.

You may find this helpful, which does exactly what you're thinking of... it's basically a set of patches to OpenSSH. HPN-13 does do AES CTR processing onto multiple threads; in addition to its original functionality which was to include various network optimizations to increase throughput. http://www.psc.edu/networking/projects/hpn-ssh/ (I've been digging into it a bit very recently because someone reported it's not playing nicely with BBSSH)

Of course, it's related to a similar fine-grained parallelism idea for crypto that I wish would be widely implemented, and that's offloading most of AES CTR mode onto a separate thread, or several separate processes since each block has a computation step that can be computed in advance in parallel with all the other blocks. I might start doing multi-gigabyte transfers over ssh if that were implemented. As it is, on even my fastest computers, multi-gigabyte transfers are very CPU bound over ssh with the ssh process pegged at 100% (and just 100%) CPU.

That's already implemented in the high performance ssh patch, available here. Scroll down to the "Threaded CTR cipher mode" patch, if that's the only part you're interested in.

I've applied it to the openssh package on my Fedora 12 system. As is, it provides about 40% increased throughput on my quad-core. You may be able to get more by tweaking it to increase the thread count.

Judging from this, it should be more like 96kB, not ~9kB.

Like the other poster, I've see 30-50 MB/s (300-500 Mbps) over a gigabit network when copying between boxes using scp. The limitations were more the frame size (not using jumbo frames on that network) along with the read/write speeds of the system on each end.

So, it's no slouch and better then SMB/CIFS.

On the other hand, I see unencrypted NFS achieve > 90 MB/s over TCP on my gigabit networks, about 2 to 3 times faster. This is without using jumbo frames.

The tiny, statically sized 64 KB send buffer that OpenSSH limits itself to really does hurt, even on low latency networks.

Check out the graph comparing unmodified OpenSSH (red) with a version patched to use larger buffers (blue) on the Pittsbugh Supercomputing Center site mentioned above:
        http://www.psc.edu/networking/projects/hpn-ssh/hpn-v-ssh-tput.jpg
        http://www.psc.edu/networking/projects/hpn-ssh/

Like the other poster, I've see 30-50 MB/s (300-500 Mbps) over a gigabit network when copying between boxes using scp. The limitations were more the frame size (not using jumbo frames on that network) along with the read/write speeds of the system on each end.

So, it's no slouch and better then SMB/CIFS.

On the other hand, I see unencrypted NFS achieve > 90 MB/s over TCP on my gigabit networks, about 2 to 3 times faster. This is without using jumbo frames.

The tiny, statically sized 64 KB send buffer that OpenSSH limits itself to really does hurt, even on low latency networks.

Check out the graph comparing unmodified OpenSSH (red) with a version patched to use larger buffers (blue) on the Pittsbugh Supercomputing Center site mentioned above:
        http://www.psc.edu/networking/projects/hpn-ssh/hpn-v-ssh-tput.jpg
        http://www.psc.edu/networking/projects/hpn-ssh/

Did OpenSSH ever fix the performance limitation on fast networks (>100Mbps)? They have static internal flow buffers that prevent fast scp/ssh! HPN has a patch but OpenSSH has to my knowledge never adopted it. http://www.psc.edu/networking/projects/hpn-ssh/

I used to work at a place where we used SCP to throw files around on the local 10/100 network. Those transfers always maxed out the network speed (11MBps or so). FTP didn't go any faster... I'm guessing your problem was elsewhere :P

It's not just about bandwidth, it's about latency. OpenSSH doesn't perform well when the latency is high. This is a known problem.

For example, between two of our offices (all have 100Mbit ISP links) with a ~300ms ping time an SCP struggles to break 170K/sec. Between two offices with only ~100ms the speed is around 350K/sec. To my home machine at only ~30ms it fully saturates my 10Mbit ADSL.

For high latency links with you need to look at alternative tools like dmscp2 to fully utilise the available bandwidth.

We've done a lot of testing for our data warehouse products over a gigabit link between two quadcore server PCs comparing the transfer of several gigabytes worth of data, between ftp and sftp, and typical times for sftp have been taking about 3 and a half hours, when the same transfer via ftp is taking about 20 minutes. For the clients, we've been using psftp and windows command-line ftp, and for the servers, War-FTP and copssh. HP has a performance patch for OpenSSH (see here), but we have been unable to locate or develop a build for Windows that has this patch. While there may be better tuned SFTP software out there, the readily available open source tools do not compare well with FTP.

There used to be server problems in http://www.wesnoth.org/ MP server. Problem was that windows clients were dropping out of server randomly because of network problems.

Problem was in server side but only windows clients were affected. Problem seemed to be that windows limits maximum packet size (default kernel buffer size for sockets) to 8kb as default which cause windows clients getting a lot larger latency than linux clients. It was about 4 times longer login time for windows vista client than linux client from same lan. (8-10 seconds for linux and 30-40 seconds for windows) So somehow server/router was preferring transmission of large packets to linux clients which cause windows clients losing connection when they didn't receive data fast enough.

Of course bfw is bad example because problem might be in multiple locations like SDL. But at least one thing is sure that 8kb is way too small for high bandwidth connections. (http://www.psc.edu/networking/projects/tcptune/)

Modern Linux kernel 2.6.17 and later has TCP auto tuning, so it can better adapt to the network and saturate it. http://www.psc.edu/networking/projects/tcptune/#Linux Windows XP default TCP window size is too small and needs registry tuning for it to be optimized high speed broadband connections. Just google for WinXP TCP tuning. Or try comparing with Vista as it has better TCP/IP stack.

It's the TCP implementations, and probably the TCP window size limits. Windows could turn in the same numbers if properly tuned.

You want to read this article for all the in-depth details: http://www.psc.edu/networking/projects/tcptune/

Windows has a default set many years ago, and never updated. Most of the Free Unix variants update every release, and some new variants even have fancy auto-scaling code. Any time you want to get over 10Mbps/second across any real latency with a SINGLE TCP stream you probably need to do some tuning, for some OS's the limit is much lower.

ISP's run into this all the time. An uninformed admin buys a GigE in LA and NY, pops up an FTP server and wonders why he can only get a few megabits a second across the "crappy network". A few settings later and behold, the same hardware can saturate a full gigabit.

Note, don't just go set your values really high, there are performance (memory used) tradeoffs....

FTP isn't going to go away until the readily available secure alternatives perform as well. Especially since data moving operations have been increasing their "need for speed" along with the amounts of data involved.

I've been involved with performance testing on a data warehouse product that must transfer umpteen-GB nightly, where we've found that FTP transfers typically perform at least 30x faster than then next fastest alternative-- scp, sftp, etc. On a 1000MB link between two computers sitting next to each other we're seeing a 20 minute FTP turn into like 3.5 hours when we switch to sftp. We've resorted to parallel connections as a work around which helps, but it's still dog-slow compared to FTP.

There's a patch to OpenSSH that helps, but FTP is still notably faster, and almost no OpenSSH program binary distributions contain the patch, so you have to have a development system and know how to use it to even try it (and on Windows, that also usually means $$$).

I agree there is a real need for a secure replacement for FTP, but have yet to seen any contenders that I can take seriously.

Is one of the "breaking standards" going to be something like a realistic MTU? 1500 bytes is technically too small for 100BASE-T, let alone something ten or even 100 times that fast. Let's do away with the jumbo frames bullshit once and for all. IPv4 can handle a MTU of up to 65KB, IPv6 can take that to something like 4.2GB.

Quick reading: http://staff.psc.edu/mathis/MTU/index.html

A couple notes about the multi-threading: The main goal was to allow SSH to make use of multiple processing cores. The stock OpenSSH is, by design, limited to using one core. As such a user can encounter situations where they have more network capacity and more compute capacity but will be unable to exploit them. The goal of this patch was to allow users to make full use of the resources available too them. The upshot of this is that its best suited for high performance network and compute environments (The HPN in HPN-SSH stands for High Performance Networking). This doesn't mean it won't be useful to home users - only that they might not see the dramatic performance gains someone in a higher capacity environment might see. Its really going to depend on the specifics of their environment.
Based on our research we decided the most effective way to do this would be to make the AES-CTR mode cipher multi-threaded. The CTR mode is well suited to threading because there is no inter block dependency and, even better, the resulting cypher stream is indistinguishable from a single threaded CTR mode cypher stream. As a result, we retain full compatibility with other implementations of SSH - you don't need to have HPN-SSH on both sides of the connection. Of course, you won't see the same improvements unless you do.
We still see this as somewhat experimental because we've not yet implemented a way to allow users to choose between a single threaded AES-CTR and multi-threaded AES-CTR mode. As such users on single core machines - if using AES-CTR may see a decrease in performance. We suggest those users just make use of the AES-CBC mode instead (which is the default anyway). Also, you need to be able to support posix threads.
Future work will involve pipelining the MAC routine and that should provide us with another 30% or so improvement in throughput.

Also, its important to keep in mind that these improvements are *not* just for SCP but for SSH as a whole. People using HPN-SSH as a transport mechanism for rsync, tunnels, pipes, and so forth may also see considerable performance improvements. Additionally, the windowing patches don't necessarily require HPN-SSH to be installed on both ends of the connection. As long as the patch is installed on the receiving side (the data sink) you may (assuming you were previously window limited) see a performance gain.

We welcome any comments, suggests, ideas, or problem reports you might have regarding the HPN-SSH patch. Go the website mentioned above and use the email address there to get in touch with us. This is a work in progress and we are doing what we can to enable line rate easy to use fully encrypted communications. We've a lot more to do but I hope what we've done so far is of use and value to the community.

Reminds me of that joke about Elaine and the horse etc.

Yet, since the first announcement of 'Cold Fusion', has anyone followed the price increase of Palladium? Its price has gone into the stratosphere. Why wonder? http://www.psc.edu/science/Wolf/Wolf.html

There won't be any period of time when the field strength will be zero magnitude. It will just get increasingly sloppy, with local north and south poles popping up in random places until they reverse.

Nice simulation here!

Also, if your servers are more than a 64k BDP away then you probbaly want to install the HPN-SSH patch from PSC. This path overcomes one of the perfromance bottlenecks in SSH (limited window size). It also give syou the option of use the NONE cipher - the neat thing is that it still uses encryption for authentication. Unfortunately, it only allows the use of the NONE cipher switching for transfering files and not shell sessions. http://www.psc.edu/networking/projects/hpn-ssh

I agree with most of what you said, but don't understand why the point of adding the bardcode bit at the end was for.

With barcodes, you can scale up your system with a tape storage robot. The barcode is mostly there for error checking to make sure a tape is in the correct spot. When I worked at the PSC, they had two fairly large tape robots. I had to rewrite the barcode generator program, so that it no longer depended on a library with a strict beerware license (No, I did not make this up).

I read once the 640K is enough for anyone...

http://www.psc.edu/science/kollman98.html

Yes it is a short period, but it is done.

If our computational power keep increasing that way it has for the last 2 years, we will be able to do this on the desktop in 10-15 years.

Earth is way overdue for a magnetic field reversal. They have an average interval of 1/4 million years and it has been 3/4 million already since the last one. Some say it is beginning with the loss of a magnetic pole in certain places in the southern hemisphere. It could be the cause of the ozone layer loss because as the field weakens it radius at the poles grows. When the field is strong the field meets at the poles in a tight radius.

Here are some cool sims from Los Alamos National Laboratory.
 
As we lose protection more radiation gets through and mother earth gets a temperature. I'm not saying that 100 years of intense burning hasn't contributed but this seems to be an ignored fact that may be contributing in a large manner.

I first heard of this from watching a NOVA program. Here is the NOVA site on earths magnetic fields with some animations.

Ok, now where did I put the SPF 10,000?

Even worse is the Dilbert Principle: http://www.psc.edu/~deerfiel/Jokes/Dilbert-princip le.html

Also, check out an animation of a spontaneous pole reversal in a simulation on a Cray C90 circa 1996.

It really irks me that supposedly professional companies sell software that doesn't follow basic lessons like that, especially for the standard libraries they provide so everybody's programs can avoid writing special file format parsers from scratch.

Some of this is because too many people still write in C when they're not good enough to do it competently, and the companies they work for aren't making sure their code is properly reviewed, and they're letting them use a language that lets you shoot yourself in the foot. Don't get me wrong - C is still my favorite programming language, small, clean, elegant, and obvious, but most people shouldn't be allowed to use it.

Ok, didn't check that bit - it was from memory.

Looking on Google again I found what I'd been think of. It's the Geodynamo if anyone want to look for it.

"Currents flowing in giant loops through the earth's core."
You're pretty much correct there on how they think it happens.

Basically the guy noticed that the direction the molton core convects is different in some places than others. And it changes the way the magnetic flux acts.

When the convects in one direction, the flux goings in one direction; when it convects in the other the flux reverses the direction (which is what I was thinking of when I said about the North/South poles).

Lots of pretty pictures on a few sites talking about the Geodynamo, I'm sure there'd be more stuff around if anyone wants to look for it.

http://www.psc.edu/science/Glatzmaier/glatzmaier.h tml
http://www.psc.edu/research/graphics/gallery/geody namo.html

The first two animations on that page show fairly when what's happening.
The core of the Earth is rather chaotic in terms of which direction the convection is happening in and therefore which direction the magnetic flux is in (this is what I'd been thinking of).
These bits change over time and move around to different points under the Earth's surface (think hotspots which move and cause chains of volanos which are all dormant apart from the ones at the end).
Which direction the flux moves in overall is essentially a complex summation of where these lines of flux are moving.
During the reversal lots of areas of convection change direction and change the direction of their flux. As they do so the overall lines of flux move and weaken, until they swap around.

This is quite an informative page on magnetic field reversals, and it talks about the Geodynamo at the end.
http://www.geomag.bgs.ac.uk/reversals.html

Obligatory Wikipedia links:
http://en.wikipedia.org/wiki/Magnetic_polarity_rev ersal
http://en.wikipedia.org/wiki/Geomagnetic_reversal
http://en.wikipedia.org/wiki/Geodynamo
http://en.wikipedia.org/wiki/Dynamo_theory

Ok, didn't check that bit - it was from memory.

Looking on Google again I found what I'd been think of. It's the Geodynamo if anyone want to look for it.

"Currents flowing in giant loops through the earth's core."
You're pretty much correct there on how they think it happens.

Basically the guy noticed that the direction the molton core convects is different in some places than others. And it changes the way the magnetic flux acts.

When the convects in one direction, the flux goings in one direction; when it convects in the other the flux reverses the direction (which is what I was thinking of when I said about the North/South poles).

Lots of pretty pictures on a few sites talking about the Geodynamo, I'm sure there'd be more stuff around if anyone wants to look for it.

http://www.psc.edu/science/Glatzmaier/glatzmaier.h tml
http://www.psc.edu/research/graphics/gallery/geody namo.html

The first two animations on that page show fairly when what's happening.
The core of the Earth is rather chaotic in terms of which direction the convection is happening in and therefore which direction the magnetic flux is in (this is what I'd been thinking of).
These bits change over time and move around to different points under the Earth's surface (think hotspots which move and cause chains of volanos which are all dormant apart from the ones at the end).
Which direction the flux moves in overall is essentially a complex summation of where these lines of flux are moving.
During the reversal lots of areas of convection change direction and change the direction of their flux. As they do so the overall lines of flux move and weaken, until they swap around.

This is quite an informative page on magnetic field reversals, and it talks about the Geodynamo at the end.
http://www.geomag.bgs.ac.uk/reversals.html

Obligatory Wikipedia links:
http://en.wikipedia.org/wiki/Magnetic_polarity_rev ersal
http://en.wikipedia.org/wiki/Geomagnetic_reversal
http://en.wikipedia.org/wiki/Geodynamo
http://en.wikipedia.org/wiki/Dynamo_theory

This round of Teragrid funding is $150 Million. PSC got $52 Million. The rest is split up among the other 7 institutions. The other major partners already got big awards under earlier rounds of funding.

Oh, and this news is a month old.

uh...you mean like the entire Cray XT3 line that scales from 1 to over 30,000 processors. Granted this is a pseudo cluster but that is generally what all big iron is now-a-days.

http://www.psc.edu/publicinfo/news/2004/2004-10-25 _cray.html

http://www.psc.edu/networking/projects/hpn-ssh/

there is a patch called HPN-SSH that addresses some issues in ssh that users encounter if they have access to faster networks. SSH has some static flow control buffer values that limit network performance. The work at PSC by Chris Rapier and Michael Stevens is really nice, is proven to work and is gaining (some) broader acceptance.

take it for a test run and if you like it, please encourage the OpenSSH folks to add it into the main trunk.

The weakening of the magnetic field is probally caused by the upcoming magnetic field flip

Mitochondrial DNA, furthermore, is inherited from the mother alone. Nuclear DNA, by contrast, bears the genetic imprint of two parents, four grandparents, eight great-grandparents, etc. Because only one parent contributes to mitochondrial DNA, genetic changes trace backward in a single line, one person in each generation, making it feasible in theory to trace all humans to a single ancestor -- so-called "mitochondrial Eve." Several major studies have used these methods to arrive at the still controversial conclusion that all humans are descended from people living in Africa about 200,000 years ago.

Don't worry, you'll get her yet; Hell is Exothermic remember?

Take Pound, a few web server machines, a database server and a NFS server (no Coda, AFS or GFS needed in most cases) and you should be set. This is a setup that I installed for a high traffic website and it is very stable.

Read the next couple of posts in that thread...

Memory addressing is not handled in a different manner, but handles way more memory. Thus it is still different (being 64-bit memory addressing instead of 32 bit memory addressing). Not a paradigm shift, but still different.

As for the much larger and more precise calculations:
They sure are. For integers (the floating point comment was just for the more precise part), the calculations can be WAY larger, since the integer values are now up to 20 digits instead of 9.

http://www.psc.edu/general/software/packages/ieee/ ieee.html64-bit floats are double precision.

The file size thing was discussed more thoroughly in the next few threads.

That would be hpn-ssh. No, it's not in 4.0. Will it be included in future releases? Maybe, it needs to be looked at more closely.

And some very interesting gravity waves will be generated!

More information is available at nlr.net including a network map. The first link that went up was between Chicago and the Pittsburgh Supercomputer Center which is run by CMU (where I go). The only problem I think we only have 1 (maybe 2) gigabit links to them, so the bandwidth isn't available onto campus.

The 2000 processor version (10 teraflop peak) going to Pittsburgh Supercomputing Center is about the size of a living room (12'x28'). Compare that with PSC's current HP machine (~3000 processors, 6 teraflop peak) which is the size of a basketball court.

The NSF just invested in a different Cray. However, your dream of drugs prices going down because of drug design on publically funded computers is a bit out there. The drug companies already take advantage of publically funded research to increase their profits. Besides, I think most of the biomed work on NSF supercomputers is much more basic than actually designing of the drug -- think protein folding and membrane channels. Yeah, it's important for drug design but far from actual production of a drug.

So, it's already being done and, no, it won't reduce drug prices.

One project on the Teragrid used 17 Teraflops. The poster hasn't done his/her research on the Teragrid.

...and many scientists with little funding but bright ideas are being left behind.

Care to cite a source?

When you apply to the PACI program you get a grant of Service Units -- i.e. time on the computers. You don't need huge amounts of funding. The requirements state that you need to be a researcher at a U.S. institution. It also helps if you can show that you actually need and can use that kind of computing power.

And, please, distributed computing and supercomputing are not synonymous in terms of what problems they address. Distributed computing cannot replace supercomputers in every case. DC is good for a limited set of problems.

Lastly, an example of Teragrid research: Ketchup on the Grid with Joysticks.

One thing that may be a serious hindrance to Microsoft edging into the supercomputing market is that people who do serious supercomputing are fairly reactionary. Note that I'm referring to people who burn the vast majority of the CPU time at the US's national supercomputing centers - astrophysicists, plasma physicists, molecular dynamicists, people who run QCD (quantum chromodynamics) simulation - and also those who work at government labs doing simulations of nuclear bombs and such. Take a look at the various supercomputing center websites - NCSA, SDSC and PSC - and look up the amount of computer times various groups use. Those doing the most computing, and getting the most science done, are doing truly old-school supercomputing

One of the main reasons for this that that these people (I'm one of them) write and use simulation codes that have a VERY long lifetime - in astrophysics there are codes that are 20-30 years old and still in wide use. This is because these codes first and foremost have to solve whatever equations you're interested in CORRECTLY, and second off, solve them FAST. People base their academic reputations on the results of these codes, and are very interested in making sure that they get the right answer. In some fields (astrophysics being the one I know the most about) people can spend 10 years developing and adding science to a code.

Now, this is a reasonable thing on a unix machine. From the user's point of view, one supercomputer really isn't all that different than another. You just need to figure out where the various libraries and compilers are, but once you do that, you type 'make' and are up and running. So if Microsoft wants to break into the traditional supercomputing market, in order to entice hard-core computational scientists into trying their products they'll have to make it so that codes written for unix systems can be ported over essentially transparently - have the same libraries, the same types of compilers, etc. etc. Frankly, that doesn't seem like a likely thing to me. But then again, I'm one of the crusty old school big-iron computational physicists, so my opinion might not be all that forward looking. All I really care about is what platforms let me get my job done the easiest, and that seems to be the various unix and unix-like systems out there right now.

While much of software will be custom applications, there are common packages that you'll find for simulatiing molecular interactions, doing sequence analysis, etc.

You can check out a list of software available on a CRAY T3E to get an idea.