OpenSSH Vulnerability Disclosed, Version 3.4 Released

Dan writes: "OpenSSH 3.4 has been released and will be shortly available on all mirrors. All versions of OpenSSH's sshd between 2.9.9 and 3.3 contain an input validation error that can result in an integer overflow and privilege escalation. OpenSSH 3.4 fixes this bug." And kylus writes: "The previously mentioned vulnerability in OpenSSH has been disclosed by ISS X-Force today on the BugTraq list. This is a potential remote root compromise, and while there is a workaround, it's advised that users upgrade to version 3.4 as soon as they can."

Comments

Why was it kept hush hush?

[wub]

Does anyone know why it was kept so hush? It's so unlike the open source community to not just put it all right out there.

Re:Why was it kept hush hush?

[datajack]

The last couple of weeks has demonstrated precisley what is wrong, from an end-user standpoint, with 'responsible disclosure' (yeuch!)

I don't know why, but ISS seemed to get under the skin of a lot of security researchers with their release of details of the recent Apache problem. This release was *directly responsible* for someone (I forget who, but thanks are due) to code a fairly simple work-around in the form of an Apache module, so that people can quickly install some protection whilst waiting for a fix to appear, and ancilliary apache addons (modssl, php etc.) to catch up with the new Apache release, so we are now maore-or-less protected whilst compiling, testing and installing new versiona of about half a dozen bits of software. Because of this release, the problem can be handled in a calm and non-disruptive manner.
Oh, and someone reported being hit with similar symptoms to this problem, well before ISS released the details.

Take that in contrast to when this OpenSSH problem hit the net. I was well aware of OpenSSH 3.3 and the new security features, and had a plan to wait till the next release (to check for implementation problems) and then upgrade all our servers in an orderly manner.
However,this morning, I opened Bugtraq to find a load of peeps that should know better (i.e. OpenSSH developers) screaming that there was a major root-exploit in the code I was currently running, that I had to upgrade immidiately, and no, they won't tell me what the problem is. Based on the available information, I made a judgement call, and suspended all incoming ssh access at the firewall until I could upgrade. As you can imagine, this pissed off a lot of customers. I also had to then reschedule my day to get, test and install this new version of SSH - I did not have time to put it through our usual QA process - to get us operational again.

To add insult to injury, when the details were released, it turns out to be a problem with a feature we do not even use and a simple config change was a suitable work-around.

Who do I get to bill for our (useless) 3 hour downtime?

Was there ever a working 'sploit?

[toupsie]

I have read much about this problem in OpenSSH and fearing the worst...checking logs to see how often my SSH version was scanned. However, as far as I know, I haven't had any break ins using a SSH exploit. Thank God for TCP Wrappers, at least that helps when you find out about these things.

Did any one of the many black hat groups out there actually work up a exploit or was this caught in time that it was just a possibility of being exploited?

Re:Was there ever a working 'sploit?

[evilviper]

I have it on good authority from a collegue of mine (okay, fine, he's a script kiddie, but he keeps up on all the '0day' exploits, so he comes in hanndy) that not only is there an exploit for OpenSSH, but there is one for the latest version of SSH.com's ssh server (3.2.0 IIRC).

He's a rather reliable source. In fact, he let me know that Apache on essentially ANY platform was vulnerable, long before IIS or Bugtrack had that info. Interestingly enough, a few days later, one of our servers (not one that I admin-I NEVER run Apache as Root) was Root'ed. It was just a small workgroup server, no real harm done.

Cheers, Theo

[gorf]

All that you need to do, as far aas I understand it, is turn Challenge/Response authentication off (which nobody uses anyway). So the line in /etc/ssh/sshd_config reads:

ChallengeResponseAuthentication no

and then restart the daemon.

Big deal.

I don't see any need to upgrade anything. Yes, privilege separation is nice in terms of future security, but I prefer the (more likely) known stability of software that has been in use for a while.

Debian security policy is that vulnerability fixes are backported (to avoid adding anything that could cause instability or further insecurity); this was made impossible by Theo's and ISS' advisory which lacked any details about the exploit. This may have been justified had the exploit not be able to be prevented by a simple configuration change (in order to give administrators time to prepare an upgrade their systems), but not for this.

Cheers, Theo, you just cried Wolf for the entire community. If there ever is a hole major enough that everyone should (or might want to) upgrade to a version which is by nature immune rather than give away the exploit by releasing a patch, noboby's going to believe you now, and probably not anyone else either.

Re:Cheers, Theo

[BJH]

Hmmm, that's weird... disabling ChallengeResponseAuthentication causes OpenSSH to show the username and hostname when asking for the password, whereas before it only showed the "Password:" prompt.
(In other words, it now displays "USER@HOST.NAME's password:")

It's not really a problem, since (obviously) you need to know what user you're trying to log in as, but it would suggest that it goes through different code paths when the option's enabled compared with when it's disabled, even if you're using the same authentication type each time.

What is ChallengeResponseAuthentication?

[rherbert]

How does this authentication method work? I just disabled it, and I was still able to log in using my RSA keys and password authentication (which are the only methods I use). The documentation says it's for s/key authentication, but what is that? How common is this authentication method, and who would use it?

okay, let me get this straight.

[Dr. Awktagon]

Okay, busy morning but glancing at the news, here's what I see:

There was a bug in the challenge/response code between 3.0-3.2.3. In fact, it's an "overflow" according the advisory. This means to me, it should be a fairly easy fix. Quote:

It is possible for a remote attacker to send a specially-crafted reply that triggers an overflow. This can result in a remote denial of service attack on the OpenSSH daemon or a complete remote compromise.

In addition, this overflow only works when SKEY and/or BSD_AUTH is enabled. But this seems to be "not enabled...in many distributions". How about Linux? However, OpenBSD has BSD_AUTH enabled (natch). Quote:

At least one of these options must be enabled before the OpenSSH binaries are compiled for the vulnerable condition to be present. OpenBSD 3.0 and later is distributed with BSD_AUTH enabled. The SKEY and BSD_AUTH options are not enabled by default in many distributions.

And now to add insult to injury, the 3.3 I installed yesterday has a new different buffer overflow, so I have to jump to 3.4 now (does it have any new bugs too?)

I don't like to jump versions on production machines. I like to fix what's running for minimum disturbance.

Can someone please explain why this vulnerability was handled this way? Why wasn't there a maintainance release that just fixed the @#$@#% problem?

I know: since the bug affected so many people, Theo thought it would be better to bury the problem in his privsep code, instead of fixing it and letting the blackhats run "diff" and find it for an easy 0-day-'sploit. In other words, security by obscurity, just like the big guys. That stinks, if you ask me.

On the other hand, I charge by the hour when I upgrade my client's machines. So thanks Theo! $-)

MacOSX update ?

[selderrr]

I wonder is Apple is going to release a minor OSX update for this. If they do, they prove themselves worthy supporters of OpenSource and creators of an OS that truly respects security.
If they don't, welll, they're still a 100 times more secure than 95% of the market

Affects who?

[MSG]

So, what do we know about who is affected? Immediately after reading the announcement, I checked Red Hat Linux's build of OpenSSH. The configure script positively reports that the affected authentication mechanisms are not available. 'ssh -v' does not indicate that challenge-response authentication methods are available either. I imagine that other Linux distros are similar?

RHL configure output:
OpenSSH has been configured with the following options:
...
Smartcard support: no
S/KEY support: no
BSD Auth support: no

Code Audits, the UNIX security model

[Nerant]

How secure any software you're running on your system(s) depends on the quality of the code audit done on the code. I'm not judging the standard of the OpenSSH's team code audit here: things will slip through given the inherent complexity of software.
Privilege separation is a step in the right direction. By minimising the amount of code running as root, it makes code audits simpler and more through, and minimises the damage any potential exploit could do in the part running as a normal user.
Stepping back from the situation, privilege separation is just a bandaid for the lousy UNIX security model. Yes, granted, UNIX / Linux (i have no experience with other UNIX systems, so i shall reserve comment) have a security model that's used, as compared to Windows 9X. (Windows 2K has a security model, but the MS culture makes it difficult to administer it, but i digress). However, this security model is too coarse grained: it grants "root" too many privileges, too many rights. This is evident in the move towards ACLs, for example in NSA's SE Linux, as well as LIDS.
We need to overhaul the security model to one that's not prone to insecure software as much. Note I said as much:No system is 100% secure, and I don't want to replace my system with a toaster.
Appreciate feedback. Thanks. =)

How soon until djbssh?

[GregGardner]

I can't wait until djb decides to write his own ssh. You can say what you want about djb and his personality, but he does know how to write some secure software. Sure, it's not the easiest thing to install and you have to create a boatload of users, but privilege separation has been in qmail since 1.0.0. Theo is getting around to it in v3.3? Never heard of any root compromises from qmail or djbdns. So hopefully this latest hole in OpenSSH has annoyed djb to the point of rolling his own.

telnet+SSL+certs

[TheLink]

if you don't need all the features of SSH, try telnet+SSL+certs

It's likely to be safer.

I've been using stunnel+telnet for years and I have had to patch/upgrade a lot fewer times than people using SSH.

Cheerio,
Link.

What is wrong with this picture

[nelsonb]

What the hell is wrong with the people at ISS? This is the second "incident" in as many weeks. The message from the Theo at OpenSSH and other Linux Vendors said the info AND the realfix would be released early next week. This certainly seemed like a very responsible method of alerting people. Give everyone a week to upgrade to 3.3 and enable an option that could help mitigate any potential damage and then release a fixed version of the program and the full details. This gives large production houses enough time to get the new version/config through change control and even gives admins who don't read bugtraq lists time enough to hear about throughanother channel. Everything was working out pretty well and ISS has to go and screw up a pretty good plan. Does ISS have a problem with queued up advisories and techs with itchy trigger fingers? Does Time some how run differently in Atlanta? I guess another lame self-justification will be forthcoming from ISS, but there is no excuse for this. What about the little people? Were you in such a hurry to get your X-Press Update advertisement out that you don't even consider the ultimate end-user? Is it so easy to forget that not everyone is in organizations that discover or releases 0-day info amongst themselves? Most administrators don't read bugtraq, and those thatdo received a polite, clear note from Theo:

Monday, June 24, 2002 11:22 PM

There is an upcoming OpenSSH vulnerability that we're working on with ISS.
Details will be published early next week.
....etc etc etc

Now ISS has up'd the ante and released it justa day and a half later. 1 and 1/2 days isn't a lot to verify that a production environment will not be adversely effected byANY new/changed element. So it would seem that "working with ISS on this issue"is synonymous"we are waiting to get blindsided". This also leads into another interesting issue. Why did ISS's reckless announcement take minutes to get through bugtraq and the OpenSSH's initial, responsible warning take 24+ hours to process? I can plainly see that Theo's letter was sent on Monday but for some reason only gets here today. I know that SMTP mail is slow..but I don't thinkmy server isTHAT slow. Fortunately, it showed up on the vuln-watch list as well and we were able to help spread the word.

> X-Force is aware of active exploit development forthis vulnerability.

I don't know if I really even believe you on this certainlyyour recent actions are not that of a company that seeks to garner trust. Of course the minute anyone suggests there is a problem with product XYZ, thousands of bored people are going to start poking around "actively" trying to develop an exploit! But blind testing from scratch would certainly have taken longer than the proposed "quiet week" before publishing details.So, lets suppose it was a more informed testing. So who knew enough about this to let it out? ISS and the OpenSSH dev team. One is made up of hard working developers who love aprogram enough give their time away to make a really great product. The other is composed of people who routinely socialize with the underground "active exploit development" community. In my opinion, at least one side would have absolutelyno motive leak their information. So I propose: A: Your analysis of the exploit development process was faulty B: there was no active development for an exploit, and you released the info for your own good.C. Someone's teamis leaking information.

In any event, there no need for any furtherunderground exploit "R&D"; everyone now has the diff blueprints to get directly to the end goal. Granted, there are people out there intelligent enough totake the time find the issue and to code an exploit without this knowledge. But these type of people wouldn't likely release it to the general populace, instead it would be used for select targets. Targets that would most likely already have security teams in placeand be up on warnings and patches. Instead we have a patch diffs in the hands of everybody and now lower skilled programmers can code the exploit. These people will spread the exploit far and wide simply for fame; only this time the targets will be everyone.

No one wins with this route you have chosen ISS. You and your X-force team used to be a respected group in my book. In the past they have provided valuable information to the security community and helped companies across the world to better secure themselves, but the handling of this and the Apache vulnerabilities are shining examples of how NOT to do things. So much for ISS being a "Trusted" center of knowledge. Trust and honor are coins you can only spend once.

Nelson Bunker, CISSP VP of Security Critical Watch The opinions expressed in this advisory and program are my own and not of any company. The big print giveth, the little print taketh away