Slashback: OpenSSH, Bio, Timeliness

Welcome to Slashback, with updates (below) on a handful of recent Slashdot posts. Most importantly, a message regarding OpenSSH 3.3 could save your system from attack -- read it; you might need to pass the word on to your vendor, too.

Things that make you want to bring back thumbscrews. A few days ago, we mentioned the release of OpenSSH 3.3; compared to previous versions, the biggest change in 3.3 is increased emphasis on privilege separation. Today, Theo de Raadt sent word of an OpenSSH vulnerability being worked on by ISS and the OpenBSD team, details of which are expected to be published early next week.

In an announcement sent to bugtraq, he wrote: "However, I can say that when OpenSSH's sshd(8) is running with priv separation, the bug cannot be exploited.

OpenSSH 3.3p was released a few days ago, with various improvements but in particular, it significantly improves the Linux and Solaris support for priv sep. However, it is not yet perfect. Compression is disabled on some systems, and the many varieties of PAM are causing major headaches.

However, everyone should update to OpenSSH 3.3 immediately, and enable priv separation in their ssh daemons, by setting this in your /etc/ssh/sshd_config file:

UsePrivilegeSeparation yes

Depending on what your system is, privsep may break some ssh functionality. However, with privsep turned on, you are immune from at least one remote hole. Understand?

3.3 does not contain a fix for this upcoming bug.

If priv separation does not work on your operating system, you need to work with your vendor so that we get patches to make it work on your system. Our developers are swamped enough without trying to support the myriad of PAM and other issues which exist in various systems. You must call on your vendors to help us."

Theo emphasizes the role of vendor cooperation in making privilege separation work on the full range of systems on which OpenSSH runs. "If the vendors don't start pulling their part," he says in an email, "by the time the bug is posted their customers will be left unprotected. These vendors who do not do the right job and instead just 'sell sell sell' are starting to become annoying. On a lot of systems today, privsep does NOT work well at all. The vendors have not been helping!"

A patched version of OpenSSH could be released as soon as Friday, incorporating vendor patches received by this Thursday.

Read More on Stallman. Vamphyri writes: "Sam Williams, author of 'Free as in Freedom', biography of GNU/Linux founder Richard M. Stallman has gone live with the online free version 1.0 of FAIFzilla.org. The paper pulp version publishers O'Reilly & Associates agreed under the terms of the GNU Free Document License and have their own version up at their site. Williams' site allows for content and corrections to be submitted by readers. He hopes for contributions to be included in later editions of the O'Reilly bio. Also: CGI coders wanted for site enhancement, paragraph and line numbering, searches etc. Maybe a CVS Tree is in order? :)"

"Urpmi Norton" doesn't work for some reason. MrResistor writes "Upon logging in to my computer at work this morning, I was greeted by a virus update notice from McAfee SecurityCenter. The update for today includes W97M/Melissa@MM, and of course McAfees newly manuf^H^H^H^H^Hdiscovered threat, the W32/Perrun JPEG virus (which was also highlighted in yesterdays update). All of the updates in the last week or so have been rated Low or No Threat (except for Perrun, which is "Low On Watch". It seems that in addition to manufacturing new threats, they're also rehashing old threats to keep subscription renewals up. Perhaps it's time for Slashdot to add an Ethics topic?"

Comments

Ethics Topic?

[einstein]

Ethics topic? I thought we had "The Almighty Buck" topic to take care of those pesky ethics...
---

OpenBSD remote hole?

[armie]

Since sshd is enabled by default in OpenBSD 3.1 (OpenSSH 3.1), and privilege separation isn't enabled by default, doesn't that mean OpenBSD 3.1 has a remote root hole?

Re:OpenBSD remote hole?

[evilviper]

I don't have an OpenBSD 3.1 box handy to check to see if priv seperation is enabled by default. However, I know it wasn't on 3.0.

But, we need not jump to conclusions. Theo was saying quite a bit about vendor support, which means he was strugling with the PORTABLE version, he made no mention of the native OpenBSD version, and we have yet to even hear the implications of this bug (hell, maybe it's not exploitable on OpenBSD, just OTHER platoforms running OpenSSH).

Again, don't jump to conclusions.

Theo D.

[The Visiting Priest]

After reading that post about OpenSSH, I
really do not understand how anyone could find
this guy difficult to work with.

To all the OpenBSD lost its claim posters..

[antis0c]

This could be flamebait, but it should be said.

Consider this, would you rather use an Operating System, where the community just shrugs off the frequently once a week remote holes with simply, "go grab the updates" ..

.. or an Operating System where the community is surprised and in disbelief that a remote hole was found after 5 years which causes entire community to start bitch fights over the right to claim its the most secure Operating System still, despite the fact the remote hole was found by the Operating System developers, and fixed before it has actually been exploited.

You don't have to be Stephen Wolfram to figure this one out.

Re:For FreeBSD users:

[John Hasler]

"For linux users, you guys are outta luck."

Nonsense. The Debian package is already out.

Linux BIAS

[jolan]

funny how this didn't make it into the main article:

We've been trying to warn vendors about 3.3 and the need for privsep,
but they really have not heeded our call for assistance. They have
basically ignored us. Some, like Alan Cox, even went further stating
that privsep was not being worked on because "Nobody provided any info
which proves the problem, and many people dont trust you theo" and
suggested I "might be feeding everyone a trojan" (I think I'll publish
that letter -- it is just so funny). HP's representative was
downright rude, but that is OK because Compaq is retiring him. Except
for Solar Designer, I think none of them has helped the OpenSSH
portable developers make privsep work better on their systems.
Apparently Solar Designer is the only person who understands the need
for this stuff.

Theo de Raadt's message in full

[joe_bruin]

From: Theo de Raadt [deraadt@cvs.openbsd.org]
Subject: Upcoming OpenSSH vulnerability

There is an upcoming OpenSSH vulnerability that we're working on with ISS. Details will be published early next week.

However, I can say that when OpenSSH's sshd(8) is running with priv seperation, the bug cannot be exploited.

OpenSSH 3.3p was released a few days ago, with various improvements but in particular, it significantly improves the Linux and Solaris support for priv sep. However, it is not yet perfect. Compression is disabled on some systems, and the many varieties of PAM are causing major headaches.

However, everyone should update to OpenSSH 3.3 immediately, and enable priv seperation in their ssh daemons, by setting this in your /etc/ssh/sshd_config file:

UsePrivilegeSeparation yes

Depending on what your system is, privsep may break some ssh functionality. However, with privsep turned on, you are immune from at least one remote hole. Understand?

3.3 does not contain a fix for this upcoming bug.

If priv seperation does not work on your operating system, you need to work with your vendor so that we get patches to make it work on your system. Our developers are swamped enough without trying to support the myriad of PAM and other issues which exist in various systems. You must call on your vendors to help us.

Basically, OpenSSH sshd(8) is something like 27000 lines of code. A lot of that runs as root. But when UsePrivilegeSeparation is enabled, the daemon splits into two parts. A part containing about 2500 lines of code remains as root, and the rest of the code is shoved into a chroot-jail without any privs. This makes the daemon less vulnerable to attack.

We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny). HP's representative was downright rude, but that is OK because Compaq is retiring him. Except for Solar Designer, I think none of them has helped the OpenSSH portable developers make privsep work better on their systems. Apparently Solar Designer is the only person who understands the need for this stuff.

So, if vendors would JUMP and get it working better, and send us patches IMMEDIATELY, we can perhaps make a 3.3.1p release on Friday which supports these systems better. So send patches by Thursday night please. Then on Tuesday or Wednesday the complete bug report with patches (and exploits soon after I am sure) will hit BUGTRAQ.

Let me repeat: even if the bug exists in a privsep'd sshd, it is not exploitable. Clearly we cannot yet publish what the bug is, or provide anyone with the real patch, but we can try to get maximum deployement of privsep, and therefore make it hurt less when the problem is published.

So please push your vendor to get us maximally working privsep patches as soon as possible!

We've given most vendors since Friday last week until Thursday to get privsep working well for you so that when the announcement comes out next week their customers are immunized. That is nearly a full week (but they have already wasted a weekend and a Monday). Really I think this is the best we can hope to do (this thing will eventually leak, at which point the details will be published).

Customers can judge their vendors by how they respond to this issue.

OpenBSD and NetBSD users should also update to OpenSSH 3.3 right away. On OpenBSD privsep works flawlessly, and I have reports that is also true on NetBSD. All other systems appear to have minor or major weaknesses when this code is running.

(securityfocus postmaster; please post this through immediately, since i have bcc'd over 30 other places..)

The Alternative to OpenSSH or SSH (commerical)

[Tadghe]

LSH (http://www.net.lut.ac.uk/psst/)

I love SSH. It's been installed on my boxen (regardless of OS) since it was stable enough to kill off telnet.
My problem with both the announcement as well as the patch is thus.

1. Theo nor any of the posters I've seen are willing to tell us what the hell is broken. Only that we must upgrade. That just don't cut it, I won't blindly patch without an idea of what is broken. The Debian security release summed it up best.

"Theo de Raadt announced that the OpenBSD team is working with ISS
on a remote exploit for OpenSSH (a free implementation of the
Secure SHell protocol). They are refusing to provide any details on
the vulnerability but instead are advising everyone to upgrade to
the latest release, version 3.3.

This version was released 3 days ago and introduced a new feature
to reduce the effect of exploits in the network handling code
called privilege separation. Unfortunately this release has a few
known problems: compression does not work on all operating systems
since the code relies on specific mmap features, and the PAM
support has not been completed. There may be other problems as
well."

2. Sudden, lack of belief in Full disclosure. Am I the only guy who remembers the days before full disclosure? The OpenBSD Security policy ( http://www.openbsd.org/security.html ) is pretty point blank (to quote)
"we believe in full disclosure of security problems. In the operating system arena, we were probably the first to embrace the concept. Many vendors, even of free software, still try to hide issues from their users"
I think posting a "fix" (ok, workaround) and not telling anyone *why* they should use it is "try[ing] to hide issues from their users"

I'll be firing up R.A.T.S and checking out LSH ( http://www.net.lut.ac.uk/psst/ ) (GNU'd SSH2ish) for my needs from here own out.

and yes, this needs a rant tag and yes I know the OSSH and OBSD teams are seperate, but they share enough philosophy and team members that I gather they have the same opinion on security.

Re:The Alternative to OpenSSH or SSH (commerical)

[Virtex]

1. Theo nor any of the posters I've seen are willing to tell us what the hell is broken. Only that we must upgrade. That just don't cut it, I won't blindly patch without an idea of what is broken. The Debian security release summed it up best.

In the world of full disclosure, it's generally considered polite to initially only notify the vendor of a product and allow them a grace period to fix the security hole. This way, when the security hole is publicized, users will (hopefully) have a patch or upgrade to secure their systems. The question here isn't whether Theo is correct in holding back the details of the exploit (which I believe he is correct in doing), but whether he should have said anything about the problem at all before releasing the full details. I think his goal was to pressure the OS vendors into helping him fix the problems in his code.

I won't say whether his choice is right or wrong, but I won't chastise him for protecting my security, either.

Re:The Alternative to OpenSSH or SSH (commerical)

[Eil]

You misinterpreted the entirety what he was trying to say. If I were in a crankier mood I'd ask you if you even read the post.

In a nutshell, he said this:

1. There is an exploitable bug in all current versions of OpenSSH.
2. We're working on a patch, but it's not done yet.
3. When it is, we'll tell you all exactly what was wrong and how we fixed it.
4. In the mean time, you can download the latest 3.3 patch and enable privilege separation to completely protect yourself from the vulnerability.

That just don't cut it, I won't blindly patch without an idea of what is broken.

There isn't a patch yet. Theo clearly stated that a patch and an explanation will be forthcoming at the same time. The whole reason he announced it early is to get admins to fix thier systems before the nefarious hackers could develop an exploit for it. (As another poster noted, it's incredibly easy for a nefarious hacker to develop an exploit if you have the source code to versions of the program with the bug and a version without. That is perhaps one of the few downfalls to open-source.)

You'll save yourself a lot of typing and needless jumping (to conclusions) if you read a bit more carefully next time.

...and my analysis

[joe_bruin]

replying to yourself is always a bad thing, but here goes...

if you cut through the bullshit (theo certainly has an interesting way of putting things), what he's saying is this:

there's a hole in sshd. we are working on a patch. if we release it now, you are all f'd, because all your systems will be compromised before you have time to patch them. we are giving you the next week to update your sshd, so that you are no longer vulnerable when we publish the bug+patch. yes, the new sshd has the bug, but is not vulnerable to it. if we fixed it now, the black hats will diff the results and be able to develop a compromise, and you still won't have a patch. oh yeah, we need your vendors' help so that you're all safe by next week.

make sense?

Re:For FreeBSD users:

[ewhac]

Thanks. I'll update my firewall tonight. And I'll be READING, DAMMIT!

Oh, and can I throw in a cheap plug for the FreeBSD Cheat Sheets while I'm here? It has a much more easy-to-follow tutorial on how to cvsup your ports, though it doesn't go into the theory behind it.

Schwab

They Had _Better_ Already Know

[John Hasler]

"Read it; you might need to pass the word on to your vendor, too."

If you need to pass the word on to your vendor you need a new vendor.

Re:For FreeBSD users:

[Glytch]

Or, install the source to openssh by hand, and solve all the damn pam errors.

At this point I would like to thank Patrick Volkerding yet again, this time for being dead-set against the wretched abortion known as PAM.

Re:ssh vulnerability disclosure?

[John Hasler]

"Another interpretation would be that he wants vendors to port the privilege separated version of sshd to their platforms..."

That is exactly what he wants.

How far BACK?

[evilpenguin]

Leaving the OS Wars aside (I run Linux, yes, but I also run FreeBSD, and I would run OpenBSD if they would just get unanal about bootable iso's): Okay, swell. 3.3 has a hole.

How far BACK does this hole extend? Does my 3.1 have it? Does EVERY copy of OpenSSH since the dawn of time have it? Can someone make this clear to me? Is it only versions that have privledge separation? Where is the boundary of this bug?

A reply to all those bitching

[hayden]

Isn't it wonderful how a security hole in an open source program brings forth all the security experts on Slashdot. And they flame someone who know a shit load about it and is dedicated to improving security to the point of being a complete arsehole.

Anyway. My guess is that this hole is something substantial, possibly very plateform dependent and any patches aren't going to be trivial. Seeing as all you people who felt the need to use fsck in their posts more than once know about as I do about this then my assumptions are as good as yours (and I don't feel the need to use the word fsck as an expletive once). Non-trivial patches mean that commercial vendors are going to take for ever to release final patches and if you are running anything open to the internet then it's likely to be ssh. Add it all up it means this could be very bad.

Now the OpenSSH team is actually two. One that develops new stuff and does code audits specifically for OpenBSD and another that takes that and ports it to other architectures.

All those bitching about full disclosure, you manage to be completely committed to a cause, idiots and miss the point of full disclosure all at once. If the bug is bad then releasing it when only the OpenBSD version of OpenSSH is patched would be an absolute security nightmare. Giving vendors advance notice is very much required in this case. When the vulnerability is announced then I'm sure it will be fully disclosed which will provide the opportunity to test a system for vulnerabilities.

As for you people who are saying Theo is being pro-OpenBSD, read the above paragraph again and answer this question. If Theo really wanted to really rip on other OS's then what could he have done with this announcement? Only OpenBSD not vulnerable and with mindless full disclosure to cover his arse. You do the maths.

The fact is Theo is a complete arsehole when it comes to security. Some see this as not a bad thing. With OpenBSD security is pretty much everything. To the vast majority of other "vendors" security is something they also do and with this Theo has a legitimate gripe. He has got a shitty reception from other vendors to something that will make a vital link in the security chain more secure. Is he making a point of this? Probably. Is he right to do that? Depends on your point of view. If it gets the "vendors" off their arses and add support for priviledge seperation in their ports then would this be considered a good thing? Most definately.

When it boils down to it, Theo would be well within his rights to patch the OpenBSD version of OpenSSH (by using priviledge seperation) and hanging the other vendors out to dry. He didn't. Deal with it.

Does NOT work with 2.2 kernels.

[netfunk]

This just bit me in the ass, so I'm passing it on.

The privilege separation code in OpenSSH 3.3 does not work with 2.2 Linux kernels.

It relies on mmap() semantics that aren't supported before kernel 2.4 (maybe 2.3.x). OpenSSH will configure, compile, and install successfully. It will start up, but it will NOT accept connections.

Your clients will get a "broken pipe" message, your syslog will get an "mmap: invalid parameter" message.

The solutions are:

• Upgrade to kernel 2.4 or higher.
  
• Don't compile in Privilege Separation.
  
• You might be able to compile privsep in and disable it, but I couldn't get this to work. Maybe I had a typo in my config file. I dunno.
  

I didn't see this anywhere until I dug into my syslog and then the OpenSSH mailing list. You have been warned.

If you do have kernel 2.4, you should read README.privsep in the openssh source distro, since you need to create a special directory and user/group for this (which also bit me in the butt...even if sshd had worked on 2.2, when I restarted it remotely, it didn't come back up because it didn't have that user...yeah, yeah, rtfm. :) )

Good luck to everyone.

--ryan.

Re:DOES work with 2.2 kernels.

[Svenne]

You just need to turn of compression. All you have to do is add the following line to your sshd_config:

Compression no

I'm running OpenSSH 3.3 with privilege separation activated on a 2.2.19 kernel.