Due Diligence?

ekr writes "The OpenSSL remote buffer overflows discovered at the end of July got a lot of press here on /. But how many people actually fixed their machines? I decided to study this question, and the results are kind of depressing. Two weeks after the release of the bug, over two thirds of the servers I sampled were still vulnerable. Even two weeks after the Slapper worm was announced, a third of the total servers were vulnerable. The paper can be found here in PDF or Postscript."

Comments

I wiped my OpenBSD boxes, reinstalled, patched

[JimmytheGeek]

I noticed connection attempts from Korea just after the announcement and decided it was time to nuke the the boxes from orbit. Not much point in having an O-BSD box you are only mostly sure of.

I had some angst with RedHat boxes, though. The update mechanism didn't change the reported version number of OpenSSH. Annoying.

Vulnerability Check

[Karamchand]

How does one check if a server is vulnerable without actually "breaking in", i.e. make oneself liable to prosection?
I skimmed through the PDF but could not find anything about this.
Thank you.

Re:Vulnerability Check

[FattMattP]

I can't believe that you got modded up for just saying "I didn't read the article." If you had bothered to read it and not skim it, you would have found the description of how they check it on page five:

The patches provided by the OpenSSL project do not change the version number. As a consequence, we cannot simply examine at the advertised version number in order to determine whether or not they have been applied. However, due to the nature of the SSLv2 problem, it's still relatively straightforward to determine whether or not a server is vulnerable and therefore by excluding servers which advertise a newer version number identify whether or not a given server has been patched.

[snip]

It's possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It's easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error.

This right under the section called "Detecting Vulnerable Servers." Maybe you should read the article next time before you post.

Re:Vulnerability Check

[DVega]

To check the version number on the "Server" field of the HTTP Reply is not enough. Some patched versions did not change this header. On the PDF they explain on page 5 that they discovered a way to check the vulnerability without crashing the server.

" It s possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It s easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error. In an unpatched version of OpenSSL, the server overruns the key_arg with whatever the extra byte is. However, as we shall now show, this overrun is harmless."

Securing OpenSSL

[Exmet Paff Daxx]

Some points to consider:

• Fear. Most Linux users are probably reeling in shock from the recent trojan inserted by elite hacking group ADM into the libpcap distribution. The old standby argument that 'checking the MD5 signatures' will save you has become null & void; ADM replaced the MD5 signatures too. The only reason the trojan was detected was because of the Google cache! This kind of thing probably has most users afraid to move to anything recently released that hasn't been extensively peer reviewed.
  
• Ignorance. Since the Slapper worm only contains offsets for a handful of platforms, many flavors of Linux are 'immune' to automated infection. While blackhat groups have offsets for nearly every implementation of Apache/SSL in existence (yes, even you x86 Solaris people), this threat isn't considered 'immediate' enough to justify the third point:
  
• Sloth. Upgrading your OpenSSL isn't as easy as it could be. You actually have to recompile Apache with ./configure flags to link it to the new version of OpenSSL which you just recently downloaded (it's not trojaned... right?). Sounds easy, but for a production server that hasn't been touched in a year, this tends to make people really nervous
  

All of this points to the fact that there is a fundamental flaw in the way that the Open Source community is securing their software. Putting MD5 signatures on the same server that the software is available from isn't even close to secure - Dave Aitel of Immunity Security keeps hammering on this point in BugTraq. And we're going to see even more of this 'Upgrade Fear' as more and more distributions get trojaned - Slash is probably next on the list.

We need to look at existing, successful solutions to this problem (like Windows Update) and catch up. Now.

Re:Securing OpenSSL

[Albanach]

There's also the issue of those running servers who, sensibly, have either gcc set to non executable, or simply have no compiler installed. It's much more difficult to compile code when there isn't a compiler, and with no gcc available, slapper can't do an awful lot.

Sure something else might come along that can, but as you point out, if you're running a server that's been up a year, changing things is never comfortable, and if you know slapper isn't going to infect you, there's much less motivation.

Re:Securing OpenSSL

[Flarners]

My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part.

That's exactly the problem the parent poster was trying to highlight! Sure, you can blindly trust that the Debian servers are secure and un-trojanned and let apt-get install it without so much as checking a key signature. Even if you do configure apt to check the signature, it fetches the public key from the same server as the packages. Thus, an attacker can easily trojan your machine with man-in-the-middle or DNS attacks, sending you to an update site with a trojanned package signed with his own public key. If someone sneaks into the real debian servers, subverting apt-get is as easy as:

• Upload new "developers" key signature.
• Sign trojanned package with new signature.
• Upload trojanned package.

and it's Game Over. Of course, Red Hat and Mandrake's solutions aren't much better. The key needs to be stored with a trusted entity like Verisign, which is how Windows Update and other commercial-grade updating systems ensure the integrity of their packages. You've never heard of Windows Update being trojanned, have you?

Re:Securing OpenSSL

[drunken monkey]

The only thing I've used the MD5 sig is for verifying download completeness. That and generating a session id string using a secret string in combination with publicly known strings.
But that secret string is as good as the security of the server it exists in.

I have higher degree of trust for the gpg key that comes on the RedHat CD's from official RedHat boxes. Nothing against other distro's, RedHat is just an example.

narbey

Servers should disable themselves...

[tjansen]

IMHO the right way is to write a system that disables servers automatically if a vulnerability is known and the administrator did not fix it in n days. Either the functionality is integrated into the daemons, and they check (via http, mail, whatever) every day whether they are affected by a problem. Or each system should run a daemon that controls all server software.
It should warn the admin before it turns the server off, of course, but a broken unmaintained server is always better than a hacked server.

Re:Servers should disable themselves...

[aridhol]

What happens when the author's website becomes obsolete? Security may be up-to-date, but you now have a bad webpage. Will the server shut itself down, or let you run with a possible compromised server? Also, consider the effect if the timing is set in the source. The first of every (month|year|week) will show an enormous load on the update server. What happens when the software can't talk to the update server? Assume everything's fine? When someone hacks the update server with a new, trojaned form, suddenly everybody will have it. Check the server's logs, you now have a list of compromised hosts.

Other than that, it's a good idea...

We Fixed Ours

[Shackleford]

I worked at an ISP at the time that this was happening and we were quite well aware of these vulnerabilites. We often referred to CERT when looking for vulnerabilities that may have affected us. It was through sites like those that we found out about the problems with OpenSSL and we made the necessary changes. I'm not sure why it was found that many others didn't do what was necessary. Perhaps there are many admins that don't understand that they need to keep themselves up to date on these matters, and of course, they are often busy with many other tasks. It's not easy being an admin. Maybe that's why there is a System Administrator Appreciation Day.

When to Patch

[Crispin Cowan]

Readers interested in this topic may be interested in this paper that we presented last week at USENIX LISA 2002:
Timing the Application of Security Patches for Optimal Uptime

Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, and Chris Wright
WireX Communications, Inc. http://wirex.com
and
Adam Shostack
Informed Security http://www.informedsecurity.com

Security vulnerabilities are discovered, become publicly known, get exploited by attackers, and patches come out. When should one apply security patches? Patch too soon, and you may suffer from instability induced by bugs in the patches. Patch too late, and you get hacked by attackers exploiting the vulnerability. We explore the factors affecting when it is best to apply security patches, providing both mathematical models of the factors affecting when to patch, and collecting empirical data to give the model practical value. We conclude with a model that we hope will help provide a formal foundation for when the practitioner should apply security updates.

Crispin
----
Crispin Cowan, Ph.D.
Chief Scientist, WireX Communications, Inc.
Immunix: Security Hardened Linux Distribution
Available for purchase

Re:When to Patch

[ekr]

Indeed. In fact, I cite your paper in mine. :)

Interestingly, if you look at the curves of people's behavior, you can see that the algorithm they're apparently following is pretty different from the one you recommend. Namely, people who are going to upgrade do so pretty much right away (mostly within 10-14 days of the stimulus that seems to have forced them to upgrade). The second interesting observation is that a substantial number of people seem to wait for a vulnerability to be released before they upgrade. One question that would be interesting to ask in the context of best practices would be how long on average vulnerabilities circulate before they are announced.

Re:this is why...

[Anonymous Coward]

I recently did some work for a colo client with an unpatched stock FreeBSD 2.6 machine. It was in perfect condition (everything worked fine, no hidden files, no core dumps, nothing funny in the logs going back months). No only that, but all the default old-skool services were running like rexec or whatever. My mouth was hanging open in disbelief but apparently it never got hacked (or it got hacked so well I couldn't find anything). The red hat linux machines on the same lan that are supposedly kept up-to-date by the clients get hacked regularly.

I'm starting to wonder if installing linux 0.99 wouldn't be such a bad idea.....

Re:Damn MCSEs

[LostCluster]

But I've said it before. Linux is now getting to the point that the clueless people who poorly run Windows boxes are now moving over to Linux. These admins are too lazy or clueless to properly secure ANY operating system, no matter how well it's designed.

Click 'Install'

[KFury]

On the mac, I just set 'Suftware Update; to run daily, and I click 'install' when it says there's a security fix.

By default, users only have to click one button (the default button) to keep their Mac-flavored BSD secure.

And they don't have to subscribe to mailing lists or be security geeks. They could be your mom and still get it right.

Not trying to rip on your mom.

I don't even know her!

No, seriously. That wasn't me, that time at the Quaker Steak n Lube!

Sysadmins need to pursue intrusion attempts

[wowbagger]

I know of a Linux system that logs and reports intrusion attempts by CodeRed/Nimda, Slapper, et. al., and mails a report to a system admin every morning.

The system admin wasn't pursuing these reports. I asked why.

His response - "Well, those are attempts to exploit a Windows server, and this is a Linux box, so they don't matter."

I made the counterpoint "If one of your system was infected, wouldn't you want to be told about it?"

If every systems admin would take the time to track down the Code Red attempts on their systems, and notify the responsible parties whereever possible, then a lot of the unpatched systems would be shut down (if not by their administrators, then by the ISP supplying connectivity).

I just don't understand an admin with an attitude like that.