Due Diligence?

ekr writes "The OpenSSL remote buffer overflows discovered at the end of July got a lot of press here on /. But how many people actually fixed their machines? I decided to study this question, and the results are kind of depressing. Two weeks after the release of the bug, over two thirds of the servers I sampled were still vulnerable. Even two weeks after the Slapper worm was announced, a third of the total servers were vulnerable. The paper can be found here in PDF or Postscript."

Comments

It's not just laziness...

[Anonymous Coward]

Many systems administrators aren't full-time and have other responsibilities. Keeping up-to-date with every security patch is very time consuming and sometimes management doesn't understand this and doesn't allocate resources for it as long as things are "working".

Re:It's not just laziness...

[dfn5]

Unfortunately keeping up with patches is a very important part of any security strategy. I am all for letting companies do things their way, but if admins don't allocate more time to security and patching then I'm afraid the government will do more than just recommend actions for Security on the Internet and will start mandating stuff. I for one don't want that to happen.

Bottom line? Improve your security while you still have the rights to do it yourself.

this is why...

[mr_gerbik]

This is why I run Windows 3.11. No worries about falling behind and not installing the latest fixes.

Securing OpenSSL

[Exmet Paff Daxx]

Some points to consider:

• Fear. Most Linux users are probably reeling in shock from the recent trojan inserted by elite hacking group ADM into the libpcap distribution. The old standby argument that 'checking the MD5 signatures' will save you has become null & void; ADM replaced the MD5 signatures too. The only reason the trojan was detected was because of the Google cache! This kind of thing probably has most users afraid to move to anything recently released that hasn't been extensively peer reviewed.
  
• Ignorance. Since the Slapper worm only contains offsets for a handful of platforms, many flavors of Linux are 'immune' to automated infection. While blackhat groups have offsets for nearly every implementation of Apache/SSL in existence (yes, even you x86 Solaris people), this threat isn't considered 'immediate' enough to justify the third point:
  
• Sloth. Upgrading your OpenSSL isn't as easy as it could be. You actually have to recompile Apache with ./configure flags to link it to the new version of OpenSSL which you just recently downloaded (it's not trojaned... right?). Sounds easy, but for a production server that hasn't been touched in a year, this tends to make people really nervous
  

All of this points to the fact that there is a fundamental flaw in the way that the Open Source community is securing their software. Putting MD5 signatures on the same server that the software is available from isn't even close to secure - Dave Aitel of Immunity Security keeps hammering on this point in BugTraq. And we're going to see even more of this 'Upgrade Fear' as more and more distributions get trojaned - Slash is probably next on the list.

We need to look at existing, successful solutions to this problem (like Windows Update) and catch up. Now.

Re:Securing OpenSSL

[Psiren]

Upgrading your OpenSSL isn't as easy as it could be.

Any decent distribution will do this work for you. That's what they're for after all. My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part.

Re:Securing OpenSSL

[Albanach]

There's also the issue of those running servers who, sensibly, have either gcc set to non executable, or simply have no compiler installed. It's much more difficult to compile code when there isn't a compiler, and with no gcc available, slapper can't do an awful lot.

Sure something else might come along that can, but as you point out, if you're running a server that's been up a year, changing things is never comfortable, and if you know slapper isn't going to infect you, there's much less motivation.

Re:Securing OpenSSL

[Flarners]

My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part.

That's exactly the problem the parent poster was trying to highlight! Sure, you can blindly trust that the Debian servers are secure and un-trojanned and let apt-get install it without so much as checking a key signature. Even if you do configure apt to check the signature, it fetches the public key from the same server as the packages. Thus, an attacker can easily trojan your machine with man-in-the-middle or DNS attacks, sending you to an update site with a trojanned package signed with his own public key. If someone sneaks into the real debian servers, subverting apt-get is as easy as:

• Upload new "developers" key signature.
• Sign trojanned package with new signature.
• Upload trojanned package.

and it's Game Over. Of course, Red Hat and Mandrake's solutions aren't much better. The key needs to be stored with a trusted entity like Verisign, which is how Windows Update and other commercial-grade updating systems ensure the integrity of their packages. You've never heard of Windows Update being trojanned, have you?

Re:Securing OpenSSL

[schulzdogg]

The old standby argument that 'checking the MD5 signatures' will save you has become null & void; ADM replaced the MD5 signatures too. The only reason the trojan was detected was because of the Google cache! This kind of thing probably has most users afraid to move to anything recently released that hasn't been extensively peer reviewed.

False. From the HLUG website (the group that discovered the trojan):
Thanks to Antioffline.com for hosting us, and Gentoo's Portage system for catching the trojaned files via checksums.

Putting MD5 signatures on the same server that the software is available from isn't even close to secure

This is true though.

Re:Securing OpenSSL

[rcw-home]

You've never heard of Windows Update being trojanned, have you?

No, but they have been cracked before:
http://www.attrition.org/security/commentary/ms16. html

It doesn't take a whole lot of imagination to come up with some very scary scenarios of what could have been put there instead of "Hacked by Chinese!" After all, how many people visiting Windows Update are running versions of IE without known run-arbitrary-code security holes?

Should have upgraded

[Hegemony]

I didn't and paid for it. Within about 3 hours time, some bastard got in, created a superuser, DoS'd nasa.gov, spawned a few irc servers, and started scanning other IP's for the same exploit. Damn they work fast.

When to Patch

[Crispin Cowan]

Readers interested in this topic may be interested in this paper that we presented last week at USENIX LISA 2002:
Timing the Application of Security Patches for Optimal Uptime

Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, and Chris Wright
WireX Communications, Inc. http://wirex.com
and
Adam Shostack
Informed Security http://www.informedsecurity.com

Security vulnerabilities are discovered, become publicly known, get exploited by attackers, and patches come out. When should one apply security patches? Patch too soon, and you may suffer from instability induced by bugs in the patches. Patch too late, and you get hacked by attackers exploiting the vulnerability. We explore the factors affecting when it is best to apply security patches, providing both mathematical models of the factors affecting when to patch, and collecting empirical data to give the model practical value. We conclude with a model that we hope will help provide a formal foundation for when the practitioner should apply security updates.

Crispin
----
Crispin Cowan, Ph.D.
Chief Scientist, WireX Communications, Inc.
Immunix: Security Hardened Linux Distribution
Available for purchase

Re:Vulnerability Check

[James_G]

How does one check if a server is vulnerable without actually "breaking in", i.e. make oneself liable to prosection? I skimmed through the PDF but could not find anything about this.

Well if you'd read the PDF instead of skimming it, you would have seen this:

Thus, we can simply connect to the HTTPS server and issue a HEAD request. The server responds with an HTTP header containing the Server: field and hence the answer we desire:

Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6

They then went on to verify that SSLv2 is not disabled, but they mention later in the paper that on only 10 hosts was this done.

Theoretically you could change the response to report the more recent version which would make this check innacurate, but why would anyone bother doing that? Far easier to just upgrade OpenSSL.

Re:Vulnerability Check

[FattMattP]

I can't believe that you got modded up for just saying "I didn't read the article." If you had bothered to read it and not skim it, you would have found the description of how they check it on page five:

The patches provided by the OpenSSL project do not change the version number. As a consequence, we cannot simply examine at the advertised version number in order to determine whether or not they have been applied. However, due to the nature of the SSLv2 problem, it's still relatively straightforward to determine whether or not a server is vulnerable and therefore by excluding servers which advertise a newer version number identify whether or not a given server has been patched.

[snip]

It's possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It's easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error.

This right under the section called "Detecting Vulnerable Servers." Maybe you should read the article next time before you post.

Damn MCSEs

[davinciII]

See, this is exactly what happens when you hire a bunch of paper MCSEs to run your........

wait, did you say Linux?

Due diligence.

[Black Parrot]

This is too easy, folks. Subscribe to your distro's update announcement list, read your mail daily, and apply the relevant patches promptly.

It's really not that hard. A typical update for me is:

1. read mail
2. ncftpget whatever.rpm
3. rpm -Uhv whatever
4. read rest of mail

By far the most time-consuming part is waiting for the RPM to download. Some say that it's even easier for source-based distros.

Run right out and get nessus.

[AugstWest]

If you haven't yet, you should definitely check out nessus.

It'll scan your machiens for known vulnerabilities and give you pointers on how to go about taking care of any that it finds. It's also got built-in updating to pull in the latest exploits.

The clients are even getting pretty spiffy these days, and the project has matured very rapidly.

Have we grown complacent?

[mao che minh]

Perhaps Linux users and administators have grown overly comfortable due to the long reign of tight security and lack of virii? Until rather recently, disclosed security advisories for FOSS could be neglected for substantial periods of time without worry. The world's hackers mostly took aim at easily exploitable IIS and Exchange servers, flimsy Win32 email clients, and major routers (like AT&T backbone routers to Asia and such). Largely ignored were the hordes of vulnerable web and mail Linux/BSD servers on campus networks and elsewhere (mostly left vulnerable due to neglect, not inherent OS issues). However, the desire to orchestrate large scale DDoS attacks and an exponential increase in the use of Linux systems has caused many hackers to take interest in conquering new grounds.

All of these years of rock solid security has made us complacent. We have to remember that, while Linux and OSS may be inherently secure, and Linux's modular design works as a fail safe against complete failure, we are still just as vulnerable if we don't remain vigilant.

Re:Have we grown complacent?

[AugstWest]

Perhaps Linux users and administators have grown overly comfortable due to the long reign of tight security and lack of virii?

I think this is a complete fallacy. Most default Linux installations, when left alone on a cable/DSL connection, have been hackable for years now. I can remember when I installed RedHat 6.2 on my gateway machine without having time to do the updates, and before midnight that night the box had been hacked.

I think that a lot of Linux users don't even realize when they've been hacked, either. Even the automated scan-and-exploit tools these days are becoming quite good at getting themselves installed on a system quietly. Unless you watch your logs on a daily basis, you often have no idea what is actually going on with your system.

Weird misconception

[dfn5]

I find that other admins patch by necessity. i.e. If something is broke, then patch it. If not leave it alone.

However, I read a stat somewhere that said that a large majority of security breaches could have been prevented by merely keeping up with patches. Therefore my philosphy is to create a patch schedule. And because I'm on Solaris things like OpenSSL are 3rd party to the OS, therefore I upgrade immediately. I rebuilt my solaris RPMs of OpenSSL that day and had it deployed to all my machines. Other things like GnuPG, IPFilter, OpenSSH, apache, sendmail, etc... they all need to be upgraded ASAP.

So all you Slashdot readers who posted that you have nothing to do but read Slashdot in that downsizing article, get off your butts and start patching. That should keep you busy full time.

Re:Weird misconception

[mao che minh]

Yes. I think that services like the "Red Hat Network" will greatly benefit end users and admins alike in this respect. Having a service that organizes errata (updates) and informs you what the current security threats are, and then shows you what systems you own/administer are vulnerable is very helpful. It gives end users an almost hands-free way of keeping themselves safe (as safe as they can in terms of updates, anyways), and can point out things that admins might have missed. I really like it.

Re:Vulnerability Check

[DVega]

To check the version number on the "Server" field of the HTTP Reply is not enough. Some patched versions did not change this header. On the PDF they explain on page 5 that they discovered a way to check the vulnerability without crashing the server.

" It s possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It s easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error. In an unpatched version of OpenSSL, the server overruns the key_arg with whatever the extra byte is. However, as we shall now show, this overrun is harmless."

Downstream Liability

[sczimme]

In the same vein, readers might be interested in a presentation we did at RSA 2002; the topic was Downstream Liability for Attack Relay and Amplification

(A two-page summary is available here.)

We described a scenario in which an intruder compromised a system, used it as a jumping-off point, and subsequently caused damages to an individual. The paper focuses on the legal side of things; IANAL but the other two authors - Tim Rosenberg, J.D. and Ron Plesco, J.D. - are.

I also state my opinion, which is that "...patches should be installed no later than ten (10) calendar days after release of the patch by the vendor". Discuss.

Or the lack of certs in OpenSSL so no one checks.

[tz]

I also checked the browsers, mainly command line a little while ago when the IE cert chain vulnerability was found. Most (wget, links, lynx) didn't bother to check the chain. Some didn't check anything at all, so any proxy server could spoof any page.

If you can see https://www.amazone.com, your browser is badly broken. amazone.com points to amazon.fr - but you should match the cert to the DNS.

Opera on the Zaurus was also vulnerable. Apple doesn't install any certificates in their OS X or Darwin OpenSSL directory.

One thing that happened between SSLeay (the original project) and OpenSSL is that the certificate chains were NOT installed by default, so everyone had a library, but no way of checking certificates since you require root certificates to check the site certificate. A second thing, probably worse is that the old default was to return an error if the certificate couldn't be validated. Now the default is TO GIVE NO ERROR IF THE CERTIFICATE CANNOT BE CHECKED. It would be better to give an error that would have to be overridden, which would cause developers to have to take a look and to actively disable security.

Curl was the only one that included any checking, but it required manually installing certs and specifying an option to turn it on. It would SILENTLY connect to SSL sites without security.

Mozilla was fine, and Konqueror fixed any problem it had, but the Opensource community should be embarrassed since the rest of the browsers security was not just flawed like IE, but DISABLED without any notice to this effect or NONEXISTENT.