Due Diligence?

ekr writes "The OpenSSL remote buffer overflows discovered at the end of July got a lot of press here on /. But how many people actually fixed their machines? I decided to study this question, and the results are kind of depressing. Two weeks after the release of the bug, over two thirds of the servers I sampled were still vulnerable. Even two weeks after the Slapper worm was announced, a third of the total servers were vulnerable. The paper can be found here in PDF or Postscript."

Comments

It's not just laziness...

[Anonymous Coward]

Many systems administrators aren't full-time and have other responsibilities. Keeping up-to-date with every security patch is very time consuming and sometimes management doesn't understand this and doesn't allocate resources for it as long as things are "working".

Re:It's not just laziness...

[dfn5]

Unfortunately keeping up with patches is a very important part of any security strategy. I am all for letting companies do things their way, but if admins don't allocate more time to security and patching then I'm afraid the government will do more than just recommend actions for Security on the Internet and will start mandating stuff. I for one don't want that to happen.

Bottom line? Improve your security while you still have the rights to do it yourself.

this is why...

[mr_gerbik]

This is why I run Windows 3.11. No worries about falling behind and not installing the latest fixes.

I wiped my OpenBSD boxes, reinstalled, patched

[JimmytheGeek]

I noticed connection attempts from Korea just after the announcement and decided it was time to nuke the the boxes from orbit. Not much point in having an O-BSD box you are only mostly sure of.

I had some angst with RedHat boxes, though. The update mechanism didn't change the reported version number of OpenSSH. Annoying.

Re:I wiped my OpenBSD boxes, reinstalled, patched

[WatertonMan]

I don't understand why you felt OpenBSD was less secure than Redhat in this regard. You can patch the software on OpenBSD fairly easily. Heavens, I've updated our OpenBSD box's Apache several times now.

Securing OpenSSL

[Exmet Paff Daxx]

Some points to consider:

• Fear. Most Linux users are probably reeling in shock from the recent trojan inserted by elite hacking group ADM into the libpcap distribution. The old standby argument that 'checking the MD5 signatures' will save you has become null & void; ADM replaced the MD5 signatures too. The only reason the trojan was detected was because of the Google cache! This kind of thing probably has most users afraid to move to anything recently released that hasn't been extensively peer reviewed.
  
• Ignorance. Since the Slapper worm only contains offsets for a handful of platforms, many flavors of Linux are 'immune' to automated infection. While blackhat groups have offsets for nearly every implementation of Apache/SSL in existence (yes, even you x86 Solaris people), this threat isn't considered 'immediate' enough to justify the third point:
  
• Sloth. Upgrading your OpenSSL isn't as easy as it could be. You actually have to recompile Apache with ./configure flags to link it to the new version of OpenSSL which you just recently downloaded (it's not trojaned... right?). Sounds easy, but for a production server that hasn't been touched in a year, this tends to make people really nervous
  

All of this points to the fact that there is a fundamental flaw in the way that the Open Source community is securing their software. Putting MD5 signatures on the same server that the software is available from isn't even close to secure - Dave Aitel of Immunity Security keeps hammering on this point in BugTraq. And we're going to see even more of this 'Upgrade Fear' as more and more distributions get trojaned - Slash is probably next on the list.

We need to look at existing, successful solutions to this problem (like Windows Update) and catch up. Now.

Re:Securing OpenSSL

[Psiren]

Upgrading your OpenSSL isn't as easy as it could be.

Any decent distribution will do this work for you. That's what they're for after all. My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part.

Re:Securing OpenSSL

[Albanach]

There's also the issue of those running servers who, sensibly, have either gcc set to non executable, or simply have no compiler installed. It's much more difficult to compile code when there isn't a compiler, and with no gcc available, slapper can't do an awful lot.

Sure something else might come along that can, but as you point out, if you're running a server that's been up a year, changing things is never comfortable, and if you know slapper isn't going to infect you, there's much less motivation.

Re:Securing OpenSSL

[Flarners]

My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part.

That's exactly the problem the parent poster was trying to highlight! Sure, you can blindly trust that the Debian servers are secure and un-trojanned and let apt-get install it without so much as checking a key signature. Even if you do configure apt to check the signature, it fetches the public key from the same server as the packages. Thus, an attacker can easily trojan your machine with man-in-the-middle or DNS attacks, sending you to an update site with a trojanned package signed with his own public key. If someone sneaks into the real debian servers, subverting apt-get is as easy as:

• Upload new "developers" key signature.
• Sign trojanned package with new signature.
• Upload trojanned package.

and it's Game Over. Of course, Red Hat and Mandrake's solutions aren't much better. The key needs to be stored with a trusted entity like Verisign, which is how Windows Update and other commercial-grade updating systems ensure the integrity of their packages. You've never heard of Windows Update being trojanned, have you?

Re:Securing OpenSSL

[schulzdogg]

The old standby argument that 'checking the MD5 signatures' will save you has become null & void; ADM replaced the MD5 signatures too. The only reason the trojan was detected was because of the Google cache! This kind of thing probably has most users afraid to move to anything recently released that hasn't been extensively peer reviewed.

False. From the HLUG website (the group that discovered the trojan):
Thanks to Antioffline.com for hosting us, and Gentoo's Portage system for catching the trojaned files via checksums.

Putting MD5 signatures on the same server that the software is available from isn't even close to secure

This is true though.

Re:Securing OpenSSL

[mwalker]

Sure something else might come along that can, but as you point out, if you're running a server that's been up a year, changing things is never comfortable, and if you know slapper isn't going to infect you, there's much less motivation.

That's exactly what the Blackhats of the world wanted to hear. Of course, they can use the exploit on you, log in, download their BINARY rootkits that don't need a compiler, and use your bandwidth to rape innocent sites like Slashdot with DoS attacks. After deleting your logs, they'll install a sniffer to see what other systems they can compromise using your NIC's visibility, and finally they'll deface your web site and pipe /dev/urandom onto your hard drive's raw interface.

Have fun!

It's really a damned shame you don't have a way of getting a securely signed OpenSSL update. While Debian has signature and key checking, it's all on a single point of failure server. You really need a trusted key that comes with the install media, but so far the only O/S which supports this is Windows. People who use Free software don't get install media and are pretty much up the creek...

Re:Securing OpenSSL

[rcw-home]

You've never heard of Windows Update being trojanned, have you?

No, but they have been cracked before:
http://www.attrition.org/security/commentary/ms16. html

It doesn't take a whole lot of imagination to come up with some very scary scenarios of what could have been put there instead of "Hacked by Chinese!" After all, how many people visiting Windows Update are running versions of IE without known run-arbitrary-code security holes?

Re:Securing OpenSSL

[passthecrackpipe]

You really should get out more. There are distro's that include a trusted key on the install media, and yes, people who use Free Software do get install media. If you don't have anything factual to see, silence will become you.....

Re:Securing OpenSSL

[TheLink]

"The key needs to be stored with a trusted entity like Verisign, which is how Windows Update and other commercial-grade updating systems ensure the integrity of their packages. You've never heard of Windows Update being trojanned, have you?"

0) How are you sure it hasn't already been trojaned?

1) Verisign just _claims_ to say the entity is who they claim they are, not that the entity is trustworthy.

2) Verisign screw up - certs issued to wrong people see Microsoft Security Bulletin MS01-017.

3) Microsoft screw up- there was an issue where the wrong types of certs could be used as CA certs. [Microsoft Security Bulletin MS02-050]

4) Network Solutions is part of Verisign. NS is not known to be very security conscious. If someone screwed both the certs and the DNS most people wouldn't notice.

5) Windows Update could become a trojan itself- make sure you read the EULA. e.g. one day you might see stuff like:
"You acknowledge and agree that Microsoft may automatically check the version of the OS Product and/or its components that you are utilizing and may provide upgrades or fixes to the OS Product that will be automatically downloaded to your computer"

And how sure are we that it will do it correctly?

Also note that Microsoft has recently said that they may break some apps.

So if windows update automatically downloads stuff which breaks some of your apps, it starts getting hard to distinguish it from a trojan.

--
I'm not saying Open Source is more trustworthy either. Most software isn't secure. Most Open Source software isn't secure. Most were never designed with security in mind - look at PHP - many of the features that make PHP PHP are actually bad for security. Look at ISC's range of software, see the history and the design/architecture of the software.

Unfortunately there are only very few who can program securely, and C just makes things worse - even fewer of the few can program securely in C.

Should have upgraded

[Hegemony]

I didn't and paid for it. Within about 3 hours time, some bastard got in, created a superuser, DoS'd nasa.gov, spawned a few irc servers, and started scanning other IP's for the same exploit. Damn they work fast.

When to Patch

[Crispin Cowan]

Readers interested in this topic may be interested in this paper that we presented last week at USENIX LISA 2002:
Timing the Application of Security Patches for Optimal Uptime

Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, and Chris Wright
WireX Communications, Inc. http://wirex.com
and
Adam Shostack
Informed Security http://www.informedsecurity.com

Security vulnerabilities are discovered, become publicly known, get exploited by attackers, and patches come out. When should one apply security patches? Patch too soon, and you may suffer from instability induced by bugs in the patches. Patch too late, and you get hacked by attackers exploiting the vulnerability. We explore the factors affecting when it is best to apply security patches, providing both mathematical models of the factors affecting when to patch, and collecting empirical data to give the model practical value. We conclude with a model that we hope will help provide a formal foundation for when the practitioner should apply security updates.

Crispin
----
Crispin Cowan, Ph.D.
Chief Scientist, WireX Communications, Inc.
Immunix: Security Hardened Linux Distribution
Available for purchase

Re:Vulnerability Check

[James_G]

How does one check if a server is vulnerable without actually "breaking in", i.e. make oneself liable to prosection? I skimmed through the PDF but could not find anything about this.

Well if you'd read the PDF instead of skimming it, you would have seen this:

Thus, we can simply connect to the HTTPS server and issue a HEAD request. The server responds with an HTTP header containing the Server: field and hence the answer we desire:

Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6

They then went on to verify that SSLv2 is not disabled, but they mention later in the paper that on only 10 hosts was this done.

Theoretically you could change the response to report the more recent version which would make this check innacurate, but why would anyone bother doing that? Far easier to just upgrade OpenSSL.

Re:Vulnerability Check

[FattMattP]

I can't believe that you got modded up for just saying "I didn't read the article." If you had bothered to read it and not skim it, you would have found the description of how they check it on page five:

The patches provided by the OpenSSL project do not change the version number. As a consequence, we cannot simply examine at the advertised version number in order to determine whether or not they have been applied. However, due to the nature of the SSLv2 problem, it's still relatively straightforward to determine whether or not a server is vulnerable and therefore by excluding servers which advertise a newer version number identify whether or not a given server has been patched.

[snip]

It's possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It's easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error.

This right under the section called "Detecting Vulnerable Servers." Maybe you should read the article next time before you post.

Re:Servers should disable themselves...

[aridhol]

What happens when the author's website becomes obsolete? Security may be up-to-date, but you now have a bad webpage. Will the server shut itself down, or let you run with a possible compromised server? Also, consider the effect if the timing is set in the source. The first of every (month|year|week) will show an enormous load on the update server. What happens when the software can't talk to the update server? Assume everything's fine? When someone hacks the update server with a new, trojaned form, suddenly everybody will have it. Check the server's logs, you now have a list of compromised hosts.

Other than that, it's a good idea...

Damn MCSEs

[davinciII]

See, this is exactly what happens when you hire a bunch of paper MCSEs to run your........

wait, did you say Linux?

Due diligence.

[Black Parrot]

This is too easy, folks. Subscribe to your distro's update announcement list, read your mail daily, and apply the relevant patches promptly.

It's really not that hard. A typical update for me is:

1. read mail
2. ncftpget whatever.rpm
3. rpm -Uhv whatever
4. read rest of mail

By far the most time-consuming part is waiting for the RPM to download. Some say that it's even easier for source-based distros.

Run right out and get nessus.

[AugstWest]

If you haven't yet, you should definitely check out nessus.

It'll scan your machiens for known vulnerabilities and give you pointers on how to go about taking care of any that it finds. It's also got built-in updating to pull in the latest exploits.

The clients are even getting pretty spiffy these days, and the project has matured very rapidly.

Have we grown complacent?

[mao che minh]

Perhaps Linux users and administators have grown overly comfortable due to the long reign of tight security and lack of virii? Until rather recently, disclosed security advisories for FOSS could be neglected for substantial periods of time without worry. The world's hackers mostly took aim at easily exploitable IIS and Exchange servers, flimsy Win32 email clients, and major routers (like AT&T backbone routers to Asia and such). Largely ignored were the hordes of vulnerable web and mail Linux/BSD servers on campus networks and elsewhere (mostly left vulnerable due to neglect, not inherent OS issues). However, the desire to orchestrate large scale DDoS attacks and an exponential increase in the use of Linux systems has caused many hackers to take interest in conquering new grounds.

All of these years of rock solid security has made us complacent. We have to remember that, while Linux and OSS may be inherently secure, and Linux's modular design works as a fail safe against complete failure, we are still just as vulnerable if we don't remain vigilant.

Re:Have we grown complacent?

[AugstWest]

Perhaps Linux users and administators have grown overly comfortable due to the long reign of tight security and lack of virii?

I think this is a complete fallacy. Most default Linux installations, when left alone on a cable/DSL connection, have been hackable for years now. I can remember when I installed RedHat 6.2 on my gateway machine without having time to do the updates, and before midnight that night the box had been hacked.

I think that a lot of Linux users don't even realize when they've been hacked, either. Even the automated scan-and-exploit tools these days are becoming quite good at getting themselves installed on a system quietly. Unless you watch your logs on a daily basis, you often have no idea what is actually going on with your system.

Weird misconception

[dfn5]

I find that other admins patch by necessity. i.e. If something is broke, then patch it. If not leave it alone.

However, I read a stat somewhere that said that a large majority of security breaches could have been prevented by merely keeping up with patches. Therefore my philosphy is to create a patch schedule. And because I'm on Solaris things like OpenSSL are 3rd party to the OS, therefore I upgrade immediately. I rebuilt my solaris RPMs of OpenSSL that day and had it deployed to all my machines. Other things like GnuPG, IPFilter, OpenSSH, apache, sendmail, etc... they all need to be upgraded ASAP.

So all you Slashdot readers who posted that you have nothing to do but read Slashdot in that downsizing article, get off your butts and start patching. That should keep you busy full time.

Re:Weird misconception

[mao che minh]

Yes. I think that services like the "Red Hat Network" will greatly benefit end users and admins alike in this respect. Having a service that organizes errata (updates) and informs you what the current security threats are, and then shows you what systems you own/administer are vulnerable is very helpful. It gives end users an almost hands-free way of keeping themselves safe (as safe as they can in terms of updates, anyways), and can point out things that admins might have missed. I really like it.

Re:Vulnerability Check

[DVega]

To check the version number on the "Server" field of the HTTP Reply is not enough. Some patched versions did not change this header. On the PDF they explain on page 5 that they discovered a way to check the vulnerability without crashing the server.

" It s possible to determine whether OpenSSL has been patched by generating a CLIENT-MASTER-KEY which is one octet too long. It s easy to see that in a patched version, this falls afoul of the length check shown in Figure 5. The result is a handshake error. In an unpatched version of OpenSSL, the server overruns the key_arg with whatever the extra byte is. However, as we shall now show, this overrun is harmless."

False

[mwalker]

Thanks to Antioffline.com for hosting us, and Gentoo's Portage system for catching the trojaned files via checksums.

Gentoo had the OLD checksums, which is the reason it was caught. Everyone who checked the new checksums got owned. The Gentoo suspicions were confirmed by checking the Google cache.

Gentoo basically caught this because they were so far behind the curve that they still had the old distribution. While it's a great argument not to use Gentoo, this kind of security-through-being-behind accident is not a security process, nor is it repeatable, nor should it be considered a success of the checksum system.

Re:False

[Anonymous Coward]

Not sure how gentoo does things, since I tend to avoid cults, but the point here isn't that they were behind the curve, or old. It is that they had checksums of the correct (untrojaned) archives on a different server/media. Thus when somebody tried to build, the modifed archive was caught. Your post is just stupid and wrong. Additionally the OpenSSH trojan was caught in the same way, only via the FreeBSD port system which includes the checksums as part of the port. If the downloaded archive doesn't match the checksum, you get a warning. It is a security process, in that by seperating the checksums from the archives you can verify them without trusting the checksum from the same repository as the archive itself. With a sufficiently strong checksum this will catch *all* third party, substitution, trojan attempts.

Re:Gentoo!

[op00to]

>2 minutes? Like an hour?

Downstream Liability

[sczimme]

In the same vein, readers might be interested in a presentation we did at RSA 2002; the topic was Downstream Liability for Attack Relay and Amplification

(A two-page summary is available here.)

We described a scenario in which an intruder compromised a system, used it as a jumping-off point, and subsequently caused damages to an individual. The paper focuses on the legal side of things; IANAL but the other two authors - Tim Rosenberg, J.D. and Ron Plesco, J.D. - are.

I also state my opinion, which is that "...patches should be installed no later than ten (10) calendar days after release of the patch by the vendor". Discuss.

Click 'Install'

[KFury]

On the mac, I just set 'Suftware Update; to run daily, and I click 'install' when it says there's a security fix.

By default, users only have to click one button (the default button) to keep their Mac-flavored BSD secure.

And they don't have to subscribe to mailing lists or be security geeks. They could be your mom and still get it right.

Not trying to rip on your mom.

I don't even know her!

No, seriously. That wasn't me, that time at the Quaker Steak n Lube!

Re:The window

[Mandi Walls]

that's the point of up2date. Neither of those things will happen.

up2date runs with gpg signatures on all packages

and it checks all dependencies. And, since the packages are built by a company trying to guarantee you can run oracle on your box rather than a couple of dudes in a basement, the packages and their dependencies are correct and current.

:)

--mandi

methodology in paper is flawed

[wuchang]

The paper looks at version numbers but does not account for back patches to old versions that fix the bug. I'm running a patched Mandrake https server which returns a version of 2.8.7/0.9.6c. Slapper requests correctly return an error message. What the paper needs to do is issue the exploit itself to determine whether or not things have been patched. Otherwise, the author overcounts the vulnerable systems out there.

Re:The window

[jroysdon]

I've had up2date break a ton of things when it installs a newer version. ypbind and xinetd are two that bit us recently. They both installed and initially tested fine, but there were subtle changes that broke other things (securenets on ypbind, and all our ssl-based email services like spop3 and simap with xinetd). Easy enough to fix, but not something you want automated when you're on vacation.

We have a cron job at 4am that mirrors the RH update directories (only downloads changes) and then emails us if there are changes. Then we install and test them on a test non-production server to verify first, then install on the production boxes, plus we already have the update(s) on a local box so 'rpm -Fvh ftp://localupdateserver/whatever.rpm' goes really fast (especially when you have a couple dozen boxes to maintain).

Or the lack of certs in OpenSSL so no one checks.

[tz]

I also checked the browsers, mainly command line a little while ago when the IE cert chain vulnerability was found. Most (wget, links, lynx) didn't bother to check the chain. Some didn't check anything at all, so any proxy server could spoof any page.

If you can see https://www.amazone.com, your browser is badly broken. amazone.com points to amazon.fr - but you should match the cert to the DNS.

Opera on the Zaurus was also vulnerable. Apple doesn't install any certificates in their OS X or Darwin OpenSSL directory.

One thing that happened between SSLeay (the original project) and OpenSSL is that the certificate chains were NOT installed by default, so everyone had a library, but no way of checking certificates since you require root certificates to check the site certificate. A second thing, probably worse is that the old default was to return an error if the certificate couldn't be validated. Now the default is TO GIVE NO ERROR IF THE CERTIFICATE CANNOT BE CHECKED. It would be better to give an error that would have to be overridden, which would cause developers to have to take a look and to actively disable security.

Curl was the only one that included any checking, but it required manually installing certs and specifying an option to turn it on. It would SILENTLY connect to SSL sites without security.

Mozilla was fine, and Konqueror fixed any problem it had, but the Opensource community should be embarrassed since the rest of the browsers security was not just flawed like IE, but DISABLED without any notice to this effect or NONEXISTENT.

gah

[nomadic]

But how many people actually fixed their machines? I decided to study this question, and the results are kind of depressing.

If you're depressed by that, you might want to see a psychiatrist. I mean, you shouldn't have that kind of reaction to such a minor issue.

Sysadmins need to pursue intrusion attempts

[wowbagger]

I know of a Linux system that logs and reports intrusion attempts by CodeRed/Nimda, Slapper, et. al., and mails a report to a system admin every morning.

The system admin wasn't pursuing these reports. I asked why.

His response - "Well, those are attempts to exploit a Windows server, and this is a Linux box, so they don't matter."

I made the counterpoint "If one of your system was infected, wouldn't you want to be told about it?"

If every systems admin would take the time to track down the Code Red attempts on their systems, and notify the responsible parties whereever possible, then a lot of the unpatched systems would be shut down (if not by their administrators, then by the ISP supplying connectivity).

I just don't understand an admin with an attitude like that.