Slashdot Mirror
Due Diligence?
ekr writes "The OpenSSL remote buffer overflows discovered at the end of July got
a lot of press here on /. But how many people actually fixed their
machines? I decided to study this question, and the results are kind of
depressing. Two weeks after the release of the bug, over two thirds of
the servers I sampled were still vulnerable. Even two weeks after
the
Slapper worm was announced, a third of the total servers
were vulnerable. The paper can be found here in
PDF
or
Postscript."
Unless there is liability accountability, people will continue to be lazy. This frequently happens when no-talent hack spend 8 hours a dat hiding from their inflated resumes.
If we don't fight for ourselves no one will.
Many systems administrators aren't full-time and have other responsibilities. Keeping up-to-date with every security patch is very time consuming and sometimes management doesn't understand this and doesn't allocate resources for it as long as things are "working".
This is why I run Windows 3.11. No worries about falling behind and not installing the latest fixes.
I noticed connection attempts from Korea just after the announcement and decided it was time to nuke the the boxes from orbit. Not much point in having an O-BSD box you are only mostly sure of.
I had some angst with RedHat boxes, though. The update mechanism didn't change the reported version number of OpenSSH. Annoying.
The servers you sampled are administered by MCSEs'. Come one now, you know that this has to be the case, because no Linux admin could ever be lazy enough not to patch his system, this is the sole right of a paper MCSE.
Yea that was me pointing at you, the long haired pasty white guy with the god complex. You have become your own worst enemy. You have become lazy and ignorant because no one knew linux and you could rule. We guess what, with getting into the big league comes the price of being exposed.
Better brush up and get on the stick because yes a lot of MCSEs are lazy, no good, paper test taking gimps but your more dangerous. You take the assumption that you are secure, but as more eyeballs look at your systmes as linux gains marketshare you going to have the same issues.
Depressing no? Enjoy the ride, your in the spot light now my friends just like the windows guys....get to your patches like a good boy.
Neck_of_the_Woods
#/usr/local/surf/glassy/overhead
like admins have time for such fiddle faddle chores...
How does one check if a server is vulnerable without actually "breaking in", i.e. make oneself liable to prosection?
I skimmed through the PDF but could not find anything about this.
Thank you.
lynx -mime_header http://www.YOURSERVER.com
Should be 0.9.6g or better.
All of this points to the fact that there is a fundamental flaw in the way that the Open Source community is securing their software. Putting MD5 signatures on the same server that the software is available from isn't even close to secure - Dave Aitel of Immunity Security keeps hammering on this point in BugTraq. And we're going to see even more of this 'Upgrade Fear' as more and more distributions get trojaned - Slash is probably next on the list.
We need to look at existing, successful solutions to this problem (like Windows Update) and catch up. Now.
If guns kill people, then CmdrTaco's keyboard misspells words.
IMHO the right way is to write a system that disables servers automatically if a vulnerability is known and the administrator did not fix it in n days. Either the functionality is integrated into the daemons, and they check (via http, mail, whatever) every day whether they are affected by a problem. Or each system should run a daemon that controls all server software.
It should warn the admin before it turns the server off, of course, but a broken unmaintained server is always better than a hacked server.
How sure are you the the administrators of the servers you sampled are also Slashdot readers? While certainly some laziness could explain your statistics, what of good old-fashioned lack of communcations? Just because a message warning about a security hole was sent out, doesn't mean it got received, or even read in a timely manner. Besides, maybe most of those administrators were taking three-week vacations just then!
I worked at an ISP at the time that this was happening and we were quite well aware of these vulnerabilites. We often referred to CERT when looking for vulnerabilities that may have affected us. It was through sites like those that we found out about the problems with OpenSSL and we made the necessary changes. I'm not sure why it was found that many others didn't do what was necessary. Perhaps there are many admins that don't understand that they need to keep themselves up to date on these matters, and of course, they are often busy with many other tasks. It's not easy being an admin. Maybe that's why there is a System Administrator Appreciation Day.
I didn't and paid for it. Within about 3 hours time, some bastard got in, created a superuser, DoS'd nasa.gov, spawned a few irc servers, and started scanning other IP's for the same exploit. Damn they work fast.
Timing the Application of Security Patches for Optimal Uptime
Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, and Chris Wright
WireX Communications, Inc. http://wirex.com
and
Adam Shostack
Informed Security http://www.informedsecurity.com Crispin
----
Crispin Cowan, Ph.D.
Chief Scientist, WireX Communications, Inc.
Immunix: Security Hardened Linux Distribution
Available for purchase
What methodology was used for this. Typically vendors patch a problem but don't "force" and upgrade on their customers. Just looking at (external version numbers) isn't going to tell you for sure..
(Mind you I'm not doubting most systems were not patched.. its just you may be getting false positives.)
prof ever. No, I'm just kidding, he's pretty interesting, his talks on the VAX really get me excited.
That's why MS wants to make apps that upgrade themselves automagically
It's not a bad idea after all, too bad you can't trust MS on anything (They use a good idea bundled with a bad one and a EULA that grants them too much)
Who should be worried about this bug? What does it affect, in particular? I'm guessing just specific webserver configurations, but do I need to patch the Linux distro I just put on this box to dual-boot? If so, how difficult is it - I'm barely getting used to Linux and the idea of recompiling a bunch of system libraries and updating lots of software doesn't sound very good to me ;)
using namespace slashdot;
troll::post();
For Gentoo users it's as easy as:
emerge update
emerge -u world
It took my Athlon 800 system >2 minutes to be fixed. I can understand the liability about why not to upgrade and apply security holes, but as IT pros, we have to weigh out the evils of this world and pick the best path for our users.
See, this is exactly what happens when you hire a bunch of paper MCSEs to run your........
wait, did you say Linux?
And my friend's server, using OpenSSL 0.9.6 as well as old OpenSSH/Apache/PHP/etc was rooted (Proberbly via OpenSSL) and someone started using it to DoS someones server. Sigh.
"With Microsoft, you get Windows. With Linux, you get the full house" - unknown
This is too easy, folks. Subscribe to your distro's update announcement list, read your mail daily, and apply the relevant patches promptly.
It's really not that hard. A typical update for me is:
- read mail
- ncftpget whatever.rpm
- rpm -Uhv whatever
- read rest of mail
By far the most time-consuming part is waiting for the RPM to download. Some say that it's even easier for source-based distros.Sheesh, evil *and* a jerk. -- Jade
If you haven't yet, you should definitely check out nessus.
It'll scan your machiens for known vulnerabilities and give you pointers on how to go about taking care of any that it finds. It's also got built-in updating to pull in the latest exploits.
The clients are even getting pretty spiffy these days, and the project has matured very rapidly.
This could be a nightmare in itself. What happens if the update server gets hacked... then all of a sudden you have systems either auto-trojaning themselves - or shutting down everywhere.
Not really a good idea... an equivilient to *gasp* "windows update" for terminal would be nice (RH8 has one, if you pay for or try RHN), where it automatically gives you a list of updates available and allows you to pick them in a dselect-style (debian) format, or something similar.
Perhaps Linux users and administators have grown overly comfortable due to the long reign of tight security and lack of virii? Until rather recently, disclosed security advisories for FOSS could be neglected for substantial periods of time without worry. The world's hackers mostly took aim at easily exploitable IIS and Exchange servers, flimsy Win32 email clients, and major routers (like AT&T backbone routers to Asia and such). Largely ignored were the hordes of vulnerable web and mail Linux/BSD servers on campus networks and elsewhere (mostly left vulnerable due to neglect, not inherent OS issues). However, the desire to orchestrate large scale DDoS attacks and an exponential increase in the use of Linux systems has caused many hackers to take interest in conquering new grounds.
All of these years of rock solid security has made us complacent. We have to remember that, while Linux and OSS may be inherently secure, and Linux's modular design works as a fail safe against complete failure, we are still just as vulnerable if we don't remain vigilant.
However, some people take vacations, or go on project, or the such. Some people even sleep, and a window of vulnerability of only a few hours can create a serious problem. Your advice is perfect for those situations which need it the least - where the system is regularly serviced by a nearly-constantly-available administrator. This by no means covers all situations.
- As should be noted from Figure 4, updating on Linux was rather easier than updating on *BSD, since all of the *BSD updates required compilation, either of the base system or from the ports/packages collection.
Huh?# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/packages/A ll/openssl-0.9.6g.tgz
Binaries installed -- no compilation required!
However, I read a stat somewhere that said that a large majority of security breaches could have been prevented by merely keeping up with patches. Therefore my philosphy is to create a patch schedule. And because I'm on Solaris things like OpenSSL are 3rd party to the OS, therefore I upgrade immediately. I rebuilt my solaris RPMs of OpenSSL that day and had it deployed to all my machines. Other things like GnuPG, IPFilter, OpenSSH, apache, sendmail, etc... they all need to be upgraded ASAP.
So all you Slashdot readers who posted that you have nothing to do but read Slashdot in that downsizing article, get off your butts and start patching. That should keep you busy full time.
-- Thou hast strayed far from the path of the Avatar.
Wow, I just read this after reading that article about sh*tty programming, and Michael Barcella saved my bacon. I have decided not to install any patches because:
I can't understand what the he** this program is doing. It's all just "use stdio.h"... WTF?? You wackos can read this stuff??
It is not written in perl, ruby, python, JavaScript or some other high level language. MB told me that C/C++ is evil. I believe him.
While I was trying to read it, a friend came up and pointed at the screen. Rule #3, no pointers.
Finally, I did not see the official seal of the united states in the upper left corner of my text editor. I never do, but after reading MB's column, I look for it.
However, I can't post this message because I am leaving the inet services off until I can understand all of the source for my TCP-IP stack. After that, I'm gonna tackle the source for Telnet.
WooT!
~Hammy
Gentoo had the OLD checksums, which is the reason it was caught. Everyone who checked the new checksums got owned. The Gentoo suspicions were confirmed by checking the Google cache.
Gentoo basically caught this because they were so far behind the curve that they still had the old distribution. While it's a great argument not to use Gentoo, this kind of security-through-being-behind accident is not a security process, nor is it repeatable, nor should it be considered a success of the checksum system.
--
What happens when you outlaw guns
Umm, 137 is netbios-ns, the netbios name service, which is for SMB/CIFS names, which is Windows Networking. Looks like somebody's scanning Windows Shares. Might want to pick a better port to laugh about. :)
--
Given enough personal experience, all stereotypes are shallow.
Thus, we can simply connect to the HTTPS server and issue a HEAD request.
One easy way I discovered to do this, assuming you've got the curl library installed, is to issue this from the command line:
curl -I [your_url__or_hostname_here]
Probably old hat to most of this group, but it proved an easy way to check several servers quickly. It's also handy if you want to make sure that your changes to Apache's httpd.conf (for ServerTokens and ServerSignature) actually "took".
In the same vein, readers might be interested in a presentation we did at RSA 2002; the topic was Downstream Liability for Attack Relay and Amplification
(A two-page summary is available here.)
We described a scenario in which an intruder compromised a system, used it as a jumping-off point, and subsequently caused damages to an individual. The paper focuses on the legal side of things; IANAL but the other two authors - Tim Rosenberg, J.D. and Ron Plesco, J.D. - are.
I also state my opinion, which is that "...patches should be installed no later than ten (10) calendar days after release of the patch by the vendor". Discuss.
I want to drag this out as long as possible. Bring me my protractor.
My Debian box was updated no more than 24 hours after I read about the problem, requiring nothing more that an apt-get update on my part. [link to Debian APT HOWTO added by me]
Um, I hope you did an apt-get upgrade , too, otherwise you have only a nice list of updated software that hasn't been installed on your Debian box.
... that keeps probing my servers!!
On the mac, I just set 'Suftware Update; to run daily, and I click 'install' when it says there's a security fix.
By default, users only have to click one button (the default button) to keep their Mac-flavored BSD secure.
And they don't have to subscribe to mailing lists or be security geeks. They could be your mom and still get it right.
Not trying to rip on your mom.
I don't even know her!
No, seriously. That wasn't me, that time at the Quaker Steak n Lube!
Kevin Fox
Most distros back port the patch to the version they were shipping. This is to keep the ABI from changing and reduce/elminate the massive amounts of turmoil that would occur if they revved to an entire new version.
The end result is that you see the 'old' vulernable version, but in actuality, you are seeing the 'old' version with patch applied.
Comment removed based on user account deletion
Wonder how many honeypots he hit?
Thanks, Steve
The paper looks at version numbers but does not account for back patches to old versions that fix the bug. I'm running a patched Mandrake https server which returns a version of 2.8.7/0.9.6c. Slapper requests correctly return an error message. What the paper needs to do is issue the exploit itself to determine whether or not things have been patched. Otherwise, the author overcounts the vulnerable systems out there.
I also checked the browsers, mainly command line a little while ago when the IE cert chain vulnerability was found. Most (wget, links, lynx) didn't bother to check the chain. Some didn't check anything at all, so any proxy server could spoof any page.
If you can see https://www.amazone.com, your browser is badly broken. amazone.com points to amazon.fr - but you should match the cert to the DNS.
Opera on the Zaurus was also vulnerable. Apple doesn't install any certificates in their OS X or Darwin OpenSSL directory.
One thing that happened between SSLeay (the original project) and OpenSSL is that the certificate chains were NOT installed by default, so everyone had a library, but no way of checking certificates since you require root certificates to check the site certificate. A second thing, probably worse is that the old default was to return an error if the certificate couldn't be validated. Now the default is TO GIVE NO ERROR IF THE CERTIFICATE CANNOT BE CHECKED. It would be better to give an error that would have to be overridden, which would cause developers to have to take a look and to actively disable security.
Curl was the only one that included any checking, but it required manually installing certs and specifying an option to turn it on. It would SILENTLY connect to SSL sites without security.
Mozilla was fine, and Konqueror fixed any problem it had, but the Opensource community should be embarrassed since the rest of the browsers security was not just flawed like IE, but DISABLED without any notice to this effect or NONEXISTENT.
Given all the crap posted yesterday claiming MS software was terrible with regard to security, I find this somewhat amusing. I was going to say then that the sysadmin patching the system is more of a security bottleneck than whether you're running Linux/BSD or Windows in most practical respects. This seems to confirm that fact. Also makes the optional automatic patching feature of Winodws look very attractive, despite all the paranoid comments I read yesterday.
Vote for Pedro
But why not make the trust symmetric? If I apt-get upgrade my packages, my client could compare all the MD5 hashes corresponding to my installed packages, checking that they are the same as those provided by the server during the apt-get update. If there's a discrepancy, my client automatically notifies the Debian server there's a problem: either I've got a trojaned package or they do. It's up to Debian security to figure out which. Meanwhile, my apt-get client tells me that my package could be compromised and do I want to disable it? This way, if there's one good and one bad version of the same package out there, it'll get found out eventually.
If only we could all use decent distributions...
I'm sitting here on my beautiful, instantly up-to-date Debian box with a terminal open to a Solaris production server. Now I'm sure there's some way to get the binary distribution of Apache to install, but I'm not sure I'll be able to actually figure it out in less time then it would take me to configure and compile the source. Of course who knows how long that will take if I have to hunt down the Solaris packages for all those "useless" tools like a C compiler that aren't installed by default.
Yes, my 31337 h4x0r friends, this is one box that won't ever be secure until I convince the boss that SPARCs should be running Linux.
Define broken.
I just loaded the site you gave on Mozilla,IE and Opera and I was presented with a dialog with each that warnedme that the cert was not matching.
Comment removed based on user account deletion
Comment removed based on user account deletion
But how many people actually fixed their machines? I decided to study this question, and the results are kind of depressing.
If you're depressed by that, you might want to see a psychiatrist. I mean, you shouldn't have that kind of reaction to such a minor issue.
- They were certainly well-known when I was in college in the mid 70s, but the PL/C dialect of the PL/I checkout compiler corrected mistakes like that at run-time. (OK, it often fixed them incorrectly, but at least least it wouldn't overrun an array.) And our professors dinged us for writing programs where that happened, and made us run the programs on input decks that were maliciously designed to check for programs that overflowed their buffers.
- They were certainly well-known when K&R wrote their books on C which warned you to be careful about bounds checking when using pointers and arrays.
- They were certainly well-known in the early 80s when everybody started complaining that the gets() and scanf() routines made it easy to overrun buffers on input when you weren't doing it by hand.
- They were certainly well-known in the late 80s when the Morris Worm wandered around a lot of the machines in the internet.
- They were certainly well known when the C++ string-handling libraries were designed to NOT overrun buffers, and when Java was designed to not even have pointers, and had array objects that checked bounds for you.
- There are enough software engineering CASE tools that try to find problems more complex than lint() looks for, though perhaps array bounds checking isn't something they check effectively.
I like C - I really like it. It's time to stop using it. It's time to stop shipping code that has array bounds problems, and security code that hasn't been proofread for them. And it's time to stop using programs that run as root when they don't need to. This isn't the 80s any more.There are other bugs out there - a popular attack is to try to abuse dotdots in path names, which there's more excuse for forgetting to check, and there are things like race conditions that are genuinely hard to check for (e.g. what happens when somebody's ripping up your temp files while your program is running), though checking return codes on system calls and doing something appropriate about failures is a good start.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
And frequently the patches cause more problems that they fix. This is why my company has switched to OSS/Linux (eat that GNU).
If you are a MS server admin you are always double checking TechNet and the other available sources because the delay between the patch on TechNet and the WindowsUpdate (critical area) can be as long as 3 weeks sometimes. Now that's sorry as can be. Lockdown tool is a joke.
I know of a Linux system that logs and reports intrusion attempts by CodeRed/Nimda, Slapper, et. al., and mails a report to a system admin every morning.
The system admin wasn't pursuing these reports. I asked why.
His response - "Well, those are attempts to exploit a Windows server, and this is a Linux box, so they don't matter."
I made the counterpoint "If one of your system was infected, wouldn't you want to be told about it?"
If every systems admin would take the time to track down the Code Red attempts on their systems, and notify the responsible parties whereever possible, then a lot of the unpatched systems would be shut down (if not by their administrators, then by the ISP supplying connectivity).
I just don't understand an admin with an attitude like that.
www.eFax.com are spammers
. . . but that was my fault because I had upgraded OpenSSH to a non-stock Red Hat 7.3 version. Thus up2date didn't upgrade it, because to it it would have been a downgrade. I'm just lucky cinik is very easily cleaned up. First and only such incident in seven years of running a Linux box 24x7.
Ironically, this never happened before up2date, because I was forced to read every single errata announcement. Now I'm doing it again, but at least up2date and/orc rc will take care of the gruntwork.
Dad? Is that you?
:).
I'm surprised you didn't get that one yet
then again, its behind my firewall, and I'm not running HTTPS, only HTTP. Very funny to watch the log though... within the 1st day, I was hit about 6 times by one IP trying to access it as an IIS server (I guess) and get it to run "cmd DIR C:"
Planning to rebuild the whole box soon, upgrading to the latest and greatest NetBSD 1.6. (its 1.5.2 now).
...because, although it is installed, we don't use it at present, so fixing it didn't seem like a priority. Also, the Cobalt patch was incompatible with our recompiled Apache setup, and Cobalt still hadn't released the source code last time I looked. (I'm sure this isn't right wrt open source licencing, but what are we going to do, sue?) So, if you probed our machine, we are in the 'security risk' list, but I don't think we should be.
Virtually serving coffee
Seriously, what would you suggest?
I personally dislike C - programming securely in C feels like clearing a minefield inch by inch.
So any suggestions?
Haven't seen docs on programming securely in Lisp. Many Lisp coders like to mix data and code, that seems scary, but I'm very very new to Lisp so is that safe? There's little out there to suggest it is or isn't.
Forth seems to have about the same problems as C - buffer overflows. Data-code mixing (but a Forth coder said a solution is to keep dictionaries separate).
Many of the other languages are more high level, very useful and recommended for certain apps, but not suitable for the low level system programming area.
The languages have to be able to hook to C apps very easily.
Comment removed based on user account deletion
And this time count the number who were patched in September when an exploit on Linux was discovered. Because there was no known vulnerability on Linux (or Win32) in July, and if you had Apache, even on BSD, you could avoid it with blowchunks or some other way of chunk filtering. Oh, and OpenSSH was never vulnerable. You can still use pre .92 safely.
... that keeps probing my asshole!!