Bug in zlib Affects Many Linux Programs

SirTimbly writes: "CNET is reporting that there is a buffer overflow problem with zlib in linux, which is used for network compression. Supposedly, someone could remotely cause a buffer overflow through mozilla, X11 and many other programs." The advisory from Red Hat is available.

Comments

more information - better article

[marks]

This article gives more information, and links to vendor advisories: http://www.linuxsecurity.com/articles/security_sou rces_article-4582.html.

Re:more info please

[Anonymous Coward]

http://online.securityfocus.com/advisories/3941

Some More Links

[Zach Garner]

Some More Links.

Linux only?

[egoots]

zlib is not os dependent. Many Windows based products/projects use it as well. Is there some linux specific issues related to this overflow issue?... or is it just a headline hype thing

Re:more info please

[Iguanaphobic]

Here.

Credit where credit is due...

[ubiquitin]

Owen Taylor at RedHat found the bug. He works on GTK among other things, as you can see from the GTK+ release notes he posted earlier this month: mail.gnome.org/archives/gtk-devel-list/2002-March/ msg00161.html

Dumb security question

[wrinkledshirt]

On the stuff I've been reading about finding and fixing buffer overflows, it seems like it's generally not too hard to spot where these things could potentially happen.

My question is this: How feasible would it be for someone to take a computer and have it do nothing but pattern-matching through all the source code in a typical Linux distribution, looking specifically for problem areas like these? Obviously we couldn't rely on it as a foolproof audit, but has something like this ever been considered?

Re:Dumb security question

[SirSlud]

Some software packages do this .. purify, etc. They're pretty expensive tho. The problem is that the logic that results in a buffer overflow error can be VERY complicated, and so its extremely difficult to spot sometimes even for the seasoned developer, nevermind a clever regex.

On the flip side, finding lots of memcpy's instead of strncpy might help you find the 'dumb' overflow bugs, but one would hope those arn't the ones we're most concerned about. :P Mostly, when copying and moving and generally playing with memory, if you spot functions without buffer limit or max byte limit arguments, you *might* be openening yourself up for trouble. Unfortunately, as I said, those are the easy ones. :) In reality, buffer overflow errors (and off-by-one bugs generally follow the 'simple errors can result from terribly complicated logic' construct of buffer overflow bugs) can be extremely difficult to spot if your input parsing/copying/moving mechanism is non-trivial.

Re:Dumb security question

[Carnage4Life]

On the stuff I've been reading about finding and fixing buffer overflows, it seems like it's generally not too hard to spot where these things could potentially happen.

From this statement I assume you are not a programmer. Buffer overflows caused by using known unsafe library functions (e.g. strcpy, strcat, gets, etc.) can be handled by simple pattern matching but actually investigating the code to make sure every memory/array access does not go out of bounds is not a simple pattern matching problem.

However some automated techniques have been developed to discover buffer overflows and similar errors in a generic manner. The most significant efforts I have seen are the Stanford Meta-level Compilation Project and the /GS switch in Visual C++.NET.

Re:Dumb security question

[jilles]

The whole issue is not to use insecure languages like C. Such languages allow all sorts of memory manipulation and typically depends on the programmer to secure. The situation is made worse by the lack of something pretty essential: good libraries for string manipulation. While 3rd party libraries are available you can't assume these to be present so many developers still use char*.

Code reviews help, testing helps, good programming helps. But neither of these practices has succeeded in eliminating this type of bugs. It is just not good enough, witness a zillion bugs and security breaches on all major OSes (including the ones deemed secure, and yes I am talking about BSD) throughout the last decade. These OSes only differ in how the issues are dealt with. The occurance of the issue is a fact of life for all of them.

There's no C developer that can claim his program is completely free of buffer overflows (many foolishly do however). There may be some undetected errors in the program, the progrm may depend on third party code that contains bugs (e.g. the compiler or one of the standard libraries). Most likely bugs in all three categories are present.

Automatic checks are indeed the solution to the problem and modern languages build these checks into the run-time environment, where they belong. Buffer overflows are a non-issue in Java, for instance. The exception of this is native code and the JVM itself (written in C).

To eliminate buffer overflows, getting rid of the C legacy is the only solution. Java is probably too controversial as an alternative right now (though arguably it is quite up to the task as far as server side development is concerned) but there are other alternatives. Rebuilding serverside services like ftp, dns, ssh, smtp, pop, etc. is mandatory since each of these services has widely used C implementations that are frequently plagued by buffer overflows. The only way to guarantee that there are none left is to reimplement them.

Re:more info please

[Bob(TM)]

This was a reference on the RedHat advisory [RHSA-2002:027-22] Vulnerability in zlib library (powertools):

Mitre
Gnome

The Mitre page says it's still under review.

advisory & zlib 1.1.4 url

[Anonymous Coward]

The advisory for zlib-1.1.3 is at:

http://www.zlib.org/advisory-2002-03-11.txt
Zli b Advisory 2002-03-11
zlib Compression Library Corrupts malloc Data Structures via Double Free

The new zlib (1.1.4) is at:

ftp://ftp.info-zip.org/pub/infozip/zlib/zlib-1.1 .4 .tar.gz

Staticly linked-implication

[joeflies]

As the article points out, anything that uses zlib staticly can't be fixed by the new zlib patch until it's recompiled.

As I'm not a programmer, what can I grep to search stuff I've compiled from source to determine what's using staticly linked zlib?

Re:Staticly linked-implication

[Stonehand]

You could write a script using 'nm' and 'grep' -- once you identify some functions in zlib. If they have a common prefix, search on that.

Of course, if you stripped the symbols out of the binaries, then the function names won't be there for nm to find and you're quite screwed -- basically you'd have to go grab the sources again and scan the Makefiles and perhaps the code itself for zlib references.

Re:Staticly linked-implication

[cicadia]

You're absolutely right -- the only thing that a binary download will fix is packages using the libz.so shared library. Most software seems to link with the library statically. This is a huge problem.

I'm currently running this command against my /usr/src directory, just to get a preliminary list of packages to recompile:

grep '-lz' `find . -name 'Makefile'` > ~/zlib-dependencies

Assuming you've still got your source tree intact since you compiled, this should find all makefiles which reference the zip library. If you've deleted any source directories, you will have to untar them and run configure again to build the makefiles.

Re:Staticly linked-implication

[umoto]

There are the right ways, then there is the easy, 99% effective way. The easy way is to search for very specific error message strings, which are sort of a fingerprint for most software. I compiled zlib then used "strings libz.a" to find these error messages:

too many length or distance symbols
invalid literal/length code

A quick grep for one of those two strings reveals quite a number of statically linked versions of zlib in /usr/bin.

Re:Version 1.1.4 fixes the problem

[Dimensio]

And the site seems to be suffering from the /. effect. Either that or they didn't patch and someone took advantage of the exploit.

No buffer overflow!

[lkaos]

From the zlib.org page:

The vulnerability results from a programming error that causes segments of dynamically allocated memory to be released more than once (aka. "double-freed"). Specifically, when inftrees.c:huft_build() encounters the crafted data, it returns an unexpected Z_MEM_ERROR to inftrees.c:inflate_trees_dynamic(). When a subsequent call is made to infblock.c:inflate_blocks(), the inflate_blocks function tries to free an internal data structure a second time.

Because this vulnerability interferes with the proper allocation and de-allocation of dynamic memory, it may be possible for an attacker to influence the operation of programs that include zlib. In most circumstances, this influence will be limited to denial of service or information leakage, but it is theoretically possible for an attacker to insert arbitrary code into a running program. This code would be executed with the permissions of the vulnerable program.

Duplicate deletions are not the same as buffer overflows and are no where near as easy to exploit. In fact, I have _never_ seen a duplicate deletion exploitation other than a simple DoS. Not to mention the fact that it requires a special series of calls from the calling program.

In summary, the world hasn't come to an end and Free Software is all-the-sudden as vunerable as closed source software. Put the pills down and relax :)

Re:No buffer overflow!

[Mike Shaver]

If the application has the "wrong" pattern of allocations and frees, it may be exploitable. One such pattern is the freeing of x, an allocation -- which gets x-(sizeof void *) -- and then the subsequent double-freeing of x.

traceroute provided an example of an exploit for a double-free in a setuid program.

ouch

[Phexro]

Is it just me, or have there been a really huge amount of security issues with Free/Open Source software this year?

It just seems like there's a new hole (or two) every week. Let's see, we've had openssh, zlib, php, mod_ssl, cvs, cups, rsync, exim, ncurses, glibc and more, just since January. We've still got two-thirds of the year to go. Anyone want to make bets on what other projects will get hit? I think we're going to see problems with XFree86, samba, and apache.

So, my question is this: Do you think that this is simply a bad time for FS/OSS security? Are we at the threshold where there are enough eyes on the code to locate these kinds of bugs? Or is the quality of FS/OSS declining?

Re:ouch

[the Man in Black]

All of whom were stamped out within hours of being found.

That's the strength of open source.

Easy Workaround!

[dtrombley]

Well, it won't prevent the DoS aspect - but, from the malloc manpage:

Recent versions of Linux libc (later than 5.4.23) and GNU libc (2.x) include a malloc implementation which is tunable via environment variables. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free() with the same argument, or overruns of a single byte (off-by-one bugs). Not all such errors can be proteced against, however, and memory leaks can result. If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored; if set to 1, a diagnostic is printed on stderr; if set to 2, abort() is called immediately. This can be useful because otherwise a crash may happen much later, and the true cause for the problem is then very hard to track down.

Seems worth it while all pour through the symbol tables of our static binaries (and recompile the stripped ones. =( )

On another note, I've always regarded security bulletins as a one-way process... For example, I couldn't find a way to tell RedHat they'd omitted this (seemingly important?) reminder. Any thoughts about this? (admittedly i didn't look very hard for very long)

Re:Should I upgrade my kernel?

[ceswiedler]

The only dynamic linking the kernel uses is modules, which aren't used for providing library routines like zlib. The kernel does not link .so files. The code is almost certainly cut-and-pasted into the ppp compression code somewhere.

One more bug OpenSSH is affected by...

[Anonymous Coward]

OpenSSh uses zlib - if you happen to compile OpenSSH statically with zlib (I think thats the default), one more upgrade cycle after the latest OpenSSH 3.0.2p1 bug... :(

Re:The article says this is only affecting Linux

[Ded Bob]

I think it does hit FreeBSD. FreeBSD uses zlib v1.1.3. v1.1.4 has the fix (http://www.gzip.org/zlib/). No security announcement has arrived in my mailbox, yet I expect it soon.

It's not a problem in zlib per se

[Starship Trooper]

This bug causes zlib to free() a malloc'ed block of memory more than once. free() on most other OS's (including Windows, FreeBSD and OpenBSD) is smart enough to check for this and will print a warning instead of destroying the heap; glibc's malloc (and by extension, Linux's) does not and will gleefully make a mess out of the whole memory space. This can cause all sorts of buggery when the next malloc() occurs, including what amounts to a buffer overflow exploit.

So, you should download the patched zlib, but you should also email the glibc maintainers and demand that they implement a sane, error-checking malloc()/free() system. Linux's current allocation model is a disaster waiting to happen.

Re:It's not a problem in zlib per se

[slamb]

This bug causes zlib to free() a malloc'ed block of memory more than once. free() on most other OS's (including Windows, FreeBSD and OpenBSD) is smart enough to check for this and will print a warning instead of destroying the heap; glibc's malloc (and by extension, Linux's) does not and will gleefully make a mess out of the whole memory space. This can cause all sorts of buggery when the next malloc() occurs, including what amounts to a buffer overflow exploit.

If you want this behavior, you can get it easily on Linux/glibc. From the malloc(3) manual page:

Recent versions of Linux libc (later than 5.4.23) and GNU libc (2.x) include a malloc implementation which is tunable via environment variables. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free() with the same argument, or overruns of a single byte (off-by-one bugs). Not all such errors can be proteced against, however, and memory leaks can result. If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored; if set to 1, a diagnostic is printed on stderr; if set to 2, abort() is called immediately. This can be useful because otherwise a crash may happen much later, and the true cause for the problem is then very hard to track down.

Re:It's not a problem in zlib per se

[Anonymous Coward]

so what you are saying is that slashdot has been wrong in the past to criticise microsoft for seeking performance ahead of robustness. glad we've cleared things up.

mjl

Re:It's not a problem in zlib per se

[Ed Avis]

AFAIK: The kernel doesn't use the glibc C library. It has its own memory management code which presumably the kernel zlib code uses. This memory manager may or may not guard against free()ing the same area twice.

But what is the use of zlib in the kernel anyway? Just to uncompress the vmlinuz image before the kernel starts? If so it's not much of a vulnerability, if you can corrupt the vmlinuz file then you can control the whole system anyway.

This is why you clear pointers after freeing them

[coyote-san]

This is why you ALWAYS set a pointer to NULL after freeing it, even if it's "totally unnecessary" because you're about to free the structure holding the pointer.

This doesn't prevent attempts to free the previously freed pointer, but that will generally do a lot less damage than freeing a real malloc'd address. And during development it's trivial to add an assertion checking for a NULL pointer before any free().

Re:Should I upgrade my kernel?

[Mr Z]

One place kernel uses zlib is to compress the kernel boot image. The kernel image then gets decompressed during bootup. So, from the standpoint of "the kernel uses zlib", the kernel is affected. There is, however, no new vulnerability introduced as far as I can tell. To attack the zlib-based decompression that the kernel performs, an attacker would need to modify the compressed kernel image that is used to boot the machine. I can think of far more fruitful ways to compromise a machine by modifying the kernel image than by trying to dork the zlib decompression that happens before the kernel even runs.

Another place the kernel uses ZLib is when mounting compressed filesystems. (Compressed RAM disks and zisofs come to mind.) In this case, you're asking a live kernel to decompress arbitrary data. These are only issues when mounting untrusted media. If you made the media yourself, then your only risk is that corrupted media might cause a kernel oops. And if you don't have cramdisk and zisofs compiled in, you're safe.

Other places the kernel seems to use ZLib (from a cursory scan of the source -- there may be others):

• jffs2 -- Journalling Flash Filesystem version 2
• ppp -- used for ppp_deflate option

In any case, the kernel is a statically linked entity, with a minor exception for modules. ZLib is not a module, therefore to upgrade ZLib in the kernel, you'll need to rebuild the kernel. And it doesn't appear to be as easy as just upgrading ZLib and rebuilding the kernel. The kernel has multiple modified copies of ZLib in its source tree. I'd wait for an official kernel patch.

--Joe

Then there's still a problem in glibc malloc()

[Starship Trooper]

*BSD's malloc manages to simultaneously provide high performance while also providing robust (and highly configurable) error checking. glibc's MALLOC_CHECK_ variable does far too much and isn't nearly as fine-grained as BSD's options. Read the "TUNING" section of FreeBSD's malloc(3) manpage. It puts Linux to shame as far as clarity, usefulness, and convenience goes. You only turn on the error checks you need, instead of a few general and poorly-implemented checks in glibc's malloc.

Why Linux can't follow in the supposedly-inferior BSD's footsteps is beyond me.

SuSE advisory (affected packages)

[hysterion]

Part 1: libz/zlib
Part 2: packages containing libz/zlib

From part 2:

The packages affected by the double-free() libz bug can be devided into
two categories:

1) packages that link dynamically against the system-provided
compression library. These packages get fixed automatically with
the update of the libz package as described in SuSE-SA:2002:010.
Please note that the processes will continue to use the old
version of the libz.so shared library if the have not been
restarted after the libz package upgrade.

2) packages that contain the compression library in their own
source distribution. These packages need an individual bugfix.
We have prepared update packages for this software that can be
downloaded from the locations as shown below.

The following is a list of the packages in category 2):
gpg
rsync
cvs
rrdtool
freeamp
netscape
vnc
kernel

Re:That's better

[cpeterso]

If they ripped them off as you say they would not be able to licence them as GPL now could they?


sure they could. The BSD license lets anyone do whatever they want, including relicense the code as GPL. There is already BSD code in the Linux kernel.

Re:This is why you clear pointers after freeing th

[greed]

Calling free() on a NULL pointer is a no-op. No check or assertion is needed.

The assertion lets you catch the logic error that led to the second free during debugging. Presumably, your code path wasn't expecting the pointer to already be free at that point; otherwise, you would have designed it to handle that case already.

Then, in production code, if you do take that path, you'll get the harmless no-op free(0). (You do build production with -DNDEBUG, right?)

Re:more info please

[ncc74656]

why the fsck would you statically link in zlib?

Um...because that's the way nearly every package that uses zlib links it? For instance, OpenSSH AFAIK will only statically link it (so if you rebuilt OpenSSH last week to fix this hole, you get to rebuild it again :-) ).

(I'm rebuilding OpenSSH on the work machines right now...I checked to see if it would link to libz.so, but it seems to only want libz.a.)

Remote upgrades : be careful

[chrysalis]

If you have to remotely upgrade the zlib library, be *very careful* .

Because SSH/OpenSSH depend on zlib, if you replace your current libz.so file with another version whoose API has a bit changed, your SSH server won't work any more.

So if you don't have access to the console, open a classical 'telnet' port for a few minutes, just during the upgrade. Once you've checked that SSH is still ok, you can remove the telnet daemon.

If SSH doesn't work any more after the zlib upgrade, recompile SSH.

Re:Remote upgrades : be careful

[Electrum]

So if you don't have access to the console, open a classical 'telnet' port for a few minutes, just during the upgrade. Once you've checked that SSH is still ok, you can remove the telnet daemon.

Since the SSH server forks after you've connected, you can safely stop the server while connected via SSH. You never need to use telnet. Just make sure that you can still connect before disconnecting from the original SSH connection.

Pointer aliasing...

[Tom7]

I dunno, most double frees come from freeing DIFFERENT copies of a pointer. Setting one to NULL won't help in this case...

(A much better solution is to use a garbage collector. ;))

Would be impossible in garbage-collected language!

[Tom7]

Like most recent security holes in linux software, this one would be unexploitable in a modern safe language. (In fact it would be *impossible* to make this error in a garbage-collected language!)

The typical response I hear to this kind of comment is that "high level languages are inefficient". (I don't belive this is true, but most other people here do.) But whatever, let's pretend they are.

Now, what kind of crazy world do we live in where we value performance more than correctness (security)?? We are seeing more and more security holes as we try to write bigger and bigger packages in C. Why do we accept this? Who here really cares more about the performance of zlib than the time it takes for them to patch all of their statically-linked software, and their risk of being rooted until they do? I sure don't.

Forget about all this "coding practices" stuff. It simply takes too much effort to produce bug-free code in C. The OpenBSD people, kings of code review, just had an exploitable bug in sshd! While we need to use C for some tasks (ie, most parts of the kernel), I think we are seriously unpowered to do this for most applications (as evidenced by the high number of simple errors made, and sometimes caught).

If we simply wrote our software in high level languages, we would automatically rule out the largest classes of security holes, which would give us a lot more time to work on more important things, like high level architecture review and optimizations. I think we'd end up with a better system. So what's keeping us?

For more discussion, see our big argument in the story about the OpenSSH root hole. http://slashdot.org/comments.pl?sid=29123&cid=3124 957

XFree86 4.2.0??

[gweihir]

The latest XFree comes with a copy of zlib 1.0.8.

Does anybody know where this is used and whether I should do a rebuild with the current 1.1.4 version?

In addition to gs, this seems to be the only software package that contains zlib in it. I found it because there is a /usr/X11R6/lib/libz.a on
my Linux system.

Traceroute hack not a double free

[Huusker]

I think the traceroute hack is an example of freeing garbage, not a double-free(). The garbage being freed happens to be part of the command line, which is how the hacker injected his /bin/sh. The traceroute exploit description did not give full details, but I don't see how it could be possible to use modify ((int*)p)-1 using the zlib vulnerability. Remember that all mallocs are sizeof(8) aligned and have a minimum size of 16 (with overhead and internal fragmentation).