"Apache Killer" Web Server Hole Plugged

CWmike writes "The Apache open-source project has patched its Web server software to quash a bug that a denial-of-service (DoS) tool has been exploiting. Apache 2.2.20, released Tuesday, plugs the hole used by an 'Apache Killer' attack tool. On Aug. 24, project developers had promised a fix within 48 hours, then revised the timetable two days later to 24 hours. The security advisory did not explain the delay."

Comments

How about 2.0

[Tea-Bone of Brooklyn]

Isn't this serious enough to warrant a 2.0.65 release that patches this too? There are still some applications that won't support the 2.2 branch.

Re:No name yet?

[The Pirou]

What, 'CVE-2011-3192' isn't exciting enough for you?

As it is effectively finished in the wild for 2.2+ so long as everyone practices due diligence, unless someone really high up the food chain manages to still get bit in a sensitive area it doesn't deserve the same status as William H. Bonney.

Re:A bit demanding, no?

[mug funky]

the should have made a post on macrumours if they wanted to be taken seriously.

Apache fix was out before the exploit was known

[Tumbleweed]

The fix is called 'Hiawatha'.

Re:Apache fix was out before the exploit was known

[Tumbleweed]

I don't know much about Comanchee. In what ways is it superior to Hiawatha? Is it more secure?

Re:A bit demanding, no?

[93 Escort Wagon]

Not to mention that any time frame given would've been an estimate, obviously. Knowing exactly how long it was going to take would've required they already knew what needed to be done - and, if that had been the case, I'd think the bug wouldn't have existed in the first place.

Not that it will help Sony at all

[robot256]

Thanks to this post, my server, at least, is fully patched now. How many years will it be before all the production systems in the world have this installed?

Re:Not that it will help Sony at all

[Anonymous Coward]

Sony doesn't have to worry about this bug. Everything over their is protected by their start of the art text-based captcha

Re:Not that it will help Sony at all

[WrongSizeGlass]

Anyone who wants to put in a little effort to scan/parse the html produced can read the characters of that captcha? It's stunning that this is what Sony considers 'security'.

partly same approach as nginx

[mzs]

http://mail-archives.apache.org/mod_mbox/www-announce/201108.mbox/%3C85111090-501E-4C80-AA8F-DD11B94FDF7C@apache.org%3E

* SECURITY: CVE-2011-3192 (cve.mitre.org)
  core: Fix handling of byte-range requests to use less memory, to avoid
  denial of service. If the sum of all ranges in a request is larger than
  the original file, ignore the ranges and send the complete file.
  PR 51714.

I remember reading how people had all sorts of ideas like sorting the ranges, ignoring gaps of less than 80 bytes, noticing that it went afoul of the standard. Around the same time nginx also did a release with the approach of sending back the entire file if the sum of the ranges was more. That was so simple, and it's okay according to RFC 2616 a server MAY ignore the range header, so it's clever too! Glad all the memory handling was fixed-up too though.

Re:A bit demanding, no?

[LordLimecat]

I was about to say, someone should contact their manager so that they can all be fired for tardiness. Who is their manager, by the way?

Re:It's time to shift our paradigm

[FatdogHaiku]

It's worse than you though, in binary!
00110001001101010011100100110001001100000011001000110111
00110001001101010011100100110100001101100011001000110001

Both your UID numbers have 32 zeros and 24 ones...

Re:No name yet?

[Cito]

bah still waiting for this to show up on debian did an apt-get update and upgrade today and only update for apache was 2.2.16 on the stable :(

Re:It's time to shift our paradigm

[maxwell demon]

It's worse than you though, in binary!
[first UID in binary]

[second UID in binary]

Both your UID numbers have 32 zeros and 24 ones...

Is there a specific reason you padded them to 56 bits?

What would have made sense to me would have been no padding (no leading zeros), or padding to 64 bits (because that's how the UIDs are probably stored internally). In the first case, it would be 30 zeros, in the second, it would be 40.

BTW, how did you get it through the lame(ness) filter? I get a filter error in just quoting your post ("Filter error: That's an awful long string of letters there.") Only after removing the binary strings, Slashdot allowed me to even preview.

Re:It's time to shift our paradigm

[stderr_dk]

It's worse than you though, in binary!
[first UID in binary]

[second UID in binary]

Both your UID numbers have 32 zeros and 24 ones...

Is there a specific reason you padded them to 56 bits?

Or why he was looking at the ASCII representation of the UIDs?

A more logical representation of the numbers would be:

(00000000)000110000101010011111101
(00000000)000110000100011011110011

12 (or 20) zeros and 12 ones.

The ASCII representations of UIDs has a Hamming distance of 6, while the more logical binary representation of the UIDs has a Hamming distance of 5.

What would have made sense to me would have been no padding (no leading zeros), or padding to 64 bits (because that's how the UIDs are probably stored internally).

32 bits would be more than enough for those UIDs.

Be prepared to back out!

[David Gerard]

I got bitten by this moving our SSL server from 2.2.9 to 2.2.20 - they changed the config processor and our SSL config broke.

Apache claim that a given "stable" series will keep a constant ABI. They seem utterly unable to comprehend that config files count as part of the ABI. Note that binary modules work the same all the way across 2.2.x ... that doesn't help when a "nothing's changed" upgrade breaks stuff.

The changes are typically tightening the rules and disabling technical violations of them. That's a noble aim, but you need to save it for the next version - you can't pull that shit midstream in a "stable" series!

We previously got bitten by Apache's incomprehension on this point when we went from Tomcat 6.0.16 to 6.0.29.

So, before upgrading anything "stable" from the Apache Foundation: Thoroughly test the result, and make sure you can back out at a minute's notice.

Re:Be prepared to back out!

[David Gerard]

Welcome to enterprise!

Re:Be prepared to back out!

[ledow]

Er, yeah, you have got to love people who complain about things like this. It's a totally valid complaint, software-wise. But if such changes affected you IN ANY WAY then you're just asking for trouble by having insufficient testing. Especially if you have the magic word "SSL" in your server description - obviously something was important enough for you to encrypt traffic, but not to test adequately.

I'm not saying the OP didn't test - but from the way it sounds, they didn't find out until the upgrade (which means that a) they weren't testing, b) they weren't keeping up with events and c) they didn't bother to wait to see if other people hit problems).

Any upgrade, change, or tweak can completely destroy any system you care to mention and even without that, things can go wrong very quickly. A major UPS manufacturer once was forced to issue a patch because they had an internal certificate embedded inside a Java app used by the UPS software that expired and, when it did, it generally took Windows Servers down with it to the point you could not log in to fix the problem. How long do you think it would take you to track that down on an affected server without having a proper known-good environment or a decent testing regime?

That's not the sort of thing you can catch in everyday testing but is ALSO the sort of thing that can happen to you at any time, for any reason, with any tiny change you ever make. If you're honestly reliant on a server continuing to work you really need to test extremely thoroughly and take WORKING backups at each stage that you deem "tested". Even a hotfix, or a config tweak, or even a reboot which fails to write a byte to disk can destroy your machine's configuration, let alone a human tinkering.

The first thing that any deployment should have is a way to deploy the entire hardware again, seamlessly, quickly, guaranteed and to a known working state. Without something like that, you're wasting your time even trying to keep things up, or diagnose problems. And with it, such "problems" are spotted in seconds after a test upgrade and instantly reverted.

It's amazing how many places have *NO* idea how to redeploy their gear to a known-working state, even with claims of backups being present.

Re:Be prepared to back out!

[asdf7890]

The changes are typically tightening the rules and disabling technical violations of them. That's a noble aim, but you need to save it for the next version - you can't pull that shit midstream in a "stable" series!

If your configuration file contains violations of the rules then your configuration is at fault, not the change that tightens the implementation of those rules. By using a configuration that doesn't match the rules (despite the fact the implementation lets you get away with it) you are relying on undefined behaviour and can not expect any guarantee that the behaviour won't change even in a stable branch. Defined/documented behaviours should not change in a stable branch unless there are exceptional circumstances (a security issue that can not be resolved any other sensible way for instance), but changes to undefined behaviour are fair game.

So, before upgrading anything "stable" from the Apache Foundation: Thoroughly test the result, and make sure you can back out at a minute's notice.

Before upgrading anything from anyone in a production environment you should test the result in a test environment first, unless you have a service level agreement that states the provider does that testing and will recompense you for any admin hassle and/or unplanned downtime resulting from a change breaking stuff (even then, I'd stil be inclined to test the change in a non-production copy of the production environment first, I consider that to be basic due-diligence).

None of our production servers (Linux or Windows) automatically install updates (and more manual updates do not get performed on production until tested elsewhere either, of course). Updates get installed in production once they have been given a least a basic kick-around in test environments first in an effort to make sure they don't break any assumptions made by our components/configuration or 3rd party code/config we are using. While more subtle changes could easily make it through this process (though I can't think of any that have off the top of my head), significant problems (like SSL being completely borken until the config is updated, as per your example) should be caught by the test environment so you don't have to diagnose and fix the problem after the updates have gone to live systems where issues like that could cause embarrasing downtime.

Re:It's time to shift our paradigm

[WrongSizeGlass]

Is your world black and white? Mine isn't. So why do our computers have to be binary?

Because computers use logic to operate. When you remove the binary 'black & white' nature of logic you start entering shades of grey. Shades of grey are fine for everyday life, but when they are applied to some things they just don't work. For examples see ethics, the law, loyalty, monogamy, honesty, etc (to see all of these in one example see politics).

Delay

[jker]

On Aug. 24, project developers had promised a fix within 48 hours, then revised the timetable two days later to 24 hours. The security advisory did not explain the delay."

I don't get this part. Even if we everything in the previous statement, 2d + 2d + 1d = 5 days. The 24 was 9 days ago.

What did I miss?

Debian, on the other hand, gave a fix 5 days later (4 days ago), way before the upstream.
http://www.debian.org/security/2011/dsa-2298

Re:Delay

[Tomun]

The summary says Apache 2.2.20 was released on Tuesday, the day after the Debian fix.

Re:Delay

[omnichad]

Even if we everything in the previous statement,
 
What did I miss?

The verb.

Re:It's time to shift our paradigm

[WrongSizeGlass]

BTW, how did you get it through the lame(ness) filter? I get a filter error in just quoting your post ("Filter error: That's an awful long string of letters there.") Only after removing the binary strings, Slashdot allowed me to even preview.

I think he added a space after the first string of binary numbers (before a <br /> tag).

00110001001101010011100100110001001100000011001000110111
00110001001101010011100100110100001101100011001000110001

Yup, that space worked.

Is there a specific reason you padded them to 56 bits?

Adding another eight leading '0's results in the "Filter error: That's an awful long string of letters there." error.

Re:No name yet?

[insane_coder]

The fix was in Debian for a couple of days now. 2.2.16-6+squeeze2 is the patched version for stable.

May not be sufficient

[mrsam]

I haven't looked at this fix in detail, but from the sounds of it, I'm not convinced that the fix is complete.

The attacker, for example, could request 999,999 individual one byte ranges of a 1,000,000 byte document. In a partial range response, each individual partial range gets wrapped into a separate MIME entity. The response from the server is basically a multipart MIME document. There's significant overhead per MIME section. Each single byte of the document gets attached to a header that, perhaps would be around 40-50 bytes long. Still quite a bit of bandwidth amplification.

Re:May not be sufficient

[ledow]

As the fecking article says, the patch:

"weeds out or simplifies requests deemed too unwieldy."

Otherwise, it wouldn't be much of a patch because it wouldn't fix this problem at all.

Re:No name yet?

[Cito]

ah ok thanks I did upgrade to 2.2.16-6 on my debian server. was wondering since the original article mentioned 2.2.20, and when I did the apt-get upgrade it only gave me 2.2.16-6 had me confused :) thanks for info then, and I've upgraded and patched to the 2.2.16-6+squeeze2 now. was hunting for 2.2.20 :P thanks again