Slashdot Mirror
Attack On a Significant Flaw In Apache Released
Zerimar points out a significant flaw in Apache that can lead to a fairly trivial DoS attack is in the wild. Apache 1.x, 2.x, dhttpd, GoAhead WebServer, and Squid are confirmed vulnerable, while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable. As of this writing, Apache Foundation does not have a patch available. From Rsnake's introduction to the attack tool: "In considering the ramifications of a slow denial of service attack against particular services, rather than flooding networks, a concept emerged that would allow a single machine to take down another machine's web server with minimal bandwidth and side effects on unrelated services and ports. The ideal situation for many denial of service attacks is where all other services remain intact but the webserver itself is completely inaccessible. Slowloris was born from this concept, and is therefore relatively very stealthy compared to most flooding tools."
be prepared to feel the slashdot-effect yourself for once!
You may have missed the 'not' in the summary there.
Opera Unite?
It's just holding sockets open; that's the "Hello, world!" of DOS attacks.
I'm finding it hard to believe that Apache is genuinely vulnerable to this. Did nobody see it coming? For real?
If you were blocking sigs, you wouldn't have to read this.
Why isn't IIS vulnerable? Does it just assume the headers are done after some amount of time? Does it have a limit to the number of headers it accepts?
Can this even be fixed without technically breaking the protocol (since it sounds like what's going on is correct behavior, theoretically)?
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Talk about a boring exploit: no chance for expanding the attack into anything other than a DOS, and if it becomes widespread enough, fairly trivial to fix... (just kill the oldest waiting client that does not have a full header when the last client is taken.) I'd be embarrassed to publish something like this....
You can have perlbal or any reverse proxy on the same machine but listening on a different port and then use iptables to redirect like this
# iptables -t nat -A -PREROUTING -d ! 127.0.0.1 -p tcp -m tcp --dport 8080 -j REDIRECT --to-ports 80
and then you don't need to change your apache configuration - and having apache listen on a different port to what users see can break some scripted sites if they read the port number from the apache config.
blog.sam.liddicott.com
And the only resolution right now that I can see is to have a connection timeout.
At least the problem is a denial of service problem and not a problem with intrusion so the damage is easily rectified - restart the web server. Not that you really want to restart it.
And I suspect that other services can be vulnerable to this type of attack too, not only web servers.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Lower Apache's timeout to below 166 seconds.
OpenBSD's pf firewall has some options that can help mitigate the "single attacker, single source IP" version of this attack. Of course if the attackers decide to spread the attack out over multiple source IPs like a DDoS, this becomes much harder to deal with until Apache has a patch.
Filter rules that create state entries can specify various options to control the behavior of the resulting state entry. The following options are available:
max number Limit the maximum number of state entries the rule can create tonumber.
If the maximum is reached, packets that would normally create state
fail to match this rule until the number of existing states decreases
below the limit. no state Prevents the rule from automatically creating a state entry. source-track This option enables the tracking of number of states created per
source IP address.
The total number of source IP addresses tracked globally can be
controlled via the
src-nodes runtime option.
max-src-nodes number When the source-track option is used,max-src-nodes will limit the number of source IP addresses that
can simultaneously create state.
This option can only be used with source-track rule. max-src-states number When the source-track option is used,
max-src-states will limit the number of simultaneous state
entries that can be created per source IP address.
The scope of this limit (i.e., states created by this rule only or
states created by all rules that use source-track) is dependent
on the source-track option specified.
Having read this more this just strikes me as incredibly stupid. Did they publish this? Surely we're just talking about a timeout implementation here where the web server will say "Ahhhh, well you didn't complete that header, bye, bye"?
I'm just waiting for a "Get The Facts" campaign for "IIS vs Apache" on the Microsoft website, along with a completely accurate comparison chart!
Obviously need to verify this, but we already run mod_cband with a per-IP connection limit of 5. This is in place to stop the over-zealous "download accelerators" from taking all our connections and DOS'ing us. I expect it would stop a single attacker using this attack, but we'd still be vulnerable to a concerted attack by MaxChildren/5 IPs.
If you keep lingerers for more then 160 seconds then no wonder this is possible.
It should be non-issues on better designed servers that keep an eye on connections anyway. Any single IP spawning lots of unfinished connections gets flagged fast and remembered for the future, so it will get limited access and bandwidth, marked as abuser etc. This is serving 101.
I just tried it against nginx 0.6.37. The attack appears to work there as well.
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
HTTP 1.1 specifies a status code for "Request Timeout" (408) and "Gateway Timeout" (504).
What is needed, therefore, is a timer running for receiving the complete header, and a second one for accepting the body. The timer for the body can be controlled by the type of request and the Content-Length header. (With, of course, a specific cap.)
Currently, Apache 2.2 has a single timeout value for all types of requests, but it is interpreted differently for the different types.
If your server only handles GETs, the obvious thing is to crank that number down. Unfortunately, for PUTs, the TimeOut value affects inter-packet time in the request, not overall request time.
Strangely, the timeout doesn't seem to run in 2.2.10 and 2.2.11 before data is received. Oh dear. That's an even simpler DoS.
Not quite as stealthy, though. At least as above.
No, it's not. It's holding an HTTP session open. That is not the same thing as a TCP socket.
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
Let me make this clearer for those that aren't very technical: It's holding an HTTP session open and Apache has a limited number of simultaneous HTTP sessions.
All someone has to do is send about 100 requests to your website and leave them open without sending any further information. Nobody else will be able to connect to your web server for a long time. The weekend is coming, so I'm expecting lots of downtime for government sites in the next couple of days...
This type is already known as "attack by a 1000 snails" type attack. It is harder to defned against than you would think. A user can be slow, but coders are hesitant to drop users that ar too slow or too fast.
A user kan just keep the TCP/IP alive by sending one byte every x seconds. If this is patched at http header level, you will see you can do the same kind of attack on the application, that can have limited php or perl sessions.
Keeping idle sessions open on the server for some period of time is not the flaw. The flaw is that some servers (in default configurations) only handles so many of these idle sessions open before reaching a limit and refusing to accept any new sessions, and that this limit is sufficiently low, and that one client can easily reach this limit.
This particularly affects web servers that fork separate processes to service multiple requests at the same time - and this includes the default configuration of Apache 2 on many systems. (Last I checked, PHP's Apache module implementation only works well with the forking proces model because of thread safety issues.)
IIS does not use this antiquated process model - it's threaded. Apache can be configured to use a threaded model too. Unfortunately, thread-unsafe code is still common.
Yes, I agree. I've seen a handful of attacks like this over the years. Maybe not exactly this one, but Apache has been vulnerable to this for years and I thought all webservers were. And I thought people just knew about it already. This one is a tough one to fix, its not like Apache can just patch something. It sounds like an architectural change is needed.
A simple connlimit declaration in IPTables shuts this down fairly easily...
I am a science fantasy fan
Not really, it just means you need more than one attacker.
"Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black
A simple connlimit declaration in IPTables shuts this down fairly easily...
Will that work for anyone using load balancers (read: people with sites worth hitting)?
If you're going to post links to isc.sans.org, can you please post links to the specific article, and not just the main page?
Here is the link to the specific article: http://isc.sans.org/diary.html?storyid=6601
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
No, that won't work. Apache will drop connections which aren't making "useful progress".
However, it's definition of "useful progress" is flawed -- you can keep sending HTTP headers, and it will keep the connection open. You only have to send one every few seconds, so it's a very low bandwidth DOS attack.
Combination - fun iPhone puzzling
IIS worker processes have a request queue. Whether or not you use asynchronous functions to handle requests, there is a fixed maximum number of threads each worker process will run to process requests. While reading from a socket, the worker thread does block but more threads are not spawned to handle connections. Instead, the worker process puts new requests into a queue until more threads are available.
... SPAM SPAM SPAM...
I believe this works because there is a timeout associated with the completion of a request. Sure, it might be difficult to distinguish a slow DoS from a slow client, but it wouldn't be impossible to set a reasonable time limit on non-POST requests. That would be a relatively easy way to fix the issue in Apache.
As far as POST goes... well, that's a different (and valid) way to perform a slow DoS attack:
Server: What would you like? Ham bacon spam, or spam eggs bacon spam with spam?
Client: I'm actually here to deliver some SPAM!
Server: How much SPAM?
Client: SPAM, SPAM, SPAM.... (3 hours later)
Slowloris can do this too. By default, IIS only reads the up to first 48KB of post data (I see much smaller numbers in practice), at which point the request is sent to an extension/app. Before this, the request doesn't leave the kernel-mode driver (http.sys). The apps can easily ignore the data or read more (on a timeout). I wouldn't be surprised if Lighttpd did the same thing (sans kernel driver).
Increase the number of connections in Windows XP post SP2.
We are Turing O-Machines. The Oracle is out there.
You can use fcgid to run PHP in a different process, and then safely run apache multi-threaded. Just FYI for those using PHP.
It's also a good deal faster, and more stable to boot.
If corporations are people, aren't stockholders guilty of slavery?