Slashdot Mirror
Attack On a Significant Flaw In Apache Released
Zerimar points out a significant flaw in Apache that can lead to a fairly trivial DoS attack is in the wild. Apache 1.x, 2.x, dhttpd, GoAhead WebServer, and Squid are confirmed vulnerable, while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable. As of this writing, Apache Foundation does not have a patch available. From Rsnake's introduction to the attack tool: "In considering the ramifications of a slow denial of service attack against particular services, rather than flooding networks, a concept emerged that would allow a single machine to take down another machine's web server with minimal bandwidth and side effects on unrelated services and ports. The ideal situation for many denial of service attacks is where all other services remain intact but the webserver itself is completely inaccessible. Slowloris was born from this concept, and is therefore relatively very stealthy compared to most flooding tools."
be prepared to feel the slashdot-effect yourself for once!
I thought IIS wasn't affected by this? At least the current versions. But I sure hope there will be a fix very soon, cause this really is a big deal.
You may have missed the 'not' in the summary there.
while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable
You didnt even read the summary?! Wow...
IIS is not vulnerable, as confirmed in TFAs and the summary. Seems as though a DoS attack has affected some peoples' eyes.
Opera Unite?
RTFS
"while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable"
It's just holding sockets open; that's the "Hello, world!" of DOS attacks.
I'm finding it hard to believe that Apache is genuinely vulnerable to this. Did nobody see it coming? For real?
If you were blocking sigs, you wouldn't have to read this.
Why isn't IIS vulnerable? Does it just assume the headers are done after some amount of time? Does it have a limit to the number of headers it accepts?
Can this even be fixed without technically breaking the protocol (since it sounds like what's going on is correct behavior, theoretically)?
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Does it timeout the HTTP header from the initial open or what?
Talk about a boring exploit: no chance for expanding the attack into anything other than a DOS, and if it becomes widespread enough, fairly trivial to fix... (just kill the oldest waiting client that does not have a full header when the last client is taken.) I'd be embarrassed to publish something like this....
You can have perlbal or any reverse proxy on the same machine but listening on a different port and then use iptables to redirect like this
# iptables -t nat -A -PREROUTING -d ! 127.0.0.1 -p tcp -m tcp --dport 8080 -j REDIRECT --to-ports 80
and then you don't need to change your apache configuration - and having apache listen on a different port to what users see can break some scripted sites if they read the port number from the apache config.
blog.sam.liddicott.com
And the only resolution right now that I can see is to have a connection timeout.
At least the problem is a denial of service problem and not a problem with intrusion so the damage is easily rectified - restart the web server. Not that you really want to restart it.
And I suspect that other services can be vulnerable to this type of attack too, not only web servers.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Nginx is a threaded web server and the new darling on the block for people on VPSs and looking for a fast an low resource web server. I wonder how that fares?
Lower Apache's timeout to below 166 seconds.
Wait, you mean the summary on /. is finally correct!?
So... I did see a pig fly past the windows.
OpenBSD's pf firewall has some options that can help mitigate the "single attacker, single source IP" version of this attack. Of course if the attackers decide to spread the attack out over multiple source IPs like a DDoS, this becomes much harder to deal with until Apache has a patch.
Filter rules that create state entries can specify various options to control the behavior of the resulting state entry. The following options are available:
max number Limit the maximum number of state entries the rule can create tonumber.
If the maximum is reached, packets that would normally create state
fail to match this rule until the number of existing states decreases
below the limit. no state Prevents the rule from automatically creating a state entry. source-track This option enables the tracking of number of states created per
source IP address.
The total number of source IP addresses tracked globally can be
controlled via the
src-nodes runtime option.
max-src-nodes number When the source-track option is used,max-src-nodes will limit the number of source IP addresses that
can simultaneously create state.
This option can only be used with source-track rule. max-src-states number When the source-track option is used,
max-src-states will limit the number of simultaneous state
entries that can be created per source IP address.
The scope of this limit (i.e., states created by this rule only or
states created by all rules that use source-track) is dependent
on the source-track option specified.
Having read this more this just strikes me as incredibly stupid. Did they publish this? Surely we're just talking about a timeout implementation here where the web server will say "Ahhhh, well you didn't complete that header, bye, bye"?
I'm just waiting for a "Get The Facts" campaign for "IIS vs Apache" on the Microsoft website, along with a completely accurate comparison chart!
Obviously need to verify this, but we already run mod_cband with a per-IP connection limit of 5. This is in place to stop the over-zealous "download accelerators" from taking all our connections and DOS'ing us. I expect it would stop a single attacker using this attack, but we'd still be vulnerable to a concerted attack by MaxChildren/5 IPs.
If you keep lingerers for more then 160 seconds then no wonder this is possible.
It should be non-issues on better designed servers that keep an eye on connections anyway. Any single IP spawning lots of unfinished connections gets flagged fast and remembered for the future, so it will get limited access and bandwidth, marked as abuser etc. This is serving 101.
Tjat would have been your screen saver.
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
HTTP 1.1 specifies a status code for "Request Timeout" (408) and "Gateway Timeout" (504).
What is needed, therefore, is a timer running for receiving the complete header, and a second one for accepting the body. The timer for the body can be controlled by the type of request and the Content-Length header. (With, of course, a specific cap.)
Currently, Apache 2.2 has a single timeout value for all types of requests, but it is interpreted differently for the different types.
If your server only handles GETs, the obvious thing is to crank that number down. Unfortunately, for PUTs, the TimeOut value affects inter-packet time in the request, not overall request time.
Strangely, the timeout doesn't seem to run in 2.2.10 and 2.2.11 before data is received. Oh dear. That's an even simpler DoS.
Not quite as stealthy, though. At least as above.
mitigate it somewhat:
In httpd.conf
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120
Unless of course this timeout is for after the header is received only... which I don't think it is... but as they say... assumption is the mother of all f*ckups.
No, it's not. It's holding an HTTP session open. That is not the same thing as a TCP socket.
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
2009-06-18 15:25:37 New Apache DOS Tool (Index,The Internet) (pending)
Enlightenment? It's just a flush in the pan.
Windows users: You probably will not be able to successfuly execute a Slowloris denial of service from Windows even if you use Cygwin. I have not had any luck getting Slowloris to successfuly deny service from within Windows, because Slowloris requires more than a few hundred sockets to work (sometimes a thousand or more), and Windows limits sockets to around 130, from what I've seen. I highly suggest you use a *NIX operating system to execute Slowloris from for the best results, and not from within a virtual machine, as that could have unexpected results based on the parent operating system.
...of one of the 14 year olds who uses this, as she runs the script.
"Dodge this." ;)
I'm finding it hard to believe that Apache is genuinely vulnerable to this. Did nobody see it coming? For real?
10.4.9 408 Request Timeout
The client did not produce a request within the time that the server
was prepared to wait. The client MAY repeat the request without
modifications at any later time.
It makes no mention of a default or required timeout.
This is just a case of the specs being written long (10 years this month) before people started adding protection measures into the specs. Luckily, unlike SMTP specs, this is easy to fix without breaking t'internet.
Let me make this clearer for those that aren't very technical: It's holding an HTTP session open and Apache has a limited number of simultaneous HTTP sessions.
All someone has to do is send about 100 requests to your website and leave them open without sending any further information. Nobody else will be able to connect to your web server for a long time. The weekend is coming, so I'm expecting lots of downtime for government sites in the next couple of days...
Apache 2.2.10 mpm_worker, anyone know how to set up that lil script to DoS it ? ive tried setting 10000 connections and its still dont want to stop working
Just tested on www.microsoft.com and I sat there in the header negotiating part for over 3 minutes before the pipe broke on remote end (I didn't do anything.) Slight change in that slowloris makes IIS* vulnerable too.
Heck one could even make it so that the header requests go out at 8 bit/s or less and just whallop any server with a lot of these requests... and have the requests FULLY valid.
Its simple for an admin to block the IP so its not that big of an issue unless it is employed as a DDoS.
Next up: Emacs: better than vi or way better than vi?
Better than vi (c'mon, what decade is this?), waaaay worse than vim.
that I should have to reboot my MacBook for a freaking web browser update? And a minor release, at that. I just rebooted last week for a damned QuickTime update (also ridiculous). Come on, Apple engineers. Get your freaking head in the game!
The article states that this is like a syn flood, but on the HTTP level, rather than the TCP level. So essentially, it sounds like a flaw in previous understanding of HTTP, which will now be rectified by patching servers INCLUDING Apache.
It's a pity the reports of affected servers wasn't more comprehensive though; I'd like to know where Cherokee and nginx stand.
Apple, you can't bring that weak ass stuff up in this humpy bumpy! And you know this, baby! WOOOO!
This type is already known as "attack by a 1000 snails" type attack. It is harder to defned against than you would think. A user can be slow, but coders are hesitant to drop users that ar too slow or too fast.
A user kan just keep the TCP/IP alive by sending one byte every x seconds. If this is patched at http header level, you will see you can do the same kind of attack on the application, that can have limited php or perl sessions.
No, that's just the Swine Flu.
"But this one goes to 11!"
Its just ironic that not so long ago, the Microsoft website was running on a set of Sun Web servers under Solaris. :)
Oh God! Apache web server having a security hole! Better write this day down. The others web servers have already an endless list of days written down, and they are not deployed like apache...
The ideal fix, would be to have a variable timeout on open connection, depending on how many open ones there are. A simple thing like auto timing out the oldest connection once the a near maximum number of connections are open should be enough. Apache should also be limit the number of connections from any one IP address, to a small number. Hope these go in Apache soon.
Keeping idle sessions open on the server for some period of time is not the flaw. The flaw is that some servers (in default configurations) only handles so many of these idle sessions open before reaching a limit and refusing to accept any new sessions, and that this limit is sufficiently low, and that one client can easily reach this limit.
This particularly affects web servers that fork separate processes to service multiple requests at the same time - and this includes the default configuration of Apache 2 on many systems. (Last I checked, PHP's Apache module implementation only works well with the forking proces model because of thread safety issues.)
IIS does not use this antiquated process model - it's threaded. Apache can be configured to use a threaded model too. Unfortunately, thread-unsafe code is still common.
Sendmail and other servers are probably vulnerable to this kind of thing. And it is not necessarily the server application itself may not be where the core of the server slowdown occurs. For example, if one were to spread this kind of attack across several different types of TCP-based protocols (SMTP/SMTPS, IMAP(S), HTTP(S), DNS(tcp version), etc then the operating system's TCP engine might start suffer from too many TCP control blocks. (And it isn't just the memory occupied - some silly implementation might do a sequential scan rather than hash lookups when matching incoming packets to TCP connection blocks.)
There is another version of this kind of attack in which rather than sending incomplete data the attacker simply is extremely lazy about sending TCP ACKs - it does so only enough to keep the connection alive. Yet another alternative is that the attacker maintains a TCP receive window that is just a tad too small to contain what the attacked machine is trying to send back.
There is a flip side of this - one can build an email server that is closely integrated with the TCP stack so that incoming mail is validated while the TCP connection is open. Then if the incoming mail is bogus the machine can go into slow ACK/small receive window mode and try to constipate the TCP stack of the spamming machine. Unfortunatly that technique was more useful before hordes of bots were used as spam amplifiers.
Yes, I agree. I've seen a handful of attacks like this over the years. Maybe not exactly this one, but Apache has been vulnerable to this for years and I thought all webservers were. And I thought people just knew about it already. This one is a tough one to fix, its not like Apache can just patch something. It sounds like an architectural change is needed.
A simple connlimit declaration in IPTables shuts this down fairly easily...
I am a science fantasy fan
What are you talking about? There is settings in IIS regarding max connections as well and presumably would be effected in the same manner if these are set too low. This has nothing to do with fork (process) or non forked (lightweight process) method of serving.... or are you telling me that IIS is currently completely on Fiber level instead of their "Threads" ?
Either way at some point you still need to do some poll on the master socket to peek if there is some requests coming your way so you can relegate it to one of the threads in the pool. Thread pool starvation and all that... you can't really fully escape this (there I even got a pun included.)
Wait, you mean the summary on /. is finally correct!?
No. Here is the link to the article. Not sure what to tell ya about the pig, though. Maybe check the dosage on your meds? ;-)
The problem is a combination of a single client being able to reach the maximum connection limit in the default configuration, and this is more likely on servers that use a forking model (becuase they must set lower limits to avoid exhausting resources.) It does not mean that servers that use a forking process model are inherently vulnerable, and I never said that it did.
Servers found to be invulnerable to this specific attack might have a default configuration that sets the session limit high and limits the amount of requests from a single host, or they might handle idle connections differently when the amount of idle connections is sufficiently high.
Not really, it just means you need more than one attacker.
"Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black
A simple connlimit declaration in IPTables shuts this down fairly easily...
Will that work for anyone using load balancers (read: people with sites worth hitting)?
Our little crappy web server even has more brains than that. Are you seriously telling me Apache never concidered this case and has no effective anti-dos measures to close slow clients when there is connection pressure?
Holding open connections is the hallmark of any DOS 101 implimentation.
Its hard to believe this is the case and I refuse to believe this somehow constititues a new discovery. Its absurd.
Isn't fork() copy-on-write? Why isn't the only difference between a threaded and a fork based server whether or not each instance has its own heap? [ Which doesn't seem too terrible... ]
I cannot believe that in this day and age, people have not come across this simple attack. Wonder why this is even considered a discovery.
IIS threads service multiple connections, news at 11.
If you're going to post links to isc.sans.org, can you please post links to the specific article, and not just the main page?
Here is the link to the specific article: http://isc.sans.org/diary.html?storyid=6601
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
Linux. Still the preferred OS of evil overlords!
Well, come on, you're using a Mac. It's not like you're doing any actual *work*...
.. as to how this even managed to get onto slashdot, but I suppose this is not the first time this has happened either.
If the entire script kiddie populous would utilize this method for attacking webservers, every sysadmins job would get a whole lot simpler. You don't need DoS mitigation devices to protect yourself against this attack, you can run a cron'd shell script that would do the same, or a single iptables rule that limits the amount of connections made per second.
Even in a DDoS scenario, this is trivial to protect against, considering it cannot be spoofed.
If more people would have experience with attempting to deal with spoofed syn flood DDoS's against HTTP, maybe less people would actually pay attention to something as ridiculous as this.
Also to note, there's a reason why the author of the security focus article titled it as 'a cheesy Apache / IIS DoS vuln..'.
gg
I second this assertion. Iptables could easily deal with this attack.
Show me packet captures and log entires, or it never happened.
No, that won't work. Apache will drop connections which aren't making "useful progress".
However, it's definition of "useful progress" is flawed -- you can keep sending HTTP headers, and it will keep the connection open. You only have to send one every few seconds, so it's a very low bandwidth DOS attack.
Combination - fun iPhone puzzling
IIS worker processes have a request queue. Whether or not you use asynchronous functions to handle requests, there is a fixed maximum number of threads each worker process will run to process requests. While reading from a socket, the worker thread does block but more threads are not spawned to handle connections. Instead, the worker process puts new requests into a queue until more threads are available.
... SPAM SPAM SPAM...
I believe this works because there is a timeout associated with the completion of a request. Sure, it might be difficult to distinguish a slow DoS from a slow client, but it wouldn't be impossible to set a reasonable time limit on non-POST requests. That would be a relatively easy way to fix the issue in Apache.
As far as POST goes... well, that's a different (and valid) way to perform a slow DoS attack:
Server: What would you like? Ham bacon spam, or spam eggs bacon spam with spam?
Client: I'm actually here to deliver some SPAM!
Server: How much SPAM?
Client: SPAM, SPAM, SPAM.... (3 hours later)
Slowloris can do this too. By default, IIS only reads the up to first 48KB of post data (I see much smaller numbers in practice), at which point the request is sent to an extension/app. Before this, the request doesn't leave the kernel-mode driver (http.sys). The apps can easily ignore the data or read more (on a timeout). I wouldn't be surprised if Lighttpd did the same thing (sans kernel driver).
How long would you set the timeout?
Let's say it's 3 seconds (if you want to support crappy dialup connections from South Africa, that sounds about right). That means that an attacker can block out your server for 3 seconds... at a time. As soon as you kill his connections, he just recreates them anew. Or, better yet, determines your timeout, and then disconnects just before your server would drop the connection (so that the logs look more benign).
Actually, IIS does use the process model, but each worker process is threaded and talks directly to the kernel-mode HTTP server.
IIS does not use this antiquated process model - it's threaded.
Well, processing requests in separate processes has the advantage that a crashing application module has no chance of bringing the server down. You may call it antiquated when you have found a method to guarantee that software never crashes. This will also give you the right to call the Halting theorem antiquated...
And yet all those web servers that don't use forking are stable enough that they haven't been rewritten...
Yes. In good load balancers (again, people with sites worth hitting) you can define rules to block or limit connections.
It is maybe interesting to notice that at least my current Linux webserver does respond to TCP connection attempts with the normal three packet handshake SYN - SYN/ACK - ACK, but the sockets are not created in ESTABLISHED state. They are still in state syn received. Only when the client sends some data will the socket go into ESTABLISHED state. On this system, a connection that does not actually send some data to the server, will not block a socket or an Apache worker process.
As soon as the client sends some data, the Apache TimeOut timer starts running, which will close the connection sooner or later. Of course, the default of 300 seconds is way to long. And yes, the number of processes and the rate at which they are generated is configurable also.
BTW, Squid uses a much shorter timeout, but is usually limited by the number of sockets per process. However, this is a limit that has to be increased on the operating system before squid can benefit. Apparently, the thread model also has its limitations. I guess it would be surprising if there were a single solution that is perfect for all types of applications. This also means that a webserver like apache must be configured for the application. Have you configured your server?
Increase the number of connections in Windows XP post SP2.
We are Turing O-Machines. The Oracle is out there.
I might subscribe to your theory if it weren't for the fact that lighttpd, which is a first-rate open-source web server, is explicitly listed as not vulnerable.
People love to hate it, but IIS has matured in to a very good web server. It's my choice over Apache.
I wish I felt the same way, since I have to run several IIS servers. I find Apache is more reliable, easier to configure, and makes better use of my cheap commodity hardware. I spend twice as much effort maintaining IIS as I do on the same number of Apache servers. I would honestly take a pay cut if they'd replace my IIS with Apache, because my life would be more enjoyable so the improvement would be worth more than money to me.
We probably have different needs, since we seem to have had different experiences.
You can use fcgid to run PHP in a different process, and then safely run apache multi-threaded. Just FYI for those using PHP.
It's also a good deal faster, and more stable to boot.
If corporations are people, aren't stockholders guilty of slavery?
I wonder if there's some way of using iptables to close very slow connections. But you need to do it to the outgoing speed.
Something you can say 'If this connection sends less than 10k outwards in a minute, close it'.
Then, these connections would be, if I understand correctly, be closed after a minute.
But, only like other methods, wouldn't block legit access from the same IP, which would interfer with proxies. And it would allow http keepalive connections that actually send data.
Granted, it would also close the keepalive connections that some webbrowers hang around keeping open even when there's no data, too, but those things usually are killed by the web server after 15 seconds or so anyway.
If corporations are people, aren't stockholders guilty of slavery?
did anyone confirm this? i am currently attacking my own server with slowloris.pl 0.7 and see no DoS. my setup: Apache/2.2.3 (Debian) sitting behind SmoothWall Express 3 firewall. i see the packets passing through the firewall. i have been running the attack script for some time now and it said it sent 90943 packets when i gave up and killed it. my config files are pretty standard, almost out of the box. can anyone post about a successful DoS using slowloris.pl 0.7?
èåæç©
"becuase they must set lower limits to avoid exhausting resources."
I would say that on a high traffic site the settings are hardly defaults. Maybe you remember one of the first pthread implementations on Linux that actually spawned things as if it was processes with shared memory. It wasn't a big deal then and it certainly isn't now that things are more hidden within one process space instead, if one looks at the threded model approach (I have done ample... and I mean AMPLE fork vs. thread testing scenarios throughout my coding ventures and I have come to realize that only some coders that seem to have drunk the Fiber cool-aid abject the idea of forking. At the end of the day it matters little because the generally bungle on race-contention instead.)
Granted a fork is heavier memory wise and uses semaphores rather than simple mutexes (but one only needs to use semaphores if sending data between processes) but with today's hardware and with the amount of memory it is less of a problem. Also the O(1) scheduler in Linux for instance would work quite effectively as well from that kind of resource scheduling point of view. I would rather have my internet banking done over a forked approach than inside of a threaded approach for security reasons, but I digress.
All this is a non issue... maybe it is an issue running Apache on Windows but I wouldn't do that in any case.
BTW: By default the thread pool would be 2048:
http://msdn.microsoft.com/en-us/library/ms682453(VS.85).aspx
Now I have had far more forked apache's running than that on Solaris/Linux/Irix (I go way back... maybe not as much as I would like in this case BUT still more than most.)
It's just holding sockets open; that's the "Hello, world!" of DOS attacks.
Yeah, a very effective 'Hello World', unfortunately.
It seems to me that if you have a single thread responding to multiple requests, a single slip in the code anywhere past the the pool is going to open every connection to the pool to every other.
If not, the pool will have to have some sort of wall between connections, and there will be a way to attack the wall.
Thus, IIS may not be "vulnerable" to this version of the attack, but it will be vulnerable to an attack modified to whatever attempt IIS makes at putting up a wall, and it will also (because we know that no programer or pool of programers is perfect) be vulnerable to all sorts of cross-talk between connections.
Computer memory is just fancy paper, CPUs just fancy pens with fancy erasers; the 'net is just a fancy backyard fence.
Well, it is a little embarrassing to have your elementary students give you the "kancho".
Computer memory is just fancy paper, CPUs just fancy pens with fancy erasers; the 'net is just a fancy backyard fence.
the bubble: everyone is vulnerable. well at least one year ago (or something) every kind of http server was. the bug: there is no bug. this is one of the flaws we live with that would require on-the-fly fixes (by sysadmin). im not sure how this became a news headline. ah wait. i also remember the "dns exploit" that was (re-)"discovered" and "fixed" by cisco,microsoft and some john doe. i also remember djb telling them ages before about it - and ofcourse providing the fix as well. but no one heard about it until the companies decided they're ready to make some profit from it. open your eyes, people. the glitch: there is. a quite-general fix was described above by someone (killing the oldest unfinished requests) but on a heavy attack this could also kill legit requests. one should really customize a tcp pattern-based filter depending on the attack combined with safe application layer rules. p.s. the "timeout" settings are useless. you can fingerprint that and force timeout resets on the server side with minimum bandwidth consumptions. see an old article with source code, etc (i think you need to fix a small compilation error to be able to use it): http://pub.mud.ro/~cia/computing/apache-httpd-denial-of-service-example.html
This affects a number of webservers that use threaded processes and ironically attempt to limit that to prevent memory exhaustion - fixing one problem created another..
1. fingerprint the timeout on serverside
2. dig the sitemap from target
3. build a list of browsers to advertise to server during request
4. buy proxies from black market
5. start requests thru proxies to target
My shame, I think I got the port numbers for dport and to-port the wrong way around....
sam
blog.sam.liddicott.com
There http://pyloris.sourceforge.net/