Fuzzing Toolkit For Web Server Testing

prostoalex writes "Dr. Dobb's Journal runs an article discussing the tools necessary for fuzzing (testing a system by generating random input in order to cause program failure or crash). Quoting: 'You are fuzzing a Web server's capability to handle malformed POST data and discover a potentially exploitable memory corruption condition when the 50th test case you sent that crashes the service. You restart the Web daemon and retransmit your last malicious payload, but nothing happens... The issue must rely on some combination of inputs. Perhaps an earlier packet put the Web server in a state that later allowed the 50th test to trigger the memory corruption. We can't tell without further analysis and we can't narrow the possibilities down without the capability of replaying the entire test set in a methodical fashion.'"

Comments

Bump Key?

[Gothmolly]

This is like using a bump key - hit a lock with random impacts and it opens. Spew enough garbage at a program and it will probably die. Eat enough food you find on the ground and you will probably get sick. Other than getting the +1, Obvious award, whats the point?

Re:Bump Key?

[caffeinemessiah]

Spew enough garbage at a program and it will probably die.

But if you can spew garbage at a program and make it die during development, perhaps you can figure out what exactly made it die and fix it. You get the +1 Obvious award, not fuzz testing.

Re:Bump Key?

[skoaldipper]

I'm no web dev, but it would appear that the primary failure point is in the parser itself (for processing POST data). A well tested robust parser under attack should simply respond, "fuzz you!".

Re:Bump Key?

[ehrichweiss]

This actually reminds me of the old hack to crash NT3.X, "telnet host:19|telnet host:53", which would take output from the random character generator port and pipe it to the DNS port which would then crash the DNS server and maybe the host itself. I remember making a friend scream in horror as I did this to his machine across his LAN where he thought it could only be done locally.

Re:Bump Key?

[beyondkaoru]

I'm no web dev, but it would appear that the primary failure point is in the parser itself (for processing POST data). A well tested robust parser under attack should simply respond, "fuzz you!". i'm not a web dev either, but i'm guessing that part of the reason for this is to take a less-tested, less-robust parser (or whatever) and eventually make it into one which is better-tested and more-robust. i mean, we do this with most code (or at least ought to). so the devs can learn how to make their program say "fuzz you!" better :)

Re:Bump Key?

[Bender0x7D1]

You could also create a robust test suite that would try to break the program in an intelligent and repeatable way. Can you do far more random tests than well-thought out tests? Yes. However, random tests, (even a lot of them), don't guarantee that you have good code coverage.

Even better, you take time to make your parser better at error handling. It can take a lot longer but is probably worth it in the long run. It won't eliminate the need for testing, but thinking through all the things that can go wrong is never a waste of time.

Re:Bump Key?

[fractoid]

Your feeble attempt to hack my neural circuits with your random data cannoERROR215 PLEASE CONTACT YOUR NEURAL ADMINISTRATOR

Re:Bump Key?

[some guy I know]

Even better, you take time to make your parser better at error handling. It can take a lot longer but is probably worth it in the long run. It won't eliminate the need for testing, but thinking through all the things that can go wrong is never a waste of time.

This should be -1 Obvious, but the sad fact is that many, many programs out there don't properly or sufficiently validate their input.
Even input from a "trusted" source, such as a DB "owned" by the program, should be checked.
There may be cases where there is a semi-valid excuse for not/minimally checking input (e.g., it would slow down a video decoder to the extent that the video would be unplayable), but in these cases at least the code should fail gracefully, and not just crash the program (like the WMV decoder sometimes does to KPlayer/Noatun/MPlayer/Totem/VLC on Linux).

The best time to add such validation is when writing the input routines.
Unfortunately, that doesn't always happen, for several reasons:

• The programmer writes a "quick and dirty" input routine that is sufficient to get his/her (well-formed) input to the rest of the program, with the view that he/she will write a more robust routine "later".
  Unfortunately, "later" never seems to come, and the original input routine ends up getting patched and patched as bugs manifest themselves.
• The input format is not finalized.
  Why write a bunch of checking code for a format that might change?
  And when the input format doesn't change, nobody goes back and revisits the input routines.
• A programmer feels that validation is unnecessary because input is coming from a "trusted" source (such as the DB I mentioned above).
• An input routine may be written for doing unit testing and the like, with the expectation that it will not be part of the production version of the program.
  Somehow, though, it manages to find its way into the production version.
• Validation code is sometimes tedious to write (i.e., not sexy), and so becomes a lower priority than the more interesting bits of the program.

Sound concept

[FoxNSox]

The concept seems to be a sound one. Sending random input to your web service in a repeatable fashion, but are the tools intuitive? Do they detect the different systems you are running? (ie. an Internet Posting Board) Do they keep track of recent exploits in the wild?

Re:Why does Slashdot refuse to cover

[Alsee]

How about some Paris Hilton stories instead?

Paris Hilton has Boooobieeees!
And here are the pictures!

Slashdot. News For Nerds.

Affirmative.

-

Repeatable Pseudorandom Numbers

[Anonymous Coward]

Any program that does this sort of testing should use a good pseudorandom number generator with a very large period and a manually-specifiable seed. If it logs where it's at in the sequence, it makes it easy to repeat a series of tests. Good generators are easy to build - use a big Linear Feedback Shift Register and SHA or MD5 hash the output.

Too bad cmpnet can't host sites. I never made it past all their interstitial ad and popup junk.

Re:Repeatable Pseudorandom Numbers

[RegularFry]

There's more to it than that to make tests repeatable. Any interesting system will have state beyond that represented by the fuzzed inputs - it's things like IO timings that will still change between runs even with the same random seed. Something as simple as a disk seek time varying between runs might make the difference between a test failing or passing.

Re:Use virtual machines and snapshots?

[FooAtWFU]

Or you could just record it all. The timing might never be exactly the same twice, but if you can just record everything you sent, and then send it again, that's a big improvement.

I'm sure the article talks about this in spades. If only we were to read it.

Can't tell

[arth1]

The article blurb says:

We can't tell without further analysis and we can't narrow the possibilities down without the capability of replaying the entire test set in a methodical fashion.

Yes, you can tell by experts eyeballing the code. Granted, this might be far more work than automated testing, but it's not like testing is the only way to isolate bugs.

replaying set?

[192939495969798999]

isn't the answer in the summary, that you obviously have it record the input as it goes, so you can literally back up and repeat any given random scenario? Without this capability, it would be like having a 3-year old bash away at the keyboard, they're just as unable to repeat anything.

Clean up after yourself

[omfgnosis]

A web application should be cleaning up after each transaction to arrive at a safe state; why wouldn't a web server do the same? If it doesn't, it should.

Re:You had me...

[Holmwood]

Heh. It is funny, but Microsoft actually started doing this a while back. Mainly because IE was so awful. (Anyone remember IE3, IE4?). I used to fuzz IE for fun. It's one of the big reasons I switched to Firefox. By IE6 my own light fuzzing seemed to show IE had gotten a lot better at dealing with really bad, even malicious HTML. So they were having some success and getting better.

But not quite good enough. Nowhere near in fact.

At CanSecWest last year, HD Moore and a student took an hour to hack together a fuzzer that found over fifty flaws in Internet Explorer. http://www.securityfocus.com/news/11387

Think about that. Two person-hours work, that leads directly to the discovery of fifty flaws. That's pretty impressive for a released product that's supposedly had a great deal of scrutiny. There are few other techniques that could discover flaws as rapidly.

The simple thing is, fuzzing is one of the cheapest things to do with one of the highest yields in bugs. Moore noted:

"Fuzzing is probably the easiest way to find flaws, because you don't have to figure out how the application is dealing with input," said Moore, a well-known hacker and the co-founder of the Metasploit Project. "It lets me be a lazy vulnerability researcher."

The idea of using a pseudo-random number generator with a known seed is good, but fuzzing is better if you actually work it so as to try and give increased code coverage (as the article notes). So rather than just spew purely "random" stuff, set up a handshake properly for a particular type of protocol, that will likely take you down a particular code path, and then go into 'random world'.

Indeed, because of the ease, I'd guess a lot of black-hat work these days is fuzzing-based, and then examining the results carefully, discovering specific vulnerabilities, and then trying to weaponize them.

Re:CAPTAIN TACO CENSORSHIP

[ascendant]

who the hell is Captain Taco, and why would I care if he is censoring us?

Re:Why does Slashdot refuse to cover

[Heembo]

How about some Paris Hilton stories instead?
OMG PONIES!

Re:Why does Slashdot refuse to cover

[Repossessed]

Paris Hilton found God in prison, and will only be showing those from now on if it somehow leads to her curing cancer,

tcpdump?

[cowbud]

Is it just me or does it some obvious that you could just dump the stream then use tcpreplay to send the stream back and anaylze the packets it is sending to the webserver? Pick intervals and ranges of packets send them and see what the webserver does. That seems like a pretty straight forward way to narrow down what is going on.

Re:You had me...

[weicco]

You are talking about IE6. That website is talking about something dated 2004! It's 2007 and IE7 has been out for a while.

How about thinning the pipeline?

[whitroth]

The thing that irritates me most are the sites that have 5,478 *different* links. Even on broadband, they take tens of seconds, sometimes over a minute, to load. I'd like one *standard* test to be that they try surfing, like 50% of the US public does, on dialup.

I won't even start on the idiots who have no compression on their cameras, and put jpgs up on their Websites that are over a meg....

            mark

Huh?

[Fuzzlekits]

I do what to servers?....

shoulda woulda coulda

[mr_stinky_britches]

A web application should be cleaning up after each transaction to arrive at a safe state; why wouldn't a web server do the same? If it doesn't, it should.

Ya, and people should always wear their seatbelts, never drink and drive, and floss everytime they brush their teeth; Yet, the majority of humans fail to abide by these 3 simple "should haves".

Maybe you should write a howto covering how you think a web server should behave, and then proceed to write your own implementation from scratch (for my network programming class I had to do this, in C++ with my own sockets library).

Oh, and nevermind the fact that the web server may not have any state information.

I'm sure you're just another troll or BOFH wannabe anyways.

c'ya and good luck, buck.

--
wi-fizzle research, straight outta oakland

Re:You had me...

[ThePerfGuy]

We've been doing this in "Performance Tester Land" for as long as there have been multi-user applications. Thing is, we actually do it rather systematically and against various application layers. I guess takes some folks a while to realize that the wheel has probably already been invented and that maybe it makes sense to talk to some of the folks who are using/building wheels to help evolve the wheel instead of starting over again... and again... and again... and....