Web App Scanners Miss Half of Vulnerabilities

seek3r sends news of a recent test of six web application security scanning products, in which the scanners missed an average of 49% of the vulnerabilities known to be on the test sites. Here is a PDF of the report. The irony is that the test pitted each scanner against the public test files of all the scanners. This reader adds, "Is it any wonder that being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!" "NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating, with Hailstorm having the second best rating of 62%, but only after extensive training by an expert. Appscan had the second best 'Point and Shoot' rating of 55% and the rest averaged 39%."

Comments

Not a surprise to me.

[ls671]

> Web App Scanners Miss Half of Vulnerabilities

Well this is no surprise to me. Designing/testing secure systems is much more than scanning for vulnerabilities.

Scanning is only one of the tool to use to accomplish the goal.

Web apps make it so easy to be insecure.

[Anonymous Coward]

The web was clearly never designed to do even a fraction of what it is expected to do today. Now, neither were computers. But at least when it comes to hardware, we're willing to throw everything away and start from scratch. We don't seem able to do that with the web.

Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder that it's so damn easy to write insecure web apps.

Furthermore, it doesn't help that the languages and frameworks commonly used to develop web apps are full of holes themselves. PHP is a very good example of this. Even in the hands of a talented and very experienced developer, it's damn near impossible to develop a site that isn't flawed in some obvious way.

We need to throw it all away. Companies like IBM, Sun, SGI and HP used to routinely do this with their computer hardware. We now need to extend that practice to our software systems. We need to start again. But will we? Probably not, and that's quite unfortunate.

Re:Web apps make it so easy to be insecure.

[Anonymous Coward]

Even inexperienced ADO.NET developers know enough to use paramaterized queries. But I'm not fucking kidding you, I still see PHP code even today where the SQL is generated via string concatenation, without properly escaping input from the user.

It gets really annoying listening to people bash php when the primary attack vector is "programmers" who don't know how to code secure applications. SQL queries written via concatenation is 100%, absolutely 100% the fault of shitty coders, and has nothing, *nothing* to do with the language. Sure, the language could force the use of parameterized queries, but that wouldn't prevent ignorant coders from doing other horrible things. The only reason php "appears" to be really bad is because all the people who *are not server-side coders*, who couldn't code in another language if they tried, wind up using php because it's the easiest language to play with and get some code that apparently "does the job".

The reason most other languages don't look so bad is because a larger percentage of the coders who use them are more knowledgable to begin with, and come from a background of writing backend code. It could simply be that 80% of coders who use ADO.NET are actually "programmers", whereas only 30% of php coders fall into that category. A huge percentage of people who use php aren't programmers in any real sense of the word - they just know a little bit of html and then start hacking (in the worst sense of the word) some php code together to shove things into a database.

You rarely find good programmers who know how to properly use php. But that's not because there aren't a lot of good php coders - it's because php is *the* language that attracts all the people who don't have any business writing server-side code. So simply put, php has a much higher ratio of bad coders vs good coders than any other language.

Re:The cat and mouse game.

[ircmaxell]

To tell you the truth, the percentage of actual vulnerabilities that it finds mean nothing to me. What matters to me is the rate of false positives. Even better would be the number of actual vulnerabilities found divided by the number of false issues found.

I had a chance to see the outputs of a few of these scanners run against a particular open source content management system. Not one of them found an actual, confirmable vulnerability. But one found over 9,000 false positives. All found a fair number of false positives. Even if could find real vulnerabilities, digging though all those false positives to find a real one is a really daunting task.

What I find works better than these scanners is hand audits by someone who knows what they are looking for. It's most definitely an intensive task, but let me ask you. What's more a better use of time, an expert doing a hand audit who may find vulnerabilities that the scanner didn't), or the expert digging through all 9000 of those "results" trying to figure which, if any, are real? I assert that the best use is going to be a combination of the two. Just don't put your faith in either one...