Web App Scanners Miss Half of Vulnerabilities

seek3r sends news of a recent test of six web application security scanning products, in which the scanners missed an average of 49% of the vulnerabilities known to be on the test sites. Here is a PDF of the report. The irony is that the test pitted each scanner against the public test files of all the scanners. This reader adds, "Is it any wonder that being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!" "NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating, with Hailstorm having the second best rating of 62%, but only after extensive training by an expert. Appscan had the second best 'Point and Shoot' rating of 55% and the rest averaged 39%."

Comments

Not a surprise to me.

[ls671]

> Web App Scanners Miss Half of Vulnerabilities

Well this is no surprise to me. Designing/testing secure systems is much more than scanning for vulnerabilities.

Scanning is only one of the tool to use to accomplish the goal.

Re:The cat and mouse game.

[ls671]

> If your website has only 1 vulnerability and no scanner detects, score 1 for the bad guys.

except that the "bad guys" mostly use scanners to discover holes ;-))

So interestingly enough, holes detectable with scanners are more exploited.

Web apps make it so easy to be insecure.

[Anonymous Coward]

The web was clearly never designed to do even a fraction of what it is expected to do today. Now, neither were computers. But at least when it comes to hardware, we're willing to throw everything away and start from scratch. We don't seem able to do that with the web.

Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder that it's so damn easy to write insecure web apps.

Furthermore, it doesn't help that the languages and frameworks commonly used to develop web apps are full of holes themselves. PHP is a very good example of this. Even in the hands of a talented and very experienced developer, it's damn near impossible to develop a site that isn't flawed in some obvious way.

We need to throw it all away. Companies like IBM, Sun, SGI and HP used to routinely do this with their computer hardware. We now need to extend that practice to our software systems. We need to start again. But will we? Probably not, and that's quite unfortunate.

Re:Web apps make it so easy to be insecure.

[Anonymous Coward]

Even inexperienced ADO.NET developers know enough to use paramaterized queries. But I'm not fucking kidding you, I still see PHP code even today where the SQL is generated via string concatenation, without properly escaping input from the user.

It gets really annoying listening to people bash php when the primary attack vector is "programmers" who don't know how to code secure applications. SQL queries written via concatenation is 100%, absolutely 100% the fault of shitty coders, and has nothing, *nothing* to do with the language. Sure, the language could force the use of parameterized queries, but that wouldn't prevent ignorant coders from doing other horrible things. The only reason php "appears" to be really bad is because all the people who *are not server-side coders*, who couldn't code in another language if they tried, wind up using php because it's the easiest language to play with and get some code that apparently "does the job".

The reason most other languages don't look so bad is because a larger percentage of the coders who use them are more knowledgable to begin with, and come from a background of writing backend code. It could simply be that 80% of coders who use ADO.NET are actually "programmers", whereas only 30% of php coders fall into that category. A huge percentage of people who use php aren't programmers in any real sense of the word - they just know a little bit of html and then start hacking (in the worst sense of the word) some php code together to shove things into a database.

You rarely find good programmers who know how to properly use php. But that's not because there aren't a lot of good php coders - it's because php is *the* language that attracts all the people who don't have any business writing server-side code. So simply put, php has a much higher ratio of bad coders vs good coders than any other language.

App scanners don't make you secure

[mysidia]

Scanners exist because people want scanners, and so people can sell a product labelled "security scanner". And get a feel-good (false) sense that everything is secure when the scanner reports no issues.

This idea started with the general idea of vulnerability scanner, tools designed to scan hosts for open ports, check software versions, and try exploits against known issues.

The problem with all of them is they can only detect anticipated vulnerabilities.

Unknown vulnerabilities are not properly detected by scanner, because they cannot be anticipated by software.

Much like Antivirus, they need pattern updates and a re-scan when new issues are discovered. Sometimes they don't get updated at all -- sometimes new vulnerabilities are discovered, but a test doesn't get created for the scanner.

Sometimes hackers become aware of security vulnerabilities that the maker of the scanner doesn't become aware of.

Sometimes the hacker can analyze the app you are running (which is industry-specific, not common), and tailor an attack against you, that the scanner vendor could never anticipate.

So are scanners worth something? Sure. But usually not nearly as much as the software vendor bills for them -- they are more fallible than even virus scanners (at least viruses, and malware are finite in number, even if a very large number --- there are more potential security vulnerabilities than one could possibly imagine).

Re:The cat and mouse game.

[ircmaxell]

To tell you the truth, the percentage of actual vulnerabilities that it finds mean nothing to me. What matters to me is the rate of false positives. Even better would be the number of actual vulnerabilities found divided by the number of false issues found.

I had a chance to see the outputs of a few of these scanners run against a particular open source content management system. Not one of them found an actual, confirmable vulnerability. But one found over 9,000 false positives. All found a fair number of false positives. Even if could find real vulnerabilities, digging though all those false positives to find a real one is a really daunting task.

What I find works better than these scanners is hand audits by someone who knows what they are looking for. It's most definitely an intensive task, but let me ask you. What's more a better use of time, an expert doing a hand audit who may find vulnerabilities that the scanner didn't), or the expert digging through all 9000 of those "results" trying to figure which, if any, are real? I assert that the best use is going to be a combination of the two. Just don't put your faith in either one...