Slashdot Mirror
Web App Scanners Miss Half of Vulnerabilities
seek3r sends news of a recent test of six web application security scanning products, in which the scanners missed an average of 49% of the vulnerabilities known to be on the test sites. Here is a PDF of the report. The irony is that the test pitted each scanner against the public test files of all the scanners. This reader adds, "Is it any wonder that being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!" "NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating, with Hailstorm having the second best rating of 62%, but only after extensive training by an expert. Appscan had the second best 'Point and Shoot' rating of 55% and the rest averaged 39%."
> Web App Scanners Miss Half of Vulnerabilities
Well this is no surprise to me. Designing/testing secure systems is much more than scanning for vulnerabilities.
Scanning is only one of the tool to use to accomplish the goal.
Everything I write is lies, read between the lines.
The key message here is that simply testing your web site with a vulnerability scanner doesn't make it secure. Well, duh.
PCI is still important because before the guidelines, most people weren't scanning their web sites at all. Even when they knew how - they couldn't convince management it was worth the trouble, time, dollars, and so on. And without scans, the number of discovered web vulnerabilities approaches 0%.
PCI isn't just about scanning your website, either. There's hundreds of things you have to do to secure everything from the physical layer up to the application layer. And having PCI be required to process credit cards makes everything much more secure. I'm talking about small businesses so cheap they don't want to put LOCKS on the doors between the outside world and the servers holding your plain-text, unencrypted credit card numbers, and who don't have the expertise to set up a web camera on their own building.
You might not like PCI, it might be inconvenient, but it's necessary to protect the general public.
Disclaimer: I am an information security professional.
> There seems to be a little bit of a disconnect between the sales force and the operations team
No kidding, I can't believe that such a company exists ;-)))
Everything I write is lies, read between the lines.
The web was clearly never designed to do even a fraction of what it is expected to do today. Now, neither were computers. But at least when it comes to hardware, we're willing to throw everything away and start from scratch. We don't seem able to do that with the web.
Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder that it's so damn easy to write insecure web apps.
Furthermore, it doesn't help that the languages and frameworks commonly used to develop web apps are full of holes themselves. PHP is a very good example of this. Even in the hands of a talented and very experienced developer, it's damn near impossible to develop a site that isn't flawed in some obvious way.
We need to throw it all away. Companies like IBM, Sun, SGI and HP used to routinely do this with their computer hardware. We now need to extend that practice to our software systems. We need to start again. But will we? Probably not, and that's quite unfortunate.
From what I recall doing this for sites that handled credit card processing (me being in the tested side), those tests are pretty much worthless.
If you had 1 vulnerability, you'd get pages of false positives or irrelevant information. I recall a particular 10 page report we got back that we were advised to fix or we'd fail on. The only item to fix was the version of the web server was just one behind current. The changelog indicated that it was to fix a vulnerability on a different platform, so it was completely unrelated to us. We'd frequently have points marked off because we couldn't be pinged or portscanned. I'd have to open the firewall up to them, just to be scanned. Our security would identify an attempted port scan as a hostile action, and react by dropping all traffic from them. Sorry my security stopped your scanning, but that's the intention of it. {sigh}
After opening the firewall to them, and changing the version number on the web server (there were reasons we couldn't do the trivial upgrade), we passed with flying colors.
For them, they were interested in the version numbers handed off by the server, not what they actually were. For example, if it was Apache, we could have it report Apache version 9.9.9, and that would have made us pass on that part without fail for years.
Serious? Seriousness is well above my pay grade.
To tell you the truth, the percentage of actual vulnerabilities that it finds mean nothing to me. What matters to me is the rate of false positives. Even better would be the number of actual vulnerabilities found divided by the number of false issues found.
I had a chance to see the outputs of a few of these scanners run against a particular open source content management system. Not one of them found an actual, confirmable vulnerability. But one found over 9,000 false positives. All found a fair number of false positives. Even if could find real vulnerabilities, digging though all those false positives to find a real one is a really daunting task.
What I find works better than these scanners is hand audits by someone who knows what they are looking for. It's most definitely an intensive task, but let me ask you. What's more a better use of time, an expert doing a hand audit who may find vulnerabilities that the scanner didn't), or the expert digging through all 9000 of those "results" trying to figure which, if any, are real? I assert that the best use is going to be a combination of the two. Just don't put your faith in either one...
If a man isn't willing to take some risk for his opinions, either his opinions are no good or he's no good
I'm going to take issue with this and say the problem is with the internet itself, RAD applications, businesses, and self-taught coders. Allow me to explain.
Half of the .NET code I write is copy/pasted from some other source, because the entire CLR is too complicated for a single person to understand. If I want to do a lookup table, there are a dozen ways to accomplish it, just using the objects provided by the runtime. I don't care how fast it is, unless it's called every page view, so I just google "C# lookup" and get piles of examples. Copy/paste, I'm done. Doesn't matter if it's from MSDN or a Microsoft blog or a random coder blog or wherever else, the code looks good and it works. I have no idea if the example failed to initialize some critical component.
My employer doesn't want to pay me to read, I am supposed to be providing output they can sell to clients/customers. So I don't get a lot of time set aside for training. The way I learned .NET was our tech lead opened up a team meeting and said "I think .NET is the way of the future, is there anyone opposed to going this way?" And the only real objection was it will take longer to produce the next version of our deliverables. Management was fine with that, so we took the leap.
We didn't sit down in a classroom and learn how things are supposed to be done. We didn't get a copy of something like Petzold's Windows bible, or Prosise MFC bible where it goes into depth about what you're doing and what things mean when the IDE puts junk in places for you. Visual Studio 2003 and above make it very easy for you to have no idea what you're doing, and still accomplish something. A quick google search can fill in all of the gaps so you have something functional.
The same with 'Learn X in 24 hours' or 'X for dummies', lots of code samples exclude error checking/handling. Oh yes, MSDN is full of these examples. Sometimes they suggest "error handling has been omitted for clarity", while sometimes it's just assumed. Other times the author has no idea they should be handling errors because it works for them.
So you have piles of coders learning on-the-fly, either because they can't afford the big book or because they have deadlines to meet. Copy/paste something without taking the time to fully understand what's happening, and you get potential problems. In short, easy access to code snippets makes you think you're able to do lots of cool stuff in a new language. Unless you take the time to understand everything you're running, every line of code, you're going to have problems at some point.
Why do you think people still make mistakes like putting form variables directly into SQL? The code snippets are out there, either in the corporate source control or on random blogs. Copy, paste, pwned.
An example, for those of you who wish to tl;dr me you can stop now.
I used MyGeneration templates to come up with database calls for our SQL database, which used Data Access Blocks or some kind of MS best practice to write functions which called stored procedures, so you could essentialy call stored procs exactly like any other function. It generates a call for every stored proc in the database, so you can make fundamental changes to the data structure, re-generate the data access library in a few seconds, and then fix the few calls where the parameters changed.
Very handy, except that the 'execute non-query' template had a bug in it, where the data connection never closed. We never had any problems with this app in production for 3 years. Suddenly in testing, we got a pooled connection exceeded timeout. Turns out the bug only shows up when the call happens most page views, when logging user visits in this case. Other non-query calls happened infrequently enough that they never exceeded the 100 connection default limit, live and in production for 3 years.
Our tech lead found MyGeneration, recommended it, and we used it ever since. Not until last month di