Slashdot Mirror
Web App Scanners Miss Half of Vulnerabilities
seek3r sends news of a recent test of six web application security scanning products, in which the scanners missed an average of 49% of the vulnerabilities known to be on the test sites. Here is a PDF of the report. The irony is that the test pitted each scanner against the public test files of all the scanners. This reader adds, "Is it any wonder that being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!" "NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating, with Hailstorm having the second best rating of 62%, but only after extensive training by an expert. Appscan had the second best 'Point and Shoot' rating of 55% and the rest averaged 39%."
> Web App Scanners Miss Half of Vulnerabilities
Well this is no surprise to me. Designing/testing secure systems is much more than scanning for vulnerabilities.
Scanning is only one of the tool to use to accomplish the goal.
Everything I write is lies, read between the lines.
Take buffer overflows for example. You can build a generic tool to create buffer overflows by feeding in long messages but there is no generic way to exploit the overflow, because every system arranges its data differently.
BTW there is a typo in the summary pitted eah scanner
http://michaelsmith.id.au
No vulnerability scanner will ever detect 100% of the vulnerabilities possible. They're still very useful, however, because no website is going to have 100% of all the vulnerabilities possible.
Think of it another way. If your website has only 1 vulnerability and the scanner detects it, then it's 100% effective.
If your website has only 1 vulnerability and no scanner detects, score 1 for the bad guys. The cat and mouse game continues.
I noticed Whitehat Security Declined to participate. I wonder why that is? We just purchased there service, I like there concept, especially as they sold it, we haven't gotten into full use of the product yet, but I can tell you some of the execution of there service could be improved. There seems to be a little bit of a disconnect between the sales force and the operations team. I would have been very interested to see how they fare in a test like this.
"Is it any wonder that being PCI compliant is meaningless from a security point of view?"
Where's that quote from? I can't find it on either the page or in the PDF...
"Want some rye? 'Course you do!" - Return to Zork
The key message here is that simply testing your web site with a vulnerability scanner doesn't make it secure. Well, duh.
PCI is still important because before the guidelines, most people weren't scanning their web sites at all. Even when they knew how - they couldn't convince management it was worth the trouble, time, dollars, and so on. And without scans, the number of discovered web vulnerabilities approaches 0%.
PCI isn't just about scanning your website, either. There's hundreds of things you have to do to secure everything from the physical layer up to the application layer. And having PCI be required to process credit cards makes everything much more secure. I'm talking about small businesses so cheap they don't want to put LOCKS on the doors between the outside world and the servers holding your plain-text, unencrypted credit card numbers, and who don't have the expertise to set up a web camera on their own building.
You might not like PCI, it might be inconvenient, but it's necessary to protect the general public.
Disclaimer: I am an information security professional.
A vendor will sell you, or often give you a free trial of, their vulnerability scanning tool. They will then turn right around and sell you a tool that is supposed to fix those problems. Does anyone else see a problem with that? One reason I prefer the FOSS tools going back to Nmap and SATAN is that they do what real intruders try to do, not what some marketing department wants them to do as a way to scare you into buying stuff.
At least when it comes to security. By the time any standard is published and a test suit is assembled, the whole threat scenario has changed by 180 degrees. We're dealing here with an industry that has a half-life period of its knowledge of about 3 months. Not the usual 2-3 years anywhere else in IT.
Don't be compliant. Either get up to speed with curent security problems or hire someone who does. Standards are worth jack, at least from a security point of view (they're still quite valuable to get contracts from companies who have been BSed into believing in the standards themselves).
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
This guy is trying to hype his own findings a bit too much. Removing half of the vulnerabilities is actually really good! If you happen to remove the vulnerability that some mass-defacement takes advantage of, you really did ad a lot of value by using the imperfect scanning tool.
One of the most common and least helpful fallacies about security is that something is either secure or it is not. Nothing is 100% secure. Removing half of the vulnerabilities is a huge improvement over removing none.
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
The web was clearly never designed to do even a fraction of what it is expected to do today. Now, neither were computers. But at least when it comes to hardware, we're willing to throw everything away and start from scratch. We don't seem able to do that with the web.
Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder that it's so damn easy to write insecure web apps.
Furthermore, it doesn't help that the languages and frameworks commonly used to develop web apps are full of holes themselves. PHP is a very good example of this. Even in the hands of a talented and very experienced developer, it's damn near impossible to develop a site that isn't flawed in some obvious way.
We need to throw it all away. Companies like IBM, Sun, SGI and HP used to routinely do this with their computer hardware. We now need to extend that practice to our software systems. We need to start again. But will we? Probably not, and that's quite unfortunate.
Scanners exist because people want scanners, and so people can sell a product labelled "security scanner". And get a feel-good (false) sense that everything is secure when the scanner reports no issues.
This idea started with the general idea of vulnerability scanner, tools designed to scan hosts for open ports, check software versions, and try exploits against known issues.
The problem with all of them is they can only detect anticipated vulnerabilities.
Unknown vulnerabilities are not properly detected by scanner, because they cannot be anticipated by software.
Much like Antivirus, they need pattern updates and a re-scan when new issues are discovered. Sometimes they don't get updated at all -- sometimes new vulnerabilities are discovered, but a test doesn't get created for the scanner.
Sometimes hackers become aware of security vulnerabilities that the maker of the scanner doesn't become aware of.
Sometimes the hacker can analyze the app you are running (which is industry-specific, not common), and tailor an attack against you, that the scanner vendor could never anticipate.
So are scanners worth something? Sure. But usually not nearly as much as the software vendor bills for them -- they are more fallible than even virus scanners (at least viruses, and malware are finite in number, even if a very large number --- there are more potential security vulnerabilities than one could possibly imagine).
If these scanners report only half the vulnerabilities, they just need to double the reported number. Simple fix, really.
Don't be compliant.
How stupid is this? PCI is a set of minimum requirements. It is all stuff that any competent admin would have done even without the standard. If you are as cool, as you apparently think you are, you will be compliant with zero work.
Did everyone miss the statement they made
they were testing against NTO's own website.
omfg, every other scanner performed poorly against a specially-constructed site
that was put together by the "winner" in the results!
wow.. that's amazing that their own product performed best. who woulda thunk it!
and later in the news: water is wet!! /s
I read TFA because summary does not make sense only to find out that TFA does not make sense.
My favorite from a past employer - one of these PCI scanning companies asked us to take down our iptables rules for a set time period while they scanned us. That's right, they wanted us to be less secure while they checked how secure we were.
We were eventually able to get an ip range from them, but not until we fought them a bit. They *would not* do the scan unless we took down our firewall. I wanted to just REJECT everything but 80 and 443 and not tell them, but the higher-ups told me to play along.
Anyway - the whole idea felt really ... wrong. And they didn't point out anything useful, either.
... Turing-reducible to the halting problem? That is, the conclusion that they miss half the vulnerabilities should be obvious.
Don't forget these results supposed to be 100% because their own test application has been scanned. It means an actual results will be much lower against a real application.
Ok, maybe I should have written "don't try to be compliant, try to be secure". I thought I'd get a reply like that one right the moment I hit submit...
Yeah, yeah, I'll use preview more now, promised.
What I wanted to express is that managers usually don't care for security. The ONLY reason my last company finally implemented bare minimum security was that they needed a certificate to get a fat contract. There was literally no other motivation for them. And, again, they only did the bare minimum of what was entirely necessary to get the cert passed. When (not if) that security theater gets finally debunked, we'll have another press conference and the CISO gets fired, a promise is made to improve security, a few k USD are blown on studies and as soon as the press has another victim, everything is pigeonholed again.
I decided to go rather than get fired.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating.
Doesn't sound exactly like ALL of them missed 50% of vulnerabilities. If I hadn't continued reading, I'd have thought that all scanners are useless.
> being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!"
Print this out and stick it on the wall, for the next time your PHB starts waffling on about compliance .. :)
> one of these PCI scanning companies asked us to take down our iptables rules for a set time period while they scanned us
Can you gave examples of companies that scan companies in the manner you describe. My understanding is that to achieve PCI compliance, you fill in a bunch of forms. I mean Heartland Payment Systems were PCI compliance, and look what happened to them.
"It's not as dumb as you may think. Security is based on a layered approach. If your firewall was down for some reason then the next layer of "security" would be your web app security .."
..
The Six Dumbest Ideas in Computer Security
Let me introduce you to the six dumbest ideas in computer security. What are they? They're the anti-good ideas. They're the braindamage that makes your $100,000 ASIC-based turbo-stateful packet-mulching firewall transparent to hackers. Where do anti-good ideas come from? They come from misguided attempts to do the impossible - which is another way of saying "trying to ignore reality." Frequently those misguided attempts are sincere efforts by well-meaning people or companies who just don't fully understand the situation
I'm going to take issue with this and say the problem is with the internet itself, RAD applications, businesses, and self-taught coders. Allow me to explain.
Half of the .NET code I write is copy/pasted from some other source, because the entire CLR is too complicated for a single person to understand. If I want to do a lookup table, there are a dozen ways to accomplish it, just using the objects provided by the runtime. I don't care how fast it is, unless it's called every page view, so I just google "C# lookup" and get piles of examples. Copy/paste, I'm done. Doesn't matter if it's from MSDN or a Microsoft blog or a random coder blog or wherever else, the code looks good and it works. I have no idea if the example failed to initialize some critical component.
My employer doesn't want to pay me to read, I am supposed to be providing output they can sell to clients/customers. So I don't get a lot of time set aside for training. The way I learned .NET was our tech lead opened up a team meeting and said "I think .NET is the way of the future, is there anyone opposed to going this way?" And the only real objection was it will take longer to produce the next version of our deliverables. Management was fine with that, so we took the leap.
We didn't sit down in a classroom and learn how things are supposed to be done. We didn't get a copy of something like Petzold's Windows bible, or Prosise MFC bible where it goes into depth about what you're doing and what things mean when the IDE puts junk in places for you. Visual Studio 2003 and above make it very easy for you to have no idea what you're doing, and still accomplish something. A quick google search can fill in all of the gaps so you have something functional.
The same with 'Learn X in 24 hours' or 'X for dummies', lots of code samples exclude error checking/handling. Oh yes, MSDN is full of these examples. Sometimes they suggest "error handling has been omitted for clarity", while sometimes it's just assumed. Other times the author has no idea they should be handling errors because it works for them.
So you have piles of coders learning on-the-fly, either because they can't afford the big book or because they have deadlines to meet. Copy/paste something without taking the time to fully understand what's happening, and you get potential problems. In short, easy access to code snippets makes you think you're able to do lots of cool stuff in a new language. Unless you take the time to understand everything you're running, every line of code, you're going to have problems at some point.
Why do you think people still make mistakes like putting form variables directly into SQL? The code snippets are out there, either in the corporate source control or on random blogs. Copy, paste, pwned.
An example, for those of you who wish to tl;dr me you can stop now.
I used MyGeneration templates to come up with database calls for our SQL database, which used Data Access Blocks or some kind of MS best practice to write functions which called stored procedures, so you could essentialy call stored procs exactly like any other function. It generates a call for every stored proc in the database, so you can make fundamental changes to the data structure, re-generate the data access library in a few seconds, and then fix the few calls where the parameters changed.
Very handy, except that the 'execute non-query' template had a bug in it, where the data connection never closed. We never had any problems with this app in production for 3 years. Suddenly in testing, we got a pooled connection exceeded timeout. Turns out the bug only shows up when the call happens most page views, when logging user visits in this case. Other non-query calls happened infrequently enough that they never exceeded the 100 connection default limit, live and in production for 3 years.
Our tech lead found MyGeneration, recommended it, and we used it ever since. Not until last month di
You flat out wouldn't get a job where I work, and if you did, you would be let go within the day. It is possible to hire intelligent, knowledgeable .net coders who know how the use the framework properly. Using statements are *basic* and entirely required knowledge for any self respecting .net developer. You are plain and simply the type of coder that PHP so greatly suffers from on a whole (and from which it derives its poor reputation).
I wouldn't get a job there because they flat out wouldn't pay what I'm worth. You want someone who understands .NET completely, you're paying a lot more than for someone who "can code" in .NET. It is possible, but expensive, and that's my point.
The expense of learning has to be paid by one of:
The employee, raising the minimum salary the employee will accept
The new employer, raising expenses in the form of providing time set aside for training
The previous employer, by allowing the employee to fit in training, making the employee more valuable.
Someone has to pay for training. Ultimately, it will be the hiring company in the form of training materials, time, or pay for experience. Typical .NET or PHP coding type jobs are paying in the range of $25-$60k/year, which is not enough to make me learn the entire .NET CLR on my own time. Catch-22, what do you do? Economic downturn and let the expensive people go, and get the lower paid people make up for it.
"Everything will now be done in .NET, but we're not allowing you training time. Here's the schedule we promised to the client, so fit your training in around that." That's how it works, unless you're into spending money on people.