Questioning Security Certifications

prostoalex writes "BusinessWeek questions the validity of security certifications in the modern world. They take a look at Federal Information Processing Standard and the certification process. Apparently 'the testing companies make money by certifying products, not catching problems' thus implying that the seal of approval might not mean a whole lot."

Comments

Not Uncommon

[theduck]

There are plenty of industries where following the money would make you think twice about the motivations of the seller of services. How about financial planners/brokers who make more cash by churning your investments? How about the auto mechanic who makes more cash by replacing your radiator when all it needed was an external cleaning (true personal experience here)? How about [fill in your own example here...everyone has one]?

At this point, if you're not always questioning whether a service provider is taking you for a ride, then you're being taken for a ride.

Haste and Waste

[j_kenpo]

Well, all things considered, most products that get security certifications probally need more QA then they actually recieve. But most security flaws usually dont get caught until later dates due to higher user base, thus making them more of a target to attackers (with the exception of buffer overflows, which shouldnt be that hard to do a search for functions that do not do range checking on input). Even with a good QA team, its more likely that a group of 3000 hackers will find security flaws in a production system better than a group of 300 QA testers, due to poor administration, default settings, whatever

Peer review

[koh]

--Automated software is a good baseline approach, but it falls far short of cunning humans hammering away at systems.

Automated software cuts costs. That's why they use it. Human security testers are expensive, even though IMHO it might be a good way for the most talented script kiddies to make a buck during summer...

--The testing companies make money by certifying products, not catching problems.

Of course they do, they're _certification_ companies, not tech support for security problems. Their job is not to catch problems in your software for you. It is to tell if a product is "secure" or not, according to tests. Which bring us to the point :

1) You can't predict the future. Tests run today can't reproduce new problems that will be discovered next year. So this "security certification" is short-termed at least.

2) There is a bias, both in the test suite used and the conception they have of "security". They're human beings too, and to them "good enough" can mean a whole less (or more) than to you.

So what is the problem ? The problem is that apps that pass their tests is instantly classified as "secure". So we have to :

- Expand the concept of "security" to give it a little more subjectify ("secure", according to company X, not just "secure, period).

- Use peer-to-peer review, which has proven good at detecting security flaws, and is quite inexpensive for free software projects.

fips

[ciscoeng]

Having been through a FIPS requirements meeting I generally agree with Schneir and Kocher: it can easily become a marketing tool if not taken seriously. While FIPS requires, say, certain crypto algorithms (DES, DH, DSA, etc) the physical boundary around the crypto hardware is pretty vague for level 1. Plus, as they mention in the article, you don't really know what method they use to test your product. Is it a monkey with a computer, a script, a Ph.D. mathematician, etc.

Charge em!

[jspayne]

It isn't clear from the article how this company does it, but in my experience with safety certification, you get charged the hourly rate for the certification process regardless if you pass or fail. Just like having your car inspected...

This removes the conflict of interest, and in fact reverses it: the certifying authority *wants* to find problems so they can bill more hours, and the developers but their butts to keep the cost of the certification down.

Jeff

It can be a joke

[Anonymous Coward]

The problem is that these certifications are often the only measure that PHB's use to decide if a product is 'safe' or not, and in this case, the certification is meaningless.

I can't count the number of times that a consultant has quizzed me about firewalls - when they're pushing a certain product (usually because they get a kickback) "This is better because it's certified!"

The problem is that (on off-the-shelf products) the certification only applies to the default configuration - and if you change it (which is pretty much every time - each site has different needs) the device needs to be re-certified... The consultants never mention that part to the client.

The best way to know if a site is secure is to have an independant security audit done by someone qualified (and I don't mean a 'general auditor' - a company that specializes in perimiter security.)

There is no security panacea

[El Volio]

If you think there is, you're fooling yourself. That said, as long as that axiom is kept in mind, something is better than nothing. FIPS (or any other certification) may not be a guarantee, but it should be a good indicator that due diligence has been performed and the software meets widely-accepted best practices.

The same applies to those practices. In and of themselves, they do not guarantee that no incident will take place. But they'll hopefully minimize the impact and frequency of those incidents. The fact that the NSA or some other entity may be able to get past your security doesn't invalidate that security entirely; depending on the environment, it may be good enough.

Information security is really all about risk management. At the end of the day, are we managing our security to the point where the risk is less than the value of the information itself? Balance business need (or whatever needs you have, if you're not a business) against the cost of extra measures. When additional measures are too expensive for the value of what you're protecting, you're secure -- at least secure enough, anyway. If everyone followed security best practices, we'd have a lot less problems than we do.

Re:NIMDA?

[cduffy]

If the goal is finding new security failures, the attacks by Nimda simply demonstrate the accuracy of the claim. This automated software is having no success in penetrating your system, whereas if it were a skilled and motivated human (rather than an automated system) hammering away at your system, they may well be able to find a new, effective attack. Similarly, once an IIS installation is appropriately patched, Nimda will have no more effect -- but a sufficiently skilled and motivated human might.

The value of a cert is it's absence

[dogfart]

Would you but a firewall product that wasn't ICSA certified? Would you buy a crypto card that wasn't FIPS 140-1 certified? No. Absence of a certification means that either the product has a serious security flaw that the test would find, or that the vendor simply doesn't care. Either of these are reasons to drop the vendor off your list. An MCSE doesn't make someone a capable system administrator by any means, but would you ever hire anyone who didn't have an MSCE to administer MS servers?

No certification can say a product is secure. A certification can only mean the product was tested and found compliant with standards. Security isn't an all or nothing characteristic. All other things being equal, a certified product is less likely to fail than one that was unable to pass the tests.