Slashdot Mirror
Questioning Security Certifications
prostoalex writes "BusinessWeek questions the validity of security certifications in the modern world. They take a look at Federal Information Processing Standard and the certification process. Apparently 'the testing companies make money by certifying products, not catching problems' thus implying that the seal of approval might not mean a whole lot."
There are plenty of industries where following the money would make you think twice about the motivations of the seller of services. How about financial planners/brokers who make more cash by churning your investments? How about the auto mechanic who makes more cash by replacing your radiator when all it needed was an external cleaning (true personal experience here)? How about [fill in your own example here...everyone has one]?
At this point, if you're not always questioning whether a service provider is taking you for a ride, then you're being taken for a ride.
How can we afford to ever sleep
So sound again
--ebtg
Well, all things considered, most products that get security certifications probally need more QA then they actually recieve. But most security flaws usually dont get caught until later dates due to higher user base, thus making them more of a target to attackers (with the exception of buffer overflows, which shouldnt be that hard to do a search for functions that do not do range checking on input). Even with a good QA team, its more likely that a group of 3000 hackers will find security flaws in a production system better than a group of 300 QA testers, due to poor administration, default settings, whatever
Automated software is a good baseline approach, but it falls far short of cunning humans hammering away at systems.
Then why is once an hour does my apache webserver have clients trying to access dll's in the log files? I am sure the IIS admins may not agree with that statement.
-Pete
Soccer Goal Plans
--Automated software is a good baseline approach, but it falls far short of cunning humans hammering away at systems.
:
:
Automated software cuts costs. That's why they use it. Human security testers are expensive, even though IMHO it might be a good way for the most talented script kiddies to make a buck during summer...
--The testing companies make money by certifying products, not catching problems.
Of course they do, they're _certification_ companies, not tech support for security problems. Their job is not to catch problems in your software for you. It is to tell if a product is "secure" or not, according to tests. Which bring us to the point
1) You can't predict the future. Tests run today can't reproduce new problems that will be discovered next year. So this "security certification" is short-termed at least.
2) There is a bias, both in the test suite used and the conception they have of "security". They're human beings too, and to them "good enough" can mean a whole less (or more) than to you.
So what is the problem ? The problem is that apps that pass their tests is instantly classified as "secure". So we have to
- Expand the concept of "security" to give it a little more subjectify ("secure", according to company X, not just "secure, period).
- Use peer-to-peer review, which has proven good at detecting security flaws, and is quite inexpensive for free software projects.
Karma cannot be described by words alone.
Having been through a FIPS requirements meeting I generally agree with Schneir and Kocher: it can easily become a marketing tool if not taken seriously. While FIPS requires, say, certain crypto algorithms (DES, DH, DSA, etc) the physical boundary around the crypto hardware is pretty vague for level 1. Plus, as they mention in the article, you don't really know what method they use to test your product. Is it a monkey with a computer, a script, a Ph.D. mathematician, etc.
Sure if you have access to the source and the time/money/skill to do your own review/testing, you can do a lot better then FIPS. But what if you just want to buy a $5000 box? It would seem to me that at least you have _some_ assurance; it might not be perfect, but it is better then none.
Really, the biggest problem with FIPS is the boundries it is drawn at. Typically only some (crypto) modules are certified; there may be gaping holes in other modules. So your crypto might be bullet proof, but someone may still be able to hack the box and sniff off any data they want
This removes the conflict of interest, and in fact reverses it: the certifying authority *wants* to find problems so they can bill more hours, and the developers but their butts to keep the cost of the certification down.
Jeff
To be useful for testing new crypto software, people need to be able to analyze source and algorithms and develop new attacks. Script kiddies, by definition, only carry out attacks built by others -- something an automated system can do just fine. Finding new vulnerabilities rather than working from a cookbook -- that doesn't take a script kiddie, it takes an expert (or five).
That's why analyzing crypto software is so friggin' expensive -- to do it right takes someone who knows a great deal about not only programming and info security but mathematics as well, and who has actual experience in the field. There are only so many people who can do it right (and I'm most certainly not among them); trying to get the job done properly using the average software engineer with 5 or 10 years of general (non-security-specific) experience won't work, much less a script kiddie of any variety.
This just in -- many individuals who possess MSCE certificates are in fact not performing on a par with senior software engineers.
A certification that doesn't mean anything? I am shocked, really shocked?!?! A goverment standard that is useless? Again I am shocked!?!?
And next you'll tell me that the "Nintendo Seal of Quality" was just a way to discriminate against third party game cartridges.
[PowerPoint] is a tool for capitalist presentation
talented script kiddies
Whoa, there's a phrase you don't see too often.
Wouldn't they be talented hackers/crackers, if they actually know their stuff?
The problem is that these certifications are often the only measure that PHB's use to decide if a product is 'safe' or not, and in this case, the certification is meaningless.
I can't count the number of times that a consultant has quizzed me about firewalls - when they're pushing a certain product (usually because they get a kickback) "This is better because it's certified!"
The problem is that (on off-the-shelf products) the certification only applies to the default configuration - and if you change it (which is pretty much every time - each site has different needs) the device needs to be re-certified... The consultants never mention that part to the client.
The best way to know if a site is secure is to have an independant security audit done by someone qualified (and I don't mean a 'general auditor' - a company that specializes in perimiter security.)
Now, before we all laugh and say "doesn't it show that the certifications are stupid?" consider this.. maybe the certification system does work, and all those other certified products are equally flaky. I've got a list of some TCSEC-certified systems here and frankly it's a pretty unappealing set of OSes. If there were as many Unicos systems (rated B1) out there as there were Windows, I betcha they'd find holes in it soon enough. The fundamental problem with any popular OS is that there will be thousands of hackers and wannabees probing away at it. I don't think there are many people reverse engineering CA-ACF2 MVS in their bedrooms.
I think the motto should be: "Security Through Obscurity" - perhaps all those horrid proprietry OSes did have a point after all.
Never email donotemail@WeAreSpammers.com
I agree that a certification isn't worth much if you don't know what it means. I think that's a big advantage of the Common Criteria (CC). While it's a lot to digest, there are guides out there to help you through it.
I work for the Canadian Common Criteria Scheme and it's my job to ensure that the Canadian labs follow the CC correctly and consistently in their evaluations. I found the article invaluable and disturbing (especially the Bruce Schneier quote), since we're obviously looking for ways to promote the CC, and the article highlights the concerns we need to address.
The same applies to those practices. In and of themselves, they do not guarantee that no incident will take place. But they'll hopefully minimize the impact and frequency of those incidents. The fact that the NSA or some other entity may be able to get past your security doesn't invalidate that security entirely; depending on the environment, it may be good enough.
Information security is really all about risk management. At the end of the day, are we managing our security to the point where the risk is less than the value of the information itself? Balance business need (or whatever needs you have, if you're not a business) against the cost of extra measures. When additional measures are too expensive for the value of what you're protecting, you're secure -- at least secure enough, anyway. If everyone followed security best practices, we'd have a lot less problems than we do.
"You can never have too many elephants on your team."
should do the certification testing. I would buy any product that has the Cowboyneal(tm) seal of approval.
From this article I get the impression that any Tom, Dick, or Harry can go out, 'perform testing' and give away FIPS certs for money.
They really should have looked into the certification process for the testing companies. I doubt that Uncle Sam lets them in without some sort of compliance standards. I guess these standards will just have to include mandatory testing by engineers instead of software.
If not, then the only thing that software developers will have to spend time guarding against are the specific areas of vulnerability that the testing houses look for.
That kind of defeats the purpose, now, doesn't it?
-S
We Apprentice Developers and Designers
At http://www.oracle.com/ip/deploy/database/oracle9i/ you will see:
Oracle now holds 15 security evaluations. DB2 has none. SQL Server has only one.
If it was easy to "buy" these certifications, I'm sure that Microsoft SQL server would have more than just one by now. (Granted, Oracle also has a bit of cash to throw around too).
Slashdot: come for the pedantry, stay for the condescension.
The purpose is to filter out most of the trolls, not to correct spelling mistakes or redundant stories.
Apparently 'the testing companies make money by certifying products, not catching problems' thus implying that the seal of approval might not mean a whole lot."
Sounds familiar. Oh yeah -- the US Patent Office makes its money off of "user fees", and it collects a bunch more when it issues a patent then when it denies one.
Hmm...
(The real answer may be in just charging for the application (either for certification or for patent, depending on whom we're talking about), and the cert is issued (or not) without extra fee. I say "may" be the answer because that approach might encourage agencies to trivially deny applications to boost re-application fee income.)
-- Alastair
No certification can say a product is secure. A certification can only mean the product was tested and found compliant with standards. Security isn't an all or nothing characteristic. All other things being equal, a certified product is less likely to fail than one that was unable to pass the tests.
"dope will get you through times of no money better than money will get you through times of no dope"
"Yes, this certificate is much more secure than a self-signed (in this case not certified) on the premise that the company or author in question gave us money to say so. Enjoy"
Private Company A pays FIPS Certifier a flate rate to have A's product evaluated. If approved, it gets certified. If denied, Company A has to change things and pay to have re-evaluated. This makes the FIPS Certifier want to fail more people, as they then have to pay to have the product evaluated again and again until they get it right. Not too hard, so why does it not work that way? Am I missing something?
The Right Reverend K. Reid Wightman,
Seriously, there are so many certifications out there now, I'm starting to wonder if it'd be profitable for me to create my own cert. and get one of the testing centers to offer it?
The cheapest I.T. industry cert. I've seen is still the A+, and that is a 2 part exam that sets you back around $180 by the time you take both pieces. Either the testing centers make an absolute killing on these things, or else a certain percentage of the profit goes back to the test's creator.
I'm thinking if you even get a 4% or 5% cut of the profits on each attempted exam, you stand to make more money than you would by actually working in the industry in a job you got, partially by becoming "certified".
Certifications are a leftover from the world where everything the military used was custom-made. Back in that world, things were so rigidly specified ("mil-spec") and known to the end user, it made some sense. From the top to the bottom, people in the military or intelligence communities were involved in the design and keeping a sharp eye on the security. The frame of thought of the designers was "zero-threat," whereby they thought of everything they possily could think of, and security was above nearly all other priorities. In this environment, it's easy to make a list of things to check for, and you can actually work that way.
:)
Now, the emphasis is on COTS products, which have almost all been designed with the commercial (read as, "less security-conscious") market. Concern has not been for security as much as marketability, ease of use, and appeal to the public. The designers often do not keep security at a high priority, if at all until the later stages of development. And the people who really understand the nature of the threat in high-security environments had no input, insight, or awareness whatsoever of the internal workings of the products. The products are so varied and different from each other, a checklist of hard facts to verify (as certifications are) is no longer sufficient to catch all the possible risks.
That said, the other problem is that no other method has yet been developed. It's easy to say, "Just do a vulnerability assessment." How? How do you do that on a constant basis on something the size of a government network? How do you make sure that nothing slips through the cracks? And above all else...how do you keep most of the contractors from getting in the way of the few (or single) contractor who gets the job of checking everyone else's work? At least certifications are neutral, in that they have no capability to be used by any single contractor against all the others.
But that's just my frustration
For your security, this post has been encrypted with ROT-13, twice.
I was involved of getting our software package FIPS 140 certified, which is the major crypto security certification. I think there's some validity to the point that the certification house (which is sort of a gatekeeper to the actual certification) has something of a conflict of interest. We are paying them for the certification, while they are the ones who check the adequacy of our security measures. FIPS is supposed to check on their work, but that was largely a rubber stamp.
Nevertheless the certification house did do a thorough check on us and did recommend a number of changes to our software. We didn't think any of them truly added security, but at least this way it was obvious that the cert company was doing their job.
The big problem is that we got that version of the software certified, taking about eight months and several employees' time. Now a few months later we come out with a new release! We can't get re-certified every time, even though they have a shortcut for recertifications. Keeping up with the short software release cycle would be way too expensive.
So we still have FIPS 140 certification listed as a feature of our product, but if a customer really wants that specific version, we have to sell him old software. As it turns out, no one does. All they really need is to be able to check the box that says we are certified, and then they're perfectly happy to take the latest software. The mere fact that we spent the time, effort and money to be certified is what really counts.
I've taken some hardware crypto products most of the way through a FIPS 140-1 level 3+ cycle, but never completed it because our customers were not willing to pay the extra money (my company had certified other products before that, and has certified some since I've left).
I find it interesting that people are starting to specify AND demand FIPS 140. When we certified our first product, using the predecessor of FIPS 140, we only ever had one customer - the US treasury. Perhaps the chicken-and-egg situation is changing, because just enough companies have FIPS 140 certified products that customers might actually be able to buy one.
In reality, most customers will not use the actual certified product, for a couple of reasons. First, it is too expensive (and takes too long) to recertify the product for every minor version change. Second, the FIPS process only allows certain algorithms (FIPS algorithms naturally) and certain cryptographic formats. If your product wants to support the wide-spread PKCS format (RSA pseudo-standard) instead of the government preferred ANSI formats (in cases where there is a difference), those PKCS commands will have to be disabled in the FIPS version of the product.
Sounds like the same problem with the US Patents Office.
They should pay them on the number of patents thrown out.
When shit hits the fan get some of these https://youtu.be/pY-GncsZ-UE
From my experience, the FIPS 140 certificate does a good job of ensuring that products live-up to their formal design specifications. The obvious question is how good were the design specifications? This is where things get interesting. To over generalize, I think FIPS 140 does a good on tamper-resistant (and respondent) hardware design, and a poor job on logical security.
A lot of the FIPS philosophy came out of the military, and the testing labs impressed me with the breadth of their physical attacks. On the other hand, the military usually has very simple logical security requirements for a crypto-box. It should be inert until authorized users properly activate it, and at that point it can perform sensitive actions. Commercial cryptography designs by contrast, usually has a set of functions that needs to be generally available. They also have a much smaller set of functions that need authorized users to control.
When we put our product through the immediate predecessor to FIPS 140-1 certification, we were the first commercial product and ended-up breaking a fair-amount of new ground (somewhat painfully as you might imagine). What we had to show was that the cryptographic commands that were available to non-privileged users were safe - because of the logical security design. Even early FIPS 140-1 processes did not really deal with these "always-on" functions very well.
Although it improved, especially with the 140-2 modifications, logical security is still the real weak point. Michael Bond's well publicized attacks on the FIP 140-1 level 4 certified IBM 4758 security module were all aimed at the "logical security" level. My favorite example of insecure by design is the PKCS #11 security module when it is used for server security.
The Cryptoki (PKCS #11) interface was designed for security tokens, and basically works a lot like the military devices. The token (smartcard, whatever) would be plugged into the client device, where it would remain inert until activated by the user password. Actually a pretty good design when used this way.
The problem is when the same design is used for a server, which is unfortunately common since several PKI vendors standardized on using PKCS #11 security modules. PKCS #11 authorization levels are all messed-up for server use. There is no concept of "always-on" commands, or multiple levels of authorization. That means that any entity (server application) that wants to access the security-subsystem must be an authorized user.
The result is that the clear password that enables the PKCS# 11 modules has to be put into the server application. Because of that clear password an attacker no longer has to break into the PKCS #11 box or steal/forge authorized user's identities. They can gain authorized user privileges merely by monitoring the communication lines between the application and box, or by analyzing the object code of the application!
You will find a number of FIPS 140 certified PKCS #11 modules, which is actually no surprise given how well PKCS #11 matches the military origins of the FIPS 140. This is a classic example of a certified subsystem that is quite secure for some uses (human insertion of a token and entry of password), but it quite insecure for others (server applications storing and using clear passwords). All the FIPS certification does in the case of PKCS #11 is tell you that the vendor has followed their design, and not if it will provide logical security in your system!
What tests and process do they do? Is this always the same? How do they learn from their mistakes? Is the process upgraded and reviewed regularly.
:-)
Not used to working with the government I see
Actually, I think the FIPS 140 process is actually a very good example of those concepts done right. Review the FIPS site.
The answer to your question about tests will be answered thoroughly, perhaps you will want to start with the derived test requirements section.
Product certicications are usually based on high level process requirements - not details/features in products. This means that while process may have been followed, all problems may not have been caught. The certification producer, and the certified entity will market the certification to people, but people should really try to be aware of what it takes to be certified and to understand the field before interpreting the meaning of a cert. Usually certifications miss very important details in an attempt to be a broad catch all and it's difficult to understand what is missed without having been there. At the very least, people should be seeking commentary from people who HAVE been there but no longer have a vested interest (to taint an opinion)
:) Thanks.
Also, unless you have achieved a particular certification, I don't think that you're qualified to comment as to its real meaning. How do you have any clue as to what it means unless you know what you've learned from it, and how you've grown in order to achieve it? Even then, people tend to lie to themselves.
Certificiations are not meaningless. But they certainly don't mean competence. Compentence is NOT knowledge. For example... I recently worked with a man, who was CISSP certified who had no clue. Someone posted a usenet article asking what a firewall deny, outgoing, to port 4000 was probably caused by (complete with destination IP, etc)... he thought that it was a trojan (this was "here's a question, go think about it for a while" type question, not a "GIVE ME AN ANSWER NOW!" question). Now, if you don't recognise port 4000, you could always reverse DNS the IP and find that the target host was icq.mirabilis.com. This is just one example - so don't just say "he was lazy and didn't invest the time in figuring it out. The man also claimed to have written White Rabbit, played with Jefferson Airplane (on stage), invented robots that could climb stairs and learn rooms by name and that he flys to Greece in the summer in a bomber (he says that its his uncle's and that it costs ~2k in gas). Obviously, he was fired, but he had a CISSP (fairly big security compentence certification).
Now, if this post sounds non-sensical, please, understand that I am currently a bit drunk, and may not be making a lot of sense.