As rightly said, sofware accreditation has been around for a while.
Orange book got there first. For each level the test were mandatory. There was no proof that the items being tested linked together to protect something. The Europeans got in with a number of methods, consolidating on ITSEC where you had to state what the security objective was and how you met it (the claim, or Target of Evaluation) and they also tested strength of mechanism. The Canadians were next and published a method. The move to Common Criteria is an attempt to make an evaluation/accreditation in one country valid in others (handy for government and military procurement).
The scales are not linear. The testing methods aren't either. E6 (EAL7) requires formal mathematical proof of the claim being met by the system, whilst E1 (EAL2) needs documentation and flowcharts. Unfortunately that just makes it harder to understand for the non-security propellor head.
Bottom line is that there aren't many properly established ways of doing this work. Only governments and the military are prepared to spend the kind of money needed to decide how to do this, who can do it, what are the testing methods to give realiable results from one evaluation to another, and how to deal with things that may be matter of opinion (just how strong is your algorithm?, does the way you are protecting the Trusted Computing Base really work) and still be reliable.
Industry has, to a large extent, avoided any kind of external review of the quality of their work (which is what evaluation comes down to). SHould we be surprised? Maybe not. IN the first two years of operation the UK ITSEC board stated that they had never had a product in for evaluation that was not found to have at least two major flaws present on initial review.
Problem is that going forwards we are betting our anatomies increasingly upon all this IT stuff working, when, from a security standpoint, the foundations are shaky at best. (No matter how well the manufacturer does, the customer has the ability to defeat security - see your local firewall.) There have been frequent calls from the security community to improve the situation, but you have to bear in mind that until someone loses their skin there is no motivating force.
In the meantime, military evaluations will continue (Orange Book, ITSEC, Common Criteria), but since industry never takes any notice of them when doing its own purechasing they will continue to be a specialist backwater. Commercial evaluations of crypto algorithm output are useful but don't tell if your keys are protected, which might be rather more useful!
Keep on trying. If people don't complain that product is not evaluated it never will be. Once you have them in the circuit you can improve the quality, but if they never get in then there's not a lot to achieve.
Slashdot Mirror
As rightly said, sofware accreditation has been around for a while. Orange book got there first. For each level the test were mandatory. There was no proof that the items being tested linked together to protect something. The Europeans got in with a number of methods, consolidating on ITSEC where you had to state what the security objective was and how you met it (the claim, or Target of Evaluation) and they also tested strength of mechanism. The Canadians were next and published a method. The move to Common Criteria is an attempt to make an evaluation/accreditation in one country valid in others (handy for government and military procurement). The scales are not linear. The testing methods aren't either. E6 (EAL7) requires formal mathematical proof of the claim being met by the system, whilst E1 (EAL2) needs documentation and flowcharts. Unfortunately that just makes it harder to understand for the non-security propellor head. Bottom line is that there aren't many properly established ways of doing this work. Only governments and the military are prepared to spend the kind of money needed to decide how to do this, who can do it, what are the testing methods to give realiable results from one evaluation to another, and how to deal with things that may be matter of opinion (just how strong is your algorithm?, does the way you are protecting the Trusted Computing Base really work) and still be reliable. Industry has, to a large extent, avoided any kind of external review of the quality of their work (which is what evaluation comes down to). SHould we be surprised? Maybe not. IN the first two years of operation the UK ITSEC board stated that they had never had a product in for evaluation that was not found to have at least two major flaws present on initial review. Problem is that going forwards we are betting our anatomies increasingly upon all this IT stuff working, when, from a security standpoint, the foundations are shaky at best. (No matter how well the manufacturer does, the customer has the ability to defeat security - see your local firewall.) There have been frequent calls from the security community to improve the situation, but you have to bear in mind that until someone loses their skin there is no motivating force. In the meantime, military evaluations will continue (Orange Book, ITSEC, Common Criteria), but since industry never takes any notice of them when doing its own purechasing they will continue to be a specialist backwater. Commercial evaluations of crypto algorithm output are useful but don't tell if your keys are protected, which might be rather more useful! Keep on trying. If people don't complain that product is not evaluated it never will be. Once you have them in the circuit you can improve the quality, but if they never get in then there's not a lot to achieve.