Slashdot Mirror
Justifying the Common Criteria Security Evaluation
lewko writes "Microsoft has just received a Common Criteria certification for Windows 2000 at Evaluation Assurance Level (EAL) 4. Security experts have been saying for years that the the security of the Windows family of products is hopelessly inadequate. Now there is a rigorous government certification confirming this. What does it all mean? This paper suggests that Microsoft spent millions of dollars producing documentation that shows that Windows 2000 meets an inadequate set of requirements, and that you can have reasonably strong confidence that this is the case. Microsoft bashing aside, the process in evaluating a security product is relevant to anyone considering the deployment of technology into their environment." The EROS operating systems he mentions looks interesting - of course, it also looked interesting three years ago.
The Common Criteria security standards deal with the design of operating systems, not the implementation. It has been certified that the security system used in Windows 2000 is "well designed"; but this says nothing about how many bugs there might be in the code.
Tarsnap: Online backups for the truly paranoid
Jonathan S. Shapiro, Ph.D.
Johns Hopkins University Information Security Institute
By now, you may have heard that Microsoft has received a Common Criteria certification for Windows 2000 (with service pack 3) at Evaluation Assurance Level (EAL) 4. Since a bunch of people know that I work on operating system security and on security assurance, I've received lots of notes asking "What does this mean?" On this page I will try to answer the question. For the impatient the answer is:
Security experts have been saying for years that the the security of the Windows family of products is hopelessly inadequate. Now there is a rigorous government certification confirming this.
Since that's a pretty strong statement, bear with me while I try to explain it in plain English.
How a Security Purchase Should Work (In Abstract)At the risk of telling you something you already know, here is how a purchaser ought to proceed when buying a security product:
Assess your needs. Determine what your requirements are.
Decide which product you are most confident will meet those needs.
Buy and deploy it.
Each of these is potentially an involved process, and most customers don't have the expertise to do them effectively. Even if you did, Microsoft (or any other vendor) isn't likely to let you examine their code and design documents in order to evaluate their product.
The purpose of the Common Criteria process is to develop standard packages of commonly found requirements (called Protection Profiles) and have a standard process of independent evaluation by which an expert evaluation team arrives at a level of confidence for some particular software product.
As a customer, this makes your life simpler, because you can compare your needs against existing requirements constructed by experts and then see how well the software you are buying meets those requirements. Security requirements are fairly hard to write down correctly, but if the resulting document is annotated properly they aren't all that hard to understand.
Obviously, if you don't know your needs (requirements) you don't stand much of a chance of getting them met. Likewise, if you don't know what requirements a software product was evaluated against, the evaluation result isn't terribly useful to you in practical terms.
How Common Criteria WorksFrom the customer perspective, a Common Criteria evaluation has two parts:
A standardized requirements specification called a Protection Profile that says what the system is supposed to do. Sometimes there will be more than one of these -- usually a general baseline protection profile and then some others describing additional, specialized requirements.
An evaluation rating. This is basically an investigation by well-trained experts to determine whether the system actually meets the requirements specified in the protection profile(s). The result of the evaluation is an "Evaluation Assurance Level" which can be between 1 and 7. This number expresses the degree of confidence that you can place in the system.
In order to understand the result of an evaluation, you need to know both the evaluation result, which will be a level between EAL1 and EAL7, and the protection profile (the requirements that were tested). Given two systems evaluated against the same protection profile, a higher EAL rating is a better rating provided the requirements meet your needs.
Knowing that a product has met an EAL4 evaluation -- or even an EAL7 evaluation -- tells you absolutely nothing useful. It means that you can have some amount of confidence that the product meets an unknown set of requirements. To give a contrived example, you might need a piece of software that always paints the screen black. I might build a piece of software that paints the screen red with very high reliability, and get it evaluated at EAL4. Obviously my software isn't going to solve your problem.
The Windows 2000 EvaluationMicrosoft sponsored an evaluation of Windows 2000 (with Service Pack 3 and one patch) against the Controlled Access Protection Profile (plus some enhancements) and obtained an EAL4 evaluation rating. This is most accurately written as "CAPP/EAL4".
Problem 1: The Protection ProfileThe Controlled Access Protection Profile (CAPP) standard document can be found at the Common Criteria website.Here is a description of the CAPP requirements taken from the document itself (from page 9):
The CAPP provides for a level of protection which is appropriate for an assumed non-hostile and well-managed user community requiring protection against threats of inadvertent or casual attempts to breach the system security. The profile is not intended to be applicable to circumstances in which protection is required against determined attempts by hostile and well funded attackers to breach system security. The CAPP does not fully address the threats posed by malicious system development or administrative personnel.
Translating that into colloquial English:
Don't hook this to the internet, don't run email, don't install software unless you can 100% trust the developer, and if anybody who works for you turns out to be out to get you you are toast.
In fairness to Microsoft, CAPP is the most complete operating system protection profile that is presently standardized. This may be the best that Microsoft can do, but it is very important for you as a user to understand that These requirements are not good enough to make the system secure. It also needs to be acknowledged that commercial UNIX-based systems like Linux aren't any better (though they are more resistant to penetration).
Note that the "Don't install software" part means that you probably shouldn't install a word processor. On several occasions Microsoft has unintentionally shipped CD's with viruses on them. A CD with a virus qualified as "malicious system development."
Problem 2: The Evaluation Assurance LevelHaving described the requirements problem, I now need to describe the problem of the EAL4 evaluation assurance level that Windows 2000 received.
As I mentioned before, EAL levels run from 1 to 7. EAL1 basically means that the vendor showed up for the meeting. EAL7 means that key parts of the system have been rigorously verified in a mathematical way. EAL4 means that the design documents were reviewed using non-challenging criteria. This is sort of like having an accounting audit where the auditor checks that all of your paperwork is there and your business practice standards are appropriate, but never actually checks that any of your numbers are correct. An EAL4 evaluation is not required to examine the software at all.
An EAL4 rating means that you did a lot of paperwork related to the software process, but says absolutely nothing about the quality of the software itself. There are no quantifiable measurements made of the software, and essentially none of the code is inspected. Buying software with an EAL4 rating is kind of like buying a home without a home inspection, only more risky.
The Bottom Line for Windows 2000In the case of the CAPP protection profile, there actually isn't much point to doing anything better than a low-confidence evaluation, because the requirements set itself is very weak. In effect, you would be saying "My results are inadequate, but the good news is that I've done a lot of work so that I can be really sure that the results are inadequate.
In the case of CAPP, an EAL4 evaluation tells you everything you need to know. It tells you that Microsoft spent millions of dollars producing documentation that shows that Windows 2000 meets an inadequate set of requirements, and that you can have reasonably strong confidence that this is the case.
ConclusionSecurity isn't something that a large group can do well. It is something achieved by small groups of experts. Adding more programmers and more features makes things worse rather than better. Microsoft has been adding features demanded by their customers for a very long time.
It is possible to do much better. EROS, a research operating system that we are working on here in the Systems Research Laboratory at Johns Hopkins University, should eventually achieve an EAL7 evaluation rating, and is expected to provide total defense against viruses and malicious code. It won't be compatible, because the most important security problems in Windows and UNIX are design problems rather than implementation problems. In fact, none of the viable research efforts toward secure operating systems are compatible with existing systems.
It remains to be seen whether EROS or one of the other attempts to build secure operating systems will prevail, but better solutions are coming.
Jonathan Shapiro is an Assistant Professor in the Department of Computer Science of Johns Hopkins University. He has been working on operating system security and assurance since 1991. His past research has yielded both formally verified security properties and dramatically improved performance results in secure operating systems. His current research focuses on tying these results together into a complete, usable system, and on evaluating and testing the correctness and reliability of the resulting system.
Dr. Shapiro is also member of JHUISI, the Hopkins Information Security Institute.
The trouble I find is that I'm able to evaluate the level diligence the IT staff at any given company has taken, I'm able to audit the level of (attempted) compliance to any documented security policy and I'm even able to assess internal security configuration and controls.
Ultimately though, I'm signing off on audit opinions that ALWAYS says and feeling a little sick about it. If we got sued, I could provide documentation proving that I diligently checked security and based on "accepted" business standards the security was implement at a reasonable level. Basically, I could cover my ass.
Is there anyone out there that has an audit program for Win2k that they would feel comfortable using to tell the auditors that they can rely on the numbers? Just curious.
Oh, BTW, the auditors could care less about Common Criteria and even though they're thick as pudding about IT, they're still smart enough o bring in outside people when they need to rely on any computer's numbers.
It was followed by a short lived, but lengthy discussion with regular readers of worldtechtribune (including the editor-in-chief apparently) and some other newsforge readers.
You may or may not find some interesting thoughts, or just more (mis)information.
since it is embedded into the os
IE is embedded into Explorer, NOT the OS (i.e. the kernel). You can easiliy run Windows with a different shell (why?).
I'm not quite sure what to make of this comment
It's MCSE, and I don't think MCSA was a typo.
but there is a microsoft certification called MCSA (like MCSE but harder apparently...).
This seems to me longer than the time for which Windows 98 was allocated, but not for server releases. I heard or read somewhere that the lifecycle had been extended, but I could be mistaken. Either way, this gives it another 2-4 years of usage. I'm not sure whether thats useful or not. Product Lifecycles
Well how much of what is secure? It seems to me that MOST of the security bugs one associates with Microsoft are problems with two programs in particular--IIS and Outlook (Express version only).
Or, y'know, the version of Outlook that was spreading all those nasty worms.... it probably had some holes too.
Vendors hated NSA's old rating process. The standards were tough, NSA did the evaluations themselves, and you only got two tries to pass. After the first evaluation, NSA told you what was wrong. If you failed on the second try, that was it - you flunked. Worst case, NSA listed your product as "Class D - This class is reserved for those systems that have been evaluated but that fail to meet the requirements for a higher evaluation class."
Later, the process became much more "vendor friendly". Evaluations are performed by outside contractors, and vendors can submit their software over and over and over again until it passes. Microsoft used this process to push NT 4 through. It took years. The evaluation process is controlled by the vendor, and there are no public reports of failure.
The "common criteria" are rather weak, down near the bottom of the old NSA criteria. And the evaluation process is almost totally under vendor control, although it does have to be performed by an outside contractor acceptable to the Government.
There's better stuff out there. Currently, the most secure OS certified is the Wang XTS-300. This is certified to level B3 of the old Red Book criteria, which is about four notches above the level Windows 2000 just reached. Various FBI and DoD systems use Wang XTS-300, which is on Wang-built Pentium II and III systems. Wang is gone, but the product has been taken over by Getronics, which keeps a low profile.
Read the data sheet for the XTS-300. It's UNIX-like, but very different inside.
Coming soon, the XTS-400, which runs Linux apps.
These secure systems enforce a "mandatory security" model. Data has a security level, an integrity level, and a list of compartments to which it belongs. Movement downward in security level or upward in integrity level is prohibited, as is movement out of a security compartment or into an integrity compartment. This is very restrictive, but it's the only approach known to have any chance of really working.
they quite obviously had to support everything that were documenting with proof, they probably factored that in.
Siggy Say, Siggy Do
The sort-of-precursor to the CC, the DOD-5200.28-STD (Orange Book) specified exactly who needed to be in the testing team. For "Division C" (Windows NT 4.0 is rated C2):
For higher security classifications, the qualifications of the testing team get higher. For Division A you need at least one individual with a bachelor's degree in Computer Science or the equivalent and at least two individuals with masters' degrees in Computer Science or equivalent.So, Safety Cap's point is well made - the method of testing and the personnel carrying it out is just as important as the technical criteria.
Never email donotemail@WeAreSpammers.com
In essence, like the author stated, many people are substituting education about security issues with Common Criteria certification. However, if the customer doesn't know what they want, or if they don't understand what Common Criteria does and DOES NOT check, then the customer still has no idea what they are getting. And like the author, I sometimes wonder if Common Criteria certification short cuts the basic security background required to write an RFP and replaces it with a check box for an EAL.
In particular, if you work on or sell a security product and want to sell to government or the European Union, it must be Common Criteria certified. What the certification proves, however, is up to the interpretation of the person implementing the product.
What's more concerning than the need to install the security patches is the large number of known and unpatched vulnerabilities, which are still exploitable on most up-to-date Windows desktops.
I think you shouldn't need to reboot more than twice to install those patches, as the hotfixes can be combined using QCHAIN.EXE.
If you're looking for checklists, Microsoft has some available. But if you've been looking, you're probably aware of them. Nevertheless, I used them when securing my network.
The Common Criteria replace the old NIST "Orange Book" specifications.
The CC is a certification standard set up by the NSA, NIST, and some European counterparts. It has an ISO number, too. It can be applied to any computer system (an OS, a browser, a PCI card) as long as you can clearly define the system boundary. The criteria keep alking about the target of evaluation (TOE) instead of calling it an OS, although most commonly you hear about CC being applied to OSes.
When you submit something for CC evaluation, you submit a very specific system with very specific configurations. Anything outside this narrw set of configurations isn't certified. The CC primarily look at design and documentation, so things like buffer overflows don't enter in to the equation. At the highest level (EAL 7), you need all kinds of (mathematical) demonstrations and proofs of sound design (probably mostly involving graph theory). At the lower levels, they require less rigorous proofs and deonstartaions. Basically there are a bunch of feature lists in the criteria and you need to convince the certifier that you have the required features. Good admin/user documentation and configuration tools are a big part of the CC. If it's secure, but not well documented how to keep it secure, you can forget it.
It's expensive to submit a system for certification, so even if the SELinux documentation and config tools were up to par, iit'd be unlikely anyone would pony up the cash to get it evaluated. In terms of software features, I think SELinux could cocievably be EAL 4 or quite possibly higher.
Copyright Violation:"theft, piracy"::Anti-Trust Violation:"thermonuclear price terrorism"<-Overly dramatic language.
As soon as you can find me a three-year old Linux distro STILL BEING SOLD AS NEW. Microsoft could easily have patched their master disc and manufactured new Win2k Server CDs at any time during these three years since the initial release but they have not done so. They are still making and selling software that they know is defective without even a token attempt at fixing the most glaring security holes in their product. In my book, this not only borders on criminal negligence, it's a fucking full-scale invasion over said border.
Hrm, thats funny. I have win2kpro cds here that are naked, or have sp1 already integrated, or have sp1 and 2 integrated. I can choose which cd to use, and i usually go for the latest one. This also is the case with win2kserver, the ones we have here have sp1 integrated. So your wrong, buy Win2k (either version) and MS will have done what you are saying they havent, and upgraded the base OS installed.
I can't see how that would be required for C2 (CAPP in the CC). The old B2 (Structured Protection) was the first level that required covert channel analysis. Granted, that's a pretty obvious covert channel, and you might see it as a kind of quasi-legitimate IPC. In that case the B1 (LSPP) level would require it to follow it's normal rules of compartments and levels.
"An object declared as type _Bool is large enough to store the values 0 and 1." -- 6.1.2.5, C99 standard.
That sounds a bit like Palladium to me (only certified code runs).
EROS has nothing to do with certified code. The idea of EROS is that you can be handed a program and run it without any fear that it will hurt you, because you will be able to restrict in what ways it interacts with the rest of the system in very specific ways. For example, some game you got as an attachment should be able to open up a window and write things to it, and that's about it. It should not be able to, say, read your mail folders, or open up network connections, and with EROS you can be pretty sure this is the case.