Slashdot Mirror


The Myth of Open Source Security Revisited v2.0

Dare Obasanjo contributed this followup to an article entitled The Myth of Open Source Security Revisited that appeared on the website kuro5hin. He writes: "The original article tackled the common misconception amongst users of Open Source Software(OSS) that OSS is a panacea when it comes to creating secure software. The article presented anecdotal evidence taken from an article written by John Viega, the original author of GNU Mailman, to illustrate its point. This article follows up the anecdotal evidence presented in the original paper by providing an analysis of similar software applications, their development methodology and the frequency of the discovery of security vulnerabilities." Read on below for his detailed analysis, especially relevant with the currency of security initiatives in the worlds of both open- and closed-source software.

The Myth of Open Source Security Revisited v2.0 The purpose of this article is to expose the fallacy of the belief in the "inherent security" of Open Source software and instead point to a truer means of ensuring the quality of the security of a piece software is high.

Apples, Oranges, Penguins and Daemons

When performing experiments to confirm a hypothesis on the effect of a particular variable on an event or observable occurence, it is common practice to utilize control groups. In an attempt to establish cause and effect in such experiments, one tries to hold all variables that may affect the outcome constant except for the variable that the experiment is interested in. Comparisons of the security of software created by Open Source processes and software produced in a proprietary manner have typically involved several variables besides development methodology.

A number of articles have been written that compare the security of Open Source development to proprietary development by comparing security vulnerabilities in Microsoft products to those in Open Source products. Noted Open Source pundit, Eric Raymond wrote an article on NewsForge where he compares Microsoft Windows and IIS to Linux, BSD and Apache. In the article, Eric Raymond states that Open Source development implies that "security holes will be infrequent, the compromises they cause will be relatively minor, and fixes will be rapidly developed and deployed." However, upon investigation it is disputable that Linux distributions have less frequent or more minor security vulnerabilities when compared to recent versions of Windows. In fact the belief in the inherent security of Open Source software over proprietary software seems to be the product of a single comparison, Apache versus Microsoft IIS.

There are a number of variables involved when one compares the security of software such as Microsoft Windows operating systems to Open Source UNIX-like operating systems including the disparity in their market share, the requirements and dispensations of their user base, and the differences in system design. To better compare the impact of source code licensing on the security of the software, it is wise to reduce the number of variables that will skew the conclusion. To this effect it is best to compare software with similar system design and user base than comparing software applications that are significantly distinct. The following section analyzes the frequency of the discovery of security vulnerabilities in UNIX-like operating systems including HP-UX, FreeBSD, RedHat Linux, OpenBSD, Solaris, Mandrake Linux, AIX and Debian GNU/Linux.

Security Vulnerability Face-Off

Below is a listing of UNIX and UNIX-like operating systems with the number of security vulnerabilities that were discovered in them in 2001 according to the Security Focus Vulnerability Archive. AIX 10 vulnerabilities[6 remote, 3 local, 1 both] Debian GNU/Linux 13 vulnerabilities[1 remote, 12 local] + 1 Linux kernel vulnerability[1 local] FreeBSD 24 vulnerabilities[12 remote, 9 local, 3 both] HP-UX 25 vulnerabilities[12 remote, 12 local, 1 both] Mandrake Linux 17 vulnerabilities[5 remote, 12 local] + 12 Linux kernel vulnerabilities[5 remote, 7 local] OpenBSD 13 vulnerabilities[7 remote, 5 local, 1 both] Red Hat Linux 28 vulnerabilities[5 remote, 22 local, 1 unknown] + 12 Linux kernel vulnerabilities[6 remote, 6 local] Solaris 38 vulnerabilities[14 remote, 22 local, 2 both] From the above listing one can infer that source licensing is not a primary factor in determining how prone to security flaws a software application will be. Specifically proprietary and Open Source UNIX family operating systems are represented on both the high and low ends of the frequency distribution.

Factors that have been known to influence the security and quality of a software application are practices such as code auditing (peer review), security-minded architecture design, strict software development practices that restrict certain dangerous programming constructs (e.g. using the str* or scanf* family of functions in C) and validation & verification of the design and implementation of the software. Also reducing the focus on deadlines and only shipping when the system the system is in a satisfactory state is important.

Both the Debian and OpenBSD projects exhibit many of the aforementioned characteristics which help explain why they are the Open Source UNIX operating systems with the best security record. Debian's track record is particularly impressive when one realizes that the Debian Potato consists of over 55 million lines of code (compared to RedHat's 30,000,000 lines of code).

The Road To Secure Software

Exploitable security vulnerabilities in a software application are typically evidence of bugs in the design or implementation of the application. Thus the process of writing secure software is an extension of the process behind writing robust, high quality software. Over the years a number of methodolgies have been developed to tackle the problem of producing high quality software in a repeatable manner within time and budgetary constraints. The most successful methodologies have typically involved using the following software quality assurance, validation and verification techniques; formal methods, code audits, design reviews, extensive testing and codified best practices.
  1. Formal Methods: One can use formal proofs based on mathematical methods and rigor to verify the correctness of software algorithms. Tools for specifying software using formal techniques exist such as VDM and Z. Z (pronounced 'zed') is a formal specification notation based on set theory and first order predicate logic. VDM stands for "The Vienna Development Method" which consists of a specification language called VDM-SL, rules for data and operation refinement which allow one to establish links between abstract requirements specifications and detailed design specifications down to the level of code, and a proof theory in which rigorous arguments can be conducted about the properties of specified systems and the correctness of design decisions.The previous descriptions were taken from the Z FAQ and the VDM FAQ respectively. A comparison of both specification languages is available in the paper, Understanding the differences between VDM and Z by I.J. Hayes et al.

  2. Code Audits: Reviews of source code by developers other than the author of the code are good ways to catch errors that may have been overlooked by the original developer. Source code audits can vary from informal reviews with little structure to formal code inspections or walkthroughs. Informal reviews typically involve the developer sending the reviewers source code or descriptions of the software for feedback on any bugs or design issues. A walkthrough involves the detailed examination of the source code of the software in question by one or more reviewers. An inspection is a formal process where a detailed examination of the source code is directed by reviewers who act in certain roles. A code inspection is directed by a "moderator", the source code is read by a "reader" and issues are documented by a "scribe".

  3. Testing: The purpose of testing is to find failures. Unfortunately, no known software testing method can discover all possible failures that may occur in a faulty application and metrics to establish such details have not been forthcoming. Thus a correlation between the quality of a software application and the amount of testing it has endured is practically non-existent.

    There are various categories of tests including unit, component, system, integration, regression, black-box, and white-box tests. There is some overlap in the aforementioned mentioned testing categories.

    Unit testing involves testing small pieces of functionality of the application such as methods, functions or subroutines. In unit testing it is usual for other components that the software unit interacts with to be replaced with stubs or dummy methods. Component tests are similar to unit tests with the exception that dummmy and stub methods are replaced with the actual working versions. Integration testing involves testing related components that communicate with each other while system tests involve testing the entire system after it has been built. System testing is necessary even if extensive unit or component testing has occured because it is possible for seperate subroutines to work individually but fail when invoked sequentialy due to side effects or some error in programmer logic. Regression testing involves the process of ensuring that modifications to a software module, component or system have not introduced errors into the software. A lack of sufficient regression testing is one of the reasons why certain software patches break components that worked prior to installation of the patch.

    Black-box testing also called functional testing or specification testing test the behavior of the component or system without requiring knowledge of the internal structure of the software. Black-box testing is typically used to test that software meets its functional requirements. White-box testing also called structural or clear-box testing involves tests that utilize knowledge of the internal structure of the software. White-box testing is useful in ensuring that certain statements in the program are excercised and errors discovered. The existence of code coverage tools aid in discovering what percentages of a system are being excercised by the tests.

    More information on testing can be found at the comp.software.testing FAQ .

  4. Design Reviews: The architecture of a software application can be reviewed in a formal process called a design review. In design reviews the developers, domain experts and users examine that the design of the system meets the requirements and that it contains no significant flaws of omission or commission before implementation occurs.

  5. Codified Best Practices: Some programming languages have libraries or language features that are prone to abuse and are thus prohibited in certain disciplined software projects. Functions like strcpy, gets, and scanf in C are examples of library functions that are poorly designed and allow malicious individuals to use buffer overflows or format string attacks to exploit the security vulnerabilities exposed by using these functions. A number of platforms explicitly disallow gets especially since alternatives exist. Programming guidelines for such as those written by Peter Galvin in a Unix Insider article on designing secure software are used by development teams to reduce the likelihood of security vulnerabilities in software applications.
Projects such as the OpenBSD project that utilize most of the aforementioned techniques in developing software typically have a low incidence of security vulnerabilities.

Issues Preventing Development of Secure Open Source Software

One of the assumptions that is typically made about Open Source software is that the availability of source code translates to "peer review" of the software application. However, the anecdotal experience of a number of Open Source developers including John Viega belies this assumption.

The term "peer review" implies an extensive review of the source code of an application by competent parties. Many Open Source projects do not get peer reviewed for a number of reasons including
  • complexity of code in addition to a lack of documentation makes it difficult for casual users to understand the code enough to give a proper review

  • developers making improvements to the application typically focus only on the parts of the application that will affect the feature to be added instead of the whole system.

  • ignorance of developers to security concerns.

  • complacency in the belief that since the source is available that it is being reviewed by others.

Also the lack of interest in unglamorous tasks like documentation and testing amongst Open Source contributors adversely affects quality of the software. However, all of these issues can and are solved in projects with a disciplined software development process, clearly defined roles for the contributers and a semi-structured leadership hierarchy.

Benefits of Open Source to Security-Conscious Users

Despite the fact that source licensing and source code availability are not indicators of the security of a software application, there is still a significant benefit of Open Source to some users concerned about security. Open Source allows experts to audit their software options before making a choice and also in some cases to make improvements without waiting for fixes from the vendor or source code maintainer.

One should note that there are constraints on the feasibility of users auditing the software based on the complexity and size of the code base. For instance, it is unlikely that a user who wants to make a choice of using Linux as a web server for a personal homepage will scrutinize the TCP/IP stack code.

References
  1. Frankl, Phylis et al. Choosing a Testing Method to Deliver Reliability. Proceedings of the 19th International Conference on Software Engineering, pp. 68--78, ACM Press, May 1997. < http://citeseer.nj.nec.com/frankl97choosing.html >

  2. Hamlet, Dick. Software Quality, Software Process, and Software Testing. 1994. < http://citeseer.nj.nec.com/hamlet94software.html >

  3. Hayes, I.J., C.B. Jones and J.E. Nicholls. Understanding the differences between VDM and Z. Technical Report UMCS-93-8-1, University of Manchester, Computer Science Dept., 1993. < http://citeseer.nj.nec.com/hayes93understanding.ht ml >

  4. Miller, Todd C. and Theo De Raadt. strlcpy and strlcat - consistent, safe, string copy and concatenation. Proceedings of the 1999 USENIX Annual Technical Conference, FREENIX Track, June 1999. < http://www.usenix.org/events/usenix99/full_papers/ millert/millert_html/ >

  5. Viega, John. The Myth of Open Source Security. Earthweb.com. < http://www.earthweb.com/article/0,,10455_626641,00 .html >

  6. Gonzalez-Barona, Jesus M. et al. Counting Potatoes: The Size of Debian 2.2. < http://people.debian.org/~jgb/debian-counting/coun ting-potatoes/ >

  7. Wheeler, David A. More Than A Gigabuck: Estimating GNU/Linux's Size. < http://www.counterpane.com/crypto-gram-0003.html >



Acknowledgements

The following people helped in proofreading this article and/or offering suggestions about content: Jon Beckham, Graham Keith Coleman, Chris Bradfield, and David Dagon. © 2002 Dare Obasanjo

14 of 206 comments (clear)

  1. Open Source still wins by Tonetheman · · Score: 3, Interesting

    It is true that you can have just as many security problems in Open Source, but those security problems are not hidden behind clever marketing. The code is out in the open and can be fixed by anyone who takes the time to do so.

    Microsoft could even have a better track record than some Open Source systems and I think that I would still choose the open source way.

    If you rely on obscurity to be your software security then you will lose every time. In the end it is the freedom to choose and to change in the open that makes a system secure.

    Tony

  2. No, Dare by ekrout · · Score: 3, Interesting

    No, Dare, you're wrong my friend.

    In fact the belief in the inherent security of Open Source software over proprietary software seems to be the product of a single comparison, Apache versus Microsoft IIS.

    No, I'd venture to say that although you are correct in citing IIS' tendency to destroy the Internet every few months when another virus comes out targeted at the Microsoft web server, there are most definitely other pieces of Microsoft proprietary crap that looks pretty lame when compared to its open source or free software counterpart.

    Ever hear of Microsoft Outlook?

    --

    If you celebrate Xmas, befriend me (538
  3. The fix is on the way by bourne · · Score: 5, Interesting

    The author's point is correct - while any Open Source package may have been audited, it isn't neccesarily audited well or at all.

    But flash-back to the recent announcement of the Sardonix Security Portal, which aims to be a central clearinghouse for tracking audits and auditors. The goal is to have a list of 1) what's been audited, 2) who audited it, and 3) what that particular auditor's track record is on other software - were holes found after they said it was clean?

    Obviously this is a new project, and it's founded on the ashes of an earlier effort that didn't get much involvement, but it's a big step in the right direction and it's got DARPA funding. And it probably will do much better jobs with Open Source software than with Closed Source.

  4. README, Changlog, LICENSE, AUDIT... huh? by kindbud · · Score: 5, Interesting

    I use vsftpd, (vs stands for "very secure" and is a goal, not a declaration of its status). Included with the source are a number of explanatory files you are familiar with: README, INSTALL, Changelog, LICENSE. There is another you probably haven't seen: AUDIT. AUDIT lists each source code file and a rating of 1-5 to indicate how much scrutiny it has received from other competent parties. 1 indicates no scrutiny and 5 means many competent programmers have reviewed it. Most of the files are rated 2 or 3.

    I think this is an excellent idea, one that should be expanded upon by other developers.

    Oh, and vsftpd 1.0.1 can be obtained from this ftp site at Oxford. It's written on Linux but I run it on Solaris with just a tweak to a #define.

    --
    Edith Keeler Must Die
  5. What a load of crap. by 7-Vodka · · Score: 2, Interesting
    I'm sorry, but I don't believe anyone can make such sweeping statements such as 'OSS is more/less secure than closed source'.

    It just doesn't jive. Some closed source software is more secure, some OSS is more secure. It depends on the talent, hardwork and organizational skills of those involved in the individual projects.

    Even if one found a methodical way to compare the mean security level of OSS and closed source software, it would be of no use!

    What use is it telling someone Closed source software is in general more secure than OSS when they're only interested in a web server? What they need to know is how secure their potential solutions are.

    Also, knowing which method in general produces more secure code won't influence a development team. They have more important things to worry about, ie. how they intend to profit from their work.

    --

    Liberty.

    1. Re:What a load of crap. by MisterBlister · · Score: 3, Interesting
      I'm sorry, but I don't believe anyone can make such sweeping statements such as 'OSS is more/less secure than closed source'.

      The article didn't say anything even close to that; what it said is that the widely-held belief (particularly among the Slashdot-crowd) that Open Source Software is somehow inherently more secure than closed source software is largely a myth. Try understanding what you read before you call it a 'load of crap'...

      It looks pretty silly when you insult the article and then make a post where all your points are pretty much the same as in the article (ie. OSS and closed source software can both be secure or insecure).

  6. Commercial vs Open Source bugs by Jordy · · Score: 4, Interesting

    There is quite a big difference between security problems in commercial operating systems such as Solaris and security problems in open source operating systems such as Debian.

    The Debian team did not write most of the software that comes with the Debian distribution. Sure they make patches and try to keep things up-to-date, but the software that is in their distribution is included for completeness more than anything else.

    Sun on the other hand did write most of the software that comes with Solaris (or at least obfuscated where it came from.) They are directly responsible for security problems with the software they distribute.

    When a security problem occurs in Apache, surely it's an Apache security problem that just happens to affect everyone who has Apache installed. If they have Apache installed on Windows, one can't claim it as a Microsoft security bug and blame Microsoft for not auditing every peice of code that happens to compile for their OS.

    No one forces the end-administrator to install 99% of the software included with an open source distribution. It is up to the administrator to only install software which they are comfortable with. If the authors of Emacs don't do frequent code audits, don't install it (that's not to say they don't.)

    Now... one thing distributions can do to make the end-administrator's job a bit easier is to include statistics along with the application for things like past security vulnerabilities, time since last vulnerability, last code audit, etc. to help them make better decisions about what to install and not to install.

    Of course, going the route of only including fully audited code in a distribution just doesn't work. If people need inn, they need inn code review or not. Granted they might take a look at the source while they are compiling it, but the chances of them finding a massive security hole with a curory glance is pretty slim.

    That's not to say that distribution vendors are free from blame; especially fully commecial vendors who should at least do some form of audit and mark which packages haven't been audited as 'unsafe,' but come on now... the real blame belongs with the administrator and the developers.

    --
    The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
  7. Open Source has more REPORTED and FIXED issues by theendlessnow · · Score: 2, Interesting

    I dare you to put an AIX box external to the internet! Sheesh. M$ doesn't get it... this guy doesn't get it either. The best way to improve quality is with everyone testing things out in the open. If you try to hush up things, nothing gets fixed (as is the case with most of the commerical *ix platforms) and holes abound. The reason you don't hear too much about the other *ix platforms is because not as many people are interested in hacking them. A piece of software with no bugs is likely a BAD piece of software... not a good one. So report your problems.... report them all.... or be like M$ and many others who try to hide them as best they can and fix them when convenient.

  8. Re:To be fair... by Brian+Knotts · · Score: 2, Interesting
    It also really makes you wonder about those Red Hat custom kernels. I reckon I'll stick with Debian.

    I suppose you could make an argument, though, that Debian sort of cheats. These stats are no doubt from debian stable. But, what percentage of debian users are actually running stable, I wonder?

  9. Re:Look at the nature of the vulnerabilities by RagManX · · Score: 2, Interesting

    But even this comparison is not sufficient to get any meaning. What about the severity of compromise? If I have a remote compromise on a Solaris system that gives me access as an unpriviledged user or is just a DoS, I'm less worried about it than a local compromise on OpenBSD that gives root, even though in the general sense I might trust OpenBSD security more than Solaris. Giving a number of vulnerabilities without a severity indicator is worthless. Even with some indicator of severity, you really can't get much meaning from the numbers. Any security expert knows that the security of the system depends more on the skill of the administrator than the OS of the system. In my opinion, articles like this are a waste of time.

    RagManX

  10. The REAL point by GSloop · · Score: 3, Interesting

    Software shouldn't be viewed any different that any other tool - cars, VCR's, Microwaves etc.

    Sure, there are some additional problems, but most come from the design and implimentation _approach_.

    If you've read "The Software Conspiracy" from Mark Minasi, you'll be enlightened about software design, and our expectations.

    Closed vs. Open isn't the point. Either can be just horrible, or quite wonderful. But the devil is in the details.

    What I think many miss is that BUGS = INSECURITY! Not all bugs will cause an insecure system, but some will.

    To make a more secure system, we need to make a bug-free system, or nearly so. Look at these software design and implimentation methods.

    Formal Methods
    Code Audits
    Testing
    Design Reviews
    Codified Best Practices

    These are the very practicies that will give good code, even bug-free code if they are followed carefully.

    Now, as part of the whole solution, you need more than a solution. You need a "push" too. Pull isn't enough by itself.

    It's my opinion that there are a couple of factors that could make this happen.

    User demand. We havn't seen much of this, but it may be growing. We also need to work to change the expectation of users. Most of us even, feel that "Oh, it just crashes sometimes" is an acceptble answer. In fact, how many of us, just add the "just reboot, it'll fix it" to the mix. I'm as guilty as anyone. But this just perpetuates the expectation that software isn't very reliable, and we shouldn't expect it to be. Lets change that.

    Finally I think the legal route should be available too. [I'll get lots of flames here, but I'm ready...] Like any other DEFECTIVE product, the user should be able to redress damage from a product that wasn't reasonably designed. [Many of you will be howling to burn me at the stake now, but read on if you can] The standard for liability is a reasonable effort. I think those that don't use a strict design and implimentation method are not using due care. These methods have been around for some time now. We just don't use them. It's also fairly clear that they can work. How well they can be implimented in real commercial products we can't know, becuase I don't know of anyone that really uses this type of design method - do you? [And not just in name. In real methodical plodding fashion...]

    Lastly, as in Minasi's book, many of you are now screaming - "It'll cost WAY TOO MUCH!"

    Bah! How much of your time is spent chasing bugs down in commercial products. Sure, it only cost $100 at the store, but you put in 35 hours figuring out how to work around bug a,c & c. It crashed, and lost your document. It took 3 hours of tech time to find and restore the right version of the data file, or worse, it wasn't backed up, and poof! Companies spend way too much on support of bad products. These costs never get allocated to the real source, but instead it's just lumped into the general support costs. That just allows the vendor to shift the cost to your company, rather than having an "honest" cost of the product up front.

    If software vendors had the real threat of laibility, they would then get serious about coding practices. If they didn't, the corp boards and shareholders would make sure it happened. A few examples, and we'd have better software.

    Finally, I think that legal liability is the only way this will happen. Until everyone is forced to a higher standard, everyone will seek the lowest common denominator. If you produce better software, but you're new, how will you charge more for it? I just don't think the "market" will fix this. [Not that the courts aren't part of the market, but many will argue they're not, incorrectly IMHO.]

    In the end, frankly, OSS might be easier to fix, but who cares? I think the design and implimentation before and while the code is written is much more important. From that perspective, I think OSS has a more difficult time imposing that regimented framework on it's coders and design people. But it's lots easier to show and embarrass the OSS people, precisely because the code is open - thus a better motivator perhaps?

    Well, I've said my piece - do your damage.

  11. Yeah, this is authoritative by Anonymous Coward · · Score: 1, Interesting
    1. Who verifies the counts for closed-source software?

    2. I see Red Hat has an "unknown" vulnerablity. WTF is that? Is it "I think there might be a vulnerability here but I don't know"?

  12. Re:hummmm not quite by Anonymous Coward · · Score: 1, Interesting
    The difference, and it's a huge one, is the level of rigor imposed on the process. I worked for IBM for years in OS development, and we used formal code reviews. These entailed getting copies of the code days in advance, reading them, and then getting together for a meeting and going through the code one line at a time, as a group. If more than a certain number of errors per fixed number of lines of code were found, the code was fixed and the process repeated.

    OSS is nothing like this, as very little code is examined in any detail, thanks to the tendency for everyone to assume that everyone else is looking at it. And even those people who do look at it 1) are almost never trained to look for errors and 2) don't take the time to do it this closely.

    Open, available source code is inarguably a Good Thing, but we shouldn't assume that it automagically fixes everything.

  13. Open Source does not GUARENTEE security. by einhverfr · · Score: 3, Interesting

    Just because a piece of code is open source does not mean that it has been audited by competent security auditors. Of course the same is true of proprietary software. I think that open source software can be more secure, however, for the following reasons (though these could apply to proprietary software as well).

    1: Access to source: Just because it has not been audited does not mean that it cannot be audited. Software can be considered more secure if the code is at least available to be audited. For this reason, I congradulate Microsoft on the shared source initiative.

    2: Independent audit: When in doubt for a mission-critical scenario, hire someone ot audit the code or part of it. This is possible with porprietary software under some licenses and with permission from the vendor, but it is always true of open source.

    3: Compartmentalized design-- application runs under minimal permissions. This is a problem with proprietary (IIS) and OS (Sendmail) software alike.

    Open Source is no guarantee for security but it helps.

    --

    LedgerSMB: Open source Accounting/ERP