Slashdot Mirror
Software Code Quality Of Apache Analyzed
fruey writes "Following Reasoning's February analysis of the Linux TCP/IP stack (putting it ahead of many commercial implementations for it's low error density), they recently pitted Apache 2.1 source code against commercial web server offerings, although they don't say which. Apparently, Apache is close, but no cigar..."
Why don't they fix them? It seems almost paradoxical, if you find .53 errors per thousands lines of code and fix them, then you'll have 0 errors. But since we can only fix errors we can detect, we only detect errors we can fix. Ok, it's too early on a Monday morning...
Moderation: Put your hand inside the puppet head!
2.1 is'nt even out yet! the latest is 2.0.46!
Cats: All your base are belong to us.
Captain: Take off every sig !!
Just because Open-Source coders can't spell when they insert comments doesn't mean that they can't write good code!
But here's the kicker: the vast majority runs Apache on either BSD or Linux. All of this code, from the kernel to the library that tells Apache how to use PHP, is open source. Every hacker on the planet has full access to the code - which means that they can review it and find vulnerabilities in it. Not many people have access to Windows or IIS code. So why does IIS and Windows come out as far less secure, and is exploited so much more?
I think the answer lies in the severity of the code defects, and the architecture and design of the operating system that powers the web server. And yes, I know that Apache can run on Windows.
Has Apache 2.1 been released as a stable, non-developmental release? If not I would say testing it for defects is a bit premature.
Reasoning found 31 software defects in 58,944 lines of source code of the Apache http server V2.1 code.
so what are the calling a defect?
And don't most NDAs for when they do let you look forbid any competetive analysis?
Or am I just too far out of that line of work to know how these things work?
He tried to kill me with a forklift!
Wouldn't that be unstable? I thought the latest was 2.0.46 or something.. If I'm not mistaken, it would be a bit like saying "Freebsd 4.8 has less bugs than Linux 2.5!"
So basically they offer a service like lclint only many times more advanced ? What is to say they haven't missed anything?
This is probably a publicity stunt for them although a good one. I think it would be a good idea for them to sell software suites of their product if they don't already.
Analytic & algebraic topology of locally Euclidean meterization of infinitely differentiable Riemmanian manifold
As far as I can see, this article says 'We have two arbitary numbers, and one is bigger than the other. From this we deduce that Apache is not as good as commercial software.'
I am TheRaven on Soylent News
According to Apache.org, Apache's latest stable version is 2.0.46. Is that a typo on their part, or are they testing a development version? Also, since 1.3.27 is widely used, it would have been interesting to see how that stacked up as well, having been developed longer.
Either way, to have only 31 errors in close to 60,000 lines of code is impressive!
libertarianswag.com
Since LOC is a poor metric, a "defect density" measurement based on that will be just as poor.
Yes, I know there's not much else to go on, but something along the lines of putting the program through its paces, stress testing, load testing, etc. would be a much better measurement than a metric based on LOC.
The difference is that now that someone has found 31 errors in the open source Apache software, they will be fixed fairly quickly whereas closed source software will have to have the company do a cost-benefit analysis, put together a team to do the fixes, probably charge to put out patches or minor upgrades (assuming the product is Microsoft's IIS ;b)...
Only two things are infinite, the universe, and human stupidity,
and I'm not sure about the former.
Why does it seem a bit odd to be testing software quality with other software? I wonder if they ran their own software through its own program, but then that gets kinda weird when a program starts noticing errors about itself... maybe it'd get depressed and start ranting at the creator on how they should have taken better care of it... ok, I need more sleep
Umm, Apache 2.1 hasn't been released yet. Current latest stable is 2.0.46.
I can only assume that they're looking through the current DEVELOPMENT codebase -- finding a higher ``defect density'' in such a development codebase compared with commercial offerings is not exactly unexpected.
They're also some automated code inspection product; the press release doesn't go into details as to the severity of the defects found or the testing methodology.
It'll be necessary to read through the full report before drawing any sound conclusions.
What bothers me about these articles is that there is more to software quality than the # of flaws-per-unit-"whatever".
Like design.
It seems to me most of the problems with Apache's main competitor in terms of software quality are the result of design and engineering choices made by MS's IIS development team.
In other words, it does exactly what they designed it to do, but what they designed it to do was a very bad idea.
"Lawyers are for sucks."
- Doug McKenzie
I know that Apache has vulnerabilities but it should come better than IIS. You can't realisticly give a verdict on IIS without looking at the libraries called.
As for the rest, I can imagine some commercial products coming in better, but not many.
See my journal, I write things there
The problems with this are:
Kevin Fox
The defect density of the Apache code inspected was 0.53 per thousand lines of source code...
We can bring this number down to 0.2 by avoiding the BSD style guidlines. No kiddings, have you seen the density of MFC code?
BSD code:
char*
foo(int bar, double baz)
{
return bar + random();
}
MS code:
char* Foo(int nBar, double dBaz) { return bar + random() + m_ExtraWindowsBugModifier(); }
So?
There are errors and there are errors. There are error that don't matter a jot, and there are errors that are show-stoppers.
I've worked on banking software containing code that was written in assembly for PD11s and developed over decades. The most horrible spaggetti code you could ever imagine. Why did the banks keep using it? Because for any particular input it always gave the correct output.
Years of bug fixing had made the code horrible and probably full of errors if you were looking at it from a purely theoretical/software engineering viewpoint. But from an input/output point of view, it was faultless.
First, are all of IIS's issues "software errors" per se? I'm wondering if all security problems would have been caught, or if that was really the goal of the analysis. Perhaps it was, but I'm not sure. One could contest that IIS has a lot of things unprotected, but that this doesn't constitute a software error.
And as you say, severity would be another issue. It's always been typical open-source style to get the mission-critical parts hardened against nuclear attack, but leaving the other bits a tad soft. I wouldn't be surprised to learn that was the case with apache.
One thing I want to know - did MS (or whoever) give these guys source or were they analyzing the binaries?
-Looking for a job as a materials chemist or multivariat
Since when are unfounded results from a company that doesn't explain what the "32 defects" were, newsworthy. Don't act like these guys are worth my time, this is bullshit.
Ignore the "p2p is theft" trolls, they're just uninformed
Prette lame when we are talking server software where apache has the lead. (apache 63% vs IIS 25% netcraft.com)
/Esben
"Nobody really checks their email any more. They just delete their spam"
Is it just me that finds this entire concept of "code defects per 000 lines" sounding like a little bullshit?
If the company has developed proprietary tools to enable them to identify defects in medium-sized software projects, which of the following business models do you think is more effective:
1. Design proprietary tools to identify defects in medium-sized software projects.
2. Fix defects
3. Profit
or
1. Design proprietary tools to identify defects in medium-sized software projects.
2. Sit around mumbling about defects, Open Source software, closed source software and why farting in the bath smells worse
3. ???
4. Profit
Secondly, where on earth did they get hold of a closed source enterprise level (which Apache undoubtedly is) web server software codebase?
"Hi, is that BEA? Do you mind if we take a copy of your entire code base so that we can peer review it against Apache's? What's that? Yes, Apache might come out on top, and we will make the results public..."
How do they define a defect anyway? A memory leak? A missing overflow check? A tab instead of 4 spaces?
It just sounds like bullshit to me...
Invoicing, Time Tracking, Reporting
Hell, there are no rules here. We're trying to accomplish something. - Thomas Edison
Another post seems to indicate this was done via software to automatically detect defects. Many (most?) security defects cannot be detected automatically, as they involve using the software in an unintended way.
...then why is it their webserver? :)
Of course it is Apache 1.3.23...
is equivalent to the error level in post-release commercial web serving software. Sounds like an endorsement to me.
Also keep in mind that defect density is just an average. If you have 31 defects in 60k lines of code, that is potentially 31 security risks, or out-of-operation risks. If the other software tested had double the lines of code (120k), the density would imply that they had slightly less than double the defects, so say 58 or 60. That implies _58_ potential security or uptime risks. In this case, imho, defect density is not a good indicator of the reliablity of the software.
:)
My general rule is that if someone is quoting statictics to you, they are lying. At least on average.
By its very nature, Open source will tend to fix important bugs and leave unimportant ones unfixed, while standard QA processes associated with commercial software will tend to fix little UI issues during the release schedule before dealing with vulnerabilities.
So seems pretty clear to me that in Open source, the ratio of showstopper bugs to miscolored widget bugs will be much lower than for commercial software.
This doesn't indicate that the commercial equivalents are better. You've got the DEVELOPMENT branch of Apache, which is derrived from the 2.0.x code which is a complete rework from the original 1.X branch of code. So it's a rather new code base and it's showing similar defect rates to a code base that has been around for a while. I'd say this prooves that open source is better.
This sig has been temporarily disconnected or is no longer in service
The longer and more content you have per line the higher the likelyhood of error/ line.
As example with one errror in 100 lines you get 1% error. Imagine you could do the whole thing in one line. Now you have 100% error.
Help fight continental drift.
I compared this to my 'other' server, for now unnammed.
My 'other' server brought me coffee, 2 pieces toast, 2 eggs OVER EASY, 4 strips of bacon, *and* Smucker's Grape Jelly with nary a mistep, or hesitation. This other server smiled, asked how my wife was, and brought me a new fork when I dropped my first one.
Congratulations, Gloria! You win the 'great server' award!
This article isn't worth the 2 dollar tip.
Why doesn't Reasoning fill the niche, and code a completely error free web server? They know other peoples mistakes, so they should know how to code an error free one.
Well, seriously, I wouldn't put much in their obvious estimation.
Any technology distinguishable from magic, is insufficiently advanced.
Ok, IIS is the obvious choice as being the second most popular web server after Apache. But I hardly think Microsoft will be letting these guys all over the IIS source code.
It could also be Zeus, SunOne or one of the other lesser known web servers out there.
Read reviews of shopping cart software
Some things I found interesting:
One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.
So you can get a sense of what a defect looks like, here is #21. The orignal uses bold and fonts improve readability, but I don't know how to reproduce that in slashcode:
DEFECT CLASS: Null Pointer Dereference
DEFECT ID 21
LOCATION: httpd-2.1/srclib/apr/misc/unix/otherchild.c : 137
DESCRIPTION The local pointer variable cur, declared on line 126, and assigned on line 128, may
be NULL where it is dereferenced on line 137.
PRECONDITIONS The conditional expression (cur) on line 129 evaluates to false.
This study makes a lot of sense to me - that the defect rate is tied to the maturity of the code base. I have long felt that Microsoft's business model where they redo the operating system in order to churn their user base and induce cash flow will always result in more defects and security problems than a model where software change is driven on a solely technical basis.
I think the next step for these folks would be to take a project that has a long history, say perhaps Apache 1.x and show defect rates over the life of the project.
Well, the reports simply state that, in the 360 files they checked (most of them header files) they found 29 cases of a potential NULL pointer dereference and 2 potentially uninitialized variables. This is from the Apache 2.1 codebase as of 31st Jan this year, about 58k lines of code.
Their automated checker also searched for out-of-bounds array accesses, memory leaks, and bad deallocations. It found none.
They also state that they ran the same checks against other codebases, and found that they did marginally better, on average.
In short, this report says that OLD development code for an unreleased opensource project is nearly as good as current commercial offerings. That's at best, when you consider the huge gamut of possible defects that this checker won't pick up. That margin probably disappears in the +/- of the sampling if you were to do a proper statistical analysis.
The report is fairly useless. It certainly should not be taken as a reason to not trust Apache; to do so would be foolhardy particularly given Apache's track record.
Oh, and Reasoning's webserver is being pounded into the ground. You can get my local copy of the reports from here.
The majority of the secruity holes are from the people setting up the web servers. The holes are usually abused by "wanna-be" hackers, or script-kiddies. The problem is that people are not educated enough to run some of these programs. Being able to understand Apache, and how to make it operate correctly is not everyone's top priority. As long as it works, people don't care how it works (as goes for many other things in this world).
Every Super Villan uses Linux.
One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.
appache got its own malloc() that kills the child (and closes connection) if it fails to allocate enough bytes.
The thing that always kills IIS, is the integration it has with Windows. This isn't a defect in IIS, or Windows, per se, but rather a defect that arises because of how they integrate with eachother. A script executes on IIS in a way that's not inately a bug, but then when it interacts with Windows, Exchange, etc, suddenly it becomes one.
Apache is just a webserver, and that's all. PHP, JSP, etc, are all separate applications treated separately. The integration does make things more efficient, yes, but also more prone to problems.
This sig has been temporarily disconnected or is no longer in service
Metric Report
They make you fill out a form that asks for your email and then do an opt out checkbox at the bottom of the form (you have to check it to NOT get spam from them). The site's a bit slashdotted right now though.
That reminds me of an old (early 1980's) product named BILF (Basic Infinite Loop Finder). It was supposed to be run against BASIC source code and it would find all infinite loops in the code, or so the vendor claimed.
A magazine reviewed the product. In their review they included a formal mathematical proof that such a program could never work. The vendor responded to the proof by saying that they would fix that problem in the next release!
1) Apache 2.1 has more bugs than some unknown commercial competitor. If the version is correct, a development (not-ready-for-release) build was pitted against a released commercial build. Not fair playing ground.
2) Reasoning does not detail the severity or kind of the bugs. Certainly, a web server not being able to handle a type of format (pdf, csv, ogg vorbis) is less severe than a security hole. Pitted against IIS, I would trust Apache even if it had more bugs, because historically it has had fewer security patches. Check out Apache's 2.0 known patches vs IIS 5.0
Well, there's spam egg sausage and spam, that's not got much spam in it.
As has been pointed out a couple of times in other comments, 2.1 is the development branch of the Apache web server - ie "beta", "buggy", "work in progress", etc. etc. In stead of reading this as "Apache has roughly as many defects as closed source web servers" let's read this as "the development version of Apache has as many defects as... well, some unidentified (beta? shiping?) version of some unknown (iPlanet? IIS?) web server". But you can be *much* more confident that these defects will be fixed in Apache than in the *other* product.
/t
Heck, forget confidence - YOU CAN JUST CHECK.
The fact that Reasoning didn't have to go and get permission from Apache to run this test - coupled with the fact that we don't even know what Apache is being compared to - is the *real* point behind this "article".
ps: IANAL but don't they have to include a copy of the Apache License given that they publish fragments of the source code in their defect report?
#!/usr/bin/english
I suspect the following code will be flagged as a defect:
as long as doOrDie() does its job and never returns a NULL then where's the defect? The guys who wrote this tester seem to want you to check any pointer dereferencing against NULL before use - I might be doing this in my doOrDie() function, I dont want to have to do it twice.Slashdot's summary of this article is way off base, and the article itself couldn't be less useful. Counting the number of "errors" in lines of code... and the ratio is supposed to mean something to us? As compared to unnamed other software? C'mon, I have better things to do with my time.
*plonk*
A hen is only an egg's way of making another egg. -- Samuel Butler
One of the best ways to get to know a large code base like Apache or something else is to find a repeatable bug and track it down. To fix a bug you do not need to understand the whole program, just the relevent parts. I've submitted bug fixes to several projects, so I must strenuously disagree, especially because, ahem, I have never submitted a bug fix to a proprietary project because its impossible.
Of course, this test of the code is purely a test of coding errors rather than errors in the code logic.
The most worrying errors in programs are generally not coding errors as they are either terminal (ie. crash) or they are benign (the error may cause memory corruption in a place where it does no harm). Of course, there are exceptions such as buffer overflows, but I'd class those, in general, into the logic error category.
Logic or algorythmic errors are far more dangerous as they can be well hidden and are more likely to make the code do things unintended. The code itself may be perfect but if the algorithm is faulty then there's a major problem.
Agrajag: "Oh no, not again!"
Why did they use the development branch of Apache, when only a handful of sites are running it? I would have found an analysis of the stable 1.3 branch, which 60% of the web-serving world uses, to be more informative.
In almost every case they listed the pathway was via a failed malloc.
Apache has it's own malloc that kills the connection (and the child) if it fails.
That code can never be reached. Their test is invalid.
First, as many posters have noted, Reasoning DID NOT TEST APACHE 2.1. They tested Apache 2.1-dev. That's dev, as in development branch. As in: I have new untested code, so don't use me on a production server until I'm released in the STABLE series.
For a valid comparison versus commercial software, the testers should have used Apache 2.0.46, the most current STABLE series release.
Second, I'd be interested to see a comparison of 2.0.46 versus 1.3.27. I have a pet theory that multithreaded C code has more bugs than single-threaded C code, and I'd like to see whether there is evidence to support it.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
You have the wrong idea here. There is a point in which you must realize what information you can release without comprimising the security of your system. While I can give you the plans to my vault, I will not give you the combination, nor the first or second numbers in it.
For the star wars geeks out there, if you were a Jedi, you don't go around telling everyone you're a Jedi, nor do you flash your light saber in public places. They do realize when to show their light saber, and when they can tell people they are a Jedi. Nor do they not tell anyone who they are, or never show their lightsaber.
You might want to check out Secrets and Lies which will give you a better understanding of security philosphy.
Every Super Villan uses Linux.
Actually, I've found that fixing bugs in large projects is about the same whether or not you are familiar with the project, provided that the author was no smoking crack at the time he wrote it.
For example, I managed to code, test, and patch a "fix" for PostgreSQL this weekend in under 2 hours, having never seen the code before.
The "fix" wasn't a bug, per se, i't just that the output of pg_dump wasn't optimal in my usage for dumping the schema for CVS revision control. I added two flags, -m -M, which molded the output to my liking.
If you haven't seen your code in two months, you and an outsider have about the same chance at finding and detecting bugs/misfeatures.
Engineering and the Ultimate
Every time I hear the "obscurity is not security" mantra I chuckle. Of course it isn't, but that doesn't make publishing the information a good idea.
Nobody's saying that the information should be published - what they're saying is that you can't rely on that information being a secret.
Is Fort Knox secure? Probably. If so, then why don't they publish the blueprints, guard rotation schedule and security policies?
That's pretty much the point you're missing - even if that information was published, it wouldn't diminish the security of Fort Knox..
If the people in charge relied on the fact that they don't publish those details, that would be obscurity, because it would lead them to make errors elsewhere. (Oh, it's OK if we leave the main vault open tonight - nobody knows that there will be no guards around it for 10 minutes at 3:30 AM tonight.)
One word: architecture.
And not just the architecture of the web server, but the architecture of the entire platform. But specifically looking at the architecture of Apache versus the architecture of IIS, you'll immediately see that the goals of the two pieces of software are not the same. Look at things like IIS's metabase - the structural details of the server's configuration are kept in an in-memory data structure, which is easily modified while the server is running. Apache, in contrast, reads its configuration at startup, and uses it to determine which modules of code are loaded, and how they are used to process requests - fixing the behavior of the web server at startup.
IIS follows typical MS enterprise software design - it has to interface with COM, and the NT security model, and active directory, and the registry, and a million other systems, all in the name of integration, and enterprise management. Apache doesn't have PHBs telling it that it needs another way for the metabase to be edited, or a new instrumentation API, or whatever else a particular large customer asked for - and can get on with just providing its facilities cleanly.
That's why IIS has so many more security holes, even if it does (as may or may not be the case) have the same raw coding error rate as Apache.
MY compiler (Microsoft C++) does catch this
if (myPointer = NULL) {
and issues a warning. Doesn't gcc?
Yes, it does. So does every other C compiler I've ever used (quite a few). I suspect the original poster may be the sort who ignores warnings....
Why?
Am I the only one who looks at reasoning's results with suspicion (even when I agree with them). Any analysis using methods that are not open and repeatable is not science. This just feels like marketing to me. (it is sad because the study of code quality is such a worthwhile pursuit)
they dont say what they used for a comparison.
when they tell us what they used, then I will believe it.
this smells microsoft.
bring it on! we want to know what it was compared against, sure as hell was NOT IIS...
Looking at their first "bug", a little manual inspection shows that it's in the "can't happen" category, even without knowing about hidden information. The code looks like this:
current_provider = conf->providers;
do {
{some safe code}
if (!conf->providers) {
break;
}
current_provider = current_provider->next;
} while (current_provider);
and they identify the second-to-last line as the "possible NULL pointer reference". Note that the "break" before that line will be taken if the pointer is NULL, so it can't happen. In fact, the static analysis could have determined this if it were a little better at propagating values.
First conclusion: subtract at least one "bug" from the 31 defects in Apache. This lowers the rate to 0.51, the same as the "average commercial code" number they quote. Yahoo!
Second conclusion: their static analysis must identify a lot of false positives, if the very first one in the list is one (I would look at more, but I should really get back to work...)