Slashdot Mirror
Bug Hunting Open-Source vs. Proprietary Software
PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."
that many of the bugs found by coverity have already been fixed.
I smell some Microsoft conspiracy behind this. :-P
I scanned through the article, it didn't seem to mention how they tested the top proprietary software. I can well understand that there are are a lot of bugs in open source code since it is written by humans. But human also right the proprietary code. How did they test it?
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
Knuth used to have this great offer where he'd send you a check for pi or e or something if you managed to find a bug in his code.
Well, what is a bug?
I doubt he'd send me a check if I told him that TeX doesn't have an easily accessible iconic user interface. No, his concept of a bug is a deviation from the specified functionality.
But what if that functionality is wrong or sucks?
Apple does really well at creating functionality that doesn't suck. They suffer from the same problems of deviations from the spec as much as anyone, but they manage to mold their spec around what users want. Microsoft, to some extent, does the same and they release products that conform to what users want (generally) because they change the spec as necessary when customers demand change.
If you are implementing towards a standard (like most OSS projects with any traction are wont to do), then you are necessarily restricted by what that spec says. If the spec says to do something inane, the standard-follower must implement it that way.
I don't really have a point here except to say that unless they say "this is what we mean by bug", there can be no way to really examine their results.
Consider the case of open source software. An open source software is tested by a whole lot of people over the world and everyone is free to take the code and test if. On the other hand in case of proprietary software this is not the case and is tested by far less number of individuals. The true measure will come when a proprietary software which has been made open source is tested against an open source software and i can not think about any :(
What do you think folks.
Isn't this an old rant? Sorry if I come out as a troll!
"Deanna Asks A Ninja: What is the circumference of a moose?!"
"It's michael pailum with his face in a pie times douglas adams squared."
This answer makes as much sense as the article.
Except "Ask A Ninja" made more sense. And was more accurate. And more entertaining.
Can I just get a Ninja hit out on this guy something so these articles will not make it slashdot anymore?
How do they find all bugs in a program? what's their secret? They have to ge really great debugers... I think this kind of conparasions is hardly never accurate, May be in open source projects the kind of bugs is diferent... Depending on how you program you make one kind of mistakes more than others, Does someone agree?
The problem is that there are different types of Bugs. things like a typo in a help file, or American spelling vs British spelling, vs a bug were the app crashes the system when installed on a system with an early version of Quicktime are clasdsified differently.
The summary just says all bugs, which is not fair if the proprietary has 5 times the number of critical or super-critical bugs.
"It is a greater offense to steal men's labor, than their clothes"
Somebody please explain to me exactly what kind of software bug can be found by automatic scanning that isn't found by standard debugging and compile-time checks. If a computer can ascertain exactly what the programmer intended to do, why do we need programmers?
The simple answer to this is that they can't. That's the point behind hiring human codeslingers to write applications. Considering that most software bugs are logic bugs (off by one, etc) that can't be directly seen in the code without actually, you know, RUNNING the program, I find it difficult to believe that AI has come to the point where it can guess the coder's intentions and infer the purpose of an application.
Additionally, where exactly did this proprietary code come from? I don't know which companies these programs were released by, but other than the SharedSource initiative which mostly caters to programming students (if they still even have this program), Microsoft would call their next o/s Windows Sh*tburger (I think I'll trademark that) before they would release their code, especially to a third party consulting firm.
I find this article highly suspect
as scanned by coverity.
;)
linux 2.6: 3,315,274 lines of code, 0.138 / 1000 lines of code.
kde: 4,518,450 lines of code, 0.012 bugs / 1000 lines of code.
based on this I would say we are doing pretty good with open source.
but we shouldn't forget that this tool only scans coding errors, not coding logic.
wine for example only has 0.112 / 1000 lines of code as well.
and we all know it by far doesn't always do what we want it to do.
...and while it is on the list on the web page, I was happy to determine that most of the issues they found were false alarms. They found three real bugs, none of which were likely to bite, and even if they did bite it is not exploitable. Nonetheless, those bugs probably wouldn't have been found otherwise, so I was happy for the scan.
Rather than brag (I won't say who I am or the name of my project), I'm just going to sit back and read all the defensive flames from self-appointed "security experts" whose open-source project didn't do so well. After all the flames from these "security experts" that I've endured, I'm going to enjoy watching them squirm.
It's karma.
Why does this surprise anyone? Propriety software traditionally undergoes a formalized, designed testing process. It's not perfect, but it's an ordered approach to boundary testing, design level implementation of quality, and more. Open source software must rely on after-the-fact testing in the form of "this broke when I tried to do this".
In the end, it comes down to black box vs. white box testing. Commercial software has a strong QA engineering component. Open Source software relies primarily on a black box testing approach.
Open source has MANY benefits and MANY advantages over commercial software. This just doesn't happen to be one of them, but unlike the commercial software, the bug fix cycle on open sourced stuff can be a LOT quicker, so it evens out in the end.
While I appreciate that PreacherTom was good enogh to bring this to us, the sentence "...no open source project had fewer software defects than proprietary code." just does not match TFA.
TFA says that no open source project is as good as the BEST of proprietary, but it also says that the AVERAGE open source is better than the AVERAGE proprietary.
The open-source development community must first graduate from Lake Wobegon University, where all of their software is just above average
This article is about how apache is more buggy than control software for a jet engine. Yes, I hope that's true. Maybe open-source is not better than the BEST stuff out there right now, but they say it explicitly -- its better than your average proprietary package.
No, *popular* open-source software is 5x as buggy as *safety-critical* closed software. The linked dissenting opinion is at least partly right; they're comparing apples to oranges.
Maybe they should try comparing open- and closed-source software that's actually trying to solve the same problem? That'd be a bit more valid of a comparison...
Well, the article with some arguments covered one thing I was going to mention, there's a big difference between software to control jet engines or nuclean powerplants and software to be used as an office suite or the like. Of course there will be quality differences between those as bugs in one can likely kill people where as bugs in the other probably won't. They have different levels of allowable bugs and required quality. The other thing which was not mentioned in the second article was this. What were the bugs and how quickly are they fixed? Having the same amount of bugs or even more bugs may not be a problem if those bugs are mere annoyances rather than something that can cause you to lose hours of work or security holes which can cause you to have secure information stolen or be used to attack someone else. How fast the bugs are fixed also matters. What if one piece of software has only 3 bugs but they take 2 weeks each to fix whereas another piece of software has 8 bugs of similar urgency to fix but has each of them fixed in only 1 or 2 days? Which is the better software in this case? Now maybe these things were taken into account, but the article certainly didn't make it clear to me if they were or not.
the report (carried by Business Week) said that the Porpriatary software that beat out the open source stuff was avionics software or controls for reactors or other heavy industrial software. That stuff is all small, done in assembly, and extensively tested.
It was not an apples to apples comparison, more like apples to diamonds. Dom't worry, just fix any real problems identified. Many of the bugs found are theoretical, not real. Many others are style questions. the experts will probably never quit arguing about what is 'good' and what is 'bad'.
There are also some very real race conditions, memory leaks, and things like that. A real list by line number would be nice.
That said, what would really matter is a comparison program by program. I don't think my quick and dirty one off is really in the same class as the Kernal, or Firefox,I havn't seen how this diferentiates.
Everybody knows 3 people with my name.
Open-source software is expensive if you want a commercial support contract (because you are asking a professional to spend a lot of time learning).
Closed-source software doesn't have the function that you want, and you cannot fix it to add the funcion that you want.
You pays your money and you takes your choice. You can always stick to pencil-and-paper, and not use this 'software' stuff at all, if you prefer.
The article makes it quite clear that the proprietary software which is much better that open source is mission-critical software. A class of software where ensuring minimum bugs is a top priority, and also a class of software which mostly does just not exist in OSS. If you are an OSS developer, would you try to develop open source air traffic control software? And even if yes, how would you do it anyway?
Basically, my own conclusion from reading the article was that it IS possible to write excellent software with very few bugs, if that is a top priority. And, that the author seems to say that while mission-critical software (which happens to be proprietary) is fortunately much better than the rest, among all that other non-mission-critical software, open source tends to be better than proprietary.
Not surprising, and quite encouraging...
they tested it by using a program that systemattically scans code for common errors.
A method known to have flaws. It raises a ton of false positives, things that might "look like" potential bugs but aren't because of the data flow. You have to do a data flow analysis to see if they really are bugs.
For example, not checking for buffer overflows when copying strings, etc, is usually considered a (potential) bug. Certainly it is when dealing with unknown input. However, in a function buried deep behind layers of other code that has already filtered out potentially overrunning strings, it is quite safe. (At least until said code is changed, so it is considered bad practise, but the fact remains that if there's no data path that can cause a problem, it's not a bug.)
Mind, there's bugs and there's bugs. The Space Shuttle software is generally considered to be about the most defect-free software around, but they'll launch with some bugs. Specifically, minor bugs in the display (ie a misspelled word, a column that doesn't quite align) are allowed because they're a much lower risk than going in and changing the software to fix them (and possibly introducing a different bug).
-- Alastair
This article focuses on the result that aerospace, financial and telecoms proprietary software has items in it that get better "quality" scores than any open-source package they tested. This is expected - if your average package crashes, no-one dies. If your plane's software crashes, so does the plane.
It also notes that average open-source beats average closed-source.
So, it tries to ask the question: "How do we make open-source stuff as good as aerospace quality?", but comes across as being more controversial than it needs to be.
I'm scared of numbers that can't be written as a fraction. It's an irrational fear.
Well first of all, you have to assume that all programmers even do the "standard debugging and compile-time checks". Even then, those checks are often hardly comprehensive. You can build some scanners that will catch rudimentary bugs that SHOULD have been caught, but were not. For example, assign things to null, test boundary conditions, etc. These are all things that should be part of a standard unit test that's delivered along with the code.
Also, things like memory leaks can be difficult to pinpoint. That's where tools like BoundsChecker are nifty.
No one is saying that a program or "AI" (as you call it) can find all bugs. But it can certainly be used to find some rather simple ones. Overall, though, I agree that you can't depend on running programs to catch bugs. You've got to have a solid QA department, which will use all the right tools to get the job done and try to maintain quality to the highest degree.
I'm not saying that OSS can't do this at all. I certainly think it can be done, but it does take more process and structure. Not having worked on any OSS projects myself, I imagine the largest, most important code-bases have a lot of this in place to drive quality. But your smaller or even mid-size OSS projects may be lacking in that department (much in the same way smaller and mid-size dev houses lack a decent QA department).
-- jchenx
You'd think just being one times less buggy would be enough.
It should come as no surprise that people here cannot accept the results of any study that shows concrete flaws in the open source software philosophy.
However, what this article says can only make sens.
Developping software is much more than coding. There is planning, designing, testing, etc involved. It's a whole organisation that requires a lot of organising. Open source people only put emphasise on the coding aspect, because that's what they care about most. Hence the result is often applications that look like crap, are difficult to use, are not documented and crash on startup 50% of the time.
In the end, the last bit of testing is always done by the user. I call it the test of time. This part of the software development cycle happens too early with open source software, almost all the time.
Of course, I am talking here of community developped software, not software where the company releases the source. Opening the source in itself is immaterial to the source.
Linux violates 235 Microsoft patents.
He's comparing "bugs" in a project such as Apache with "bugs" in the software controlling a jet engine on an airplane.
He refuses to accept that different projects have different requirements. When the project results in people dying if it fails, you spend a LOT more money and time finding all the "bugs".
When the worst that happens is that you don't see a web page, your money/time requirements are not so high.
Even so, from his finding, Open Source is, on average, better than the closed source projects (not counting the closed source projects that result in loss-of-life in the event of a failure).
He's an idiot for confusing the different requirements.
The selection of programs from the two populations of programs (open source, proprietary) are not going to be comparable: vendors of proprietary software have a say over which code gets scanned, and they are going to select a different population of programs than the company selected for open source projects. This isn't a fixable problem: there is no way of doing this sort of study so that you can compare the two data sets. The best they could do is compare something like OpenOffice against Microsoft Office, or Apache against IIS.
Furthermore, Coverity simply cannot accomplish what they claim to accomplish: there is no way of detecting "bugs" automatically--if there were, compilers would already be doing it. Coverity effectively does little more than compare code against a set of internal coding conventions; that can be useful if it's done right, but it's not a measure of code quality. Some completely correct code will score thousands of violations against their tool, while other code may contain thousands of bugs, none of which register. Furthermore, it is likely that a lot of their customers are Windows based and that Coverity is biased towards Windows-based coding conventions, giving more false positives on non-Windows code. Before publishing such comparisons, Coverity first would need to demonstrate that their tool does not contain such biases.
Finally, and perhaps most importantly, the company isn't publishing its data, so nobody can verify or even evaluate their claims. Not only do they fail to publish their raw data (obviously, they can't do that for proprietary software), they also fail to list their summary statistics by vendor and project (which they could, but obviously won't do). They don't even give a summary statistic by class of application, class of organization, and code size. Their results are meaningless because they're not reproducible.
These numbers tell you nothing about FOSS code quality relative to commercial code quality. What they tell you is that Coverity apparently doesn't know how to do statistics, misrepresents what their product can do, and doesn't know how to report experimental results properly. Now, do you want to put your trust in such a company?
This is just smart marketing. Imagine they put up a survey that did not make any controversial claims (something like, open source and proprietary software are comparable), then would that generate as much heat? Now many people hear about the company because more people talk about this now than if the survey said something less controversial.
Now to compare every open source software application to aerospace software is really comparing apples to oranges. There is a big difference in the expected quality between an editor and an aerospace application. It's alright even if my editor crashes once in every 20 times I invoke it. Is that acceptable with an aeroplane?
I'm sure the folks at Coverity understand all this. But if they really speak what is right, they will not get all the eyeballs and publicity. In classic slashdot lingo:
1. Do something (anything) that involves open source and proprietary software
2. Make claims that sound outrageous / controversial
3. Profit! (with all the free publicity)
I'm much more funny, interesting and insightful than the moderators think
It's not that OS may or may not have more bugs, it's that we can actually FIX them. Proprietary bugs are at the company's discretion.
- This is the typical meaning behind the word "bug".
- That's where your TeX complaint would fall under. It's "by design" that it doesn't have an iconic user interface, but that doesn't mean it's something that shouldn't be addressed ever
- This is actually a result of the bug tracking system that we use. Rather than sending e-mail, which often gets lost, we often track work items as bugs. For example, "Need to turn off switch X on the test server when we get to milestone Y"
To further complicate things, there is a severity and priority attached to every bug. Severity is a measure of the impact the bug has on the customer/end-product. It can range from 1 (Bug crashes system) to 4 (Just a typo). Priority is a measure of the importance of the bug. It ranges from 0 (Bug blocks team from doing any further work, must fix now), to 3 (Trivial bug, fix if there is time). (I don't know why the ranges don't match, BTW, seems silly to me)
As anyone who works on large-scale project probably knows, there are always a wide range of bugs, across all the pri/sev levels. To me, a simple count of all the bugs isn't terribly useful. A project could have a ton of bugs, but most of them being DCRs (which are knowingly going to be postponed till the next release) and/or low pri/sev bugs. Or maybe it's the beginning of the project and they're all known work items. Or a project could have only a few bugs, but with all of them being critical pri/sev ones.
So, whenever I see a report that simply talks about bug count, I take it with a huge grain of salt. If I had to guess (I skimmed the article), it seems like OSS projects have far more bugs, but perhaps lower pri/sev since the product itself has been evaluated as being higher quality. In the end, it's the quality that the customer really cares about.
-- jchenx
From the summary:
From the article:
In other words, they're comparing the best proprietary code---in this case some aerospace controller program---with the best open-source code. Of course, we would expect the controller program to be more closely scrutinized. There hasn't been life-and-death deployment of open source software nor a need for it.
On the other hand, the article doesn't clarify whether the top quality proprietary software can be purchased by a consumer. I'm guessing these programs are generally not available to personal computer users because of their highly specialized nature (e.g., running a nuclear power plant or jet turbine engine).
I once had a signature.
"Propriety software traditionally undergoes a formalized, designed testing process"
You're kidding right, what about that US university booking that wouldn't accept applications from 'overseas' students with addresses in the UK. Or the Airline Radio system that borked every 2^32 millisecs seconds when a 32 bit buffer cycled round to zero.
"Open source software must rely on after-the-fact testing in the form of "this broke when I tried to do this"."
"Open Source software relies primarily on a black box testing approach."
You've got that the wrong way round, closed source is the blackbox.
Re:Why is this surprising?
davecb5620@gmail.com
He's selling a service/product ("bug" scanning).
If you required that he match the apps/categories, then he wouldn't be able to match aircraft software to any Open Source project. Without the highly tested, life-critical proprietary apps, his case would collapse.
Which is why he only differentiates based upon "proprietary" or "open".
The '11 of the top 15' is also grossly misleading. There were twice as many proprietory projects as OSS projects - you'd expect them to take 10 out of the top 15 slots. Deviation by 1 from that is lost in the noise.
I agree with your analysis - I've been on the fixing end of a lot of these kinds of reports, and have known that the flagged error can never occur, but the linting nazis insist that there must be zero warnings at any cost.
I thought Voyager was the ultimate in stable code, not the space shuttle?
FatPhil
Also FatPhil on SoylentNews, id 863
According to http://scan.coverity.com/ , subversion has 15 lines of code.
That's a bug, surely. Hypocrites!
FatPhil
Also FatPhil on SoylentNews, id 863
I happen to work in QA for a very commercial software firm (MS in fact, although it's in the games group). I agree whole-heartedly with your comments.
Commercial software has no "lock" on QA. Good fundamentals can be practiced anywhere. And I've certainly seen many testers coming from other commercial firms that have no idea what it does to be a good tester. (Definately instances of that in MS as well, we're a very large company)
I think the only difference "commercial vs OSS" has on QA is perhaps the environment that you're in. OSS projects can benefit from generally being smaller, so they can be more agile, try better software engineering methods, etc. On the other hand, it naturally can take a lot longer to change a larger, commercial firm, away from traditional the waterfall model, etc. On the flip-side, commercial firms benefit from just being able to spend more money: hiring more testers, doing more formal usability studies, etc. Larger OSS projects probably don't have problem recruiting good talent, but I imagine the smaller ones might. Of course there are always exceptions to the above generalizations, and I am in no way trying to say that one is necessarily better than the other. (And I could be wrong about some of them as well, I have not contributed to any OSS projects myself)
But to generalize that commercial software always has a stronger QA engineering component than OSS is bull.
-- jchenx
But as I read it - they didn't audit the proprietary code base - they counted the errors found, per 1,000 lines of code, and errors fixed.
Now, if Open source is better at finding more subtle errors, even if it fixes them, doesn't his methodology penalize OS code against proprietary code where they didn't find and correct the error in the first place?
Pug
An Invisible Entity of Vast Power whose existence must be taken on faith alone: Liberal Media
Open source software, it's easy to find and it can been downloaded and evaluated on the spot. When looking through commercial software I get frustrated with the difficult to search marketing material, the lack of fully functional downloads, and limited documentation before purchase. Open source, on the other hand, can be easily found by searching sites that do a lot of the indexing work for you such as SourceForge.net and all the documentation is readily available as plain text or HTML. Obviously, this does not apply to all commercial and open source software. I've seen a lot of good and bad on both sides.
What they have not done is compare comperable projects -- IE to Firefox, Open Office to MS Office, Windows to OS-X to Linux-KDE. There is, as far as I know, no Open Source software product that is really intended for mission critical applications -- I guess maybe SSH might qualify, but I don't see it in their list.
So, I think what we have here is a comparison of Apples to Turnips using a dubiously calibrated error-o-graph machine that uses an unknown technology to perform undefined tests on software.
Don't get me wrong. I sure as hell wouldn't run a nuclear power plant with Linux-X-Windows-whatever. Nor with Windows -- neither Windows 9 nor NT based Windows. They don't meet my admittedly subjective standards of quality either. But if we waited for near perfect software quality, we'd still be trying to get text mode right. Personally, I 'd vote for that because I think building major structures on weak foundations will likely lead to big trouble a decade or three out, but I think I'd lose that vote about 93 to 1 with maybe 6 abstensions.
You can't see ANYTHING from a car, You've got to get out of the goddamned contraption and walk...Edward Abbey
I have audited many many programs. Never once have I been ignored, or had to argue the case for anything I found and reported.
Still I know its not easy to write secure code, even with my auditing experience I've still released code with exploitable holes. You need to get everything right every time, the attacker only has to find one hole to take over...
The more lines of code the more bugs it will naturally have, cannot be helped. People spend time working on fixing said bugs, some slip through nonetheless, some are hard to replicate some are glaringly obvious. A Typo in a message for the user, is annoying and while easy to fix is pretty low on the scale of needs to be fixed. A memory overflow causing say a moriphene drip to open up and rush a whole lot of painkiller into the body, much more than it can handle while a nice way to go.. the patient is still dead, and a bug in such a circumstance is pretty HIGH on the needs to be fixed scale. Not all bugs are the same, however when you have the potential for ANYONE to review the code to make sure its up to certain standards the chances that code that threatens life and limb or the bottom line, gets through is radically reduced. Not to say they won't get through even the most stringent review, but the more eyes on potential problems, and the more people who might be potentially able to FIX said problem, the better it is for everyone. Proprietary code reduces these checks and measures, literally reducing the eyeballs that MIGHT see something another did not. Would you trust your life to code that was reviewed by a small group anywhere from six months ago to a year or so, vs, code thats constantly looked over by everyone who wants to be involved in the project including the organization involved in having said code run the mission critical application?
Okay, now I'm not much of a big name coder here, but from my experience here at university in learning how to code data structures one thing is always true; it's not what you code, it's how you code. Open Source does not mean good tidy coding practices, open documentation, and etc. It just means everyone else can see your code, that's all. As for it being better at finding bugs, that would be true if documentation practices were more strict, but many closed source projects have the same ailment of not being able or willing to document features, patches, and bugs of their program. Ultimately, whenever I read something like this I realize their assumption is based on that some how certain practices are automatic to open source vs closed source, which is not true since open source licenses do not require the programmer at anytime to program a certain style, it only handles the release of the source code and nothing else.
:)
If the author of this article knew that, then the article would probably not exist at all. So, the tag [FUD] fits perfectly.
Quality of Programmers is critical...
Which would you rather have 100 monkeys programming on a project or 10 skilled programmers?
More programmers and more 'eyes' on a project does not mean it is going to be inherently more bug free. In fact with a group of bad programmers in the mix, it can cause severe harm to a project.
I'm not knocking Open Source, but for people to just expect it to be better because more people have access to work on it, have obviously not met as many programmers as I have.
There are a lot programmers that put time into project (and yes Open Source) that have no business developing a VB application for a 10yr old kiddie game, yet they are taking part in large scale coding projects that truly would be better off with them not working on it.
When working with XWindows years ago, I ran into a few people that scared the hell out of me and other people. They had no vision or scope past the specific things they were trying to do, and would often come up with modifications or 'features' that would break more than it added to the project.
In the Windows world of 3rd Party developers I have also found 100s of people I wouldn't want them to develop Hello World. As they had no concept or regard to security, Unicode or many other features that would fail when the applications would run on a non-English system with a user having administration privledges.
You can even find many commericial products in the 3rd party Windows world that also have these problems, but are yet produced by big companies are popular products.
I wish that all ideas would be welcomed into a project, but the people having the final say could trump crap programmers and crap ideas if they are detrimental to the project.
When you look at the Linux Kernel or BSD, you can quickly understand why Linus and others don't want to let the 'deciding' control into the masses, or both of these core OS would become crap in a matter of months of unregulated programmer additions.
SIGSEGV caught, terminating
wait... not that kind of sig.
Another curiosity: vim has 276,398 versus emacs with 233,375 ? Who said emacs was bloated?
What is the reason for all of the complaints? So what if comparing open source to mission critical proprietary is an apples to oranges comparisons. When attempting to better yourself to become the best possible you don't compare yourself against your peers - you compare yourself against your betters. It may be the case the OS(open source) can't use the same methods as mission critical PS(proprietary software), but it at least known that there a better defect rate is actually achievable in reality and is something to strive for.
Typical PS advocates could also make arguments for an apples to oranges comparison - we have constraints placed on us by customers and managers, we have to be concerned about making a profit, we have to beat others to market and so on (and yes OS advocates will make arguments against this - the point is everybody can make arguments about why a comparison is unfair - but a difference in quality exists regardless of fairness of comparison). In the end, it doesn't matter - typical OS is better and the typical PS needs to improve, just at OS needs to improve compared to mission critical PS.
What this article really enforces is the idea that no model is perfect (which should have been obvious to everyone but apparently isn't) and all of the models have room for improvement and the people using these different models should learn from each other.
There site doesn't seem to give me the source code to the scanner, just keeps asking me if I want a free trial?
How can anyone be sure of how reliable the study is without reading the program that the entire article is based on, if the security scanner is flawed then the article i pointless.
Would you believe someone who told you he had built a device which could measure something that noone new existed, but refused to allow you to see his device, I sure wouldn't.
As many people have said, the scanner can not detect ALL faults.
The question is How was it taught to find faults (maybe taught is a bad word, maybe it should be 'designed').
If it looks for common programming errors how did they determine what was a common programming error?
Asking programmers is a risk, do they always know when they've made a mistake?
The only real way is to look at code where a bug has been found. Proprietery software won't let you do this, Open Source will.
If they built thhe system based on common mistakes in Open Source systems, then tey are more likly to detect mistakes in Open Source code, as open source and proprietry code development techniques differ but may produce differant kinds of bugs.
...and if so, did it report any bugs?
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
A guess is that this is because much of emacs' functionality is implemented in elisp code, which is not part of the core program and so not included in the source line count, whereas most of vim is implemented directly.
I find it curious that the open source projects most touted for the quality of their code (OpenBSD and offspring OpenSSH) are one of the few large-ish open source projects that *are not* tested and compared ...
Having trouble figuring out your requirements and identifying saftey critical systems?
Having trouble identifying what your actually supposed to be doing?
Then I suggest you give the Norm Internationale - IEC 61058, a read! It's a favourite of mine...
In general most of the engineers throw their hands up in the air and scream blue murder when I quote this stuff at them... (We're a Medical Device company). But I f you want to get a good idea of whats required..
Cheers,
P.S I'd suggest a quick peruse of part 3 Appendix B
- No Dynamic Objects, Limited use of pointers, limited use of recursion....
It'l' take you a wee while to wade through it... (If you can actually lay your hands on a copy)
This is the usual completely meaningless accounting, with a myriad of methodological flaws. You cannot make a general statement about bugginess of open-source vs. closed-source code. There are just too many variables to conduct a statistically meaningful study. The reason why open source code is better bug-wise is very simple: the user of the code can fix the bugs. As a developer, I hate to depend on closed-source code. Why? Because *every* moderately large code, open or closed, has bugs, and they invariably show up at some point. I much rather go fix the issue myself that wait for some random amount of time (including forever) for someone else to fix the bug for me.
https://bugzilla.mozilla.org/show_bug.cgi?id=13604 9
1 9
A bug in Netscape/Mozilla/Thunderbird that was first reported ELEVEN YEARS AGO.
https://bugzilla.mozilla.org/show_bug.cgi?id=2045
A case where a programmer with a engo trip can veto a solution to what the user's (customers) demand.
The whole "anyone can checkout OSS code and fix it" is a crock -- only those with 1) the prorgramming skills to do anything 2) the urge to do anything to begin with and 3) no reason to leave their parent's basement.
I personally prefer software written and controlled by those with a profit motive and consequences for failure.
How freaking long did it take ANY form of *nix to support WPA (or wireless for that matter) YEARS after it was available out of the box for commercial OSes?
Heck I use OSS only because I have the patience and skills to mitigate the shortcomings, not because I can't get laid and have too much free time.
If this was MS, then it would be comparing OS to OS. Instead, they are saying that it is software that is running nukes, medical, aviation, etc. All things that have undergone certification. I would guess that Green Hills is doing a MS style FUD job though. Almost certainly they picked closed source that they knew has been around for eons vs. OSS that is relatively new.
I prefer the "u" in honour as it seems to be missing these days.
to Jira (or Bugzilla or whatever) on the Open Source projects and it could submit the bug reports automatically and after only ... three days to three years all those bugs would get fixed.
Probably true, but there is bound to be plenty of narrow-interest proprietary software out there (the stuff Coverity probably knows and little cares about) that is bug-infested crap and not worth $10, much less several thousands of dollars. Office, despite its bloat, ought to be largely bug-free since so many people everywhere use it and finance development for it (you'd think!), but I've noticed obvious bugs. Plenty of mal-features, but a lot of OSS is like that too.
Here's an interesting question - what's so bad about taking "only" 4 out of the 15 top spots when OSS projects were only about 1/3 of the total pool being rated? ("50 of the mos popular", "over 100 companies", "more than 150 open-source and proprietary software applications that we have analyzed") Let's hope the 11 proprietary ones in that elite group were the ones made to control critical systems like airplanes and nuclear reactors, eh? Hate to think that such weren't as good as the best of OSS!
For sure, MS Windows and MS Office are closed source so "we" cannot analyse them.
However, Debian (which I am running here) and Open Office are not closed source.
Simple logic will tell us that someone or someones in MS are tasked with keeping a close eye on open source and how it compares with MS own operations, hell, this is MS core business strategy, just ask Jobs...
MS aren't likely to tell anyone else these results, but, we can maybe infer them from MS actions.
eg
"Vista sucks" ---> MS ain't worried about Debian / KDE
or
"vista allows you to pause and resume multiple file transfers, and don't abort the whole list on one error" ---> MS is very worried about *nix in general coming to the desktop
http://slashdot.org/~GuyFawkes/journal
Proprietary software tends to have requirements of what bugs they MUST fix. In this case once the product goes 'live/GA' users can request bugs to be fixed AFTER 'live/GA'. Not just security issues either. We usually do a few maintainence fixes, after GA because of what our users require us to do.
I also think that there is a QA issue in open source. Open source does not usually have a QA department, it has a beta period for people to find bugs. Proprietary software almost ALWAYS has a QA department or QA period where stuff is tested over and over for bugs and then these bugs are fixed.
Maybe open source needs OPEN QA!
Only 'flamers' flame!
Does slashdot hate my posts?
First off, yes, the article is fairly worthless for countless reasons, but that's not what I want to talk about.
This is not to say Coverity is worthless as well. It is not doing things that a compiler can simply do and it does find errors that you *might* stumble across in debugging, or you might not (my place of work recently evaluated it, and I'll venture a guess that it's one of the proprietary folks included in the study). It tries to execute various possible branches of code to see if you might be working with pointers which might be null or if you might have a buffer overrun. Yes, some of the "bugs" it reports are false alarms, but it will also draw your attention to areas where there might be legitimate problems (and it will just as easily not catch bugs). Complaints in here that it's worthless as a compiler/debugger could do it and Windows-programming-practices-centric (I honestly have no idea what that means) are pretty baseless, and I'm guessing, coming from people who have never even used the product. Am I advocating using this product? Not necessarily, but these kinds of tools *do* have a place in any serious development shops QA process.
... department of homeland security is funded by the government. Government is funded by tax dollars. Tax money comes from big corporations like Microsoft and Google. It's obvious that his is a biased report. No way open source software has ever had any flas ever.
This doesn't make any sense. It assumes that they found all the bugs in close source software or they trust the proprietary vendor to disclose all defects.
Yeah right. Like a vendor making money selling a product is going to be truthful when telling just how crappy his product really is. ROFLMAO
The race isn't always to the swift... but that's the way to bet!
I just ran across an interesting blog entry by Federico Mena Quintero that linked to two interesting papers : When should a test be automated? and Classic testing mistakes.
Automated tests are often more expensive to write than manual tests, so when should you do automated tests versus manual ones? The answer is not always automated tests. Also typical unit testing is only part of what kind of testing needs to happen.
Using Ring Buffer Logging to Help Find Bugs was also another good link that Federico had in another entry.
OK, what is the definition of quality? Most people like to say "free of bugs", but that isn't a true indicator of quality. My working definition is that a high quality product meets or exceeds the client's expectations. Now with OSS, my expectations aren't really that high. I mean, if it crashes, or the UI is kind of clunky, or it is hard to install... I just kind of expect there to be some of that. But if I pay for something, I expect it to be of higher quality in these regards. And if a client hires someone to create software for them, it had better meet their requirements. Of course, that is all pretty much a game nowadays, where promises are made, requirements are poor, and it all just turns into a used car sale. Roll out the dog and pony show, let the smooth talkers "work" the clients to explain why they didn't get what they asked for...
My beliefs do not require that you agree with them.
not only that, but it has 15 lines of code and they fixed 23 bugs in it. that's some buggy code!
The other interesting thing I found:
Gnome: 31,000 lines
KDE: 4,500,000 lines
I always thought KDE was bloated (commence flame wars)
being vague is almost as cool as doing that other thing...
openoffice code is a mess because i is very old... remember that there was a staroffice in DOS time, the same code was update over and over and over before release to the opensource community...
since then many people try to clean it, but its hard and risky to clean a such big app
most projects have a coding style that everyone should follow, and many force you to comply if they want their code to be accepted
Higuita
Coverity, of course, knows that reports like this will be written up in exactly the way this summary was, clearly associating their company with the idea of enumerating the bugs in a piece of source code. While not illegal, this type of marketing is of course deceptive; while published papers describe the type of defects (or non-defects) actually detected, the overwhelming volume of commentary will reflect the broader, and incorrect, view that Coverity == bug-finder. It would be just as meaningful (which is to say, not very) to publish the number of lint warnings or missed opportunities to qualify pointer arguments with the const keyword, and neither would require an expensive piece of overhyped software.
Just Say No to Coverity's marketing gimmicks.
Now that is what I call compact code! I don't know of any other project that's pulled off so much functionality in so few lines of code.
But the bug/lines of code ratio of 23/15 is a little high for my tastes. Perhaps it's time to look at Visual SourceSafe.
The codebase is very old, contains a bunch of legacy stuff nobody really understands, as the codebase has passed hands from a German company to Sun to the OpenOffice.org foundation. It's also picked up a layer of java along the way (for whatever reason).
It's too bad because it actually works kinda okay, but it's a real effort to get your hands dirty with.
Blender is also like that... it seems when a codebase has 'gotten around' it tends to pick up the bad habits of all the hands its been through.
MySQL is a bad state because it's really only developed by MySQL AB -- no one else is contributing to it so they have no reason to make it any more maintainable than it is. PostgreSQL, on the other hand, had the luxury of being the fruit of some academic research projects and was rewritten once or twice, so it's a little more maintainable.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Proprietary Software Sucks.
May it be because they are the government's pawn, and government is now run by cartel-loving people, and they want to push digital rights, patents, restrictions on the internet, more secretive, closed code so that they can better 'protect' their supporting cartels' digital 'rights' ?
Read radical news here
Quality in closed source projects is declining due to predating open-source projects. Money is not being made in closed-source projects anymore so wages are declining, people are getting fired and ultimatelly quality suffers. Nearly all healhy closed-source firms are threatened or in the phase of being liquidated by predating open source projects. There is no point of starting a new closed-source proprietary project anymore, even if you start one then there will be a new o.s. project soon to wreck it. The more successfull you are then the possibility that some german student will start getting busy on sourceforge.org is even higher.
...and there are code reviews. One of the customers I work at hired EDS to do a code review (not one of my apps). EDS sent five people...none of them actual programmers. They spent the first two days in a conference room figuring out how to approach the job. 80 man-hours later the EDS team reaches the startling conclusion that they're going to need programmer support for the actual code review.
And people say Big 5 consultants aren't worth the money. Pshaah.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
so did I. Now we have proof!!!
Hell, open-source is a nice idea, but it DEFINITELY lacks QA EVERYWHERE! Even KDE -- the soooo user-friendly desktop -- does not care for proper and reliable operation. How often did I need to go into ~/.kde and fix things manually because the calendar did not work any more etc.etc.etc....
In the end, it is all a matter of work. Much money means much work (usually). So there you go. I just hope that the independence of open-source software leads to a rather prefect end-product (somewhen).
Do I look studpid?
I will have to check in the mirror again as I might be living in a different universe where my Linux boxes always have to be remotely rebooted on a weekly basis and my Windows machines have 400 days worth of uptime.
-Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
Proof that what, GNOME has no browser?
--
WHO ATE MY BREAKFAST PANTS?
It's not a bug, it's a feature! :P
Staring at a white background [on a computer screen] while you read is like staring at a light bulb — Maddox
"The best of closed-source software is found in so-called mission-critical applications: things like jet engines, nuclear power plants, telephone systems, medical devices. These are things that simply can't fail, or people may die. "
This type of quality (CMM level 5, I might hope) doesn't happen overnight. It can only be reached by subjecting the whole development process to rigorous methods. Also, someone is hired and appointed as the responsible person for the software in question. Not just any software company is capable of reaching CMM level 5, even if it was Mr Gates himself who founded it (I'm not even sure MS writes software at CMM 2). We're talking about best-of-the-breed software companies here.
In contrast, no warranty is given on software that is released under the GPL. It is in the nature of the GPL that anyone can contribute, but that if you wish to use it, you do so at your own risk.
Which makes me draw the conclusion: Duh- *Of course* open source software has more bugs than mission critical nuclear power plant / jet engine / hospital equipment software.
Whoever said "The 2002 NIST study estimated that $22 billion of the cost of buggy software would vanish with better testing" doesn't realize that there is more to software quality than testing. Software quality only happens it is guaranteed in every step of the development process, from specification to rollout.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
What the article really tell is that although the quality of OS application is on the average higher than Closed Sources applications, there exist a class of OS application that has higher quality, notably airplane software.
Well since there is a very limited number of airplane manufacturer (for a large number of very bad reasons), and since it is quite difficult to have a handy Aibus 380 or Boing Skyliner around for testing, it is clearly not a very "OS Friendly" field of development.
Actually what is really compared is:
Open Source applications
Closed Source applications
and a class of very specific Bespocken applications.
Now the methodology is not very scientific, and therefore the results quite disputable.
But what it might show if anything, is that if there would be more openess in the airplane industry and more cooperation and information sharing, planes would probably be cheaper and safer.
So to make a slightly too fast shortcut: patents are bad not only in the software industry but in practice, everywhere.
damn right and why should it? a browser should NOT be a native part of a DE. that is a major sign of bloat.
Just a guess but is it multiplication? Like golf the lowest score gets the prize? (ie: the lowest score = x 0). Hence they 2nd numbering system starts @ 0. HTD
Ahh, bingo. I believe that's it. I know of some groups that just need one score to focus on, and will simply multiply priority by severity. But if a bug is truly pri 0 (blocking all other tasks), you want that to be worked on no matter what its severity is.
-- jchenx
First, the author is not against open source. In fact, Coverity is helping projects like FreeBSD by providing tools for developers.
Second, he is not comparing apples to oranges. He is using a metric, bug-rate in a wide sample. Then, he finds, interestingly enough, that proprietary software is very much scattered, but the ones on the top are 5 X less buggy than open source. This begs the question: why? This is what people should be discussing. Not saying he works for Microsoft and all that childish bullshit...
The *real* question is: what are those methods people developing mission-critical software use that open source hackers do not? My hunch is: formal methods, safe languages.
For instance, Ocaml http://www.astree.ens.fr/ was used in a sofware verification system for the Airbus A340 fly-by-wire system. Haskell is used by Galois Connection http://www.galois.com/ to develop secure protocols for the DoD. And there are many other examples, just look at the clients of vendors of Erlang (well known), Common Lisp, and Eiffel.
As long as the open source community sticks to C (and C++), we're all going to remain in this ridiculous situation that we are in today. In this day and age you can use a fast compiler for safer languages like ML, Lisp or Eiffel, but people insist programming like we're in the 70s.
Main difference between the BSD license and the GPL license: one is from California and the other is from Massachusetts