Slashdot Mirror

Just thought of this: Make it stipulation of GPL that if you publically report bugs or bug counts in GPL software, that you must also produce a detailed account of how to reproduce the bug, and you must provide that report to the maintainer of the current source (who you got it from, or the root source as listed in the code). Possibly a two-week window between notification (and acknowledgement) and publication.

Not all bugs are easily reproducible - and not all bugs are found by tripping over them. Consider, for example, bugs found by various of the warnings enabled by GCC's -W options. I.e., you get reports saying "this code path has these problems", not reports saying "this code path blew up when I did XXX".

I just looked at an old report from Coverity on one of the free-software projects with which I'm involved - one of the problems it found was in a chunk of code

if (cfg->in)use) {
report an error;
return;
}
if (cfg != NULL) {
process what it points to;
} else {
report an error and clean up;
return;
}

where it quite appropriately pointed out that we were checking whether cfg was null after dereferencing it rather than before dereferencing it. We subsequently fixed that problem.

It might be possible to construct a scenario where the application would crash due to that bug - or it might not; that bug is in "framework" code, and if the code using that framework code doesn't happen to pass an argument that would cause cfg to be null, there won't be a crash, but some code in the future might pass such an argument (which might be an argument that comes from user input, so it's not as if passing such an argument is a bug - perhaps the code using the framework code is expecting that code to tell the user of the error).

Even if it's possible to construct such a scenario, the software that found the problem doesn't have a deep enough understanding of the code to say "hey, if you open up the app on a file like with this in it and select this menu item and type this into the dialog box that pops up and then click 'OK', it'll crash", so it's not as if the software that's reporting this problem (non-publicly - to see the reports on an app, you have to be a "member" of the project whose code is being scanned, and sign up for an account) can give "a detailed account of how to reproduce the bug".

I checked out the Coverity website and saw on the list of projects the aalib ASCII art library which according to the history hasn't been updated for something like 7 years.

Damn we better protect ourselves from Terrists hiding their WMD's in ASCI art

http://scan.coverity.com/

Look at http://scan.coverity.com/. This is a great project to improve the stability of open source projects by looking for all sorts of coding errors that can be very hard to spot manually. It may not be true that with enough eyeballs, all bugs are shallow. But is it very clear that the Coverity eyeballs are exceptionally good at exposing lots of bug. It is all clear that the open source developers are excellent at fixing these bugs. If KDE can get 4.0 out the door and drive their Coverity defects closer to zero, then I think that we will see a very robust, efficient KDE 4.0 by year's end. The number of defects in my Linux/X.org/KDE 'desktop stack' has dramatically dropped, at least as measured by Coverity. Sorry for sounding like an advertisement. I know that there are other ways to find defects, but I am just so impressed with how open source developers have closed thousands of coding errors that have been identified by these automated code audits. This is the sorts of constant improvement that quietly leads to better stability and security.

> i am the original author of a quite succesful program , which is now OS (since last year), but I am not more proud of it

Open source is not magic. If you write a closed source software and then release it as open source, it will be the same shit it was before. The reason why open source software generally is better is because of the 3 rules from which you have t sacrifice at least one when making software:
- Time
- Money
- Quality

Because open source is often done on free time, without salary and deadlines. There is no need to worry about time or money. So all the effort can be put on the quality.

But of course again, there is no magic. If you are a bad programmer, writing open source program doesn't make the result better. But if you are a good programmer, time is usually the only thing that is limiting the quality of your programs.

But this doesn't make any difference between closed source program on free time and open source program on free time. But open source has another benefit. For example I have actually red the whole source code of one program and I did find some bugs from there, which might have never been spotted without someone reading the code. Or look this project, which has revealed several bugs from open source projects: http://scan.coverity.com/

Then open source has patches. There is one product which was really good and I enjoyed using it, but it had bugs I didn't like. It wasn't open source and it had only one developer, which then stopped developing it and the project was pretty much dead after that. No fixes, no development. If it had been open source, I would have fixed the bugs for free on my free time. But because it wasn't I couldn't do this. This is the main reason why I prefer open source over anything else. If it is broken and no-one else cares about fixing it, at least I have an option of doing it myself.

Check out http://www.coverity.com/. Not free, but they demonstrate their product on a few big open source projects like Mozilla.

Try a static analysis tool like Coverity... http://scan.coverity.com/

I work on a commercial static analysis tool called CodeSonar. It costs money, but we do offer free trials.

Our major competitors in this space are Coverity and Klocwork.

All three tools can (to some extent) infer how a program will behave at run-time, so they find more subtle bugs than tools that just look for suspicious patterns in your code.

I strongly suggest you look at coverity.

They have excellent checks as well as the best framework for creating custom tests that I have ever come across.

NOTE: I am not affiliated with coverity, just a very satisfied user.

But FindBugs does not cover the C/C++ codebase...

C/C++ checkers:
  http://www.coverity.com/ (commercial)
  http://www.dwheeler.com/flawfinder/ (OSS)

Why oh why are we in 2007 seeing code like this in security apps? input verification in the classical C way with pointer arithmetic on strings.
(and no, the error isn't there, it's just the first thing I came across in the snort source)
Why are they even using C? Suprise, they make exploitable buffer overflow attacks! And they still have one verified, non-fixed issue detected by coverity, plus 33 "uninspected and pending" according to coverity's scan.


int CheckRule(char *str)
{
        int len;
        int got_paren = 0;
        int got_semi = 0;
        char *index;

        len = strlen(str);

        index = str + len - 1; /* go to the end of the string */

        while((isspace((int)*index)))
        {
                if(index > str)
                        index--;
                else
                        return 0;
        } /* the last non-whitspace character should be a ')' */
        if(*index == ')')
        {
                got_paren = 1;
                index--;
        }

        while((isspace((int)*index)))
        {
                if(index > str)
                        index--;
                else
                        return 0;
        } /* the next to last char should be a semicolon */
        if(*index == ';') ...

I thought it was called Coverity (http://www.coverity.com/html/prod_prevent.html)

I have also noticed a huge improvement in KDE's stability. With the recent Coverity scans, we see that KDE is on and off the 0 defect list. KDE seems to be the most active projects on the Coverity scan, I notice more more week to week change in KDE than in any other project. In 3.4 million lines of code, Coverity has uncovered over 1,200 bugs. All bugs have been identified and all but 10 have been closed. KDE has been on the zero defect list, but there is new development going on so new bugs do appear. Not only is KDE gaining the features you mention, but they are doing it while cleaning up the code base. KDE development seems to have a great deal of momentum, especially in Europe.

That stuff sounds kinda neat, but after reading the SLAM summary paper, it looks pretty similar to the Stanford Checker extensions to gcc, which eventually turned into Coverity. Seems like there are some differences: I don't think that the Checker has to do the rewrite to the Boolean language thing, but the whole idea of rules like "match lock with unlock" seems analagous. Recent Linux kernels have featured some reasonably neat gcc stuff that warns if you don't check return codes and the like.

The Vulcan thing sounds pretty cool, though. Wonder what the overhead is?

Or instead of guessing, you could read the article. They didn't keep the bugs secret, they showed them to the open source teams. They have helped reduce the bugs. You can see what they have done with some projects at http://scan.coverity.com/ and you can furthermore read the report by going to the link on the left hand side of the page. Registration is required, but you will get a lot of other reports as well.

they tested it by using a program that systemattically scans code for common errors.

I don't know if the closed source statistics are online somewhere, but these are the open source statistics.
http://scan.coverity.com/

and if you ask me the "Defect Reports / KLOC" is pretty low, and such software would normally be considered "good" software.

My understanding also is that configuration management comes into play, in that the DoD is unsure that the open source software has been developed with appropriate CM. Remember that CM is more than a simple CVS type system: it is also a system where the changes are reviewed before they are incorporated for impact and for security.

There are also assurance issues, as was noted above. There are some attempts to assess the quality of open source software; I'm aware the DHS has an initiative with Coverity; see http://scan.coverity.com/.

Of course they welcome this. Just look at the results page for the Coverty scans and see how many defects have been fixed in major open source projects.
http://scan.coverity.com/

http://www.coverity.com/news/nf_news_06_27_05_stor y_9.html

Sorry about the link.
Corrected link. Unfortunately there are only 2 examples since there are trade secrets involved with bug reports.
This might look like a slashvertisement, but I didn't submit the original story (which does pick up on a press release)

According to the blog post that announced it, Coverity were scanning 3.9 million lines of KDE code. Although the reports are a bit wonky at the moment, I'm sure Apache has more than 9 lines of code!

Coverity's own site shows how many defects each product has fixed. the number of outstanding defects on AMANDA is now zero. zdnet reported the fixes back in April.

Those that follow amanda-hackers will know that there was less than a week between when coverity released the report on March 6th and it was announced that all bugs were fixed in AMANDA on March 12th.

if you look at the coverity site ( http://scan.coverity.com/ ) you will see that there are already multiple projects who have brought there bugs down to zero. samba being on of the earliest.

Fixing security issues and improving coding style to avoid security issues has been a very bog concern in the Etehreal/Wireshark project over that last couple of years. for instance, unsafe string operations are now no longer tolerated in the code (e.g. the strxxx functions). A couple of people have run various source code analyzers against Ethereal/Wireshark, and each time, the developers where quick in fixing the issues found. Even for coverty, the statistics look very good compared to most other open source projects:
http://scan.coverity.com/
Also, given the rather quick release cycle, the fixes quickly make it into releases. So all in all, Etehreal/Wireshark is a very good project regarding security.

Marcel

Coverity has a very successful business line of this http://coverity.com/ and is spun off from research at Stanford that I believe has found the most bugs in linux.

For one thing, the closed-source nature of the whole anti-malware market is a fertile breeding ground for exactly this sort of problem.

CODE is a fertile breeding ground for this sort of problem. Closed source, open source, it doesn't matter. Check out the Coverity page on their work with open source projects. Notice all the defects they've found in open source.

Also, I GUARANTEE you that Symantec and every other security software firm has extensive review processes to try to prevent precisely this sort of thing.

I went to the Coverity website and got the following when I clicked on their Product tab:

404
Not Found

The requested URL /products/nf_index.html was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

There are products available, memprof, Coverity nessus which can be used to find and fix common forms of previous bugs. These fix everything from repeating previous security flaws (I note a previously unknown DoS flaw I found in Asterisk's skinny codec ages ago which emulated a bug in cisco call manager exactly, which I found with Nessus), to bad programming, or programming mistakes (Coverity), to memory leaks (memprof). These types of bugs are unacceptable, there are tools out there to detect them DURING THE PRODUCT DEVELOPMENT CYCLE. I am not saying that you can fix every bug every time, but 5 digit numbers of open bug reports are unacceptable.

If the Coverity bug report for FOSS software is true, NetBSD is amazingly well done and bug-free.http://scan.coverity.com/

They have actually removed existing features also to make it not-bloated. And about polishing, what do you think places is? It is an attempt to improve existing feature to make it better. They are also planning to improve the search-engine for user to be able to edit the search engine list with more easily way, isn't that polishing? They also fix bugs all the time, including memory leaks: http://www.squarefree.com/2006/02/04/memory-leak-p rogress/

In Firefox 3.0 they are also planning to switch to Cairo, which is an attempt to increase speed. They are also fixing bugs that can potentially cause crashes or other problems, as you can see from this list: http://scan.coverity.com/

FreeBSD: 626
Linux: 628

X.org: 713
OpenOffice.org: 1024

Why don't the powers that be look at this tool?

http://www.coverity.com/
http://scan.coverity.com/

Why don't the powers that be look at this tool?

http://www.coverity.com/
http://scan.coverity.com/

Quick response from Amanda development community. Amanda moved from bottom to the top of list!!!! 108 defects addressed within a week. I think this shows the strength of the open source software development methodology. Zero defects found means it will be in top of the list.
See http://scan.coverity.com/ for latest Coverity testing results.
Paddy
Amanda developer
http://wiki.zmanda.com/ for Amanda information

oops

http://scan.coverity.com/

If you read the actual results of the study, you will see that it doesn't even mention MySQL.

http://scan.coverity.com/

You will also notice that PERL and Python are far less defective.

I really hate the way mainstream tech news/fud sites spin these buzzword-compliant mis-informing articles.

I assume he means the baseline of 0.434 bugs/1000 lines, and that if they removed PHP from the LAMP stack, that average bug count would go down even further.

Spot on, as you can see on scan.coverity.com:

• PHP features 205 defects for 431,327locs, or 0.475 defects/kloc
• Perl has 91 defects for 431,327locs, or 0.19 defects/kloc
• Python is very slightly lower than perl (but with a noticeably smaller codebase) at 49 defects for 259,908locs or 0.189 defects/kloc
• Apache-httpd features 32 defects in 127,817 locs, or 0.25 defect/klock

I noticed that on Coverity's demo page, there's no mention of MySQL. If they *do* scan MySQL for bugs, why not have it on the front page?

I know that there is a Stamford University, and everyone always jokes that it's for people who want to pretend they went to Stanford, but, this just makes things really confusing. The Register article says Coverity used a verifier from Stamford University, when really the program came from Stanford. In fact, AFAIK, UCONN-Stamford doesn't even have a CS department.

I read about this a couple days ago and spent some time on the company's site looking for an explanation of what they are doing that is so new. The answer I came up with is "Nothing". There is no information on their websites about specifc products or services. Looks like another snake-oil security startup.

There are other companies and even some academic groups (PROTOS from the University of Oulu, to name one) who have been doing real things in this area for years. There are also companies that take a source-code centric approach.

For several years now, there have been products that check for whole classes of vulnerabilities in applications. Such approaches are not limited to just known vulnerabilities in existing apps -- they check for common programming or configuration errors in custom applications as well. They are making it sound like checking for these things before systems go into production is a new concept. That's the whole point of security auditing.

Slashdot:

I work at Coverity (a static source code analysis company), I personally have been trying to get in contact with someone at Blizzard to demo our static source code tool. Maybe this forum would be an appropriate place to at tell them about our technology.

Coverity's breakthrough product, Coverity Prevent, automatically detects critical software defects that hamper the development process and frustrate your important customers. In addition, Coverity Prevent pinpoints security vulnerabilities in source code to reduce the risk of malicious attack. In a matter of hours, our customers such as Juniper Networks, Veritas Software, palmOne, Synopsys, and nVidia found hundreds of critical, crash-causing bugs and security vulnerabilities at compile time with no manual testing or changes to the code, build, or existing processes. Our technology was recently featured in Forbes:
http://www.forbes.com/home/free_forbes/2005/0314/0 60.html

Coverity's technology came out of years of research at the Stanford Computer Science Department, where it was used to automatically detect thousands of bugs in open source projects such as Linux and Apache. What resulted was a product that customers find easy to use and that scales to large, complex software projects.

Coverity Prevent is the only solution that can achieve 100% code and path coverage, detecting costly defects and security vulnerabilities in parts of the code that are difficult to test. Furthermore, using Coverity Extend, our customers can develop custom checks to uncover defects that are unique to their product or defects that have been triggered in the field.

We would like to be used by every game developer in the industry but are having some difficulty in contacting the appropriate people. We have almost every other industry except gaming using our tool.

http://www.coverity.com/customers/

For more information see
http://www.coverity.com/
http://linuxbugs.coverity.com/

Bugs found in the linux kernel:

http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit% 2Ftorvalds%2Flinux-2.6.git&a=search&h=HEAD&s=cover ity

Slashdot:

I work at Coverity (a static source code analysis company), I personally have been trying to get in contact with someone at Blizzard to demo our static source code tool. Maybe this forum would be an appropriate place to at tell them about our technology.

Coverity's breakthrough product, Coverity Prevent, automatically detects critical software defects that hamper the development process and frustrate your important customers. In addition, Coverity Prevent pinpoints security vulnerabilities in source code to reduce the risk of malicious attack. In a matter of hours, our customers such as Juniper Networks, Veritas Software, palmOne, Synopsys, and nVidia found hundreds of critical, crash-causing bugs and security vulnerabilities at compile time with no manual testing or changes to the code, build, or existing processes. Our technology was recently featured in Forbes:
http://www.forbes.com/home/free_forbes/2005/0314/0 60.html

Coverity's technology came out of years of research at the Stanford Computer Science Department, where it was used to automatically detect thousands of bugs in open source projects such as Linux and Apache. What resulted was a product that customers find easy to use and that scales to large, complex software projects.

Coverity Prevent is the only solution that can achieve 100% code and path coverage, detecting costly defects and security vulnerabilities in parts of the code that are difficult to test. Furthermore, using Coverity Extend, our customers can develop custom checks to uncover defects that are unique to their product or defects that have been triggered in the field.

We would like to be used by every game developer in the industry but are having some difficulty in contacting the appropriate people. We have almost every other industry except gaming using our tool.

http://www.coverity.com/customers/

For more information see
http://www.coverity.com/
http://linuxbugs.coverity.com/

Bugs found in the linux kernel:

http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit% 2Ftorvalds%2Flinux-2.6.git&a=search&h=HEAD&s=cover ity

Slashdot:

I work at Coverity (a static source code analysis company), I personally have been trying to get in contact with someone at Blizzard to demo our static source code tool. Maybe this forum would be an appropriate place to at tell them about our technology.

Coverity's breakthrough product, Coverity Prevent, automatically detects critical software defects that hamper the development process and frustrate your important customers. In addition, Coverity Prevent pinpoints security vulnerabilities in source code to reduce the risk of malicious attack. In a matter of hours, our customers such as Juniper Networks, Veritas Software, palmOne, Synopsys, and nVidia found hundreds of critical, crash-causing bugs and security vulnerabilities at compile time with no manual testing or changes to the code, build, or existing processes. Our technology was recently featured in Forbes:
http://www.forbes.com/home/free_forbes/2005/0314/0 60.html

Coverity's technology came out of years of research at the Stanford Computer Science Department, where it was used to automatically detect thousands of bugs in open source projects such as Linux and Apache. What resulted was a product that customers find easy to use and that scales to large, complex software projects.

Coverity Prevent is the only solution that can achieve 100% code and path coverage, detecting costly defects and security vulnerabilities in parts of the code that are difficult to test. Furthermore, using Coverity Extend, our customers can develop custom checks to uncover defects that are unique to their product or defects that have been triggered in the field.

We would like to be used by every game developer in the industry but are having some difficulty in contacting the appropriate people. We have almost every other industry except gaming using our tool.

http://www.coverity.com/customers/

For more information see
http://www.coverity.com/
http://linuxbugs.coverity.com/

Bugs found in the linux kernel:

http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit% 2Ftorvalds%2Flinux-2.6.git&a=search&h=HEAD&s=cover ity

Are they proposing building a 'i think know what you meant' version on lint or something?
Not quite, but we're working on it :)
http://linuxbugs.coverity.com/

The Linux kernel has not been quirky for me.Perhaps you should file a bug report, if you haven't already. FreeBSD 5.3 supposedly has stability problems of its own.

http://www.newsforge.com/article.pl?sid=04/12/14/1 518217

Also according to Coverity source code analysis tools, Linux has less bugs detected per lines of code than FreeBSD. Ofcourse, this cannot detect every kind of bug, but it cannot be argued by a logical person that because FreeBSD has more bugs per lines of code, it is actually more stable.

http://www.coverity.com/news/news_06_27_05_story_9 .html

http://linuxbugs.coverity.com/linuxbugs.htm
And we all know the changelogs don't have fixes for all of those at once, so there are still many left in 2.6.11.

Linux is buggier than you think. Factually (this is not trolling, shut the fuck up, the AC who keeps replying to me), NetBSD's bugs found with the same software: http://mail-index.netbsd.org/tech-kern/2005/02/23/ 0037.html

No mention of Free or Open or DFly, but they should be somewhere in between, from the bug reports I've seen around.

I was going to give you a link to where the bug database was but it seems to have turned into a promotional page - Coverity Linux "bug database" page. Normally that link goes to a bugzilla like database showing you a list of bugs, what type of error was found and a highlighted code snippet. All very nifty.

Here's kernel hacker davej talking about coverity bugs. So they do let the kernel folks know about the bugs they have found.

However this hits upon something I don't like about this press release. I wonder whether the headline should be "Tool finds less bugs in Linux than rival OSes when previous bugs found by the tool are fixed in Linux". Assuming fixes for all problems that means only new and changed code gives a chance for these errors to occur (which is a subset of the entire code).

Were all codebases that were compared given coverity bug databases and a chance to fix the reported bugs too and over what time scale? If the Linux kernel code gets 6 monthly reports and say OpenBSD kernel only gets reports one every two years and you test towards the end of those two years is that fair test? What does it say other than our tool finds less bugs in code that is regularly checked by previous versions of our tool? Is that in itself a good justification?

From the ACM paper "bugs as deviant behavior", we find
"We demonstrate that the approach works well on complex, real code by using it to find hundreds of errors in the Linux and OpenBSD operating systems. Many of our bugs have resulted in kernel patches."
This paper appeared in 2001 and represents the work of several researchers, some of whom were PhD students in CS. Some of them went on to form a private company (Coverity ) which plans to offer (for pay, of course) the use of their software to companies. It would be possible for Sun to use this software to look for errors in Solaris. However, the fact that the source codes for Linux and BSD were freely available allowed these researchers to test their prototype code and to submit bug reports years before Sun or MS had the option of using the tools developed by this group. Who knows what other groups of CS students or faculty are doing the same type of exercise right now; they may develop tools which will help private companies someday while using OSS to test their code now. (By the way, you might ask why such a group might help OSS projects. The answer is illustrated in the quote above; this company can already claim credit for finding bugs for which kernel patches have been released. You cannot buy better PR than this.)

The "Linux" code base just includes the kernel.

No, they included more than the kernel. Only 1% of the bugs they found were in the kernel. You can see a bit of a breakdown of what the bugs were and where they were found here.

It is indeed possible to prove some programs correct (cf the famous Knuth quote, Beware of bugs in the above code; I have only proved it correct, not tried it.). However it is usually difficult, requiring annotations or other manual intervention to specify invariants. Other, more automatic kinds of checking are possible, but they're always incomplete; first of all, because it's impossible to check any kind of program property in general, and second, to achieve any kind of reasonable performance.
I actually work for a company that sells such a checker (sorry, not FOSS, got to eat). Found hundreds of bugs in the linux kernel alone.
As for proof-carrying code, it's code that carries around a proof of what it computes; the proof can be checked at the end of the program. Problem is that there is no guarantee that the proof will actually be correct, nor that the program will terminate. I almost published a paper on some sort of language where you could only carry around correct proof, so if you could get to the end of your program, you did the right thing (partial correctness). However the type of proofs you could have was very limited.