Do Static Source Code Analysis Tools Really Work?

jlunavtgrad writes "I recently attended an embedded engineering conference and was surprised at how many vendors were selling tools to analyze source code and scan for bugs, without ever running the code. These static software analysis tools claim they can catch NULL pointer dereferences, buffer overflow vulnerabilities, race conditions and memory leaks. Ive heard of Lint and its limitations, but it seems that this newer generation of tools could change the face of software development. Or, could this be just another trend? Has anyone in the Slashdot community used similar tools on their code? What kind of changes did the tools bring about in your testing cycle? And most importantly, did the results justify the expense?"

Comments

In Short, Yes

[Nerdfest]

They're not perfect, and won't catch everything, but they do work. Combined with unit testing, you can get a very low bug rate. Many of these (for Java, at least) are open source, so the expense in negligible.

Re:In Short, Yes

[Anonymous Coward]

The proper answer would be: No. A fully working static code analyzer would be like solving the Halting Problem, which has been proven to be impossible. Essentially you can just try to catch as many potential problems as you can, but you can never catch all.

Re:In Short, Yes

[tritonman]

yea they work, I wouldn't spend a lot of money on them though, a decent compiler will give you info for stuff like buffer overflows. Most of the bugs will need to be caught at runtime, so if you are on a tight budget, definitely skip the static tools for the more useful ones.

Re:In Short, Yes

[crusty_yet_benign]

In my experience developing win/mac x-platform apps, Purify (Win), Instruments (OSX) and BoundsChecker (Win) have all been useful. They find obvious stuff, that might have led to other issues. Recommended.

Re:In Short, Yes

[Goaway]

You don't need to be perfect to be useful.

Re:In Short, Yes

[Simon80]

There's also valgrind, for Linux users, and mudflap, for gcc users. I haven't tried mudflap yet, but valgrind is a very good runtime memory checker, and mudflap claims to do similar things.

Re:In Short, Yes

[FBSoftware]

Yes, I use the formal methods based SPARK tools (www.sparkada.com) for Ada software. In my experience, the Examiner (static analyzer) is always right (> 99.44% of the time) when it reports a problem or potential for runtime exception. Even without SPARK, the Ada language requires that the compiler itself accomplish quite a bit of static analysis. Using Ada, its less likely you will need third-party static analysis tool - just use a good compiler like GNAT.

Re:In Short, Yes

[Entrope]

My group at work recently bought one of these. They catch a lot of things that compilers don't -- for example, code like this:

int array[4], count, ii;

scanf("%d", &count);
for (ii = 0; ii < count; ++ii)
{
scanf("%d", &array[ii]);
}

.. where invalid input causes arbitrarily bad behavior. They also tend to be better at inter-procedural analysis than compilers, so they can warn you that you're passing a short literal string to a function that will memcpy() from the region after that string. They do have a lot of false positives, but what escapes from compilers to be caught by static analysis tools tend to be dynamic behavior problems that are easy to overlook in testing. (If the problem were so obvious, the coder would have avoided it in the first place, right?)

Re:In Short, Yes

[HalWasRight]

valgrind, BoundsChecker, and I believe the others mentioned, are all run-time error checkers. These require a test case that execises the bug. The static analysis tools the poster was asking about, like those from Coverity and Green Hills, don't need test cases. They work by analyzing the actual semantics of the source code. I've found bugs with tools like these in code that was hard enough to read that I had to write test cases to verify that the tool was right. And it was! The bug would have caused an array overflow write under the right conditions.

Re:In Short, Yes

[samkass]

Due to its dynamic nature and intermediate bytecode, Java analysis tools seem to be especially adept at catching problems. In essence, they can not only analyze the source better (because of Java's simpler syntax), they can also much more easily analyze swaths of the object code and tie it to specific source file issues.

In particular, I've found FindBugs has an amazing degree of precision considering it's an automated tool. If it comes up with a "red" error, it's almost certainly something that should be changed. I'm not familiar with any C/C++ tool that comes close.

Re:In Short, Yes

[Anonymous Coward]

At my place of work, we just use OCaml. Most of the features and benefits of these third-party analysis tools are built into OCaml itself.

By using assertions and proper unit testing, virtually all errors can be caught early on. Our product is used on a daily basis by approximately 600 corporate users, and over the past six years, we've only had 37 bug reports originating outside of the company. None in the past two years, in fact. And this is even after heavy modification to our application.

OCaml's static typing catches many of our errors at compile-time. The assertions and unit tests catch problems that manifest themselves at runtime. And in the end our software is top-notch.

Re:In Short, Yes

[Shotgun]

That sort of code should be caught by a code review. Rule #1: Body cavity search all data at the boundaries.

But I guess dumb developer checkers have a place in the toolbox.

Re:In Short, Yes

[Anonymous Coward]

Weren't Valgrind and Purify in the hands of incompetent programmers the cause of the recent Debian OpenSSL issue?

Re:In Short, Yes

[Idaho]

There's also valgrind [valgrind.org], for Linux users, and mudflap [gnu.org], for gcc users. [..] valgrind is a very good runtime memory checker,

It most certainly is, but it's not a static checker ;) It detects problems at runtime, so only when they actually occur. It does not tell you about *potential* memory leaks that are only triggered in corner cases that you happen not to check.

Re:In Short, Yes

[Anonymous Coward]

When I was doing low-level Win programming in C++ (no foundation classes), I found BoundsChecker quite helpful in catching hard to trackdown bugs. It was worth the cost.

I've not used anything since switching from programming in C++ to C#. Strong type checking makes such programs less useful to me.

Re:In Short, Yes

[Anonymous Coward]

The proper answer would be: No. A fully working static code analyzer would be like solving the Halting Problem, which has been proven to be impossible. Essentially you can just try to catch as many potential problems as you can, but you can never catch all. I hate it when the halting problem is trotted out as "proof" that formal verification is impossible. If you like to put intractable recursion in your code then you probably shouldn't be a programmer. (Maybe you could draft legislation instead.) In practice, you should be able to prove (at least informally) that your program halts when it's supposed to.

The only real significance of the halting problem is to demonstrate that there can be some pretty absurd programs out there. It is not an indictment of static analyses. Nor is it an excuse to have less than total confidence in the correctness of your code.

Re:In Short, Yes

[Jeffrey Baker]

Yes, but any moron can catch bugs like that one with a simple "rgrep scanf ." and fire the programmers in question. Static analysis not required.

Re:In Short, Yes

[grammar fascist]

They're not perfect, and won't catch everything, but they do work.

More like can't catch everything. Or they'll "catch" perfectly fine things. Anybody remember Rice's Theorem from computability theory?

(What? You didn't think you'd ever need computability theory again?)

It's a generalization of the Halting Problem. The basic idea is that there is no algorithm for any non-trivial property ("does this program halt on every input?" "does this array get written out-of-bounds?" "does this value overflow?") that can verify every program. All such properties are undecidable. This means that either your static checker 1) must be wrong sometimes, or 2) doesn't always halt. Not halting is generally seen as bad, so strong typing systems halt with false positives and other static checkers tend to halt with false positives and negatives.

As such, they're good for advice only.

Catching errors dynamically is also incomplete, but for more obvious reasons.

Re:In Short, Yes

[pnewhook]

Would it not make sense to run this tool to catch these types of errors before wasting everyones time in a code review?

By the time you get to code review and test, you should be catching logic errors, not stupid syntactical and poor code style ones. If the tool helps a developer clean up and catch the obvious stuff, then testing can be much more productive catching the real problems.

Basically if the tool helps reduce errors then it is useful. Same comment goes for code complexity checkers. No tool will catch everything though, but then again you shouldn't be depending on it to.

Re:In Short, Yes

[nighthawk]

I'd have to second this. I'm still amazed that C and C++ are still used for any software of importance. The small increase in ease of use because they are so flexible (read sloppy), is more than offset by the huge amount of time spent scrubbing out all the stuff static analyzers are needed to find.

Re:In Short, Yes

[dollargonzo]

I disagree. Think of a loop where a break condition depends on the validity of, say, Goldbach's conjecture. No static analyzer can tell when (or if) such a program will halt. This is an extreme example, of course, but any sufficiently complicated expression that affects what code path is taken and/or termination of the program will pretty much have the same problem when it comes to static analysis. "Intractable recursion," as you call it, is computationally equivalent to unbounded loops. So, technically, anything that is in the body of an unbounded loop, except for the most trivial cases, a static analyzer cannot catch.

Re:In Short, Yes

[leonmergen]

I disagree. Think of a loop where a break condition depends on the validity of, say, Goldbach's conjecture. No static analyzer can tell when (or if) such a program will halt. This is an extreme example, of course, but any sufficiently complicated expression that affects what code path is taken and/or termination of the program will pretty much have the same problem when it comes to static analysis. Wouldn't just taking all the different codepaths solve the problem in the context of this article ?

Re:In Short, Yes

[NoOneInParticular]

Strictly speaking, if you construct a halting problem around a commercial static code analyzer to prove they can't solve all problems, you would likely be slapped with a lawsuit (a) for creating an unauthorized derivative work, and (b) for obtaining their source code through illegal means. Remember, a halting problem is constructed around a halting checker.

Re:In Short, Yes

[vrmlguy]

It's a generalization of the Halting Problem. The basic idea is that there is no algorithm for any non-trivial property ("does this program halt on every input?" "does this array get written out-of-bounds?" "does this value overflow?") that can verify every program. All such properties are undecidable. Such properties may be undecidable for every possible program, but that doesn't mean that they are undecidable for a specific program. main(){printf("Hello, world!")} is decidable, for instance. Non-trivial program can also be decidable in a practical sense, if not an absolute sense. Consider this bit of Python:
with open('somefile', 'r') as f:
  for line in f:
    print reversed(line)
As long as the iterator returned by open halts, so does the program, which means that as long as 'somefile' isn't a symlink to /dev/zero or some similar nonsense, we're safe.

Re:In Short, Yes

[_Swank]

Though, to be fair, "rgrep scanf" is a crude form of static analysis. So you haven't avoided it entirely.

Re:In Short, Yes

[Hotawa Hawk-eye]

Except that the rgrep is itself a very basic static analysis, right?

Re:In Short, Yes

[just_another_sean]

By the time you get to code review and test, you should be catching logic errors, not stupid syntactical and poor code style ones. If the tool helps a developer clean up and catch the obvious stuff, then testing can be much more productive catching the real problems.

Sounds like a good way to teach developers about these stupid errors as well. As someone whose knowledge of programming is self taught I learned a long time ago to pay attention to all errors, warnings and output from tools like lint to add to my understanding of the correct way to do things.

Re:In Short, Yes

[renoX]

Yup, but in the hand of incompetent programmers everything can be the cause of an issue.

As an aside, the original code should have explained better what it was doing to avoid this, so it's not only a maintenance issue.

Re:In Short, Yes

[synthespian]

Don't forget Polyspace, which I personally have never used - but I would love to. Polyspace is a Standard ML (a higher-order functional programming language) success story, because it relies heavily on the SML MLton compiler. Another thing that makes it a success story is the fact the Mathworks (makers of Matlab) bought it.

The C/C++ version does MISRA-C (the C used in the automotive industry) too.

There's also a version for Ada, of course.

Re:In Short, Yes

[pthisis]

I disagree. Think of a loop where a break condition depends on the validity of, say, Goldbach's conjecture.

That would be one of the absurd programs the GP was slamming. But a program where the break condition depends on, say, the user's input isn't amenable to static analysis and is perfectly reasonable and useful.

But you don't need to be perfect to be decent. A lot of static analysis can't tell what will happen, but can warn you if some code is unreachable, if no path will ever free memory, if a loop runs off the end of a memory allocation, etc.

The Linux kernel uses a lot of static checking tools to pretty great effect (sparse, for one, is extremely helpful, and the Stanford checker found a lot of problems too).

Re:In Short, Yes

[OldHawk777]

So, So, technically, anything that is in the body of an unbounded loop (would do what?), except for the most trivial cases, a static analyzer cannot catch (would it matter?).

NNFR; I am not a programmer/SE

Re:In Short, Yes

[Llywelyn]

I remember quite distinctly spending an hour inserting all of the issues into a code review that FindBugs came up with, and then going through and explaining each one. That time could have easily been spent by the author of the program, which would have then freed me up to focus on higher level problems that the static code checker can't detect (logic flaws, etc). Many of them can also detect things quickly, such as deadlocks and improper synchronization, that it can take a developer a while working with a pad of paper to catch.

It isn't a substitute for a proper code review, but it can make code reviews substantially more efficient and productive.

Re:In Short, Yes

[HuguesT]

For me, the tiny number of Ada libraries that I can use is a big problem. I don't have time to rewrite everything and I'm not working on a defence or aerospace contract. Is there *any* open source Ada code somewhere ? apart from GNAT ?

Re:In Short, Yes

[edxwelch]

If your goal is a low bug rate, neither unit testing nor automatic code check tools will have a huge impact. Automatic code check tools will flag down some minor bugs alright, and require little effort, so definately worthwhile.
Code quality isn't something that's just tacked on at the end of the process. If the design process is done well, there is less chance to bugs creeping in - it's just natural that logical well-designed code is going to be less buggy.

Re:In Short, Yes

[Bill Dog]

This is similar to what I was thinking -- the time at which an organization should be catching such errors is neither in the code review nor in running an analysis tool, it's during the interview.

Re:In Short, Yes

[Anonymous Coward]

That's what all the useless people say.

Re:In Short, Yes

[Anonymous Coward]

You don't need to be perfect to be useful.

I am perfect you insensitive clod!

Re:In Short, Yes

[Anonymous Coward]

Indeed the problem is obvious. Why are you using scanf at all? You are asking for trouble...

Re:In Short, Yes

[Goaway]

Perfectly USELESS!

Re:In Short, Yes

[Anonymous Coward]

My experiences programming in mercury were a real eye-opener. You have to annotate your predicates/functions with modes and determinism, but programming this way makes it very hard to get buggy code through the compiler at all. You'll end up fixing compiler errors for an hour, and then the program will run exactly as you intended it to. Then you can actually focus on any high-level mistakes you made due to misunderstanding the spec.

Re:In Short, Yes

[5pp000]

I hate it when the halting problem is trotted out as "proof" that formal verification is impossible.

I do too. The Halting Problem doesn't show that you can never prove that a program terminates; it doesn't even show that you can't write a program that proves that some other programs terminate. All it shows is that you can't write a program that is guaranteed to always tell you (in finite time), for any other program, whether that program terminates.

Re:In Short, Yes

[neokushan]

I hope you realise I just spent a good 2mins googling around for an explanation of a for loop with 4 parts to it instead of the 3 I was used to seeing. I genuinely thought it was some special, relatively unknown and underused part of the C spec that I'd just not seen before.
Then I realised it was just the HTML screwing up a less-than symbol. Then I felt a bit silly.
Then I just had to tell someone....

Re:In Short, Yes

[isomeme]

Exactly. But the user has to understand the imperfections reasonably well to make an imperfect product truly useful. I've seen developers look at a reasonably clean source analysis report and take that as proof that the code is bug-free. That is worse, in my view, than not giving them the tool at all. Proper training has to be part of the picture.

Re:In Short, Yes

[einhverfr]

I am not sure that you can prove anything more than the fact that the right factors will lead eventually to a halt assuming proper inputs.

However consider the following problem: you have a control system for a nuclear reactor. Under certain circumstances it is necessary to do an emergency shutdown of the reactor fast. You can indeed prove that it will shut down under the right circumstances, but proving that it will do so in the time required may be beyond the scope of a code analysis tool, even on the assembly-language level. I believe that this is a fundamental mathematical impossibility because of the fact that many sorts of issues which can arise which are beyond the scope of the proof.

In short there *are* limits as to what one can validate in a software system, but this does not mean that it is a waste of time. Instead it is just something where one needs to be aware of certain limits of this sort.

Re:In Short, Yes

[NotZed]

Although there is nothing in this use of scanf that warrants that statement (in the pseudo code sense at least - like any other c function you need to check the return code). No different to using Int32.TryParse(string, out foo) and ignoring it's return code.

scanf used on integers can be perfectly safe. And any other input mechanism that read an integer would still fail with that loop - it has nothing to do with scanf. Like goto's, scanf is misunderstood and blindly called bad without understanding why - which is the problem, not the function(ality) itself. Anyone who doesn't understand their use enough to know when it is unsafe should not be calling for anyone who uses it to be fired.

Back on topic, I think money would be better spent on better programmers or training than only slightly useful tools which give you a false sense of security.

Re:In Short, Yes

[Kamineko]

For shame.

Re:In Short, Yes

[fishbowl]

You claim (correctly) that non-trivial problems can be decidable. But the issues of decidability
show up quickly, when we talk about static checking for correctness.

Is part of your input to the checker, an assertion that the property shall be decidable for the given program?

If we have that, then we have a decider for halting, or anything that reduces to halting, and the only decision we have to make is to repeat your overriding assertion.

I agree that it might be the case, when we run a static code checker, that we are asserting to the checker, that the property being evaluated is decidable. The code checker isn't trying to be a UTM. It's performing tasks that have already been vetted as being possible -- or at least by asserting this, we accept failure as our error.

Rice's theorem tells us that a static analysis tool will be required to make approximations, and that we are forced to accept false positives and false negatives as a result.

I'm going to leave further discussion on the subject to those slashdotters who grade papers for an automata course (you see the darndest ideas on those sometimes, and occasionally wish they were right.)

Re:In Short, Yes

[Simon80]

Talking about "logical well-designed code" is very vague, and doesn't convey any useful information. I can suggest two ways to minimize bugs: make your interfaces hard to misuse (i.e. see this and this), and use testing to make sure that your interfaces work as advertised.

Re:In Short, Yes

[Simon80]

I realize that, but since the parent went ahead and suggested several commercial runtime memory checkers, I wanted to tack on a couple of free ones for any readers that haven't heard of them.

Re:In Short, Yes

[plover]

That's what the static analyzers do. They don't bother discerning whether or not the true or false path can be taken, they simply follow both paths and report something like "Null pointer dereference in line 222 when 'if' clause at line 123 is false." It doesn't bother to figure out if it's possible for the condition to actually become true or false -- if you coded a decision path, it makes the assumption that either could be followed.

The analyzers wouldn't be very useful if they had to fork at every code branch. A program with not-too-many nested decision branches would quickly become unexaminable.

Re:In Short, Yes

[benhattman]

The only real significance of the halting problem is to demonstrate that there can be some pretty absurd programs out there. It is not an indictment of static analyses. Nor is it an excuse to have less than total confidence in the correctness of your code. Nuts to that. The true significance of the halting problem is to demonstrate that a class of problem exists for which no solution can ever be provided even with infinite computation. Such a solution justifies the "it's good enough" approach to hard problems, which has proven essential to producing workable solutions in many domains. For instance, the one we are talking about, where you might be able to find some errors but certainly not all of them.

Re:In Short, Yes

[CorSci81]

This was exactly the point I was going to make. I spend a very large fraction of my time on any code project of importance just designing the damn thing. It annoys the hell out of people I work with sometimes, but generally by the time I finish the design job the code just flows naturally. And at the end of the day most of my compile time errors are just stupid typos and once I fix them my programs rarely have any unknown (i.e. I know what I'm doing has ways to break it and I flag them) runtime errors. Then again, I mostly design static libraries for other developers and I trust them to not do stupid things with my code (like ignoring 5 error returns before finally passing a function bad input that should've been caught WAY before).

Re:In Short, Yes

[poopdeville]

A program that depends on Goldbach's conjecture for its validity over all inputs isn't particularly absurd. Have you never written a program that decomposes natural numbers into a sum of primes? Never? Wow, you must write pretty boring programs. Additive prime decomposition is a straightforward way to partition a space for combinatorial problems.

Re:In Short, Yes

[poopdeville]

Nuts to that. The true significance of the halting problem is to demonstrate that a class of problem exists for which no solution can ever be provided even with infinite computation.

The halting problem (alone) doesn't do that. You can model Turing's oracles as computations at limit ordinals, and a single oracle is sufficient to solve the plain old halting problem. Then again, the argument can be adapted to show that oracle machines have insoluble problems. (And that there is no last ordinal, and that uncountable sets are bigger than little omega, and so on).

Unbounded and infinite are not synonyms.

Re:In Short, Yes

[porpnorber]

I hate it when the halting problem is trotted out as "proof" that formal verification is impossible. If you like to put intractable recursion in your code then you probably shouldn't be a programmer. (Maybe you could draft legislation instead.) In practice, you should be able to prove (at least informally) that your program halts when it's supposed to. [...] The only real significance of the halting problem is to demonstrate that there can be some pretty absurd programs out there. It is not an indictment of static analyses. Nor is it an excuse to have less than total confidence in the correctness of your code.

Apparently you aren't in the formal software verification business. The canonical example of the unverifiable piece of software is the software verifier.

And no shit! If someone tells you, "I'm always right, and you know that this is so because I say so, and I'm always right!" are you going to believe them?

So not only is formal software verification obviously impossible, but unverifiable programs that we would obviously like to verify obviously exist.

The thing you perhaps ought to be taking issue with is the idea that if it's not perfect it can't be damned useful. That, I will agree, is silly. But your original argument seems to be targeted squarely at itself!

Re:In Short, Yes

[FBSoftware]

What kind of Ada libraries are you looking for? There are lots of open source Ada projects on Sourceforge or on various personal developer web sites. The licenses vary, not all are GPL. Search comp.lang.ada archives or look at the links from http://www.adaic.org/links/libs.html.

Re:In Short, Yes

[TapeCutter]

Yes, what the halting problem states is that it is impossible to figure out if an algorithim halts without actually following the the algorithim until it...well...halts. If you can get to the stop point faster with the same results then you have found a more efficient algorithim.

In no way does the halting problem mean that you can/cannot prove a particular algorithim halts.

Most of the tools available (eg: bounds checker) take a long time to diagnose a chunk of code. They are not a debugging tool, they are a testing tool that can be very usefull when testing for null pointers and memory leaks. You still need a human to: a) configure and run the tool, b)evaluate the results, and c) fix any problems found.

Re:In Short, Yes

[Anonymous Coward]

Practical application of the halting problem: Give a Wiimote to a 2-year old kid or a dog or something, and see if he ever finishes a Mario Kart Wii race on one of the tracks where Lakitu drops you behind where you fall off the track (Mushroom Gorge comes to mind).

The kid may very well never finish.

Re:In Short, Yes

[Anonymous Coward]

Thanks for those references.

Re:In Short, Yes

[lpq]

Yeah -- if you think about it -- if humans can "pre-plan" a design, why can't a program be written to catch pre-planning errors or errors in implementation? Computers are already superior for managing complex resource constrained schedules allowing factories to run much more efficiently than planning and resource scheduling done by humans. Why should programs, ultimately, be any different? Oh, that's right, bugs are a requirement for job security...

Guess there are reasons to make sure static code analyzers don't work.

Re:In Short, Yes

[TheRaven64]

And, as a corollary, it is possible to prove whether any program which can be represented by a finite automaton will terminate. This means any program that has a finite limit on its stack size, which means any program running on a real computer. The algorithm for doing so is trivial but has huge space requirements.

Re:In Short, Yes

[Anonymous Coward]

You mean under the wrong conditions.

There, fixed for you!

Re:In Short, Yes

[jthill]

Everybody knows a single divisor that division can't handle. Is division impossible?

Re:In Short, Yes

[Shoten]

Some of these tools do catch it before code review. A number of them are not meant to work in a code review process, but rather operate real-time as the developer writes code. The philosophy behind this runs along the lines of 'the horses are out of the barn for existing apps; instead, focus on what's in development now and help the coders learn from their mistakes directly at the same time.' The biggest advantage to this that I see is that anyone who is writing code not only learns of their own foibles, but they're actually incented to write code more securely in the first place, since they have to rewrite it on the spot if the tool flags a problem. And it could also be argued that code that won't pass through a developer's workstation again (for updates or other changes) is so difficult to fix that the time/effort is better spent on the next set of apps.

Re:In Short, Yes

[KDR_11k]

Proofs require creativity at times, the halting problem is about algorithms which have zero creativity. All it really tells you is that you need a human in the loop.

Re:In Short, Yes

[vrmlguy]

Is part of your input to the checker, an assertion that the property shall be decidable for the given program?

If we have that, then we have a decider for halting, or anything that reduces to halting, and the only decision we have to make is to repeat your overriding assertion. No, I don't make that assertion, because in practice I don't care that the decider itself may (apparently) never halt. The decider has three outputs: "Yes, it's good", "No, here's a list of the problems I've found", and "Can you give me another minute?"

Staic analysis is a lot like the traveling salesman problem. The fact that there's no general solution doesn't preclude the existence of useful tools. A static analysis just has to try to transform the program into a simpler one, and if it takes too long then we still have useful information.

Re:In Short, Yes

[KDR_11k]

If you use sparse conditional constant propagation you can catch at least a decent number of contradictions or tautologies, would surprise me if such an analysis tool wasn't capable of that.

Re:In Short, Yes

[0xFCE2]

This is an extreme example, of course, but any sufficiently complicated expression that affects what code path is taken and/or termination of the program will pretty much have the same problem when it comes to static analysis.

But for programs where it really matters (the usual example: aviation), you should have a very tight specification. And that specification hopefully doesn't contain "sufficiently complicated" expressions. So analysis tools may work pretty well on software written with such specifications, simply because the specification doesn't allow any behavior for which the halting problem would matter.

Re:In Short, Yes

[Anonymous Coward]

That's a philosophical, not theoretical, objection. (How do we know that any proof is correct? Can we prove the proof? How do we know the meta-proof is correct?) The advantage of a formal verifier is that confidence in the verified programs can be reduced to confidence in the verifier. But we can have independent teams code verifiers. We can audit the code. We can put the theoretical framework through peer review. In short, it's easier to debug the verifier than it is to debug everything else.

Re:In Short, Yes

[Anonymous Coward]

Nuts to that. The true significance of the halting problem is to demonstrate that a class of problem exists for which no solution can ever be provided even with infinite computation. Such a solution justifies the "it's good enough" approach to hard problems, which has proven essential to producing workable solutions in many domains. Not so. Even if the halting problem (and similar problems) were to have a solution, we would still be using the "it's good enough" approach for reasons of algorithmic complexity. Just like we can't feed an arbitrary provable theorem into a computer and have it discover the proof in a reasonable amount of time, despite decidability. Undecidability is not the real issue here.

Re:In Short, Yes

[magisterx]

An informal proof is not a proof. However, for most given pieces of code it is possible (and fairly easy) to formally prove that it will halt, it is merely that you can prove that the general case cannot be solved. In other words, for any given piece of code you can probably prove that it will or will not halt, but it is possible to construct code for which it cannot be proven.

Re:In Short, Yes

[Anonymous Coward]

ignoring it's return code

"its".

Re:In Short, Yes

[Baki]

The cost is immense:

I have seen some large projects where loads of people were wasting their time with configuring such tools, with trying to analyze the results. Others had to "fix" the problems but then it became clear in many cases that the tool produced false positives, resulting in more tweaking of the rules/configuration of these tools.

Also, many became so focussed on the tools that they replaced (partially) normal code reviews and common sense; the result was that the most horrible problems went undetected, but according to the predefined rules everything was ok. There was a lot of emphasis on coding style and syntactic issues, but whether the software really did not contain functional bugs was automatically deprioritized.

So no, I would not say that such tools have no expense, even if the tool itself is free.

Re:In Short, Yes

[Anonymous Coward]

It's kind of sad that this was modded funny.

I work in a fairly boring domain. I maintain and modify a members-only website (that reports statistics for a larger industry). As part of my job, I often have to optimize Oracle queries. Mind you, we have a DBA on hand for "really hard" problems, but my department hasn't sent any his way since I got there. How do I do it? I get a pad of paper and pull out books on Lattice and Category theory (for reference, at this point). I use combinatorial methods extensively too, both for query optimization and for algorithm analysis. I get a full week's worth of work done in a day -- everybody on the team is assigned the "same" amount roughly -- and get to do homework the rest of the week. (Algebraic topology, currently)

Re:In Short, Yes

[Anonymous Coward]

If you'd run a static analysis tool over that code, you'd have saved yourself some trouble.

Re:In Short, Yes

[fishbowl]

>The decider has three outputs: "Yes, it's good", "No, here's a list of the problems I've found", and "Can
>you give me another minute?"

That's exactly what I mean by "an overriding assertion."

I haven't said that analysis tools can't be useful.
I have only said that we must accept approximate results.

Re:In Short, Yes

[bemo56]

By the time you get to code review and test, you should be catching logic errors, not stupid syntactical and poor code style ones. If the tool helps a developer clean up and catch the obvious stuff, then testing can be much more productive catching the real problems.

I worked on a software project a while ago and was tasked with finding problems caused by some previous developers.

Buy using these tools, I found code with "mistakes" which, more than often led to the stuff that wasn't obvious and would have only been found by chance.

While these tools wont catch the "Serious" problems, they can lead to the poor code where these problems usually live

Yes.

[Dibblah]

It's another tool in the toolbox. However, the results are not necessarily easy to understand or simple to fix. For example, see the recent SSL library issue - Which exhibited minimal randomness due to someone "fixing" an (intended) uninitialized memory area.

Re:Yes.

[mdf356]

If you're using uninitialized memory to generate randomness, it wasn't very random in the first place.

Not that I actually read anything about the SSL "fix".

Re:Yes.

[Anonymous Coward]

Sigh. That bug wasn't from fixing the use of uninitialized memory, it was from being overzealous and "fixing" a second (valid, not flagged as bad by Valgrind) use of the the same function somewhere near the first use.

Re:Yes.

[Anonymous Coward]

I think the actual details weren't very widely reported anyway. Apparently two statements were removed; one read from uninitialised memory, but the other was completely valid. Since the second one was responsible for most of the randomness, removing it reduced the keyspace to the point where it can be brute forced.

Re:Yes.

[moocat2]

Assuming you are talking about the SSL issue in Debian - the original 'issue' they tried to fix was reported by Valgrind. Valgrind is a run-time analysis tool.

While the parent makes a good point that results are not always easy to understand or fix - since the original post is about static vs run-time analysis tools, it's good to understand that they each have their problems.

Re:Yes.

[ezzzD55J]

Yes, only getpid(). Idiotic.

Re:Yes.

[Josef Meixner]

If you're using uninitialized memory to generate randomness, it wasn't very random in the first place.

It is only one source for the entropy pool and the SSL "fix" was a Debian maintainer running valgrind on OpenSSL, finding a piece of code where uninitialized memory was accessed, "fixed" it and a "similar piece" and accidently removed all entropy from the pool. The result of that is, that all ssh-keys and ssl-certs created on Debian in the last 20 months are to be considered broken. (Debian Wiki SSLkeys on the scope and what to do)

Re:Yes.

[DougBTX]

From the should-I-comment-out-these-lines post, apparently both lines were flagged, or at least the person using the tool thought they were flagged, which has much the same effect.

Re:Yes.

[xenocide2]

Of course the second one was flagged, there was a fucking #ifdef PURIFY right next to it. An entropy pool basically has randomness added to it. The problem is that the destination is one of the operands in every case, so there's no initial starting value by design. Static analysis tools decide that adding a random number to an uninitialized field doesn't make the field initialized. So the pool is never initialized since that would remove some magic entropy that probably doesn't exist but certainly doesn't hurt.

Re:Yes.

[Anonymuous Coward]

Everybody keeps telling this, but:

Was valgrind complaining AT ALL about the second MD_Update() (the one from ssleay_rand_add()) ? The probability of a programmer modifying two separate pieces of code --not exactly related-- on questionable look-alive (the usage of MD_Update() is quite different in the two) is really slim.

I'd rather buy into the conspiracy theory, where he deliberately slipped this ;-)

Re:Yes.

[Anonymous Coward]

from that thread:

Valgrind never reported a bogus error message to me. The problem isn't always that clear where the problem is however.

Afaik, both are actually being used, the one without the #ifndef PURIFY just doesn't seem to be used that much.

That's absolutely disgusting. That error-and-trial, cargo-cult, rolling-eyes superstitious approach to code. It's a kind of poetic justice he fucked up in the end.

Re:Yes.

[Omnifarious]

This particular programmer has been known in the past to check in code to CVS that failed to compile. So the fact he did this doesn't surprise me.

Just like compiler warnings...

[mdf356]

Here at IBM we have an internal tool from research that does static code analysis.

It has found some real bugs that are hard to generate a testcase for. It has also found a lot of things that aren't bugs, just like -Wall can. Since I work in the virtual memory manager, a lot more of our bugs can be found just by booting, compared to other domains, so we didn't get a lot of new bugs when we started using static analysis. But even one bug prevented can be work multiple millions of dollars.

My experience is that, just like enabling compiler warnings, any way you have to find a bug before it gets to a customer is worth it.

Re:Just like compiler warnings...

[gmack]

I found even tools like lint or splint can catch interesting bugs but not nearly as many as the time I enabled GCC's type checking for varargs on a software project a company I work for was developing.

Re:Just like compiler warnings...

[ccguy]

Here at IBM

any way you have to find a bug before it gets to a customer is worth it. You must be <30. You definitely weren't there in the OS/2 years.
So what product did you say you were working on? :-)

Re:Just like compiler warnings...

[WatersOfOblivion]

Hey! I'm less than thirty and I remember coding for OS/2 Warp! Of course, I was 11 at the time...

Re:Just like compiler warnings...

[mdf356]

I work on the VMM for AIX.

All software sucks, but the ones I know the internals of suck less. :-)

Re:Just like compiler warnings...

[Sponge Bath]

So what product did you say you were working on?

OS/3 - The Empire Strikes Back

Re:Just like compiler warnings...

[pnewhook]

Here at IBM we have an internal tool from research that does static code analysis.

That wouldn't be SCAMP would it? Love that tool but haven't seen it since a really old version I had on a developer kit. I emailed the original author to see if it is still being written but never got a response back.

Re:Just like compiler warnings...

[pnewhook]

Wow - jump to conclusions much? Get a grip.

Re:Just like compiler warnings...

[Bat Country]

My experience is that, just like enabling compiler warnings, any way you have to find a bug before it gets to a customer is worth it.

Thank god for people like you.

Re:Just like compiler warnings...

[mdf356]

The new tool is called BEAM (Bugs, Errors And Mistakes). I have no idea if there's plans to productize it. So far my code has found almost as many bugs in BEAM as BEAM found in my code.

Re:Just like compiler warnings...

[fishbowl]

>I hate having to work with dicks like you, because you misrepresent your experience and knowledge far too often.

At 14 I wrote an inventory control system, complete with history reports targeted for a specialized accounting system, archives, it worked for multiple vendors, recorded shipping and receiving data, allowed some customization of the data entry screens for the client, and even had my own validating input field editor in Z-80 Assembly.

You'd probably love to work with a dick like me, once you got to know me. But I'm not going to understate my experience and knowledge :-)

OSS usage

[MetalliQaZ]

If I remember correctly, one of these companies donated their tool to many open source projects, including Linux and the BSDs. I think it led to a wave of commits as 'bugs' were fixed. It seemed like a pretty good endorsement to me...

Re:OSS usage

[NewbieProgrammerMan]

IIRC, they donated the results of their analysis, not the actual tool itself. If somebody *did* actually donate a free license to use their software to those projects, I missed it and would love a link to the details. :)

Re:OSS usage

[chromatic]

That was Coverity. We've found the results very useful for Parrot. You have to be smart about how you interpret the results and how you fix actual bugs, but their tools did reveal many dubious constructs and actual bugs.

Re:OSS usage

[Anonymous Coward]

I tried Parrot 0.6.1 recently, and I'm sorry to say, it's a buggy piece of shit. It'd segfault on nearly every sample or example script that I tried.

I thought that it might have just been my PC or my compiler or something, so I tried it on my wife's Mac, and has just as much trouble there.

Yeah, I should probably report these bugs. But since I don't think Parrot will ever reach a usable state, it doesn't seem worth my time.

Maybe you guys should run Coverity on your code again. There seems to be a lot of problems with it.

Re:OSS usage

[diegocgteleline.es]

Well, Linux (at least the kernel) is mainly analyzed with Sparse, a tool started by Linus himself.

Re:OSS usage

[chromatic]

Yeah, I should probably report these bugs.

Yeah, you probably should, since we have a large test suite and we do our best to keep it 100% passing on all supported platforms.

Re:OSS usage

[Anonymous Coward]

parrot ? isn't that dead yet ?

would you spend another decade perfuming that pig ?

could you spell 'FAILURE' ?

Static analysis tools

[MadShark]

I use PC-lint religiously for my embedded code. In my opinion it has the most bang for the buck. It is fast, cheap and reliable. I've found probably thousands of issues and potential issues over the years using it.

I've also used Polyspace. In my opinion, it is expensive, slow, can't handle some constructs well and has a *horrible* signal to noise ratio. There is also no mechanism for silencing warnings in future runs of the tool(like the -e flag in lint). On the other hand, it has caught a (very) few issues that PC-Lint missed. Is it worth it? I suppose it depends if you are writing systems that can kill people if something goes wrong.

Re:Static analysis tools

[somersault]

I suppose it depends if you are writing systems that can kill people if something goes wrong. The best way to avoid bugs in that case would be for the developers to test the systems themselves - then they'd be a lot more careful! Plus it helps natural selection to weed out the sloppy coders :) In that case you'd probably want to write all the code from scratch though to ensure that nobody else's bugs kill you.

Re:Static analysis tools

[sadr]

Here's another vote for PC-Lint by http://www.gimpel.com/

You really can't beat it for the money, and it is probably as comprehensive as some of the other more expensive products for C and C++.

Re:Static analysis tools

[MadShark]

We do test them ourselves. We chain the equipment down during testing so it can't tip over. We've still bent/broke things due to going outside mechanical system limits(that were not in the spec by the way). It gets even more interesting when you are 50 feet in the air, in a 30 MPH wind when it is -30 F outside. If the machine doesn't kill you, the cold will.

They do work

[Anonymous Coward]

Static analysis does catch a lot of bugs. Mind you, it's no silver bullet, and frankly it's better, given the choice, to target a language+environment that doesn't suffer problems like dangling pointers in the first place (null pointers, however, don't seem to be anything Java or C# are really interested in getting rid of).

Even lint is decent -- the trick is just using it in the first place. As for expense, if you have more than, oh, 3 developers, they pay for themselves by your first release. Besides, many good tools such as valgrind are free (valgrind isn't static, but it's still useful).

Re:They do work

[Alphasite]

You should look at Spec# who does really care about null pointers among other things such as contracts.

Re:They do work

[Kjella]

(null pointers, however, don't seem to be anything Java or C# are really interested in getting rid of). Frankly, I don't see how you could avoid it as long as you have indirect references. Imagine you said pointers are the work of the devil, and you could only have references to valid objects. Well, a lot of the time you'd make some sort of dummy object that you're not supposed to call as a temp. If you did call it, it'd be a mistake and you'd actually like some kind of "DummyObjectException" instead of "NullPointerException" but the result would be exactly the same. Even with some sort of design by contract I don't think you could code your way out of creating some sort of "contract breach" exception.

Re:They do work

[renoX]

>Frankly, I don't see how you could avoid it as long as you have indirect references.

Avoid totally?

No.. But reduce significantly the issue see Nice which have by default non-null types (with X x = ..; x can't be null) and if you want to declare nullable variable you have to add a '?' at the end of the typename ie X? x; so the type system helps catching errors.

Re:They do work

[owlstead]

"null pointers, however, don't seem to be anything Java or C# are really interested in getting rid of"

Oh really? See the Java JSR-308 and the work of Michael Ernst and his MIT collegues (e.g. @NotNull, @ReadOnly, @Interned and @Immutable annotations) for more info.

Yes

[progressnerd]

Yes, some static analysis tools really work. FindBugs works well for Java. Fortify has had good success finding security vulnerabilities. These tools take static checking just a step beyond what's offered by a compiler, but in practice that's very useful.

Re:Yes

[Pollardito]

These tools take static checking just a step beyond what's offered by a compiler, but in practice that's very useful. that's a good point in that compiler warnings and errors are really just the result of static analysis, and i think everyone has experience in finding bugs due to those

Re:Yes

[Deanalator]

An important thing to remember is that these security based static analysis tools are tools designed to be used by security analysts. While it is good for developers to see the reports, and understand common security vulnerabilities, too often I see people hooking up static analysis to their code repository, and thinking that they have solved all future security problems.

Re:Yes

[Anonymous Coward]

Whilst they may not be useful for your average developer, they're really really useful for your developer who has a clue about security. Unfortunately, most developers don't, so you need to waste the time of your security consultants.

Re:Yes

[FormOfActionBanana]

I disagree. Fortify and Ounce both spend a lot of effort on developer remediation; for example Fortify displays in the IDE a detailed article about the issue found, how developers fall into the trap, exploit examples and how to safely remediate the code. Truncated example of this sort of information here: http://www.fortify.com/vulncat

It's true that security analysts are the ones best suited to triaging away the false positives.

Re:Yes

[museumpeace]

interesting.
I still am skeptical about static checking be very effective in seeing race conditions...but then Coverity also as dynamic checking tools: https://event.on24.com/eventRegistration/EventLobbyServlet?target=registration.jsp&eventid=110122&sessionid=1&key=678B7E9A4BA76B1FB9659E52F0946162&partnerref=5SD&sourcepage=register

Change bug source

[192939495969798999]

The best thing these tools can do is to tell everyone what they probably already know -- that a particular coder or coder(s) are responsible for a whole ton of the errors in the code. I think it'd be much better to move that coder to some other part of the company ... it would be way cheaper than trying to fix all their bugs.

Re:Change bug source

[GenMoo]

Even the best coders make mistakes... Your comment shows that you're not so familiar with coding

Re:Change bug source

[Alphasite]

Quite the contrary, I think he's got a point. That's mainly the reason why great companys (specially google) try so hard to get the best coders they can.

Of course best coders still make mistakes but lousy coders make a lot of them, belive me "I've seen things you people wouldn't belive". Every coder makes mistakes but some coders or maybe so called coders which in fact happens to have a learn xxx in 21 days course makes lots of mistakes.

Re:Change bug source

[smellotron]

I think it'd be much better to move that coder to some other part of the company...

You mean like the "not working here anymore" division?

Static analysis tools

[wfstanle]

I am presently working on an update to static analysis tools. Static analysis tools are not a silver bullet but they are still relevant. Look at them as a starting point in your search for programming problems. A lot of potential anomalies can be detected like the use of uninitialized variables. Of course, a good compiler can use these tools as part of the compilation process. However, there are many things that a static analyzer can't detect. For this, you need some way to do dynamic analysis ( execution based testing). As such the tools we are developing also include dynamic testing.

Yes, they work.

[Anonymous Coward]

You will probably be amazed at what you will catch with static analysis. No, it's not going to make your program 100% bug-free (or even close), but every time I see code dies on an edge case that would've been caught with static analysis, it makes me want to kill a kitten (and I'm totally a "cat person" mind you).

Static analyzers will catch the stupid things - edge cases that fail to initialize a var, but then lead straight to de-referencing it; memory leaks on edge-case code paths, etc. that shouldn't happen but often do, and get in the way of find real bugs in your program logic.

Re:Yes, they work.

[anonymous_wombat]

... it makes me want to kill a kitten (and I'm totally a "cat person" mind you). Yes, I am a cat person too. I like killing kittens, especially if I can use a nail gun. I got 5 with one nail one time. Well, I think the first nail got them, although the next 26 might have finished the job. I'm not sure.

Re:Yes, they work.

[merreborn]

very time I see code dies on an edge case that would've been caught with static analysis, it makes me want to kill a kitten

Wow, I've heard of some strange fetishes, but that takes the cake...

Re:Yes, they work.

[Anonymous Coward]

it makes me want to kill a kitten You know that's slang for masturbation, right?

Re:Yes, they work.

[Gorobei]

Google "god kills a kitten" and you will understand.

Of course they can work

[Idaho]

Such tools work in a very similar way to what is already being done in many modern language compilers (such as javac). Basically, they implement semantic checks that verify whether the program makes sense, or is likely to work as intended in some respect. For example, they will check for likely security flaws, memory management/leaking or synchronisation issues (deadlock, access to shared data outside critical sections, etc.), or other kind of checks that depend on whatever domain the tool is intended for.

It would probably be more useful if you could state which kind of problem you are trying to solve and which tools you are considering to buy. That way, people who have experience with them could suggest which work best :)

Re:Of course they can work

[Yvanhoe]

It looks like you are reading memory from unallocated memory space, you should comment line 427 in ssl_seed_generator.h

I also see how it could bring a distribution to its knees. But I agree that they will probably be worthwhile 90% of the time.

Re:Of course they can work

[mrogers]

Depends on whether one interprets "you should comment" as "you should document" or "you should comment out", I guess. :-)

Re:Of course they can work

[jlunavtgrad]

Well I'm coding for an embedded linux environment in C and C++, so java tools won't help. I am more interested in the tools that can generate a call-stack and trace through the execution path to show you how a null-pointer dereference might come about. I was looking at Klockwork Insight, but the price tag is practically my salary for the number of licenses we would need. Has anyone worked with that tool specifically?

Re:Of course they can work

[laddhebert]

Yes, we run Klocwork 7.7.x in production an 8.0 in test. Our users (embedded developers, wireless mostly, along with some IDE developers) have had great success with Klocwork. The price tag is up there, but you need to do an ROI exercise to determine if it is worth it. For us, it WAS worth it. We also have Coverity in house, but many users are leaving that tool to use Klocwork due to ease of use, number of false positives, along with a decrease in cost since they can use a shared pool of licenses. Polyspace is on the landscape, but it is commonly thought of as a tool to use once you've fixed all the problems that Klocwork and Coverity discovered.
Hope that helps!
/-l

Re:Of course they can work

[procrastitron]

Are you doing new development or maintaining an already large code base? For new development I would suggest looking at Cyclone, which adds static analysis (in the form of type checking) to the language itself.

Re:Of course they can work

[Coryoth]

Such tools work in a very similar way to what is already being done in many modern language compilers (such as javac). Basically, they implement semantic checks that verify whether the program makes sense, or is likely to work as intended in some respect. The keywords here, for the purposes of good static analysis tools, are "work as intended". Knowing what code is intended to do isn't trivial, and it can be severely limiting in what static analysis tools can check for (or, alternatively, they can spit back a lot of false positives). A good solution is provide the static analysis tool with some hints as to what the programmers intentions are -- that means annotations as used by splint, or ESC/Java2, or something similar. The benefits for static analysis (and API documentation depending on the tools) can far outweigh the "extra" work*; consider the sorts of errors ESC/Java2 can successfully catch for instance. Of course you have to be interested in doing static anaylsis to begin with, btu the benefits are definitely available.

* Note that "extra" is in scare quotes because ultimately most of the annotations are usually things you should be documenting anyway, so it's not so much "extra" work as documentation that gets done sooner rather than later.

Re:Of course they can work

[Idaho]

Well I'm coding for an embedded linux environment in C and C++, so java tools won't help. I am more interested in the tools that can generate a call-stack and trace through the execution path to show you how a null-pointer dereference might come about. I was looking at Klockwork Insight, but the price tag is practically my salary for the number of licenses we would need. Has anyone worked with that tool specifically?

Someone please mod the parent up as it's the original poster clarifying his question.

I have not used this particular tool so I have no idea. (basically, beyond valgrind and gdb/ddd my knowledge of C debugging tools is pretty outdated)

It sounds like what you're looking for is not necessarily *static* checking (checking that happens at compile time). If you want to get better debugging information in the case that crashes occur at runtime, you are probably looking for "instrumentation" of the existing program, so that a tool can gather information about it at runtime (similar to a profiler or normal debugger). Or do you really want to feed it the source code only (without actually running the program) and tell you about potential paths that might lead to null-pointer access? Considering it's embedded software, you probably want the checking to be static, i.e. outside of the embedded device itself?

In the case of C, static checking of null-pointer access is impossible in the general case, because C uses a flat, unprotected (within the process itself) memory model and allows pointer math, which means someone could be doing complex pointer math and no amount of static checking is going to save you in that case. If you are willing to ignore such "borderline" cases, search for "c null pointer reference static checking" and you'll find quite some papers about this, which generally means there's bound to be some tools implementing those ideas. If your code involves lots of pointer math to improve efficiency, I think you'll find that most static checkers are unlikely to do a very good job. I'm afraid I can't help you with pointers to specific tools, as it's been at least 5 years since I last wrote any C code :(

In any case if you're considering to buy a tool, just send an e-mail to the tool supplier and tell them you would want to test the tool. Surely if you're considering to pay them about a programmers annual salary they should be willing to let you test the tool first, for that amount of $$.

Testing cycle

[mdf356]

I forgot to answer your other question.

Since we've had the tool for a while and have fixed most of the bugs it has found, we are required to run static analysis on new code for the latest release now (i.e. we should not be dropping any new code that has any error in it found via static analysis).

Just like code reviews, unit testing, etc., it has proved useful and was added to the software development process.

Yes! Uh, sorta.

[BigBlueOx]

Ya can't beat a good "Lint party" after all the testing is done! You'll find all kinds of cool stuff that slipped through your testing suites.

However, static code analysis is just one part of the bug-finding process. For example, in your list, in my limited experience, I have found that buffer overflows and NULL pointer derefs get spotted really well. Race conditions? Memory leaks? Hmm. Not so good.

YMMV. Don't expect magic. Oh to hellwithit, just let the end-users test it *ow!*

Yes.

[Anonymous Coward]

The Astrée static analyser (based on abstract interpretation) proved the absence of run-time errors in the primary control software of the Airbus A380.

Yes

[kevin_conaway]

Add me to the Yes column

We use them (PMD and FindBugs) for eliminating code that is perfectly valid, yet has bitten us in the past. Two Java examples are unsynchronized access to a static DateFormat object and using the commons IOUtils.copy() instead of IOUtils.copyLarge().

Most tools are easy to add to your build cycle and repay that effort after the first violation

MIT Site

[yumyum]

I took a very cool graduate-level class at MIT from Dr. Michael Ernst about this very subject. Check out some of the projects listed at http://groups.csail.mit.edu/pag/.

Very useful in .Net

[Toreo asesino]

While not the be-all-and-end-all of code quality metrics, VS2008/Team Foundation Server has this built-in now so you can stop developers checking in completely junk code if you so wish - http://blogs.msdn.com/fxcop/archive/2007/10/03/new-for-visual-studio-2008-code-metrics.aspx

FxCop too has gone server-side too (for those familiar with .Net development). It takes one experienced dev to customise the rules, and you've got a fairly decent protection scheme against insane code commits.

Re:Very useful in .Net

[SpryGuy]

If you're doing C# development, you should really check out JetBrains "ReSharper". Version 4.0 is due out soon, which supports all the C#3.0 syntaxes (extension methods, LINQ, lambda expressions, etc), but even 3.x is a worthwhile tool. It does real-time syntax checking (as you type) so you don't have to compile to find out you have a syntax error, as well as tons of refactorings, and very useful static code analysis.

Once you develop with Resharper, you really can't go back to using VS without it... it's like coding with stone knives and bear skins.

Re:Very useful in .Net

[Toreo asesino]

You know what, I did a couple of years back for VS2005; some of the functionality was real nice, but I found it dragged the IDE into the ground. Has it got any better since?

Re:Very useful in .Net

[SpryGuy]

Version 3.x still grinds things down if you have a 'website' in your solution, but it's faster loading and working in non-website/ASP.NET projects than previous versions.

I've been beta testing Version 4.x, due out soon, and performance is substantially better in every way, even with Websites and ASP.NET projects.

Definitely check out 4.0 when it's released. You can give it a 30-day trial before deciding to buy.

Yes, But...

[SwashbucklingCowboy]

Yes, static code tools do work well for finding certain classes of issues. However, they are not a panacea. They do not understand the semantics that are intended and cannot effectively replace code reviews.

The more things change...

[BrotherBeal]

the more they stay the same. Static code analysis tools are just like smarter compilers, better language libraries, new-and-improved software methodologies, high-level dynamic languages, modern IDE's, automated unit test runners, code generators, document tools and any number of other software tools that have shown up over the past few decades.

Yes, static code analysis can help improve a team's ability to deliver a high-quality product, if it is embraced by management and its use is enforced. No, it will not change the face of software development, nor will it turn crappy code into good code or lame programmers into geniuses. At best, when engineers and management agree this is a useful tool, it can do almost all the grunt work of code cleanup by showing exactly where problem code is and suggesting extremely localized fixes. At worst, it will wind up being a half-assed code formatter since nobody can agree on whether the effort is necessary.

Just like all good software-engineering questions, the answer is 'it depends'.

Not Yet, In My Personal Experience.

[RJCantrell]

In my own corner of the world (.NET Compact Framework 2.0 on old, arcane hardware), they certainly don't. Each time I get optimistic and search for new or previously-missed static analysis tools, all roads end up leading to FxCop. Horrible signal-to-noise ratio, and a relatively small number of real detectable problems. That said, I'm always willing to submit myself to the genius of the slashdot masses. If you know of a great one, feel free to let me know. = )

Re:Not Yet, In My Personal Experience.

[FishWithAHammer]

I believe Gendarme might be of some use. Just don't invoke the Portability assemblies and I can't see why it'd fail.

Re:Not Yet, In My Personal Experience.

[RJCantrell]

At first glance, this looks promising. Thanks for the link!

Re:Not Yet, In My Personal Experience.

[boatboy]

I've never had problems with FxCop - it catches tons of common and not-so common real-world errors, lets you turn off rules you don't care about, and links to usually helpful MSDN articles to explain the rules. I'd say everything in the Security, and Design categories are well worth the few minutes it takes to run a free tool. Having it baked into VS08 is even better, but I was a big fan of it with 05/2.0 as well.

Re:Not Yet, In My Personal Experience.

[FishWithAHammer]

No problem. I'm ripping Gendarme apart and including its functionality in my Google Summer of Code project, so I'd be interested in hearing any feedback from you regarding it. Feel free to drop me a line. =)

Re:Not Yet, In My Personal Experience.

[SpryGuy]

Check out ReSharper, from JetBrains.

Much greater utility, real-time, as-you-type analysis, lots of additional features like solid refactorings and code cleanup and formatting, life templates, code generation, better "intellisense", better code navigation, etc. A very highly useful tool, one that I recommend to anyone who will listen.

Useful for planning tests

[Chairboy]

At Symantec, I used to use these tools to help plan tests. I wrote a simple code velocity tool that monitored Perforce checkins and generated code velocity graphs and alerts in different components as time passed. With it, QA could easily see which code was being touched the most and dig down to the specific changelists and see what was going on. It really helped keep good visibility on what needed the most attention and helped everyone avoid being 'surprised' by someone dropping a bunch of changes into an area that wasn't watched carefully. During the final days of development before our products escaped to manufacturing, this provided vital insight into what was happening.

I've since moved on, and I think the tool has since gone offline, but I think there's a real value to doing static analysis as part of the planning for everything else.

Coverity Reports Open Source Security Making Grea

[doug]

Is this what you were talking about?

http://it.slashdot.org/article.pl?sid=08/01/11/1818241

- doug

Potential issues are the biggest drawback

[Anonymous Brave Guy]

I've also used Polyspace. In my opinion, it is expensive, slow, can't handle some constructs well and has a *horrible* signal to noise ratio.

The signal-to-noise ratio is pretty horrendous in most static analysis tools for C and C++, IME. This is my biggest problem with them. If I have to go through and document literally thousands of cases where a perfectly legitimate and well-defined code construct should be allowed without a warning because the tool isn't quite sure, I rapidly lose any real benefit and everyone just starts ignoring the tool output. Things like Lint's -e option aren't much good as workarounds either, because then even if you're hiding an issue that might be a phantom problem today, you'll still be hiding it if it becomes a real problem tomorrow. :-(

Coverity & Klocwork

[Anonymous Coward]

We have had presentations from both Coverity and Klocwork at my workplace. I'm not entirely fond of them, but they're wayyyyy better than 'lint'. :) I much prefer using "Purify" whenever possible, since run-time analysis tends to produce fewer false-positives.

My comments would be:

(1) Klockwork & Coverity tend to produce a lot of "false positives". And by a lot, I mean, *A LOT*. For every 10000 "critical" bugs reported by the tool, only a handful may be really worth investigating. So you may spend a fair bit of time simply weeding through what is useful and what isn't.

(2) They're expensive. Coverity costs $50k for every 500k lines of code per year... We have a LOT more code than this. For the price, we could hire a couple of guys to run all of our tools through Purify *and* fix the bugs they found. Klocwork is cheaper; $4k per seat, minimum number of seats.

(3) They're slow. It takes several days running non-stop on our codebase to produce the static analysis databases. For big projects, you'll need to set aside a beefy machine to be a dedicated server. With big projects, there will be lots of bug information, so the clients tend to get bogged down, too.

In short: It all depends on how "mission critical" your code is; is it important, to you, to find that *one* line of code that could compromise your system? Or is your software project a bit more tolerant? (e.g., If you're writing nuclear reactor software, it's probably worthwhile to you to run this code. If you're writing a video game, where you can frequently release patches to the customer, it's probably not worth your while.)

Re:Coverity & Klocwork

[Anonymous Coward]

Where I work we have signed up corporate wide for the Coverity product. It does work as expected and the false positive rate is acceptably low.

These tools are not the be-all end-all of bug finding; you still need to have competent programmers to start with and you need to enforce unit testing and good design.

Re:Coverity & Klocwork

[EPAstor]

I did some work running Coverity for EnterpriseDB, against the PostgreSQL code base (and yes, we submitted all patches back, all of which were committed).

Based on my experience:

1) Yes, Coverity produced a LOT of false positives - a few tens of thousands for the 20-odd true critical bugs we found. However, the first step in working with Coverity is configuring it to know what can be safely ignored. After about 2 days of customizing the configuration (including points where I could configure it to understand that certain methods fixed its common complaints already), the bug list dropped to 200. 2 days of configuration by an inexperienced user dropped it to a total false positive rate of about 75%, if that, though only about half of those proved to be critical bugs. I think that's good enough to be useful.

2) Can't argue - horrendously expensive.

3) Postgres is smaller than what you're talking about... but still, the speed was a pain, but the system was useful enough to make up for it, particularly once we had the system configured to know what bugs were real and automated its running every few nights. While EnterpriseDB still had the license, it was a great supplement to nightly testing.

Trends or Crutches?

[bsDaemon]

I'll probably get modded to hell for asking but seriously -- all these new trends, tools, etc - are they not just crutches, which in the long run are seriously going to diminish the quality of output by programmers?

For instance, we put men on the moon with a pencil and a slide rule. Now no one would dream of taking a high school math class with anything less than a TI-83+.

Languages like Java and C# are being hailed while languages like C are derided and many posts here on slashdot call it outmoded and say it should be done away with, yet Java and C# are built using C.

It seems to me that there is no substitute for actually knowing how things work at the most basic level and doing them by hand. Can a tool like Lint help? Yes. Will it catch everything? Likely not.

As generations of kids grow up with the automation made by generations who came before, and have less incentive to learn how the basic tools work, an incentive which will diminish, approaching 0, I think we're in for something bad.

As much as people bitch about kids who were spoiled by BASIC, you'd think that they'd also complain about all the other spoilers. Someday all this new, fancy stuff could break and someone who only knows Java, and even then checks all their source with automated tools will likely not be able to fix it.

Of course, this is more of just a general criticism and something I've been thinking about for a few weeks now. Anyway, carry on.

Re:Trends or Crutches?

[Lord_Frederick]

Any tool can be considered a "crutch" if it's misused. I don't think anyone that put men on the moon would want to return to sliderules, but a calculator is only a crutch if the user doesn't understand the underlying fundamentals. Debugging tools are just tools until they stop simply performing tedious work and start doing what the user is not capable of understanding.

Re:Trends or Crutches?

[bsDaemon]

In the 7th grade I left my calculator at home one day when I had a math test. I did, however, have a Jepsen Flight Computer (a circular slide rule) that my dad (a commercial airline pilot) had given me, because I was going to a flying lesson after school.

I whipped out my trusty slide rule and commenced to using it. The teacher wanted to confiscate it and thought that I was cheating with some sort of high-tech device... mind you it was just plastic and cardboard. I'm sure you've all seen one before.

I'm only just about to turn 24, so 7th grade was not long ago for me.

The point is, students should be required to know how to do things by hand. a PhD in physics 20 years ago clearly knew how to do calculus by hand. if he wants to use a Ti-92 to do it now, that's his business. Good that those aren't allowed in class (even if they are allowed on the SAT and AP exams, well the ti-89 is).

Intro to comp sci shouldn't be taught with Java any more than elementary school math should be "intro to the calculator." You just cripple people's minds that way.

I ended up getting a BA in English the first time around because of personal reasons I was too messed up to concentrate on maths. I'm now starting to take a 2nd degree in MechE - and I'm making a good faith effort to do as much by hand as possible, because I don't want to fuck something up in the future because I figured the calculator was giving me a right answer and I had no idea of where the proper answer should be.

summary: using a tool to check your answer is one thing. Relying on it to get the answer in the first place is lazy, stupid, and potentially dangerous.

Re:Trends or Crutches?

[flavor]

Did you walk uphill in the snow, both ways, when you were a kid, too? At one point in time, high-level languages like ASSEMBLER were considered crutches for people who weren't Real Programmers. Get some perspective!

Look, people make mistakes, and regardless of how good a programmer you are, there is a limit to the amount of state you can hold in your head, and you WILL dereference a NULL pointer, or create a reference loop, at some point in your career.

Using a computer to catch these errors is just another flavor of metaprogramming. Get over it, and go be more productive with these tools, instead of whining for the days when you coded on bare metal with your bare hands and you liked it.

Arrgh.

Re:Trends or Crutches?

[gmack]

These tools require skill. Blindly fixing things that Lint shows up can introduce new bugs or conversely using lint notation to shut the warnings off can mask bugs.

I also don't think new languages help bad programmers much. Bad code is still bad code so now instead of crashing it will just memory leak or just not work right.

On a software project I worked on before our competition spent two years and two million dollars did their code in visual basic and MSSQL and they abandoned their effort when no matter what hardware they threw at it they couldn't get their software to handle more than 400 concurrent users. We did our project in C and with a team for 4 built something in about a year that handled 1200 users on a quad CPU P III 400mhz Compaq. Even when another competitor posed as a client and borrowed some of my ideas (they added a comms layer instead of using the SQL server for communication) they still required a whole rack of machines to do what we did with one out of badly out of date test machine.

C is a fine tool if you know how to use it so I doubt it will go away any time soon.

Re:Trends or Crutches?

[fishbowl]

>a PhD in physics 20 years ago clearly knew how to do calculus by hand.

An undergrad in physics in my school knows how to do calculus by hand,
which is a requirement to be accepted into the degree program.

Long before one is hooded for a PhD here, mere "calculus" is a first language,
a primary tool for expressing research.

Re:Trends or Crutches?

[bsDaemon]

I always loved Bio more than physics, so I'm not really talking from any previous experience here -- I did intern as a C programmer at the Thomas Jefferson National Accelerator Facility out of high school, though.

At my first undergrad institution, I could have taken "Calculus with MATLAB" and relied on the CAS to do everything for me if I so chose, and that's the sort of thing which quite frankly, we should all be able to to agree is b.s.

Re:Trends or Crutches?

[EastCoastSurfer]

What level of detail is enough for someone to you know? You seem to imply C, while the programmers around when C came onto the scene would say you need to know ASM. Wait, does someone then need to know the details of how a transistor works or how electricity works on the molecular level in order to write good software? You can easily enter a never ending spiral of decomposition where you never end up solving the actual problem you set out to solve.

These tools are not crutches. They make possible the large, complex software that is written today. You're right, we put men on the moon with a pencil and slide rule. How many lines of code were required to accomplish this feat? Probably not more than could be hand checked by a talented person or two. Now look at modern OSs, RDBMs, etc... Code bases that are so large and complex, these tools are not crutches, but necessary parts of the development process.

Now, if you want to argue that software doesn't need to be as complex as it has become I think you might have a point. A look at the recent directions of languages like Python or Ruby shows that simplification is the new thing.

Re:Trends or Crutches?

[FlyingGuy]

Well I agree & disagree with you...

Bring from the old school of programing, I am quite literally programmed to never, ever, allocate without making damn sure I de-allocate.

The same thing goes for stuffing anything into arrays, doubly so for arrays of string, or any kind of buffer. Always check to see that you are not overflowing, the time penalty is not that critical, but failing to check and overflowing will be the thing that brings your code to a screaming halt, if not the entire box, depending on how low level the code is you are writing.

An argument can be made such as, "Its Java and I don't allocate memory directly but on the other hand you do allocate entire new objects and all of those have impact on memory. We rely on other peoples code to handle all that properly.

Perhaps this is the case, but in my opinion a competent programmer should never let a bit of code out of his or her hands that has the possibility of seg-Faulting or PMV'ng in the 1st place. It may perhaps be tedious to validate input, check your bounds, keep track of what you have asked for from the machine and make sure you give it back, but that is part of the job, and in the OP's opinion, with which I concur, programmers are beginning to forget the base fundamentals of their craft, and are relying far to heavily on OPC ( Other Peoples Code).

Re:Trends or Crutches?

[JoeMerchant]

It doesn't really matter that Java was built using C, early C was built using Macro-assembly (later C was built using early C), and Macro-assemblers were written in machine code. There are still people who understand the machine code - very few, but at the moment, that's all we need.

There are places where this kind of layering has led to real nasty black boxes (such as hard drive controllers) - but for the most part, it's a good thing - allowing people to move away from the technical details on to the things they're really interested in.

If you really want to get back to basics, you should be growing your own food, while defending your territory from raiding parties, and then start attempting to refine semi-conductors and silicone into microchips.

Re:Trends or Crutches?

[bsDaemon]

I would venture to guess that an electrical engineer (I am not one) would be able to write more competent code than someone who only knows "how to program," particularly in a super high-level language.

I've only just started messing with C again after a long while. I'll probably want to learn assembler or verilog as well.

My argument, however, is mostly about how we are creating a situation in which future generations are likely to become so dependent on high-level tools that the number of people who are qualified to create those tools are going to dwindle. And its not even really about software development, per se.

Its the old argument, "why remember what I can look up?" extruded to "why do by hand what i can do with the calculator," "why build what i can buy," etc.

The course of human development has largely been in search of answers to two questions: "if i'm down here, how can i get what's up there," and "man - that was hard. how can i automate it so that I don't have to do it anymore."

However, our schools seem to be skipping over teaching the "how do i get whats up there" (say, drawing a graph or working out an equation by hand), skipping "how did i automate that," stipulating "that was hard," and teaching the kids the tool that makes sure they won't have to actually do stuff.

Its making us lazy, fat and bandy. its double-plus ungood. I'm all for using tools to make your life easier once you know how to do something -- its why I started messing with programming in the first place - figure out how to do a math operation, then write a program to do it for me. however, its unacceptable to skip the "figure it out" part.

Re:Trends or Crutches?

[StrawberryFrog]

Languages like Java and C# are being hailed while languages like C are derided and many posts here on slashdot call it outmoded and say it should be done away with, yet Java and C# are built using C.

You are oversimplifying to the point were you are making no sense at all. Languages like Java, C#, Ruby, Python, Perl, etc. are built using C. C is built using assembly. If you think assembly is therefor the perfect tool for every job, you are sorely wrong. It is the right tool for making C compilers, and these days that;'s about all. C compilers in turn are good tools for making device drivers, .net virtual machines, Java runtimes, dynamic languages etc. These aforementioned languages are in turn the best tools that we currently have for run-of-the-mill real-world web, desktop and server programming tasks. C is not anymore. It should not be done away with, but as a "general purpose language" for all tasks, it's rather past its prime. It's becoming a niche skill, like assembler before it. And there's nothing wrong with this.

It seems to me that there is no substitute for actually knowing how things work at the most basic level and doing them by hand.

Knowing some assembler will make you a better C coder. Knowing some C will help you understand Java or C# better. Essential or all that's needed? Not at all.

Re:Trends or Crutches?

[coyote-san]

The lunar missions were actually pretty straightforward. You had to have a solid knowledge of physics and/or engineering to understand the math, but Newtonian mechanics has been understood for a LONG time and by the lunar missions it was mostly a question of getting accurate values for the constants.

They still got it wrong, btw. The control system on the lunar lander was unstable on the first few missions, but fortunately nobody tripped it. I think I read that it would have been non-survivable.

On the other issue... you're right that there are a lot of people who never develop the mental skills needed to stand alone. On the other hand, a lot of people never developed them before either. They didn't use their tools available to them either. They often didn't see problems even when you pointed it out to them.

And... when you get down to it, they were still (mostly) good enough for their jobs. 99% of the tasks are boring drudge work (relatively speaking) and you don't need the full set of tools you need to do the most critical work. More power to them, as long as they know when they're over their head.

That's why I'm a big fan of static analysis etc. It won't weaken skills that much, overall, and should dramatically improve the quality of the code when used appropriately.

Re:Trends or Crutches?

[SwashbucklingCowboy]

C is built using assembly.

Nope, C is built using C. Check out self-hosting.

Re:Trends or Crutches?

[hansamurai]

It's getting pretty crazy the calculator crutch schools are creating these days. I mentor a fifth grader and she was grabbing for her calculator when I asked her to add and subtract double digit numbers. I honestly don't know how to break her out of this habit.

Anyways, I found my TI-86 completely useless at college. It didn't help me in calculus or linear algebra or my AI classes. I'm just glad I worked hard during high school on math because I needed those skills for school. A calculator wouldn't have helped me.

Re:Trends or Crutches?

[Anonymous Coward]

I ended up getting a BA in English the first time around because of personal reasons I was too messed up to concentrate on maths. I don't want to sound like I'm making light of your problems (whatever they were), but... you were too messed up to work at your maths, yet you still managed to get a BA in English?

Either you're smarter than you think, or that's pretty damning of the English degree course (and the kind of anecdote snotty maths students could use as a veiled insult towards people studying English:-) ).

(Anyway, good luck with the second degree; I fucked mine up totally first time round for a whole load of personal reasons (also), then went back a few years later and passed with flying colours- and second time round I had a similar attitude to yours here).

Re:Trends or Crutches?

[Kingrames]

Would you consider a fighter Jet to be a crutch? You certainly don't have to know all the inner workings of it to pilot it. If you did, we'd have a serious shortage of pilots. (and yes, I know they do know a LOT about their planes)

Truth be told, BASIC doesn't make you stupid. Programming in C doesn't make you smarter. They're just tools.

Re:Trends or Crutches?

[svnt]

I'll bite. Are esoteric knowledge of edge cases and math-by-pencil capabilities really that valuable if a machine can take care of them for you?

We have a limited capacity. Would you rather use it memorizing something you can have a machine take care of, or put it to better use elsewhere?

You are analyzing these tools from a position where they (presumably) provide little value to you. Imagine what your brain could do if parts of it weren't tied up with processes designed to circumvent weaknesses in the way a programming language was designed.

Some people (and even more twenty years ago) would say that a GUI is an unnecessary "crutch". Take care that you don't hold to your ideas so strongly that they (and thereby you) ossify.

Re:Trends or Crutches?

[BlueTrin]

First C compiler was probably written in asssembly ...

Re:Trends or Crutches?

[jilles]

Having a tool point out your errors and mistakes can be quite useful in learning how not to make those mistakes. Basically that improves the quality of programmer output. How is that a bad thing?

Regarding the rest of your comments. That is called progress. It's the reason that you don't have to chase your dinner with a club to kill it before you make some fire with two rocks. I can imagine several reasons why it could be useful to know how to actually do that but I'm fine with the pizza boy delivering my dinner.

C is just one of the many things in a long line of inventions. Before programming languages were invented you actually had to be intimately familiar with how the machine in front of you (or rather in the huge room you were in) processed the bits you manually fed to it and how to interpret what came back. Many such skills were lost when people started to use compilers. Most programmers today, C or otherwise, have probably only a very limited notion of how their own PC processes their instructions. C was designed for a computer architecture that is vastly more simplistic than what is used today. Most C programmers think they understand their CPU because they think they know how their C code maps 1 to 1 to processor instructions. That's naive at best these days. Do you know how to fix your CPU architecture when it breaks? Do you think that's a problem for doing your job?

To fix the JVM in case of bugs in the bytecode compiler you'd actually need those skills. That's why it's called a virtual machine. C is just a means to mediate between the machine level architecture and the virtual machine level architecture.

Re:Trends or Crutches?

[peccary]

IIRC, the first C compiler was written in... B.

Re:Trends or Crutches?

[Anonymous Coward]

Georgia Tech has the following setup:
Intro to Programming (Python)
Intro to OOP (Java)
Then two courses have that as a prerequisite:
Data Structures and Algorithms (Java)- the last 1000-level CS programming course; and Computer Organization and Programming.
This latter class starts you out with logic circuits; then you use a trimmed-down assembly language; then you use C.

Frankly, I'm glad I had the Java experience before I dove into assembly and C, because otherwise I wouldn't have had any sort of context of what I would have *wanted* my assembly code to do.

And it makes enough sense to me now that I can go back and do things the other way- I can easily see how to make data structures in C using pointers instead of object references, and they didn't even have to teach us that. It just automatically made sense once we learned how to point to a struct.

Teaching high level languages only is a sure path to failure, but starting out and learning the fundamentals of how to use conditionals and looping instead of dealing with individual assembly instructions and having to deal with trap vectors and so forth is a huge plus. It means you actually have a clue what you're trying to *accomplish* with the assembly or C code.

Once you know how to do stuff in assembly and C, it's pretty straightforward to do it in a language like Java.

Re:Trends or Crutches?

[StrawberryFrog]

True, but a C compiler must output assembly. So in order to write a C compiler, you must understand assembly.

Re:Trends or Crutches?

[BlueTrin]

Oh wow I did not even know that there was a B language =/. Is there a A language before B ?

Re:Trends or Crutches?

[SwashbucklingCowboy]

Not really. Compilers are typically written as front-ends and back-ends. The front-end translates the source code into an intermediate form and the back-end translates the intermediate form into machine language.

Re:Trends or Crutches?

[SwashbucklingCowboy]

I suspect it was written in B.

Re:Trends or Crutches?

[ioshhdflwuegfh]

My argument, however, is mostly about how we are creating a situation in which future generations are likely to become so dependent on high-level tools that the number of people who are qualified to create those tools are going to dwindle. And its not even really about software development, per se. We live you know in a technological world. Technology becoming more sophisticated, complicated, harder to understand and explain, makes the world only more of a technological world, but just because the tools are getting more complex does not mean that there will be no more those who can build those tools to specification. One part of your worries is exactly that you're afraid that there will be no more humans in the future because of the technology.

You are in fact worried that the technology will supplement natural thinking, which you deem as bad, and also believe that some special diet must be held in schools so that natural thinking is preserved and that the technology should only be kept as a supplement to the natural thinking:

[teaching the kids the tool that makes sure they won't have to actually do stuff... is] making us lazy, fat and bandy. its double-plus ungood. Now you should use your natural intelligence to figure out all the contradictions in your demands.

Re:Trends or Crutches?

[Anonymous Coward]

a lot of these tools cover for deficiencies in the language or common apis. if Java had a NotNull type annotation and the compiler enforced it and flagged any dereference of a variable that wasn't marked NotNull then you wouldn't need a lint tool to check for null pointer problems.

Re:Trends or Crutches?

[peccary]

Algol -> Cambridge Programming Language (CPL) -> Basic CPL -> B -> C

I think I recall Dennis Ritchie mentioning that the first C compiler was written in B (before rewriting it in C) but perhaps he said the language was based on B but the compiler was written in BCPL. There must be something written down on the interwebs about this.

Standard and Custom static analysis tools

[peterofoz]

Static analysis is another tool in the toolbox. Its a great indicator of overall code quality and care taken by a developer which may predict code quality during dynamic testing.
"You can put lipstick on a pig, but its still a pig".
"You can lint check bad code and add comments, but its still bad code".
At least with the static tools run early in the development process, you can identify the code pigs and make a decision to rebuild parts or team up an experienced developer with a new one. Using RegEx tools like AWK, you can even build your own static analysis tools. We did this for Y2K checking some years ago and will probably need to do it again in 2036.

To a degree, yes

[gweihir]

You actually need to tolerate a number of false positives in order to get good coverage of the true bugs. That means you have to follow-up on every report in detail and understand it.

However these things do work and are highly recommended. If you use other advanced techniques (like Descign by Contract),they will be a lot less useful though. They are best for traditional code that does not have safety-nets (i.e. most code).

Stay away from tools that do this without using your compiler. I recently evaluated some static analysis tools found that the tools that do not use the native compilers can have serious problems. One example was an incorrecly set symbol in the internal compiler of one tool, that could easily change the code functionality drastically. Use tools that work frrom a build environment and utilize the compiler you are using to build.

Re:To a degree, yes

[Anonymous Coward]

Surely any setup of a compiler independent static analysis tool would be the mimicry of the the real compiler settings/defines/types?

Re:To a degree, yes

[gweihir]

If you had read my posting, you would know that I have found instances where this is not true. Not surprising as this is very hard to get in the first place.

Jetbrains IntelliJ IDEA and Resharper

[SpryGuy]

I've used the above two tools ... the IntelliJ IDEA IDE for Java development, and the Visual Studio plug-in Resharper for C# development ... and can't imagine living without them.

Of course, they provide a heck of a lot more than just static code analysis, but the ability to see all syntax errors in real time, and all logic errors (like potential null-references, dead code, unnecessary 'else' statements, etc, etc) saves way too much time, and has, in my experience, resulted in much better, more solid code. When you add on all the intelligent refactoring, vastly improved code navigation, and customizable code-generation features of these utilities, it's a no-brainer.

I wouldn't program without them.

Re:Jetbrains IntelliJ IDEA and Resharper

[dlim]

I'll second the ReSharper. It's incredible.

Yes, absolutely

[Llywelyn]

FindBugs is becoming increasingly widespread on Java projects, for example. I found that between it and JLint I could identify a substantial chunk of problems caused by inexperienced programmers, poor design, hastily written code, etc. JLint was particularly nice for potential deadlocks, while FindBugs was good for just about everything else.

For example:

• Failure to make null checks.
• Ignoring exceptions
• Defining equals() but not hashCode() (and the other variations)
• Improper use of locks.
• Poor or inconsistent use of synchronization.
• Failure to make defensive copies.
• "Dead stores."
• Many others

At least in the Java world, I wish more people would use them. It would make my job so much easier.

My experience in the Python world is that pylint is less interesting than FindBugs: many of the more interesting bugs are hard problems in a dynamically typed language and so it has more "religious style issues" built in that are easier to test for. It still provides a great deal of useful output once configured correctly, and can help enforce a consistent coding standard.

Low startup cost and great benifits

[iamwoodyjones]

I have used static analysis as part of our build process on our Continous Integration machines and it's definitely worth your time to set it up and use it. We use FindBugs with our Java code and have it output html reports on a nightly basis. Our team lead comes in early in the morning and peruses them and assigns them to either "Suppress" or fix the issues. We shoot for zero bugs either through suppressing them if they aren't bugs or by fixing them. FindBugs doesn't give too many false positives so it works great.

Could this be just another trend?

I don't worry about what's "trendy" or not. Just give the tool a shot in your group and see if it helps/works for you or not. If it does keep using it otherwise abandon it.

What kind of changes did the tools bring about in your testing cycle?

We use it _before_ the test cycle. We use it to catch mistakes such as "Whoops! Dereferenced a pointer there, my bad" before going into the test cycle.

And most importantly, did the results justify the expense?

Absolutely. The startup cost of adding static analysis for us was one developer for 1/2 a day to setup FindBugs to work on our CI build on a nightly basis to give us HTML reports. After that, the cost is our team lead to check the reports in the morning (he's an early riser) and create bug reports based on them to send to us. Some days there's no reports, other days (after a large check-in) it might be 5-10 and about an hour of his time.

It's best to view this tool as preventing bugs, synchronization issues, performance issues, you name it issues before going into the hands of testers. But, you can extend several of the tools like FindBugs to be able to add new static analysis test cases. So if a tester finds a common problem that effects the code you can go back and write a static analysis case for that, add it to the tool and the problem shouldn't reach the tester again.

Buyer (User) Beware

[normanjd]

We use them more for optimizing code than anything else... The biggest problem we see is that there are often false positives... A senior person can easily look at recomendations and pick whats needed... A junior person, not so much, which we learned the hard way...

In short, YMMV

[Moraelin]

My experience has been that while in the hands of people who know what they're doing, they're a nice tool to have, well, beware managers using their output as metrics. And beware even more a consultant with such a tool that he doesn't even understand.

The thing is, these tools produce

A) a lot of "false positives", code which is really OK and everyone understand why it's ok, but the tool will still complain, and

B) usually includes some metrics of dubious quality at best, to be taken only as a signal for a human to look at it and understand why it's ok or not ok.

E.g., ne such tool, which I had the misfortune of sitting through a salesman hype session of, seemed to be really little more than a glorified grep. It really just looked at the source text, not at what's happening. So for example if you got a database connection and a statement in a "try" block, it wanted to see the close statements in the "finally" block.

Well, applied to an actual project, there was a method which just closed the connection and the statements supplied as an array. Just because, you know, it's freaking stupid to copy-and-paste cute little "if (connection != null) { try { connection.close(); } catch (SQLException e) { // ignore }}" blocks a thousand times over in each "finally" block, when you can write it once and just call the method in your finally block. This tool had a trouble understanding that it _is_ all right. Unless it saw the "connection.close()" right there, in the finally block, it didn't count.

Other examples include more mundane stuff like the tools recommending that you synchronize or un-synchronize a getter, even when everyone understands why it's OK for it to be as it is.

E.g., a _stateless_ class as a singleton is just an (arguably premature and unneded) speed optimization, because some people think they're saving so much by a singleton instead of the couple of cycles it takes to do a new on a class with no members and no state. It doesn't really freaking matter if there's exactly one of it, or someone gets a copy of it. But invariably the tools will make an "OMG, unsynchronized singleton" fuss, because they don't look deep enough to see if there's actually some state that must be unique.

Etc.

Now taken as something that each developper understands, runs on his own when he needs it, and uses his judgment of each point, it's a damn good thing anyway.

Enter the clueless PHB with a metric and chart fetish, stage left. This guy doesn't understand what those things are, but might make it his personal duty to chart some progress by showing how much fewer warnings he's got from the team this week than last week. So useless man-hours are spent on useless morphing perfectly good code, into something that games the tool. For each 1 real bug found, there'll be 100 harmless warnings that he makes it his personal mission to get out of the code.

Enter the snake-oil vendor's salesman, stage right. This guy only cares about selling some extra copies to justify his salary. He'll hype to the boss exactly the possibility to generate such charts (out of mostly false positives) and manage by such charts. If the boss wasn't already in a mind to do that management anti-pattern, the salesman will try to teach him to. 'Cause that's usually the only advantage that his expensive tool has over those open source tools that you mention.

I'm not kidding. I actually tried to corner one into;

Me: "ok, but you said not everything it flags there is a bug, right?"

Him: "Yes, you need to actually look at them and see if they're bugs or not."

Me: "Then what sense does it make to generate charts based on wholesale counting entities which may, or may not be bugs?"

Him: "Well, you can use the charts to see, say, a trend that you have less of them over time, so the project is getting better."

Me: "But they may or may not be actual bugs. How do you know if this week's mix has more or less actual bugs than last weeks, regardless of wh

Re:In short, YMMV

[AmaDaden]

As I understand it these things work but turning the code in to a state diagram. It maps out the code by stepping through it and replacing variables that have a values with sets of possible values. So if the code has the user enter a value for the int X then the sate of X is now what ever the user was permitted to enter. Then when the code hits an "if(X > 10)" statement the map branches so that on one branch it the state is like the IF executed and on the other it is like it did not.

The problem is that to do a full map of this would take far more processor time then any group would be willing to give so programs that do this need to take clever little short cuts. This cause the user that the parent is talking about.

Do I understand it correctly?

Re:In short, YMMV

[clem.dickey]

In short, beware of tools which produce graphs. Or worse, pie charts.

My story: management spent 25,000 on a malloc checker which produced pie charts. Our code didn't use malloc - at least not directly - so the code tested clean. Waste of money, but management liked the pie charts.

On the other hand, lint, gcc warnings (use more than -Wall, which is not really "all"), and Coverity are all good things to use.

Re:In short, YMMV

[JaneTheIgnorantSlut]

Also beware of managers who insist that each item identified by the tool needs to be somehow addressed. I inherited a body of code full of comments to the effect that "the tool says this is a a problem, but I looked at it and it is not".

Re:In short, YMMV

[TemporalBeing]

Enter the clueless PHB with a metric and chart fetish, stage left. This guy doesn't understand what those things are, but might make it his personal duty to chart some progress by showing how much fewer warnings he's got from the team this week than last week. So useless man-hours are spent on useless morphing perfectly good code, into something that games the tool. For each 1 real bug found, there'll be 100 harmless warnings that he makes it his personal mission to get out of the code.

I've found that eliminating compiler warnings will do a lot for finding bugs. Sure, there may be a number of "harmless" ones, but cleaning them up will still do a lot of good to the code too, and make the other not-so-harmless ones stand out even more. It also gives good practice for resolving the issues so that you become more proactive than reactive to bugs in the code. Just 2 cents.

Re:In short, YMMV

[giuntag]

Was it a tool by CAST software, by any chance?

The grand-unified-find-interrelations-between-your-sql-and-your-code was the most over hyped useless garbage I have met in a while.
It eg. found in delphi code that a button named the same as a db column was somehow tied to the db.
Or in php code, it was not even able to tell what KIND of databases the code would connect to, since db connections depended on ini files, and it could not grok includes based on variables.

Talk about some false positives...

Re:In short, YMMV

[Anonymous Coward]

Also beware of programmers who believe the comments of inherited code without actually looking into it.

Re:In short, YMMV

[vux984]

I've found that eliminating compiler warnings will do a lot for finding bugs. Sure, there may be a number of "harmless" ones, but cleaning them up will still do a lot of good to the code too, and make the other not-so-harmless ones stand out even more.

The last bit is the big one. If you've got a project that compiles with 3000 warnings, you don't look at them. And your stupid typeo:

if (x = y) { //do something... }

doesn't get caught...

But if your code is warning clear, when you do get the compiler warning... "Warning: x=y is an assignment not a comparison" hey you might just have saved an hour or so of debugging.

Re:In short, YMMV

[vrmlguy]

Well, applied to an actual project, there was a method which just closed the connection and the statements supplied as an array. Just because, you know, it's freaking stupid to copy-and-paste cute little "if (connection != null) { try { connection.close(); } catch (SQLException e) { // ignore }}" blocks a thousand times over in each "finally" block, when you can write it once and just call the method in your finally block. This tool had a trouble understanding that it _is_ all right. Unless it saw the "connection.close()" right there, in the finally block, it didn't count. I've tried and failed to understand what you're talking about here, specifically the bit about "the statements supplied as an array".

Also, if you have a thousand places you need to copy your "close the connection" block, that imples thousands of places where you're communicating with the database. Refactor all of those into a very few methods which include the "close connection" block, and you'll not only get fewer complaints from the static ananlysis tool, you're have better code as well.

Re:In short, YMMV

[older coder]

Certainly there are tools as clueless as the one you mention and certainly there are bosses that are equally clueless, but methinks you were burned somewhere along the way and that has swayed your thinking. IMO, these tools have come a long way. The ones I have used provide a lot of useful information. The better ones can be taught or told to ignore certain rules and can have new rules added that apply to a specific product or organization. I remember using lint early in my career and using a tool another engineer had written that could be used to ignore certain messages and could also diff the output of lint so that you could see what messages were different from a previous run. Lint was far less intelligent than what I have used recently and yet it was still useful. In fact, because it was free to use it, almost free to run it, the only cost was the time to process the output. In our group, we decided to get our code to pass lint (with the modifications shown above) before we compiled (I actually would do them in parallel). Back them, the cc compiler (using K&R C) didn't flag many errors at all. So, my bias now is to get some good tools and run them religiously. The challenge, as you say, is in how you handle the output. That decision should be left up to the developer or to the development team - never the manager. My $.02.

Re:In short, YMMV

[garaged]

keeping code warning clean would be an absolute requisite for that to work, wanna bet on percentage of FLOSS projects that keep code clean enough to see that one ??

Re:In short, YMMV

[jgrahn]

The problem is that to do a full map of this would take far more processor time then any group would be willing to give so programs that do this need to take clever little short cuts. This cause the user that the parent is talking about.

Do I understand it correctly?

Well, not exactly. Some of those short cuts can be made in a clever way (as opposed to a "clever" way) so that an analysis can take less than a decade and still be useful.

At least that's roughly what a Polyspace salesman once told me.

Re:In short, YMMV

[Moraelin]

Compiler warnings, yes, at least for a decent warning level.

Going out of the way to satisfy a tool, whose only reason to exist is to flag 10 times more stuff than -Wall, I found actually counter-productive.

And I don't mean just as in, WOMBAT (Waste Of Money Brains And Time.) I mean as in: it teaches people to game the tool, actually hiding their real bugs. And it creates a false sense of security too.

I've actually had to deal with a program which tested superbly on most metrics of such a tool. But only because the programmer had learned to game it. The program was really an incoherent and buggy mess. But it gamed every single tool they had in use.

A. to start with the most obvious, some bright guy there had come up with an own CVS script which didn't let you check in, unless you had commented every single method, and every single parameter and exception thrown. Bout damn time, eh? Wrong.

1. This forced people to effectively overwrite the comments inherited from better documented stuff. E.g., if you had a MyGizmoInterface interface, which was superbly documented, and the MyGizmoImpl class implementing it, it forced you to copy and paste the JavaDoc comments instead of just letting JavaDoc pick them from the interface. So instead of seeing the real docs, now everyone had docs all over the place along the lines of "See MyGizmoInterface.gizmoMethod()" overwriting the actually useful ones there. Or some copied and pasted comments from 1 year ago, where one of the two gradually became obsolete. People would update their comments in one of the two, but let the other say something that wasn't even true any more. Instead of having them in one place, and letting JavaDoc copy them automatically.

2. The particular coder of this particular program, had just used his counter-script or maybe plugin, to automatically generate crap like:

/**
  * Method description.
  *
  * @param x method parameter
  * @param y method parameter
  * @param idx method parameter
  * @param str method parameter
  */

I mean, _literally_. Hundreds of methods had "Method description" as their javadoc comment, and thousands of parameters total were described as "method parameter."

B. It also included such... brain-dead metrics as measuring the cohesion of each class, by the ratio between number of class members to class methods.

He had learned to game that too. His code tested as superbly cohesive, although the same class and indeed the same method, could either send an email, or render a PDF, or update an XML in the database, depending on which parameters they got. But the members to methods ratio was grrrreat.

That's really my problem with it:

A. Somewhere along the way, they had become so confident in their tools, that noone actually even checked what javadoc comments those classes have. Their script already checks that there are comments, hey, that's enough.

B. Somewhere along the way, everyone had gotten used to just gaming a stupid tool. If the tool said you have too many or too few class members, you'd just add or remove some to keep it happy. If it complained about complexity, because it considered a large switch statement to have too many effective ifs, you just split it into a several functions: one testing cases 1 to 10, one testing 11 to 20, and so on. Which actually made the code _less_ readable, and generally lower quality. There would have been ways to solve the problems better, but, eh, all that mattered was to keep the tool happy, so noone bothered.

That's why I'd rather not turn it into a religion. Use the tool, yes, but take it as just something which you need to check and use your own judgment. Don't lose track of which is the end, and which is merely a means to that end.

Re:In short, YMMV

[daedae]

There was a static analysis tool, whose name escapes me, that was based on looking for weird code patterns (like initializing a variable twice without any intervening uses). The idea was that certain things, while not bugs on their own, generally indicated a lack of understanding on the programmer's part and so going back to look at these sections of code tended to bring out other bugs. Same idea, I guess... fixing things that aren't necessarily bugs exposes things that are.

Re:In short, YMMV

[Anonymous Coward]

As someone that have been on the two sides, I can tell you that the "pretty colors" that management love are actually very very useful. My developers often had disgust to the those pretty graphs, and did everything in their power to make them useless (because they were not "perfect") without understanding they were shooting themselves in the foot because no measure is an absolute no-no to upper management ("if you can't measure it, it does not exist")

Re:In short, YMMV

[GravityStar]

First; I agree with some of the things you say. But I have to disagree on the following:

Other examples include more mundane stuff like the tools recommending that you synchronize or un-synchronize a getter, even when everyone understands why it's OK for it to be as it is.

E.g., a _stateless_ class as a singleton is just an (arguably premature and unneded) speed optimization, because some people think they're saving so much by a singleton instead of the couple of cycles it takes to do a new on a class with no members and no state. It doesn't really freaking matter if there's exactly one of it, or someone gets a copy of it. But invariably the tools will make an "OMG, unsynchronized singleton" fuss, because they don't look deep enough to see if there's actually some state that must be unique. A unsynchronized singleton is not a problem on a singlecore CPU system. But it may fail on SMP system or on a multicore CPU system, and when it does fail, it may do so spectacularly.

I'll explain: You create the unsynchronized singleton on CPU1. CPU2 now retrieves the singleton. But it's possible CPU2 has out-of-date cache on the memory that is occupied by your singleton. If that happens, your singleton will not work correctly.

Don't create a unsynchronized singleton. Either synchronize; or create the singleton in a static initializer; or just call new anytime you need a instance of that class.

A link to a related problem:
http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

You may believe I'm wrong on some subtle point, or for some particular use case. I'm not. In Java, do not use a unsynchronized singleton on a multithreaded system.

Re:In short, YMMV

[Emb3rz]

He's saying that he's closing any of a number of database connections -- which one specifically depends on which object variable he passes to the function. This is, in his opinion, a better way to solve the problem as opposed to manually writing/copy+pasting the code that is, aside from which connection object it references, identical.

Re:In short, YMMV

[Allador]

Wow, nice story. Quite amazing, as you'd think the devs could just do things the right way for not much more work than gaming the tools.

This kind of thing though, is ultimately a failure of management, whoever leads/runs the dev team. They should be able to see this kind of thing happening and either apply some proper motivation, change the procedures, or let some bad devs go.

Mind you, bad developers as well. But if I were the owner, the dev mgr would get the brunt first on something like this.

Re:In short, YMMV

[jlarocco]

Does that make his point any less valid? Just because some "FLOSS projects" don't clean up their warnings, doesn't mean the rest of us shouldn't.

Re:In short, YMMV

[KlomDark]

Ask a stupid person if they are a singleton or a simpleton.

Wait, that would be mean.

Re:In short, YMMV

[Elrond, Duke of URL]

I'd have to agree with that. If you make a habit of getting rid of compiler warnings as you go, it's very little extra work and can catch some actual bugs.

And, as you said, it makes new problems stick out like a sore thumb and they are far less likely to be overlooked or ignored.

Re:In short, YMMV

[DarkGreenNight]

FLOSS? Talk about commercial code too.

When I compiled the first time at work, after university where I always compiled marking the warnings as error), I was flabbergasted by the sheer amount of warnings that were rampant in the code.

Re:In short, YMMV

[Anonymous Coward]

If you have that level of trouble, find a new place of work. Seriously.

Coverity Prevent Rocks

[Arakageeta]

My large C/C++ project (2,000,000+ SLOC) started using Coverity Prevent about a year ago. Its results have truly been invaluable. We simply have too much code for standard human code reviews or for detailed run-time coverage analysis (ex. Insure* or valgrind). Prevent has caught many programming errors (some extremely obscure and/or subtle) and saved use a ton of money and time.

* I really like Insure, but it is difficult to set up on a system composed of many shared libraries. However, there are some bugs that really need run-time analysis to catch.

Re:Coverity Prevent Rocks

[wili]

The parent's claim about having "too much code for standard human code reviews" is just laughable. Proper reviewing takes a fraction of the time writing the code in the first place does. How do you end up in a situation where you have people banging out millions of lines of code but no-one has time to do human code reviews? Is all of the code machine-generated? Also, why can't the parent do detailed run-time coverage analysis? Is his team lacking a testing framework for this? Why are their banging out millions of lines of code without proper run-time tests?

Re:Coverity Prevent Rocks

[FlaSheridn]

My day job is using Prevent, so I'm obviously biased, but it does catch a lot of important bugs when used properly--which does take some work. Among other things, you should probably adjust the set of checkers it runs, disabling some of the default ones, and enabling a lot of the non-default ones. (If you're already a customer, Coverity's forum has a discussion thread about this; I'd urge you to contribute your experiences.) It's good about letting you mark false positives, as well as the far more common false don't-cares, so that you only see them once--and marking a bunch of false positives is a lot faster than finding its bugs would be without static analysis.
Prevent's defect manager has an option to compare successive runs of the tool, which I've found useful for tracking bugs which have been fixed by developers _not_ using the tool. If a developer fixed some bugs independently, that's independent validation of the value of those bugs.
A previous poster complained about using its metrics, and I think I disagree, though we're not using it that way. It does look to me like the areas where it has the highest defect density (even allowing for false positives) are the areas of our code which need the most attention. The usual disclaimers apply: You can only use metrics as a tool for making judgments, not a replacement, and you've got to avoid the usual "Whatever we can measure easily is all that matters" fallacy.

Count also the forming of good programming habbits

[deep-deep-blue]

Another good point for using lint, is that after a while a programmer learns the the way, and the outcome is a better code in a shorter time. Of course I also found that are a few ways to avoid lint errors/warnings in a way that lead to some very ugly bugs.

Infinite loop detector

[nuttyprofessor]

How about a tool that will tell me if my program will
eventually halt or not for a given input? I'd pay big money for that!

Re:Infinite loop detector

[FunkyELF]

This can be accomplished with an emulator or a VM.
Run the code you wanna test in a VM and give it the input.
If the state of the virtual machine is ever exactly the same as it was previously, you have a loop, don't you?
The virtual machine only has so many possible states, and the input is finite as well.


The machine is deterministic, so if you're at state X while after reading input Y, you do 1.23 million operations and you're back at state X again while not moving to the next chunk of input you'll hit state X again in exactly 1.23 million operations no?


I think you can solve the halting problem for finite deterministic machines. The only machines it doesn't work on are theoretical ones with infinite memory.
The problem is that to solve the halting problem for a real machine, you need a machine with more memory than particles in the known universe.


On a machine with 4gb of memory there would be 2 ^ (4 * 1024 * 1024 * 1024 * 8) different states in memory alone.
I remember reading somewhere about brute force decryption of somethign encrypted with 1024 bit key with todays microprocessors...
Just having a chip that would sequentially count to 2 ^ 1024 would require more energy than the sun puts out in 10 years.
So having a machine do anything usefull with that would be even worse.
If the virtual machine has X bits of memory, you'll need 2 ^ X bits of memory to keep track of which states have been visited.
Then you start getting into that whole thing about the number of known particles in the universe being less than the number of bits needed.

Re:Infinite loop detector

[nuttyprofessor]

If you can determine if the following program will
ever halt then you will have solved a problem that
no mathematician has solved!

main() {
    bigint n, a, b, i;
    for (n = 1; ; n++) {
          a = n*n;
          b = (n+1)*(n+1);
          for (i = a+1; i< b && !prime(i); i++)
                ;
          if (i == b) break; /* no prime found, halt program */
    }
}

Re:Infinite loop detector

[FunkyELF]

Again, in the real world with a real machine with a finite amount of memory, your bigint can only be as big as your memory will allow.
I'm not saying it wouldn't take millions of years. I'm not saying that it wouldn't take more memory than you can fit on Earth.

Re:Infinite loop detector

[nuttyprofessor]

If your computer runs out of memory, take a snapshot
of its state and continue on a computer with more memory.
Keep this up on newer generations of computers -- pass
it off to your ancestors -- continue until the sun turns
into a red giant if need be. Hopefully we will have developed
quantum computer which can hold more memory than there
are particles in the universe. Keep it going. Will it ever halt?

Many of never all

[mugnyte]

  Short version:

      There are real bugs, with huge consequences, that can be detected with static analysis.
      The tools are easy to find and worth the price, depending on the customer base you have.
      In the end, that cannot detect "all" bugs that could arise in the code.

  Worth it?
      Only you can decide, but after a few sessions learning why tools flag suspect code, if you take those suggest to heart, you will be a better coder.

Yes

[DerekSTheRed]

My company uses Ounce for static analysis. It's great at helping find the tedious bugs and potential security violations. For instance, server side input validation for web sites is much easier and more accurate with static analysis. As others have said, it's not perfect but having an extra set of digital eyes looking at the code helps. The sooner bugs are fixed in the development cycle, the cheaper it is to fix.

Absolutely

[Fippy Darkpaw]

At my little corner of Lockheed Martin we use Klocwork and LDRA to analyze C/C++ embedded code for military hardware. Since the various compilers for each contract aren't nearly as full-featured as say, Visual Studio or Eclipse, I've found static code analysis tools invaluable. Can't comment on the cost/results ratio though, since I don't purchase stuff. =)

Linux kernel devs use sparse for static analysis

[ncw]

The linux kernel developers use a tool originally written by Linux Torvalds for static analysis - sparse.

http://www.kernel.org/pub/software/devel/sparse/

Sparse has some features targeted at kernel development - for instance spotting mixing up kernel and user space pointers and a system of code annotations.

I haven't used it but I do see on the kernel mailing list that it regularly finds bugs.

signal to noise

[www.sorehands.com]

That is the problem for a beginner. When you first configure PC-Lint you need to tune the configuration to ignore stuff that you don't have a problem with, ie. assignments within a test. After than you need to configure your project for lint, setting up the lint files to include the correct headers and such. Then the noise is not too bad. Just make sure when you think something is noise, it is not really noise.

Re:signal to noise

[McGregorMortis]

If you're tuning it to ignore assignment within a test , ie "if( x=y ) {}", then you're missing the one of the great points of using PC-Lint.

That code is simply in poor taste, even if it works. What PC-Lint, and good taste, say you should do is change the code to "if( (x=y) != 0 ) {}". This will satisfy PC-Lint, and also makes your intention very clear to the next programmer who comes along. And, best of all, it doesn't generate a single byte of extra code, because you've only made explicit what the compiler was going to do anyway.

Re:signal to noise

[Anonymous Brave Guy]

When you first configure PC-Lint you need to tune the configuration to ignore stuff that you don't have a problem with, ie. assignments within a test.

The problem is, sometimes that is idiomatic and correct, and sometimes it's a bug. You can't just blanket disable it without losing some of the benefit. In fact, I would say that is the typical problem a beginner causes: instead of fixing the problem, hack around it, even though you lose some good things as a side-effect.

As another responder has already pointed out, your particular example is unfortunate anyway, because you can always code around it explicitly if you really mean it. Many other things aren't so easy, though. For example, Lint is always whining about possible out-of-bounds array access when in fact the code is provably safe, but you surely wouldn't want to miss a genuine potential out-of-bounds error.

Re:signal to noise

[Joutsa]

With PC-lint, it is this easy to disable a warning for just one line:

x = *(p + 10); //lint !e416 "Potential buffer overflow". I know what I am doing here

Re:signal to noise

[Anonymous Brave Guy]

Sure, but as I pointed out earlier in the thread, this is a flawed solution: if you ever change that code (or perhaps something it depends on), it might no longer be appropriate to suppress the Lint warning, but the comment won't go away by itself.

Besides, any comment where a developer has written "I know what I am doing here" should automatically be flagged for triple independent code review, since it is automatically in the most likely 1% of places in the code base to find a serious error. ;-)

Re:signal to noise

[Joutsa]

Sure, but as I pointed out earlier in the thread, this is a flawed solution: if you ever change that code (or perhaps something it depends on), it might no longer be appropriate to suppress the Lint warning, but the comment won't go away by itself.

I don't think that changing the code itself is a problem - the lint warning suppression is a big red flag for everyone who is editing the code. A bigger problem is that the underlying data structure that cannot be accessed without questionable pointer arithmetic might change. Even then, it will be easier to grep for that particular warning suppression than for every use of the data type.

WHOA... nice timing

[w00f]

YOU sir, have amazing timing! I just wrote a 2-part article on this topic! Interesting... mine was published http://portal.spidynamics.com/blogs/rafal/archive/2008/05/06/Static-Code-Analysis-Failures.aspx The Solution: http://portal.spidynamics.com/blogs/rafal/archive/2008/05/15/Hybrid-Analysis-_2D00_-The-Answer-to-Static-Code-Analysis-Shortcomings.aspx Comments welcome!! Interesting that this topic is getting so much attention all of the sudden

Re:Linux kernel devs use sparse for static analysi

[ncw]

s/Linux Torvalds/Linus Torvalds/ - I keep making that typo ;-)

make up for language deficiencies

[nguy]

Generally, these tools make up for deficiencies in the underlying languages; better languages can guarantee absence of these errors through their type systems and other constructs. Furthermore, these tools can't give you yes/no answers, they only warn you about potential sources of problems, and many of those warnings are spurious.

I've never gotten anything useful out of these tools. Generally, encapsulating unsafe operations, assertions, unit testing, and using valgrind, seem both necessary and sufficient for reliably eliminating bugs in C++. And whenever I can, I simply use better languages.

Re:make up for language deficiencies

[Anonymous Coward]

"these tools make up for deficiencies in the underlying languages"..."whenever I can, I simply use better languages."

Exactly what i was thinking; But language-idioms bug-pattern-matching would be great to have in Blackbox, EiffelStudio or Dolphin SmallTalk anyway

Man on the moon - pencil, slide rule and computer!

[Folger]

Ever heard of the Apollo Guidance Computer ? Even if they had several computers on board the space vehicles, they surely used computers to design the space vehicles.

We use Compuware DevPartner Studio

[gravy.jones]

At my company, we use Compuware DevPartner Studio and have found it to be a very comprehensive package. I have used it for performance optimization, memory leak detection, and resource misuse. I have not used its ability to find "dead" code, but that exists. It plugs into Visual C++ 6 and Visual Studio .NET and it takes minimal time to get used to it. Others may know of it from its legacy name "Bounds Checker"

Watch the differences!

[tikal_work]

Something that we've found incredibly useful here and in past workplaces was to watch the _differences_ between Gimpel PC-Lint runs, rather than just the whole output.

The output for one of our projects, even with custom error suppression and a large number of "fixups" for lint, borders on 120MiB of text. But you can quickly reduce this to a "status report" consisting of statistics about the number of errors -- and with a line-number-aware diff tool, report just any new stuff of interest. It's easy to flag common categories of problems for your engine to raise these to the top of the notification e-mails.

Keeping all this data around (it's text, it compresses really well) allows you to mine it in the future. We've had several cases where Lint caught wind of something early on, but it was lost in the noise or a rush to get a milestone out -- when we find and fix it, we're able to quickly audit old lint reports both for when it was introduced and also if there are indicators that it's happening in other places.

And you can do some fun things like do analysis of types of warnings generated by author, etc -- play games with yourself to lower your lint "score" over time...

The big thing is keeping a bit of time for maintenance (not more than an hour a week, at this point) so that the signal/noise ratio of the diffs and stats reports that are mailed out stays high. Talking to your developers about what they like / don't like and tailoring the reports over time helps a lot -- and it's an opportunity to get some surreptitious programming language education done, too.

Doesn't do me any good

[Anonymous Coward]

Static analysis tools never work for me, I don't declare anything static.

Re:s and x

[Anonymous Coward]

Linus is not Linux

Re:Man on the moon - pencil, slide rule and comput

[bsDaemon]

Yes, I have heard of it. However, it's hardly a Quad Xenon workstation, is it? That's still just part of what I'm talking about. Their computers took up rooms (on the small side), and were less powerful than some calculators today.

It's like when Wheeler died and he was called "one of the last great titans of physics." One fellow slashdotter called this an unfair characterization as it is unfairly biased against the people who are doing work today, which he sees as no less important, comparing it to if say, Linus Torvalds were to die and was called one the "last great titans of computing."

The best computers they had available when they were designing the bomb were like UNIVAC. They had to do their math by hand, and much of their calculations were largely based on assumptions. These days the computer does a lot of the work and doing things by hand is for "suckers."

Comparing Wheeler to Linus is absurd. It'd have been more poinignt to compare Wheeler to Grace Hopper or someone, frankly.

That's just my two cents though. Your exchange rate my vary.

open source tool list?

[fraselsnarf]

"...Many of these (for Java, at least) are open source, so the expense in negligible."

can you list a few? This summer I am picking back up a couple of old embedded projects, and am interested in learning more about the open source embedded tools available.

Re:open source tool list?

[Gavagai80]

http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

To summarize...

[kclittle]

Static analysis is a tool. In good hands, it is a valuable tool. In expert hands, it can be invaluable, catching really subtle bugs that only show up in situations unlike anything you've ever tested -- or imagined to test. You know, situations like what your customers will experience the weekend after a major upgrade (no joking...)

Re:who proved Astrée ...?

[zimtmaxl]

It may be the best tool in the world - I admit I do not know it. But the word "proved" makes me suspicious. To me this sounds like the typical - and wide spread - management speak to make business decision makers and their insurrers sleep well. Thank you! This gives the perfekt example that the misleading wording is even used by educational bodies.
Is this is a proof or do some mistakenly think they're safe?

Who "proved" Astree to be error free in the first place?!

Static analysis is part of the basics

[Incster]

You should strive to make your code as clean as possible. Turn on maximum warnings from your compiler, and don't allow code that generates warnings to be checked in to your source repository. Use static analysis tools, and make sure your code passes without issue there as well. These tools will generate many false positives, but if you learn to write in a style that avoids triggering warnings, quality will go up. You may be smarter than Lint, but the next guy that works on the code may not be. Static analysis tools are just another tool in the tool box. Also use dynamic analysis tools like Purify, valgrind, or whatever works in your environment. Writing quality code is hard. You need all the help you can get.

Define "work"

[crmartin]

They will indeed find certain classes of bugs, and code that is lint-free (especially with more modern versions of lint has fewer defects. Other metrics, like McCabe cyclomatic complexity, can also point out areas in which bugs have a high probability.

On the other hand, no tool can find 100 percent of bugs. This is a theorem (via Turning's halting and equivalence theorems), and also because some bugs are places where the code is doing what it was supposed to do, but that isn't what the user actually wanted.

Coverity

[Anonymous Coward]

I've used coverity. It's not cheap, unless your project is opensource, but it is very good. Very low false positive rate, and finds some stuff that would be pretty tricky to find through code reviews etc due to its ability to track things all across the project.

Preventing bugs is only part of what it gives you

[McGregorMortis]

I use PC-Lint religiously, and have been for maybe 18 years now. In my opinion, it has made me a better programmer. It teaches you to be disciplined, because whenever you get sloppy, it barfs all over your disgusting code. The discipline you learn from it remains even after you stop using it. You start to think like Lint.

The First Commandment for C Programmers states "Thou shalt run lint frequently and study its pronouncements with care, for verily its perception and judgment oft exceed thine."

This is as true now as it was when the prophet (PBUH) spoke.

I love PC-Lint, but it does have some drawbacks. They added C++ support to it, but it really is having a hard time with the crazy template meta-programming tricks found in the Boost library. Using Boost will bury you under an avalanche of spurious warnings. Sometimes, it will even crash. In this case, studying its pronouncements with care is liable to give you a stroke.

Coverity: we like it

[Anonymous Coward]

We've had fantastic luck with Coverity. It's expensive, but it catches a lot of bugs with a very low false positive rate. It's good about explaining, step by step, "okay, what if this variable has this value, and this if/else goes this direction, and this other thing happens..." so you can tell why it gives the error in question.

So the false positives are pretty much always when an environment variable or a file contains something we don't expect it to. And those are only sort of false positives -- those things *could* actually happen, we just don't guarantee the results if they do :-)

Yup.

[Anonymous Coward]

People do need to keep in mind that the statically typed languages that they most commonly work with are quite primitive when it comes to type systems. It's no surprise, for example, that a static analysis tool can find spots where one can get null pointer exceptions, since it's easy to design a language with a type system that guarantees they can't happen.

To make such a language, you just have to distinguish the types of variables that are allowed to be null, from the type of those that aren't, and force code that deals with the former to deal with the possibility the variable holds null. Nice does this; and of course, the general labeled union types available in ML and Haskell make nulls completely unnecessary.

Not really new

[Archtech]

Static analysis tools are by no means new. DEC, for instance, had DEC Source Code Analyzer (SCA) on the market in the mid-1980s. Logically, it was implemented as an add-on option to the DEC Language-Sensitive Editor (LSE). That way, SCA could assume all of LSE's functionality, and concentrate on adding extra features. Moreover, the arrangement ensured absolute consistency between the two products.

Of course static analysis cannot find all the problems in a piece of code. But when used in conjunction with compilers, editors, profilers, coverage analyzers, and automated test systems, it adds another piece to the jigsaw. And it's obviously better to detect errors sooner rather than later, thus shortening the code-test loop.

Languages with built-in static analysis

[WatersOfOblivion]

I've never used lint or any of the other tools being talked about, but I have experience with the ML family of languages (OCaml, mainly), which have type-inference. Type inference is static analysis and catches, in my experience, a great deal of the bugs caught by those other tools, and better yet it's a core part of the language. Having to be aware of types forces you to catch the bugs before you make them, and your code won't compile until you've fixed the bugs. It, of course, doesn't solve logic bugs, but just about everything else gets caught. Obviously, this doesn't work for existing an codebase (and so this post is OT), but for greenroom projects, it is worth investigating.

Static analysis induces lazyness.

[stengah]

I would rather put my efforts in developping my code with a safe-by-construction language (for instance with the Coq proof assistant), or use tools like Why to guarantee that my code won't blow up. Time spent debugging with static analysis tools should have been spent proving the correctness of the program beforehand.

ROI

[Anonymous Coward]

I used to work for a static analysis company. You get the most return for the first run.

The analyzer, like any automaton, matches patterns based on a set of criteria. These criteria are often simple in nature, as they are simple to implement. The more complex software defects, such as cross functional variable reference tracking are often ignored. Complex matching can be done, but it's simply too expensive to implement and may produce a large number of false positives. Run time errors, obviously are not static and cannot be caught. Running against C will return much higher quality results than against C++, as the object oriented model is almost impossible for the analyzer to follow.

I second valgrind

[jberryman]

There's also valgrind, for Linux users

It's great for finding all those elusive bits of code that might be accidentally seeding a pseudo-random number generator somewhere.

KWIM

[KGBear]

Show me a program than will know what I mean when I say (or type) "allow my friends to see my stuff" and does what a human would do. Then I'll start believing in automated debugging.

Re:KWIM

[james.widman]

But this discussion isn't about automated *debugging*; it's about the automated *detection* of certain kinds of bugs. That's very different: with the former, you're asking the machine to actually make a change to the code, and I think we can all agree that at present this facility exists only in science fiction.

But with automated bug *detection*, you're merely asking the machine to issue diagnostics about constructs that could lead to undefined behavior. A machine can be well-suited to this task, because there are many kinds of bugs whose detection does not require any knowledge of the intentions of the human programmer; some bugs are just going to make the program crash. And static analysis tools have had a great deal of success at flagging those kinds of bugs for several decades now (not to mention the modest success that can be enjoyed with the limited static analysis provided by modern compilers).

For example, a static analyzer can perform a data flow analysis (similar to analyses performed by compilers for the purpose of determining which optimizations would be safe), and in the process determine whether it's possible for an object to be accessed while it's in an uninitialized state. Typically, static analysis tools also process each translation unit in a program, and so are able to report about things like unreferenced entities; violations of the One Definition Rule (which are generally invisible to compilers, since they only see one translation unit per run); the polymorphic destruction of an object whose class's destructor is non-virtual; and many other things that can go wrong.

Re:KWIM

[KGBear]

http://www.gergely.risko.hu/debian-dsa1571.en.html

Q.E.D.

A double edged sword

[xenocide2]

Static analysis tools are common in the open source world. The lint name is well known enough that many projects make theirs a pun on it, ala lintian. A few years ago a local root exploit in X was discovered by running these sorts of checks. But generally, static analysis tools require human review -- with large code bases they generate large numbers of false positives, especially the dumber ones. This leads to trouble for perfectionists, a common trait among software developers interested in bug fix analysis. For example, the recent massive Debian vulnerability was caused by an overzealous developer trying to fix static analysis flags. One of these flags was valid, one was not, and removing both removed nearly all entropy from the RNG.

In the more general sense, static analysis cannot find all bugs. There's a trivial proof: a program stuck in an infinite loop is a bug, but finding all such loops would solve the halting problem. Handling interrupts and the like also causes reasoning problems, as it's very hard, if not computationally intractable, to prove multi-threaded software is safe. So static analysis won't rid the embedded world of watchdog timers and other software failure recovery crap.

Re:A double edged sword

[Free the Cowards]

For example, the recent massive Debian vulnerability was caused by an overzealous developer trying to fix static analysis flags. One of these flags was valid, one was not, and removing both removed nearly all entropy from the RNG. I'm curious as to where this came from, because I'm seeing it a lot, but it's not actually correct.

Both lines in question were equally valid. The trouble is that the lines which accessed uninitialized memory were also doing real work. The first time they were called, they would read uninitialized memory and mix the new randomness into it. On subsequent calls, it would do the same thing, but the buffer would have the previous randomness in it rather than just uninitialized memory.

So both places were accessing uninitialized memory and were equally "wrong". The reason they were doing this was simply because it made no difference whether it was initialized or not, and thus it was easier not to initialize it.

The correct fix to the warning would have been to initialized the buffer prior to the first access. Instead, the developer commented out the lines which accessed it, which ended up disabling the whole thing.

Re:A double edged sword

[xenocide2]

I think you misunderstand. One of the lines flagged did something worthless (add uninitalized memory into the pool or something I cannot recall) and the other did something important (add sources of entropy). The patch that fixes it undoes only one of the comments. The other has been argued to be of little consequence either way. Adding a fixed constant to an entropy pool doesn't make it less random, which is basically what it does since the memory's unlikely to have been touched by that program.

At any rate, this was definitely a case of an overzealous perfectionist removing memory debugger warnings, and an upstream that does a borderline practice without explanation in code or the FAQ.

Re:A double edged sword

[Free the Cowards]

Maybe I do misunderstand, but as far as I know this isn't quite it. The standard way to add entropy to a pool is to use a cryptographic hash, and perform the operation pool = hash(pool + entropy). In this case, the pool started out uninitialized, so the first call to the hash function made valgrind complain even though it was harmless. Then the culprit commented out the whole pool update, thus destroying the effectiveness of the pool.

Another line to use.

[Ungrounded Lightning]

Me: "ok, but you said not everything it flags there is a bug, right?"
Him: "Yes, you need to actually look at them and see if they're bugs or not."
Me: "Then what sense does it make to generate charts based on wholesale counting
            entities which may, or may not be bugs?"
Him: "Well, you can use the charts to see, say, a trend that you have less
          of them over time, so the project is getting better."
Me: "But they may or may not be actual bugs. How do you know if this week's
          mix has more or less actual bugs than last weeks, regardless of what the
          total there is?"
Him: "Well, yes, you need to actually look at them in turn to see which are actual bugs."
Me: "But that's not what the tool counts. It counts a total which includes an
            unknown, and likely majority, number of false positives."
Him: "Well, yes."
Me: "So what use is that kind of a chart then?"
Him: "Well, you can get a line or bar graph that shows how much progress
          is made in removing them."

Your next line is:

Me: "So you're selling us a tool that generates a lot of false warnings
          and a measurement on how much unnecessary extra work we've done to
          eliminate the false warnings. Wouldn't it make more sense not to use
          the tool in the first place and spend that time actually fixing real bugs?"

To work this question must be asked with the near-hypnotized manager watching.

Re:Another line to use.

[Allador]

The 'me' in this case is missing the point.

You dont just run the tool over and over again and never adapt it to your code.

If it produces a bunch of false positives, then you go in and modify the rules to not generate those false positives.

Thats half the point of something like this, you need to tune it to your project.

The flip side is that if you see some devs over and over making the same kind of mistake, well you can write a new rule in it to flag that kind of thing.

If you have an endless number of false positives, that doesnt ever go down, then you are either:

1. Not using the tool correctly.
or
2. Not working on a project that is amenable to this tool.

IME, the vast majority of time its #1. Now you may find that for certain small or narrowly scoped projects, or those worked on by 2 super-gurus, that the overhead for learning and tuning the tool for that project isnt worth it. But thats something you'd have to find out yourself, and it differs from project to project.

Re:Another line to use.

[benhattman]

Me: "ok, but you said not everything it flags there is a bug, right?" Him: "Yes, you need to actually look at them and see if they're bugs or not." Me: "Then what sense does it make to generate charts based on wholesale counting entities which may, or may not be bugs?" Him: "Well, you can use the charts to see, say, a trend that you have less of them over time, so the project is getting better." Me: "But they may or may not be actual bugs. How do you know if this week's mix has more or less actual bugs than last weeks, regardless of what the total there is?" Him: "Well, yes, you need to actually look at them in turn to see which are actual bugs." Me: "But that's not what the tool counts. It counts a total which includes an unknown, and likely majority, number of false positives." Him: "Well, yes." Me: "So what use is that kind of a chart then?" Him: "Well, you can get a line or bar graph that shows how much progress is made in removing them." Your next line is: Me: "So you're selling us a tool that generates a lot of false warnings and a measurement on how much unnecessary extra work we've done to eliminate the false warnings. Wouldn't it make more sense not to use the tool in the first place and spend that time actually fixing real bugs?" To work this question must be asked with the near-hypnotized manager watching. Meh, I would respond with a second graph based on the first one titled "Static Code Analysis Warnings Before/After This Consultant Was Hired." It would show the warnings produced by the tool one week before your org started using it vs the warnings each week since the consultant was hired.

Maybe that would be enough to convince him that a more warnings from the tool does not necessarily mean he'll keep his job.

Horses for courses

[Simon Brooke]

Yes, they kind-of work; it depends what you're doing and what you're trying to achieve.

Example: recently I was doing a quality audit of a bit of code which ran an industrial control unit in a safety critical application. The software was written using an obsolete, closed source C compiler and the processor was an old Hitachi; there is a GNU C compiler for it but because the code was non-ANSI we couldn't use that (the object was to audit the existing code for safety, not to change it). So dynamic analysis of the running code, other than black box testing of the complete controller, was not possible.

My job was to demonstrate that the code could not fail unsafe. I used QA/C, which I found very useful, and VectorCast, which turned out not to be useful on this particular project because it needs to interact with the compiler. The compiler would only run under 16 bit DOS, VectorCast under 32 bit Linux or Windows, so it proved to be impossible to get them to communicate (this doesn't mean VectorCast wouldn't be useful on other projects).

In summary, you wouldn't want these to be the only tools in your audit toolbox. But to get to understand a piece of not-very-well structured legacy code quickly, they're pretty useful.

false positives vs. false negatives

[Chris Snook]

Because they cannot solve the halting problem, there are many instances where they will see a questionable piece of code, and have to decide whether they should flag it and risk a false positive, or ignore it and risk a false negative. This is where the magic happens, at least in the high-end commercial code analysis tools. If it always errs on the side of false positives, the output will be ignored in all but the most thoroughly audited fields. If it always errs on the side of false negatives, it's worthless. A lot of work goes into analyzing which practices commonly cause problems in the real world, and fine tuning the problem detection code to look for those, while perhaps passing up certain classes of bugs that are very rare and very computationally difficult to identify.

Re:false positives vs. false negatives

[Anonymous Coward]

I believe you are just trying to spout terms you learned in college. Most of the problems that can be found by these tools have absolutely nothing to do with "the halting problem."

Re:false positives vs. false negatives

[Chris Snook]

For languages where no free static analysis tools exist, you're probably right, but if you're using C, things like lint will find the simple stuff that doesn't require analyzing execution paths. Taking it to the next level requires entering halting problem territory, and using heuristics or configuration options to limit your search to things that are likely to be problems and avoid spewing garbage that doesn't matter.

Javascript tools

[mandreiana]

What Javascript tools do you recommend? (similar to Findbugs for Java) What about ActionScript? Thanks

Re:who proved Astrée ...?

[johannesg]

Hey, it's just the French. I deal with them regularly, and *any* software they write is always "perfect" even if it can be demonstrated to be a piece of crap in front of half a dozen decision makers. It's a cultural thing I think.

As to how they proved Astree with perfect, they ran it on itself until they had zero errors, obviously...

Re: Coverity Reports Open Source Security Making G

[Anonymous Coward]

Coverity does not check for unitialize variables that is float or double.

Lint has never found any bugs for me.

[Anonymous Coward]

Lint has never found any real bugs for me. Not one. Nor have a couple other C++ static analyzers that I've tried (though I've felt that Coverity might still be useful even though it hasn't found bugs for me). People say that these code analyzers are invaluable, but I think it depends on the quality of the programmer. Some people might not like to hear it, but the fact is that some programmers are much more accurate than others.

However, by saying that these tools never find bugs for me, I'm not saying that my code is bug-free. The existing bugs in my code tend to be higher level logic bugs that these tools don't understand.

Re:who proved Astrée ...?

[Daishiman]

From the website:

ASTRÃE analyzes structured C programs, with complex memory usages [17], but without dynamic memory allocation and recursion. This encompasses many embedded programs as found in earth transportation, nuclear energy, medical instrumentation, aeronautic, and aerospace applications, in particular synchronous control/command such as electric flight control [22], [23]. There is a load of research on proving program correctness for limited subsets of languages (especially with no dynamic memory allocation). On functional languages these proofs can be almost trivial, not so much on imperative languages.
Correctness proofs go as far back as computer science itself, so it's an extensively studied area, which is unfortunately mostly relegated to academia on most cases because of the restrictions it places on the expressive power of such systems. But don't underestimate its ability to not get people killed.

they are useful

[elmartinos]

Its not a trend, it is something developers have been doing for a long time. We have a build system here that automatically compiles and runs unit tests, and when something fails the developers gets an email. We try to automate as much as possible, so we also have several static code analysis tools like PMD, Findbugs, Checkstyle installed. All of them are not perfect, but they all detect at least some problems; its better than nothing. It is also important that these tools can be switched off so that they don't get annoying. PMD does this very nicely, you can disable checks on a method based granularity with a simple annotation at places where appropriate.

KDE uses

[oever]

The English Breakfast Network

It helps to find potential problems and also checks for licenses, spelling errors and coding conventions.

Halting problem bullshit

[Animats]

Several posters have cited the "halting problem" as an issue. It's not.

First, the halting problem does not apply to deterministic systems with finite memory. In a deterministic system with finite memory, eventually you must repeat a state, or halt. So that disposes of the theoretical objection.

In practice, deciding halting isn't that hard. The general idea is that you have to find some "measure" of each loop which is an integer, gets smaller with each loop iteration, and never goes negative. If you can come up with a measure expression for which all those properties are true, you have proved termination. If you can't, the program is probably broken anyway. Yes, it's possible to write loops for which proof of termination is very hard. Few such programs are useful. I've actually encountered only one in a long career, the termination condition for the GJK algorithm for collision detection of convex polyhedra. That took months of work and consulting with a professor at Oxford.

The real problem with program verification is the C programming language. In C, the compiler has no clue what's going on with arrays, because of the "pointer=array" mistake. You can't even talk about the size of a non-fixed array in the language. This is the cause of most of the buffer overflows in the world. Every day, millions of computers crash and millions are penetrated by hostile code from this single bad design decision.

That's why I got out of program verification when C replaced Pascal. I used to do this stuff.

Good program verification systems have been written for Modula 3, Java, C#, and Verilog. For C, though, there just isn't enough information in the source to do it right. Commercial tools exist, but they all have holes in them.

Re:Halting problem bullshit

[stengah]

Indeed. In fact, most of the interesting algorithms come with a proof of correctness and termination. The problem is, traditional programming languages are not expressive enough to specify and prove termination within the language itself. But there are many ways to achieve this : either annotate the program with logical assertions, allowing an external automated demonstration program to *try* to check soundness and completness; or program directly with Coq or PVS. Some links : http://why.lri.fr/ & http://coq.inria.fr/.

Re:Halting problem bullshit

[The_reformant]

I have angel processes which restart my loops when they terminate.

Re:Halting problem bullshit

[A R Baboon]

The real problem with program verification is the C programming language. In C, the compiler has no clue what's going on with arrays, because of the "pointer=array" mistake. You can't even talk about the size of a non-fixed array in the language.

__builtin_object_size works rather well if you need the compiler itself insert instrumentation. C affords users the option to have API defined zero-copy references, instrumented (reference tracked) zero-copy references, or copying. It sounds like you have interacted with programmers who have made poor choices.

This is the cause of most of the buffer overflows in the world. Every day, millions of computers crash and millions are penetrated by hostile code from this single bad design decision.

Once again, the decisions are available to a programmer. Better compilers provide the needed tools. Sounds like you want -fstrict-aliasing

That's why I got out of program verification when C replaced Pascal. I used to do this stuff.

Good program verification systems have been written for Modula 3, Java, C#, and Verilog. For C, though, there just isn't enough information in the source to do it right. Commercial tools exist, but they all have holes in them.

I think you need to read:
http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html
and:
http://gcc.gnu.org/onlinedocs/gcc/Variable-Attributes.html

Compiler attribution combined with splint directives like /*@fallthrough@*/ in a switch statement make software rather checkable without sacrificing speed or readability.

Re:who proved Astrée ...?

[fbjon]

Who "proved" Astree to be error free in the first place?! The creators of Astrée, presumably. Proving in the scientific sense that a piece of software is correct can definitely done, it's just really expensive most of the time. In any case they claim that Astrée is sound, i.e. catches all errors, but that the precision can be adjusted to reduce or increase the number of false positives, depending on how much time you have. The A380 fly-by-wire analysis was apparently the first case where no false positives were reported (and no true positives either, of course). According to the page, the analyzer checks C code that doesn't contain dynamic allocation or recursion, so it's probably not applicable to most non-critical software anyway.

Analog of hardware "false path".

[Ungrounded Lightning]

In the design of synchronous digital circuits the tools measure how long it takes signals to propagate through networks of logic from one layer of flops to another, try to optimize the organization and physical layout of the logic to meet timing requirements, and tell you how fast you can run your clocks and still have everything settled at flop inputs in time for them to correctly capture the data.

They do this by making worst-case assumptions: How long after a clock the output of a flop is stable, how long it takes to get through gates of each type, how soon before a clock it must arrive at the input to the next flop, and so on.

But one of these assumptions is that ANY input to the combinatorial logic might change on one clock and that change might propagate through any available path to affect any output, which must be stable by the next clock. This assumption might be wrong for a number of reasons.

For instance: A complicated logic block might be used differently on different cycles. An adder might be adding two values near the output of a combinatorial block on some cycles, two near the beginning (with its output driving something else complicated) on others. The tools see that there's a long path on the way to the adder's input and another coming from it's output. But they may not see that both can't happen at the same time (either though not checking all the possible combinations of inputs, or not being able to see the circuitry that creates the guarantee - which may be in another block that's not available to the tool).

Without this knowledge the tools may perform unnecessary optimizations to "fix" the non-problem: Using fast gates (with higher power and more silicon area), rearranging the logic to improve the long path (ditto and lengthening some that DO get used). And/or it may report timing violations for the user to fix (masking any shorter but real ones) or prescribe an unrealistically low clock rate (reducing the part's performance).

To avoid this, such tools provide the ability for the users to declare such paths to be "false paths", which should be ignored in optimization. (Unfortunately, the paths often cross module boundaries, so the ability to declare them is generally provided OUTSIDE the source code, in some separate configuration file for the build.)

IMHO many of the bogus warnings from static analysis tools are a similar problem. As a result, such tools need a couple of similar features to solve it.

At a minimum they need the ability for the code author to say "This is really OK." in a way the tool can process. This gets rid of the bogus warning - clearing the output so that REAL warnings won't be buried in the false cries of wolf and will be acted upon.

A useful addition would be an ability to say "This is really OK because of X". That way the tool could then check that X actually still holds and sound off if it gets broken later. (Unfortunately, X can be pretty general. So you still need the ability to say "This is really OK because I told you so.")

This already arose in C++ and ANSI C strong type checking. But idioms were available to tell the compiler you really meant it. (Cast to void then cast to another type, store in a union as one thing and load as another, argument type of void or pointer-to-void, encapsulating such idioms in typecast defined operators, etc.) Now we have another checking tool that needs its own flavor of communication from the designer to the tool.

They *can* be awesome

[bokmann]

Yes, static analysis, especially with some of the open source tools available in the Java space, can be awesome. Check out PMD and FindBugs for two awesome examples of this. Both are free, and do everything you could ever want. If you are willing to pay money for a tool like this though, you are focusing on the wrong things.

No, these tools won't catch entire categories of bugs that can only crop up while your application is running. For those, use unit tests, profilers, etc. For an explanation beyond code for this, check out the book "Godel Escher Bach, an Eternal Golden Braid", specifically the discussion about recording records that when played, destroy the record player playing them. No, its not real, its a thought experiment.

Another fan (+ fan of PC Lint)

[mike_diack]

I've been using static analysis tools since 1993 now and I'm totally convinced of their merits. I've found no compiler that yet comes close to being as capable at finding problems as the best static analysis tools are now.

I develop medical software for cancer treatment professionally (which is keenly watched by the FDA) and thus am exceptionally keen to maintain bugfree code. While static analysis can't guarantee it, it can make a huge difference to code quality and reliability - and yes, to answer the original questioner, yes, they can catch stuff like buffer overruns, pointer abuse, memory leaks etc, and in some cases, resource/locking abuses etc. I will admit that gcc is very good these days, but even it isn't as good as these tools.

In particular I'm a very big fan of Gimpel's PC Lint - which is astonishingly good and astonishly inexpensive.

If you wish to try a demo of it, interactively, try their demo tool at:

http://www.gimpel-online.com/OnlineTesting.html

Also, they do a monthly "bug of the month" in a code sample - see whether your compiler can spot the issue/potential issue:

http://www.gimpel.com/html/bugs.htm

For anyone integrating it into Visual C++ (PC Lint is a command line tool), take a look at Riverblade's Visual Lint also:

http://www.riverblade.co.uk/

Disclaimer: I don't work for any of these companies (Gimpel or Riverblade) but I'm a huge fan of both of their products - I believe they've made more difference to my development than any number of UML tools etc - I tend to dump tools that don't make a real difference to me, and these are the only tools that I've used year in year out...

(Gets off soapbox)

Seriously though - whatever you choose, do investigate static analysis - it really is worth it.

Yes, they do - in the right hands

[jalapenokitten]

I've been using PC-Lint since about 1997, and have found some pretty nasty bugs using it - in most cases well before they had surfaced in the field. Given that, using such tools is (for me at least) a no-brainer. You do however need to a a particular mindset and take the time to learn how to use such tools effectively - how to tune your warning policy to your codebase, and what issues to look out for which could indicate real trouble (e.g. I've seen behavoural bugs caused by variable name scoping issues which lint caught but which at first glance would appear to be "lint noise"). I have a red flag list of issues I watch for in a new codebase, which works rather well in practice. I should add a disclaimer that these days I make most of my living from working with PC-Lint, so I'll acknowledge my interest in the subject up-front before anyone else points it out!

Only solution:

[SanityInAnarchy]

Only solution is to work for a manager who is either technically competent or trusts your judgment.

Of course, the chain can be a lot more complicated than that -- you could have a technically competent boss whose boss is a PHB -- but the basic theme is, the person ultimately making the decisions either must understand the real impact of those decisions, or must be delegating to someone who does.

Anything other than that, and your corporation is dysfunctional. I realize many, maybe even a majority of corporations are dysfunctional, but it's good to know.

Short-term solution: Adopt issue-tracking software and have it generate graphs. Then, when the snake-oil salesman comes around, you whip out your chart and say, "I have a chart here of actual bugs closed, and new bugs opened, and average total bugs open over a given week. You have a chart of potential bugs remaining, and no way to mark one of them as not a bug. Why would we want your chart when we have my charts?"

Definitely worth it

[Anonymous Coward]

They're definitely worth it.
We've used McCabe IQ here and it's very helpful.
IQ focuses more on metric (grading code quality) and coverage (lots of little 'i got here's in the code). The combination of the two is nice since you can get an idea of where your testing effort is going wrong as well as highlighting suspect code.

It's expensive but you only need to find a few 'deal-breaker' bugs to increase sales / decrease costs. So in short these tools are definitely worth it.

Static analysis bolted on to GCC

[Anonymous Coward]

Mozilla is making some interesting static analysis plugins for GCC. It looks like something that could be easy to utilized by open source projects.
See Dehydra static analysis tool

Worth the Investment (Time and $$)

[SlySpy007]

We did a fairly detailed study of several of the more popular tools on the market and found that the Coverity product is pretty good in terms of effectiveness and rate of false positives. One finding was that the effort to properly configure the tool was non-negligible but that the results were worth the effort.

We also do a lot of work with the formal verification tools like Spin. Again, the effort can be substantial (some have likened Spin modeling to a black art) but the results that you get can be well worth it.

I suppose the last word is that if your definition of static analysis tools consists solely of Lint and its derivatives, you will get very little benefit (if any) from the effort. The same goes for assuming that you can buy a tool, install it, and get results immediately -- it just doesn't work that way. The tools, like everything else in the software realm, are not silver bullets, but rather another arrow in the quill.

Static Code debugging.

[lordmage]

Is interesting, but what I found is that I use Several Methods:

1. -Wall -pendantic, etc. Rule: No Warnings, or errors of course.
2. Insure++ from Parasoft. Absolutely Wonderful. Catches about everything you can think of and we work in our Debug Areas with it. It catches a lot of issues before they even become bugs.
3. Logic Flow issues are solved with DEBUGGERS, and PRINTFs :)

Insure++ is a major money saver in the long run.

Err? Q.E.D.

[IBitOBear]

My (potentially flawed) understanding is that Purify was reporting the function calls as "possibly" using a variable that was not initialized.

The relevant point was that the function in question initialized a buffer with random data (e.g. from a random number source) [e.g. _not_ "from uninitialized memory" but "_into_ uninitialized memory"] but the tool could not tell what the function was doing, so it said "look here, possibly uninitialized data".

The programmer, having no clue how to imterpret the (false positive) result simply removed the code.

This is what happens when someone who doesn't know what he is doing takes the advice of a tool that cannot understand the semantic importance of an otherwise opaque action.

If the guy had looked up the function call and seen what it did (instead of just removing it) he would have seen it as a false positive.

Thus Is It Proved: powerful tools are dangerous in the hands of the untrained.

Re:Err? Q.E.D.

[agbinfo]

I don't know about this particular instance but I find that a lot of useless code is often left in applications because nobody knows what the code does and, even though it's probably dead code, nobody is willing to take the time to refactor the code or remove the dead code.

The problem is that dead code very often leads to maintenance nightmares and sometimes to an avalanche of bugs. The code gets ugly fast because programmers code around it. I've even seen test scripts written to return "OK" results for buggy code because they thought the code was doing something useful. I'm not kidding.

Like I said, I don't know about this particular instance, and security and encryption tools are a different beast, but I've found that it's sometimes better to remove code when nobody knows what it does then to maintain the code. If it was useful, you'll find out soon enough, in most cases before it's released, and you can put the code back with proper documentation.

Very much, but there are limits

[bill_kress]

They can help good programmers quite a bit, and they can help bad programmers become good programmers. In both cases it takes someone with a desire to learn and not just some 9-5 coder here for the chick-magnet status.

The other thing you may want to factor in is the language. Static Typing allows much more reliable static analysis than dynamic typing. A language can communicate to the tool--for instance, if you tell a language that a variable can't be null, either the language or a test-mode tool should be able to assert that easily.

Java has some really nice tools built in. Some have more. Hints like "This variable can't be null", or "This variable must be assigned by the time the constructor ends and can't be changed" really help if applied correctly.

For example. In java, this code will produce a compile error:

private String isFive(int i)
{
  String s;
  if(i==5)
      s="i was five!";
else if(i > 5)
      s="i remember when i was 5";

  return s;
}

since there is a path through the code that allows s to be returned without having ever been assigned.

In this case, it's pretty obvious, but sometimes you have many paths through your code including exceptions, loops, and if statements--it's really handy to have it pop up a note saying that a variable was used before being assigned.

Because of this really helpful warning, it's a really bad idea in java to initialize variables to null:
    String s=null;

Although sometimes it seems like it's the only way to "Eliminate" that warning, what you are really doing is defeating valuable static analysis tools. Aside from that, it's totally redundant.

In the above case, an else clause would fix it. Even:
else
    return null;

Note that the string isn't assigned, but since that path returns it is never used in that path, so the compiler is happy.

The warnings generated by the error analyzers in eclipse, when cranked up, are also amazingly helpful--but again, more helpful in some languages than others.

Every project should turn them up and possibly add additional error plugins, and then eliminate every single warning. You do this by looking at each error and making a deliberate decision to mask or fix. Sometimes you have to mask, but looking and deciding alone will fix bunches of potential bugs.

Also, never make the decision to mask based on verbosity. Errors should only be masked if there is just no way around it--although in some cases an entire class of errors have to be ignored because of a pattern that is just too pervasive or too hard to eliminate or doesn't match your deliberately chosen coding style (Some settings actually can enforce a coding style).

Sorry about the really long example. I just wanted to demonstrate the amount of thought that could go into a single detection mechanism--and this one is basic.

Embedded systems and static analysis

[Anonymous Coward]

To answer your question: It depends.

Whether it is "worth it" to use static analysis tools depends on how much bugs cost you. Static analysis tools can find important defects in code, and it can find them in places that are difficult to test well (e.g., error cases).

The reason you saw a lot of this in an embedded systems conference is that it's a lot more expensive to send a patch out for a dishwasher than it is for a desktop app. Plus the embedded systems market includes the medical, military, and casino industries, all of whom have reasons to be very conservative when it comes to software quality issues.

I think you'll hear a lot less about static analysis among developers of desktop apps, where issuing updates is inexpensive.

Is this "another trend" or "going to change the face of software development"? As computers get faster and algorithms get better, static analysis will continue to provide more accurate results faster. So I think the trend will continue. I think probably it will get integrated with a lot more people's dev processes over the next decade or so, and hopefully software quality will improve as a result. How radical a change is that really, compared to other technologies (e.g., widespread adoption of OOP) over the last 15 years? *shrug*

Disclaimer: I am an employee of Coverity.

You misunderstood my point

[Moraelin]

Yes, I think you either mis-understood my point, or are wrong about a subtle distinction: the case when it's a _fake_ singleton, just as a pseudo-optimization.

If it's a _stateful_ singleton (i.e., a real singleton), then yes, damn better synchronize it even on single-CPU system. No arguments there.

The problem is that there are classes which really shouldn't be singletons at all, and are a mis-use of that pattern. They don't actually _have_ any state, which could be stale on another cache. (Which actually is the least of your concerns there.)

E.g., as a trivial example, consider the following kind of singleton:

public class LoggerSingleton {
    private LoggerSingleton instance = null;
 
/** So noone can instantiate it */
    private LoggerSingleton() {
    }
 
    public void log(String s) {
        System.out.println(s);
    }
 
    public void log(Throwable t) {
        t.printStackTrace(System.out);
    }
 
    public static LoggerSingleton getInstance() {
        if (instance == null) {
            instance = new LoggerSingleton();
        }
        return instance;
    }
}

Yeah, there are better loggers available, but it's just an example. Bear with me.

The class has no state, and holds no resources. It's just a collection of idempotent methods.

It's not a _real_ singleton, it's just someone's misguided attempt at saving the (ridiculously low) overhead of a "new LoggerSingleton()" everywhere it's used. It doesn't matter if two threads get two different copies of it. Worst case scenario is, literally, one of them wasted 8 bytes on the heap, which will be garbage collected later. That's all. There is no way to malfunction because of that.

Synchronizing that, simply isn't necessary. It also defeats the whole "optimization" there, since synchronization is actually more expensive than just not making it "singleton" in the first place.

Now I'm not saying it's right to have that kind of singleton, but it's the kind of "optimization" you tend to get from people coming over from C++. They occasionally even write pools for such objects.

But ultimately it doesn't matter much.

_If_ you have plenty of time, and/or you're at architecture design stage, yeah, tell the people to not do it.

But it's kinda annoying to see an overpaid consultant insisting that synchronizing those is top-priority, and he's saving everyone from some great catastrophe with it. In a project which is already overdue and in trouble. There are better ways to use those resources, than "fixing" non-problems.

Re:You misunderstood my point

[GravityStar]

I understand your point. Please understand my point that there is a very subtle reason why this code can fail on multiprocessor systems.

Please take the time to read about the Java Memory Model and about Memory Barriers. The reason why this code can fail is very subtle and is covered by my previous message.

As a example, a instance of this particular class can fail on CPU2 when calling getClass() on it. I'm not kidding.

Re:You misunderstood my point

[Moraelin]

I actually do know a bit about those, I already knew about the double-checked lock problem since 2002 or so, and I'm still not entirely sure how this can fail, unless the multiprocessor system is broken. I have also seen plenty of this kind of singleton running cheerfully on Sun, Intel and IBM multi-processor machines, and none of them failed so far.

So, well, please enlighten me.

Re:You misunderstood my point

[GravityStar]

Failure can happen when CPU1 creates the singleton;
CPU2 retrieves the pointer to the singleton;
If, and only if, CPU2 already has (stale) cache information about the memory the singleton occupies (I admit, this could be rare condition);
methods executed on CPU2 using that memory will now fail in unexpected ways.

It's a race condition that you could test for a million times and not have a fail; but with my luck, put it on a production machine and the JVM will have a native crash.

Anyway, YMMV, but I don't go hunting for this kind of bug in the codebase. If I have to edit that file for any other reason though, I'll change it to be correct.

Re:You misunderstood my point

[Moraelin]

Hmm, well, on second thought, I guess it _could_ fail if any multi-cpu system is actually as broken as to need explicit instructions to re-synchronize the cache. I'm pretty sure Intel didn't need those since, oh, at least the Pentium Pro, and, again, neither did Sun's Sparc processors (at least the 64 bit ones, not sure if the earlier too), and neither do IBM's P5 or P6 systems.

Your link mentions the Alpha as one which does, which kind of surprises me. I would have expected more from those, but ok. If anyone ports Java 5 to a decade-old, defunct CPU, I suppose it's possible to have a problem.

Still, I suppose for purity sake, you have a point.

Re:You misunderstood my point

[Moraelin]

_If_ that happens, then IMHO Java's own memory allocation is broken on that machine.

I can imagine stale data in the reference variable written by "getInstance()", which is (as far as I understand it at 1AM) the problem on Alphas in your link. But that's highly unlikely to be a problem, since it will then usually see a stale "null" and proceed to allocate a copy. It would take something like having only half the pointer stale, to actually have a problem. Not very likely, since it would also require a kind of alignment that's avoided by the compiler, and causes a fault on some machines. But ok, I'll concede that possibility.

Stale contents of the object? Hmmm.

Java's own memory allocation has to synchronize _somehow_, since all CPUs advance the same pointer in the heap. If that memory allocation isn't in a critical section in the JVM itself (which would be kinda required for CPU2 to see stale data in the memory taken by the object itself), then that machine has a bigger problem. Then two threads trying to allocate an array at the same time, without a cache synchronisation in between, would get the same chunk of memory. And possibly a corrupt heap, as each advances the free memory pointer by a different amount.

IMHO you'd notice that kind of malfunction very fast, with or with synchronization.

Re:You misunderstood my point

[Anonymous Coward]

While the JVM certainly doesn't make any guarantees about consistency of user memory the JVM must make consistency guarantees about the memory of its own state for objects. getClass can't return the wrong results and can't get some random uninitialized value from the cache because it would violate type safety.

The specific problem cited by your source is "Thus, a thread which invokes getHelper() could see a non-null reference to a helper object, but see the default values for fields of the helper object, rather than the values set in the constructor." Information pertaining to type safety cannot be included in the list of default values.

Sigh

[Anonymous Coward]

I'm sitting here, reading this post and peoples comments about static analysis, while I should be fixing Lint errors.

I have 136 of them to fix, all of them are re-using variable name errors and "this variable might have not been initialized" errors, because it's state machines that use a switch statement in a for loop. All the damn variables are initilized in state 1, which is were the damn thing always starts, but Lint can't figure that out. I've been at this for days because an 85K source file was ported into a new project and the previous project didn't lint.

Next I have to check a text analyzer, it has 3000 lint errors, all because the analyzer is based on c strings and Lint is worried there might be array overflows in the specific walks. This is code which has been heavily used since the late 1980s without ever having any problems. Management wants 0 Lint errors, it's not that there's actually any problems. Sigh. I hate Lint.

Just to clarify

[Moraelin]

Just to clarify my previous point, the point of contention there is almost invariably the "getInstance()" method. Such tools trip over it, hard, and there's usually no way to tell them, basically, "it doesn't freaking matter if another thread or another CPU essentially bypasses the singleton and gets a different instance of a stateless class."

The actual methods of the singleton, well, you use your own head. In my logger example it works because it essentially wraps around System.out which is already synchronized, so there's no need to synchronize the outer layer again. In other cases it's really stateless and idempotent, and again it doesn't matter there either. In very few cases, they might actually need to be synchronized.

But anyway, my minor peeve there is with tools and people getting tripped by the unsynchronized "getInstance()" in a situation where it just doesn't matter. I'm not even opposed to turning it into a non-singleton, but seeing someone who should know better (or claims to know much better) spend pages in his report and _hours_ worth of meetings just over the lurking evil of that "getInstance()", just tells me they might have exaggerated their claim to greatness, ya know?

Sort of

[Shadow-isoHunt]

They work, but not nearly as well as bunny - which is free. It's a drop in replacement for gcc, and does 9 types of fuzzing/analysis, reporting changes in behavior of the program.
http://code.google.com/p/bunny-the-fuzzer/wiki/BunnyDoc

"solutions"

[sohp]

Many of the work well, as can be seen from all the endorsements in other comments.

There is one thing to watch for though -- companies selling you software or consulting around the issues their tool highlights.

A few companies will practically give you a tool that they claim finds lots of critical bugs, either through static or runtime analysis, and by the way, for a more considerable sum, offer you "fixes" for these "bugs". This is something that companies sell to a PHBs desire to use the tool to generate metrics and measure progress by how many problems the tool spits out get fixed.

Of course there are companies that offer genuine value in finding and fixing software problems, but every red flag in the room should fly if the company proposes to offer you a free or almost-free "no obligation" assessment. Once the sales dogs are in the door, watch out!

GCC has some nice features

[Anonymous Coward]

I recently discovered the use of __attribute__ ((format(printf, ...)) in GCC to tag all our own printf-like functions.

We recently started compiling on 64bit and used to the above to pickup all malformed printf style arguments in our code. There were over a thousand! Things like %ld where parameter was only int, etc.

Yes

[ehovland]

A lot of the yes responses in this thread are dead on. The current crop of static analysis tools have improved tremendously over the previous generation. And the gold standard of this generation for C/C++ is Coverity Prevent. FindBugs and Coverity Prevent for java. Klockwork is gaining in C/C++.

Now, be careful. This is another tool. It means that the problem may still lie with the programmer. All reported defects should be seen not just as a bug to fix but the number of defects related to similar code may point to a design issue.

And to all of you punks out there saying there are a lot of false positives. There are way fewer false positives then you think. These tools can point out bugs that you don't understand until you have really sat down and figured out what is going on. Understand the defects before discounting them.

I work for a research laboratory for a space agency and static analysis tools are soon going to be a requirement for flight software. And this is after running these tools on a lot of flight code to see if these tools are any good.

Unfortunately, even after being very strict with compilers (-Wall -pedantic) and using static analysis tools until there are near zero defects bugs still appear. It is maddening. But having near zero defects almost makes finding these bugs easier.

What happended to OS.2 ?

[Anonymous Coward]

Remember that OS called OS.2? 32-bit flat memory, multitasking/threads? 16-bit drivers. Wha. 16-bit drivers? Surely some mistake. Chkdsk that tooks hours on a 1 GB drive? and always complained about the, yes, /. (no pun intended) directory? That IBM?

You're making a big mistake

[Anonymous Coward]

Java's memory model needs to be the same on all CPU architectures. What the exact behavior is on x86 is irrelevant to deciding whether your Java program is correctly synchronized; what matters is the Java Memory Model. The Java Memory Model says that, in the case you're talking about, any non-final fields of a newly constructed object may be stale from the point of view of a thread that has a reference to that object.

Java's own memory allocation has to synchronize _somehow_, since all CPUs advance the same pointer in the heap. If that memory allocation isn't in a critical section in the JVM itself (which would be kinda required for CPU2 to see stale data in the memory taken by the object itself), then that machine has a bigger problem. Then two threads trying to allocate an array at the same time, without a cache synchronisation in between, would get the same chunk of memory. And possibly a corrupt heap, as each advances the free memory pointer by a different amount.

But how granular is the synchronization needed to correctly allocate memory from a heap shared by all threads? Presumably, the threads just need to synchronize on a single pointer. What if you're on a machine that allows you to synchronize on just that?

Also, what if some crazy guy implements a VM where each thread gets its own young generation, and therefore, allocation doesn't need to synchronize at all?

You're reasoning about this problem in terms of implementation, not semantics. Java's memory model tries to specify a minimal semantics, so as to allow maximum implementation flexibility.

false positives (considered harmful)^2

[Anonymous Coward]

I'm seeing a handful of posts decrying false positives. With my experience using Findbugs for Java, we haven't found too many false positives at all! In fact, things that we originally thought were false positives, with a little thought, turned out to be improvable code. It was worth changing a line of code or two here or there to make the so-called false positive go away, because the resulting code was both prettier and more useful.

One concrete example: I used to call toString() on String arguments to constructors to make sure that NullPointerException was thrown if nulls were passed in. Findbugs objected to this. At first, I thought of this as a false positive, because I really wanted the toString method to be called on the String object as a defense against nulls. But with some further thought, I realized that I should explicitly test for null in the construtor code itself. This was not only less surprising to anyone reading the code, it also allowed me to throw a NullPointerException with a meaningful message.

Yes, static analysis tools really help. Ask folks to bring their Findbugs output to code reviews. Make it easy and routine to run Findbugs from Ant. (I'm also very impressed with Coverity, but we couldn't afford it.)

This post is a redundancy analysis test

[iminplaya]

How many of you will get on a plane that's never been flown? Or buy a car without taking it for a test drive?

Now, before I buy software, I want to at least see a screen shot.

Unless you become lazy

[benhattman]

Even if such a tool only catches a couple errors, it is probably worth the investment. If there is one intermittent error on a subset of your target platforms, and if this tool catches that error it can easily save hours of debugging work. Considering engineering rates, these will pay for themselves quickly.

Unless, engineers begin to rely on them! If I stop thinking about referencing null pointers because my tool catches 90% of them, I haven't gained a thing.

Used on C source code.... YES ... PLEASE YES!

[syn1kk]

The type of coding I do is applied mathematics ( see Digital Signal Processing ) where most of the code is procedural C code ( see NOT object oriented ) and we abuse the rules of C a lot. There are tons of small abuses and warnings. Most of these could be easily found but are not found until it freezes our embedded system and then we have to physically hit a jumper or power cycle the box.

So if we had one of these it would prevent that sort of thing from happening a lot more ... make our code much more robust... and increase our productivity.

-----

I don't see how it would be helpful for Java though... because those languages already have *very* robust warnings from the compiler... and the languages themselves are designed to be more dummy proof ( C trusts that you know exactly what you are doing... java trusts that you don't know what you are doing ).

-----

So for languages like C where there is lots of potential to screw up these would be excellent. But for more dummy proof languages like Java / C# the payoff would be less.

Re:Used on C source code.... YES ... PLEASE YES!

[RPoet]

As a Java coder I may not know what I'm doing, but FindBugs for Java is tremendously helpful. Java is not not a foolproof language, nor is any other language as far as I know. The compiler will warn you to a certain degree, but FindBugs will go much further, and also spot anti-pattern usage and advise you about best practice. It will spot potential race conditions, see potential for resource waste (database connections not being closed, etc), and much more.

Re:Used on C source code.... YES ... PLEASE YES!

[RPoet]

Correction: "Java is not not a foolproof language" should have just a single negation, not a double :-)

FindBugs works

[Anonymous Coward]

Yes, use FindBugs frequently. Happy with it.

Perl::Critic

[Anonymous Coward]

For Perl, Perl::Critic is excellent:

I don't have a lot to add, except that if you program Perl and don't use it, you shall spend more time on debugging.

Linux kernel.

[rew]

Some people ran these kinds of tools on the Linux kernel, and found some suspicious code. It is similar to turning on -Wall on your gcc compiler: Some bugs suddenly turn up (with exact line number!) without having to go into the testing phase. This means you have more time to test for real bugs that only show during runtime.

Static analysis is in large part what your brain d

[Anonymous Coward]

One reason why static analysis works is
because this is in large part what you
do with your brain when your read code.

There's obviously a limit to how much
a static analysis tool understands about
what your code is supposed to do, but in
a relatively restricted scope, they can catch
a ton of things.

GCC, Splint and Disipline Will Do For Syntax

[A R Baboon]

GCC has a number of options that add static checking. Additionally splint (when properly parametrized) will catch a number of other common gaffs. Who ever said C does not express enough to add checking clearly does not understand the problem. The syntactic sugar in many popular languages actually adds complexity. Some implicit garbage collection, pooling and threading mechanisms add non-deterministic qualities which make *static* checking a np problem. If you are smart enough to keep your aliasing confined and tagged there is little danger in C and the simple syntax makes static checking easier.

Also of note valgrind is an excellent tool suite but it is not a *static* checker.

Splint never found as many Problems as Ada

[krischik]

I tried Splint - relay hard - tried all the annotations - but it was never able to find as many possible problems as an Ada compiler could find. Split was a failure and the project is dead since 2003 [1].

Note that I am fully literate in C and Ada (and C++ and Java) - so I do fully understand the problem.

If anybody does not understand the problem the it's the die hard C programmers who never relay tried another programming language. It's not all syntactic sugar, it's about always knowing how many elements are in an array. With emphasis on "always" - not sometimes - but always.

Have a look at:

http://en.wikibooks.org/wiki/Ada_Programming/Type_System

Last not least, the sad truth is that 80% of all programmers are not smart enough. Note that since you read /. you most likely belong to the other 20% but you should not project your level of C expertise to the other 80%.

Martin

[1] http://sourceforge.net/project/stats/?group_id=34302&ugn=splint

anonymous

[Anonymous Coward]

It is useful when you inherit a project with poor coding and documentation. Particulary when there are a lot of files , it allows to see better the relations and where are defined such and such function or class.

Signal-to-noise ratio

[mlimber]

You will need to do some tuning with a tool like Gimpel's PC-Lint, so it's best to use it from the get-go rather than try to start linting part way through or at the end of your project. Then you do some initial setup and make minor tweaks as you go, so the tuning process is not at all painful. Many tools, like PC-Lint, also integrate into IDEs like Visual Studio, and some have free add-on programs like ALOA (http://pera-software.com/htm/english/aloa.htm) which analyzes the output of PC-Lint to generate some metrics for the quality of the code.

I have found such tools to be invaluable. I had code like this:
class Lock {/*...*/};
void Foo( Mutex& m )
{
Lock(m);
// ...
}

This is valid syntax, but I intended to use that lock instance for the duration of the function, so the first line should have read "Lock lock(m);". Multithreading is tricky enough, and I looked at the real code for a long while, reading right over this bug. PC-Lint found it for me right away (thankfully, it was already tuned, and I should have been using it before running my code).

But even though PC-Lint is pretty good, it ain't perfect. I have found that it has some trouble with advanced C++ templates (e.g., policy-based design). I have submitted bug reports for many of these problems, and they do seem responsive in working them in to the patches.

Finally, here's an article from 2006 discussing the available static analysis available tools for C, C++, and Java and describing how and why to integrate it into your development process.

Re:who proved Astrée ...?

[anpe]

It's not "error free", it's _run-time error free_. Which according to the GP's link means that no undefined behaviour according to the C standard or user added asserts may happen.
So for example, the program won't ever divide by zero or overflow an integer while summing.

Re:who proved Astrée ...?

[anpe]

*any* software [the French] write is always "perfect" even if it can be demonstrated to be a piece of crap in front of half a dozen decision makers. You know, any "demonstration" aimed at half a dozen PHBs often relies on the looks of your Powerpoint presentation.
So, basically, you're good at picking fonts, and that pie chart was really gorgeous. Congratulations.

NASA Flight Software and Source Code Analyzers

[Anonymous Coward]

NASA has analyzed several static analyzers and continues to apply a couple to all the flight software projects under analysis. Most of the issues we identify are from the output of these analyzers. Coding errors identified by such analyzers generally have little relationship to design and requirement issues. Issues found during requirement and design phases of development our generally more valuable to the projects but they do not prevent coding errors. Static Analyzers do find the type of errors listed in the original post. Any development project that neglects to use lint-type analyzers routinely through development lifecycle will effectively raise the cost of development and increase risk. Integration and Test can identify some of the problems such as race conditions and memory leaks. However, the complexity of today's applications, NASA flight software included, nearly always demands more resources than are available to Integration and Test Analysis. Much of the time, if not all of the time, spent testing is to validate that the code functions under nominal conditions. The amount of stress testing applied often will not uncover many of the straightforward issues that a "simple" static analyzer will. The output of false-positive results by static analyzers are their weak point. Commercial efforts have attempted to incorporate algorithms to filter out false-positives with some success. The ratio of false to positive results can be 10:1 or higher. An analyst is required to review and identify real errors. But it is usually only the first build analyzed that has an inordinate number of false-positives. Thereafter, results can be differenced and the net output of new flagged issues is significantly reduced. Unquestionably, eliminating all coding errors possible with static analyzers, improves the ease of startup of testing, how quickly testing proceeds (eliminating the need to re-execute test runs) and improves the level of assurance that test results provide. Altogether V&V of requirements and design do not prevent coding errors and static analyzers are not a replacement for exhaustive testing. The latter two analysis processes are complimentary and they are all necessary to assure that the software performs as intended and responds to adverse conditions appropriately.

Static Tools - those that do work.

[ipr1118]

I think you should try Coverity Prevent. You will be amased how one pointer set to NULL only under certain set of circumstances and passed through several functions gets dereferenced - and Prevent will catch it. I have been in the industry for many years and had a clear idea what is possible what is not. Well, my ideas had to change after using Prevent. See http://www.coverity.com./ There are some other tools on the market, but their depth is not even close.

Re:Static Tools - those that do work.

[Anonymous Coward]

We've reviewed Coverity and compared against several other tools. The tool does not flag the breadth of errors that other tools do. The intelligent filtering and elimination of false-positives could actually be discarding issues. In attempting to eliminate false-positives, the tool has a smaller set of coding errors that are most easily verfied as absolutely true (positives). Comparing features and performance, Coverity is prohibitively expensive. If one simply differences the output from static analyzers, then after the first build is processed through an analyzer, subsequent net output from the analyzer is easily reviewed.

Good Article on This Topic

[Anonymous Coward]

A good article on this exact question can be found here, with a case study:
http://www.embedded.com/columns/technicalinsights/207000574?_requestid=815557