Programmers Learn to Check Code Earlier for Holes

Carl Bialik from WSJ writes "Many companies are teaching programmers to write safer code and test their security as software is built, not afterward, the Wall Street Journal reports. This stands in contrast to an earlier ethos to rush to beat rivals with new software, and, of course, brings tradeoffs: 'Revamping the software-development process creates a Catch 22: being more careful can mean missing deadlines.' The WSJ focuses on RIM and Herb Little, its security director, who 'uses Coverity every night to scan the code turned in by engineers. The tool sends Mr. Little an email listing potential red flags. He figures out which problems are real and tracks down each offending programmer, who has to fix the flaw before moving on. Mr. Little has also ramped up security training and requires programmers to double-check each others' code more regularly.'"

Comments

This just in:

[r_jensen11]

Writers are encouraged to proofread.

Re:This just in:

[Anonymous Coward]

No point you proofreading you own code. You see what you think you've written, not what you've actually written, therefore don't spot any problems with it.
The trick is to get 2-3 other people to review it.

1. The earlier you spot a defect, the cheaper it is to fix.
2. Test results are only as good as the test code written.
3. Edge cases don't normally show up in test code. Test cases are typically designed to show that the code works, rather than finding the boundary where it fails.
4. You can suggest better ways of writing the code/learn new tricks during code reviews.

static_analysis++

[tcopeland]

Static analysis is great stuff. I've worked on an open source Java static analysis tool, PMD, for the past few years and I've gotten lots of feedback from folks who have used it to find all sorts of things in their code. Just a quick scan for unused variables can yield some excellent results, and the copy/paste detector works quite nicely too. And there's a book, too!

Coverity's doing a nice job with their tech marketing, too - l think a couple of open source projects are using the stuff they found to clean things up. At least, there's been a fair amount of traffic on the Ruby core list about some things Coverity's scan found. Good times...

Re:static_analysis++

[Stellian]

Enough whith Coverity allready. It's like the 50th slashdot article that talks about this.
FYI, it costs about 50.000 $ for a medium sized project (500.000 lines), and is no more than a lint on steroids. Here is a somewhat cheaper competitor.
None of this tools is a mach for a manual audit performed by a professional.

Re:static_analysis++

[Coryoth]

Also, back on topic, try writing financial software some time. It's like a different world. Everything is unit tested, and the unit tests don't so much check for bugs as prove that your code works.

Unit tests don't prove your code works any more than drawing a few right angled triangles and measuring the sides proves Pythagoras' theorem. If you want to prove your ode works you use a theorem prover. To do tht you usually need to provide more detailed specification (beyond just type signatures) about how your code is intended to function. That tends to be more work, though if you really need to know your code is going to work it can often save time in the long run (over ridculously long and exhaustive testing). There are things out there that provide toold support for theorem proving aout your code: SPARK Ada along with the SPARK tools provides a powerful theorem prover, and HasCASL with CASL tools (including the HOL theorem prover) provides string theorem proving for Haskell. Even ESC/Java2 utilises a theorem prover (called Simplify) to provide extended static checking of Java code. I'm sure there are more examples.

My point is not that Unit testing is bad (it's very good), but that you shouldn't overstate its effectiveness. Unit tests are a great way to provide a reasonable degree of assurance that your code will hopefully ork as intended. It isn't a substitute for actual assurance however. It really depends on exactly how sure you need to be - how much an error will cost, and whether that can be tolerated.

Jedidiah,

Re:static_analysis++

[Coryoth]

I've worked in industry as a mathematician. When we say we're going to prove something we actually prove it, rather than just tossing out a few random examples for demonstration. Given that a piece of software is, at its heart, just a lot of mathematics, and the fact that it really is possible to prove things about code in the real sense of the word, I would be very careful about saying you "prove" your software works.

Jedidiah.

Re:static_analysis++

[IamTheRealMike]

FYI, it costs about 50.000 $ for a medium sized project (500.000 lines)

Yes it's incredibly expensive. Yet, plenty of well known companies pay for it, so I suspect it's worth it to them.

is no more than a lint on steroids.

Er, no. No, no, wrong, no.

I've got access to the Coverity results for WineHQ. It's already found many problems that evaded both manual code review and unit testing. Its rate of false positives is remarkably low once properly configured. A lot of these problems would only occur in obscure circumstances or on error paths - but these are precisely the kind of errors that unit testing tends not to reveal. It can detect problems like race conditions or memory leaks that lint cannot. The recent X security bugs were revealed by the tool first.

I've seen tools like this before, but not one as good as this. I've never used competing commercial products, so cannot speak as to their effectiveness, but for a large C++ codebase I would certainly be happy to have such a tool helping me out.

Microsoft have used similar programs developed by MS Research on the Windows codebase for some time now and they're apparently very effective. Quite a lot of security problems revealed by them were silently fixed along with other problems in updates.

None of this tools is a mach for a manual audit performed by a professional.

Totally wrong. Every patch that gets checked into Wine passes code review by at least Alexandre who is without question the best programmer I've ever met. He is easily as good as Linus but his much quieter and more conservative personality means he doesn't get Linus' press attention (a good thing, imo). And all the patches are posted to a public mailing list where several other people can and do review patches too.

Static analysis can reveal problems that simply don't get spotted by the human eye because they're too complicated to follow, because they occur in very weird situations, or because the code evolves over time under the direction of many different people and inconsistencies creep in.

I hold any bet

[Opportunist]

After missing a few deadlines, the marketing goons will push to abandon security for more crap on the shelves.

After all, that's how the software market works. People buy anything. "LOOK! THE NEW (insert program/OS name here)! I MUST HAVE IT!"

Stable?
Secure?
Mem-leak free?
In one word: FINISHED?

Who cares? It's new, it's shiny, it's been all over all the mags and preview pages, the hype is on, WANNAHAVE!

And as long as we keep buying the unfinished crap, it won't change.

Yes, I'm sure everyone in the tech departments would see this as the right way to go. Test your software, preferably during development, not afterwards. Go through memleak tests, go through stability tests, have some experienced whitehats poke at it, and if it survives, let it go into beta.

If anyone gets that idea past marketing, I will bow down to him.

Re:I hold any bet

[NineNine]

You're right... The problem is that software consumers already have a mindset that makes broken programs OK, and the way to fix them is by buying the new version. One of the worst offenders that I've seen in Intuit. Intuit is famous for releasing new programs every single year, regardless of whether or not anything has actually changed. They're also notorious for simply not fixing old code after the new version is released, with the official response of "Your problem is fixed in the new version. Buy that one." This is grossly fraudulent, in my opinion.

The problem is that we all, as consumers, already accept this kind of shit as acceptable. I wish I knew a way to reverse this, but realistically, I don't see this mindset changing any time soon.

Catch 22?

[Tourney3p0]

Revamping the software-development process creates a Catch 22: being more careful can mean missing deadlines.

Alright, so writing better code means you might miss a deadline. But not writing better code means.. things are exactly as they've always been, or the software development cycle will be revamped appropriately?

Not much of a catch 22.

QA is.....

[Wisp]

The new Black!

I do this personally.

[cableshaft]

I usually do some quick general design and planning beforehand, then go in and write the software one element at a time, testing to make certain it works properly before moving on to the next. The benefits seem to far outweight doing it the other way, for me, as it reveals problems I wouldn't have noticed in the planning stages in the design or implementation early, and it also helps isolate where any bugs would be located at, so I'm not checking all over the place.

I'm not sure if it really saves me any time in the long run, but I'm much more comfortable coding this way, which is probably more important.

Also, so far, I've been the only coder for my projects at work and my games at home, so it *might* not be quite as effective for large teams, although what I've read on XP seems to suggest that it can still be very effective.

Catch-22

[kentyman]

Revamping the software-development process creates a Catch 22: being more careful can mean missing deadlines.

That's not a Catch-22. That's just a tradeoff.

Slippery slope

[metamatic]

Jeez, next thing programmers will be expected to document their code.

What will the XP weenies do then?

Ain't gonna last

[FiveDollarYoBet]

Those aren't security holes.... They're undocumented network transfer features!

It sounds good and all but there's a direct correlation between the deadline and how bullet proof the code is.

insert sig here

Well I learned that at Uni

[Yrd]

Correct-by-construction programming is a fundamental part of a proper education in software engineering, I would have thought.

Where did these people learn to code?

Re:Well I learned that at Uni

[truthsearch]

In the real world... where the client says, "I don't care about security, just get it done!" Of course they start to care after a break-in, so they have things fixed in hind-sight.

That's why...

[GillBates0]

I always make sure I use the highest quality bits when I program. You'll find none of those low-quality, flimsy and occasionally perforated bits in my code.

Agreed, periodic checking for holes has it's own value, but nothing beats using the best quality, industrial-strength (tm) bits to start with, moreso while developing reliable software in the post-911 world.

This Just In From Microsoft

[Metabolife]

After taking this training routine, Microsoft says that Vista will be delayed another 2 years.

gets() and people

[mkiwi]

Sometimes it amazes me what people do with the C programming language, for good or for bad. Take some pro programmers who I caught using gets() instead of fgets(). I'm not a rocket scientist, but I'd say anything that uses gets() is a serious problem, since that function does no bounds checking and is prone to attacks.

How do people learn to code like this? Is it just early habits that do not go away?

OT: not a Catch 22

[cain]

The example in the write up is not a catch 22. A catch 22 requires two things be done, each one before the other, thus neither can be done.

Thinly veiled ad?

[Mr Z]

Is it just me, or does the article just read like a thinly veiled advertisement for Coverity? It's reads like a generic commercial template: "Meet Bob. Bob thought everything was fine. But then he discovered he had Problem X. That's when Bob discovered Company Y with Solution Z." (etc. etc.).

Re:Thinly veiled ad?

[cant_get_a_good_nick]

Seems they've been astroturfing for a while. wasn't that long ago they did a big writeup on flaws Coverity found in certain FLOSS projects. at least then they found some bugs and helped fix.

I'm all for tools like this. YOu can find a billion text editors on sourceforge.net but very few good programmers tools. Just this smells like an add for me.

Good at publicising themselves

[derek_farn]

Tools are a cost effective way of checking source for lots of different kinds of problems. I have no direct experience of the Coverity tool, but see that they are certainly good at getting lots of publicity. A List of static analysis tools is available on Wikipedia.

Deadlines are set wrong

[192939495969798999]

If being careful makes you miss the deadline, then the deadline is set wrong. Shipping a product with security holes that you knew about + could've fixed with a bit more time is how we got into the position we're in. Pushing back a release date to fix them first should be the rule, not the exception.

Obligatory Fight Club

[Weaselmancer]

Narrator: A new program written by my company is shipped on time, but with bugs. The network stack locks up. The OS crashes and burns and scrambles the hard drive. Now, should we initiate a code review? Take the number of licenses in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a code review, we don't do one.
Business woman on plane: Are there a lot of these kinds of bugs?
Narrator: You wouldn't believe.
Business woman on plane: Which software company do you work for?
Narrator: A major one.

Wow. A 'Developer' article

[ishmalius]

I'm just so happy that a "Developer" article actually made the front page. I have been afraid that the tech level of the audience of Slashdot has been falling lately. Compare it to the number of "Game" articles on the front page.

But to stay with the topic, analysis tools are just that: tools. They are not a cure to chronic software problems. Developers are not excused from the responsibility of at least attempting to write quality code.

Some current project development methods really contribute to buggy and insecure code. Example: XP. I really think that some aspects of XP programming are a bad idea. Namely, the "code as fast as you can" aspect of it is fraught with errors. A more thoughtful, disciplined approach might seem like it is terribly slow. Yet being inherently less buggy, it can reach the target faster than the sloppier, more haphazard approach. This is much like the Tortoise and the Hare. Or maybe a better analogy would be like a rally driver who is more careful with his fuel and tires.

Don't get me wrong. Some parts of XP are fine. The Buddy System is an excellent way to get things done quickly by short-circuiting the collaboration cycle.