Scott Trappe's Answers About Code Quality

We got some excellent questions for Scott Trappe, the code quality expert who runs Reasoning. In return, he gave us some excellent thoughts on how to improve code quality, along with some insight about how he reached his now-famous "Linux TCP/IP stack is better than most proprietary TCP/IP stacks" conclusion.

1) sample size and conclusions
by tim_maroney

How can any conclusions about the relative virtues of two development methodologies with a universe in the millions of components be drawn from a single sample, and one as small and atypical as a TCP/IP stack?

Scott:

You raise an excellent point, and answering it will take some exposition, so bear with me.

You're right, these results do not "prove" that Open Source is superior to proprietary development, nor were we trying to prove superiority of either development process. However, it does counter the numerous articles and studies written alleging that Open Source software is inherently inferior in quality because there is no formal development process, no comprehensive test plan or infrastructure and no dedicated resources to provide follow-on support and maintenance. Our study is a single counterexample which discredits those assertions. We are in the process of inspecting other Open Source projects to see if these results are anomalous or not, and will report our findings.

We chose to examine the TCP/IP stack because it's "atypical" in a way that makes it perfect for comparison: it has a very well-defined set of published requirements that have been stable for several years; adherence to published standards is an essential element of any implementation; there are hundreds of books and articles covering design alternatives, performance measurements, and sample implementations; and there are publicly available conformance tests.

Thus, unlike most software projects, all the implementations we looked at had a common set of requirements to meet and access to a wealth of information about how to design a high performance, well behaved implementation. All of the implementations have been in existence for several years. Thus in comparing them, we didn't have to consider the quality of the requirements document, the uniqueness of the design challenges, nor the resources available for validation.

In short, all the ingredients exist for all of these implementations to be of very high quality. The commercial implementations should enjoy the significant advantages that other studies have pointed out, i.e., resources available to them to insure comprehensive testing, response to bugs, etc. The five commercial implementations were developed at large corporations that have well-deserved reputations for developing high quality products, and invest significant resources to insure that quality. I don't believe it is a question of developer skill; the companies all have smart, capable, experienced engineers.

That an Open Source implementation would compare so favorably to commercial implementations was a surprising result and one that we thought others would be interested in. The question remains, what is different about the Open Source development process that, at least in this case, resulted in such a difference in measured quality? One of the most obvious differences between Open Source and proprietary development projects is the peer review process -- code inspection -- that is very intense for major Open Source projects, and underemployed in most commercial development projects because of resource and time constraints.

2) What needs to happen?
by argmanah

If open source has such a direct correlation to better quality, why do you feel companies are still keeping their source proprietary? Do you think that we should try and convince them to open source their code in every case, and if so, what do you think needs to happen before they can be convinced to change their minds?

Scott:

As I mentioned in the answer to the previous question, we haven't proven that Open Source is always superior, just that it can be. One of the clear advantages enjoyed by Open Source development projects is the luxury of time: the developers can take as long as they deem necessary to get it "right." Commercial projects rarely have this opportunity. Customer demands or competitive pressures often force aggressive schedules, and teams simply run out of time.

Even if Open Source was always superior in quality, quality and profitability is not the same thing. Most commercial software companies are leery of Open Source because they don't see how to make money at it. Red Hat, perhaps the most successful company with a business model based on Open Source reported only a tiny profit last year. Until executives are convinced that making their software Open Source will not negatively impact their profitability or valuation, it won't happen.

I think trying to lobby companies to make ALL their code Open Source will fail. However, I can see a hybrid approach where companies make some of their software Open Source and keep some of it closed. For example, some of the engineers and marketing staff inside Reasoning lobbied me to make our "Code Collection" tools Open Source. They convinced me that this provided a benefit for our customers at relatively little cost. Other companies may be persuaded to do the same on at least a limited basis.

3) Influence of project size
by arvindn

The parallelizability of bug-fixing is quite clearly very effective for high-visibility projects such as the linux kernel and apache. However, considering that most open-source projects have only between 1 and 5 developers, how popular do you think a project needs to be for it to significantly benefit from people looking at the source code?

Scott:

I think a project of ANY size benefits from peer review, and that it is even more important if there is only a single developer! That said, there is no question that as the number of active developers on a project grows, the opportunity for misunderstandings and miscommunication grows geometrically. A significant portion of the errors we find are clearly the result of two (or more) developers working on different parts of the code not understanding the interfaces between their components.

4) Quality Software vs Fewer Bugs?
by gosand

I work in software Quality Assurance, and have for going on 10 years now. My experience tells me that true software bugs are only part of the quality of software. So much can get lost in the software development lifecycle. An unclear requirement can travel through the lifecycle and come out the other end as a bug to the customer. Usability is another part of quality. It could be bug-free, but if it is really difficult to use or doesn't fit the needs of the customer, it may not matter.

It sounds like your company focuses on analyzing the code bugs, and not necessarily the perceived bugs. What are your opinions on this? I know that locating and eliminating the bugs *is* a critical part of software QA, but do you feel that bug-free ensures true quality? A bug-free Open Source project may still be too difficult to use or confusing for the non-technically inclined.

Scott:

You are absolutely right, and we don't claim that Illuma is a panacea. For the two issues you raised, there are good methods available today. Conventional black-box testing is a good way to confirm that an application meets its specified requirements, and usability testing with "real" users is important to determine if the customer's needs are truly being met. The problem that we see is that typically there are so many "code bugs" in the software that QA ends up spending most of their time running into those and not having enough time to adequately confirm conformance to the requirements. What Reasoning seeks to do is change that by providing developers with frequent, in-depth feedback on code bugs so that when QA does get the application, it is largely functional and robust.

5) General quality of programming
by pro-mpd

Do you find that the quality of the programming depends upon the geographic location of the programmers? So, for instance, an open source program will be troubleshot and combed over by people from potentially a dozen different countries. Closed source software is checked by people where it is written. Since, as a general rule, education varies in quality and areas of emphasis around the world, does it help having people attacking a program from many different angles (i.e., open source, cheked world wide) rather than simply drawing from a set of people who may share many of the same abilities, backgrounds, etc.?

Scott:

You have really posed two different questions: (1) how does software quality vary based on the location of the developers, and (2) do geographically distributed teams result in better quality because of the (implied) diversity of the developers?

To answer the first question, there are certainly significant differences in the quality of education around the world, and I believe cultural factors also play a role. However, numerous studies have shown that the programmer productivity and quality varies over a huge range - 10 to 1 or more, even when controlling for ethnicity and education. So I believe quality depends more on the individuals that make up the team than does their location.

For the second question, I disagree with the underlying assumption. Open Source projects could well be composed of people from similar backgrounds and education. In contrast, many commercial development teams, whether geographically concentrated or distributed, have a tremendous diversity among their developers. Again, I think the individual programmers are by far the most significant factor, and that location plays a much smaller role.

6) Issues behind test cases for proprietary vs. open
by Tekmage

One of the bigger challenges facing open source projects as compared to their proprietary equivalents is how to manage confidentiality of test cases. With companies such as Red Hat and Ximian involved, it's certainly less of an issue for their core products and projects they over-see, but there will always be cases where there is friction when the best/only person who can fix a particular problem is on the outside, unable to work with the test cases in question.

What are your thoughts on this trade-off between test case management and confidentiality as it relates to proprietary v.s. open source code development?

Scott:

This is a common problem regardless of the state of the source. For example, a semiconductor company may run into a serious defect with a device simulation program. They may view their "test case" -- the chip they are designing - as being so proprietary that they are not willing to share it with anyone outside, regardless of whether the application is Open Source or closed. Open Source may have an advantage if it is practical to bring in an outside expert on the application and do the bug-fixing work on the customer's premises. The framework that exists for open source projects (SourceForge.net, etc.) is better suited for this than most proprietary development environments. However, companies will always look for a competitive edge to deliver superior results relative to their competitors. If the company is based on Open Source, then they must identify other means to differentiate themselves. Red Hat, Suse, and others differentiate based on (a) components included in their distribution; (b) a "certification" that the integrated components will work together; and (c) quality of service and support. Thus, keeping test suites proprietary is a logical way for a company like Red Hat, for example, to differentiate its version of Linux from others, and I think this will be common in businesses based on Open Source.

7) How do you maintain your neutrality?
by arvindn

Given that on more than one occasion "independent institutions" which conducted similar studies (and concluded that closed source is superior) were revealed to have been sponsored by the other side [microsoft.com], how do you convince other people of your neutrality? Since you are selling a service [reasoning.com], not a product, I would guess that the confidence of your customers in your independence is pretty important from a business perspective. How do you win and keep that confidence? The article notes that you agree with ESR's pro open-source reasoning. Wouldn't the perception of your having a OSS bias be something you'd want to avoid?

Scott:

As I answered in question #1, we didn't set out to prove Open Source superior or inferior to proprietary software. Our mission is to give our customers a competitive edge by helping them deliver great software--by accelerating their development process, saving time and money. Code inspection has been demonstrated to be the single most effective way to eliminate defects, and we offer a way for organizations to perform thorough inspections quickly and cheaply. When we started our comparison, we didn't know how it would turn out, but the results confirm our belief that code inspection -- which happens naturally for major Open Source applications -- has a dramatic impact on software quality.

8) So if open source is so good...
by anthony_dipierro

Where can I get the source code to these automated inspection tools?

Scott:

How much are you willing to pay? :-)

Seriously, as I discussed in the answer to question #2, I haven't seen a way to provide Reasoning's shareholders with an equivalent (much less superior) return by making our source code Open. I think this is one of the most significant challenges that advocates of Open Source have yet to successfully address.

9) The future of automated code inspection
by phamlen

According to the article, it appears that you look for buffer overflows, freeing memory early, and other memory issues.

What errors are currently hard to detect automatically but which you would really like to be able to find? What is the next category of errors that you're trying to detect with automatic code inspection?

To give you some ideas, what about:

• "unrefactored" code - code which has a lot of duplication and should be cleaned up
• "untested" code - code (or branches in the code) that are currently untested by unit tests?
• "programmer intention" errors - code which doesn't do what the programmer intends

Scott:

Well, I've always wanted an editor with a "DWIM" command (Do What I Mean) so I wouldn't have to think so hard when I'm writing -- whether it's code or just a letter. :-)

More seriously, as was alluded to in the answer to question #4, automated inspection can detect "errors of effect" but not "errors of intent". By that, I mean that we cannot validate that the application does what the requirements document says it should. For example, if the requirement for a routine is that it calculate the geometric mean of a list of numbers, but instead it calculates the arithmetic mean, we cannot detect that. Today, we detect NULL pointer dereferences, references to uninitialized variables, array bounds violations (a superset of buffer overflows), memory leaks, and bad deallocations.

In general, a static analysis approach is limited to identifying defects resulting from (a) internal inconsistencies -- such as setting a pointer to NULL at one point in the program and dereferencing that pointer at a later point; and (b) violations of constraints imposed by the language semantics or standard library functions.

So, staying within those bounds, I'd like to find: (a) Concurrency-related defects: deadlocks, race conditions, etc. These are notoriously difficult to find and even harder to test. (b) Resource leaks: similar to memory leaks but applied to other types of resources such as file descriptors, handles, etc. (c) Protocol violations: misusing an API, such as trying to read or write a file after it has been closed. (d) Dead code: code that could never be executed. (e) Performance bottlenecks: either inefficient algorithms (such as using Bubble sort), or indications of "hot spots" in the code.

Your ideas are good too; so far we have focused on "critical defects" -- those likely to make the application abort, hang, or silently corrupt data. Your first example is more in the category of "poor programming practice". I can see integrating this type of analysis in the future, because there is a correlation between overly complicated routines and the likely presence of bugs.

10) Test first
by neurojab

What do you think about the new "test first" software development methodology? For those that haven't heard of it, it's a method wherein the test cases for a program are written, and no code is written that doesn't cause a failing test case to pass. All test cases are automated and run after every code change. Would you advocate this in an open-source project? This would mean every contributor would write test cases for each new feature, and add it to a project's common test case repository... What do you think?

Scott:

In my experience, programmers like to write code. Period. They don't like to write documentation, they don't like to write system tests, and they don't like to write unit tests. Programmers are also optimists--how else could they tackle building these enormously complex systems and think they had any chance of working? Programmers like instant gratification (who doesn't?). They enjoy coming up with a solution to a problem and seeing that solution implemented immediately.

Because programmers are optimists, that is reflected in their unit tests. Time and time again I've seen developer-written tests that demonstrate the feature works -- because the tests reflect the thinking of the developer about how the feature will be used. They rarely do a good job of testing corner cases, limits, or "unusual" situations (like running out of memory or other finite resources).

I think the "test first" methodology is too at odds with what motivates programmers to do what they do. Would Linux have ever been created if Linus' original postings to the net had been test cases for a UNIX-like operating system? And invited others to write more test cases? How many would have responded? How many would have become excited about the prospect of building an Open Source operating system if the first year was going to be spent writing unit tests?

Maybe I'm just a skeptic, but Test First reminds me of so many other software development methodologies proposed over the years that promise great benefits but rarely deliver them. That said, I am excited by other aspects of some of the newer methodologies, such as the "program in pairs" in Extreme Programming.

Comments

Independent code checking

[TopShelf]

The point about Open Source code verification being stronger due to people in different locations checking the code is a particularly good one. We "supposedly" have code reviews within my IS department, but since they're being done by other members of the programming team, there's a personal familiarity there that undermines truly independent analysis. Too often, code is simply scanned over and rubber-stamped rather than thoroughly reviewed.

Re:Independent code checking

[tcopeland]

Yup, code reviews are fraught with pitfalls.

Here's another scenario: you write a class, it gets reviewed by someone and they suggest you use a Factory pattern. You rewrite it to use Factory, and then it gets reviewed by someone else and they say "why are you adding in that Factory pattern? that just makes it more complicated to use - you're over abstracting!". Both reviewers are right in a way... it all depends on how the class is being used.

There's a nice discussion of alternatives to code reviews here.

tom

Re:Independent code checking

[scott_trappe]

I think the issue is objectivity rather than geography. How able are you to comment thoroughly on someone else's work when you eat lunch with them every day? And you both are facing tight deadlines for your project?
One benefit of using an outsourced automated inspection service like Reasoning's Illuma is that there is no axe to grind, no preconceived notions, and no fear of retribution ("give me a bad review and I'll give you a bad review").

Re:Independent code checking

[SquareOfS]

I also worked in a CMM'd company (oh, the temptation to find a new meaning for that acronym).

I saw two major problems with our code reviews, other than the one noted above, which was also present.

First, timeframe issues. Inspections quickly become the enemy (rightly or wrongly) when you have irrational deadlines. Then the deadline causes managers to exert pressure to "streamline" inspections, which, regardless of the intent of the manager, means that inspections get turned into puppet theater, and good engineers learn to hate them. Presto: vicious cycle.

Second, "inspector distance". By this I mean that the available inspectors in most closed-source development models are likely too close to the implementation to inspect it appropriately. They've either (1) written the requirements, (2) consulted with or mentored the developer who actually wrote the code, (3) done the architectural design, etc. (Or all of the above.)

Now, the same thing clearly is true (to a degree) in open-source development, and those people are likely to be the best inspectors available. But I think there's an important psychological effect in inspection-at-a-distance. Reviewing the code of someone who doesn't just sit two cubes away from you with whom you talk about the product over coffee helps open one's eyes.

Why Didn't They Ask The Metrics Question?

[Carnage4Life]

My question was modded to 5 in the thread asking for interview questions and it seems to be very relevant when discussing claims like one piece of code is more or less buggy than another. Since Slashdot refused to ask him the question can someone with an inside track answer the following question?

I assume some of this information may be "company secrets" but I'm very interested in learning what metrics are used to determine which source code is "buggier" than others. Is this something as simple as running lint + "gcc -Wall -ansi -pedantic" then piping the output to "wc -l" ?

Are there checks for use of unsafe functions like gets and the str* family of functions in C? Are there more complex data flow analysis algorithms at play here like those in the used in Stanford's Meta-level compilation techniques?

Inquiring minds want to know. A pronouncement like OS foo has more/less bugs than OS bar is meaningless without a definition of what having more/less bugs means.

What about HIS best practices?

[192939495969798999]

I wonder, what special tactics (if any) does he truly recommend for programming within the company he works for?

Blah, everyone knows, we're just too lazy

[Ars-Fartsica]

Everyone knows how to improve their code. Peer review, reuse, acutally knowing how to program in their language of choice, etc etc....yet no one does it. Why not just fess up - programmers are lazy because we can get away with it.

Lets face it, msot of us do printf debugging and assume that if it doesn't core dump, it works.

Re:Blah, everyone knows, we're just too lazy

[tcopeland]

Wellll.... maybe.... but I think that for many Java projects, at least, a culture of JUnit tests is developing.

Seeing a JUnit report page like this certainly gives me a bit more confidence in a project...

Tom

Umm

[arvindn]

(Answer to Q.3):

I think a project of ANY size benefits from peer review, and that it is even more important if there is only a single developer!

Maybe my question was not quite clear. I didn't mean how big should a project be for peer review to be benificial. I meant how big should a project be so that it gets any peer review at all in the first place. In particular, I was asking about the thousands of OSS projects that have between 1 and 5 developers. Do these get any significant amount of patches/bugfixes to justify their being open source?

Any opinions?

test case management as competitive edge

[Tekmage]

Thanks for answering my question (Q6), Scott. The IC example brings it home for me, given that it's my own area of expertise. A good part of the challenges I face day-to-day are coming up with simpler, non-proprietary test cases that capture the same symptoms and "reproducability" of a given bug.

I like the point you make, about maintaining proprietary and confidential test suites being a key ingredient in developing a business (aka making money and being profitable) based on Open Source solutions. The business relationship is built on trust and survives based on a sustainable, mutual relationship. It isn't dependent on code remaining secret, and if the relationship breaks down (lost trust and confidence) the business/individual using the software isn't stranded.

Thanks for the insight!

Detection of unrefactored code

[Ed Avis]

"unrefactored" code - code which has a lot of duplication and should be cleaned up

The other day I hacked up a similarity tool which gives an index between 0 and 1 of the cross-redundancy between files. (It gzips the concatenation of the two files and compares this with gzipping them independently.) Maybe a version control system could run this on all new file checkins, so a file is rejected if it is too similar to some file already under version control. This would cut down on cut-and-paste of existing code, which is generally a Bad Thing (refactor instead).

Re:Testing

[pmineiro]

In a corporate environment, isn't this what testers are for? You don't waste the programmers time on this ... In fact, they shouldn't be the same people.

I agree that ultimately, one needs a seperate QA department for the end game assessments, because after a long time of staring at the same code, a set of people will no longer be able to see any further defects.

However, I think it's important for programmers to take an adversarial approach to their own code. I definitely don't feel that it "wastes programmer time" to write their own tests, and therefore think about how to test the code. In my experience this leads to constructing more easily testable components which in turn increases modularity and improves design.

Also, on a social note, I believe it in unnatural and counterproductive to consider one activity (initial generation of inevitably defective code) to be more important or worthy of developer time than another activity (identification and elimination of introduced code defects) when both activities are critical to final software quality. Basically, then nobody wants to write tests because it is considered an inferior task!

-- p

How to make $$ with Open Source

[PyTHON71]

"I haven't seen a way to provide Reasoning's shareholders with an equivalent (much less superior) return by making our source code Open. I think this is one of the most significant challenges that advocates of Open Source have yet to successfully address."

Sell your binaries and give away the source. That's how easy it is.

Free Software/Open Source does not equal no-cost binaries. It never has, and it never will. If a company offers a binary, it's a gift. If the company feels obliged to give away a binary, they've misunderstood FS/OS. If a hacker feels that the company is obliged to provide binaries, he's less of a hacker and more of a slacker. All a hacker has a right to is unrestricted access to and use of the source code. Binaries are not a right; they are a privilege.

Linux distros lose money because they give away binaries to everything, pure and simple. Why they give them away, I don't know. I suspect it's part ignorance, part hope that a 'free' product will find an audience that will get hooked now and pay for support later. I suspect that RedHat has taken this a step further -- release a basic distro that's buggy enough to encourage users to upgrade to RHAS. SUSE has a better solution: binaries are available four weeks after the product goes on sale. Who can't wait four weeks for free binaries, if you can't afford the boxed set?

To be blunt, companies that are serious about turning a real profit from FS/OS products must put their source code on their FTP server and charge for about everything else except basic documentation. If a hacker can compile OpenWhazzit all by his lonesome, let him. He poses no threat to the profit margin. If he posts a HOWTO on the subject, there's still no problem. He and everyone else who reads it will probably document and/or fix the bugs they find in the same way. He may even give away or sell binaries of FreeOpenWhazzit, but should that really matter to the bottom line? Do they think Boeing or Pepsico or Mitsubishi would pass up 1,000 copies of BlueGlove Advanced Server or in favor of Joe's Blue Garage Linux? Odds are, even if Joe's distro represents a siginificant increase in quality, they would still buy the name brand. And so will the average home user who wants to use a real, inexpensive OS. That is, unless BlueGlove Advanced Server is so bad that Joe's distro finds a niche in the marketplace, in which case, shame on BlueGlove for selling a bad product.

That means, in the end, more people are using your product, so more people are buying your product, and more people are fixing your product at not cost to you. I'd say that's what you want your investors to hear.

And before the flames arrive, understand this: I have nothing against giving away binaries, as long as it's understood by everyone that they are gifts, not obligations. And if businesses want to survive, they're going to have to learn this lesson quickly. That will mean that hackers will either have to get used to compiling or switch to Debian.

Daniel.

Re:How to make $$ with Open Source

[Glyndwr]

See Transgaming's WineX, which works exactly like this. Of course, the detailed howtos on Franksworld about how to compile WineX out of the (open access) CVS does make me less likely to subscribe.

But doesn't this just encourage developers to ship slightly broken source? Say, all the Makefiles deleted or something?

Very telling example...

[Anonymous Brave Guy]

Hey, strcpy is no buggier than anything else if you use it properly. I recently was forced to change my code as follows:

char *foo = new char[strlen(bar) + 1];
strcpy(foo,bar);

to

char *foo = new char[strlen(bar) + 1];
strncpy(foo,bar,strlen(bar));

What's the problem with this?

Your example is particularly apt to a discussion about the quality of reviewers. You comment that those reviewing your code suggested (wrongly) that the second version was safer. My immediate suggestion, had I been doing the review, would have been to use std::string (or some other string class if you're not allowed templates/standard library stuff for whatever reason) instead of low-level pointer and array manipulation.

This is the big problem with code reviews: not only do the reviewers need to be unbiased and able to speak their minds freely, they also have to be well informed and qualified to offer the opinions for which they are asked. 'Course, there aren't that many suitably qualified people around, and you then get the question of whether their peer time would be better used conducting training (for example, in general principles like how to avoid using low-level constructs in high-level code in a language like C++) than in individually reviewing single pieces of code, and probably making the same criticism of half the developers in the department in different places.

Code reviews in themselves are clearly a good thing, but if you have a scarce resource (very good programmers), you have to look at the different ways you can take advantage of their skills, and using them for code reviews may not be the most constructive use of their finite amounts of time.

Length sensitive checks on strings

[tjwhaynes]

Hey, strcpy is no buggier than anything else if you use it properly. I recently was forced to change my code as follows:

char *foo = new char[strlen(bar) + 1]; strcpy(foo,bar);
to
char *foo = new char[strlen(bar) + 1]; strncpy(foo,bar,strlen(bar));

Shame your reviewers knew enough to know about strncpy() but not enough to know about strnlen() and how to handle the out-of-range conditions from strnlen()... But then of course, you must have rewritten your code better than you indicate here.

So what were you programming? Ah - you were working on Mozilla. Still, I'm sure that is completely stable and crashpr

NO CARRIER

Re:Length sensitive checks on strings

[binaryDigit]

So what were you programming? Ah - you were working on Mozilla. Still, I'm sure that is completely stable and crashpr

Well if you're referring to the various buffer overrun bugs that allow "arbitrary code" to execute, this would not be an example since the memory is allocated on the heap and guaranteeing the heap would be in the right condition to create say an overwrite of a vtable would be very difficult without explicit knowledge of the code itself and this code is not open source. Most of those issues deal with having fixed size stack buffers anyway.

This function takes a string and copies it to a working buffer. It's on the heap, its in a method that is supposed to handle arbitrarily long buffers. If it's an honest mistake (which the method can't know), then the chances of running into another 0 character are pretty high before you get too far. Either way, the worse case is that you end up absorbing all available virtual memory and the program stops with an exception (this is how we have our new handler set). In this case, having the exception is better, as it helps prevent "silent" behviours from occuring.

You're right but it almost never happens.

[SuperKendall]

The first place I worked for, had a great QA department (which I started in). They wrote and maintained thousands and thousands of unit tests, and when things failed they would also discuss with the developers if the test or the code was broken (because even tests can be buggy). They even went so far as to have code compiled on checkin and run nightly batches of all unit tests on all platforms the code supported - code checked in that broke compilation was rejected, and nightly unit test results went out each day with the people that had checked in code the day before looked at most closley to sove the issues. It all sounds very draconian but really it worked pretty well for the developers (and I should know, as I became a developer for a while after the QA stint) and didn't feel at all intrusive to work with - since if you coded well, things all went smoothly and the process never touched you.

That was a small company ( 100 employees) who really, really cared about the quality of thier code - and it showed, they had a very stable and well-written product. It was the only time I've seen real SCM and QA in action and it was great.

Slowly over time I moved into larger and larger companies - never again did I see that level of attention to the SCM and QA process. Instead I see a lot of beurocracy that calls itself SCM, but is really just wastes a lot more of my and my team's time than maintaining my own CVS tree. There is more emphasis on knowing who to blame when something goes wrong, than to help produce working code leveraging as many people as possible to fix something that might be wrong.

So you are right, the QA writing tests is the idyllic case. But alomost no-one does it, at least at the moment. The funny thing is that using code workshops in India to do QA tests while keeping development in the US would probably be the optimal model, since it would actually take advantage of the time difference and not be too affected by seperation, but instead companies go too far (as usual) and just ship everything overseas (well, some companies anyway).

I Heart Unit Testing

[dubl-u]

In a corporate environment, isn't this what testers are for? You don't waste the programmers time on this,

Both sorts of testing are helpful.

Functional tests make sure that the program meets the requirements. Good QA people are invaluable.

But even if I had the world's best team of testers, I'd still do test-first programming. Why?

• Faster feedback. No matter how fast a testing team is, it's not fast enough for me. With automated unit tests, I know within seconds that my code is good.
• Clearer interfaces. When I do test-first programming, I'm always thinking about what my code looks like from an internal perspective. This makes the interfaces to it much clearer for others to deal with.
• Automated documentation. When I wnt to know what a chunk of code is supposed to do or how it's supposed to be used, I just look at the unit tests.
• Rot prevention. Having automated unit tests gives me confidence that the code will continue to work, even when I'm not looking at it. Change in libraries? Junior programmers making changes? I'm not worried; the tests will catch the problems.
• Refactoring support. If you don't have good test coverage, doing serious refactoring is impossible.

As far as I can tell, every programmer does manual unit testing, generally by putting in print statements as they develop and then looking for stuff in the output. Doing automated unit testing just means that you take those manual checks and automate them. And isn't automating tedious manual work what computers are all about?