Slashdot Mirror
Scott Trappe's Answers About Code Quality
1) sample size and conclusions
by tim_maroney
How can any conclusions about the relative virtues of two development methodologies with a universe in the millions of components be drawn from a single sample, and one as small and atypical as a TCP/IP stack?
Scott:
You raise an excellent point, and answering it will take some exposition, so bear with me.
You're right, these results do not "prove" that Open Source is superior to proprietary development, nor were we trying to prove superiority of either development process. However, it does counter the numerous articles and studies written alleging that Open Source software is inherently inferior in quality because there is no formal development process, no comprehensive test plan or infrastructure and no dedicated resources to provide follow-on support and maintenance. Our study is a single counterexample which discredits those assertions. We are in the process of inspecting other Open Source projects to see if these results are anomalous or not, and will report our findings.
We chose to examine the TCP/IP stack because it's "atypical" in a way that makes it perfect for comparison: it has a very well-defined set of published requirements that have been stable for several years; adherence to published standards is an essential element of any implementation; there are hundreds of books and articles covering design alternatives, performance measurements, and sample implementations; and there are publicly available conformance tests.
Thus, unlike most software projects, all the implementations we looked at had a common set of requirements to meet and access to a wealth of information about how to design a high performance, well behaved implementation. All of the implementations have been in existence for several years. Thus in comparing them, we didn't have to consider the quality of the requirements document, the uniqueness of the design challenges, nor the resources available for validation.
In short, all the ingredients exist for all of these implementations to be of very high quality. The commercial implementations should enjoy the significant advantages that other studies have pointed out, i.e., resources available to them to insure comprehensive testing, response to bugs, etc. The five commercial implementations were developed at large corporations that have well-deserved reputations for developing high quality products, and invest significant resources to insure that quality. I don't believe it is a question of developer skill; the companies all have smart, capable, experienced engineers.
That an Open Source implementation would compare so favorably to commercial implementations was a surprising result and one that we thought others would be interested in. The question remains, what is different about the Open Source development process that, at least in this case, resulted in such a difference in measured quality? One of the most obvious differences between Open Source and proprietary development projects is the peer review process -- code inspection -- that is very intense for major Open Source projects, and underemployed in most commercial development projects because of resource and time constraints.
2) What needs to happen?
by argmanah
If open source has such a direct correlation to better quality, why do you feel companies are still keeping their source proprietary? Do you think that we should try and convince them to open source their code in every case, and if so, what do you think needs to happen before they can be convinced to change their minds?
Scott:
As I mentioned in the answer to the previous question, we haven't proven that Open Source is always superior, just that it can be. One of the clear advantages enjoyed by Open Source development projects is the luxury of time: the developers can take as long as they deem necessary to get it "right." Commercial projects rarely have this opportunity. Customer demands or competitive pressures often force aggressive schedules, and teams simply run out of time.
Even if Open Source was always superior in quality, quality and profitability is not the same thing. Most commercial software companies are leery of Open Source because they don't see how to make money at it. Red Hat, perhaps the most successful company with a business model based on Open Source reported only a tiny profit last year. Until executives are convinced that making their software Open Source will not negatively impact their profitability or valuation, it won't happen.
I think trying to lobby companies to make ALL their code Open Source will fail. However, I can see a hybrid approach where companies make some of their software Open Source and keep some of it closed. For example, some of the engineers and marketing staff inside Reasoning lobbied me to make our "Code Collection" tools Open Source. They convinced me that this provided a benefit for our customers at relatively little cost. Other companies may be persuaded to do the same on at least a limited basis.
3) Influence of project size
by arvindn
The parallelizability of bug-fixing is quite clearly very effective for high-visibility projects such as the linux kernel and apache. However, considering that most open-source projects have only between 1 and 5 developers, how popular do you think a project needs to be for it to significantly benefit from people looking at the source code?
Scott:
I think a project of ANY size benefits from peer review, and that it is even more important if there is only a single developer! That said, there is no question that as the number of active developers on a project grows, the opportunity for misunderstandings and miscommunication grows geometrically. A significant portion of the errors we find are clearly the result of two (or more) developers working on different parts of the code not understanding the interfaces between their components.
4) Quality Software vs Fewer Bugs?
by gosand
I work in software Quality Assurance, and have for going on 10 years now. My experience tells me that true software bugs are only part of the quality of software. So much can get lost in the software development lifecycle. An unclear requirement can travel through the lifecycle and come out the other end as a bug to the customer. Usability is another part of quality. It could be bug-free, but if it is really difficult to use or doesn't fit the needs of the customer, it may not matter.
It sounds like your company focuses on analyzing the code bugs, and not necessarily the perceived bugs. What are your opinions on this? I know that locating and eliminating the bugs *is* a critical part of software QA, but do you feel that bug-free ensures true quality? A bug-free Open Source project may still be too difficult to use or confusing for the non-technically inclined.
Scott:
You are absolutely right, and we don't claim that Illuma is a panacea. For the two issues you raised, there are good methods available today. Conventional black-box testing is a good way to confirm that an application meets its specified requirements, and usability testing with "real" users is important to determine if the customer's needs are truly being met. The problem that we see is that typically there are so many "code bugs" in the software that QA ends up spending most of their time running into those and not having enough time to adequately confirm conformance to the requirements. What Reasoning seeks to do is change that by providing developers with frequent, in-depth feedback on code bugs so that when QA does get the application, it is largely functional and robust.
5) General quality of programming
by pro-mpd
Do you find that the quality of the programming depends upon the geographic location of the programmers? So, for instance, an open source program will be troubleshot and combed over by people from potentially a dozen different countries. Closed source software is checked by people where it is written. Since, as a general rule, education varies in quality and areas of emphasis around the world, does it help having people attacking a program from many different angles (i.e., open source, cheked world wide) rather than simply drawing from a set of people who may share many of the same abilities, backgrounds, etc.?
Scott:
You have really posed two different questions: (1) how does software quality vary based on the location of the developers, and (2) do geographically distributed teams result in better quality because of the (implied) diversity of the developers?
To answer the first question, there are certainly significant differences in the quality of education around the world, and I believe cultural factors also play a role. However, numerous studies have shown that the programmer productivity and quality varies over a huge range - 10 to 1 or more, even when controlling for ethnicity and education. So I believe quality depends more on the individuals that make up the team than does their location.
For the second question, I disagree with the underlying assumption. Open Source projects could well be composed of people from similar backgrounds and education. In contrast, many commercial development teams, whether geographically concentrated or distributed, have a tremendous diversity among their developers. Again, I think the individual programmers are by far the most significant factor, and that location plays a much smaller role.
6) Issues behind test cases for proprietary vs. open
by Tekmage
One of the bigger challenges facing open source projects as compared to their proprietary equivalents is how to manage confidentiality of test cases. With companies such as Red Hat and Ximian involved, it's certainly less of an issue for their core products and projects they over-see, but there will always be cases where there is friction when the best/only person who can fix a particular problem is on the outside, unable to work with the test cases in question.
What are your thoughts on this trade-off between test case management and confidentiality as it relates to proprietary v.s. open source code development?
Scott:
This is a common problem regardless of the state of the source. For example, a semiconductor company may run into a serious defect with a device simulation program. They may view their "test case" -- the chip they are designing - as being so proprietary that they are not willing to share it with anyone outside, regardless of whether the application is Open Source or closed. Open Source may have an advantage if it is practical to bring in an outside expert on the application and do the bug-fixing work on the customer's premises. The framework that exists for open source projects (SourceForge.net, etc.) is better suited for this than most proprietary development environments. However, companies will always look for a competitive edge to deliver superior results relative to their competitors. If the company is based on Open Source, then they must identify other means to differentiate themselves. Red Hat, Suse, and others differentiate based on (a) components included in their distribution; (b) a "certification" that the integrated components will work together; and (c) quality of service and support. Thus, keeping test suites proprietary is a logical way for a company like Red Hat, for example, to differentiate its version of Linux from others, and I think this will be common in businesses based on Open Source.
7) How do you maintain your neutrality?
by arvindn
Given that on more than one occasion "independent institutions" which conducted similar studies (and concluded that closed source is superior) were revealed to have been sponsored by the other side [microsoft.com], how do you convince other people of your neutrality? Since you are selling a service [reasoning.com], not a product, I would guess that the confidence of your customers in your independence is pretty important from a business perspective. How do you win and keep that confidence? The article notes that you agree with ESR's pro open-source reasoning. Wouldn't the perception of your having a OSS bias be something you'd want to avoid?
Scott:
As I answered in question #1, we didn't set out to prove Open Source superior or inferior to proprietary software. Our mission is to give our customers a competitive edge by helping them deliver great software--by accelerating their development process, saving time and money. Code inspection has been demonstrated to be the single most effective way to eliminate defects, and we offer a way for organizations to perform thorough inspections quickly and cheaply. When we started our comparison, we didn't know how it would turn out, but the results confirm our belief that code inspection -- which happens naturally for major Open Source applications -- has a dramatic impact on software quality.
8) So if open source is so good...
by anthony_dipierro
Where can I get the source code to these automated inspection tools?
Scott:
How much are you willing to pay? :-)
Seriously, as I discussed in the answer to question #2, I haven't seen a way to provide Reasoning's shareholders with an equivalent (much less superior) return by making our source code Open. I think this is one of the most significant challenges that advocates of Open Source have yet to successfully address.
9) The future of automated code inspection
by phamlen
According to the article, it appears that you look for buffer overflows, freeing memory early, and other memory issues.
What errors are currently hard to detect automatically but which you would really like to be able to find? What is the next category of errors that you're trying to detect with automatic code inspection?
To give you some ideas, what about:
- "unrefactored" code - code which has a lot of duplication and should be cleaned up
- "untested" code - code (or branches in the code) that are currently untested by unit tests?
- "programmer intention" errors - code which doesn't do what the programmer intends
Scott:
Well, I've always wanted an editor with a "DWIM" command (Do What I Mean) so I wouldn't have to think so hard when I'm writing -- whether it's code or just a letter. :-)
More seriously, as was alluded to in the answer to question #4, automated inspection can detect "errors of effect" but not "errors of intent". By that, I mean that we cannot validate that the application does what the requirements document says it should. For example, if the requirement for a routine is that it calculate the geometric mean of a list of numbers, but instead it calculates the arithmetic mean, we cannot detect that. Today, we detect NULL pointer dereferences, references to uninitialized variables, array bounds violations (a superset of buffer overflows), memory leaks, and bad deallocations.
In general, a static analysis approach is limited to identifying defects resulting from (a) internal inconsistencies -- such as setting a pointer to NULL at one point in the program and dereferencing that pointer at a later point; and (b) violations of constraints imposed by the language semantics or standard library functions.
So, staying within those bounds, I'd like to find: (a) Concurrency-related defects: deadlocks, race conditions, etc. These are notoriously difficult to find and even harder to test. (b) Resource leaks: similar to memory leaks but applied to other types of resources such as file descriptors, handles, etc. (c) Protocol violations: misusing an API, such as trying to read or write a file after it has been closed. (d) Dead code: code that could never be executed. (e) Performance bottlenecks: either inefficient algorithms (such as using Bubble sort), or indications of "hot spots" in the code.
Your ideas are good too; so far we have focused on "critical defects" -- those likely to make the application abort, hang, or silently corrupt data. Your first example is more in the category of "poor programming practice". I can see integrating this type of analysis in the future, because there is a correlation between overly complicated routines and the likely presence of bugs.
10) Test first
by neurojab
What do you think about the new "test first" software development methodology? For those that haven't heard of it, it's a method wherein the test cases for a program are written, and no code is written that doesn't cause a failing test case to pass. All test cases are automated and run after every code change. Would you advocate this in an open-source project? This would mean every contributor would write test cases for each new feature, and add it to a project's common test case repository... What do you think?
Scott:
In my experience, programmers like to write code. Period. They don't like to write documentation, they don't like to write system tests, and they don't like to write unit tests. Programmers are also optimists--how else could they tackle building these enormously complex systems and think they had any chance of working? Programmers like instant gratification (who doesn't?). They enjoy coming up with a solution to a problem and seeing that solution implemented immediately.
Because programmers are optimists, that is reflected in their unit tests. Time and time again I've seen developer-written tests that demonstrate the feature works -- because the tests reflect the thinking of the developer about how the feature will be used. They rarely do a good job of testing corner cases, limits, or "unusual" situations (like running out of memory or other finite resources).
I think the "test first" methodology is too at odds with what motivates programmers to do what they do. Would Linux have ever been created if Linus' original postings to the net had been test cases for a UNIX-like operating system? And invited others to write more test cases? How many would have responded? How many would have become excited about the prospect of building an Open Source operating system if the first year was going to be spent writing unit tests?
Maybe I'm just a skeptic, but Test First reminds me of so many other software development methodologies proposed over the years that promise great benefits but rarely deliver them. That said, I am excited by other aspects of some of the newer methodologies, such as the "program in pairs" in Extreme Programming.
The point about Open Source code verification being stronger due to people in different locations checking the code is a particularly good one. We "supposedly" have code reviews within my IS department, but since they're being done by other members of the programming team, there's a personal familiarity there that undermines truly independent analysis. Too often, code is simply scanned over and rubber-stamped rather than thoroughly reviewed.
Stop by my site where I write about ERP systems & more
I wonder, what special tactics (if any) does he truly recommend for programming within the company he works for?
stuff |
Second, re:
However, it does counter the numerous articles and studies written alleging that Open Source software is inherently inferior in quality because there is no formal development process, no comprehensive test plan or infrastructure and no dedicated resources to provide follow-on support and maintenance. Our study is a single counterexample which discredits those assertions.
This is yet another case of ESR-driven confusion between open source licensing and community-based development. There is no requirement that open-licensed code include patches from any teenager who can compile hello.c, let alone that it be developed without a formal development process or test plan.
Not that his point is completely misplaced, since the Linux TCP stack _is_ a community effort, but it's wrong to define open source in terms of community. The overwhelming majority of open projects are not developed the way Linux and KDE are.
What I'm listening to now on Pandora...
that's why slashdot never get slashdotted when it link to itself! you probably run l4hurd with linux tcp/ip stack right?
[self dealloc];
Seriously, as I discussed in the answer to question #2, I haven't seen a way to provide Reasoning's shareholders with an equivalent (much less superior) return by making our source code Open. I think this is one of the most significant challenges that advocates of Open Source have yet to successfully address.
Really? No open source advocates have addressed this at all?
My journal has hot
What the hell is that? .NET
I click on 'Read More' and the first thing is see is this HUGE ad for Microsoft Visual Studio
I know you have to pay the bills, but come on.
That was the LAST ad I expected to see under a thread that has "Linux....better than proprietary...(ie, MS)"
Slashdot has the Open Source 'slant', but pushes down Satan's (cough cough) Ads.
Maybe I'm over-reacting....No, I'm not.
Ryan Harden
All in all, I agree with much of what he has to say, but there was one thing that bugged me.
He said that TCP/IP was a good example because the requirements, and specifications were already there and were the same for everyone. That may have made the example easier to analyze, but in my opinion it gives an incomplete picture. How good the requirements, specification, and design are effects how good the implementation is. In order to have a complete comparison of open source and closed source you need to compare how well each of these steps (or their equivalents) occurs in each.
I personally suspect that the "think it through first" part could also be better in open source (if less formal) because there's no deadline pressure involved. Also, the people programming had to convince themselves that it was a worthwhile endeavor somehow, which requires them to convince themselves that at least the requirements are better.
The BSD stack is rather .. free.
Seems to be a rather solid implementation, perhaps even more so.
Lets face it, msot of us do printf debugging and assume that if it doesn't core dump, it works.
"Programmers like instant gratification"
Well then, we should opensource it! Gratification was meant to be free.
In fact, they shouldn't be the same people. That way there are two sets of people going over the requirements. Sometimes a developer will interpret a requirement differently than a tester. It allows ambiguous requirements to be found during the development cycle rather than having a person write the test, write the code, and have the customer say "I didn't want it to work like that."
Overrated Moderation: This posts sucks... because.
I think a project of ANY size benefits from peer review, and that it is even more important if there is only a single developer!
Maybe my question was not quite clear. I didn't mean how big should a project be for peer review to be benificial. I meant how big should a project be so that it gets any peer review at all in the first place. In particular, I was asking about the thousands of OSS projects that have between 1 and 5 developers. Do these get any significant amount of patches/bugfixes to justify their being open source?
Any opinions?
Thanks for answering my question (Q6), Scott. The IC example brings it home for me, given that it's my own area of expertise. A good part of the challenges I face day-to-day are coming up with simpler, non-proprietary test cases that capture the same symptoms and "reproducability" of a given bug.
I like the point you make, about maintaining proprietary and confidential test suites being a key ingredient in developing a business (aka making money and being profitable) based on Open Source solutions. The business relationship is built on trust and survives based on a sustainable, mutual relationship. It isn't dependent on code remaining secret, and if the relationship breaks down (lost trust and confidence) the business/individual using the software isn't stranded.
Thanks for the insight!
--The more you know, the less you know.
One of the benefits of the open source model is that even projects with only a single developer can get peer review. The reasons are simple, and obvious once they are stated. The only barrier to entry as a reviewer on an open source project is knowledge, of the project, the language, and the address to send bug reports to. It is often touted that open source turns every user into a potential co-developer. It should not be overlooked that projects with a large user base can receive not just bug reports, but also patches, from that user base.
I've found that when I investigate a problem to determine if there is a bug, the investigation often yields a solution because I have the source to inspect. Thus, when I submit a bug report, I can frequently include a candidate patch to fix it. Even projects with a single active developer can often have a number of occasional or even one-time contributors.
The net will not be what we demand, but what we make it. Build it well.
The real reason? I think it's because open-source coders get 'credit' for what they do. In a corporate job, the typical software engineer gets paid, and that's all. If the product even gets to the outside world and is successful, it's the CEO of the company that will take the credit.
It might be interesting to look at exceptions to that, though. I wonder how the GPLed code coming out of some place like Red Hat, happens to stack up against amateur-written code. (Do Red Hat programmers have the "luxury of time"? I suspect not, but don't really know enough about how the place works.)
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
However, it does counter the numerous articles and studies written alleging that Open Source software is inherently inferior in quality because there is no formal development process, no comprehensive test plan or infrastructure and no dedicated resources to provide follow-on support and maintenance ...
We chose to examine the TCP/IP stack because it's "atypical" in a way that makes it perfect for comparison: it has a very well-defined set of published requirements that have been stable for several years; adherence to published standards is an essential element of any implementation; there are hundreds of books and articles covering design alternatives, performance measurements, and sample implementations; and there are publicly available conformance tests.
Ok, I love open source as much as, no more than, the next guy. But being knee deep in LSAT preparation materials, I immediately spotted the weakness in the argument here.
1. Open source is believed to be inferior because of the lack of formal development process.
2. TCP/IP is an atypical situation where an external standards body has created requirements, test cases, and the other trappings of a formal development process.
3. TCP/IP open source implementations are greatly superior to non-open source implementations.
Scott goes on to talk about the virtues of increased code inspection in open source projects. I think this is a fair and interesting point, namely, when the "formal development process variables" are held constant (i.e. all other things being equal), the increased code inspection leads to greatly increased code quality.
However, it seems clear to me that the original contention that open source lacks formal development processes and therefore produces inferior code has not been directly addressed. We are left to wonder whether the virtues of increased code inspection can overcome the lack of formal development processes.
-- p
I don't really write tests for initial project requirements, but once the project is on the way all Bugs must have a test script (how to reproduce the bug).
If the test can be automated then test data is produced to verify that the bug has been corrected.
Also, I frequently write 'test scripts' when consulting change requirements. This helps resolve the requirements, makes testing easy and covers saves you neck if the client says that's not what they asked for(the tests are unambiguous).
Now, if only they put referances to the standards in the kernel source tree, I could do some 'real' reviewing.
thank God the internet isn't a human right.
That luxury of time comes both before and after the release for open source. Without the pressure of a specific feature list and release schedule being directly tied to a revenue stream it is obvious that open source projects have more leeway before the release date. However, the Release Early, Release Often paradigm has two parts. It presents a model that makes a project available to people all the way along the adoption curve. There is less pressure for make-or-break releases. This presents an opportunity for a more continuous cycle of releases and improvements.
The net will not be what we demand, but what we make it. Build it well.
The other day I hacked up a similarity tool which gives an index between 0 and 1 of the cross-redundancy between files. (It gzips the concatenation of the two files and compares this with gzipping them independently.) Maybe a version control system could run this on all new file checkins, so a file is rejected if it is too similar to some file already under version control. This would cut down on cut-and-paste of existing code, which is generally a Bad Thing (refactor instead).
-- Ed Avis ed@membled.com
Seriously, as I discussed in the answer to question #2, I haven't seen a way to provide Reasoning's shareholders with an equivalent (much less superior) return by making our source code Open. I think this is one of the most significant challenges that advocates of Open Source have yet to successfully address.
And advocates of Open Source never will, nor should they try.
Your products will: survive because they are unreproducable; survive because they go open source; or be replaced by an open source package that ships with 20 operating system distributions a month after its first stable release.
Roll those dice however you like. "Open Source advocates" shouldn't have to care, and we CERTAINLY should not have to hand-hold you through the decision. If you were making the decision for more than econimic reasons, then we could talk, but open source software development is fundamentally the denial of the idea that software is a valuable commodity (rather that programmers and support), so we're not going to help you understand how to preserve its status as such.
*Successfully* address.
Success is a value judgement, and without knowing his criteria for "success", you can't know whether ESR's works meet those criteria.
So, for instance, an open source program will be troubleshot and combed over by people from potentially a dozen different countries.
pro-mpd seemed to be attempting to steer a cultural-related response out of Scott Trappe. However, I really think geographically-distributed culture has little to do with the actual code quality.
What is more significant is that distributed developers tend to communicate mostly through written e-mail or newsgroups. Their very act of communication is documenting their decisions and design plan!
Many "bugs" are due to Joe-developer-A having a coffee-room discussion with Joe-developer-B, where the idea is good but not thought through and certainly not written down. So, when they go to implement it, they probably lost the insight they briefly had during their discussion and the end result isn't very good. This scenario is harder to achieve when there are e-mail archives to remind everyone what's going on.
Healthcare article at Kuro5hin
If a program being developed has an urgent customer demand, it applies pressure on the developers. Commercial developers can end up releasing too soon, and selling the customers an inferior product. Release early and release often means that open source developers also give the "customers" incomplete programs. But they are not selling to the customers, though, so they can clearly label this stage as Beta. If there is a genuine urgent need for the program, though, the open source developers now get an important advantage. Some of those anxiously waiting customers may decide that their itch is too big, and they stop waiting and assist the development team. They scratch their itch by sending in bug reports and possibly fixes for the parts that don't yet fill their needs, or they do whatever else they see that will help fulfill their own needs faster. The open source project starts moving faster (as long as the communication load doesn't start to get in the way).
I think the they was merely refuting the implied absolute of 'inherently inferior'. In other words:
1. All Open Source implementations are inferior because of the lack of formal development process
2. Comparing the TCP/IP handling, we found that the Open Source implementations are superior to proprietary implementations
Ergo: The common belief is not an absolute truth.
I don't think it was their intent to show that a lock of formal development is more or less likely to produce inferior code, only that it isn't an absolute result.
Of course, moving from pure logic to statistics it is clear that one test result isn't really enough to disprove the original assertion. However, it is enough to plant doubt and possibly lead to more comparisons along these lines.
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
Sell your binaries and give away the source. That's how easy it is.
Free Software/Open Source does not equal no-cost binaries. It never has, and it never will. If a company offers a binary, it's a gift. If the company feels obliged to give away a binary, they've misunderstood FS/OS. If a hacker feels that the company is obliged to provide binaries, he's less of a hacker and more of a slacker. All a hacker has a right to is unrestricted access to and use of the source code. Binaries are not a right; they are a privilege.
Linux distros lose money because they give away binaries to everything, pure and simple. Why they give them away, I don't know. I suspect it's part ignorance, part hope that a 'free' product will find an audience that will get hooked now and pay for support later. I suspect that RedHat has taken this a step further -- release a basic distro that's buggy enough to encourage users to upgrade to RHAS. SUSE has a better solution: binaries are available four weeks after the product goes on sale. Who can't wait four weeks for free binaries, if you can't afford the boxed set?
To be blunt, companies that are serious about turning a real profit from FS/OS products must put their source code on their FTP server and charge for about everything else except basic documentation. If a hacker can compile OpenWhazzit all by his lonesome, let him. He poses no threat to the profit margin. If he posts a HOWTO on the subject, there's still no problem. He and everyone else who reads it will probably document and/or fix the bugs they find in the same way. He may even give away or sell binaries of FreeOpenWhazzit, but should that really matter to the bottom line? Do they think Boeing or Pepsico or Mitsubishi would pass up 1,000 copies of BlueGlove Advanced Server or in favor of Joe's Blue Garage Linux? Odds are, even if Joe's distro represents a siginificant increase in quality, they would still buy the name brand. And so will the average home user who wants to use a real, inexpensive OS. That is, unless BlueGlove Advanced Server is so bad that Joe's distro finds a niche in the marketplace, in which case, shame on BlueGlove for selling a bad product.
That means, in the end, more people are using your product, so more people are buying your product, and more people are fixing your product at not cost to you. I'd say that's what you want your investors to hear.
And before the flames arrive, understand this: I have nothing against giving away binaries, as long as it's understood by everyone that they are gifts, not obligations. And if businesses want to survive, they're going to have to learn this lesson quickly. That will mean that hackers will either have to get used to compiling or switch to Debian.
Daniel.
Free software, not Iraq, because Bill Gates is evil & Saddam is just misunderstood.
Your example is particularly apt to a discussion about the quality of reviewers. You comment that those reviewing your code suggested (wrongly) that the second version was safer. My immediate suggestion, had I been doing the review, would have been to use std::string (or some other string class if you're not allowed templates/standard library stuff for whatever reason) instead of low-level pointer and array manipulation.
This is the big problem with code reviews: not only do the reviewers need to be unbiased and able to speak their minds freely, they also have to be well informed and qualified to offer the opinions for which they are asked. 'Course, there aren't that many suitably qualified people around, and you then get the question of whether their peer time would be better used conducting training (for example, in general principles like how to avoid using low-level constructs in high-level code in a language like C++) than in individually reviewing single pieces of code, and probably making the same criticism of half the developers in the department in different places.
Code reviews in themselves are clearly a good thing, but if you have a scarce resource (very good programmers), you have to look at the different ways you can take advantage of their skills, and using them for code reviews may not be the most constructive use of their finite amounts of time.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Hey, strcpy is no buggier than anything else if you use it properly. I recently was forced to change my code as follows:
char *foo = new char[strlen(bar) + 1]; strcpy(foo,bar);
to
char *foo = new char[strlen(bar) + 1]; strncpy(foo,bar,strlen(bar));
Shame your reviewers knew enough to know about strncpy() but not enough to know about strnlen() and how to handle the out-of-range conditions from strnlen()... But then of course, you must have rewritten your code better than you indicate here.
So what were you programming? Ah - you were working on Mozilla. Still, I'm sure that is completely stable and crashpr
NO CARRIER
Anything I post is strictly my own thoughts and doesn't necessarily have anything to do with the opinions of IBM.
Scott gives some perceptive and interesting answers, but his discussion of Test First is incorrect due to a pair of misunderstandings about how it works.
I'm going to use the term "Test Driven Development", or TDD, since that's the current usage.
First, TDD doesn't start by coding a lot of tests. That would indeed be discouraging and counterproductive. It starts by coding _one_ test, and then immediately implementing just enough code to satisfy that one test. The point here is immediate feedback.
Second, TDD isn't about producing tests; it's about designing software. When you write a TDD test, you're actually producing a component of the product's detailed design. Most programmers enjoy design; the reason most give to dislike it is that design usually lacks immediate feedback. With TDD, any reason to dislike design is gone, since the feedback is immediate.
So when Scott says that TDD doesn't take into account the psychology of programmers, he's simply incorrect -- it does.
Of course, the original questioner made the same mistake of identifying TDD with testing. To him: TDD isn't about testing. TDD produces a lot of useful tests, but analysing the effectiveness of the tests isn't something that TDD looks at. A good programmer or team will have to put some effort into a test plan which incorporates the TDD tests together with whatever else is needed to verify and validate the requirements.
-Billy
The first place I worked for, had a great QA department (which I started in). They wrote and maintained thousands and thousands of unit tests, and when things failed they would also discuss with the developers if the test or the code was broken (because even tests can be buggy). They even went so far as to have code compiled on checkin and run nightly batches of all unit tests on all platforms the code supported - code checked in that broke compilation was rejected, and nightly unit test results went out each day with the people that had checked in code the day before looked at most closley to sove the issues. It all sounds very draconian but really it worked pretty well for the developers (and I should know, as I became a developer for a while after the QA stint) and didn't feel at all intrusive to work with - since if you coded well, things all went smoothly and the process never touched you.
That was a small company ( 100 employees) who really, really cared about the quality of thier code - and it showed, they had a very stable and well-written product. It was the only time I've seen real SCM and QA in action and it was great.
Slowly over time I moved into larger and larger companies - never again did I see that level of attention to the SCM and QA process. Instead I see a lot of beurocracy that calls itself SCM, but is really just wastes a lot more of my and my team's time than maintaining my own CVS tree. There is more emphasis on knowing who to blame when something goes wrong, than to help produce working code leveraging as many people as possible to fix something that might be wrong.
So you are right, the QA writing tests is the idyllic case. But alomost no-one does it, at least at the moment. The funny thing is that using code workshops in India to do QA tests while keeping development in the US would probably be the optimal model, since it would actually take advantage of the time difference and not be too affected by seperation, but instead companies go too far (as usual) and just ship everything overseas (well, some companies anyway).
"There is more worth loving than we have strength to love." - Brian Jay Stanley
but what do I do when mommy and daddy kick me out ?!?
At the several commercial software houses I've worked, we won't touch/link to open source code since it inevetiabally includes much legal hand wringing about GNU lawsuits over 'why we don't open our code'
Public domain is the only way to go for true Open source code.
Both sorts of testing are helpful.
Functional tests make sure that the program meets the requirements. Good QA people are invaluable.
But even if I had the world's best team of testers, I'd still do test-first programming. Why?
As far as I can tell, every programmer does manual unit testing, generally by putting in print statements as they develop and then looking for stuff in the output. Doing automated unit testing just means that you take those manual checks and automate them. And isn't automating tedious manual work what computers are all about?
I thought using the "new" operator in C++ on built-in data types gives you zero'ed memory, like calloc() instead of malloc() in C.
I have to toot on my own trumpet.
Check out kbugs.org. These are the smatch results from testing the 2.5.65 kernel. We found 1400 possible bugs in the 2.5.65 kernel but probably over half of those are false positives.
Smatch is an open source checker that finds similar sorts of problems to the Reasoning software. For example, both look for dereference bugs.
The bad news is that smatch is still in the pre-alpha stages and it only works on C for now. And also the kbugs.org site is crappy.
The cool thing about smatch is that you can write checks which are custom to your code. Mostly it is used for the kernel, but Michael Stefaniuc has used it for Wine specific bugs as well.
Most developers have realized this problem (where the great idea dissolves before you code it), and find a good mix of written and verbal communication.
There's a lot to be gained in getting a few people together in a room with a whiteboard to brainstorm -- you can review (and discard) a much greater volume of ideas, because the feedback is almost instantaneous... you can stop explaining as soon as the other people understand the idea, you get non-verbal hints that someone else sees a problem with the first part of your idea, etc..
At the end, you should have a written list of the ideas that seemed to have the most merit, and someone works on the details then emails the rest to review results later that day.
Obviously email can be used to achieve the same results, but it can be a lot more work, partly just because you have to guess how much explanation is necessary to get your idea across (and if you guessed wrong, you either need *more* emails to clear up misunderstandings, or you just wasted a lot of time).
There are only 10 types of people: those who understand decimal, those who don't, and, uh, 8 other types I forget.
My rhetorical question wasn't supposed to actually be asked!
I missed this the first time round and probably should have posed this as a question, but its a bit late now.
Basically, what does Reason offer over running gcc -Wall -pedantic -anything else, running lint on the code, and then running the code with something like valgrind to detect dynamic memory problems? Okay, its a single tool as opposed to many of these and probably produces nicer reports, but what I think I'm really asking is, for the poor student like me, is there anything similar/have I missed out any really useful tools from the above (lint/valgrind/gcc with all warning options)?
Yes, the GPL is an aggressive license, and yes, you get into trouble if you link to GPLed libraries and then release the result under a standard closed source license.
That's just the way the GPL is, (by design) everyone knows it by now (or should). Get over it. Saying "OSS use may == legal trouble" is using such a broad brush that I'm sure that someone could easily accuse you of spreading FUD.
There's no reason not to use GPLed source in building your closed source app. Want to use gcc? Fine. Want to use gnu make and bash to handle your automated builds? Fine. Want to use the ncftp client to automatically move your nightly builds from one place to another? Fine.
Just don't link in GPLed code into your proprietary executeable. (And, ok, I could see some worry about not wanting to link to LGPL'ed modules either if you're the kind of place that wants to slap sixteen kinds of "do not reverse engineer under penalty of whatever we can think of" clauses into your license agreement)
But use? Fine.
Also, there's a big range of licenses before you hit public domain that generally cause no problem being integrated into proprietary development. There's the X11 license, the very similar new BSD license, the old BSD license with advertising clause, the Apache license, the Perl Artistic license, etc.
Implying that open source libraries are somehow unique in the potential for licensing issues in code that uses them just tells me that you have not dealt with enough proprietary libraries or that you fail to realize that the payment of royalties is a licensing issue.
I don't think I agree with his analysis of writing tests first; or rather, I believe he misses the larger point. Yes, programmers will tend to write tests that mirror the programs they are testing, but I find they also tend to force developers to at least think about testing, what's correct, and how things fail. It's not perfect and if everything is well-designed it may not matter, but in real applications, writing tests first can cut out at least a few code/run/fix cycles.