Slashdot Mirror
A Framework For Quality Assurance?
"This framework might include:
- A "General Practice Agreement" or "Ethics Agreement" that states ground rules for open source projects and programmers in general. (I am using the word Agreement lacking a better word). This would be like the contracts that bind lawyers, doctors, architects and other professionals. Developers could sign the "Agreement" by placing a graphic on their site.
- Self-test guidelines. The guidelines would include a checklist that needs to be completed before any production release.
- Documenting results of the self-test on the project site and/or distributing the test suite with the software.
Most projects already have some form of testing. Documenting and publishing these tests would add to the credibility of the project. An ethics standard might be a more difficult issue. How would you come up with an ethics standard that is acceptable to all? A statement like "I shall document my code well" or " I shall make my software extensible and customizable" would be easy to adhere to. "I shall base my work on industry standards (when possible)" might not appeal to all programmers.
Such a framework would need the backing of prominent open source programmers to be developed and implemented. I believe that the assurance of quality would bring more respect to open source and open source programmers. If such a framework already exists (and escaped my well traveled mouse) please let me know. Otherwise, I request the Slashdot community to kick start the development of such a framework. What should a "General Practice Agreement" contain? What would be good self-test guidelines?"
For anyone wanting to begin using a quality methodology in their programming, I make the following suggestions:
1) Read [Steve McConnell's] three books. It's a good start. What he writes is based on solid research, which he shares with you. His books aren't the complete answer to all problems, but reading them and using a bunch of the tools/methodologies he describes is a great way to begin doing things better.
2) Do a Google search on "Software Capability Maturity Model" and start researching. Eventually you'll come across the Software Engineering Institute, and their [summary of CMM], which is well worth reading.
3) Do a Google search on "Bell Canada Trillium" and start researching. The [Trillium model] is well-respected, and is based on ISO, CMM and other best practices. Where it differs is that it actually tells you what to do; the others tell you what you need.
4) Do as much as you can with the structures that are described, plan on how to do them all, and adapt them to your needs. Identify what works and what doesn't, and fix those things that don't.
--
--
Don't like it? Respond with words, not karma.
Eh, you haven't had problems with Sendmail?
And here I thought Sendmail was on top of the list of buggy programs which lead to security problems. Guess I was wrong.
Je ne parle pas francais.
Yes, some software does need to be subjected to a formal QA process. Do you really think any OSS tool is going to be incorporated into Space Shuttle's systems without this?
Of course, the latest version of nethack probably doesn't need a *formal* QA check - the players ensure swift resolution of any problems.
Many commercial organizations use RedHat because of the support. They assume that some QA has been done to minimize the need for that support. This work is what helps build the RedHat brand.
John Doe buys Colgate toothpaste because he trusts the brand - and what it implicitly stands for. If OSS is to take over the world, it needs to build those brands.
"This is the kind of bullshit thinking that drives companies out of America and to the waiting hands of Countries that welcome them for being just a company,"
and then you say: "QA is very crucial for what I do, so I believe I am right in my thoughts."
It seems to me your thoughts could do with a bit of clarification.
No, this is the kind of thinking that turns American corporations in global behemoths capable of crushing all competition. If open source wants to compete, ultimately it'll need to use some of the same techniques. Don't just dismiss QA because nothing you've ever done was important enough to need it...
I mean look at how big Microsoft is when they release software that is notoriously buggy. People seem to be quite content to have unguaranteed occasionally buggy software. OSS doesn't need a guarantee, it's guarantee is that if you have the skill and knowledge you can fix your own damn bugs yourself. It's guarantee is that if you e-mail the makers of the software and ask them nicely, they'll probably fix your problem (if its significant). Ultimately it isn't really a "guarantee" but those OSS products that grow and thrive generally have this feature.
As a developer I have worked with many products that were deffective. That is to say, they made claims about what they were capable of doing that were blatantly wrong. With one product I even ran into a situation where I told them that it didn't work with another product that they said it did. Rather than fixing it, they simply modified their website to no longer claim that it worked that way. Generally these products are black box commercial software, and if they break I can't do a damn thing to fix them. If I'm lucky the fix pack for the software will come out a couple months down the line and fix it, but I may have to wait until the next version.
On the other hand, with products like Tomcat, Apache, Linux, etc, I can dig in and look at what's wrong. I don't have to wait for a bloody fix pack or new version (if I know what i'm doing). And I have to say that despite having no guarantees of anything, I've found that the most responsive and helpful technical support I've ever gotten has been through an OSS mailing list and I didn't have to pay a dime for the software or the support. After working with all OSS stuff in college (like I could afford commercial software), I found it tedious and frequently ineffective to try to get phone support from a commercial vendor.
---
This sig has been temporarily disconnected or is no longer in service
Also, bear in mind that most commercial software comes with no warranty either. Just check the "shrink wrap" or "click wrap" agreeemnts. "THIS PROGRAM COMES WITH NO WARRANTY, EXPRESS OR IMPLIED."
sure, there may be formal QA in commercial softwre, but what does it really mean if the user can't get his money back?
My journal has hot
As another QA engineer, I second the motion. All too often QA gets relegated to the bottom of the food chain, but they belong on the top right next to the developer.
Some of the major quality problems I see with a most non-commercial OSS are:
1) Lack of specifications and requirements. A project that does not have a detailed plan right from the beginning is courting crud. (and a major reason why a lot of commercial code is crap as well). I offered my unpaid services to a project a while ago saying "I will write a comprehensive test plan for you if you give me access to your specs and reqs". Unfortunately, the only documentation they had was the source code tree and a README that said "sombody write this...".
2) Not adhering to the specs and reqs. This is almost as bad. All you users out there, get and read the GNOME and KDE user interface guidelines. If an individual GNOME or KDE program doesn't follow their self-stated rules, log a bug immediately.
3) Passing the blame on to someone else. It's all to easy to blame another one of the myriad projects in a Linux or BSD system as being the culprit, but that's a copout.
4) Passing off beta (or even alpha) software as "stable". Do you think just because Microsoft can get away with it that you can too? Don't call your project complete until every last one of your serious bugs are fixed, and issue a list of all ramaining minor bugs and annoyances with work-arounds. And this brings up another point. Get someone independant of the developer to rate bug severity.
A Government Is a Body of People, Usually Notably Ungoverned
Open software enjoys the benefit of massive code review which provides dramatic improvements in overall quality.>/em>
So what? Code review is not quality. It's only the first tiny step on the long road to quality.
The problem is I can't say how many people are testing it and how.
Absolutely! QA is one of those non-glamorous jobs that OSS often ignores. But its a needed role.
A Government Is a Body of People, Usually Notably Ungoverned
As a QA engineer, the software needs my approval to ship. Of course the CEO can ship it anyway, but it takes his explicit veto to do so, and he would find an empty engineering department if he ever did so. I suspect that there are quite a few companies like this, especially those that have been around for more than a few years.
But some companies do put deadlines ahead of QA. If you work for such a company GET OUT NOW!
One of my questions I like to ask QA candidates is if they would sign off on the software if their manager told them to. So here I am, fifteen minutes into the interview with another 45 minutes left, and the candidate says "yes" to the question. I reply, "I'm sorry, we don't hire rubber stamps for $25 an hour when they're only two bucks at Office Depot."
A Government Is a Body of People, Usually Notably Ungoverned
Sounds like Mozilla :)
We have a growing community of QA and testing volunteers who have been working and learning QA process on one of the largest open source projects out there.
mozilla.org provides daily Mozilla builds for 3 to 4 platforms here. We provide an open (and kickass) bug reporting and tracking tool Bugzilla Our QA and testing docs are getting better all the time (Mozilla QA) with published daily smoketests as well as detailed functional test suites for all areas of the product.
If you're intererested in getting involved with a one of a kind open source QA and testing project please take a look at our Getting Involved pages or stop in to #mozillazine or #qa on irc.mozilla.org We have a weekly help session (BugDay) every Tuesday for new folks interested in getting involved. So if all this talk about open source quality has sparked your interests let us know.
-Asa
Mozilla QA and Stuff
I trust you, and I agree. I also make my living as a developer.
FINAL test (at any level) really does need to be done by a third party.
The QA person should be in on code walk throughs - but at that level, he is more of a "throw questions at the developer", and the developers will be able to tell the QA prson if it will work.
AFTER the developer has run his OWN unit test, you feed the unit to QA, and trust me, unless you're a VERY strange developer, a GOOD QA person will find things that you didn't
Once you are into final test, QA becomes invaluable. Developers just don't have the time to dedicate to things like full regresion testing, and that alone can save your butt.
OK, let's just say your developing Windows Software (Boo, Hiss, I know - it's just easier to give a screwed up example)
Have you:
Installed on Win95 (Base, NO SPs - unless you don't support that)
Win95B?
Win98?
Win98se?
NT4.0?
NT2k?
(and then run a FULL test of everything?)
The have you installed Microsoft Office over you app, and run all the test again? (all OSes please)
The have you done it in opposite order (Base OS + Office) then install your app
If you want a really stable system, you'd better try all of these (Plus different versions of IE etc)
Of course, we have the same problems in Linux - Have you tried it on as many differnt flavors as you support? With other software installed (the worse behaved the better)?
This is where a GOOD QA department makes their living. Gad, I wish me present company had one
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
Open Source really doesn't need a set QA system, it already has one. It's called peer review. Say Hacker X creates a program with obvious bugs. He/she releases it, and Hacker Y downloads it. Because Hacker Y has access to the source code, he/she can find the bug and fix it.
While peer review cuts back on bugs, it's no substitute for QA. What you fail to mention is that a hundred other people download the same version as Hacker Y, and it's just as buggy as ever, in fact more so than a project that had QA. Also, there is less pressure on the developers to test properly, since "we can always just release a new version" is in the back of their minds.
I think the simpler we make the whole process, the better.
I agree. This doesn't mean ignoring even the most rudimentary QA, instead putting out a release that probably has a lot of bugs, and hoping that other people will find them.
Open Source really doesn't need a set QA system, it already has one. It's called peer review. Say Hacker X creates a program with obvious bugs. He/she releases it, and Hacker Y downloads it. Because Hacker Y has access to the source code, he/she can find the bug and fix it.
This is a myth. Problems:
1. Most people, even hackers, don't have the incentive or time to track down a bug in huge software products.
2. Fixes made by people who aren't familiar with a code base are unreliable. Changing code is scary if you don't understand the .
3. Most people who touch OSS code don't do regression testing. Heck, most OSS doesn't have test suites available.
4. Many "bugs" end up being differences of opinion or simply a user not understanding things properly. Jumping in and fixing those is bad.
I have a qualified disagreement with this statement also.
While I agree that THE developer is not the best person to check their own code, ANOTHER developer (esp. if the other developer hasn't been working too closely w/the 1st one, but has a firm understanding of the internals of the existing code) is an excellent way of improving one's code.
It's a question of the granularity of testing - a QA test guy can only test the overall product like a black box, exploring errors that can only be activated directly through the external interface, and if they discover a problem, they can only describe the symptoms.
A _developer_, on the other hand, can test & review the code from the inside out, resulting in robust components & code maintainability which an external tester only has an indirect interest (and a very limited set of ways that they can reach some of the internal code). The developer can also point directly to the source of problems, saving incredible amounts of redundant debugging effort (e.g., to reconstruct that really annoying bug which only shows up under a certain race condition...)
A good QA test suite is still essential to make sure that final overall product looks & behaves like it was supposed to, but you don't make robust bottom up, incremental improvements by blackbox testing.
Unfortunately, most of the companies I have seen/worked at aren't willing to expend the resources necessary to have another developer THOROUGHLY review & test each other's code. (As far as I can tell, the managers seem to think that this kind of activity is "not productive", and/or it will cost too much to hire enough developers to do such testing in a reasonable amount of time.) Usually, I end up in a few code reviews where a bunch of people scan the code quickly, looking for typos or uninitialized pointers or other such simple problems.
I didn't intend to say that the QA teams are composed of grunts, but that unless they are MUCH more talented developers than the actual developers (which is definitely not the case if they are usually composed of "entry-level" programms), they aren't going to be able to come up with ways of testing the internals of the code & pinpointing errors as well as people who were on the original development team.
For the most part, the QA team will test the interfaces to the various components of the code (not just the UI interface), as well as correct any superficial problems.
For finding & correcting deep-down, fundamental design-type problems, you need experienced developers with a gut-level instinctive understanding of the code that they're working with (unless you're talking about a trivial amount of code).
Furthermore, the best place to find & correct bugs is early in the development process - BEFORE a lot of structure has been built around that code. By the time a serious bug is discovered by a QA team, the developers are going to have to do a lot of work to patch the problem - and in the face of a deadline, this usually results in sloppy work. Whereas, if the bug is discovered earlier in the development process, it takes much less effort to the correct the problem.
You can still say that the QA team is responsible for finding THOSE kinds of bugs earlier & earlier in the development cycle, but then the QA team pretty much gets merged into the development team - which was my point to begin with, that the QA process should really be an intimate part of the development process, from bottom-up and top-down.
A much simpler solution is to just publish the test plan for the package. Then the user of the software can decide if it has been adequately tested. Just because the author (or a third party) claims that it has been tested to a certain level doesn't mean anything, especially if the certification is accompanied by a disclaimer of liability.
I work for a software consulting company and one of the deliverables on every project is the test plan. But every customer is not willing to pay for exaustive testing (and in some cases really don't have any need for that level of QA.) On some projects, the test plan might be "we ran through the changes and it appears to work". In other cases, it would be thousands of test cases, with both automated regression tests and manually executed cases. For a program that prints labels for a CD, you don't really need extensive QA. For a compiler, you do.
It certainly wouldn't HURT for some person or group to publish different levels of certification, along with suggested test plans, tools for managing test cases and plans, and automated test tools (perhaps based on DejaGNU, etc.) Sounds like an excellent Open Source project in fact. Providing tools to actually help authors and development teams would be much more useful (and much more work) than creating a committee to write up some bogus standards that aren't really appropriate for any specific project.
If the author of this question is really serious about the quality of OSS, then I'd suggest that they just go test something. If you're concerned about the quality standards for a specific piece of software, contact the author and volunteer to set up an automated regression test system and generate sufficient test cases to prove it works. I'm sure that the developer would welcome the assistance. Open Source works much better when led by example, rather than by creating some arbitrary standards.
This kind of thing (QA or testing, whatever you want to call it) goes into the category "Things that you could do to aid an open source project even though you aren't necessarily a programmer"
Anybody with experience in larger projects will be able to verify that when projects of any form or shape get beyond a certain number of people involved, there is overhead involved in maintaining the project.
One could think of administrative duties (such as maintaining a web site and communications beyond a message board and a mailing list) but also the things done in commercial software development that are not actually coding.
In other words: why not have QA people (good ones, not bureaucratic, satanic, ignorant organisational outcasts) as a part of an OSS project. That would free the developers from (often tedious) testing work and it would speed up the elimination of bugs.
A possible scenario is that QA people that (like programmers) want to contribute their time would do nothing other than have a box with a known (and published) configuration (VMWare!) that they would install new revisions of a product on and run extensive test suites on it, feeding back the bugs into the project.
btw: this is quite common for Big OSS Projects, but the smaller ones could benefit from it just as much.
Food for thought, I hope.
Schmolle
(QA Manager -- FT.com)
I agree with this... I feel that a developer can, with discipline, build a comprehensive test plan that scours every requirement, etc. but you just can't beat a 3rd party for this. It's far too easy to gloss over areas you "think" are working, or to not find a bug because you just didn't think "anyone would ever DO that".
Further, 3rd partys (or any non-developer) can really help point out interface issues. I know what a boolean operator is, and how to build a query string using them, but I wouldn't dream of asking a search engine user to enter a query in disjunctive normal form, etc, yet if I recently found myself using a bunch of "tree" terminology when trying to write prompts and messages for an application I was working on that organizes user contributions in a tree structure. It took a non-CS person to try out our interface and laughingly ask what it meant for a form field to be "non-empty", etc... And so on...
--8<--
--8<--
For IBM, this is no problem; IBM can say: "we'll make it work or give you back your money." That's credible. For Joe's Garage Software, this statement may not have the same level of credibility. If Joe has some past history in business, he may well be able to get a performance bond to stand behind his work, and ensure that funds will be there to fix problems.
It seems to me that this problem really bites the folks who have a really new package, and no past history. I'd guess that for sendmail or apache, there aren't any difficulties here; lots of places will provide support for them. You don't even need the developers anymore for that. Third-party support has been the answer to this question so far in free software. But linuxcare isn't going to offer support for some thing they've never even heard of.
Maybe the beginning of an answer lies in that last paragraph: software with a good reputation WILL be supported by third parties, and you (or your client) will be able to hire some outfit like linuxcare to ensure that your application runs for a client, IF you have earned a good reputation. So, to provide quality assurance in free software:
First, build it and make it good.
Second, keep developing it, make it better, and get users to adopt it.
By now, you should have a good reputation and lots of firms standing ready to stand behind your work.
I'm not sure how we fit a quick IPO and early retirement in here, but maybe you shouldn't expect that out of the free software model.
Nels
See what I've been reading.
I would agree with some form of Quality Assurance. The easiest thing would of course be clear code documentation. I know that many a programmer wouldn't be too fond of it, but it would help for implementation. Companies are for more interested in software that has some kind of built-in reliablity level. We're looking to get some software coded for us, and documentation of the code is really vital to us, so that we can make changes or easily fix any bugs found.
The power of accurate observation is commonly called cynicism by those who have not got it. - G.B. Shaw
A developer can be the best person to test his code. But only if he truly respects the value of testing. This is a respect that must be cultivated in most people. Unfortunately, most projects (volunteer or commercial) fail to do this. Commercial projects generally forsake the effort and dedicate a different team to testing. Most (not all) volunteer projects simply forsake it (but ESR's bazaar priciples can make up for this lapse).
I don't disagree that even a consciencious programmer can be blind to his own mistakes. However, if he cares about quality, he will also be the most acutely aware of the weaknesses and possible problem areas in his code. This should let him target his testing to maximum effect. A dedicated tester on the other hand will basically shoot at random parts of the code.
Don't get me wrong--I'm glad my company has testers. They are more cost-effective at catching many bugs, especially those that only show up when all the pieces are put together. But they should only start testing after developers do their own testing and debugging. Otherwise, their time is wasted reproducing, confirming, isolating, and reproducing simple bugs that the developer could diagnose and fix in minutes.
If there can only be one tester, it should be the developer, if he is consciencious.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
Mabye in very mature development organizations. But lets face it: the space of inputs is enormous and testing resources are limited. Since tests are basically discrete (to use a rough analogy, a single test covers a measure zero subset of input space), coverage in any rigorous sense is a mirage. Inevitably, entire categories of possible inputs will be missed.
Saying that testers fire "at random" was obviously an exaggeration. They certainly have some notion of what's important to test. But they don't have the sniper's accuracy of a careful developer.
Also, the QA department should be involved right from the very beginning, even before a single line of code is written.
Bah, this has way too much overhead for most projects. Give me a realistic example of an early design mistake that would likely be found by a tester.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
I know I'm by no means to bring this up. I've seen this addressed by far wiser folks than myself on both sides of the issue, but for this topic I believe it is appropriate.
At what point in time does ease of use and quality start to eat into the profits of a company that bases it's entire revenue stream on support? Just so I have someone to pick on in this discussion, let's pull a RedHat out of a hat. They're a fair example being that they essentially don't sell software, but a service supportting a software package.
Mind you, I don't mean to suggest that RedHat makes any attempts to make Linux difficult to use, and there's certainly ample evidence to suggest that they're invested in projects to ease usability. Where the concern comes from is in the long term, when ease of use starts to collide with the only product they truly sell, support.
In the traditional software market, support is not a profit center. Instead, it's considered to be a liability which charges folks for this service as a way to cover costs. By it's very nature this model has to strive to make ease of use a top priority to be profitable.
If software configuring becomes too simple, this is going to have an impact on support oriented companies. Perhaps the very notion isn't even possible. Still, one method must strive (not saying it's ever gotten there) for support free solutions, while the other must involve enough difficulty as to require a support agreement to make money.
The line must be drawn here. This far. No further.
There is no quality assurance with any free software. That's why fee-based software is so much more popular (like Windows).
- I don't care if they globalize against free speech. All my best free thoughts are done in my head.
You beat me to it. There is NO WAY the developer is the best person to assure quality. In fact, that person is very often the absolute worst.
Nice of you to put in that comment about OSS projects, but I have to disagree. I've found most OSS projects to be far inferior in quality to commercial projects (and to the people who can't read, that does not mean that all OSS has inferior quality to commercial software).
--
Sometimes it's best to just let stupid people be stupid.
It occured to me that I can't see anything stopping anybody from selling GPL software with a warranty, a warranty provided by the company that sells the software, not the developers. The no-warranty is there to make sure the developers will not get sued for failings, but businesses selling free software should be able to provide a warranty, in connection with e.g. a support program.
I don't know if this allready exists, but I think it would benefit the OSS community, as such a company is likely to do extensive and formalized tests of OSS software, and come back with patches if they find bugs.
Also, it may impress the suites that a software company offers warranty for a product others develop, out of their strict control.
Employee of Inrupt, Project Release Manager and Community Manager for Solid
You want something that you can develop and change for yourself, yet you want a warranty or guaranty that it will work just like you expect it to... COME ON! At very least, where's your pioneering spirit? Do you also want to climb Mount Everest but expect someone to guaranty that you will not fall or hurt yourself in any way?
Stop whining and release your creative spirit! If it doesn't work like you want it to, then add the code that enables it do sing or dance for you!
--
Vote Homer Simpson for President!
This reminds me of George Carlin talking about prostitution: "Selling is legal, Fscking is legal. Why isn't selling fscking legal?!"
"Ancillary does not mean you get to rule the world." --U.S. Circuit Judge Harry Edwards, speaking to the FCC's lawyer
Damn right on that one. Tim sweeney coded the Unreal Engine, complete with mouselag, sound system lag, and network code that outright sucked (even he admits to this). And he STILL won't address the lag problems in Unreal Tournament; he seems too arrogant to even admit defeat.
"Ancillary does not mean you get to rule the world." --U.S. Circuit Judge Harry Edwards, speaking to the FCC's lawyer
Remember the printf() bug scattered about ALL the libraries in Linux, Unix, and BSD! That was Linux's back orifice, and it could've been solved by the engineers actually checking their syntax! Gee! Ever notice how the bible-thumping software engineers never seem to check their syntax?
"Ancillary does not mean you get to rule the world." --U.S. Circuit Judge Harry Edwards, speaking to the FCC's lawyer
Releasing the source code to your software means taking the risk of having someone hack it for malicious purposes. If it wasn't for Pure Server, Q3 would have cheaters all over the world due to one unethical programmer fiddling around with the source code (wait, that's already happened!). Q2 is a good example too; remember the Ratbot? Counter-Strike has the aimbot. Let's face it, there's plenty of immature programmers out there who don't want to play fair (in more ways than one).
"Ancillary does not mean you get to rule the world." --U.S. Circuit Judge Harry Edwards, speaking to the FCC's lawyer
While it is well known that the Free/Open method of software development produces high quality software, some users still need formal QA. I actually run a QA department in a software company and see a real need for this. For example, pharmaceutical companies in the US are required by the FDA to validate software used for production purposes. I think this is very resonable.
Open software enjoys the benefit of massive code review which provides dramatic improvements in overall quality. That is, if it got a massive review. With most projects, just don't know and that is the QA problem. Quality systems require that projects be developed according to documented procedures and that "quality records" be produced that show that the documented procedures were followed. You often hear the phrase "Say what you do, do what you say."
I happen to be working with the final beta of KDE 2 (which is very impressive, by the way) as we speak and I know that a lot of people are looking at it and are finding bugs and reporting them to the developers. The problem is I can't say how many people are testing it and how. I don't know if a particular feature or environment is being ignored by the testers. I don't know if all the documentation has been reviewed, etc.
I think it would be useful for the community to have a set of agreed upon standards for development. We already have some, like the GNU project's coding standards, but we need more and they have to include a method of producing a trail of quality records that backs up any claim that a project followed its standards.
--
Howard Roark, Architect
Howard Roark, Architect
I believe in a Man's right to exist for his own sake.
In commercial software houses, changes are made all the time by people who don't understand the code. Often times, the person who wrote the code left several years ago, and noone is really even sure it works completely with the latest library versions. However, with free software, projects tend to be much smaller, and so figuring out what is going on is fairly trivial
Sorry, that's wishful thinking. In a commercial development house, there are people who work with the codebase day in and day out. Even if the original author has left (very common), there's always someone who can scribble on a white board for 10 minutes to set you straight. Often what looks like an obvious fix turns out to be incorrect, making you glad you had that 10 minutes of scribbling. Maybe the biggest problem with fixes to OSS is that you don't know the reputation of the person making the fix. Maybe it's from a careless programmer. Maybe it's from someone who can program but doesn't understand the importance of thorough testing. Who knows?
And these two numbers, hours tested and how many doing the testing are orders of magnitude greater for Linux than for any Microsoft OS. So while Microsoft spends orders of magnitude more dollars on "Quality Assurance" programs, Linux comes out more stable. You tell me. Which is the better indicator of product reliability?
You don't know what you're talking about. For some bizarre reason, Linux advocates simultaneously claim that Microsoft is the worst run software company in the world, and then go on to use Microsoft to prove the "superior" OSS development model.
There are other software companies, you know. Ones that make Linux quality look like the crap that it is. Yes, I said crap. Only in comparison to, say, Win/95 is Linux stable and reliable. It is riddled with security holes and has many bugs. You personally just don't see them because you don't stress your box in any sort of way.
Linux is not even close to the best version of Unix. Take a look at AIX sometime, or even Solaris. Got news for you: commercial Unix blows Linux out of the water. Or heck, look at the AS/400 if you want to see reliability.
The bottom line is that the OSS development model is far from proven, and in fact, has very few examples of producing top quality software. Apache is one. And I can't think of another. Almost all the best software is closed source and commercial. And one of the reasons is having a QA and testing staff.
--
Sometimes it's best to just let stupid people be stupid.
The best person to assure quality is the developer? No chance! I assume you've never worked in a commercial development enviroment?
As a QA tester, i can assure you that a developer is not the best person to assure quality. They just don't see their own bugs, or code that isn't to spec. (This may not always apply to OSS projects). The only person who can assure quality is a third party (Non developer), who does a full QA test on it. Trust me on this one.
Syllable : It's an Operating System