A Framework For Quality Assurance?

Midh Mulpuri asks: "These days licenses like GPL, MPL and QPL make it easy for developers to release software as open source without the hassle of having to write up "TERMS and CONDITIONS" for each project. While the software is released as open source, it is mostly released with no warranty whatsoever. That is reasonable considering the software is often 'free'. Major projects often find service providers offering fee-based support. But for most part, the burden of providing service falls on the community and the developers. Further, there is no quality assurance other than the reputation of the project (and the developers). While for many this is enough assurance, the lack of warranty and quality assurance might still leave some people uncomfortable. To make 'free' software more appealing we would need a better way to assure quality. The greatest quality assurance for software is the developer. A framework should be developed to make it easier for the developer to assure users of quality." Does Open Source really need a framework for Quality Assurance?

"This framework might include:

1. A "General Practice Agreement" or "Ethics Agreement" that states ground rules for open source projects and programmers in general. (I am using the word Agreement lacking a better word). This would be like the contracts that bind lawyers, doctors, architects and other professionals. Developers could sign the "Agreement" by placing a graphic on their site.
2. Self-test guidelines. The guidelines would include a checklist that needs to be completed before any production release.
3. Documenting results of the self-test on the project site and/or distributing the test suite with the software.

Most projects already have some form of testing. Documenting and publishing these tests would add to the credibility of the project. An ethics standard might be a more difficult issue. How would you come up with an ethics standard that is acceptable to all? A statement like "I shall document my code well" or " I shall make my software extensible and customizable" would be easy to adhere to. "I shall base my work on industry standards (when possible)" might not appeal to all programmers.

Such a framework would need the backing of prominent open source programmers to be developed and implemented. I believe that the assurance of quality would bring more respect to open source and open source programmers. If such a framework already exists (and escaped my well traveled mouse) please let me know. Otherwise, I request the Slashdot community to kick start the development of such a framework. What should a "General Practice Agreement" contain? What would be good self-test guidelines?"

Comments

Re:QA and support based biz

[Howard Roark]

I think I can shed some light on this. In my company we track technical support calls and attempt to characterize them. Over time, as our efforts to improve product quality have paid off, we notice that the character of our technical support calls have changed. Instead of "Help! I can't your product to work!" calls we get more and more "Help! How can use your product to solve my particular problem" calls.

Just as Red Hat makes it easier to install their product, their support offerings will change too. You actually don't make as much money on support helping a single user install a buggy version as you do helping a large customer with a large implementation solution.
--
Howard Roark, Architect

We need QA for some things...

[Howard Roark]

While it is well known that the Free/Open method of software development produces high quality software, some users still need formal QA. I actually run a QA department in a software company and see a real need for this. For example, pharmaceutical companies in the US are required by the FDA to validate software used for production purposes. I think this is very resonable.

Open software enjoys the benefit of massive code review which provides dramatic improvements in overall quality. That is, if it got a massive review. With most projects, just don't know and that is the QA problem. Quality systems require that projects be developed according to documented procedures and that "quality records" be produced that show that the documented procedures were followed. You often hear the phrase "Say what you do, do what you say."

I happen to be working with the final beta of KDE 2 (which is very impressive, by the way) as we speak and I know that a lot of people are looking at it and are finding bugs and reporting them to the developers. The problem is I can't say how many people are testing it and how. I don't know if a particular feature or environment is being ignored by the testers. I don't know if all the documentation has been reviewed, etc.

I think it would be useful for the community to have a set of agreed upon standards for development. We already have some, like the GNU project's coding standards, but we need more and they have to include a method of producing a trail of quality records that backs up any claim that a project followed its standards.
--
Howard Roark, Architect

Re:Why OSS doesn't need QA...

[Junks Jerzey]

In commercial software houses, changes are made all the time by people who don't understand the code. Often times, the person who wrote the code left several years ago, and noone is really even sure it works completely with the latest library versions. However, with free software, projects tend to be much smaller, and so figuring out what is going on is fairly trivial

Sorry, that's wishful thinking. In a commercial development house, there are people who work with the codebase day in and day out. Even if the original author has left (very common), there's always someone who can scribble on a white board for 10 minutes to set you straight. Often what looks like an obvious fix turns out to be incorrect, making you glad you had that 10 minutes of scribbling. Maybe the biggest problem with fixes to OSS is that you don't know the reputation of the person making the fix. Maybe it's from a careless programmer. Maybe it's from someone who can program but doesn't understand the importance of thorough testing. Who knows?

Re:Sorry, but "Quality Assurance" is a crock of sh

[Reality Master 101]

And these two numbers, hours tested and how many doing the testing are orders of magnitude greater for Linux than for any Microsoft OS. So while Microsoft spends orders of magnitude more dollars on "Quality Assurance" programs, Linux comes out more stable. You tell me. Which is the better indicator of product reliability?

You don't know what you're talking about. For some bizarre reason, Linux advocates simultaneously claim that Microsoft is the worst run software company in the world, and then go on to use Microsoft to prove the "superior" OSS development model.

There are other software companies, you know. Ones that make Linux quality look like the crap that it is. Yes, I said crap. Only in comparison to, say, Win/95 is Linux stable and reliable. It is riddled with security holes and has many bugs. You personally just don't see them because you don't stress your box in any sort of way.

Linux is not even close to the best version of Unix. Take a look at AIX sometime, or even Solaris. Got news for you: commercial Unix blows Linux out of the water. Or heck, look at the AS/400 if you want to see reliability.

The bottom line is that the OSS development model is far from proven, and in fact, has very few examples of producing top quality software. Apache is one. And I can't think of another. Almost all the best software is closed source and commercial. And one of the reasons is having a QA and testing staff.

--

Huh? Don't think so

[Vanders]

The best person to assure quality is the developer? No chance! I assume you've never worked in a commercial development enviroment?

As a QA tester, i can assure you that a developer is not the best person to assure quality. They just don't see their own bugs, or code that isn't to spec. (This may not always apply to OSS projects). The only person who can assure quality is a third party (Non developer), who does a full QA test on it. Trust me on this one.