Slashdot Mirror
QA Under The Open Source Development Model
carrowood writes "A survey was conducted questioning open source developers from both large and small projects concerning their quality assurance practices. A
research paper based on the survey result was just published in the Journal of Systems and Software.
Some comparisions between QA practices of open vs closed source projects are made with some interesting observations. While on the whole it looks like open source QA can be as good as that in traditional software development, there were a few areas pointed out where the open source community does not do so well, such as regression testing and setting release dates.
A thought provoking read."
...then you can go and buy software developed out of a motivation for financial gain.
Surely open source projects exist for other motivations, and hence have less (if any) need to aim towards release dates.
After all, in the business world the sheer fact that you _have_ to get something out by a particular date (because marketing / the bean counters said so) contributes significantly to the quality problems.
Ultimately, while development can, in certain cases, be done in a vacuum, QA cannot (and should not). It's by nature a collaborative and interactive process.
I have nothing but respect for the few (good) QA engineers I've worked with.
Well, the fact that no releases dates are setted are more a good point than a bad one ! Of course, in an ideal world where software would be released on dates (!), they won't have bugs either. But in the real world, must proprietary software aren't on schedules, and anyway, when they are, this is often at the detriment of the number of remaining bugs and/or dismiss of some features.
In the free software world, the software is released when ready. So, of course they don't set release date (generally speaking -- some projects have regular releases). But I hardly see that as an obvious bad point. It could be on the contrary one of the strength of the free software.
At least, programmers on free software releases when they are happy with the code.
I suspect clients requiring ISO 9001 just go for commercial software. One part of OSS often is to refuse warranty and alike - IMO this does not quite fit into ISO 9001.
In my many years of experience, I have to say that regression testing is not as common in proprietary software development as it should be (and frequently not as common as claimed). Furthermore, while I don't want to say that regression testing is less important for Libre Software, I will say that I think it's probably more important for proprietary software, where the programmers are writing for a paycheck, rather than for pride, and are frequently under intense deadline pressure (which in turn, frequently leads to testing/QA of all types being skimped).
:)
As for no release dates, anyone who doesn't recognise that as an advantage of Libre Software simply lacks any clue about the process. Sure, there are downsides, but nothing's perfect. Free, reliable, on-time, pick any two.
When I release code I realy dont care about getting any certifications and accridations linked to it. I write code to be usefull not to be noticed. You use open source at your own risk, and the beauty of it is if you dont like it you can just change it. Why would you need a quality assurance process in place for open source. If its open and its quality people will use it. If it doesnt work then people will just blow it off. Does any one else share in my oppinion?
Faith_Healer -- The antethsis to almost everything, and the worlds worst speller.
Free software is not used by "ALL possible users", it's used by interested users. The size of the user-base varies greatly from project to project. And no, speaking from experience, I can assure you that not all bugs are found w/in 4 months. Subtle timing or edge-case bugs can lurk for years before leaping out to destroy someone's critical data. (And this is true with both proprietary and libre software.)
One thing this study didn't seem to look at though, was the size of the user-base of the projects studied (of course, this is a hard thing to measure, but interesting). I think it would be useful to see what sorts of correlations (if any) there are between QA practices of a project, size of the user-base (popularity), and the overall quality of a project. Can a large user-base help make up for poor QA practices, or is a project with better QA more likely to attract users due to its higher reliability? Do users even care about quality and reliability, or do they just say they do? Interesting questions for which I don't have answers.
Do any open source projects get audited for ISO 9001 compliance?
Who would pay for it? ISO 9001 auditing is VERY expensive.
My journal has hot
The open source development model allows tremendous flexibility, allowing members of a development team to be dropped or added at a moment's notice. With the source readily available, one can become familiar with a project's code before applying to be given access to the CVS or equivalent repository. Gradual accretion may produce code in a style not unlike that of James A. A. Joyce's Ulysses manuscripts, but, like James A. A. Joyce, all of the core developers can easily jump from point to point in the code and comprehend the necessary sections and the C/PHP/Python/Java equivalent of their allusions.
Unfortunately, as a result of this decentralised development system, commercial QA, support and RHQ are not as readily available. I'm a middle manager and my company has had a double-sided experience with the MySQL AB organization. They produce a fine product which is perfect for a medium sized corporation such as the one I work for (which shall remain nameless). MySQL worked very well for us, but unfortunately at one point we started receiving segmentation faults when there were more than 30 connections or if a query was greater than 2,048 characters in length. We have reported these bugs to MySQL AB but they have not yet fixed them in their latest gamma/production release. However, they have been very polite and are always willing to cooperate with us; even if small portions of the code are not yet fixable and have escaped the relatively poor QA of the EOD, their TOS have always been reasonable and our MD has always been able to CWT regarding the slight problems and BS our way around them.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
I'm tired of all this talk about open source development vs. closed source development. That's not the issue here.
It doesn't matter so much whether the source is open or closed as it does who is in charge of the project. Any company could use standard software development methods to produce open source software. Similarly, any company could hire developers all over the world, and have them work together on a project without ever having met.
It's not an open vs. closed thing, and it has little to do with licensing. It just happens that most companies use standard development techniques and don't release as open source.
Other areas of problems are attributable to slovenly or "don't give a damn" attitudes--unused variables, unreachable code, "magic number" constants, and so on. Ignoring values returned by a function are very common. Maybe it is acceptable with a library function, but why return a value if you aren't going to use it? It's better to make the function into a procedure by returning void. On a more theoretical level, the use of weak typing even when the language allows for tighter specification of variables. Strong typing is designed to prevent such oddities and inadvertently multiplying a color by date.
However, in the end it all comes back to complexity. And that is where the biggest improvement in OSS quality can be obtained.
For example, anyone with half a clue in the IT profession is aware that you NEVER NEVER NEVER take on board a x.0 release of any Microsoft product
Sure, there are bugs in any "x.0" release, and not just from Microsoft. Still, every company has its own IT environment, and in many cases even a first release may be stable enough -- YMMV, each one has to decide for themselves.
Also, you have to compare the risk of an "early" deployment with the potential benefits that you'll get with the upgrade, such as better performance, security, new features or whatever.
Mattias
And no, speaking from experience, I can assure you that not all bugs are found w/in 4 months. Subtle timing or edge-case bugs can lurk for years before leaping out to destroy someone's critical data. (And this is true with both proprietary and libre software.)
I'll second that! I have been responsible for, and paid to test a reasonably large administration tool for the last three years. This product was developed and tested for at least 4 years prior to my employment. Even after 4 years of someone else actively testing it, and another 3 years of active testing, I am *STILL* finding new bugs in scenarios that I hadn't previously considered.
I have been noticing a rather disturbing trend:
First, a new feature or large code change is introduced.
Second, I spend at least as much time developing a test plan and doing an initial test pass as the developer spent coding the change, and frequently up to 5 times as long. I probably find 40 percent of the bugs in this pass, and when I am finished, all of the major scenarios that I can initially think about work correctly.
Third, I spend another block of time writing automated tests while the developer fixes some of the lower priority bugs. I might find another 5 percent of the bugs in this stage.
Fourth, I work on some other area of testing for a while, and then I hit this feature again looking for test holes and I find another 5 percent.
Fifth, I repeat this loop of looking for test holes and discuss the features with other testers, do testcase reviews, etc. and find another 20 percent of the bugs over a period of months.
Sixth, sometime later the developer mentions what seems to be an innocuous little bit of information, that turns out to be a critical omission in the spec and I find another 15 percent of the bugs.
At this point, it is several months later and we have only found 85 percent of the bugs.
Ten percent of the remaining bugs will be found by co-workers, beta testers and customers.
The final five percent, might not be found for a number of years even with heavy testing.
Even with years of time, every possible combination of inputs in every possible configuration and every possible usage scenario is not possible to test. For the program that I happen to test, this works out to be in the range of 10^85 unique tests. In order to test every possible input, I'd be testing until after the sun burns out and this program isn't that big.
Fortunately, equivalence classes bring this down very significantly, and I can complete a test pass in about 40 days. The danger is in incorrently assuming that a set of values all belong to an equivalence class.
I would say that the process used is predicatable but the outcome for each project can still be better, worse, or the same as a previous project.
Why? Because using the same process to develop different projects is like using the same algorithm for every mathematical problem, in general it's going to be inappropriate most of the time.
Each project typically has its own goals, deadlines, customers, implementors, managers and economics. To be successful, the process has to reflect those realities.
Trying to come up with a single process that handles all situations optimally requires that you deeply understand all the projects you will ever create in the future.
The practical reality is if you feed the ISO beast enough money and make the right kind of politically correct noises you get entry into the club and get to use your certification as a marketing tool.
Sometimes government agencies require ISO or CSI certification which is a great way for old, large and slow companies to hold off their more nimple competitors by requiring complicated processes that are too expensive for smaller companies to afford and are likely to slow them down.
In my experience, most (not all, of course) Open Source projects aren't concerned with backward compatibility outside of the scope of the project itself. Regression testing in OSS is folded into the bug testing.
That's one of the downsides of OSS. The biggest example: When you upgrade your libc, you have to recompile all of your dependent apps. One thing Windows and Solaris have going for them is that you can run a 5 year old binary on the newest version of the OS and it will almost always work.
It's a big burden on the development team to provide support for old interfaces, but this sort of thing is where OSS has a long way to go. It gets really expensive for individual persons/companies to support (bugfix, etc.) packages that are a few revisions old.
Translation:
It's really just a shiny bullet point for PHBs that the ISO is selling.