Slashdot Mirror
QA Under The Open Source Development Model
carrowood writes "A survey was conducted questioning open source developers from both large and small projects concerning their quality assurance practices. A
research paper based on the survey result was just published in the Journal of Systems and Software.
Some comparisions between QA practices of open vs closed source projects are made with some interesting observations. While on the whole it looks like open source QA can be as good as that in traditional software development, there were a few areas pointed out where the open source community does not do so well, such as regression testing and setting release dates.
A thought provoking read."
Quality assurance under the open source development model
Luyin Zhao, Sebastian Elbaum.
The open source development model has defied traditional software development practices by generating widely accepted
products (e.g., Linux, Apache, Perl) while following unconventional principles such as the distribution of free source code and
massive user participation. Those achievements have initiated and supported many declarations about the potential ofthe open
source model to accelerate the development of reliable software. However, the pronouncements in favor or against this model have
been usually argumentative, lacking of empirical evidence to support either position. Our work uses a survey to overcome those
limitations. The study explores how software quality assurance is performed under the open source model, how it differs from more
traditional software development models, and whether some of those differences could translate into practical advantages given the
right circumstances. The findings indicate that open source has certainly introduced a new dimension in large-scale distributed
software development. However, we also discovered that the potential of open source might not be exploitable under all scenarios.
Furthermore, we found that many of the open source quality assurance activities are still evolving.
Copyright 2002 Elsevier Science Inc. All rights reserved.
Keywords: Software development models; Open source; Quality assurance; Survey
All the comments so far have been arguing that "no set release date means more QA". From my experience with open source software, this is complete crap. In many cases, commercially used open source software is developed by organizations such as Red Hat and IBM who, we hope, have significant QA teams and resources.
But, for the projects contributed by individuals, with a single or small group of developers -- who wants to do testing? Everybody wants to be a developer, and the more new features, the more bang, the more people will be drawn to the program, so more often than not new features and frequent releases rather than testing are the focus. Most of the time, you can guess from the sheer frequency of builds with new features that QA is not being taken into account. This system works great for projects that enthusiasts use -- individuals request new features, play with the product, and in return, report bugs. This does not work for companies of any size who expect software to work properly right out of the box, where the "feedback loop" of reporting bugs is a last resort.
Open source programmers and commercial programmers alike are under pressure to release early and release often. And in both the open source world and the commercial world, there are groups that excel at QA and groups that don't take QA seriously enough.
Communism was just a red herring.
Interesting thing is, having been through the ISO 9001 rigamarole, I can tell you that it does NOT necessarily enhance quality. ISO 9001 has very little to do with quality, regardless of what their claims are to the contrary. ISO is all about process and repeatibility. It simply validates that you have a well-defined process. You can have a wonderful process, with everything down to the number of times the developers go to the bathroom and for how long documented and validated, and still have a product that's a piece of crap.
An organization I used to work for did the ISO thing. The software had no fewer problems after ISO than it did before. It's just a nice thing to put on my resume, really.
Scott Adams of Dilbert fame put it best: If you never went through an ISO audit, you probably don't know what ISO is about. If you did go through an ISO audit, you definitely don't know what it's about.
Karma: Frotzed (mostly due to the Frobozz Magic Karma Company)
Since when does QA set the release date in open OR closed software? Certainly not with any company I've worked for.
The role of QA is, unfortunately for the state of software, rapidly diminishing. Open Source has only rarely done QA in a professional manner due to it's very nature. And in closed source in today's economy something has to give between time, money and quality. We all know it's not going to be money and time to market is viewed as the all-important element, leaving one thing to suffer.
I have literally worked for a startup where it was essentially "It boots? SHIP IT NOW!" That's how sad it is with respect to professional development these days.
These days the only people that care about Quality are the customer and QA.
Open source projects do not necessarily "exist for other motivations", or at least not completely. Commercially-sponsored open source projects (e.g., RealNetworks' HelixCommunity.org) and commercial applications built using open source components (e.g., IBM Websphere) most likely have release date considerations. Even non-commercial projects have to release *sometime*...
Commercially-sponsored open source projects can attempt to dictate release schedules for the open source projects to match their commercial needs. I say "can attempt" because too much of that may be seen as "heavy-handed" tactics by the community and therefore stifle collaboration.
Commercial projects that use open source components are probably largely stuck lobbying for lots of milestone builds, from which they will pick one to use for shipping. There's evidence that Sun (OpenOffice.org) and Netscape (Mozilla.org) worked this way with their products.
Also, open source projects do need to have at least soft release dates, if for no other reason than to try to maintain community momentum. If the project just keeps coding...and coding...and coding...without ever packaging up a release, the user portion of the community may wander away to something that's creating usable releases more often.
There's no doubt that non-commercial open source projects can leverage their status to maximize quality -- slips in preferred timetables to deal with quality issues will generally be seen as A Good Thing by the community.
The Busy Coder's Guide to Android Development
True. However, if your development processes are unpredictable and unreliable, your product is most likely going to be a piece of crap. You may have superstar coders who will "save the release" by working through the night for weeks, but the result is not going to be a well thought out and maintainable piece of code you can build future versions upon.
Like you, I've been through an ISO audit and I can affirm your observation that it guarantees nothing. However, if you don't go through something like it, you won't even know if your process is poor.
http://mail.gnome.org/archives/gnome-hackers/200 2-June/msg00041.html is the historically significant mail for GNOME in the post-2.0 era.
We're sticking to 6-month releases (in fact, 2.2 hit the streets after 5 months since the first bit of time was bugfixing 2.0). The advantages to having time-based releases are if the features aren't ready, they wait while the bugs from the finished stuff are removed and the new features released. This is instead of adding more features, making the software (presumably) buggier and buggier, which would make the release QA much harder and more painful.
6 month releases means that a feature that misses a given release will not have long to wait before it can be in the main development branch again.
So far, it's working.
Andrew Sobala, GNOME QA dude, variable hacker, and release team member
Commercial software is pretty thin on the warranties as well. Last time I checked, MS only offered a vague 90 day warranty that the product would do "substantially" what it is supposed to.