Slashdot Mirror
QA Under The Open Source Development Model
carrowood writes "A survey was conducted questioning open source developers from both large and small projects concerning their quality assurance practices. A
research paper based on the survey result was just published in the Journal of Systems and Software.
Some comparisions between QA practices of open vs closed source projects are made with some interesting observations. While on the whole it looks like open source QA can be as good as that in traditional software development, there were a few areas pointed out where the open source community does not do so well, such as regression testing and setting release dates.
A thought provoking read."
Do any open source projects get audited for ISO 9001 compliance? They have a quality certification that many [enterprise] customers desire.
I know at work ISO audits are both fun and exciting! (must contain laughter)
--------
Free your mind.
Considering the fact that open source software is often not commercial, it doesn't sound strange to me that release dates are not very strict. As there often is no budget, there is no maximum allowed time that may be spent on the feature set. Thus, software is released whenever the planned features have been built in, rather than cutting the desired feature set to fit the release date.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
...then you can go and buy software developed out of a motivation for financial gain.
Surely open source projects exist for other motivations, and hence have less (if any) need to aim towards release dates.
After all, in the business world the sheer fact that you _have_ to get something out by a particular date (because marketing / the bean counters said so) contributes significantly to the quality problems.
Ultimately, while development can, in certain cases, be done in a vacuum, QA cannot (and should not). It's by nature a collaborative and interactive process.
I have nothing but respect for the few (good) QA engineers I've worked with.
Well, the fact that no releases dates are setted are more a good point than a bad one ! Of course, in an ideal world where software would be released on dates (!), they won't have bugs either. But in the real world, must proprietary software aren't on schedules, and anyway, when they are, this is often at the detriment of the number of remaining bugs and/or dismiss of some features.
In the free software world, the software is released when ready. So, of course they don't set release date (generally speaking -- some projects have regular releases). But I hardly see that as an obvious bad point. It could be on the contrary one of the strength of the free software.
At least, programmers on free software releases when they are happy with the code.
Lemme guess... "When it's done" isn't good enough?
N4st0r, trixx0r h0bb1tz0rz! Th3y st0l3 0ur pr3c10uzz!
1. Introduction
As everyone knows, Open Source software is the wave of the future. With the market share of GNU/Linux and *BSD increasing every day, interest in Open Source Software is at an all time high.
Developing software within the Open Source model benefits everyone. People can take your code, improve it and then release it back to the community. This cycle continues and leads to the creation of far more stable software than the 'Closed Source' shops can ever hope to create.
So you're itching to create that Doom 3 killer but don't know where to start? Read on!
2. First Steps
The most important thing that any Open Source project needs is a Sourceforge page. There are tens of thousands of successful Open Source projects on Sourceforge; the support you receive here will be invaluable.
OK, so you've registered your Sourceforge project and set the status to '0: Pre-Thinking About It', what's next?
3. Don't Waste Time!
Now you need to set up your SourceForge homepage. Keep it plain and simple - don't use too many HTML tags, just knock something up in VI. Website editors like FrontPage and DreamWeaver just create bloated eye-candy - you need to get your message to the masses!
4. Ask For Help
Since you probably can't program at all you'll need to try and find some people who think they can. If your project is a game you'll probably need an artist too. Ask for help on your new Sourceforge pages. Here is an example to get you started:
Thousands of talented programmers and artists hang out at Sourceforge ready to devote their time to projects so you should get a team together in no time!5. The A-Team
So now you have your team together you are ready to change your projects status to '1: Pre-Bickering'. You will need to discuss your ideas with your team mates and see what value they can add to the project. You could use an Instant Messaging program like MSN for this, but since you run Linux you'll have to stick to e-mail.
Don't forget that YOU are in charge! If your team doesn't like the idea of giant robotic spiders just delete them from the project and move on. Someone else can fill their place and this is the beauty of Open Source development. The code might end up a bit messy and the graphics inconsistant - but it's still 'Free as in Speech'!
6. Getting Down To It
Now that you've found a team of right thinking people you're ready to start development. Be prepared for some delays though. Programming is a craft and can take years to learn. Your programmer may be a bit rusty but will probably be writing "hello world" programs after school in no time.
Closed Source games like Doom 3 use the graphics card to do all the hard stuff anyhow, so your programmer will just have to get the NVidia 'API' and it will be plain sailing! Giant robot spiders, here we come!
7. The Outcome
So it's been a few years, you still have no files released or in CVS. Your programmer can't get enough time on the PC because his mother won't let him use it after 8pm. Your artist has run off with a Thai She-Male. Your project is still at '1: Pre-Bickering'...
Congratulations! You now have a successful Open Source project on Sourceforge! Pat yourself on the back, think up another idea and do it all again! See how simple it is?
Open Source... Quality Assurance... I didn't know you could use those 4 words together in a sentence...
:)
"51% projects have one developer" - Now we have proof geeks really can't work well with others
- Adam L. Beberg - The Cosm Project - http://www.mithral.com/
In my many years of experience, I have to say that regression testing is not as common in proprietary software development as it should be (and frequently not as common as claimed). Furthermore, while I don't want to say that regression testing is less important for Libre Software, I will say that I think it's probably more important for proprietary software, where the programmers are writing for a paycheck, rather than for pride, and are frequently under intense deadline pressure (which in turn, frequently leads to testing/QA of all types being skimped).
:)
As for no release dates, anyone who doesn't recognise that as an advantage of Libre Software simply lacks any clue about the process. Sure, there are downsides, but nothing's perfect. Free, reliable, on-time, pick any two.
Or that the itches they're scratching are quite personal ;-)
When I release code I realy dont care about getting any certifications and accridations linked to it. I write code to be usefull not to be noticed. You use open source at your own risk, and the beauty of it is if you dont like it you can just change it. Why would you need a quality assurance process in place for open source. If its open and its quality people will use it. If it doesnt work then people will just blow it off. Does any one else share in my oppinion?
Faith_Healer -- The antethsis to almost everything, and the worlds worst speller.
I skimmed the article. I'm not sure what they were trying to prove. Release dates didn't jump out at me. As a software engineer, I don't see what release dates by you. Regression testing is obviously missing.
BUT HERE's the free idea. Maybe someone (aka doctoral candidate) could prove that because Free software is used by ALL possible users in whatever way they are going to use it within, say 4 months, that it doesn't matter if there's some obscure bug, in say, the Intellivision drivers for Linux, because no one uses those. I.e., software has an astronomical amount of states, any 1 of which could be broken, but after all (5 billion people) actually use the software, all HUMANLY possible states are VERIFIED.
Quality assurance under the open source development model
Luyin Zhao, Sebastian Elbaum.
The open source development model has defied traditional software development practices by generating widely accepted
products (e.g., Linux, Apache, Perl) while following unconventional principles such as the distribution of free source code and
massive user participation. Those achievements have initiated and supported many declarations about the potential ofthe open
source model to accelerate the development of reliable software. However, the pronouncements in favor or against this model have
been usually argumentative, lacking of empirical evidence to support either position. Our work uses a survey to overcome those
limitations. The study explores how software quality assurance is performed under the open source model, how it differs from more
traditional software development models, and whether some of those differences could translate into practical advantages given the
right circumstances. The findings indicate that open source has certainly introduced a new dimension in large-scale distributed
software development. However, we also discovered that the potential of open source might not be exploitable under all scenarios.
Furthermore, we found that many of the open source quality assurance activities are still evolving.
Copyright 2002 Elsevier Science Inc. All rights reserved.
Keywords: Software development models; Open source; Quality assurance; Survey
All the comments so far have been arguing that "no set release date means more QA". From my experience with open source software, this is complete crap. In many cases, commercially used open source software is developed by organizations such as Red Hat and IBM who, we hope, have significant QA teams and resources.
But, for the projects contributed by individuals, with a single or small group of developers -- who wants to do testing? Everybody wants to be a developer, and the more new features, the more bang, the more people will be drawn to the program, so more often than not new features and frequent releases rather than testing are the focus. Most of the time, you can guess from the sheer frequency of builds with new features that QA is not being taken into account. This system works great for projects that enthusiasts use -- individuals request new features, play with the product, and in return, report bugs. This does not work for companies of any size who expect software to work properly right out of the box, where the "feedback loop" of reporting bugs is a last resort.
Open source programmers and commercial programmers alike are under pressure to release early and release often. And in both the open source world and the commercial world, there are groups that excel at QA and groups that don't take QA seriously enough.
Communism was just a red herring.
The open source development model allows tremendous flexibility, allowing members of a development team to be dropped or added at a moment's notice. With the source readily available, one can become familiar with a project's code before applying to be given access to the CVS or equivalent repository. Gradual accretion may produce code in a style not unlike that of James A. A. Joyce's Ulysses manuscripts, but, like James A. A. Joyce, all of the core developers can easily jump from point to point in the code and comprehend the necessary sections and the C/PHP/Python/Java equivalent of their allusions.
Unfortunately, as a result of this decentralised development system, commercial QA, support and RHQ are not as readily available. I'm a middle manager and my company has had a double-sided experience with the MySQL AB organization. They produce a fine product which is perfect for a medium sized corporation such as the one I work for (which shall remain nameless). MySQL worked very well for us, but unfortunately at one point we started receiving segmentation faults when there were more than 30 connections or if a query was greater than 2,048 characters in length. We have reported these bugs to MySQL AB but they have not yet fixed them in their latest gamma/production release. However, they have been very polite and are always willing to cooperate with us; even if small portions of the code are not yet fixable and have escaped the relatively poor QA of the EOD, their TOS have always been reasonable and our MD has always been able to CWT regarding the slight problems and BS our way around them.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Since when does QA set the release date in open OR closed software? Certainly not with any company I've worked for.
The role of QA is, unfortunately for the state of software, rapidly diminishing. Open Source has only rarely done QA in a professional manner due to it's very nature. And in closed source in today's economy something has to give between time, money and quality. We all know it's not going to be money and time to market is viewed as the all-important element, leaving one thing to suffer.
I have literally worked for a startup where it was essentially "It boots? SHIP IT NOW!" That's how sad it is with respect to professional development these days.
These days the only people that care about Quality are the customer and QA.
I'm tired of all this talk about open source development vs. closed source development. That's not the issue here.
It doesn't matter so much whether the source is open or closed as it does who is in charge of the project. Any company could use standard software development methods to produce open source software. Similarly, any company could hire developers all over the world, and have them work together on a project without ever having met.
It's not an open vs. closed thing, and it has little to do with licensing. It just happens that most companies use standard development techniques and don't release as open source.
Other areas of problems are attributable to slovenly or "don't give a damn" attitudes--unused variables, unreachable code, "magic number" constants, and so on. Ignoring values returned by a function are very common. Maybe it is acceptable with a library function, but why return a value if you aren't going to use it? It's better to make the function into a procedure by returning void. On a more theoretical level, the use of weak typing even when the language allows for tighter specification of variables. Strong typing is designed to prevent such oddities and inadvertently multiplying a color by date.
However, in the end it all comes back to complexity. And that is where the biggest improvement in OSS quality can be obtained.
I like to make this point about ISO-9000: if they are not now ISO-9000 certified, McDonalds could be in about five minutes - they have a proceedure for everything. Does this mean that McD's good is "good quality"?
IF you have a proceedure, AND IF that proceedure included required analysis of failures, AND IF that proceedure requires improvements to the proceedures to correct the failures, THEN you MIGHT begin to approach quality over time.
However, far too many company's ISO proceedures fail to require analysis and actively discourage improvement to the proceedures ("You want to change the proceedure! OK, here's a ton of paperwork to fill out, and you will have to get the signatures of fourteen people who would rather not sign anything other than their own paychecks. Have Fun!").
ISO is really just a big peer-pressure MLM scam the way it is run now.
And yes, the company I work for is ISO 9001 qualified.
www.eFax.com are spammers
http://mail.gnome.org/archives/gnome-hackers/200 2-June/msg00041.html is the historically significant mail for GNOME in the post-2.0 era.
We're sticking to 6-month releases (in fact, 2.2 hit the streets after 5 months since the first bit of time was bugfixing 2.0). The advantages to having time-based releases are if the features aren't ready, they wait while the bugs from the finished stuff are removed and the new features released. This is instead of adding more features, making the software (presumably) buggier and buggier, which would make the release QA much harder and more painful.
6 month releases means that a feature that misses a given release will not have long to wait before it can be in the main development branch again.
So far, it's working.
Andrew Sobala, GNOME QA dude, variable hacker, and release team member
I don't know of ISOed OS software, but I am aware of organisations that have gone through a quality audit who use open source software. The main issue is having a test and internal release procedure, so you don't, for example, roll out Perl 5.8.1 without ensuring that your users are aware that their old version is being replaced. You *don't* need to get Larry Wall's personal inprimateur on the package, you just need to have a documented procedure.
In this way, security and QA are very much related. Zero security and no quality checks are fine, as long as those people using the system are aware of this and agree.
See my journal, I write things there
In my experience, most (not all, of course) Open Source projects aren't concerned with backward compatibility outside of the scope of the project itself. Regression testing in OSS is folded into the bug testing.
That's one of the downsides of OSS. The biggest example: When you upgrade your libc, you have to recompile all of your dependent apps. One thing Windows and Solaris have going for them is that you can run a 5 year old binary on the newest version of the OS and it will almost always work.
It's a big burden on the development team to provide support for old interfaces, but this sort of thing is where OSS has a long way to go. It gets really expensive for individual persons/companies to support (bugfix, etc.) packages that are a few revisions old.