Slashdot Mirror
QA Under The Open Source Development Model
carrowood writes "A survey was conducted questioning open source developers from both large and small projects concerning their quality assurance practices. A
research paper based on the survey result was just published in the Journal of Systems and Software.
Some comparisions between QA practices of open vs closed source projects are made with some interesting observations. While on the whole it looks like open source QA can be as good as that in traditional software development, there were a few areas pointed out where the open source community does not do so well, such as regression testing and setting release dates.
A thought provoking read."
Do any open source projects get audited for ISO 9001 compliance? They have a quality certification that many [enterprise] customers desire.
I know at work ISO audits are both fun and exciting! (must contain laughter)
--------
Free your mind.
Considering the fact that open source software is often not commercial, it doesn't sound strange to me that release dates are not very strict. As there often is no budget, there is no maximum allowed time that may be spent on the feature set. Thus, software is released whenever the planned features have been built in, rather than cutting the desired feature set to fit the release date.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
Kudos for editors for publishing the story.
...then you can go and buy software developed out of a motivation for financial gain.
Surely open source projects exist for other motivations, and hence have less (if any) need to aim towards release dates.
After all, in the business world the sheer fact that you _have_ to get something out by a particular date (because marketing / the bean counters said so) contributes significantly to the quality problems.
Ultimately, while development can, in certain cases, be done in a vacuum, QA cannot (and should not). It's by nature a collaborative and interactive process.
I have nothing but respect for the few (good) QA engineers I've worked with.
Well, the fact that no releases dates are setted are more a good point than a bad one ! Of course, in an ideal world where software would be released on dates (!), they won't have bugs either. But in the real world, must proprietary software aren't on schedules, and anyway, when they are, this is often at the detriment of the number of remaining bugs and/or dismiss of some features.
In the free software world, the software is released when ready. So, of course they don't set release date (generally speaking -- some projects have regular releases). But I hardly see that as an obvious bad point. It could be on the contrary one of the strength of the free software.
At least, programmers on free software releases when they are happy with the code.
Lemme guess... "When it's done" isn't good enough?
N4st0r, trixx0r h0bb1tz0rz! Th3y st0l3 0ur pr3c10uzz!
1. Introduction
As everyone knows, Open Source software is the wave of the future. With the market share of GNU/Linux and *BSD increasing every day, interest in Open Source Software is at an all time high.
Developing software within the Open Source model benefits everyone. People can take your code, improve it and then release it back to the community. This cycle continues and leads to the creation of far more stable software than the 'Closed Source' shops can ever hope to create.
So you're itching to create that Doom 3 killer but don't know where to start? Read on!
2. First Steps
The most important thing that any Open Source project needs is a Sourceforge page. There are tens of thousands of successful Open Source projects on Sourceforge; the support you receive here will be invaluable.
OK, so you've registered your Sourceforge project and set the status to '0: Pre-Thinking About It', what's next?
3. Don't Waste Time!
Now you need to set up your SourceForge homepage. Keep it plain and simple - don't use too many HTML tags, just knock something up in VI. Website editors like FrontPage and DreamWeaver just create bloated eye-candy - you need to get your message to the masses!
4. Ask For Help
Since you probably can't program at all you'll need to try and find some people who think they can. If your project is a game you'll probably need an artist too. Ask for help on your new Sourceforge pages. Here is an example to get you started:
Thousands of talented programmers and artists hang out at Sourceforge ready to devote their time to projects so you should get a team together in no time!5. The A-Team
So now you have your team together you are ready to change your projects status to '1: Pre-Bickering'. You will need to discuss your ideas with your team mates and see what value they can add to the project. You could use an Instant Messaging program like MSN for this, but since you run Linux you'll have to stick to e-mail.
Don't forget that YOU are in charge! If your team doesn't like the idea of giant robotic spiders just delete them from the project and move on. Someone else can fill their place and this is the beauty of Open Source development. The code might end up a bit messy and the graphics inconsistant - but it's still 'Free as in Speech'!
6. Getting Down To It
Now that you've found a team of right thinking people you're ready to start development. Be prepared for some delays though. Programming is a craft and can take years to learn. Your programmer may be a bit rusty but will probably be writing "hello world" programs after school in no time.
Closed Source games like Doom 3 use the graphics card to do all the hard stuff anyhow, so your programmer will just have to get the NVidia 'API' and it will be plain sailing! Giant robot spiders, here we come!
7. The Outcome
So it's been a few years, you still have no files released or in CVS. Your programmer can't get enough time on the PC because his mother won't let him use it after 8pm. Your artist has run off with a Thai She-Male. Your project is still at '1: Pre-Bickering'...
Congratulations! You now have a successful Open Source project on Sourceforge! Pat yourself on the back, think up another idea and do it all again! See how simple it is?
Open Source... Quality Assurance... I didn't know you could use those 4 words together in a sentence...
:)
"51% projects have one developer" - Now we have proof geeks really can't work well with others
- Adam L. Beberg - The Cosm Project - http://www.mithral.com/
In my many years of experience, I have to say that regression testing is not as common in proprietary software development as it should be (and frequently not as common as claimed). Furthermore, while I don't want to say that regression testing is less important for Libre Software, I will say that I think it's probably more important for proprietary software, where the programmers are writing for a paycheck, rather than for pride, and are frequently under intense deadline pressure (which in turn, frequently leads to testing/QA of all types being skimped).
:)
As for no release dates, anyone who doesn't recognise that as an advantage of Libre Software simply lacks any clue about the process. Sure, there are downsides, but nothing's perfect. Free, reliable, on-time, pick any two.
Or that the itches they're scratching are quite personal ;-)
When I release code I realy dont care about getting any certifications and accridations linked to it. I write code to be usefull not to be noticed. You use open source at your own risk, and the beauty of it is if you dont like it you can just change it. Why would you need a quality assurance process in place for open source. If its open and its quality people will use it. If it doesnt work then people will just blow it off. Does any one else share in my oppinion?
Faith_Healer -- The antethsis to almost everything, and the worlds worst speller.
Well, that's easy to understand. Commercial products are generally released to a schedule, whereas open source is released to a standard.
For example, anyone with half a clue in the IT profession is aware that you NEVER NEVER NEVER take on board a x.0 release of any Microsoft product - you ALWAYS ALWAYS ALWAYS wait for the first service pack as the release date was set by marketing types who wanted profile over quality. Even if the product is buggy people get their bonuses, and people talk about the product (no such thing as bad publicity).
Open source products are typically released by people who don't want to spend large amounts of time answering emails from users about problems, and madly fixing bugs that slipped through.
It's like back in the old days when if a vendor released a product, andthe product had problems, they fixed the problems for free rather than just make you wait for the next release and buy it. Yes kids, that did in fact happen.
I skimmed the article. I'm not sure what they were trying to prove. Release dates didn't jump out at me. As a software engineer, I don't see what release dates by you. Regression testing is obviously missing.
BUT HERE's the free idea. Maybe someone (aka doctoral candidate) could prove that because Free software is used by ALL possible users in whatever way they are going to use it within, say 4 months, that it doesn't matter if there's some obscure bug, in say, the Intellivision drivers for Linux, because no one uses those. I.e., software has an astronomical amount of states, any 1 of which could be broken, but after all (5 billion people) actually use the software, all HUMANLY possible states are VERIFIED.
Quality assurance under the open source development model
Luyin Zhao, Sebastian Elbaum.
The open source development model has defied traditional software development practices by generating widely accepted
products (e.g., Linux, Apache, Perl) while following unconventional principles such as the distribution of free source code and
massive user participation. Those achievements have initiated and supported many declarations about the potential ofthe open
source model to accelerate the development of reliable software. However, the pronouncements in favor or against this model have
been usually argumentative, lacking of empirical evidence to support either position. Our work uses a survey to overcome those
limitations. The study explores how software quality assurance is performed under the open source model, how it differs from more
traditional software development models, and whether some of those differences could translate into practical advantages given the
right circumstances. The findings indicate that open source has certainly introduced a new dimension in large-scale distributed
software development. However, we also discovered that the potential of open source might not be exploitable under all scenarios.
Furthermore, we found that many of the open source quality assurance activities are still evolving.
Copyright 2002 Elsevier Science Inc. All rights reserved.
Keywords: Software development models; Open source; Quality assurance; Survey
All the comments so far have been arguing that "no set release date means more QA". From my experience with open source software, this is complete crap. In many cases, commercially used open source software is developed by organizations such as Red Hat and IBM who, we hope, have significant QA teams and resources.
But, for the projects contributed by individuals, with a single or small group of developers -- who wants to do testing? Everybody wants to be a developer, and the more new features, the more bang, the more people will be drawn to the program, so more often than not new features and frequent releases rather than testing are the focus. Most of the time, you can guess from the sheer frequency of builds with new features that QA is not being taken into account. This system works great for projects that enthusiasts use -- individuals request new features, play with the product, and in return, report bugs. This does not work for companies of any size who expect software to work properly right out of the box, where the "feedback loop" of reporting bugs is a last resort.
Open source programmers and commercial programmers alike are under pressure to release early and release often. And in both the open source world and the commercial world, there are groups that excel at QA and groups that don't take QA seriously enough.
Communism was just a red herring.
and I have my new release.
When does any commercial project match that ?
I hope cvs won't ask me for a credit card number in the future....
>>such as regression testing and setting release dates
Scheduling (Release dates) have nothing to do with QA. A schedule is a separate leg of the stool upon which a project sits; quality, scope and cost being the others.
Release dates for open source projects are more honest. They say beta or alpha when they really mean beta or alpha, and they say release when it actually works.
Most proprietary software fails for reasons that manufacturing failed in the 1980s. Top down, hieararchical processes simply do not work. You need to have cross functional teams, have a quality culture that allows developers to take risks and the initiative, and interact directly with requirements. Most of today's processes simply don't do that, preferring instead to build software in the same way that GM used to build cars. We get buggy bloated software that is unreliable for the same reason that we used to get buggy bloated cars that are unreliable, because the process that builds them stink!
This is my sig.
4a. Pretend to be a supporter of OSS
You can always play it the Linus & co way, and get suckers to write code for you. Then you get all the credits, become a diva, and what you only have to give back to the community is fuck all. Is very simple, and quite frankly the potential is there.
The way I handle QA usually is to write, test and release the code myself and then request feedback from the user community. QA is not just about bug testing, but it is also about making sure that the program actually enables people to get their work done. So usually, I release beta then rc then final but request feedback at every point regarding general usability.
Also maintaining a roadmap is also essential, but it should be flexible based on user feedback.
Ideally one of the strengths of OSS is that it provides for more contact between developers and users. This changes the dynamics of QA and hopefully will result in better, more productive software.
LedgerSMB: Open source Accounting/ERP
Look at a company like MySQL AB vs. Microsoft. MySQL 4's release dates slipped and slipped, and when questioned, Monty responded that the software will be ready when it's ready. A company like Microsoft is driven by the marketing team. They set a launch date, and bugs or no bugs, the software will be launched with maximum fanfare on that date.
I think it's obvious which practice delivers the most stable software.
The open source development model allows tremendous flexibility, allowing members of a development team to be dropped or added at a moment's notice. With the source readily available, one can become familiar with a project's code before applying to be given access to the CVS or equivalent repository. Gradual accretion may produce code in a style not unlike that of James A. A. Joyce's Ulysses manuscripts, but, like James A. A. Joyce, all of the core developers can easily jump from point to point in the code and comprehend the necessary sections and the C/PHP/Python/Java equivalent of their allusions.
Unfortunately, as a result of this decentralised development system, commercial QA, support and RHQ are not as readily available. I'm a middle manager and my company has had a double-sided experience with the MySQL AB organization. They produce a fine product which is perfect for a medium sized corporation such as the one I work for (which shall remain nameless). MySQL worked very well for us, but unfortunately at one point we started receiving segmentation faults when there were more than 30 connections or if a query was greater than 2,048 characters in length. We have reported these bugs to MySQL AB but they have not yet fixed them in their latest gamma/production release. However, they have been very polite and are always willing to cooperate with us; even if small portions of the code are not yet fixable and have escaped the relatively poor QA of the EOD, their TOS have always been reasonable and our MD has always been able to CWT regarding the slight problems and BS our way around them.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
The open source development model gives tremendous flexibility and leeway for various methodologies such as XP or the more traditional but cumbersome waterfall method, allowing members of a development team to be dropped or added at a moment's notice. With the source readily available, one can become familiar with a project's code before applying to be given access to the CVS or equivalent repository. Gradual accretion may produce code in a style not unlike that of James A. A. Joyce's Ulysses manuscripts, but, like James A. A. Joyce, all of the core developers can easily jump from point to point in the code and comprehend the necessary sections and the C/PHP/Python/Java equivalent of their programmative allusions.
Unfortunately, as a result of this decentralised development system and the reduced usage of traditional ways of programming large-scale projects like WM and RDP due to the geographic dispersion of developers, commercial QA, support and RHQ are not as readily available. I'm a middle manager and my company has had a double-sided experience with the MySQL AB organization. They produce a fine product which is perfect for a medium sized corporation such as the one I work for (which shall remain nameless). MySQL worked very well for us, but unfortunately at one point we started receiving segmentation faults and other similar insidious faults (such as KPs and the like) when there were more than 30 connections to multiple database servers or if a query was greater than 2,048 characters in length. We appreciate that these difficulties can be hard to track down and correct and so our engineers have reported these bugs to MySQL AB but they have not yet fixed them in their latest gamma/production release. However, they have been very polite and are always willing to cooperate with us; even if small portions of the code are not yet fixable and have escaped the relatively poor QA of the EOD, their TOS have always been reasonable and our MD has always been able to CWT regarding the slight problems and make our way around them.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Since when does QA set the release date in open OR closed software? Certainly not with any company I've worked for.
The role of QA is, unfortunately for the state of software, rapidly diminishing. Open Source has only rarely done QA in a professional manner due to it's very nature. And in closed source in today's economy something has to give between time, money and quality. We all know it's not going to be money and time to market is viewed as the all-important element, leaving one thing to suffer.
I have literally worked for a startup where it was essentially "It boots? SHIP IT NOW!" That's how sad it is with respect to professional development these days.
These days the only people that care about Quality are the customer and QA.
I'm tired of all this talk about open source development vs. closed source development. That's not the issue here.
It doesn't matter so much whether the source is open or closed as it does who is in charge of the project. Any company could use standard software development methods to produce open source software. Similarly, any company could hire developers all over the world, and have them work together on a project without ever having met.
It's not an open vs. closed thing, and it has little to do with licensing. It just happens that most companies use standard development techniques and don't release as open source.
If it is true, then you haven't done your homework very well. File dialog is purely aesthetic, and there are many in use. PCI modem support is hard to reverse engineer for software modems - PCI true hardware modems will Just Work. As with windows there is a choice of screensavers... need I go on?
That said, you do mention a couple of issues which are valid for the novice user and limit the uptake of Open Source environments on the consumer desktop.
I unintentionally duplicated this comment because my browser was taking too long to post the comment in the first place and I stop/reloaded.
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
What I find interesting about this algorithm is that it is applied individually by each node; there seems to be no need for nodes to share data over some complicated protocol as in many distributed systems. Yet (I think we can believe Clarke) this change improves response time through the system as a whole. It's a validation of the basic Freenet model of systems acting alone but providing a service greater than the sum of its parts.
There are excellent tools for development and version control (gcc/make/kdevelop/cvs). What about testing? My first thoughts were shell and/or "expect". What do others use? (I know it's a broad question... but was curious what others use.)
Other areas of problems are attributable to slovenly or "don't give a damn" attitudes--unused variables, unreachable code, "magic number" constants, and so on. Ignoring values returned by a function are very common. Maybe it is acceptable with a library function, but why return a value if you aren't going to use it? It's better to make the function into a procedure by returning void. On a more theoretical level, the use of weak typing even when the language allows for tighter specification of variables. Strong typing is designed to prevent such oddities and inadvertently multiplying a color by date.
However, in the end it all comes back to complexity. And that is where the biggest improvement in OSS quality can be obtained.
How could you miss such an obvious stage?
Escher was the first MC and Giger invented the HR department.
why is is that IBM open source proejcts have more regression testing than SUn's own?
:)
Inquiring minds want to know
Don't Tread on OpenSource
Comercial software has relase dates for one reason: money. If they don't release it at the right time they don't make money. Release too soon and you get a reputation for bugs. (beta programs help relase earlier, but you still can't do them too soon). Release too late and someone else will beat you in.
However it is more complex. I've worked with projects targeted at telecoms. Back then they had a lab, and nothing was put into production until they bought a bunch of them, and it was in the lab for 3 months while THEY tested it. However you couldn't get something into the lab anytime you were ready. They opened the door for a short time every 3 months. Miss the day where they bring something in, and you wait 6 months before they will make a major order. Release on that day, and they would normally buy in 3 months. Once you were in the lab you could provide bug fixes under the claim "We never saw that in our testing..." so you had to relase a product on their schedual no matter what. If you didn't, someone else did, and 3 months latter they get the order, and unless they screw up baddly you may never even get into the lab.
Open source doesn't care about your scedual, it cares about good. So we have release when it is ready. Thats why freeBSD announced a schedual ship of ONE year, and once the extra year was up released a 5.0 version and called beta quality and recomended you not use it for anytime important. Of course smaller projects are not so good, but then they don't have as many users who care either.
I like to make this point about ISO-9000: if they are not now ISO-9000 certified, McDonalds could be in about five minutes - they have a proceedure for everything. Does this mean that McD's good is "good quality"?
IF you have a proceedure, AND IF that proceedure included required analysis of failures, AND IF that proceedure requires improvements to the proceedures to correct the failures, THEN you MIGHT begin to approach quality over time.
However, far too many company's ISO proceedures fail to require analysis and actively discourage improvement to the proceedures ("You want to change the proceedure! OK, here's a ton of paperwork to fill out, and you will have to get the signatures of fourteen people who would rather not sign anything other than their own paychecks. Have Fun!").
ISO is really just a big peer-pressure MLM scam the way it is run now.
And yes, the company I work for is ISO 9001 qualified.
www.eFax.com are spammers
Before you get your fucking balls in a tangle know this: Neil Peart is the fucking drummer from Rush so close your fucking penis moistener ok fuckio.
The tool you use should depend on what system you are testing. If it's windows software then 'expect' is probably not quite much help! Neither would it help if your programming in KDE ( or Gnome before anyone has a go.)
....
The main problem with any tool, is that the results of a test need to be known precisely for it to work. This may sound a strange thing to say but consider the following.
If you are developing a system that uses a well designed protocol, eg errrmmmm lets say http ( no comments about how well it's design, just go with the flow ). Then you can be quite sure that if you stuff x down one side of the protocol then y will be returned. In this case, you should be able to automate the testing from the word go.
but
If your developing a user interface for Windo... I mean KDE, then its for more difficult to determine what the exact result of a test will be. After all, just how wide will the developer make a text box! Most tools will flag an error if a componant is only slightly different size. There is also the problem that there are bugs which usualy only happen through silly use, eg double clicking rather than single clicking. And thats not even addressing the times when a screen design just doesn't work well for a user.
The point I'm making is that there is no single solution. And even if you have an automated tool, then it may only be of use once the product is stable and in a regression test phase.
8 years of testing ( and no, I don't secretly want to be a developer ) has taught me that the best tool, is someone sitting there using the system, and doing what users do - i.e. stupid things.
http://mail.gnome.org/archives/gnome-hackers/200 2-June/msg00041.html is the historically significant mail for GNOME in the post-2.0 era.
We're sticking to 6-month releases (in fact, 2.2 hit the streets after 5 months since the first bit of time was bugfixing 2.0). The advantages to having time-based releases are if the features aren't ready, they wait while the bugs from the finished stuff are removed and the new features released. This is instead of adding more features, making the software (presumably) buggier and buggier, which would make the release QA much harder and more painful.
6 month releases means that a feature that misses a given release will not have long to wait before it can be in the main development branch again.
So far, it's working.
Andrew Sobala, GNOME QA dude, variable hacker, and release team member
" there were a few areas pointed out where the open source community does not do so well, such as regression testing and setting release dates."
/usr/src/regress/ directory. The tests are numerous and extensive.
It really depends. Setting release dates is not mandatory for an opensource project. Those projects are not made by a company for a customer requiring a specific deadline. OSS projects are mostly made during spare time. So the most reasonnable release dates are "when the code will be completed".
Regression tests are another story, but it really depends on projects and how they are managed.
For instance OpenBSD has very strict release dates. Every 6 months. And this schedule has always been respected.
OpenBSD also always introduces regression tests for new commands or new features implemented in existing commands. Have a look at the
{{.sig}}
Process is the heart of engineering. There is at least one thing that links civil engineers to electrical engineers to software engineers, it's process. In science to, there's that little matter of reproducibility of results. Having a well defined process is the only way to get there.
I'd argue that if you don't have process, you're hacking. Which is not always a bad thing, but not many people would bet the company, their livelihood on hacking out code.
The company has to be able to adapt to losing a key programmer or manager. Maybe your company has a few guys who really know what they're doing. What guarauntee do you have that the project can survice they living.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
While I would always prefer a quality product released late to a buggy one released on time, some customers do depend on reliable release dates. If a customer has a release date for a big package like Oracle, or Solaris, or whatever, they can schedule the appropriate upgrades or testing periods, or whatever their IT policy dictates.
WARNING: Linux-specific stuff ahead
Currently, AA and AC are cool for regression testing. Its nothing like what the "Big Boys" (read: well-funded corporations) do, but its a far cry from the pre-2.x days.
We could all help out with that, via well-written bug-reports, questions, etc. Just make sure its an actual bug and not a vendor quirk. Make sure those reports go to the right people, too - use the LKML as a last resort. (Speaking from experience).
As far as release management goes, it's *still* up to Linus. Not that, I've ever had a problem with that, it's *his* project, after all. If I'm feeling lazy, I'll just wait till the big vendors pick it up.
C|N>K
A schedule is a separate leg of the stool upon which a project sits; quality, scope and cost
Knowing that a proper stool only requires three legs, the Debian project decided to throw the schedule in the bin.
Read, L
If you are going to make the ultimate band the bass player better be that fucker from Primus or the band is not the ultimate.
The singer? Man, the dude from led zepellin. He can actually sing n shit not like the scrubs from these 90s shitstain bands.
When the company goes to buy repacement hardware it is no longer available for the old software. Because the hardware companies are in cahoots with the software companies, forced upgrades become possible, and now have become standard business practice in computerised warehousing software.
This was why IBM was so great, they really did try to take in the needs of the customer first and made hardware manufactures ensure backward compatibility. With the introduction of MS style get rich quick philosophy to hardware, firmware, and software the business customer has become a thing to be milked not a valued resource.
OH THE SHAME I fell off the wagon and use sigs again!
I do as much formal QA as I'm required to by client sites. Sometimes that means unit and regression test frameworks, sometimes it just means making it "good enough" for daily use.
Regardless of the reasons for the formal QA, you won't find too many people who actually enjoy doing it. It's a necessary part of doing a good job, like documentation. But as with anything that requires tedious, painstaking detail, it's hardly something I want to do with my free time.
And there is the crux of it: my free time. I see no reason someone can't take a branch of a project and apply formal QA processes if they need or want to, but it's another thing to ask people to stop having fun programming, and treat their hobby as a job. It's not. It's relaxation. It's entertainment. It's a thought provoking challeng. It is not going through endless checklists of test cases to see if you broke something.
I do not fail; I succeed at finding out what does not work.
Since when does QA set the release date in open OR closed software? Certainly not with any company I've worked for.
I've worked for Microsoft, where QA literally decided the ship date. If it wasn't ready to ship, then it wasn't ready to ship.
(which, of course, provokes flames from the Slashdot crowds as Microsoft never can stick to a release date. Consistency is not a feature of Slashdot...)
I don't know of ISOed OS software, but I am aware of organisations that have gone through a quality audit who use open source software. The main issue is having a test and internal release procedure, so you don't, for example, roll out Perl 5.8.1 without ensuring that your users are aware that their old version is being replaced. You *don't* need to get Larry Wall's personal inprimateur on the package, you just need to have a documented procedure.
In this way, security and QA are very much related. Zero security and no quality checks are fine, as long as those people using the system are aware of this and agree.
See my journal, I write things there
might even be a winghing pommie.
This space is intentionally staring blankly at you
From the website:
Aegis is a transaction-based software configuration management system. It provides a framework within which a team of developers may work on many changes to a program independently, and Aegis coordinates integrating these changes back into the master source of the program, with as little disruption as possible.
Aegis enforces a development process which requires that change sets ``work'' before they may be integrated into the project baseline. Works includes requiring that change sets build successfully, and (optionally) that they include and pass tests. It also ensures that code reviews have been performed.
Needless to say it is not very popular because you have less to debug with it and save time for doing other things
This space is intentionally staring blankly at you
YUO is teh geenious!
In my experience, most (not all, of course) Open Source projects aren't concerned with backward compatibility outside of the scope of the project itself. Regression testing in OSS is folded into the bug testing.
That's one of the downsides of OSS. The biggest example: When you upgrade your libc, you have to recompile all of your dependent apps. One thing Windows and Solaris have going for them is that you can run a 5 year old binary on the newest version of the OS and it will almost always work.
It's a big burden on the development team to provide support for old interfaces, but this sort of thing is where OSS has a long way to go. It gets really expensive for individual persons/companies to support (bugfix, etc.) packages that are a few revisions old.
QA (for those who don't know) stands for Quality Assurance. For many small companies, that means testing. Is bigger companies, it means lots of different testing, such as installation, integration, subsystem, system, performance, load, stress, regression, automated, etc. When you get to those companies with the CMM rating (and possibly ISO, I am not sure) QA is not about testing, that is for the test group. QA is about making sure that processes are followed, metrics are collected, entry and exit criteria are met between phases, etc.
For the last 10 years I have been in different companies that had these different ideas of what QA is all about. From the article, I will assume that it is referring to testing, because it talks about OSS. I actually asked an "ask Slashdot" question similar to what this article addresses last year. It got rejected. I am guessing that the OSS community knows little about real Quality Assurance. It relies mainly on the expertise of the developer, their stake in the software, and the "many eyes" of the end users. It seems to work OK for some projects, but not for all.
I am currently working for a company that does software for hospitals. I can now better understand why the OSS model does not fit every software solution. One of the things we do in QA/Testing here is verify the requirements. For all of you OSS people with cocked heads and puzzled looks on your faces, requirements are what you get from your customer that define what they want/need the software to do. Requirements analysts go over them, refine them, create more of them. A whole team of people inspect them, then we create test cases for them while development designs the system around them. When we get the code, we test it against the requirements. In theory, if everything is good, then it meets the customer expectations. Oh, unless they have changed their minds, or a different customer has a different idea of how it should work. (not what it should do, but HOW it should do it)
OK, [/rant].
I think that software QA will be around for a while, because proprietary software will be around for a while. As it should be. I don't think it is OSS or nothing. At least I hope not, I make my living doing QA. And QA isn't just for the rejects of the software development pool, I have a BS-CS, and have chosen to do this field. If your management looks at QA as non-important and puts new developers there just to get them "up to speed", then you have a whole different set of problems.
My beliefs do not require that you agree with them.
Where the core team each adds a new feature and fixes all crash/data loss and the most severe bugs the first week of each month, and then releases at the end of the month. Then the next month we do the same.
Having 11 short development cycles in a year will really help get new features to the apps team in a shorter period of time.
Plus having the features added in isolation really helps find new bugs that we added with the feature.
Open source people might want to consider a new release after each new feature or after each new set of related features.
It is relativitly easy to have a list of all the features you want to add, a dependency list of how the features relate to each other, and the current months feature that is to be implemented.
Then you write the new test cases, run the tests and make sure the old tests pass and the new tests fail, then you add the feature and keep running the test cases until all the test cases pass.
Quality needs to be built in up front. Most OSS projects will fail to do this, but this anomaly of software development isn't confined to OSS. Most commercial software will fail to do this as well.
However, in software development methods that try to test in quality, the most effective method for reducing defects is the code review, and this is definately a major strength of OSS. The conclusion drawn from this would be that OSS has a high QA coefficient.
I would think that OS projects could greatly benefit from automated regressions testing, probably even more so than commercial projects. OS projects can have problems with both the quantity and quality (meaning no disrespect - I'm referring to the level of familiarity with the code base) of personnel available at any given time. So for example, if a submitted patch can be run against an automated test suite by the submitter, it can take some validation load off the core development team. Likewise, if the core developers must run similar tests before checkin, it takes some load off the QA team (as The Pragmatic Programmer put it, "A human should find a bugs only once.").
It even has benefits for the longevity of the project as such tests are part of the doucmentation of the project and would assist a new development team that took over a project that had been abandonded.
You will not drink with us, but you would taste our steel? - Walter Matthau, The Pirates
If they don't release it at the right time they don't make money.
Wrong. Try to find a direct correlation between release dates and money, and with very few exceptions you will find none. The exceptions are mainly in those areas where consumer spending patterns are seasonal, and expecially the Xmas shopping effect. A game that misses an expected Xmas release slot when it has been hyped up for several months previously could make a substantial loss instead of a profit. However, this applies to very few other types of software.
Except in a few exceptional cases as above, there is only one motivating force for particular release dates: management wanting to manage, and inevitably putting quality last after manpower and other resource issues. The key observation here is that management invariably continues to apply its release-date worldview even when there are NO date-sensitive items on the PERT chart that constrain development time. It's in their genes.
As a contractor, I've seen it in dozens of companies. It's just management bollocks and nothing else, in at least 95% of the cases on projects I've worked on. (I've never worked in the games industry though.)