Slashdot Mirror
QA != Testing
gManZboy writes "Original author of Make and IBM Researcher, Stu Feldman has written an overview of what should be (but is sadly perhaps not) familiar ground to many Slashdotters: Quality Assurance. He argues that QA is not equivalent to 'testing', and also addresses the oft-experienced (apparent) conflict between QA-advocates and 'buisiness goals.'"
From TFA:
QA is described as making sure a project is "measurably meeting expectations and conforming to requirements"
At my job, requirements are often one-sentence requests with no needed detail whatsoever. If it then doesn't go to a business analyst in the IT department, that's what the programmers work from. When the QA process starts, it makes it easy to say that you've complied with all details of the requirements.
I'm a big tall mofo.
IME, Quality = Knowledgeable_Staff_On_Good_Salary + No_Deadlines. Unfortunately, this formula larglely is not compatible with business world. So, in the mean time, customers should be grateful if software has been _somehow_ tested and mostly works.
The real problem for linux re; QA is that its now TOO LATE to get decent QA (you need it before coding begins)
QA != Testing
Doh! I had this in high school software development course.
Quality assurance is a process that runs through the entire project, testing is a component of that process.
When building software there is a tendency to lump quality assurance and testing together precipitously at the end of a project. The distinction that is made in this article is an important one, true quality and successful projects are obtained by having quality assurance as a project long process. Then you have quality assurance during requirements, design, development and yes even testing.
Let me be the first to quote Linus Torvalds: "Testing? What's that? If it compiles, it is good, if it boots up it is perfect."
8 of 13 people found this answer helpful. Did you?
From TFA: "The quality assurance process is a process for providing adequate assurance that the software products and processes in the product life cycle conform to their specific requirements and adhere to their established plans."
I think the QA process in the OSS community is more than adequatly covered by the numerous alpha, beta, PR, RC and all the other release mechanisms. The larger your user base, the more chances of a high quality final release (which in turn creates an even larger user base, which is a goldmine for the next release etc etc). That is, imho, one of the biggest advantages that OSS has over proprietary software. No QA department in the world would beat a large enough group of college IT students with too much time on their hands.
Just
Good QA begins when the Business Case proposing a new project gets reviewed.
It continues constantly until the Project Post Mortem gets reviewed.
QA should be involved in every activity regarding the projet in between. (including reviewing the requirements ellicitation process)
Happily, when I worked in QA (for a telecoms test equipment manufacturer) that was how we did things. We, in QA, were responsible to the QA Director, and the Managing Director, and nobody else - that gets engineering in touch with you early and ofen...
I assure you that quality is the thing we lost,
while chasing after the deadline and trying to
meet our budget!
Hmm wait a sec. - I assured something, I
mentioned quality, I own a red(5%)-yellow(80%)-
green(15%) tie - I qualify for CQM
(Corp. Quality Management).
JK
..Slashdot editors could apply QA to the spelling in posted stories?
It's "business", not "buisiness".
and clever != good marks in exams
testing doesn't make the software any better but testing do find bugs which developers missed. Quality assurance is to make sure that the software is of good enough quality before release and testing does confirm the case.
I am harvesting funny/good quotes. Please help by putting them in your sigs
testing can only prove the presence of an error, not its absence.
On another note, QA and QM methodes may sound incredibly dull and based upon "duh - how else should I do this, dumbass?", but are in fact highly sophisticated.
Not because they are not readical new, but because they are radically in their consistency. Think of something, and its error and faults, then of their causes, and their effects and impacts. Prefferably add fault probabilities, too. Then start over again.
It is constant feedback throughout the whole design process that is most important.
Powerful is he who overpowers his temptations.
No, I'm saying that if Linux were to have a QA strategy before the next great project begins, then the product has the potential to be even better.
Why on this damn website will any criticism, no matter how true/justified be taken as the most vile bile ever spewed?
And why is any response to criticism met by words to the effect of "And, of course, YOU think that WINBLOW$ is better? Well let me tell you something, you . . . "?
Six Sigma is one of the many quality processes out there that can apply to everything from design to manufacturing. The idea is to remove variation so that everything you do is always the same. Quite boring but effective.
This can apply to the way code is written and does in some companies.
Evolution or ID?
Every software company I've worked for, the QA department was always separate from Development. Then the classic mistake was always repeated... bring in QA at the tail end of a project and expect them to certify it and find all the bugs.
Madness!!
I think that the real issue here is the difference between code that works, and code that meets the business rules and need. Anyone can make good code, and have it compile and execute. The problem comes when that great code still doesn't fit the need that it was supposed to fill in the first place. This issue has two hurdles if it is to be overcome. First, are coders that have no business knowledge. Second, business pros that have no software development experience. The coders complain that they weren't given the proper details of the project, and the business guys complain that the coders know nothing about business. I think that all biz people need to take a basic programming course, and all coders need to take a business class. The gulf of poor communication between the two camps is quite large without it.
My
What a revolutionary point of view/complaint this article gives! In the sense that "revolutionary" means that history goes through a loop and comes back to where it was before...
It is interesting to see though, how every so many years the same ideas pop up in new guises. Edger Dijkstra for instance said more or less the same thing about Software Engineering and its mantra of process phases and planned testing. And the same argument can (and has) been brought against Kent Beck's Extreme Programming methodology.
Oh well, just goes to show you: the only lesson ever learned from history is that nobody ever learns from history.
If you have good quality you can cut down or completely out testing. Think of it like a math equation. Y = f(x). Y is the output and f(x) is the input. If you control the inputs with no variation then the output will always be the same so no need to test it. Honda has done this with engines. They save money because they don't test every engine. Yet, all their engines always work.
Evolution or ID?
If you take the time to do a good QA, you get complaints that it's taking too long to release the product. If you don't QA the product, you get complaints about problems in your product. It's a no win scenario. You just have to balance both.
A way to look at it is, there is a lump sum for a project. The amount of money (time is money) to produce a product. This amount is made up of base price for product plus rework (which is cost of poor quality). If you can minimize the cost of the rework you can either/or increase profits, cut costs (including time to get product out), improve the design as it is higher quality.
Evolution or ID?
An excellent example of how anecdotal evidence can lead you to incorrect conclusions.
The SEI was approached by the military a couple decades ago. The military had a problem; when it contracted out software development work, it would sometimes get back what it was looking for, it would sometimes get it on time. Sometimes it was late, sometimes it didn't work, sometimes it did the wrong thing, and sometimes they got nothing at all.
The SEI went about polling a large number of contractors, trying to see what was common amongst the ones who delivered. They found there was actually a very strong correllation between a number of processes and practices and high-quality under-budget software. The result is the Capability Maturity Model or CMM for short, which divides companies up into 5 "levels".
The kind of organization you describe is quite definately a "level 1" company, the kind with the highest risk, and the lowest quality. Most companies, even small ones, should strive to follow the practices of at least level 3, as the benefits are quite tangible; no more late projects, and vastly fewer defects.
I mentioned it in another post, but my dad has a good web site that deals with quality issues (IE only, unfortunately). And, if you're looking to improve the quality of your software, his current contract is going to expire soon.
The writer talks about separating QA from the Development group. In our organization, this was a large part of the problem. First, there was a tendency for the development group to "throw it over the fence" and expect QA to find problems that the engineers couldn't be bothered to look for.
The QA staff, on the other hand, rarely had engineers of a sufficient caliber that had the insights to search for and find the most insidious problems. Not only that, they (QA) occupied the no-man's land between business users and development, understanding neither area with any clarity.
Yeah? Well I think you're overrated too.
The Japanese have been all about quality for many years. Their cars are less expensive yet more reliable. They do things in general at a lower cost, that are more reliable and faster.
Not a bad biz model. In the car example they eve build the cars on US soil with these methods, so, it is something we can do and doesn't break out backs.
Evolution or ID?
The jellybean example is absurd. I know an algorithm which will, by looking at only 100% of the beans, find 100% of the defective ones.
ummm... so he's an expert on quality, but can't be bothered to make his Web page conform to standards?
Good thing he's not a cobbler, or you'd be barefoot.
Garg
Garg
Alumnus, Xavier's School for Gifted Youngsters
There are 2 places to add QA to linux.
1) The first is really just a patch, but to QA the current code base and the new incoming code.
2) To add QA to the process of generating the code so that it is controlled enough the output is garunteded.
This would take quite a long time to get in place but in the end it would produce a much better and cleaner code base. This also would give ti respect from more professional organizations.
Evolution or ID?
Was your Dad's site not properly tested?
I'm too cool for a sig.
Quality Assurance = "do the buttons all do what it says on them?"
I like muppets.
Wow, I might enshrine this one in a text file as a great, real-life example of a red herring.
You sir are abso-fucking-lutley correct.
Linux QA comes after the software is released.
It has much better QA then even the professionals but it makes the software look bad when your userbase is doing QA qithout their express knowledge.
This is the downfall of Linux. After software is released it is supposed to be polished, with linux the detailing doesn't start until after release.
You only get two:
If design is flawed, what should QA do?
...And more, see F. Brooks' "Mythical Man-Month" for example, or Alen Holub's "Rope Long Enough to Shoot Your Leg".
If docs are porly written, and incomplete, how does one decide what's bug an what's feature?
If the docs depict the program's behavior, not define it, what can QA do?
And yes, if everythign is done right from the beginning, the QA people would have enough time to do something except testing.
Of course, only third of the two ways to write bugless programs works...
WYSIWIG, but what you see might not be what you need
I know testing only shows the presence of trouble, but if fully automated, you can first replicate a problem, and secondly show it has gone away,
Whereas proofs of correctness need to be redone every time you change code.
Test-centric dev (XP, the Agile methodologies) is standard in Java Dev these days, and there are good Python test tools too. Its a shame that C++ has lagged a bit, though CppUnit works well.
We have used CppUnit to test code, with CruiseControl (on sourceforge) to check out, rebuild and retest the app *every* time something changes in CVS. This is the future: you know all your tests work, all the time.
In fact, I've seen "Knowledgeable_Staff_On_Good_Salary + No_Deadlines" produce the worst kind of quality possible -- no working product at all.
I will agree that ignorant staff will degrade or kill quality, poor pay doesn't help, but tight deadlines can cut either way.
The real key to quality products of any type is:
1) deciding what you want to build, and deciding exactly (i.e., good specs);
2) deciding how you want to build it, and deciding exactly (i.e., good architecture);
3) ensuring that these two elements are matched to the capabilities of your team and budget (e.g., don't try to cram all the R3 featuers into R1); and
4) to create feeedback loops throughout the process to check that you are doing what you think you are doing (e.g., peer code review, pair programming, data acquisition and recording in manufacturing processes). Testing should be merely a "double check".
With these steps, and especially ensuring that the demands are only a bit beyond the capabilities of the team, even a basically competent team on modest pay can produce great things in short times.
Without adequate planning, deadlines and QA, the most brilliant, high-paid teams with no deadlines will produce crap, if they produce anything at all. As Sun Tsu said: 'every battle is won before it's fought'.
I can see the problems this creates. Dev teams blame QA for being fussy; QA hate the dev team for releasing such junk.
We have to embrace test-centric development more. JUnit, CppUnit, PyUnit, they make it easy to write tests. But to convince the developers to write good tests, that is another matter. I often find it is harder to write a decent test for something complex than it is to implement the complex thing, but without that test, how do you know it works?
The same quality concepts can work for both. It just depends on how you implement them.
Evolution or ID?
Can someone please forward this to the good old folks at Mozilla. The written QA requirements (on Mozilla) are so cursory that whole features have dropped off the map simply because nobody bothered to check to see if they still worked (i.e. in 1.4 multiobject drag and drop stopped working). Might also help the parsing engine, which continues to have kruft from the Netscape days (like how is interpretted as instead of as a single broken tag to be ignored [and you can use any tags you want, the second one can even contain parameters, allowing you to really annoy people by adding extra HTML that only works in Firefox/Mozilla, not IE/Opera]). Though really as Slashdot has reported before, Mozilla could really use a more robust and systematically tested parser just to avoid potential buffer overrun exploits.
Bottom line: OSS could get way ahead of commercial software simply by doing proper QA and unit testing (not just the UNIX "it seems to work" test, the "are out of range inputs properly detected or does the program just choke and die") on par with what the best commercial developers have been doing. Just because you have to do all this paperwork and repetive checking when working for "The Man", doesn't mean it's an evil thing that should be thrown out. Sometimes the man actually has some good ideas, even if he spends most of his time shouting about how you didn't dot your i s on that last flowchart.
I must admit to never having any 'buisiness goals'
Replace the "QA" references with "security" and see a trend in application development...
/*Dave
Comment removed based on user account deletion
QA is made indeed by the users, the policy of "release early" is all about that indeed, but I wouldn't generalize your statement on the entire OSS movement.
:) ).
Counterexamples: Debian has a community-driven QA system and takes care that its (long awaited) releases are bug-free. The same can't be said about OSs from Microsoft (what are Service packs made of?) or even Apple (users of OSX prior to 10.2 should remember about that
---- MISSING MISCELLANEOUS DATA SEGMENT --- [sigdash] trolololol
QA vs. Business vs. PHB
Been there done that.
The first thing they drilled into us when our organization switched to a different QA process was that 'quality' did not mean 'good'. Quality just meant that the product or service conformed to certain requirements.
Where I am now, 'quality' means ISO-900x. It is a bureaucratic process for keeping things from happening. I used to be able to find rules and procedures. We had documents that were clear and easy to find. Now we have a rule that the documents that we need can only exist as one copy so that we don't have different versions circulating. I realize that ISO has all kinds of wonderful ideals but the practice around here is something other than wonderful.
I realize that people will say that my organization is using the ISO process wrong and I agree but we still pass all the audits. ie. The QA on the QA system itself does not ensure goodness.
Linux has the other half -- while it lacks a pre-defined QA team, the programmers are not angry with testers.
The core GNU/Linux programs are usually (re)?implementing some well-designed arch/protocol/format/whatever.
Also the docs are much better (although often require googling) -- no "competitors" to learn your secret tricks and hidden APIs.
And BTW an encountered bug is usually sQuAshed, not workarounded -- 'cause geeks and major distros will just recompile everything broken by the remedy.
Some more OSS QA starts when a freshmeater abandons his poorly designed project and joins a better one.
And yes, best open-source progs have been heavily sponsored to become as consistent as they are. But I don't see it as a "major failing"...
WYSIWIG, but what you see might not be what you need
in some (many, most?) cases QA becomes the definitive answer to what the requirements actually are. In some cases the requirements are not very precise, so it is up to the developer to translate them into the code and it is up to the QA to decide whether the developers' interpretation is good enough for the purposes of the system. In reality if you have a QA department, the QA defined test-cases become the guide to how the program behaves under various conditions. These test-cases define the system better than the requirements in some (many, most?) cases.
You can't handle the truth.
One would assume you can get Open Office by way of apt. It came with my distribution, so I'm not sure. The specific application you're looking for in the suite is called "Impress".
Quality Assurance is merely an assurance. It is not a guarantee. It is some corporate hack who knows that some clients (especially govt) will succumb to the wiles of rhetoric. And will even exploit the vagueries of it in the contract when the client starts pushing back for the results they expected. Quality Assurance; broadening the gap between client satisfaction and liability.
He's an expert on process and software quality, not HTML.
:P He said he wrote it in some Microsoft tool and keeps claiming he's planning on rewriting it. Sadly, my father is very much in the clutches of Microsoft.
But, you're right, his site is pretty deplorable.
I find that software Q&A for applications/OS's/etc. for use by the general public is not necessarily the priority it should be due in part to the fact that they can always "patch" something if a bug arrises. But, on some occasions, patches are created after the damage is done. Although I will say that even if QA is a priority, there will always be a few bugs that will slip by the testing, due to the sheer amount of various situations that the software experiences (A plethora of different hardware setups, configurations, etc.)
However, one company that has stood out in this field is the video game developpers Blizzard Entertainment. Sure, they have a reputation of not being able to adhere to shipping dates, but for good reason: They want to make sure that the product they ship is near perfect. I'm sure we're all willing to wait a bit more if that time is being used to test products.
- "I reject your reality and substitute it with my own", Adam Savage
Cost, Time and Quality and you can only have 2 out of the 3.
For instance, you can have a project done fast and cheap but the quality will be lacking. Or you can have a project done correctly and quickly but it will cost you a fortune!
QA is part of that "having it done correctly" piece that most companies tend to cut out. Most companies can only grasp the Time and Cost factor and fail at the whole "Quality" component when doing pre-project analysis. I do not have enough fingers to count the projects I've witnessed that have been running behind schedule (which is usually a BS timeframe to begin with) that decide to cut time from the QA process to make the deadline.
The reasons for "falling behind schedule" and all those other fun things that go into bad project management have been discussed before so I won't even mention it.
If you have read this comic before, you remember "QA Confidential." Funny, funny stuff.
If you haven't, the site is no longer up, but here's a mirror:
QA Confidential
Earn a free iRiver
Sorry if that sounded harsher than I meant.
But the old saw I alluded to - "the cobbler's son has no shoes" - points out our tendency to focus in on something so much we ignore the obvious. Heck, I've done it... most of us have. But first impressions are important. ("I wouldn't go to that cobbler, his kids have no shoes." ) Something you might want to point out to him.
So now I will overcome my bias, fire up IE and see what he has to say...
Garg
Garg
Alumnus, Xavier's School for Gifted Youngsters
There are the following two "standards."
Does it render in Internet Explorer?
Does it render in Mozilla?
His site passes both.
Did you buy your shoes from an ISO 9000 certified shoe manufacturer? Good thing you did, or you'd be barefoot.
That last paragraph makes about as much sense as your argument.
For more information, click here.
For more information, click here.
Counterexamples: Debian has a community-driven QA system and takes care that its (long awaited) releases are bug-free. The same can't be said about OSs from Microsoft (what are Service packs made of?) or even Apple (users of OSX prior to 10.2 should remember about that :) ).
And again, this is polish after the fact.
The Debian community QA system is not really QA but rather product polish. No large changes are made prior to release. How often does the Debian community completely revamp a release because something could be done more intuitively, or easier, or perhaps possibly because it would just plain provide a better user experience.
Debian has an operating philosophy and that is fine, they are free to do whatever they want but don't pretend that polishing a product is QA.
As for the Jab at windows, I will say that while there are a lot of service paks for windows to fix horrible design flaws in the security subsystems, the QA for windows was done semi-properly the first time.
QA is not a bug hunt, it is a complete picture of how the user and in this case, OS, will interact. I think they did a great job with the UI personally but failed in the security part of QA.
Also, release in the Linux community rely on other software to become a usable system. How can any distro invest the time, effort, and money into doing QA for every package they install. They can't.
The quality of the product is not in focus. If you try talk about things like Lint and Purify with persons representing Quality you will get an answer that that isn't about quality at all...
So the whole Quality business is something that is invented to support itself and not the end customer of the product. In the long run the customer is actually more interested in the quality of the product than any provided documentation that states that this product was created with our Superior Methods and Ultimate Skill. That documentation doesn't help at all if the product crashes twice a day...
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
QA is not equivalent to 'testing'
... at least in my experience ;)
No 5h1t, QA is production
"It is constant feedback throughout the whole design process that is most important."
Well it's more than that. Reading anything by Deming for details.
> The Debian community QA system is not really QA but rather product polish. No large changes are made prior to release. How often does the Debian community completely revamp a release because something could be done more intuitively, or easier, or perhaps possibly because it would just plain provide a better user experience.
.....
You DON'T.
Its that simple, it is better to have a product that is maybe not ideal for every case now then nothing now but a perfect product in 2 years, especially when having the imperfect product now doesn't preclude fixing what you found to be wrong and enhancing what you found can be enhanced.
Release often, release early. That may be against the specific idea of QA and releases that you have in your mind, but it seems to work really well from a real world point of view (unlike from a theoretical textbook point of view maybe)
> Debian has an operating philosophy and that is fine, they are free to do whatever they want but don't pretend that polishing a product is QA.
> As for the Jab at windows, I will say that while there are a lot of service paks for windows to fix horrible design flaws in the security subsystems, the QA for windows was done semi-properly the first time.
Have you ever seen Windows 1.0 That my friend was the first time they could have done QA, and I am very sure they did no QA there whatsoever. Lets try again, WIndows 2.0? uh....
Actually, the first real attempt at QA was for Windows 3.0, that is the 3rd major release of that product. How do you mean they got QA the first time around? (not even talkign about proper QA, just an attempt at it)
> QA is not a bug hunt, it is a complete picture of how the user and in this case, OS, will interact. I think they did a great job with the UI personally but failed in the security part of QA.
They ripped off the Workplace shell, didn't manage enough QA to manage an anywhere consistent user experience for another 2 or so releases, and are still not close to the quality and user experience of what they started out ripping off. You call that a great job? please..
To be technically accurate, testing is NOT a part of Quality Assurance at all. In the strict CMM model, they have nothing to do with each other. SQA (Software Quality Assurace) is all over the board, and I don't even think testing is mentioned much.
However, this is one of those things where the industry has misued a term, and it has stuck. I just left a company where I was the SQA leader, and I came to a small company where I am the QA manager. Needless to say, my roles are VASTLY different. But there is a reason I left the other company. :-)
My beliefs do not require that you agree with them.
If it only works in IE then obviously it wasn't QA'd properly and we shouldn't bother visiting your link since the person can't even get something as simple as a web page tested properly.
Believe me, if I started murdering people, there would be none of you left.
This one has been in my craw for years (as evidenced by my use of the word "craw"). The industry has taken the term QA and changed the meaning. QA (or technically, SQA) has absolutely nothing to do with testing. Testing has nothing to do with SQA. Read the CMM model, testing isn't really even mentioned while SQA is an entire KPA on its own (at level 2) and is peppered throughout the other areas as well. (sorry, KPA is Key Process Area)
I want to say that I am not necessarily advocating one way over another. But I don't like it when terms are misused. Testing is testing, QA is QA. I am not going to why, except to say that QA applies to the entire development process and deals with adherence and improvement of process.
I just left a Fortune 25 company in an SQA role to go to a small company where I am the QA manager. That means testing manager. So I am not saying that the CMM is the solution to the world's software problems. But it has its place, and if you are going to use it, then use it as it was supposed to be used. Unfortunately, I am now part of the problem.
My beliefs do not require that you agree with them.
Perhaps if you read TFA, and understood what QA really is, you would know that your statement doesn't make any sense. Yeah, you were trying to be funny, but it is obvious now why this story is necessary.
My beliefs do not require that you agree with them.
In that case you have a specification! In the form of your static prototype.
A prototype and a specification do not contain the same information. A prototype consists of a single concrete instance of the thing a specification describes. It contains more information than the specification in some respects (the concrete design choices the implementor has made to fill in the gaps the specification is silent on) but more importantly it is also missing information that is absolutely required in a specification.
For example: What are the tolerances on particular processes? What are the scaling and responsiveness requirements? And so on.
Nothing that does not capture this kind of information is a specification, and so far as I know only a document can capture this kind of information.
Don't get me wrong: prototypes are useful. But they are not specifications, because they do not contain the all of the information that a specification must contain.
--Tom
Blasphemy is a human right. Blasphemophobia kills.
Quality is for people who intend to be dead ended in their jobs. My directors and executives stay in their jobs for about one year at which time their mistakes, lazyness and venal corruption become someone else's problem. Quality is for losers.
Have you ever seen Windows 1.0 That my friend was the first time they could have done QA, and I am very sure they did no QA there whatsoever. Lets try again, WIndows 2.0? uh....
Are you honestly comparing the QA done for the first two releases of windows to what QA would be in the world today.
Windows 1.0 was fairly progressive for it's time. Not because of what it was but because of the accessibility. You contradict yourself above when you say Its that simple, it is better to have a product that is maybe not ideal for every case now then nothing now but a perfect product in 2 years, especially when having the imperfect product now doesn't preclude fixing what you found to be wrong and enhancing what you found can be enhanced.
Windows 1 and 2 QA was on par with the QA of that time period.
They ripped off the Workplace shell, didn't manage enough QA to manage an anywhere consistent user experience for another 2 or so releases, and are still not close to the quality and user experience of what they started out ripping off. You call that a great job? please..
Consistent with what? With all the other mainstream GUI's out there. Give me a break. How can a 1.0 release of software be consistent. Perhaps a 2.0 release can but with the advances that were happening in computing at such a rapid pace during that time, why be consistent when doing so would mean you are left behind?
Like it or not Windows, brought computing to the mainstream. That was its goal. I think the market share speaks for itself in showing that QA was a success.
You should:
apt-get a life
Apocalypse Cancelled, Sorry, No Ticket Refunds
I completely disagree. That is not the problem. The problem is that business people can't make their requirements clear, and developers need to be able to get out of their developer mindset.
In my 12 years in software testing and QA, I have found that for the most part developers think like developers and cannot seem to get out of that realm. When translating requirements, it leads to huge miscommunications. Likewise, whoever writes the requirements states them poorly. This leads to misinterpretations by developers and everyone else.
The business people should be able to state clearly what they need. Someone needs to be able to take those needs, translate them into good requirements, and developers need to be able to code to those requirements.
IMO, miscommunication causes many more problems than just coding errors.
My beliefs do not require that you agree with them.
Technicallyk, Quality Assurance has nothing to do directly with the quality of the software. It deals directly with the quality of the processes that go into making the software. Software doesn't pass Quality Assurance, the processes are what QA is concerned with.
My beliefs do not require that you agree with them.
Deadlines are good, but when the deadline is set in stone and development slips, it's usually testing that must take up the slack. That means three weeks of testing gets cut to three days and the product gets shipped even if serious problems are found because there is no time in the schedule for development to go back and fix the problems.
I've been on various QA/Testing (no, they are not the same, but my team performs them both) teams for the better part of three years now.
... and the defect is thrown to the wind, because marketing refuses to update their first (and apparently final) draft of specs to accomodate the new feature that would make the product much more palletable to a prospective consumer.
/. We need some more comments from people in the industry, whose job titles actually include the phrase "Quality Assurance" or "Software Testing" or something.
;D
The most severe downfalls I've seen are when the marketing teams and development teams do not adequately communicate with the QA team.
In order for QA to be successful, the QA team must be aware of and in tune with marketing's idea of how the product is envisioned to perform.
Nothing is more frustrating than logging a bug that is a critical flaw in the operation of a product to get the developer response "spec discrepancy"
I was disappointed to enter this thread and see stabs at Microsoft testing and other mindless drivel I'm used to seeing at
Of course going AC here, lest my co-workers lynch me.
Cost, Time and Quality and you can only have 2 out of the 3.
Every time I hear this I cringe, because it is complete crap. On a small level or to someone who has no quality assurance practices in place, it appears to be true, but for large projects/products quality assurance is inversely proportional to cost and time precisely because it cuts down on the amount of time required to develop and test a product to its specifications. Quality assurance procedures will help ensure that errors are caught early on in the project lifecycle and develop standard procedures for project development which may have an initial hit on timelines, but in the end save time. The really nice thing about good quality assurance procedures is that they generally get easier as people get familiar with them and time goes on.
For instance, you can have a project done fast and cheap but the quality will be lacking. Or you can have a project done correctly and quickly but it will cost you a fortune!
I'm assuming that when you say quickly here you are thinking of thowing more manpower at the solution because, as the time to complete a project increases the cost increases and vice versa. Quality does not mean features and flawless software. It means that your client gets what they ask for on budget and it works like specified.
Cost is directly proportional to time, quality assurace might add some time to the project, but it has been my experience that the time it adds is usually offset during testing and not having to add features that a client expects during / after internal testing.
10: PRINT "Everything old is new again."
20: GOTO 10
I wrote the parent.
I pretty much agree with everything you said. When the organization started the ISO process they gave it to an idealistic, hard working guy who tried to do all the 'right' things. He got bogged down in the process and management got impatient because things weren't happening fast enough. They appointed someone who 'just got the job done'. The result was that the 'right' things didn't get done but we got to crow that we were now ISO compliant.
Your comment about killing rogue projects points to another thing wrong with the process. Skunkworks, back-burner and rogue projects are a major source of innovation. Kill those and you ensure that the company will ossify and die. Peters and Waterman in their book "In Search of Excellence" point out the importance of such projects for such companies as 3M. Clayton Christensen in "The Innovator's Dilemma" points out that most important innovation in companies comes from such projects and that companies that can't cope with disruptive innovation are replaced by companies that can. Ergo, you can use the ISO process to commit slow corporate suicide.
My grandfather worked in QA for 25 years at Drackett testing products such as Drano, Windex, various Glade cleaning and scent products, mops, cleaning cloths and so forth.
From what I could gather about their work environment, it was middle-managed, and led to promoting the ones with the most ambition (as opposed to know-how). But then again the company was sold to MeadJohnson and restructured. But products weren't rushed and often "bugs" were discovered before shipments arrived.
As far as Quality Assurance, it was serious business. A cleaning product that produces it's payload via a spray bottle can run into loads of problems. Incorrect pressure can cause weak or excessive sprays and nozzles made out of the wrong plastics can break down after a few uses. Trust me, we saw a lot of products that did and didn't make it to market (including Nutrament which sadly did, although I hated it). In fact I've later cursed "OrangeClean" because their spray mechanism suffers from a problem I've heard of plenty of times. Every product truely had a story and everything had been thoroughly tested - we all made sure that happened. Before working in QA my gp worked as a chemist so he's pretty smart too. He really had the ability to look at the entire project and seeing possible conflits - a knack he still has.
My point is that if you call yourself QA you need to have some know-how and you need to truely test everything. Not every software project needs to meet governement software guidlines but QA should be your bug testers, anyone brave enough to run the software in a production (like) environment. Project managers should be heading up things like compliance, if I'm not mistaken. It makes more organizational sense. Teams should be formed around their functions, eliminating management levels and unneeded personnel.
Let your "QA" team work on things like sending design considerations back to the project managers/leaders and alerting them to potential breakdowns. QA == Testing, it does, it does, it does. I'm sorry but the process includes a lot of testing. They don't have to test the final product, they can test prototypes and then the final product - but they should never make the business decision. (Never)
Get your Unix fortune now!
Project manager: Here Josh, here's your test plan.
QA guy: Thank you. *tests* The results I get for #3 are not what you said to expect. Looks like there's an error.
Engineer: *thinks about it* Oh, ok, yeah. That's not really an error. Ignore that one.
QA guy: *steam comes out of ears*
www.HearMySoulSpeak.com
If they are they are illigimate.
1. Please. We are all professionals. If you need to be in a separate dept so as to not hurt someones feelings... you have the wrong people.
2. Not saying the entity is NOT a full time job. Developers are not QA.
3. Most organizations tend to blow off the QA depts - so that excuse doesn't hold water.
If they were part of the process - from the beginning - there would be less time lost due to software quality. Work with developers within it's own discipline. It can be done!
When I worked in a mainframe group at a major airline writing code for internal use in flight ops, we had a small team of a dozen or so experienced programmer/analysts (perhaps 15 years of experience on average) who each knew their business/application area rather well in addition to being quite competent technically.
We also tended to work directly with a dedicated set of business analysts who were also quite experienced, being dedicated "end users" from various operational areas, and it was a collaborative effort to design and implement projects large and small.
After years of working with the same experienced people, the system worked *very* well. We had processes in place to help ensure that proper testing and documentation was done and that no unauthorized production loads were made -- otherwise, we basically trusted people to use their best judgement.
Since the folks who were writing the code were also supporting the application 24x7 on a rotating basis, they had a vested interest in keeping the system stable.
I think that demonstrated (at least to me) that sometimes a full-blown separate QA process isn't required. By doing things in a somewhat abbreviated way, however, the group was a lot more agile, and quality fixes could literally be coded and loaded in a matter of hours (in some cases).
When I worked for Unisys on application development for paying customers, however, we had a much more formalized process. We had dedicated business analysts writing the func specs, programmer/analysts to write code to those specs, and dedicated QA people who designed and helped implement formal test scripts (both manual and automated) before the product was rolled out.
The size of the group and the nature of the product made that level of QA more important, I think, but it was also implemented knowing that the software development cycle in place was a slow and deliberate process.
Moral of the story: Maintaining a local system for internal corporate use is sometimes a VERY different process from developing commercial software for external customer use, and the two situations can sometimes differ greatly in approach while still maintaining a very high level of quality.
I also think it depends quite a bit on the quality of the people you have in place, and also on the level of experience those people have with the product, the technology, and in working with each other. Experienced people can work wonders if you let them.
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
In reality, Windows rose to dominance because it was an inferior product brought to market at the right time.
businesses chose to use windows because they wouldn't have to rewrite all their legacy/mainframe/in-house apps, because underneath it all, you still had DOS. and if your app didn't play nice with windows task-switching of DOS apps, you could exit wondows entirely, and have essentially a clean slate for the custom app to use.
> Are you honestly comparing the QA done for the first two releases of windows to what QA would be in the world today.
I am saying that it isn't proper QA to todays standards indeed, answering the claim that it had sortof proper QA from the start.
> Windows 1 and 2 QA was on par with the QA of that time period.
Having worked for a company that at the time was involved in the development of Windows (upto and including version 3), I can honestly say it was not on par with QA at that time period. It was an experiment, no more and no less. MS had to have a GUI thing seeing the succes of Apple. Only after 2.1 did they start to consider what their users might want from it, and the difference that made is history.
> Consistent with what? With all the other mainstream GUI's out there. Give me a break. How can a 1.0 release of software be consistent. Perhaps a 2.0 release can but with the advances that were happening in computing at such a rapid pace during that time, why be consistent when doing so would mean you are left behind?
Consistent with itself for a start. The 'make action X perform the same thing everywhere' kinda idea.
They ripped off the Workplace shell for WIndows95 btw, so no 1.0 release here. From that point it took them to XP to make it at least moistly consistent with itself.
I'm ignoring the 1.x, 2.x and 3.x era there, bt I should have said that at least they tried to think up something there themselves, and managed somewhat with 3.0 and esp. 3.1 after 2 initial attempts that looked and worked like shit. It had to be thrown away mostly (but not completely) for 95.
So, to conclude, the first incarnation did not have QA, also not to the standards of that time, and definitely not something you could call somewhat proper QA nowadays.
THe 3rd incarnation had for the time proper QA, and we'd probably recognize it as proper QA still nowadays. The result of which was somewhat dramatic btw.
The 4th incarnation (WIndows 95, ignoring NT 3 and 4 here for a bit) used an UOI that was a ripoff of an IBM project that MS used to have a share in, a ripoff that didn't manage to be at least consistent with itself for quite some time.
I'm ignoring NT 3.x and 4.x here because this was about the UI, and both basicly retain the WIndows 3.x UI.
Also, one can argue that the Windows 9x UI is vastly superior to the one of 3.x and I agree when looking at how it looks and what functionality it offers. That said, the Windows 95 interface has serious quality issues, which is somewhat surprising because there was a very good example out there that MS had access to, and that did get proper QA from the start.
They copied the look and feel, but not the underlying ideas and design (one of the reasons why shortcuts work as bad as they do once you move or delete the target) and as said, didn't do proper QA there.
My contention is that the requirements and design process is more broken than the rest of the software development process. I believe that this is the actual cause of the rise of "Test driven" or Test First" development process currently being peddled as the silver bullet, here what I believe is a more forthright characterization of the methodology:
Due to the fact that the requirements creation process for software is irreparably broken, we have devised a way to sidestep it.
By having programmers/developers/software engineers write tests first, we actually are having them create software encoded and automatically testable requirements documents - it has less to do with testing and more to do with recording design, requirements and assumptions.
Without a functioning requirements process, the development process is hindered, but Testing (commonly known as QA) process is really screwed.
I'm not calling Scrum bizzare. Scrum seems like a good way to develop a piece of software where the requirements churn is very high. I have a hard time thinking of such a system off the top of my head. It might be an acceptable way to develop a low-churn system where you want your time-to-market to be next to zero, provided the cost of a failure is very low, and redeployment costs are minimal, and you don't mind burning some cash down the road to get to market first. A web-based email system, for example, might be a good candidate for scrum. I'd also think that you would need a fairly small team size for scrum to work properly, although I don't have data to back that up.
A mission critical system, though, where the cost of failure is high (say, death), is not really a good candidate for Scrum. If you worked at Boeing and found "777 left wing" in your process backlog, you'd probably want to seek employment elsewhere.
With any mission critical system, you need a very clear understanding of the requirements before you start design - otherwise there is a high probability that your design will fail to meet one of those requirements, and therefore the system will likely fail. The whole point of scrum is that you basically make up features (and, therefore, requirements) as you go along, and hack them in. Your product backlog is basically just a list of features you want but haven't had time to hack at yet.
The requirements of a mission critical system do change, as new features do get added (communications gear is a good example of a critical system with low-medium requirement churn), but scrum is still, IMO, inappropriate, as the rate of change is generally far too low.
Scrum also seems to lead to a lot of rework. If you find a requirement buried in your product backlog which you hadn't previously considered carefully, you may find you have to rewrite a substantial ammount of design to add that requirement. Again, in a system where your requirement churn is high, this is perhaps unavoidable, but in most systems (especially mission critical systems) the majority of your requirements are known ahead of time, so all that rework just becomes waste. The waterfall model would serve you better.
Finally, Scrum has a few pitfalls. Since your product backlog is generally full of features, it's easy to end up doing things like assigning features to designers instead of subsystems. The "sprint" mentality also seems to make it easy to shirk the responsability of producing correct documenation. You can do (or fail to do) these things with the waterfall method too, of course, it just seems like Scrum almost promotes them.
As for "There's nothing inherent in any agile methodology that precludes CMMI compliance," I'll point you to the Agile Manifesto. Let's just look at those first two points here;
1) "Individuals and interactions over process and tools"
How does this relate to CMM?
"Success in Level 1 organizations depends on the competence and heroics of the people in the organization and cannot be repeated unless the same competent individuals are assigned to the next project. Thus, at Level 1, capability is a characteristic of the individuals, not of the organization." - http://www.dfas.mil/technology/pal/cmm/lvl1desc.ht m
So the first point of the Agile Manifesto seems to scream out CMM level 1.
2) "Working software over comprehensive documentation"
Even if you had excellent quality software with poor documentation, the software would be difficult to change and maintain, unless you assign tasks to people who are intimately familiar with the existing code base (Sound like CMM 1 again?). If a person moves to a new project, there is a siginifigant learning curve. This is compounded by the unfortunate fact that, at my company, developers are assigned features, rather than subsystems. A new developer must t
Testing = finding bugs
QA = determining which features are required, and whether or not they work as intended for the end-user.
I'll admit though often there is overlap between the two. If your bug-testing team keeps complaining about a certain feature being counter-intuitive, or whatever, then QA should listen and decide to fix the problem (even if it does not qualify as a "bug").
QA Testers don't break software; it's broken when we get it.
Half of writing history is hiding the truth.
the reason that "testing must take up the slack" could be because of lack of reasonable deadlines/gates in the earlier development stages?
How often does the Debian community completely revamp a release because something could be done more intuitively, or easier, or perhaps possibly because it would just plain provide a better user experience.
As I understand your comment, it's happening Now
QA is not a bug hunt, it is a complete picture of how the user and in this case, OS, will interact. I think they did a great job with the UI personally but failed in the security part of QA.
You are talking about SOFTWARE. QA is a broader subject than bug hunting, but bugs are the most important thing to get rid of, if you talk about quality, IMHO.
---- MISSING MISCELLANEOUS DATA SEGMENT --- [sigdash] trolololol
What I do not like from what I have seen of this book is that it seems to place all the blame for increasing featurism on the developer. In my actual experience however this is not the case. Requirements creep is usually caused by users/managers requesting more features. When the developement cost and time increase and the product has decreased stability, reliability, and useability, everyone is left thinking "Where did we go wrong here?" The book is right in the fact that unnecessary/additional requirements cause lots of problems. It just is not always the developers fault. I think the movie "Pentagon Wars" portrays a more accurate picture of how this is in the real world.
Come on, one of the cornerstones of our culture is "Mistrust authority, promote decentralisation". It really hurts to see authoritarian systems being promoted like that.
When you start talking about open source, I'd agree. But, when you talk open source, you're talking about a whole different beast. Most OSS projects have no Quality Assurance to speak of, other than the discipline applied by the developer to his own work. There are a lot of different discussions about why OSS succeeds, despite doing most of the things that classical engineering would shun, but regardless of which arguments you buy or don't buy, two things are obvious; OSS works, and the bazaar is nothing like the cathedral.
Once you start talking about the cathedral, you're no longer talking about our culutre, you're talking business. At the end of the day, I'd rather get my for-work projects done before the deadline with a minimum of fuss, so I can go home and work on more interesting projects. In order to do that, though, I need to the cooperation of the people around me, many of whom are not uber-hackers like you and me, but people who got into computers because that's where the money is, even though they'd really rather be doing something else.
The cathedral requires a common process which everyone must follow, otherwise we end up working late, past deadline, over-budget, and eventually go under.
You've missed the important bit about the model - you're not looking at all of the beans (or routes through a process to remove the analogy) at once, you're looking at a sample. And each time you pick a sample you see some beans you've seen before. Hence to pick enough samples that you've seen all the beans you needs to pick a vast number of samples.
Similarly each time you go through a process in a bit of software you're mostly going through the same code over and over again until you get to a new branch you've not seen yet. You can't (by testing the software as a whole) just look at the individual parts.
NOTE: This is a genuine question / request for advice. If you suggestions somehow suggest I can fix all of my problems by ramping up on an alternate skill set (Linux, Apache, PHP, Java, MySQL, PostgreSQL), I've already thought about it. However, I currently spend 4hrs a day commuting and I need to pay my bills, so that's not an option right now.
I've been creating web apps for 6+ years using ASP, SQL and HTML. I'm fairly attentive to detail, try to make sure my code is legible / portable / documented in-line, etc. However, I'm currently working on a monstrous behemoth of an application and the need exists for some way to automatically test my app out. Can anybody suggest free or trial-based tools/software packages that can help me unit-test my code before I submit it to end users from second-stage testing?
I would expect the head of process quality at Ford to know that quality means that the product does what it is supposed to do; that is, it conforms to specifications. In addition, I would expect the head of process quality at Ford to know that the World Wide Web is not limited to Microsoft software, but in fact has a set of specifications of its own. Whenever I write or update a web site, I run each page through the W3C markup validation service. I recommend that site to your (and his) attention; it is a good starting point for learning how to write good HTML, and thus quality web sites.
John Sauter (J_Sauter@Empire.Net)
If only you could adapt the same algorithm to software...
I saw this fallacy years ago (>10 years) when I developed software. Tried to change testing procedures. Like trying to fight city hall. I expect his article to change not a thing, sadly.
You should't have said that. Now you'll have every nutty BSD zealot coming out of the woodwork pointing to their dead OS saying that they corner the market on open source QA. Well, it's easy to corner the market on QA when you use OUTDATED software. Sorry, but latest and greatest is where it's at. The latest stuff is usually patched to the gills against new vulnerabilities, where as old code is lacking in features. If you want a feature filled OS, go with Linux or Windows. If you want easy to use and stable, go with MacOS X. If you want archaic, never up to date, backwards, hard to use, insanely complex, unfriendly, bug ridden code, then by all means go with BSD.
Hmmm...
s/notch/noth/
s/nothing/noth/
Survey says?
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
- Use Validators and Load Generators to Test Your Web Applications
It's at GoingWare's Bag of Programming Tricks, and is published under the GNU Free Documentation License.I was inspired to write it after I got a call in the middle of the night from the CTO of a now-failed dot-com, desperate for help because his eCommerce application fell over under a load of less than fifty users when they deployed it live, just in time for Christmas. This despite the fact that their servers were high-end Sun hardware.
Despite the best efforts of our valiant hardware engineers to obey Moore's law, careless mistakes by software engineers can and often do reverse hardware performance gains.
Request your free CD of my piano music.
The algorithm in the Powerpoint document is absurd because nobody tests software like that (for reasons you outline above, among others). Or jellybeans for that matter.
Right?
So what metrics do you use? Do you just tell your team to get their defect/kloc down, without giving them any kind of plan or direction as to how?
The fact of the matter is, if you have a process, and you follow it on two different projects, you'll get two products of very similar quality at the end. If you don't follow a process, you'll get two products of random quality at the end.
CMM 1 you get for free. CMM 2 gives you guidelines as to how to establish a process, so you'll be "repeatable". CMM 3 gives you guidelines as to how to make your process better. When you get up into CMM 5, you have a process that makes itself better.
Again, you have to implement the "intent" of CMM, not the "letter". You don't want to measure your team against CMM, you want to use CMM to reduce your defects/kloc and your effort/kloc.
Quality isn't something that happens after the fact. Good products are the outcome of good design. Good designs are prototyped, rigorously evaluated, and redesigned as flaws are found. It's not even a software-specific concept.
People do test software like that - web applications and computer games being two examples that spring to mind. If you _only_ test them like that then you're doomed, but conversely if you don't test a product the way the user experiences them then you're equally in trouble.
One of my main Day Jobs up to a few years ago was working in QA for Major Computer and Software Manufacturers.
The idea of:
Testing = finding bugs
QA = determining which features are required, and whether or not they work as intended for the end-user.
Is fine in theory, but rarely happens in practice. It usually ends up that QA is ignored and conflated into bug testing. And even then, it often doesn't matter.
Example: I was working on a team that developed an Important Piece of Software That Is Very Popular These Days. We Had No Specification.
None. After some terrifying meetings with the CEO, we somehow brought it to a 1.0 release. I didn't want to have to go through that little nightmare again, so at the 1.0 post mortem meeting, I asked "So, we built 1.0 without a spec - what exactly are we going to do next? What is the Specification for 2.0?"
The lead programmer looked right at me and said "The Spec for 2.0 is 1.0."
We had shipped 1.0 with over 850 bugs, with half a dozen known (if somewhat obscure) crashing bugs, and with several features "turned off" because we couldn't get them to work.
At that point I knew I had to get the fuck out of there. I wasn't going to spend over 2 hours a day driving just to help this rolling train wreck. I left as we shipped 2.0.
From there I went to a company That Was Also Extremely Famous (but now defunct) where QA was more of an expensive after thought. they hired a great team, but the engineers were so disjointed that the product kept changing every other month.
The stress level was so high at that company, of the 120 employees, half a dozen attempted suicide in the 9 months I worked there. At one point, there was such a row in basic engineering philosophy, two of the main programmers got into a fist fight. When the money dried up, we all got laid off.
We can go on and on about how important QA is, but the fact is, we're in a business that makes products, and when the business is more than a dozen people jammed in a garage or airless office space, the products tend to be driven by marketing droids. Left to their own devices, Engineers will produce complex objects that don't necessarily work or fulfil a worthwhile function, or do it in a way that is elegant and useful. Left to QA, the product never gets out the door, because Software Engineering *ISN'T*. SE is more like knitting or quilt making than an Engineered Science. Bridge Builders use Engineers - they have books full of equations that have been proven over the years and they use these solid tested things in a creative way to solve a problem : how to get from here to there.
When a bridge fails, or a building collapses, they just look at the evidence and compare it to the known working materials sciences, engineering principles, etc. and figure out how it failed.
With Software everything is written in a series of codes - at the machine level, we know what they do. But once you get into the meat and potatoes of software development, it all gets very wiggly very quickly. That's why TESTING is needed. And QA should be brought in even before the coding begins - when the UI is being developed, when the notion of the application's purpose and methods are being developed.
But, as noted above: if QA runs the show: it never ships, as there are always improvements to be made. Always.
So, you have the maketing droids who have the ear of the business managers who then set arbitrary and insane deadlines. The result? QA can't touch everything, or they conspire with Engineering that some sections are "good enough" and they let it go, so they can focus resources on testing problem areas, in order to meet the absurd deadline.
The end result is always the same: The public gets buggy software.
The only question is : How Buggy Is It?
They flip the crank, they do the late nights, they get the product out. QA and Eng do their littl
Step 1: Receive vague, often times contradictory goals for a new project. Step 2: Drag management, kicking and screaming, into meetings to discuss timelines and project details. Step 3: Some random vice president attends one meeting and kneejerks the entire project into chaos. Step 4: Everyone loses interest. Step 4: Blame IT. Sheesh. Buzz phrases may sound impressive in meetings, but must be backed up by a chnage in management behavior.
Original author of Make and IBM Researcher, Stu Feldman
make = pain. So now I at least know who to blame.
I've been saying this for decades, even here on Slashdot.
Nice to see this guy has finally gotten it!
The problem is, the price benefits of Quality are well hidden with today's accounting methods, while the price is glaringly obvious to those Harvard grads who never learned how to program but run the company.
Last hired, First fired! It's the official slogan of QA departments everywhere.
If you follow the CMM, your company will improve it's software and product production. If you are having problems following it, then you really don't understand it.
Or Software Development either.
It's not an easy process, you can't instill it overnite, it takes months and years to get up to say Level 3.
But it works, everytime.
Wow, someone else on Slashdot who knows what Quality is and understands it!! I think that makes like 4 of us.
"Technicallyk, Quality Assurance has nothing to do directly with the quality of the software."
QA has nothing to do indirectly with the quality of software either IMHO.
Kind of an off-the-topic question...
But isn't part of QA in manufacturing to intentionally introduce an error, and to see if it is caught by your process?
That's a fun thing to contemplate...
Injecting a requirements error, to see if it is caught. Injecting a design error. Injecting a documentation error. Let alone the act of injecting a coding error - like a crash or a memory leak.
Wow. The imagination reels, trying to picture those things being caught repeatably.
Education is the silver bullet.
I am Quartz's father. I found the replies that were generated by my son's posting are very interesting. My appologies to those who do not have MS Internet Explorer but contrary to what my son claims, I have no intention of changing the page. It just isn't worth the effort to hand craft it in HTML that everyone can read. The objective of the presentation was to show how futile it is to attempt to improve the reliability of a software system of any significant size by backend testing. The impact of any economically feasible test program on the typical defect ridden system is not significant. It then goes on to show how to use some reasonably easy to collect metrics to produce a verification plan that will have a significant impact and that is economical (assuming a reasonable but not exceedingly onorous or rigerous development process which is not changed). One other idea that is very important is the concept of quality as the lack of waste ($)and therefore that quality improvement is the reduction of waste. Waste is not just waste in the development organization. I suspect that Microsoft caused more waste with its operating systems and office tools than they spent on their development. The waste occured through lost productivity on the part of the users. For those who didn't like the analogy built of jelly beans (those blue ones sure taste good!), how about an analogy based on another food group. Mushrooms! (which unlike software defects can be eaten or sold). Consider a wooded park with many trails that fork and rejoin (like the logical paths through a piece of software). On one warm summer morning there are N mushrooms growing in the park. There is a person who walks the same path in the park every morning and who likes muchrooms. On this morning he/she find X mushrooms along the regular path. On a second summer morning there are 2N muchrooms in the park. The person follows the same path and finds 2X muchrooms (the assumption being that the path is representative of the park as a whole). Case 1: Fraction found X/N number left in the park (N-X) Case 2: Fraction found 2X/2N number left in the park (2N-2X) If this analogy holds true, then a given amount of test effort would produce the same fraction of the defects in a software system, regardless of the number present and the more defects a given amount of testing produces, the more defects will be left in the system. The analogy holds not just for software but for blue jellybeans as well. Given that an equal effort will find an equal fraction of the defects, then the analogy can be used to estimate the total defect load at the start of test. If 2E units of test effort were exepended and the first E units found X defects and the second E units found Y defects, then the equation X/N = y/(N-X) should hold true (YX). This can be solved for N. Try it on the JellyBean model examples. It gives much better estimates than the extraplation of the historical data. There is only one way to get a reliable system out of test. That involves putting a reliable system into test. Once the reliability is determined, additional testing is waste. If there are defects, other far more economical methods of finding them exist. Quality is free. But it is not a gift. You have to make an investment and you have to use common sense and maybe a little arithmatic. Testing is intuitively seductive because it looks like it is the right thing to ds. But it doesn't deliver the goods.
Ah, the joys of PowerPoint's "export as HTML" feature.
For more information, click here.
Basically, anywhere within a java program where an exception could legally be thrown, the tool would cause your program to throw one of the allowed exceptions.
Sure, java has checked exceptions, and you have to either catch and handle them, or propagate them, but requiring this doesn't guarantee you do it correctly. Also there are some exceptions, like running out of memory, that aren't checked or declared in a throws clause, that could happen just about any time.
Java being an interpreted language makes it possible to do this to a binary, which I don't think would be possible for a C++ executable, but I think some kind of source code preprocessor would allow such testing for C++ programs. Maybe a good open source project for someone with more time on their hands than I.
Request your free CD of my piano music.
My understanding of the ISO 900x standards is that they strive, not so much for quality, as for repeatability and tracability.
The drill is not to make the best product. The drill is to not make later products that are worse - or accidentally different in bad ways - than the current product. If that means documenting your mistakes so you can make them over and over again, the same way every time, that's just fine.
The customer chose your product, warts and all. He built his processes around it (possibly including a wart-removal step). ISO 900x lets him rest assured that he can get more copies without suddenly finding them different in ways that break HIS processes.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
There seems to be a lot of confusion about what Quality is about; people should consider reading Deming, it works for the Japanese.
Whenever I hear people talk about "new" quality ideas, such as "picking the low hanging fruit" also traditionally known as "chasing fires" my skin crawls.
Equally enlightening are those who suggest "QA is for Losers" because attitudes - and pay skills - that reflect those attitudes have driven many of the best QA people I know - including myself - from the field - in many cases those of us with BSCS leave the computer industry completely because when we attempt to obtain developer positions the hiring manager assumes we must be idiots because we did QA!
Few people seem to recognize that design, specification, and paradigm errors are the largest contributors to software problems. Bugs in code are generally the easiest problems to find - and if there are a lot of problems in code, it's a design error. So, we see "black box testing" and shoddy products.
if QA runs the show: it never ships, as there are always improvements to be made. Always.
Then you're doing QA wrong.
One of the big parts of having a spec and a QA process is to know WHEN TO STOP.
When you get the function of a part right, and the tests have been run and show it's right, you MOVE ON. You only come back to it if the debugging of a later part reveals a hidden bug that the earlier tests missed (or couldn't test without the availability of the later part).
When you've moved on from the last part you're DONE.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
I'm a QA Analyst for a large software firm creating a Point of Sale program. It is complete crap.
p a=showpage&pid=277) We're way understaffed and they will be more understaffed as every QA person is looking for a new job.
It's complete crap because of the reasons you stated above. Our company thinks that QA is a division and not a process. All we are allowed to do is run through some use cases and make sure the thing's still broke. There is no review of requirements (they only write the Use Cases after the module's been built), no code reviews, no Unit Testing, no nothing and the app bearly runs because of it.
QA is not part of the development process. It's only tacked on after the fact. We often get builds that don't even run only to find out that developers didn't even try to run the app after trying to fix it. They've been getting away with this because the QA department manager reports to the development manager and the QA manager's a real wimp. The developer's excuse is that if they had to test it, it would slow them down. They also use the famous "It's not our job."
It's been in development for three years and we are currently trying to sell it to customers. One has already dropped us and the two remaining will probably leave soon. We get nightly builds and a weekly build that we are supposed to test each week. It takes at least half a day, sometimes two days, to even get it running.
It's also so big that it can't all be tested in every release (See this article, also in the Queue last month - http://www.acmqueue.org/modules.php?name=Content&
You're right on with the list you mentioned and we can't get any upper management here to listen to us. The previous poster isn't the only one who's missing out.
Needless to say, if you are approched to buy "The Next Generation Java POS" and it seems to have more bugs than it should, run as fast as you can. It's worse than you think. They don't call it a POS system for nothing.
or software engineering (whatever that is) this might be true, but software development is different from classic engineering. The money factor is not linear. To put more money into a software project would be more workers since you essentally have no other costs. But this rarely helps quality or schedule (Mythical man month et al).
"If you decrease time"
I said nothing about decreasing time, what I meant is that the presence of deadlines or gates in a project is important, because it forces a project to move from reflection to practice. It also helps the team to only do what they beleive is absolutley nessesary, and this promotes simplicity, which I beleive is essential to quality.
The hard part in project management is to keep the reflection/practice cycle in balance, and to enforce respect of the deadlines, but still be able to push them if nessecary.
[Perryman]
Q/A's job is to make sure everything is as it should be. A military example: Switches switch, screws are tight, paperwork is done correctly, no corrosion, no loose parts, etc etc. This applies the same way for this, analogously
I am betting one of two things:
1. You haven't seen it done well.
2. The effects of QA aren't always obvious.
I can give you a personal example. At a basic level, part of QA is making sure that you follow the processes that you say you are going to follow. Where I worked before, we said we were going to have a project plan for every release, even if it was a service pack release. It didn't have to be a massive one, as we might have for a full release, but it had to exist. This was to make sure that it didn't affect the schedules of other things that were going on. Well, as part of my QA duties I pointed out that we didn't have a project plan for a rather large service pack, and after discussing it with various managers, one was created.
Now I can see how you might say "big deal". But having that project plan probably saved a lot of hours of work for some people. If there is a project plan, it gets looked at, it gets updated, it gets watched, it gets a schedule made, and that schedule is worked in with the other schedules. Without all that, it would have been one of those "in our spare time" releases that always seemed to lead to trouble for everyone. Eventually, it would have become a hot issue and impacted all the other projects.
Not really an obvious or glamorous example, I know. But it definitely affected the quality of our products. Quality isn't just lack of bugs, it is meeting schedules and feature lists.
But this is also why I don't think that QA is necessary for every project or every company. Some just really don't require it. (and remember, I am not talking about testing here)
My beliefs do not require that you agree with them.
An observation. You say that it is not worth the effort to make HTML that everyone can read. And yet you criticize Microsoft for not making software that everyone can use. Aren't both decisions about quality? And in both cases isn't the issue not a matter of knowing enough about to get quality, but simply not always caring about it when the costs are born by others?
Quality is a matter of both delivering useful value and a matter of avoiding defects. You need a team/person that is responsible for both. The answer is to make the team responsible for quality, not to attach a QA team to a programming team that isn't doing satisfactory work.
"At a basic level, part of QA is making sure that you follow the processes that you say you are going to follow. Where I worked before, we said we were going to have a project plan for every release, even if it was a service pack release."
But what if your process didn't include a requirement for a project plan? Then you'd be fully compliant with your process, but you wouldn't have the advantages of the project plan.
Quality comes from doing the appropriate things for a particular project to make sure a product meets all the requirements. Following a generic or company-wide set of process rules may or may not increase the quality of a product.
I am not sure where you get the idea that I criticized Microsoft for not making universally usable software. The web page I put up is accessable to anyone that uses windows. The same applies to MS Word. Linux users have to use some other word processor.
My web page was put together to keep me amused and as a resource for looking for work. I don't know what costs might be born by others as a result.
Quality (or the lack of it) is an economic issue. If someone thought it was worth the cost to buy MS Windows just to read my web page, I would be very flattered but it is not likely to happen. The vast majority of PCs have Windows and MS IE.
Try to keep a focus on the topic which is the relationship between quality and testing. Its the message that is important, not the package. If it means a lot to you to look at my site, go visit a friend who has windows.
Quality comes from doing the appropriate things for a particular project to make sure a product meets all the requirements. Following a generic or company-wide set of process rules may or may not increase the quality of a product.
I understand where you are coming from, and I agree. But another part of QA is evaluating the process and improving it. You don't just take a set of processes that you didn't write and maintain and use them blindly. The process didn't require a project plan - but if the project manager didn't have one, it had to be documented why one wasn't necessary. It wasn't, which is why I simply stepped in and asked the question. You are right that you need to do the appropriate things for a particular project, and if that is going to happen it needs to be built into the process.
Everyone turns their nose up at process, but what is funny is that most people really like it when it is implemented correctly. The problem is that too many companies don't implement processes that work well, and they are too rigid. If they are working, and people just don't like it, then that is an education issue, not a process issue. It sounds all hoity-toity, but in reality all you have to do is create processes that work and that people will use, document them, and revisit them on a periodic basis and tweak them if necessary.
My beliefs do not require that you agree with them.
Where I am now, 'quality' means ISO-900x. It is a bureaucratic process for keeping things from happening.
Isn't a bureaucratic process DEFINED as something that keeps things from happening?