Slashdot Mirror
QA != Testing
gManZboy writes "Original author of Make and IBM Researcher, Stu Feldman has written an overview of what should be (but is sadly perhaps not) familiar ground to many Slashdotters: Quality Assurance. He argues that QA is not equivalent to 'testing', and also addresses the oft-experienced (apparent) conflict between QA-advocates and 'buisiness goals.'"
IME, Quality = Knowledgeable_Staff_On_Good_Salary + No_Deadlines. Unfortunately, this formula larglely is not compatible with business world. So, in the mean time, customers should be grateful if software has been _somehow_ tested and mostly works.
Quality assurance is a process that runs through the entire project, testing is a component of that process.
When building software there is a tendency to lump quality assurance and testing together precipitously at the end of a project. The distinction that is made in this article is an important one, true quality and successful projects are obtained by having quality assurance as a project long process. Then you have quality assurance during requirements, design, development and yes even testing.
Let me be the first to quote Linus Torvalds: "Testing? What's that? If it compiles, it is good, if it boots up it is perfect."
8 of 13 people found this answer helpful. Did you?
This is a sign that there was no quality assurance during the requirement gathering. Which probably means you were not actually starting your "QA process" , but were actually starting "testing".
..Slashdot editors could apply QA to the spelling in posted stories?
It's "business", not "buisiness".
and clever != good marks in exams
testing doesn't make the software any better but testing do find bugs which developers missed. Quality assurance is to make sure that the software is of good enough quality before release and testing does confirm the case.
I am harvesting funny/good quotes. Please help by putting them in your sigs
testing can only prove the presence of an error, not its absence.
On another note, QA and QM methodes may sound incredibly dull and based upon "duh - how else should I do this, dumbass?", but are in fact highly sophisticated.
Not because they are not readical new, but because they are radically in their consistency. Think of something, and its error and faults, then of their causes, and their effects and impacts. Prefferably add fault probabilities, too. Then start over again.
It is constant feedback throughout the whole design process that is most important.
Powerful is he who overpowers his temptations.
The writer talks about separating QA from the Development group. In our organization, this was a large part of the problem. First, there was a tendency for the development group to "throw it over the fence" and expect QA to find problems that the engineers couldn't be bothered to look for.
The QA staff, on the other hand, rarely had engineers of a sufficient caliber that had the insights to search for and find the most insidious problems. Not only that, they (QA) occupied the no-man's land between business users and development, understanding neither area with any clarity.
Yeah? Well I think you're overrated too.
There are two parts to quality. The second part of the IEEE definition is "The degree to which a system, component, or process meets customer or user needs or expectations".
Although Feldman leaves out the second part (I believe it comes from another standard), he alludes to its importance in his discussion of how stringent QA must be, indicating that software for different purposes will have different quality requirements according to the needs of its users.
Quality Assurance is not possible in the absence of requirements and specifications. Although we (the company I work for) often receive requirements with minimal detail, we have addressed the quality problem by writing a (relatively) detailed specification up front, and presenting it to the customer. Effectively we're saying "this is what you're going to get for your money, okay?". It's just prudent practice, but it gives us a goal and a way to achieve quality (by both definitions).
You can find more on the combining the technical and business approaches to quality in my essay The Quality Gap.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
Yeah, I use it and am in a certified team. It's vastly overrated, and no substitue at all for people who know what they're doing. It might complement people who know what they're doing, but then such people would have come up with their own valid processes anyway, hence your initial correlation.
And it's hardly helped the US military come in on time and under-budget, now has it?
!
Cheers,
Ian
I used to work as a QA person.
In our company QA was a separate organisation, for 3 simple reasons;
1 - you are auditing and commenting on other people's work, not in a peer review "did you think about doing it like this" way, but in a "That is not acceptable; redo" way. Close colleagues in a department are NOT suitable for that role; you cannot be expected to say that about the person in the next cubicle's work, whereas a department with that as their job will be accepted when they do it.
2 - Keeping up to date on the quality requirements, combined with performing your live QA duties for the engineering department was a full time job. Or at least, it certainly was if the company wanted to keep its ISO9001 certification.
3 - Its a case of the buck stopping here. In our company project proposals, requirements and plans HAD to be signed off by QA before the funding got released for the project. At the same time, due to our doing telecoms stuff, we had a legal responsibility to sign off that the EMC conformity, physical safety and electrical safety tests had been conducted properly and passed. (and that meant constantly checking updates of the various national standards to ensure the company standards used the strictest requirements in each case). Random engineer is not good enough. (you have to have passed the right courses to audit each various section, need to be a qualified ISO9001 auditor to do the internal audits for that etc)
Professional QA is a full and seperate job. (but I did get to play with the 20KV discharge equipment!)
If design is flawed, what should QA do?
...And more, see F. Brooks' "Mythical Man-Month" for example, or Alen Holub's "Rope Long Enough to Shoot Your Leg".
If docs are porly written, and incomplete, how does one decide what's bug an what's feature?
If the docs depict the program's behavior, not define it, what can QA do?
And yes, if everythign is done right from the beginning, the QA people would have enough time to do something except testing.
Of course, only third of the two ways to write bugless programs works...
WYSIWIG, but what you see might not be what you need
For good reading on the design/requirements problem, I recommend Alan Cooper's The Inmates Are Running the Asylum. Talks a lot about how products can meet all their requirements and yet still fail because the requirements weren't right to begin with.
Can someone please forward this to the good old folks at Mozilla. The written QA requirements (on Mozilla) are so cursory that whole features have dropped off the map simply because nobody bothered to check to see if they still worked (i.e. in 1.4 multiobject drag and drop stopped working). Might also help the parsing engine, which continues to have kruft from the Netscape days (like how is interpretted as instead of as a single broken tag to be ignored [and you can use any tags you want, the second one can even contain parameters, allowing you to really annoy people by adding extra HTML that only works in Firefox/Mozilla, not IE/Opera]). Though really as Slashdot has reported before, Mozilla could really use a more robust and systematically tested parser just to avoid potential buffer overrun exploits.
Bottom line: OSS could get way ahead of commercial software simply by doing proper QA and unit testing (not just the UNIX "it seems to work" test, the "are out of range inputs properly detected or does the program just choke and die") on par with what the best commercial developers have been doing. Just because you have to do all this paperwork and repetive checking when working for "The Man", doesn't mean it's an evil thing that should be thrown out. Sometimes the man actually has some good ideas, even if he spends most of his time shouting about how you didn't dot your i s on that last flowchart.
That sounds very good. In theory.
Having worked in a CMM 3 company for a couple years, my opinions of the thing are quite different: CMM, and processes in general, are a tool that managers use to offload their work on the engineers.
We used to spend vast amounts of time peer reviewing all sorts of useless documents, making estimates for project planning, and so on, additionally to the architecture and coding work.
This didn't do anything at all for quality. Deadlines slipped like always (often more, because of the time lost to irrelevant stuff). Spec documents were just as Ground-Control-To-Major-Tom-like as usual.
It did, however, give the managers the warm fuzzy feeling that overcomes control freaks everywhere when they're sure they can track, number, file and index everything that goes on around them. Without having to do any actual work. Without even knowing the first thing about the product we were making (without CMM, a prerequisite for anyone attempting to write any sort of project plan).
One of our line managers admitted all of this quite openly, one of his favourite sayings was "Since we have processes, I can go home at four every day". We didn't. We got to stay till 8.
In my experience, CMM should be avoided like the plague. It's complete and utter waste of time, and encourages empty hierarchies.
in some (many, most?) cases QA becomes the definitive answer to what the requirements actually are. In some cases the requirements are not very precise, so it is up to the developer to translate them into the code and it is up to the QA to decide whether the developers' interpretation is good enough for the purposes of the system. In reality if you have a QA department, the QA defined test-cases become the guide to how the program behaves under various conditions. These test-cases define the system better than the requirements in some (many, most?) cases.
You can't handle the truth.
Let's also not forget that the DoD has had a number of programs over the years that attempt to determine whether such methodologies work, and/or attempt to determine what the best methodology might be. Of course, everyone using such a methodology invariably reports that it works fantastically, either because they want the next deal, or because they want their particular methodology to be King of the Hill.
I worked for a DoD contractor for a while, so I've seen it from the inside -- and, I'd say that using DoD-funded development projects as a measure of anything is ludicrous. Years after my DoD experience, I remember interviewing for a lead hardware engineer. I needed a guy who could build a Z80-based microcontroller board. I had one tech to give him, that's it. And, I needed the board laid out and working in 4 months. I knew this was possible, because I had worked with plenty of hardware engineers who could do this in their sleep, with one layout and no rework. Remember, this is 4mhz, folks. Crosstalk? What crosstalk? Hell, armed with a book and help from the vendor in the form of boilerplate designs, even I could have taken a stab at it, and the last time I hacked hardware was years ago in a college course.
Anyway, this guy was from a large defense contractor, R******n. Turns out he was PART OF A TEAM that had built a Z80 CPU board over the last 18 months. His particular responsibility had been the DRAM circuit. According to him there were 20 other hardware engineers on the project. Yup, he said TWENTY. That's right. T-W-E-N-T-Y.
The $64,000 question is, what the heck was this guy doing for those 18 months? I was stunned. So was he, when he realized what was expected of him in the "real" world. I don't care how MIL-spec'd his board had to be, or how much vibration and radiation testing they had to do, or how many $22,000 toilets they had to flush it down to test it, 18 months and 20 people is ridiculous. Period.
I found someone else for the position. He built the board, delivered it ahead of schedule, and it worked fine. And while he was doing that, in parallel he designed and built another board for an RF hand-held. I guess he wouldn't have fit in at R******n. Nothing against R******n, though. Largest employer in the state. Love you guys. Keep everyone working.
The problem is, there are two motives to reach CMM 3;
a) It looks good to our customers.
b) It reduces our cost.
Companies that strive for motive A often will do their best to meet the requirements of CMM to the letter, without actually changing what they do on a day to day basis. "CMM says we need to have a baseline and configuration management for our code, so I want everyone to check their work into this new CVS thing, at least once a month", for example.
It's easy to "meet the letter" of CMM without at all meeting the intent. At my company, for example, there's a core group who is trying to push "scrum" as a software development methodology, and they make all kinds of bizzare claims that this is somehow consistent with CMM 3, pointing to specific wording within CMM, and making claims that such and such is equivalent to CMM, even if it doesn't quite meet it. Meanwhile, I try to envision a mission critical system like a 767, or a space shuttle, or an ambulance dispatch service produced with scrum, and it makes me afraid to go outdoors.
Some people are afraid of change.
"From what I've seen of QA, it is testing, just not in the "Testing" phase. It is having well-defined objects, interfaces, input ranges, output ranges, unit tests and so on to make sure that when you assemble everything together, it has few bugs left. Basicly, weeding them out at an earlier point in the process."
No offense, but you missed out:
Ensuring that the requirements the SW is built to match are complete / correct in the first place.
Ensuring that the SW is built in a way that is suitably efficient for the project.
Ensuring that the SW has at least been thought about in terms of being built for re-use.
Ensuring that there was at least some thought about "is there something already here that we could re-use or modify?"
Ensuring that the SW is built in a method that lends itself to maintenance and modification without tearing out of hair.
Ensuring that some form of profiling or metrication has been performed, in case the project as a whole needs optimisation (being able to look at the metrics for each unit speeds that first "where to optimise" pass SOOOO much)
Ensuring that throughout all those processes the correct feedback was fedback to the people who actually DID all those things you just ensured...
ALL of that is part of the QA for software development, very little of it actually involves testing the software does its job right...
One of my main Day Jobs up to a few years ago was working in QA for Major Computer and Software Manufacturers.
The idea of:
Testing = finding bugs
QA = determining which features are required, and whether or not they work as intended for the end-user.
Is fine in theory, but rarely happens in practice. It usually ends up that QA is ignored and conflated into bug testing. And even then, it often doesn't matter.
Example: I was working on a team that developed an Important Piece of Software That Is Very Popular These Days. We Had No Specification.
None. After some terrifying meetings with the CEO, we somehow brought it to a 1.0 release. I didn't want to have to go through that little nightmare again, so at the 1.0 post mortem meeting, I asked "So, we built 1.0 without a spec - what exactly are we going to do next? What is the Specification for 2.0?"
The lead programmer looked right at me and said "The Spec for 2.0 is 1.0."
We had shipped 1.0 with over 850 bugs, with half a dozen known (if somewhat obscure) crashing bugs, and with several features "turned off" because we couldn't get them to work.
At that point I knew I had to get the fuck out of there. I wasn't going to spend over 2 hours a day driving just to help this rolling train wreck. I left as we shipped 2.0.
From there I went to a company That Was Also Extremely Famous (but now defunct) where QA was more of an expensive after thought. they hired a great team, but the engineers were so disjointed that the product kept changing every other month.
The stress level was so high at that company, of the 120 employees, half a dozen attempted suicide in the 9 months I worked there. At one point, there was such a row in basic engineering philosophy, two of the main programmers got into a fist fight. When the money dried up, we all got laid off.
We can go on and on about how important QA is, but the fact is, we're in a business that makes products, and when the business is more than a dozen people jammed in a garage or airless office space, the products tend to be driven by marketing droids. Left to their own devices, Engineers will produce complex objects that don't necessarily work or fulfil a worthwhile function, or do it in a way that is elegant and useful. Left to QA, the product never gets out the door, because Software Engineering *ISN'T*. SE is more like knitting or quilt making than an Engineered Science. Bridge Builders use Engineers - they have books full of equations that have been proven over the years and they use these solid tested things in a creative way to solve a problem : how to get from here to there.
When a bridge fails, or a building collapses, they just look at the evidence and compare it to the known working materials sciences, engineering principles, etc. and figure out how it failed.
With Software everything is written in a series of codes - at the machine level, we know what they do. But once you get into the meat and potatoes of software development, it all gets very wiggly very quickly. That's why TESTING is needed. And QA should be brought in even before the coding begins - when the UI is being developed, when the notion of the application's purpose and methods are being developed.
But, as noted above: if QA runs the show: it never ships, as there are always improvements to be made. Always.
So, you have the maketing droids who have the ear of the business managers who then set arbitrary and insane deadlines. The result? QA can't touch everything, or they conspire with Engineering that some sections are "good enough" and they let it go, so they can focus resources on testing problem areas, in order to meet the absurd deadline.
The end result is always the same: The public gets buggy software.
The only question is : How Buggy Is It?
They flip the crank, they do the late nights, they get the product out. QA and Eng do their littl