Slashdot Mirror
QA != Testing
gManZboy writes "Original author of Make and IBM Researcher, Stu Feldman has written an overview of what should be (but is sadly perhaps not) familiar ground to many Slashdotters: Quality Assurance. He argues that QA is not equivalent to 'testing', and also addresses the oft-experienced (apparent) conflict between QA-advocates and 'buisiness goals.'"
IME, Quality = Knowledgeable_Staff_On_Good_Salary + No_Deadlines. Unfortunately, this formula larglely is not compatible with business world. So, in the mean time, customers should be grateful if software has been _somehow_ tested and mostly works.
This is a sign that there was no quality assurance during the requirement gathering. Which probably means you were not actually starting your "QA process" , but were actually starting "testing".
and clever != good marks in exams
testing doesn't make the software any better but testing do find bugs which developers missed. Quality assurance is to make sure that the software is of good enough quality before release and testing does confirm the case.
I am harvesting funny/good quotes. Please help by putting them in your sigs
No.
What you have described is a large bug-hunting exercise.
QA is a process by which errors are supposed to be PREVENTED, not FOUND OUT.
That's why QA != Testing
Better QA == fewer bugs to find (it assures you are building quality)
Better Testing == more bugs found (it is, in fact, closer quality verification)
Every software company I've worked for, the QA department was always separate from Development. Then the classic mistake was always repeated... bring in QA at the tail end of a project and expect them to certify it and find all the bugs.
Madness!!
I think that the real issue here is the difference between code that works, and code that meets the business rules and need. Anyone can make good code, and have it compile and execute. The problem comes when that great code still doesn't fit the need that it was supposed to fill in the first place. This issue has two hurdles if it is to be overcome. First, are coders that have no business knowledge. Second, business pros that have no software development experience. The coders complain that they weren't given the proper details of the project, and the business guys complain that the coders know nothing about business. I think that all biz people need to take a basic programming course, and all coders need to take a business class. The gulf of poor communication between the two camps is quite large without it.
My
If you have good quality you can cut down or completely out testing. Think of it like a math equation. Y = f(x). Y is the output and f(x) is the input. If you control the inputs with no variation then the output will always be the same so no need to test it. Honda has done this with engines. They save money because they don't test every engine. Yet, all their engines always work.
Evolution or ID?
The writer talks about separating QA from the Development group. In our organization, this was a large part of the problem. First, there was a tendency for the development group to "throw it over the fence" and expect QA to find problems that the engineers couldn't be bothered to look for.
The QA staff, on the other hand, rarely had engineers of a sufficient caliber that had the insights to search for and find the most insidious problems. Not only that, they (QA) occupied the no-man's land between business users and development, understanding neither area with any clarity.
Yeah? Well I think you're overrated too.
If design is flawed, what should QA do?
...And more, see F. Brooks' "Mythical Man-Month" for example, or Alen Holub's "Rope Long Enough to Shoot Your Leg".
If docs are porly written, and incomplete, how does one decide what's bug an what's feature?
If the docs depict the program's behavior, not define it, what can QA do?
And yes, if everythign is done right from the beginning, the QA people would have enough time to do something except testing.
Of course, only third of the two ways to write bugless programs works...
WYSIWIG, but what you see might not be what you need
Can someone please forward this to the good old folks at Mozilla. The written QA requirements (on Mozilla) are so cursory that whole features have dropped off the map simply because nobody bothered to check to see if they still worked (i.e. in 1.4 multiobject drag and drop stopped working). Might also help the parsing engine, which continues to have kruft from the Netscape days (like how is interpretted as instead of as a single broken tag to be ignored [and you can use any tags you want, the second one can even contain parameters, allowing you to really annoy people by adding extra HTML that only works in Firefox/Mozilla, not IE/Opera]). Though really as Slashdot has reported before, Mozilla could really use a more robust and systematically tested parser just to avoid potential buffer overrun exploits.
Bottom line: OSS could get way ahead of commercial software simply by doing proper QA and unit testing (not just the UNIX "it seems to work" test, the "are out of range inputs properly detected or does the program just choke and die") on par with what the best commercial developers have been doing. Just because you have to do all this paperwork and repetive checking when working for "The Man", doesn't mean it's an evil thing that should be thrown out. Sometimes the man actually has some good ideas, even if he spends most of his time shouting about how you didn't dot your i s on that last flowchart.
" Imagine a complex program containing many code paths"
Mmmkay.
"QA spends a day to code a test suite which exercises every code path"
erm... "QA spends a day"
Yeah, right.
You do realise that a FULL code path test suite will, perforce, be LARGER than the source code it tests?
When doing QA, I used to start writing the test cases for software when the REQUIREMENTS document arrived, so that they were ready for use during the tail end of coding and for the unit testing. Its a BIG job.
And you design tests from the reqs, not from the code - how will you trap a completely missing boundary case, if you build tests from the source? Or the design?
Requirements drive source and test design, separately so that the assumptions in the former cannot pollute the latter.
That sounds very good. In theory.
Having worked in a CMM 3 company for a couple years, my opinions of the thing are quite different: CMM, and processes in general, are a tool that managers use to offload their work on the engineers.
We used to spend vast amounts of time peer reviewing all sorts of useless documents, making estimates for project planning, and so on, additionally to the architecture and coding work.
This didn't do anything at all for quality. Deadlines slipped like always (often more, because of the time lost to irrelevant stuff). Spec documents were just as Ground-Control-To-Major-Tom-like as usual.
It did, however, give the managers the warm fuzzy feeling that overcomes control freaks everywhere when they're sure they can track, number, file and index everything that goes on around them. Without having to do any actual work. Without even knowing the first thing about the product we were making (without CMM, a prerequisite for anyone attempting to write any sort of project plan).
One of our line managers admitted all of this quite openly, one of his favourite sayings was "Since we have processes, I can go home at four every day". We didn't. We got to stay till 8.
In my experience, CMM should be avoided like the plague. It's complete and utter waste of time, and encourages empty hierarchies.
Let's also not forget that the DoD has had a number of programs over the years that attempt to determine whether such methodologies work, and/or attempt to determine what the best methodology might be. Of course, everyone using such a methodology invariably reports that it works fantastically, either because they want the next deal, or because they want their particular methodology to be King of the Hill.
I worked for a DoD contractor for a while, so I've seen it from the inside -- and, I'd say that using DoD-funded development projects as a measure of anything is ludicrous. Years after my DoD experience, I remember interviewing for a lead hardware engineer. I needed a guy who could build a Z80-based microcontroller board. I had one tech to give him, that's it. And, I needed the board laid out and working in 4 months. I knew this was possible, because I had worked with plenty of hardware engineers who could do this in their sleep, with one layout and no rework. Remember, this is 4mhz, folks. Crosstalk? What crosstalk? Hell, armed with a book and help from the vendor in the form of boilerplate designs, even I could have taken a stab at it, and the last time I hacked hardware was years ago in a college course.
Anyway, this guy was from a large defense contractor, R******n. Turns out he was PART OF A TEAM that had built a Z80 CPU board over the last 18 months. His particular responsibility had been the DRAM circuit. According to him there were 20 other hardware engineers on the project. Yup, he said TWENTY. That's right. T-W-E-N-T-Y.
The $64,000 question is, what the heck was this guy doing for those 18 months? I was stunned. So was he, when he realized what was expected of him in the "real" world. I don't care how MIL-spec'd his board had to be, or how much vibration and radiation testing they had to do, or how many $22,000 toilets they had to flush it down to test it, 18 months and 20 people is ridiculous. Period.
I found someone else for the position. He built the board, delivered it ahead of schedule, and it worked fine. And while he was doing that, in parallel he designed and built another board for an RF hand-held. I guess he wouldn't have fit in at R******n. Nothing against R******n, though. Largest employer in the state. Love you guys. Keep everyone working.
The problem is, there are two motives to reach CMM 3;
a) It looks good to our customers.
b) It reduces our cost.
Companies that strive for motive A often will do their best to meet the requirements of CMM to the letter, without actually changing what they do on a day to day basis. "CMM says we need to have a baseline and configuration management for our code, so I want everyone to check their work into this new CVS thing, at least once a month", for example.
It's easy to "meet the letter" of CMM without at all meeting the intent. At my company, for example, there's a core group who is trying to push "scrum" as a software development methodology, and they make all kinds of bizzare claims that this is somehow consistent with CMM 3, pointing to specific wording within CMM, and making claims that such and such is equivalent to CMM, even if it doesn't quite meet it. Meanwhile, I try to envision a mission critical system like a 767, or a space shuttle, or an ambulance dispatch service produced with scrum, and it makes me afraid to go outdoors.
Some people are afraid of change.
"From what I've seen of QA, it is testing, just not in the "Testing" phase. It is having well-defined objects, interfaces, input ranges, output ranges, unit tests and so on to make sure that when you assemble everything together, it has few bugs left. Basicly, weeding them out at an earlier point in the process."
No offense, but you missed out:
Ensuring that the requirements the SW is built to match are complete / correct in the first place.
Ensuring that the SW is built in a way that is suitably efficient for the project.
Ensuring that the SW has at least been thought about in terms of being built for re-use.
Ensuring that there was at least some thought about "is there something already here that we could re-use or modify?"
Ensuring that the SW is built in a method that lends itself to maintenance and modification without tearing out of hair.
Ensuring that some form of profiling or metrication has been performed, in case the project as a whole needs optimisation (being able to look at the metrics for each unit speeds that first "where to optimise" pass SOOOO much)
Ensuring that throughout all those processes the correct feedback was fedback to the people who actually DID all those things you just ensured...
ALL of that is part of the QA for software development, very little of it actually involves testing the software does its job right...
1) deciding what you want to build, and deciding exactly (i.e., good specs);
Too often, I've stumbled across over-specified systems that, as a result, are delivered incredibly late. And then, because of time constraints, the whole project is de-scoped and bodged work-arounds are built so that functionality can be 'added later'.
At the design stage, politics often slows things down. I prefer the continuous approach: When you have enough design, start coding.
Did he inhale?
Having said that, many of the ideas that are in the CMM are good ideas. Each item has some justification, and can be helpful to generate better software systems. After all, the cost-plus contracts are based on justified costs, not just every cost that the contractor can dream up. There has to be some plausible justification for every cost item.
So, for non-military purposes, we can use the CMM model and associated literature. We just have to use them intelligently. We cannot blindly implement every detail, because that will make every project over-budget and late. And, we cannot blindly reject every CMM element just because we think the CMM is bloated and inefficient. We should instead consider which management methods and processes are actually appropriate for the current project at hand. The CMM books provide a good inventory of things that we might want to do. We just have to pick the items that are appropriate for the particular project, in the particular company and business environment.
Try 15. (In software and in manufacturing)
The most severe downfalls I've seen are when the marketing teams and development teams do not adequately communicate with the QA team.
I have to disagree, the most severe downfalls I've seen are when the ship to customer date isn't allowed to slip but the date for development submitting to test is. Test end up having to do a month's work in a week and there is no time to fix bugs. So it shouldn't be a surprise when the product ships with known, unfixed crtitcal bugs and the customer ends up cancelling the entire contract.
In my experience what you need to produce some sort of "quality" product is a sales team that doesn't over promise, senior management that are aware that you can't just tack on some testing at the end and it'll be alright, clear specs, and enough time to actually do the job.
Oh, and a test team that are commited to the role, not a bunch of wannabe programmers who see it as something to do for 3 months just to get a foot in the door.
Good communication helps, as does a good working relationship between teams but neither is entirely essential.
Having said the above, it's also my belief that the strongest impediment to producing high quality software is the end user license. As long as software producers can disclaim any and all responsibilities, fitness for purpose etc there is no incentive for them to do it right.