Slashdot Mirror
QA != Testing
gManZboy writes "Original author of Make and IBM Researcher, Stu Feldman has written an overview of what should be (but is sadly perhaps not) familiar ground to many Slashdotters: Quality Assurance. He argues that QA is not equivalent to 'testing', and also addresses the oft-experienced (apparent) conflict between QA-advocates and 'buisiness goals.'"
IME, Quality = Knowledgeable_Staff_On_Good_Salary + No_Deadlines. Unfortunately, this formula larglely is not compatible with business world. So, in the mean time, customers should be grateful if software has been _somehow_ tested and mostly works.
This is a sign that there was no quality assurance during the requirement gathering. Which probably means you were not actually starting your "QA process" , but were actually starting "testing".
and clever != good marks in exams
testing doesn't make the software any better but testing do find bugs which developers missed. Quality assurance is to make sure that the software is of good enough quality before release and testing does confirm the case.
I am harvesting funny/good quotes. Please help by putting them in your sigs
No, I'm saying that if Linux were to have a QA strategy before the next great project begins, then the product has the potential to be even better.
Why on this damn website will any criticism, no matter how true/justified be taken as the most vile bile ever spewed?
And why is any response to criticism met by words to the effect of "And, of course, YOU think that WINBLOW$ is better? Well let me tell you something, you . . . "?
No.
What you have described is a large bug-hunting exercise.
QA is a process by which errors are supposed to be PREVENTED, not FOUND OUT.
That's why QA != Testing
Better QA == fewer bugs to find (it assures you are building quality)
Better Testing == more bugs found (it is, in fact, closer quality verification)
Every software company I've worked for, the QA department was always separate from Development. Then the classic mistake was always repeated... bring in QA at the tail end of a project and expect them to certify it and find all the bugs.
Madness!!
I think that the real issue here is the difference between code that works, and code that meets the business rules and need. Anyone can make good code, and have it compile and execute. The problem comes when that great code still doesn't fit the need that it was supposed to fill in the first place. This issue has two hurdles if it is to be overcome. First, are coders that have no business knowledge. Second, business pros that have no software development experience. The coders complain that they weren't given the proper details of the project, and the business guys complain that the coders know nothing about business. I think that all biz people need to take a basic programming course, and all coders need to take a business class. The gulf of poor communication between the two camps is quite large without it.
My
If you have good quality you can cut down or completely out testing. Think of it like a math equation. Y = f(x). Y is the output and f(x) is the input. If you control the inputs with no variation then the output will always be the same so no need to test it. Honda has done this with engines. They save money because they don't test every engine. Yet, all their engines always work.
Evolution or ID?
A way to look at it is, there is a lump sum for a project. The amount of money (time is money) to produce a product. This amount is made up of base price for product plus rework (which is cost of poor quality). If you can minimize the cost of the rework you can either/or increase profits, cut costs (including time to get product out), improve the design as it is higher quality.
Evolution or ID?
The writer talks about separating QA from the Development group. In our organization, this was a large part of the problem. First, there was a tendency for the development group to "throw it over the fence" and expect QA to find problems that the engineers couldn't be bothered to look for.
The QA staff, on the other hand, rarely had engineers of a sufficient caliber that had the insights to search for and find the most insidious problems. Not only that, they (QA) occupied the no-man's land between business users and development, understanding neither area with any clarity.
Yeah? Well I think you're overrated too.
If design is flawed, what should QA do?
...And more, see F. Brooks' "Mythical Man-Month" for example, or Alen Holub's "Rope Long Enough to Shoot Your Leg".
If docs are porly written, and incomplete, how does one decide what's bug an what's feature?
If the docs depict the program's behavior, not define it, what can QA do?
And yes, if everythign is done right from the beginning, the QA people would have enough time to do something except testing.
Of course, only third of the two ways to write bugless programs works...
WYSIWIG, but what you see might not be what you need
I can see the problems this creates. Dev teams blame QA for being fussy; QA hate the dev team for releasing such junk.
We have to embrace test-centric development more. JUnit, CppUnit, PyUnit, they make it easy to write tests. But to convince the developers to write good tests, that is another matter. I often find it is harder to write a decent test for something complex than it is to implement the complex thing, but without that test, how do you know it works?
Then you just have to convince everyone to stick to what they've agreed to when they ask you to change everything halfway through coding!
Can someone please forward this to the good old folks at Mozilla. The written QA requirements (on Mozilla) are so cursory that whole features have dropped off the map simply because nobody bothered to check to see if they still worked (i.e. in 1.4 multiobject drag and drop stopped working). Might also help the parsing engine, which continues to have kruft from the Netscape days (like how is interpretted as instead of as a single broken tag to be ignored [and you can use any tags you want, the second one can even contain parameters, allowing you to really annoy people by adding extra HTML that only works in Firefox/Mozilla, not IE/Opera]). Though really as Slashdot has reported before, Mozilla could really use a more robust and systematically tested parser just to avoid potential buffer overrun exploits.
Bottom line: OSS could get way ahead of commercial software simply by doing proper QA and unit testing (not just the UNIX "it seems to work" test, the "are out of range inputs properly detected or does the program just choke and die") on par with what the best commercial developers have been doing. Just because you have to do all this paperwork and repetive checking when working for "The Man", doesn't mean it's an evil thing that should be thrown out. Sometimes the man actually has some good ideas, even if he spends most of his time shouting about how you didn't dot your i s on that last flowchart.
"I mentioned it in another post, but my dad has a good web site that deals with quality issues (IE only, unfortunately)."
I'm sure the irony of that statement is not lost on anyone. A site, giving advice on good quality, is itself a quality disaster. You'll understand if I don't take his credentials to heart.
" Imagine a complex program containing many code paths"
Mmmkay.
"QA spends a day to code a test suite which exercises every code path"
erm... "QA spends a day"
Yeah, right.
You do realise that a FULL code path test suite will, perforce, be LARGER than the source code it tests?
When doing QA, I used to start writing the test cases for software when the REQUIREMENTS document arrived, so that they were ready for use during the tail end of coding and for the unit testing. Its a BIG job.
And you design tests from the reqs, not from the code - how will you trap a completely missing boundary case, if you build tests from the source? Or the design?
Requirements drive source and test design, separately so that the assumptions in the former cannot pollute the latter.
That sounds very good. In theory.
Having worked in a CMM 3 company for a couple years, my opinions of the thing are quite different: CMM, and processes in general, are a tool that managers use to offload their work on the engineers.
We used to spend vast amounts of time peer reviewing all sorts of useless documents, making estimates for project planning, and so on, additionally to the architecture and coding work.
This didn't do anything at all for quality. Deadlines slipped like always (often more, because of the time lost to irrelevant stuff). Spec documents were just as Ground-Control-To-Major-Tom-like as usual.
It did, however, give the managers the warm fuzzy feeling that overcomes control freaks everywhere when they're sure they can track, number, file and index everything that goes on around them. Without having to do any actual work. Without even knowing the first thing about the product we were making (without CMM, a prerequisite for anyone attempting to write any sort of project plan).
One of our line managers admitted all of this quite openly, one of his favourite sayings was "Since we have processes, I can go home at four every day". We didn't. We got to stay till 8.
In my experience, CMM should be avoided like the plague. It's complete and utter waste of time, and encourages empty hierarchies.
Let's also not forget that the DoD has had a number of programs over the years that attempt to determine whether such methodologies work, and/or attempt to determine what the best methodology might be. Of course, everyone using such a methodology invariably reports that it works fantastically, either because they want the next deal, or because they want their particular methodology to be King of the Hill.
I worked for a DoD contractor for a while, so I've seen it from the inside -- and, I'd say that using DoD-funded development projects as a measure of anything is ludicrous. Years after my DoD experience, I remember interviewing for a lead hardware engineer. I needed a guy who could build a Z80-based microcontroller board. I had one tech to give him, that's it. And, I needed the board laid out and working in 4 months. I knew this was possible, because I had worked with plenty of hardware engineers who could do this in their sleep, with one layout and no rework. Remember, this is 4mhz, folks. Crosstalk? What crosstalk? Hell, armed with a book and help from the vendor in the form of boilerplate designs, even I could have taken a stab at it, and the last time I hacked hardware was years ago in a college course.
Anyway, this guy was from a large defense contractor, R******n. Turns out he was PART OF A TEAM that had built a Z80 CPU board over the last 18 months. His particular responsibility had been the DRAM circuit. According to him there were 20 other hardware engineers on the project. Yup, he said TWENTY. That's right. T-W-E-N-T-Y.
The $64,000 question is, what the heck was this guy doing for those 18 months? I was stunned. So was he, when he realized what was expected of him in the "real" world. I don't care how MIL-spec'd his board had to be, or how much vibration and radiation testing they had to do, or how many $22,000 toilets they had to flush it down to test it, 18 months and 20 people is ridiculous. Period.
I found someone else for the position. He built the board, delivered it ahead of schedule, and it worked fine. And while he was doing that, in parallel he designed and built another board for an RF hand-held. I guess he wouldn't have fit in at R******n. Nothing against R******n, though. Largest employer in the state. Love you guys. Keep everyone working.
Cost, Time and Quality and you can only have 2 out of the 3.
For instance, you can have a project done fast and cheap but the quality will be lacking. Or you can have a project done correctly and quickly but it will cost you a fortune!
QA is part of that "having it done correctly" piece that most companies tend to cut out. Most companies can only grasp the Time and Cost factor and fail at the whole "Quality" component when doing pre-project analysis. I do not have enough fingers to count the projects I've witnessed that have been running behind schedule (which is usually a BS timeframe to begin with) that decide to cut time from the QA process to make the deadline.
The reasons for "falling behind schedule" and all those other fun things that go into bad project management have been discussed before so I won't even mention it.
The problem is, there are two motives to reach CMM 3;
a) It looks good to our customers.
b) It reduces our cost.
Companies that strive for motive A often will do their best to meet the requirements of CMM to the letter, without actually changing what they do on a day to day basis. "CMM says we need to have a baseline and configuration management for our code, so I want everyone to check their work into this new CVS thing, at least once a month", for example.
It's easy to "meet the letter" of CMM without at all meeting the intent. At my company, for example, there's a core group who is trying to push "scrum" as a software development methodology, and they make all kinds of bizzare claims that this is somehow consistent with CMM 3, pointing to specific wording within CMM, and making claims that such and such is equivalent to CMM, even if it doesn't quite meet it. Meanwhile, I try to envision a mission critical system like a 767, or a space shuttle, or an ambulance dispatch service produced with scrum, and it makes me afraid to go outdoors.
Some people are afraid of change.
"A good specification is one, but if you are so inclined a static prototype can often achieve as much if not more from the customer's perspective."
In that case you have a specification! In the form of your static prototype.
Nowehere does it say that a specification HAS to be solely a written document...
People should not be afraid of their governments - Governments should be afraid of their people.
"From what I've seen of QA, it is testing, just not in the "Testing" phase. It is having well-defined objects, interfaces, input ranges, output ranges, unit tests and so on to make sure that when you assemble everything together, it has few bugs left. Basicly, weeding them out at an earlier point in the process."
No offense, but you missed out:
Ensuring that the requirements the SW is built to match are complete / correct in the first place.
Ensuring that the SW is built in a way that is suitably efficient for the project.
Ensuring that the SW has at least been thought about in terms of being built for re-use.
Ensuring that there was at least some thought about "is there something already here that we could re-use or modify?"
Ensuring that the SW is built in a method that lends itself to maintenance and modification without tearing out of hair.
Ensuring that some form of profiling or metrication has been performed, in case the project as a whole needs optimisation (being able to look at the metrics for each unit speeds that first "where to optimise" pass SOOOO much)
Ensuring that throughout all those processes the correct feedback was fedback to the people who actually DID all those things you just ensured...
ALL of that is part of the QA for software development, very little of it actually involves testing the software does its job right...
1) deciding what you want to build, and deciding exactly (i.e., good specs);
Too often, I've stumbled across over-specified systems that, as a result, are delivered incredibly late. And then, because of time constraints, the whole project is de-scoped and bodged work-arounds are built so that functionality can be 'added later'.
At the design stage, politics often slows things down. I prefer the continuous approach: When you have enough design, start coding.
Did he inhale?
In that case you have a specification! In the form of your static prototype.
A prototype and a specification do not contain the same information. A prototype consists of a single concrete instance of the thing a specification describes. It contains more information than the specification in some respects (the concrete design choices the implementor has made to fill in the gaps the specification is silent on) but more importantly it is also missing information that is absolutely required in a specification.
For example: What are the tolerances on particular processes? What are the scaling and responsiveness requirements? And so on.
Nothing that does not capture this kind of information is a specification, and so far as I know only a document can capture this kind of information.
Don't get me wrong: prototypes are useful. But they are not specifications, because they do not contain the all of the information that a specification must contain.
--Tom
Blasphemy is a human right. Blasphemophobia kills.
I've been on various QA/Testing (no, they are not the same, but my team performs them both) teams for the better part of three years now.
... and the defect is thrown to the wind, because marketing refuses to update their first (and apparently final) draft of specs to accomodate the new feature that would make the product much more palletable to a prospective consumer.
/. We need some more comments from people in the industry, whose job titles actually include the phrase "Quality Assurance" or "Software Testing" or something.
;D
The most severe downfalls I've seen are when the marketing teams and development teams do not adequately communicate with the QA team.
In order for QA to be successful, the QA team must be aware of and in tune with marketing's idea of how the product is envisioned to perform.
Nothing is more frustrating than logging a bug that is a critical flaw in the operation of a product to get the developer response "spec discrepancy"
I was disappointed to enter this thread and see stabs at Microsoft testing and other mindless drivel I'm used to seeing at
Of course going AC here, lest my co-workers lynch me.
Cost, Time and Quality and you can only have 2 out of the 3.
Every time I hear this I cringe, because it is complete crap. On a small level or to someone who has no quality assurance practices in place, it appears to be true, but for large projects/products quality assurance is inversely proportional to cost and time precisely because it cuts down on the amount of time required to develop and test a product to its specifications. Quality assurance procedures will help ensure that errors are caught early on in the project lifecycle and develop standard procedures for project development which may have an initial hit on timelines, but in the end save time. The really nice thing about good quality assurance procedures is that they generally get easier as people get familiar with them and time goes on.
For instance, you can have a project done fast and cheap but the quality will be lacking. Or you can have a project done correctly and quickly but it will cost you a fortune!
I'm assuming that when you say quickly here you are thinking of thowing more manpower at the solution because, as the time to complete a project increases the cost increases and vice versa. Quality does not mean features and flawless software. It means that your client gets what they ask for on budget and it works like specified.
Cost is directly proportional to time, quality assurace might add some time to the project, but it has been my experience that the time it adds is usually offset during testing and not having to add features that a client expects during / after internal testing.
10: PRINT "Everything old is new again."
20: GOTO 10
In reality, Windows rose to dominance because it was an inferior product brought to market at the right time.
businesses chose to use windows because they wouldn't have to rewrite all their legacy/mainframe/in-house apps, because underneath it all, you still had DOS. and if your app didn't play nice with windows task-switching of DOS apps, you could exit wondows entirely, and have essentially a clean slate for the custom app to use.
Having said that, many of the ideas that are in the CMM are good ideas. Each item has some justification, and can be helpful to generate better software systems. After all, the cost-plus contracts are based on justified costs, not just every cost that the contractor can dream up. There has to be some plausible justification for every cost item.
So, for non-military purposes, we can use the CMM model and associated literature. We just have to use them intelligently. We cannot blindly implement every detail, because that will make every project over-budget and late. And, we cannot blindly reject every CMM element just because we think the CMM is bloated and inefficient. We should instead consider which management methods and processes are actually appropriate for the current project at hand. The CMM books provide a good inventory of things that we might want to do. We just have to pick the items that are appropriate for the particular project, in the particular company and business environment.
the reason that "testing must take up the slack" could be because of lack of reasonable deadlines/gates in the earlier development stages?
The tools you are looking for exist, but they're not cheap and they need someone to create test cases for them and run them. I assume it's a web app you're talking about? Really, you need to hire a professional and give them professional tools. There's no good way to test your app cheaply. What you really need is a copy of Mercury Interactive's Astra or Segue Silk and someone to run it for you.
If you have to do it cheaply and by yourself, good luck and have fun writing those tests. Do a web search and have fun evaluating tools and writing tests. Here is the first hit I got on google. I've never used it but maybe you could look into ieUnit.
Oh, and my advice is to find a job closer to home. That commute will literally kill you. No, really, I mean it, you'll die.
if QA runs the show: it never ships, as there are always improvements to be made. Always.
Then you're doing QA wrong.
One of the big parts of having a spec and a QA process is to know WHEN TO STOP.
When you get the function of a part right, and the tests have been run and show it's right, you MOVE ON. You only come back to it if the debugging of a later part reveals a hidden bug that the earlier tests missed (or couldn't test without the availability of the later part).
When you've moved on from the last part you're DONE.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way