Slashdot Mirror
Should Developers Do All Their Own QA? (itnews.com.au)
An anonymous reader quotes IT News:
Fashion retailer The Iconic is no longer running quality assurance as a separate function within its software development process, having shifted QA responsibilities directly onto developers... "We decided: we've got all these [developers] who are [coding] every day, and they're testing their own work -- we don't need a second layer of advice on it," head of development Oliver Brennan told the New Relic FutureStack conference in Sydney last week. "It just makes people lazy..."
Such a move has the obvious potential to create problems should a developer drop the ball; to make sure the impact of any unforeseen issues is minimised for customers, The Iconic introduced feature toggles -- allowing developers to turn off troublesome functionality without having to deploy new code. Every new feature that goes into production must now sit behind one of these toggles, which dictates whether a user is served the new or old version of the feature in question. The error rates between the new and old versions are then monitored for any discrepancies... While Brennan is no fan of "people breaking things", he argues moving fast is more beneficial for customers.
"If our site is down now, people will generally come back later," Brennan adds, and the company has now moved all of its QA workers into engineering roles.
Such a move has the obvious potential to create problems should a developer drop the ball; to make sure the impact of any unforeseen issues is minimised for customers, The Iconic introduced feature toggles -- allowing developers to turn off troublesome functionality without having to deploy new code. Every new feature that goes into production must now sit behind one of these toggles, which dictates whether a user is served the new or old version of the feature in question. The error rates between the new and old versions are then monitored for any discrepancies... While Brennan is no fan of "people breaking things", he argues moving fast is more beneficial for customers.
"If our site is down now, people will generally come back later," Brennan adds, and the company has now moved all of its QA workers into engineering roles.
Sort of. QA does development, but not on the project, they develop automated testing scripts and the like. The skill sets are different. A developer should do basic testing, but the developer is way too close to the code and is very likely to miss things. I see developers who assume no testing is needed, it was an obvious fix, and then it turns out to have bugs. Developers are often not good at integration testing, because they may have troubles seeing the big picture of how all the parts fit together.
QA can read the product's code certainly, they can be involved in the code reviews, but they shouldn't develop that code either. For a good QA person, testing, creating the test plans from the functional specs, automating the tests, and so forth, is a full time job in itself. In a lot of ways it is harder than development.
Every time I have seem companies try to merge two groups or departments together it has not turned out well. And this cost saving measure won't turn out well either.
You need a series of gates and software/firmware/hardware has to pass through all those gates. The later a bug is found the more expensive it is to fix it. When the customer finds a bug it can be catastrophic; as in layoffs are going to happen because of loss of revenue. So find the bugs early, and make sure you've got all the gates set up.
- Developer must unit test the code - developer must to the basic test of flipping it on, making sure that something happens and that it does what the developer thinks it does. Finding and fixing a bug here is very cheap.
- Other developers must code review the changes; spread out the knowledge to more than one person, have other sets of eyes looking at the code, let someone point out where the clever change is breaking the design or violating preconditions from elsewhere in the project.
- QA must then test the feature; positive and negative tests. Run it for long periods, try to break it, compare it one by one to each requirement and specification. This can happen before all features are complete.
- QA then tests the integration; not just one feature but all of the features, so it's done once there's a code freeze for the release. Maybe it repeats some of the earlier tests, but it should run through a full regression, use it like a customer would use it.
- If it's applicable, do a full system test. End to end, from hardware board to back end server. Test different combinations. Finding a bug here is expensive; but it's still less expansive than having the customers find the bug.
If you still find bugs after that, then review how the bug slipped past, fix the gates, and do better next time.
I've been a software engineer, a QA engineer, traditional IT SysAdmin, and have now found my niche in SysOps/DevOps/SRE/whatever-we're-calling-it-this-week. And good lords of Kobol, QA is absolutely necessary. The usual reasons definitely apply. A separate pair of eyes will find things you've been blind to because you're too close to your own work. Building things and breaking things are different skillsets and different mindsets. There will be edge cases that even the spec didn't anticipate. Blah blah blah.
But try being on the operations front line some time, and you'll REALLY appreciate a good QA team. Right now, I'm responsible for multiple products from teams that, due to acquisitions and re-orgs and different companies doing things differently, some have dedicated QA and some let the developers test their own code. I see the errors and failures before anyone else. And the QA'd stuff invariably causes me less grief than the "we trust the developers" stuff. Hell, if I had my way, we'd have QA vet ops changes and updates too; because I'm under no illusions of infallibility myself.
Imagine all the people...
Developers should absolutely have unit and integration tests and only deliver a product to QA that they think will be immediately passed onto the customer. But within many organizations this just doesn't make people happy, and unhappy people make bad decisions. QA can give a seal of approval that gets better buy in within the company. This should not be the way it is but that's life.
Lastly, a good QA will look to see that what was produced is what was actually asked for and promised. This is no small thing, because many programming teams can lose sight of what the contract actually says. "But it's better" is something that should be negotiated, not just delivered.
But that said, QA can be a major road block. It is not uncommon that you throw things over the fence only to find that QA has hyper focused on something that isn't a requirement. For instance you might have some system that does echo location to find the distance to a wall. Your requirements call for one successful echo location every 10 seconds. You fire out 10 per second and around 50% fail. The reality is that you are getting roughly 5 ranges per second and never will you miss your target of one every 10 seconds. Yet QA gets all wound up about the 50% failure rate and reports that the product has a 50% failure rate. This is where you begin convincing the accountants that the QA department needs a head count reduction.
In aerospace, independent verification and validation is a critical part of the regulatory process. Every change has a reason, every reason is documented, and every change is reviewed by someone else.
[
Should Developers Do All Their Own QA?
If you can do that without deleterious consequences, then your work is trivial and/or not overtly complex.
Otherwise, such a thing is insane, specially if you work in developing actual products (shrink-wrapped software or software bundled in hardware), or complex services. ) And let us not even mention software related to avionics, weapons or medical devices (where QA is done by a separate organization to remove bias.)
The primary concern is always bias, bias that come from first-hand knowledge of a thing (or assumptions about a thing.)
It is a lot easier and effective to have a 3rd party dedicated to methodical black box and end-to-end integration/UA testing than to get a developer to test his own creation to the breaking point.
There is a reason why organization hire external teams to perform penetration testing of their infrastructure. He who knows nothing of the internals a-priori is the perfect person to black-box test the shit out of something with nothing but specs.