Slashdot Mirror
Should Developers Do All Their Own QA? (itnews.com.au)
An anonymous reader quotes IT News:
Fashion retailer The Iconic is no longer running quality assurance as a separate function within its software development process, having shifted QA responsibilities directly onto developers... "We decided: we've got all these [developers] who are [coding] every day, and they're testing their own work -- we don't need a second layer of advice on it," head of development Oliver Brennan told the New Relic FutureStack conference in Sydney last week. "It just makes people lazy..."
Such a move has the obvious potential to create problems should a developer drop the ball; to make sure the impact of any unforeseen issues is minimised for customers, The Iconic introduced feature toggles -- allowing developers to turn off troublesome functionality without having to deploy new code. Every new feature that goes into production must now sit behind one of these toggles, which dictates whether a user is served the new or old version of the feature in question. The error rates between the new and old versions are then monitored for any discrepancies... While Brennan is no fan of "people breaking things", he argues moving fast is more beneficial for customers.
"If our site is down now, people will generally come back later," Brennan adds, and the company has now moved all of its QA workers into engineering roles.
Such a move has the obvious potential to create problems should a developer drop the ball; to make sure the impact of any unforeseen issues is minimised for customers, The Iconic introduced feature toggles -- allowing developers to turn off troublesome functionality without having to deploy new code. Every new feature that goes into production must now sit behind one of these toggles, which dictates whether a user is served the new or old version of the feature in question. The error rates between the new and old versions are then monitored for any discrepancies... While Brennan is no fan of "people breaking things", he argues moving fast is more beneficial for customers.
"If our site is down now, people will generally come back later," Brennan adds, and the company has now moved all of its QA workers into engineering roles.
We're lazy as shit.
QA done right can add value to a project by rooting our ambiguous requirements and holding developers accountable. Developers are worried about a lot of issues like function, aesthetics, security, performance, scalability. Having QA work from the same specs to derive the functional tests from a different perspective can provide a huge value to developers. But you have to involve QA early in the game.
No, for the same reason professional writers need an editor. Someone else will see things that you didn't. Yes, they will.
Didn't Microsoft move to this model just before the Windows 10 upgrade, uh, issues?
https://www.windowscentral.com...
I've done development and QA work, and QA approaches testing from from a whole different perspective. The problem is context switching. It's difficult for a developer to approach a piece of functionality from the user's perspective (especially if they don't necessarily know all the parameters for how the functionality is going to be used) It takes time to switch out of that mode, and, in an effort to make the code perfect, you can come short of ever making the code good to begin with.
A QA engineer is going into testing with the mindset of an end user. What are they going to expect to have happen? What are they likely to do? What corner cases aren't worth exploring? It's useful for developers to have some experience in this, but I don't think it's reasonable to expect them to be experts.
In general, you want developers to spend time and cognitive effort making sure the code is functional and maintainable. Heaping on the cognitive load of completely switching gears and thinking like an end user isn't always the best use of their resources.
My Other Computer Is A Data General Nova III.
Longer version: NOOOOOOOOOOOOOOOOOOO.
I've been at places where managers get a variation on this lovely idea: Hey, maybe you would be more careful coders if you had to do your own testing. You're all so sloppy!
Oddly enough, this is mostly from non-coders, or folks without technical acumen.
Here's the deal: There ARE coders that claim to not make mistakes, and create bug-free code. Those are known as inexperienced or self-deluded coders, or folks only making one-off projects negotiating exact terms that make it not REALLY programming so much as configuring.
Even just from a simple level as just writing down words, if you've ever written a work longer than a couple of paragraphs, you'll virtually always reach a point where there's some subtle or stupid mistake you make without noticing it. Even VERY experienced journalists and writers need proofreaders on any serious project.
It's not a matter of taking your work seriously, or being unable to face responsibility - you're always going to be blind to the flaws in your own code, and although testing crews ARE expensive - they exist for many, MANY important reasons. They're cheaper than the same churn without a proper quality process. Sure, some of that effort may be being duplicated, or used badly - but few possibilities would be worse than expecting everyone to 'man up' and expect bug count to go down.
In other words - certainly, I can test my own code if you'd like, but I'm a VERY expensive testing staff member, and my perspective is going to cause me to overlook a lot of things - since it already 'makes sense' to me and I won't spot the inconsistencies a percent of the time.
Ryan Fenton
EVERYONE on your team should be a developer.
But QA should come from that pool for one project. Swap them out for the next, etc.
Developers working at QA know where the mistakes are likely to be.
Developers testing their own shit subconsciously avoid the shoddy parts.
This also keeps the team as one.
It's like a platoon of Marines taking turns burning the shit.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
This is dumb.
QA people think like QA people. Developers think like developers. And developers can't bring a fresh eye to their own code.
Heck, most developers can't even develop remotely comprehensive test data.
There is some evidence in the nuclear and aerospace industries that having things checked twice results in people being lazier (because they think the other person will catch it).
But for mainstream business developers, I've never seen them able to test as well as QA does.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
Not for their own code. Developers are lousy at testing their own code, because they handle the errors that they think of, and won't make the ones they don't.
I suspect that developers are even poor choices for testing other developers code, because they're less likely to make ID10T errors.
I think we've pushed this "anyone can grow up to be president" thing too far.
You need a series of gates and software/firmware/hardware has to pass through all those gates. The later a bug is found the more expensive it is to fix it. When the customer finds a bug it can be catastrophic; as in layoffs are going to happen because of loss of revenue. So find the bugs early, and make sure you've got all the gates set up.
- Developer must unit test the code - developer must to the basic test of flipping it on, making sure that something happens and that it does what the developer thinks it does. Finding and fixing a bug here is very cheap.
- Other developers must code review the changes; spread out the knowledge to more than one person, have other sets of eyes looking at the code, let someone point out where the clever change is breaking the design or violating preconditions from elsewhere in the project.
- QA must then test the feature; positive and negative tests. Run it for long periods, try to break it, compare it one by one to each requirement and specification. This can happen before all features are complete.
- QA then tests the integration; not just one feature but all of the features, so it's done once there's a code freeze for the release. Maybe it repeats some of the earlier tests, but it should run through a full regression, use it like a customer would use it.
- If it's applicable, do a full system test. End to end, from hardware board to back end server. Test different combinations. Finding a bug here is expensive; but it's still less expansive than having the customers find the bug.
If you still find bugs after that, then review how the bug slipped past, fix the gates, and do better next time.
I've been a software engineer, a QA engineer, traditional IT SysAdmin, and have now found my niche in SysOps/DevOps/SRE/whatever-we're-calling-it-this-week. And good lords of Kobol, QA is absolutely necessary. The usual reasons definitely apply. A separate pair of eyes will find things you've been blind to because you're too close to your own work. Building things and breaking things are different skillsets and different mindsets. There will be edge cases that even the spec didn't anticipate. Blah blah blah.
But try being on the operations front line some time, and you'll REALLY appreciate a good QA team. Right now, I'm responsible for multiple products from teams that, due to acquisitions and re-orgs and different companies doing things differently, some have dedicated QA and some let the developers test their own code. I see the errors and failures before anyone else. And the QA'd stuff invariably causes me less grief than the "we trust the developers" stuff. Hell, if I had my way, we'd have QA vet ops changes and updates too; because I'm under no illusions of infallibility myself.
Imagine all the people...
As a developer solo-managing a legacy codebase, it is still important to have various rounds of external QA.
The first issue is that the developer can be so far into the code that they can completely miss what is obvious or un-intuitive to a non-technical user. This may also involve legacy bugs or interactions with code they havn't written themselves.
The second issue is that the the spec itself may not have been fully defined, or what you have is the developer's interpretation of the spec, and this may require clarification or feedback once someone has seen the end result.
The third issue is that a developer will focus their attention on the things they consider most important. Sometimes the best way to achieve this is to just give a developer a list of minor bugs/features, which is a great way of focusing attention by clarifying the spec and letting a developer speed though a bunch of quick fixes which have a predefined spec (defining the spec is half the mental effort).
So in general a developer should be capable of doing a first round of internal QA on their own code, but a second pair of eyes is still occasionally needed, as the developer has a very peculiar set of perception filters.
Developers should absolutely have unit and integration tests and only deliver a product to QA that they think will be immediately passed onto the customer. But within many organizations this just doesn't make people happy, and unhappy people make bad decisions. QA can give a seal of approval that gets better buy in within the company. This should not be the way it is but that's life.
Lastly, a good QA will look to see that what was produced is what was actually asked for and promised. This is no small thing, because many programming teams can lose sight of what the contract actually says. "But it's better" is something that should be negotiated, not just delivered.
But that said, QA can be a major road block. It is not uncommon that you throw things over the fence only to find that QA has hyper focused on something that isn't a requirement. For instance you might have some system that does echo location to find the distance to a wall. Your requirements call for one successful echo location every 10 seconds. You fire out 10 per second and around 50% fail. The reality is that you are getting roughly 5 ranges per second and never will you miss your target of one every 10 seconds. Yet QA gets all wound up about the 50% failure rate and reports that the product has a 50% failure rate. This is where you begin convincing the accountants that the QA department needs a head count reduction.
In aerospace, independent verification and validation is a critical part of the regulatory process. Every change has a reason, every reason is documented, and every change is reviewed by someone else.
[
Should Developers Do All Their Own QA?
If you can do that without deleterious consequences, then your work is trivial and/or not overtly complex.
Otherwise, such a thing is insane, specially if you work in developing actual products (shrink-wrapped software or software bundled in hardware), or complex services. ) And let us not even mention software related to avionics, weapons or medical devices (where QA is done by a separate organization to remove bias.)
The primary concern is always bias, bias that come from first-hand knowledge of a thing (or assumptions about a thing.)
It is a lot easier and effective to have a 3rd party dedicated to methodical black box and end-to-end integration/UA testing than to get a developer to test his own creation to the breaking point.
There is a reason why organization hire external teams to perform penetration testing of their infrastructure. He who knows nothing of the internals a-priori is the perfect person to black-box test the shit out of something with nothing but specs.