Slashdot Mirror
Ask Slashdot: Building a Software QA Framework?
New submitter DarkHorseman writes: I am looking into a new position with my employer and have the opportunity to work with the development and QA team to further the creation of a Quality Assurance Framework that will be used into the long-term future. This is software that has been in continuous development, in-house, for >10 years and is used company-wide (Fortune100, ~1000 locations, >10k users, different varieties based on discipline) as a repair toolset on a large variety of computers (high variability of SW/HW configuration). Now is the time to formalize the QA process. We have developed purpose-built tools and include vendor-specific applications based on business need. This framework will ideally provide a thorough and documentable means by which a team of testers could help to thoroughly ensure proper functionality before pushing the software to all locations.
The information provided by Lynda.com along with other sources has been invaluable in understanding the software side of QA but I have seen very little in terms of actual creation of the framework of the process. What would you consider the best resources to prepare me to succeed?
Even if your QA needs are for smaller projects, what advice do you have for formalizing the process?
Tons of marketing speak and a link to Lynda.com. Write a document explaining its use, get marketing to put a price tag on it, and call it good.
Presumably you are using modern tools to compile and build your software, manage source code, and manage your project work. Many of these tools will either incorporate or integrate with bug tracking software and testing frameworks. If there's a native bug tracker available, select it. If there's a native test framework available, use it.
What you need is a least-friction option, where testers, analysts, and developers can all see the bugs, write up the bugs, test the bugs, and fix the bugs. You don't need "The Most Advanced Framework Available Today", you don't need "The Best Test Tracking and Reporting Software Ever Produced", you need a solution that works well for all the people involved. Having a third party tool where the developer has to stop working, log in to the bug tracker, read the bug details, switch back to the development environment, make some changes, switch back to the bug tracker, write up the findings, switch to the test framework, execute a test, switch back ... All that switching is a huge productivity killer. The smoother the integration, the more effective and efficient the engineers will be - and that's where your expenses really lie.
Here's the problem. Some organizations say "hey, let's evaluate and buy the bestest test software out there" without giving a thought to the developers. So the QA department runs off on their own, buys a tool, and starts building tests in it that the developers can't run. If the developers can't run the tests, they don't know if they're fixing the problems correctly, so they waste tons of time. Worse, if a developer makes a change that breaks some test, they won't know until that result is reported to them, possibly days, weeks or even months later, depending on your QA cycles. During the intervening time, the developer continues to write code based on their original faulty change, creating technical dependencies on what may be a completely flawed base assumption. When the test finally reveals the flaw, the developer's choices are limited to: A) rewrite everything according to the better architecture uncovered by the flaw, or B) make a scabby patch so the test passes. If you choose A, the software's release will be expensively delayed. If you choose B once, you'll likely choose it again, you're incurring technical debt, all your software is likely to be crap, and no good developers will want to work for you. The correct answer is of course C) don't produce tests the developer's can't run themselves on demand, or tests that aren't automated as a part of the build process.
John
I am looking into a new position with my employer and have the opportunity to work with the development and QA team to further the creation of a Quality Assurance Framework that will be used into the long-term future.
You haven't taken the position yet? Don't!
Unlike other industries (such as the auto industry, where QA is often the highest ranking and most powerful professional short of plant manager, respected for their global understanding of both engineering and the math of probability and statistics, empowered to order things fixed and, if appropriate, stop the production until they are), the pointy-haired bosses of software development generally view QA people (even the write-the-tools types) as failed developers. "If you were good you'd be a developer." is a typical quote. A QA position on your resume is an albatros around your neck.
One of the better programming talents I know (and I know a lot of the big names), when starting out, took a QA position as a stop-loss. After that she couldn't get a developer position - even those where they CLAIMED to give her a developer position actually put her in the QA department.
QA is who they cut first, when things get tight. This is massively stupid - because it's who you need to get OUT of the jam. But understaffing QA shows up on the bottom line a couple years out, after the PHB has lined up his next positio, while understaffing development shows up quickly.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
I have built and seen succeed and crash-n-burn spectacularly a variety of automated test frameworks for large enterprises. Let's start with the successes:
- High availability / Robust
- Staging environment for automated test developers
- Performance metrics
- Easy to understand test results
The failures were due to:
- Brittle and poorly designed tests which don't run the same in the CI system versus the tester's machine
- Testers committing bugs into the test environment
- No performance metrics
- Hard-to-understand failure results require duplication and deep analysis
As you can tell, the failures are the opposite of the successes. Allow me to further explain.
The most important item is that the tests always work and always be running. This means test machines and back-up test machines. Running the same test on three different machines is even better because you can throw away or temporarily ignore outliers. Outliers need to be addressed eventually, but for day-to-day developers and managers just want to know if the code they committed causes failures. Having the test be in any way unreliable causes faith in the tests to disappear. The tests must run, and they must run well. Test environments go down or require maintenance and you want to be able to continue to run tests during these downtimes. Treat the tests like a production system.
Next, a big improvement I've seen is to have automated test developers contribute new work to a separate-but-equal staging environment. Automated test teams run on an Agile/Scrum iteration and only "release" their new tests at specific times. Another thing which reduces faith in test results if the tests are breaking due to the fault of automated test developers-- which happens all too often.
Automated tests are the ideal platform for generating performance metrics.
Lastly, a big pet peeve of mine is for understandable test failures. Test failures should obviously describe the set-up, expected and actual result. If test failures are obtuse and require a lot of time to analyze and triage-- that is wasted time that could have been spent fixing the root cause.
Good luck! If past experience is any indicator, you will be spending far more time and money than you ever imagined to create a robust system that developers and managers will have faith in.
That's not what I meant.