Slashdot Mirror
Ask Slashdot: Building a Software QA Framework?
New submitter DarkHorseman writes: I am looking into a new position with my employer and have the opportunity to work with the development and QA team to further the creation of a Quality Assurance Framework that will be used into the long-term future. This is software that has been in continuous development, in-house, for >10 years and is used company-wide (Fortune100, ~1000 locations, >10k users, different varieties based on discipline) as a repair toolset on a large variety of computers (high variability of SW/HW configuration). Now is the time to formalize the QA process. We have developed purpose-built tools and include vendor-specific applications based on business need. This framework will ideally provide a thorough and documentable means by which a team of testers could help to thoroughly ensure proper functionality before pushing the software to all locations.
The information provided by Lynda.com along with other sources has been invaluable in understanding the software side of QA but I have seen very little in terms of actual creation of the framework of the process. What would you consider the best resources to prepare me to succeed?
Even if your QA needs are for smaller projects, what advice do you have for formalizing the process?
Copy them.
> Now is the time to formalize the QA process
It's long past time for a formal QA process. Maybe you mean "better late than never"?
Tons of marketing speak and a link to Lynda.com. Write a document explaining its use, get marketing to put a price tag on it, and call it good.
This sounds like one of those quests to solve world peace and end hunger. QA framework is about as generic as a statement your can make.
The only framework you need is to get everyone on board with your end goals. Rest everything is circumstantial is my take on it.
Presumably you are using modern tools to compile and build your software, manage source code, and manage your project work. Many of these tools will either incorporate or integrate with bug tracking software and testing frameworks. If there's a native bug tracker available, select it. If there's a native test framework available, use it.
What you need is a least-friction option, where testers, analysts, and developers can all see the bugs, write up the bugs, test the bugs, and fix the bugs. You don't need "The Most Advanced Framework Available Today", you don't need "The Best Test Tracking and Reporting Software Ever Produced", you need a solution that works well for all the people involved. Having a third party tool where the developer has to stop working, log in to the bug tracker, read the bug details, switch back to the development environment, make some changes, switch back to the bug tracker, write up the findings, switch to the test framework, execute a test, switch back ... All that switching is a huge productivity killer. The smoother the integration, the more effective and efficient the engineers will be - and that's where your expenses really lie.
Here's the problem. Some organizations say "hey, let's evaluate and buy the bestest test software out there" without giving a thought to the developers. So the QA department runs off on their own, buys a tool, and starts building tests in it that the developers can't run. If the developers can't run the tests, they don't know if they're fixing the problems correctly, so they waste tons of time. Worse, if a developer makes a change that breaks some test, they won't know until that result is reported to them, possibly days, weeks or even months later, depending on your QA cycles. During the intervening time, the developer continues to write code based on their original faulty change, creating technical dependencies on what may be a completely flawed base assumption. When the test finally reveals the flaw, the developer's choices are limited to: A) rewrite everything according to the better architecture uncovered by the flaw, or B) make a scabby patch so the test passes. If you choose A, the software's release will be expensively delayed. If you choose B once, you'll likely choose it again, you're incurring technical debt, all your software is likely to be crap, and no good developers will want to work for you. The correct answer is of course C) don't produce tests the developer's can't run themselves on demand, or tests that aren't automated as a part of the build process.
John
I am looking into a new position with my employer and have the opportunity to work with the development and QA team to further the creation of a Quality Assurance Framework that will be used into the long-term future.
You haven't taken the position yet? Don't!
Unlike other industries (such as the auto industry, where QA is often the highest ranking and most powerful professional short of plant manager, respected for their global understanding of both engineering and the math of probability and statistics, empowered to order things fixed and, if appropriate, stop the production until they are), the pointy-haired bosses of software development generally view QA people (even the write-the-tools types) as failed developers. "If you were good you'd be a developer." is a typical quote. A QA position on your resume is an albatros around your neck.
One of the better programming talents I know (and I know a lot of the big names), when starting out, took a QA position as a stop-loss. After that she couldn't get a developer position - even those where they CLAIMED to give her a developer position actually put her in the QA department.
QA is who they cut first, when things get tight. This is massively stupid - because it's who you need to get OUT of the jam. But understaffing QA shows up on the bottom line a couple years out, after the PHB has lined up his next positio, while understaffing development shows up quickly.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
...how to build a software company.
Anyway, there is a lot to QA, but everything starts with the build and bug/feature tracking systems. QA is anchored to these two pillars. Formalizing (and documenting) how these systems work and interact is critical to correctly tracking issues, reproducibility and inter-QA-to-developer communication. A strong nightly build system coupled with a properly used tracking tool (like Bugzilla, Jira, etc) are pretty much the cornerstone of any framework.
The build system itself breaks down into the nightly automated build process as well as source revision control system. Testing should start before QA ever sees a build by testing the build after checkins and running developer implemented unit tests. Tagging release candidates and making sure that QA is working with the correct build is critical to optimizing QA efficiency. Nothing is worse than having QA run tests on the wrong build.
Next, QA testing itself gets divided into two major categories: automated and manual testing. Automation has a high initial cost of development, but pays off in the long run with lower labor, especially for mundane tasks that are too tedious to possible test by human.
Automation typically is closely linked into the build system. Manual testing breaks down into a bunch of different sub-categories, but all manual testing should follow documented test plans. Test plan documentation can get overboard, but at least you want a wiki of some kind (Confluence, etc) to document how development wants QA to perform the testing.
Manual testing is often where labor is the biggest factor, and thus it tends to get the most attention from management. These folks tend to be either "testers" good at testing vs. system integrators who are good at staging environments that look like real world customer environments. Much here of course depends on the nature of your product and your customer.
However, with both manual and automated testing, it is important to have infrastructure for creating clean environments to test with. This usually involves some form of virtualization. QA needs to be able to produce clean images of systems to test with. This part of QA has never been easier, unless you are a hardware company, and actually have to still use a lot of physical equipment instead of VMs.
Release quality is determined by comparing the list of open issues targeting a given milestone and their status. Again, a well integrated tool for issue tracking is critical in the process of release management.
Lastly, I would not wait 10 years to formalize a QA plan. Every software startup that I know of had a QA director starting to fill this plan out within a few months of getting seed money. QA is integral to software (and hardware) engineering.
Someone you trust is one of us.
I could tell you how to build a Software QA Framework. But...
If you're good at something, never do it for free!
Pay me my consulting rate, and I'm happy to oblige.
I have built and seen succeed and crash-n-burn spectacularly a variety of automated test frameworks for large enterprises. Let's start with the successes:
- High availability / Robust
- Staging environment for automated test developers
- Performance metrics
- Easy to understand test results
The failures were due to:
- Brittle and poorly designed tests which don't run the same in the CI system versus the tester's machine
- Testers committing bugs into the test environment
- No performance metrics
- Hard-to-understand failure results require duplication and deep analysis
As you can tell, the failures are the opposite of the successes. Allow me to further explain.
The most important item is that the tests always work and always be running. This means test machines and back-up test machines. Running the same test on three different machines is even better because you can throw away or temporarily ignore outliers. Outliers need to be addressed eventually, but for day-to-day developers and managers just want to know if the code they committed causes failures. Having the test be in any way unreliable causes faith in the tests to disappear. The tests must run, and they must run well. Test environments go down or require maintenance and you want to be able to continue to run tests during these downtimes. Treat the tests like a production system.
Next, a big improvement I've seen is to have automated test developers contribute new work to a separate-but-equal staging environment. Automated test teams run on an Agile/Scrum iteration and only "release" their new tests at specific times. Another thing which reduces faith in test results if the tests are breaking due to the fault of automated test developers-- which happens all too often.
Automated tests are the ideal platform for generating performance metrics.
Lastly, a big pet peeve of mine is for understandable test failures. Test failures should obviously describe the set-up, expected and actual result. If test failures are obtuse and require a lot of time to analyze and triage-- that is wasted time that could have been spent fixing the root cause.
Good luck! If past experience is any indicator, you will be spending far more time and money than you ever imagined to create a robust system that developers and managers will have faith in.
That's not what I meant.
The objective of QA is to verify that the developers have been doing proper testing and not to do the testing yourself.
Deploy Kaizen. Also, in a big, varied organization using just a single test team is like utilizing focus groups on testing political messages. It is just not enough. In other words, aim for lightness, easy feedback and flexibility in the identified points of variability.
100% correct. Plus QA is frequently offshored as it is just as effective.
Dev's are there own testers and QA is the end users.
also we have that easy update system. So release now patch / update later.
You could look at standards like iso9001, iso27001 which elaborate procedures for total software (+HW) quality - from origin to deployment. 9001 is mostly for critical applications - medical, transportation. 27001 is more for data security - HIPAA, PCI-DSS, etc. Pick and choose depending on what you are doing.
QA starts at the (or designer) developer and then goes on to a QA group. So you need to establish software code quality standards and then make sure they are met by doing code reviews. Developers need to be trained as far as preventing code exploits - sql injection, cross site scripting, and others. Make sure all inputs and outputs are validated.
Writing specifications that include testing procedures for QA is also helpful for a QA group. For final QA, automated testing is very useful - developers can even write QA test code with the product code.
Getting buy-in from management and the developers to actual follow the process is the hard part.
Too late already. The horses are out of the barn.
You can not retrofit good QA to an existing development ecosystem.
Give up. Don't take the job, you're just the fall guy.
Write a python program to "drive" the code paths of your software. As it expands, factor it out into a Python module. After a while, this module will be the glue that's running your whole company. At least mine is. I think mine is sentient by now.
Well, the question is so random, open-ended, and vague with no details that anyone could say anything in response to it. "I have a 10+ year code base of whatever that needs to do something and I want to write some software. Might as well make it a framework" Yeah, go for it.
> QA is who they cut first, when things get tight. This is massively stupid - because it's who you need to get OUT of the jam.
In general, debt is normally dumb. Except when it's for an offsetting investment. OCCASIONALLY, in business debt is smart. For example, being first to market for a rapidly growing sector may well justify debt. Survival mode may justify debt. Technical debt, poor code and systems, IS debt. Get the product out now, pay the cost of poor code later. That's NORMALLY a bad idea. Occasionally, it's the correct move.
An example from my very recent experience is replacing the annual licensing of a $xx million / year third party program with one built in-house. Two weeks before the deadline, it's better to get it done poorly and fix it later than to pay another $XX million to license the third-party app for another year. But we WILL have to fix it later. SOMETIMES quality actually isn't the most important thing. Which bugs the shit out of me.
Perhaps look into the following standards and consider emplementing one or both of them:
* ISO/IEC 9126 https://en.wikipedia.org/wiki/ISO/IEC_9126
* ISO/IEC/IEEE 29119 https://en.wikipedia.org/wiki/ISO/IEC_29119
It must be a slow news day for /.
Rule of thumb is to use similar languages/technologies/frameworks that the developers use to track their workflow. That way they'll help you when you break stuff.
--bm
QA is not just about testing. Testing is Quality Control; just one aspect of a QA framework.
What does the product manual say about what functions it has? Are they correct? Are there any procedures in the manual for the operator/owner? Are they correct?
what's wrong with selenium and cucumber? am i missing something?
lololol
Now, after >10 years? Sudden outbreak of pandemonium?
I've been a software tester for more than 10 years. I was regarded as an excellent software developer (by peers and managers) when I *chose* to become a software tester because I decided that I like science more than engineering, which in my opinion, is the key difference between software testing and software development.
I have not yet come across any managers (pointy-haired bosses) who genuinely believe that testers are failed developers, but I have come across many developers who regard testers as lesser beings. One even told me with a blank look on his face - "I think you are good enough to be a developer?".
It doesn't help that there are indeed many failed developers working as software testers, spouting nonsense like "I am not technical" or "Testing is so easy".
Software testing is science. It's an intellectual activity. It's challenging. It's very difficult to perform efficiently and effectively. Anyone claiming otherwise simply misunderstands software testing. (If you are one of those people, I recommend that you look up James Bach.)
Sure, I could have continued my career as a developer because I don't like others thinking of me as a failed developer. But in the end, I chose my path based on what I liked rather than what others (incorrectly) thought about the profession.
We need more capable people to perform meaningful software testing to dispel the myths in the industry, including ones posted below such as "QA is just as effective when offshored". Telling people to don't become testers because others incorrectly view them as failed developers is not a helpful advice, neither for those people or the industry as a whole.
Ran across this - does it fit the bill?
https://www.cons3rt.com/
If you're writing a big, complicated testing framework, don't forget to write tests for the tests. I'm not even joking
It is true that we developers do look down on software testers. They are paid less as well. Frequently they are offshored. I'm glad you like it, but you chose the wrong path if you are looking to maximize your usefulness and earning power in the industry. No offense, but that is the truth.
Untested software isn't very useful
"in continuous development, in-house, for >10 years"
A little late for the rodeo, aren't you?
Now is not the time to think of this sort of thing; that time was 10 years ago.
Working for a top recruiter, he said QA is a definite negative point. I've read that the greatest QA teams are hated by programmers. My best guess it's a bit like racism, QA people need not apply.
QA people need not apply.
It is true that we developers do look down on software testers. They are paid less as well. Frequently they are offshored. I'm glad you like it, but you chose the wrong path if you are looking to maximize your usefulness and earning power in the industry. No offense, but that is the truth.
Wow.. sounds like you come from a twisted corporate culture. Why dis someone who's work protects you from making a fool of yourself in front of customers?
Testing is critically important in a successful software project.
I've never seen it "just as effective". There are delays in reports due to time zone, and someone will need to be the chump who has to handle the latenight or early morning concalls. Bug reports are often unreadable, unreproducable, lacking vital information or just don't make sense. Software versions under test are often out of sync. Lots of problems glossed over due to inexperience, lack of interest, or trying to make a milestone and keep the bug count down. Frequent glaringly obvious bugs that the offshore teams claim they never saw in test and can't reproduce on their hardware/software/platform/gizmo.
But the rest is all true, I've been around long enough that QA was 95% in house and it was a decent paying profession. Now it's almost entirely offshore and for those that remain it will be one of the first groups that gets the axe.
Offshoring QA is a common mistake that companies make because they think it's a cut and dry task. But every company I've worked with who off shored the QA department has ended up with less quality and more developer time spinner their wheels on false positive test failures than with a QA staff in house. The worst I've seen is when the offshore QA team stops their test plan after a single bug is found. Then its a 24 turn-around before the test plan can continue. This causes huge delays in releasing the software.
There is a tremendous advantage in co-locating QA and developers. But if you have upper management that are number crunchers, they'd rather save a buck than have a productive work force.
There have been several other answers that seem to hit aspects of this, but none address how a framework can mean multiple different things at different levels. Here are a non-exhaustive list of some of the levels you might consider:
* Automation framework(s) ...
- Automated tools and checking
- Load Test
- Manumation (mix of automation and manual)
* Institution Vocabulary/Training - Taking something like BBST so everyone has the same idea of what you mean by QA. http://bbst.info/?page_id=11
* Test Process
- How the testing cycle appears.
* Development Process
- How testing integrates into the bigger scheme.
- Who does unit tests? Are they dev-only or are testers embedded in this process?
- Are testers involved at design time? Are developers involved at test time?
* SDLC
- Where does testing occur. How do you know when the system is ready for release? What is the mission of testing? (See BBST above)
* Bug Lifecycle/Advocacy
- How you write bugs
- Who is assigned bugs and when is a bug 'done' (do you retest in production?)
- How are bugs triaged?
* Team Organization
- Are QA silo'd or embedded or
- How does automation fit in? Who does it? Are they silo'd or embedded?
- Who do they report to (and how are conflicts resolved)?
- Is QA one happy family or specialized?
- What about cross-cutting concerns like security? Is that part of QA?
* Documentation
- How is it organized? Wiki? Text files in git?
- Is it all in one central place or divided by project?
* Rank/Role
- How do you separate a Tester 1 vs a Tester 2 (insert your labels here)?
- How do you determine what type of person you want to hire?
- Hire junior people and do lots of training/process or hire senior people and give them the freedom to solve problems themselves?
- Central Planning vs Team Leadership (E.G. Waterfall vs Agile)
- What skill sets you want or need - http://blog.red-gate.com/wp-content/uploads/2014/02/Test-Engineering-Skills-v3.pdf
* QA vs QC
- Do you intend on improving the entire process? Then you are QA. Otherwise you are just QC.
* Testing Models (http://www.satisfice.com/tools/htsm.pdf)
- Session-Based Test Management: http://www.satisfice.com/sbtm/index.shtml
- Thread-Based Test Management: http://www.satisfice.com/blog/archives/503
- Tours: http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards
I'm afraid I've run out of time, but hopefully that helps. Some of my links were stolen from my blog: http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html Good luck!
As a customer of such software, I second that, and I resent being asked to do the job the vendor was too stupid to put on the payroll just to get their product to work.
Someone had to do it.
"... but I have seen very little in terms of actual creation of the framework of the process. What would you consider the best resources to prepare me to succeed?"
What books have you read on QA? What books have you read on software processes, in general?
You want to set up the "framework of the process." OK, fine, you don't have a technological challenge, it's an organizational challenge. You have been handed a tar baby, a maze of twisty corridors that all look alike. *Now* they want 10 years of cruft organized. We, out here, have no idea what type of software it is, etc. Nobody can give you specific information on how to solve your problem in a short paragraph.
The best thing you can do is to read some books on the QA process. This will give you a clue. I was just at Powell's book store, and they had some books on software QA. From what you've written, anything will be a big help. Go, read, plan, get organized.
Basic guidelines:
What is the software supposed to do? (Basic specifications. If the managers and devs say they don't need specifications, run! That's what I did at my last position. They didn't actually want QA, and I didn't need to be there.)
How is the conformance to the business specifications verified and tracked?
How is the quality (robustness against failures) verified and tracked?
How is the usability verified and tracked?
You have to start with specifications, then implement verification, and then tracking. This isn't a huge monolithic thing, but a continual and verifiable process.