A Bible for Software Testing?

An anonymous reader asks: "I'm soon to be starting a position in software testing and wondered (well hoped) if Slashdot readers had recommendations for reading, in terms of dealing with testing from the trenches and management of the process. I've read a number of general software engineering texts, but what I'm looking for is a specific 'bible' on software testing that will get me in the right mindset, before I begin."

Comments

get certified, don't worry too much

[oiarbovnb]

You should think about getting certified as a CSTE (Certified Software Test Engineer) or CQA (Certified in Quality Assurance) from the Quality Assurance Institute.

By the way, I'm a software tester, and I hate my job. And don't worry about being too over-prepared for your job. All the testing jobs I've had (4 now) have been pretty simple. The first day on the job, you'll be introduced to your new computer, and if you are lucky, your co-workers. After that they will give you a bunch of documentation to look through and in about 2-4 weeks you will begin testing from scripts. It's the easiest, most mind-numbing job for a computer professional. Not fun if you ask me... If you are detail-oriented, and not problem-solving oriented, it might be a good match for you. But don't expect to do much thinking on the job...

Re:get certified, don't worry too much

[oiarbovnb]

Also, if you want to impress management, go look at some Mercury tools and get time reading about them. I don't know if you can get a demo for training for them, but WinRunner will really save testers (and then Management, of course) time in Regression testing. Another great website for testers is Sticky Minds. There you will find references to all the books your mind can eat up. Good luck!

Re:get certified, don't worry too much

[SuperguyA1]

I've been a Q/A engineer for about 6 years now and I love it(my current job). I've done QA in a situation like yours and it is mind numbing but it doesn't have to be. In a smart company with a decent QA team the challenges are just as great as a development team (sometimes more). I do a lot of scripting and I have to understand our products as well as the developers, at least in a blackbox setting, to find all the nasty hidden corner cases. Plus I get to break stuff and get paid. For anyone else looking at QA keep in mind there are jobs that suck in any area.

Re:get certified, don't worry too much

[Quikah]

Contract software testing is menial labor. You just follow along with a script and report any defects you find (games are the worst types of apps to test BTW, mind-numbing and really kills any enjoyment you get from gaming). I agree it is very easy and boring. Probably why a lot of it is being dumped to India. Career QA professionals are the ones that are writing the scripts, scoping out the tests, setting up the testbeds, etc. This is very fun work to me.

If you want to continue in QA I suggest you look into getting some experience with automated tools. You will do a lot of coding and problem solving with those. It is a good fit for someone who enjoys coding but either is not good enough or just doesn't want to be a full time developer. Also there is a big demand for these skills.

Re:get certified, don't worry too much

[oiarbovnb]

Haha... I'm in a dumb company with a horrible QA team. The management has a lot of "vision" for the QA team, but no "execution" skills. I agree that testing *could* be fun, but in my past 4 jobs, I haven't found it to be so. Perhaps I am not detail oriented enough. You seem to be, as you say that you have to find all the "nasty hidden corner" areas of your app. Well, best of luck to you and your QA careers (both SuperguyA1 and parent AskSlashdot poster). I know I'll be looking to enroll in school to do something else where I get to problem solve and work with people more.

Re:get certified, don't worry too much

[Peterl]

Automation can save significant amounts of time and money if the project:

• Has a large enough number of regressions to make the expense of automating less than the cost of all of the regressions.
• Has APIs and/or UIs that are stable enough over the project (and hopefully across versions) to allow the automation to work without lots of maintenance.
• Has management and development behind the automation effort so that time is properly allocated and dev understands the needs of QA.

Re:get certified, don't worry too much

[oiarbovnb]

I swear you stole that from my presentation to my management about testing automation... :)

Re:get certified, don't worry too much

[Peterl]

I do QA for a living. I mostly work on designing and implementing tools for automated testing, and performance testing. I code, write test plans, work with management and dev to help decide project goals and requirements, provide input on hardware and software capital expenditures for the QA team, etc. Not all QA jobs involve following "button pushing" scripts that somebody else wrote.

If you are unhappy with the level of challenge at work, then I suggest you improve your skillset and look for work elsewhere.

Re:get certified, don't worry too much

[oiarbovnb]

While you are correct that not all QA jobs involve executing the scripts that someone else wrote, a lot of QA is testing as that is what the discipline is for. Testing software. A QA department, while it can, doesn't have to order hardware, or write test plans. Testers test, first and foremost. You may be in a managerial position of some sort, I don't know. But it sounds like the poster of the article will be doing software testing, as I traditionally think of it, which is reading and executing test scripts. This job is pretty boring, and any temp should be able to do the job as well as I can (or better because they might be happier doing it considering their alternative might be flipping burgers). You are right that I should "improve my skillset and look for work elsewhere," and am planing on doing just that...thanks for the advice though.

Re:get certified, don't worry too much

[Peterl]

Excellent! That mind reading drug I bought from that spammer is working! :-)

Actually I've been over and over that topic with management at my various jobs.

Re:get certified, don't worry too much

[Mod Me God]

I am not a QA professional, but my unit had a bit of be-spoke software written a few months ago which has gotten around to being implemented. I had to do the QA/user acceptance tests a month ago. I literally drew the short straw - it sucked. Following instructions - nothing to solve, just doing what was written on a 300 sheet pad, commenting (with a tick or a cross, possibly some more verbose abuse if i was extremely pissed off).

My experience 'only' took a week but was just so mindless - if you do QA for a living make sure you can have some creative/productive input or you risk losing your mind (unless you like mindless repetative 5h17).

Just my 2p/c/d/b

Re:get certified, don't worry too much

[Anonymous Coward]

I'd like to disagree.

In my experience I found that there are tons of automation opportunities if you keep your mind open. Compile good statistics. Automate your boring work away, and you'll be doing great (if your Boss lets you).

Re:get certified, don't worry too much

[dubl-u]

Automation can save significant amounts of time and money if the project [...] Has a large enough number of regressions to make the expense of automating less than the cost of all of the regressions.

On my last few projects, I did both automated unit tests and automated end-to-end tests, right from the beginning of the project. Developers are the ones in control of the unit tests, but I like the product managers and/or QA to be in control of the end-to-end tests.

In my experience, having test suites like this makes development go much faster than it otherwise would, even with very high-quality developers. And the payoff is ongoing; fully automated QA tests mean that you can change the system with complete confidence, releasing daily if you like.

My experience matches what Martin Fowler talks about in his blog entry on very low defect projects, meaning less than one bug per month.

Re:get certified, don't worry too much

[caseydk]

Andy Hunt, of Pragmatic Programmer fame just released a series on Unit testing while emphacising *NOT* screwing productivity.

I just got to see a presentation from him this past Saturday. He's a sharp guy.

Re:get certified, don't worry too much

[caseydk]

Oh, and I forgot:

"Debugging : the nine indispensable rules for finding even the most elusive software and hardware problems"

Here's some reviews

This lays out some rules incredibly clearly... some of the rules will cause you to say "Duh, that's obvious" but there will be others that make you say "Doh, I should have been doing this already!"

My 0.02

Testing Computer Software

[jhoger]

Testing Computer Software

by Kaner, Falk, Nguyen

You can't go wrong with this one.

Testing Computer Software

[Quikah]

Testing Computer Software

by Cem Kaner, Jack L. Falk, Hung Quoc Nguyen, Jack Falk, Hung Q. Nguyen

If you plan on doing this as a career I am sure you will encounter something by James Bach, IMO he is overated and a bit of an ass (sent me outside a classroom because I didn't have any questions for him?! So I came up with a lame question I already knew the answer to and proceeded to fall asleep for the rest of the lecture).

Good Beginner Text

[TechnoBoffin]

I work testing desktop app integration, and a very good book I've seen that's quite accessible to the beginner is Testing Computer Software by Kaner, Falk, and Nguyen. I bought it and passed it around our lab just for the first few chapters which deal with the mindset behind software testing, such as why even bother testing when you can't assure 100% bug-free code. Later chapters cover how to effectively log bugs, how to test things besides actual code (devices, localization, manuals, etc.), and an overview of how to manage a software testing team. From experience, I can tell you that not 100% will be applicable to the particular job you're about to start, but it will meet your "get me in the mindset" requirement.

Here's what's on my bookshelf

[steve.m]

I find these books useful:
Lessons learned in software testing - A good introduction
Software Testing - A whole load of thing you'd never think off
Software test automation:Effective use of test execution tools - A bible for implememting automated testing
How to break software - crashing apps by forcing error conditions
How to break software security - similar to above, but with security in mind

How To Break Software

[RomSteady]

How To Break Software by Dr. James Whittaker.

I was able to attend a "virtual lecture" by Dr. Whittaker thanks to a former employer. He not only understands the root causes for most bugs, but understands the core competencies that the best software testers have.

The Art of Software Testing

[speedy1161]

Here
Sure it costs a fortune, and its old, but the stuff in it stands the test of time. Definatly a must have if you're testing any software that just cannot have a critical failure.

Software Testing in the Real World

[GlenRaphael]

I rather like the book Software Testing in the Real World: Improving the Process by Edward Kit. This book is very short, so you can read it in a reasonably finite time. It's aimed at the managerial end of things -- figuring out how to iteratively improve the testing process in an existing organization.

If you're not at a managerial level and need more localized nuts-and-bolts information, I'll second the recommendation everybody else is making - Testing Computer Software by Kaner/Falk/Nguyen.

If you want to get certified, you can find out about that here. As part of running tests to certify people, the Software Quality Institute maintains a current list of books here for Certified Software Test Engineers and here for Certified Software Quality Analysts.

(I'm signed up to take the CSTE next month.)

Re:Software Testing in the Real World

[MImeKillEr]

(I'm signed up to take the CSTE next month.)

I feel your pain. I'm to sit for my CSTE in September. The study guide from QAI is one of the most bloated, poorly-written books I've ever had the misfortune of reading.

And since the CSTE isn't like an MCSE, there's not a plethora of other (read:better) study guides on the market to turn to.

Re:Software Testing in the Real World

[GlenRaphael]

I'm to sit for my CSTE in September. The study guide from QAI is one of the most bloated, poorly-written books I've ever had the misfortune of reading.

I guess I'm glad I didn't buy it, then! My plan was just to review the various and sundry QA-related books I've bought in the past (including the two I mentioned in my post) and buy one or two new ones. If I take their claims about the test at face value, combining that sort of review with my relevant industry experience should suffice for the CSTE.

The CSQA test, on the other hand, I'll probably get the study guide for. But first things first...

From the Software Testing Bible ...

[yamcha666]

He who blue screens shall not suffer, For he is blessed by the Lord our Root...

Ok. Cheap joke. I know.

Depending on the type of testing...

[Satan's Librarian]

Every company I've worked at seems to have their own standards idea of what testing software is. What a "Software Tester" does can be anything from following a written 'script' and pushing buttons in the product to writing test harnesses for unit testing, building the configuration management and build machines, and building automated tests to first check to see if each feature works - then make sure it stays working once it does.

What you'll need depends on what the corporate culture is for QA there and how much initiative and experience you're bringing to the table. Several people have suggested good books on testing and breaking software already, so just a few things I'd recommend on the practical...

If you're taking the programmer angle, take a look at "Building Secure Software" by Viega and McGraw. If the product runs on Windows, I'd also suggest "Debugging Applications for .NET and Windows" by Jon Robbins.

Whether you code or not, I'd also recommend GUI Bloopers by Johnson - there's a lot more to testing software than just finding where it crashes, and GUI issues are one place where problems are often swept under the rug. Look for inconsistencies in how the software responds to the user, and point those out to developers.

Use revision control software like CVS and keep any scripts and automation projects you do in there - they can be almost as important as the product's code. Also make sure there's a way to track bug histories available - I've walked into some shops that did it all purely by email and word of mouth, which is a great way to ship forgotten bugs. Bugzilla and Mantis are the two I've used.

Remember that the developers and management should strongly appreciate it when you find problems with the software, but more often than not they'll be behind schedule and a bit bitchy when you do your job well and find issues with their code - it's part of being in testing. They'll likely want to brush off things they find 'minor' entirely near ship date - you can keep those marked as low priority, but get them into the bug tracker!

Try to learn how to let people know there's a problem w/o being abrasive, and if you're working with a good team you'll likely find the developers are happy to help you out if you need anything to make your process smoother. Don't be afraid to ask for features that make testing easier.

And probably most importantly - when you do find a problem, make sure you know *exactly* how to repeat it or do your best to find out and let the developers know. That can mean the difference between a 5-minute fix by a developer and a few days of developer time before a fix. There are a couple of QA guys I've worked with that I wish I could drag with me whenever I switch products/companies simply because of how detailed their bug reports are.

Just my $0.2...

Software Testing Class Site

[rjune]

Check out this class site:
http://www.eng.mu.edu/corlissg/198.2001/
T here is a lot of useful information including the required and recommended texts along with other resources.

good book not quite bible level

[timbu2]

I like Boris Beizer's "Black Box Testing", which is not really about white box testing. It has really good examples based on filling out U.S. tax forms. It helps to understand testing at both specific and abstract levels.

Careful with that QA moniker

[gosand]

Every company I've worked at seems to have their own standards idea of what testing software is. What a "Software Tester" does can be anything from following a written 'script' and pushing buttons in the product to writing test harnesses for unit testing, building the configuration management and build machines, and building automated tests to first check to see if each feature works - then make sure it stays working once it does. What you'll need depends on what the corporate culture is for QA there and how much initiative and experience you're bringing to the table.

Just a note - Software QA (i.e. Quality Assurance) can mean anything from a tester in the trenches to someone who only works on process improvement. Note that in the CMM, Software Quality Assurance has nothing to do with testing. It is a process job. Keep this in mind, depending on where you are working.

Certified, shmertified

[gosand]

You should think about getting certified as a CSTE (Certified Software Test Engineer) or CQA (Certified in Quality Assurance) from the Quality Assurance Institute.

By the way, I'm a software tester, and I hate my job. And don't worry about being too over-prepared for your job. All the testing jobs I've had (4 now) have been pretty simple.

You must be a contractor. I have been in software testing for 11 years now, and it can get very complicated. Of course, every place you work looks at testing differently. If you get in somewhere that values testing, your job can get very complicated - "Here, learn how this entire program works in a week. Oh, and learn how it interfaces with these two other products."

It's the easiest, most mind-numbing job for a computer professional. Not fun if you ask me... If you are detail-oriented, and not problem-solving oriented, it might be a good match for you. But don't expect to do much thinking on the job...

Sorry, but you don't sound like a professional to me. Sure, it can be boring and un-technical, but if you take that attitude then you'll be doing the same job forever. There are tons of opportunities to work directly with developers to debug code, or get into complex system configurations, or performance testing and tuning, or tracking down some elusive customer problem. It all depends on where you are working, and their level of sophistication. Test automation is a whole nuther ball-o-wax too. There are requirements, design, and sometimes code inspections. And if where you are working is into the whole CMM thing, then QA is not testing, but is process oriented.

Believe me, a good quality tester is a valuable asset. They think differently than programmers, which can strike a very good balance. If a workplace doesn't value their QA/testing people, then it probably shows in their products.

Re:Certified, shmertified

[oiarbovnb]

You must be a contractor. I have been in software testing for 11 years now, and it can get very complicated. Of course, every place you work looks at testing differently. If you get in somewhere that values testing, your job can get very complicated - "Here, learn how this entire program works in a week. Oh, and learn how it interfaces with these two other products."

Nope, not a contractor. I was a contractor for 2 of the 4 positions, but the first and last jobs were straight up jobs. I never got certified but am thinking about doing it.

Sorry, but you don't sound like a professional to me.

You are right that I'm not very professional about my job. I don't enjoy the work, and need to get into another field. The day I walked out of college I was handed a high-paying software testing job, and took it. I regret having done that because now I have no "actual" experience in other areas of software development and it is hard to make a switch.

Test automation is a whole nuther ball-o-wax too.

This is one of the areas of testing that I actually enjoy, as it allows me to do some problem solving and programming. You don't need to tell me I'm in the wrong job, I already know this. It sounds like there are some people out there that really enjoy their jobs in QA, but I don't for the reasons I already pointed out - I'm not detail-oriented, and this is a detail-oriented position.

Re:Certified, shmertified

[HeyLaughingBoy]

Believe me, a good quality tester is a valuable asset

Second that! I may want to swear at some of our testers when they send my "fixed" bugs right back at me, but they do find important defects (and just as important, they will document the steps & environment necessary to duplicate them). We the developers suggest test cases, but the better testers go well beyond that and design their own test cases that really stress the system.

It's not a job I would want (did enough hardware testing early in my career), but I respect those who are good at it.

Re:Certified, shmertified

[gosand]

You are right that I'm not very professional about my job. I don't enjoy the work, and need to get into another field. The day I walked out of college I was handed a high-paying software testing job, and took it. I regret having done that because now I have no "actual" experience in other areas of software development and it is hard to make a switch.

Actually, I have found that it has helped me. You know what colleges turn out? Programmers. I got a BS-CS, so I did my share of programming. My job out of college was doing configuration managment, and from that I went on to testing. I did a LOT of shell scripting those first few years. (not quite programming, but is better than nothing) Sure, programmers make more than I do, but I also know several who are out of work. The market got saturated. They don't teach testing in school, at least they didn't when I was there. It was at best a chapter in a software development course. So my advantage is that I do something that they don't teach in school, you need someone who has experience to do it well.

You don't need to tell me I'm in the wrong job, I already know this. It sounds like there are some people out there that really enjoy their jobs in QA, but I don't for the reasons I already pointed out - I'm not detail-oriented, and this is a detail-oriented position.

That sucks that you don't like your field, and I can see where it would be hard to switch. I enjoy it about as much as I can, but it isn't really an enjoyable job. I am a detail-oriented person, so maybe that helps. The one thing about it is that you can only go so far up the ladder without getting into management (shudder). At least in programming, you can get into architecture design and whatnot.