Practical Software Testing Resources?

rhartness asks: "I've been a software engineer by profession for a few years and a programming enthusiast for much longer. As my experience has increased, so has the size of the projects that I have had to work on. My software testing method involves trying to do everything I can think of that the end user might try to do. Hopefully, this will break the application if there is a bug within my code. The current project that I am working on involves numerous tiers within a smart client environment. Trial and error testing is no longer sufficient — there is simply too much that could happen. Searching the Internet for software testing resources provides an abundant amount of information but it's often quite philosophical and verbose. What are some practical resources that Slashdot readers use for testing your software projects?"

Comments

release it!

[Hell O'World]

Release it with a big BETA label. Let your customers find the bugs.

First, understand testing

[faster]

Read "The Art of Software Testing" by Glenford Myers. Even if you only get a few chapters in, you'll get a lot of ideas that will improve your testing. "Testing Computer Software" by Cem Kaner is another good one.

automatic unit testing

[coyote-san]

A huge step is to put in unit testing / regression testing that's run nightly, and idealy with every build. You should at least cover the basics, e.g., at the persistence level can you create/read/update/delete a record? If you load a parent object, do dependent objects get loaded as well? At the presentation layer you can verify that a missing field will set the right error message.

You still need to do additional testing, but this will catch the underlying errors that can cause flakiness -- and worse, bad workaround -- at the higher levels in your code.

Automate

[Timesprout]

My software testing method involves trying to do everything I can think of that the end user might try to do

You need to automate your unit testing and let someone else perform functional and integration testing, prefereably qualified testers. Developers almost always test only positive flow which is totally inadequate when end users get their illogical mitts on the application.

Automate, automate, automate.

[Z0mb1eman]

Step 1: write a systematic test plan. There are all kinds of acronyms and books out there on the subject - I doubt you need it. At the very least you can start with CRUD (Create, Read, Update, Delete) and go from there. Cover the success path, cover the failure paths.

Step 2: automate. This isn't optional if you're planning to maintain a project beyond the 1.0 release. For specifics, it depends on your project. Is it mostly a Java app (http://www.junit.org/)? Is a lot of the logic in the database (http://www.dbunit.org/)? Is it a web app, regardless of language (http://webtest.canoo.com, http://wtr.rubyforge.org/)?

Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.

Hire a tester.

[Bender0x7D1]

While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

Testing is an important part of the development process. If you have an expensive and complex application, do yourself a favor, and hire a test team. Don't try to do it yourself, unless you have no choice. You may do an OK job of it, but it sounds like you don't have the experience to do it properly. This isn't a critique of your skills, it is a fact that you have a different skillset. You don't expect an embedded programmer to develop web apps or vice-versa. Could they do it? Sure. But it will be a long and difficult road as they make the mistakes an experienced person already known how to avoid.

If you have decided that comprehensive testing is important, (and some people decide it isn't), then do the right thing and get professional (test) help.

My take

[Pootie Tang]

Let me start by saying that there is no one size fits all
solution. You described the nature of your projects to a degree, but
not the nature of your employment.

Do you work for a company or are you a contractor? Is your
responsiblity just for your code or the application as a whole (which
is another way of asking how big the team is in most cases)? Are the
users of the software someone close to you, you can ask them "does this
do what you want" or is this going to go on the shelf in a store and
you have no idea who might wind up using it?

I think these kinds of things factor into the equation. For example a
number of people have suggested having dedicated testers. My guess is
if you had them, you wouldn't be asking slashdot. And if you don't,
it's probably not something you can make happen by yourself, though it
might be a good point to bring up to your coworkers/boss.

I disagree with the Unit Testing advice that is being given out
too. You should have unit tests, but it will only get you so far. Unit
tests are effective at guarding against bugs being introduced when
making changes, but far from sufficient to make sure there are no bugs
in the entire system. In my experience the people who most heavily
rely on unit tests are also the ones who produce the buggiest code. I
think some of the comments make them out to be more of a panacea than
they actually are, and worse yet can lead to a false sense of
security.

Automation is extremely helpful. However if your issue is that " there
is simply too much that could happen", it's hard to realistically
automate all those paths. Presumably a problem with the things like
CRUD operations are going to be tested in virtually any path through
the application, which is why I don't think unit tests are necessarily
what you are looking for. Still do automate everything you can, just
be aware of the limitations.

My advice is as follows:

1) Keep doing what you're doing. You won't be able to catch
everything, but if you stop trying to do mimic what you think an
end user will do, you're going to miss a lot of things.

2) Write code that is likely to be bug free. Easier said than done,
but it's extremely important. Comment things, use descriptive
names, clear separation of concerns. You already know this, but
don't lose sight of it.

3) Use a logging framework. Have an error reporter for client side
apps. Write assertion checks into your code. If you try to insert a
row into the database, check the return value to see how many rows
were inserted. If you expected 1 and got something else, make sure
it's logged so you can find it later. Make sure you log as much
about the data/application state as reasonable so you can tell what
circumstances cause that problem. Make sure it's logged in such a
way that you can distinguish between serious errors like this and
as-you-go debugging output.

4) Do code reviews with other programmers. Develop best
practices. Re-evaluate them periodically to make sure they are
still "best".

5) Have a short write, compile, run cycle. Test that code as soon as
you can after writing it. If you find one of those "hmmm... I think
this should work", try to verify as soon as you can before you
forget. Don't wait until the end of the day and then conclude
"guess it worked". Test corner cases while the corner cases are
still apparent to you. If getting just created code to run is a big
pain in the ass (which could mean o

Re:My take

[jt2190]

...when requests come up (on an almost weekly bases) to change the code and rework specific modules entirely at the whim of management, things start to get complicated.

Willy-nilly changes are a huge risk to your success. Your management needs to focus on a goal, and let you reach it. It's like running twenty-five miles of a marathon and then having them move the finish line one you: Pretty soon, you drop dead from exhaustion. You can help them focus by making a list of all the stuff that they don't keep changing their minds about. Get that stuff built first. You can add window dressing later.

...as this project grows, I am reaching my mental limits of recalling and remembering what each method in that application does and why I even wrote some of them in the first place.

This is exactly why you want unit tests in place. The unit tests are your assumptions about how the code is supposed to behave, expressed in code. The ability to rerun hundreds or thousands of these tests very quickly against the code base, to make sure that things still work the same way as when you wrote them is a huge time saver.

By definition, unit tests don't test the assembly of the units, just the units themselves, so you'll still need to perform other types of testing to get complete coverage (funtional, integration, etc.) A good tester (i.e. someone who could write the program themselves but enjoys breaking code more than making it) can pay huge dividends here.

Re:My take

[Pootie Tang]

Given the additional information, I have a little more to say. It reinforces my impression that you are basically on the right track, there are just no silver bullets. However I think unit testing is probably more significant for you than I assumed at first. I think jt2190 expressed the value of unit tests very well.

I also agree with jt2190 about the significance of requirement changes. This is absolutely the area where you stand to make the most gains. If your boss does not have professional software development experience him/herself than he/she almost certainly does not appreciate how severe the consequences of changes are. Even if you tell them, they won't fully get it.

The fact that your boss is the brain child is a double edged sword, but mostly positive I think. Try to push more of the testing burden on your boss. After all, it's your boss who best understands how they want it to work. Explain that he/she can't just test the part that changed, but must as least give a cursory test to everything to check for side effects. That will help reinforce the importance of avoiding changes. Assuming your boss is as busy as most bosses are, it will also increase your chances of getting a dedicated tester hired (most likely from zero to slightly non-zero if you are the only dedicated developer).

When you get a new requirement or a requirement change, question it. Make sure you understand the underlying concern that is supposed to be solved by that requirement. This is an art, I'm sure you're getting better at it already. More importantly make sure THEY understand both the concern as well as the solution they are proposing. Specifying good requirements, just like good testing, is very hard and a lot of it comes from misidentifying the problem itself. If you have a good way of producing a mockup that allows them to explore the solution hands-on without needing to invest a lot of time to create it, it will go a long way. That's tricky as there are two closely related traps: the quick-and-dirty mockup evolves to production code without adequate planning or the quick-and-dirty mockup gets thrown out entirely despite the fact that some of it was valuable.

One thing I noticed you didn't say is that you are under pressure from your boss to reduce bugs. That leads to me to believe you realize how time consuming bug fixes can be and are self-motivated to find a solution. If so, that's a good sign.

One last thing that I should have mentioned initially as it applies to almost all situations, clean up that old code. It's inevitable you'll find crufty code as you go back to things. Sometimes it started that way for whatever reason (most often time pressure), sometimes it evolved that way. Cleaning it up (also known as refactoring) pays off 95% of the time. Unit tests are perfect when you want to change the code but not the behavior.

write a test plan, and/or hire a pro

[javaxman]

First, you should write a test plan. Actually, scratch that... first, you should write a design document. You have one of those, right? If not, you don't know what you're testing, and will eventually fail to cover some portion of your product, and that portion will likely break. Once you have a halfway decent design document, write a test plan that refers to your design document.

I'm going to second the notion I've seen in other posts that if your project really is big, there should be a programmer or analyst dedicated to testing it. You can hire someone who is less of a programmer and more of a tester if you want, but you'll get the best coverage if you can find someone with real programming chops who is able to design and implement tests. If you're doing your project right, that QA team will be every bit as involved in every portion of your project as you are.

Without knowing more about your particular project, it's very difficult to suggest appropriate tools. Different types of systems require different approaches to testing.

Verification?

[RAMMS EIN]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took. I don't know if there are any verification tools for the programming language you are using (the ones I used were all for toy languages), but that was my first thought.

Re:Verification?

[Raenex]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took.

In the real world very little software is formally verified. It's very expensive and time-consuming, and the vast majority of programmers don't know how to do it. Most projects just cannot afford it.

You're making too big a deal out of his "software engineer" label. There's no such thing as a software engineer, in the sense that for software there's nothing like the standards that engineers have to meet in other disciplines.

Ahem... try this

[plopez]

Go to wikipedia, look up software testing. Click aon some of the referenced links, then chase those links and you will find a wealth of information.

Just google +"software testing" you will find tons of links.

you might try these guys: http://www.softwareqatest.com/
They have some nice links as well.

Or would you rather start with a newsgroup? Go to google news groups and seach on software testing again. Lots of knowledgable and helpful folks hang out on those boards.

BTW, a colorful phrase for "trying to test everything a user might do" is "soap opera testing". Google it, the story behind the name is sort of fun.