Practical Software Testing Resources?

rhartness asks: "I've been a software engineer by profession for a few years and a programming enthusiast for much longer. As my experience has increased, so has the size of the projects that I have had to work on. My software testing method involves trying to do everything I can think of that the end user might try to do. Hopefully, this will break the application if there is a bug within my code. The current project that I am working on involves numerous tiers within a smart client environment. Trial and error testing is no longer sufficient — there is simply too much that could happen. Searching the Internet for software testing resources provides an abundant amount of information but it's often quite philosophical and verbose. What are some practical resources that Slashdot readers use for testing your software projects?"

Comments

release it!

[Hell O'World]

Release it with a big BETA label. Let your customers find the bugs.

TDD, JUnit, FIT

[Ramses0]

TDD, JUnit (or equivalent), FIT. Well-tested, independent tiers. GUI test as a last resort, or with humans.

--Robert

First, understand testing

[faster]

Read "The Art of Software Testing" by Glenford Myers. Even if you only get a few chapters in, you'll get a lot of ideas that will improve your testing. "Testing Computer Software" by Cem Kaner is another good one.

Re:First, understand testing

[rhartness]

Does anyone else have any thoughts on "The Art of Software Testing"? It appears that there was an update to the book in 2004-- the first since it's initial publishing in 1976. Is it worth getting the second ed. over the first?

Re:First, understand testing

[aevans]

"quite philosophical and verbose"

automatic unit testing

[coyote-san]

A huge step is to put in unit testing / regression testing that's run nightly, and idealy with every build. You should at least cover the basics, e.g., at the persistence level can you create/read/update/delete a record? If you load a parent object, do dependent objects get loaded as well? At the presentation layer you can verify that a missing field will set the right error message.

You still need to do additional testing, but this will catch the underlying errors that can cause flakiness -- and worse, bad workaround -- at the higher levels in your code.

Automate

[Timesprout]

My software testing method involves trying to do everything I can think of that the end user might try to do

You need to automate your unit testing and let someone else perform functional and integration testing, prefereably qualified testers. Developers almost always test only positive flow which is totally inadequate when end users get their illogical mitts on the application.

Re:Automate

[Timesprout]

I also need to test my posts for closing tags before I hit the submit button

Automate, automate, automate.

[Z0mb1eman]

Step 1: write a systematic test plan. There are all kinds of acronyms and books out there on the subject - I doubt you need it. At the very least you can start with CRUD (Create, Read, Update, Delete) and go from there. Cover the success path, cover the failure paths.

Step 2: automate. This isn't optional if you're planning to maintain a project beyond the 1.0 release. For specifics, it depends on your project. Is it mostly a Java app (http://www.junit.org/)? Is a lot of the logic in the database (http://www.dbunit.org/)? Is it a web app, regardless of language (http://webtest.canoo.com, http://wtr.rubyforge.org/)?

Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.

Re:Automate, automate, automate.

[ygthb]

Step 1: write a systematic test plan. Cover the success path, cover the failure paths. Gather all your data sources

Step 2: automate. follow the above advice if you want to create our own harness. But for an easier way check out the commercial tools (trying to be non-blatant, i am a vendor, check my profile)

Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.

An update to the previous poster.

Re:Automate, automate, automate.

[Z0mb1eman]

>follow the above advice if you want to create our own harness. But for an easier way check out the commercial tools

Commercial tools are definitely worth evaluating, yes, but as a programmer, I have yet to find one that actually makes my life easier (let alone be worth the usually exorbitant prices). They're great for places like banks, which might have a large budget and relatively low technical expertise, but (generalizing here) they don't sound like what the poster is looking for.

Re:Automate, automate, automate.

[ygthb]

Z0mb1eman,

It is up to the individual development group, but getting the fox out of the henhouse is as important as developing a test plan for accurate results. Programmers are taught to think in a certain way, that is not a bad thing. I can always tell when a programmer developed the test plan, or the subsequent tests. This is not to say that it is all bad. Programmers make great unit testers. But when it comes to wholistic testing, and what your users are going to subject your application to, a professional tester leading a group of CS reps or BA's is a much better solution.

It is just a story of two different skill sets. Critically addressing a problem with code, or critically addressing an application to identify problems.

ART

Re:Automate, automate, automate.

[Anonymous Coward]

Step 1: write a systematic test plan. There are all kinds of acronyms and books out there on the subject - I doubt you need it. At the very least you can start with CRUD (Create, Read, Update, Delete) and go from there. Cover the success path, cover the failure paths.

Step 1.5: show the test plan to someone else who's reasonably familiar with the project being tested. Let them ask questions about the test plan. Add any new test cases they think of (or that their questions cause you to think of) that you feel are important to test to the test plan. Repeat if you have time (and if you don't have time, show the test plan to someone else then start on step 2 while they're reviewing it.) The developer must, if possible, be one of the people who reviews the test plan. [Your test cases may trigger the developer to realize they hadn't thought about how to handle a certain case, which may lead to a modification to the feature specification and/or your test plan.]

Step 2: automate. This isn't optional if you're planning to maintain a project beyond the 1.0 release. For specifics, it depends on your project. Is it mostly a Java app (http://www.junit.org/)? Is a lot of the logic in the database (http://www.dbunit.org/)? Is it a web app, regardless of language (http://webtest.canoo.com, http://wtr.rubyforge.org/)?

Step 2.5: have someone else code review your tests, just like you'd have someone code review the project code. When they find bugs in your test, fix them. When they find bugs in the project code ("What, you're testing to make sure 2+2=5? That's what the code returns? That needs to be fixed."), pass them back to the developer. When they offer suggestions for other tests to be added, update your test plan and create those new tests.

Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.

Step 4: If possible, generate some measurement of code coverage and look for lines of code that are not being tested. Add test cases for those lines to your test plan, and add automated tests to the test bed. If you can't think of a way to exercise that line of code, ask the developer. If neither of you can think of a way to exercise that code, ask if it's dead code and needs to be removed. While code coverage is not the be-all and end-all of measuring how effective testing is, if you're not exercising a line of code you'll never find a bug that may be hiding there.

Depending on what you're actually doing, there may be other steps that you can/should run or other suggestions the Slashdot community can make. For instance, I test numerical software, and one of my steps is to think about how the weird edge cases of IEEE arithmetic (Inf, NaN, denormal numbers, etc.) interact with each other and the code.

Hire a tester.

[Bender0x7D1]

While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

Testing is an important part of the development process. If you have an expensive and complex application, do yourself a favor, and hire a test team. Don't try to do it yourself, unless you have no choice. You may do an OK job of it, but it sounds like you don't have the experience to do it properly. This isn't a critique of your skills, it is a fact that you have a different skillset. You don't expect an embedded programmer to develop web apps or vice-versa. Could they do it? Sure. But it will be a long and difficult road as they make the mistakes an experienced person already known how to avoid.

If you have decided that comprehensive testing is important, (and some people decide it isn't), then do the right thing and get professional (test) help.

Re:Hire a tester.

[Raenex]

While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

I've run across a few excellent QA people. They're extremely valuable if you can find them.

a few I've tried

[Darth_Burrito]

I've tried all of these but always end up getting sidetracked. Usually, the pain comes in terms of trying to apply tests to poorly designed or rapidly changing program models or in trying to set up test data in a db.

• JUnit
• NUnit
• PhpUnit (I think there are multiple versions)
• Php SimpleTest

Of these, I actually like SimpleTest the best so far, but I don't have more than a passing familiarity with any of them.

In terms of code quality, I think I've got more mileage out of goodish coding practices (separation of business/presentation logic, lots of acceptance testing) and libraries that support rapid development (eg DB_DataObject, HTML_QuickForms, Propel, Smarty).

Another favorite book

[DruBear]

I'm a big fan of "Software Testing In The Real World: Improving The Process (ACM Press)", Edward Kit (ISBN 0201877562.) It got me thinking about testing at a time when I hadn't really gotten the "testing bug."

TDD and Continuous Integration, also hire testers!

[bADlOGIN]

First off, you should be practicing TDD (Test Driven Development) as discribed here: http://en.wikipedia.org/wiki/Test_driven_developme nt as
  well as Continuous Integration as discribed here: http://www.martinfowler.com/articles/continuousInt egration.html.

The next thing you need to do is hire some testers. TDD and CI can help make fantastic improvements in the quality of your code, but you will never replace the time and effort of another human working to break your code.

Production

[eison]

The most common software testing solution that effectively tests a system of the complexity you have described is called "Production". Amazing how effective the "Production" environment is at locating all of the hard to find bugs. Anything less just won't do.

Short of 'production', Mercury makes some excellent tools, but they're more for automated regression or load than for full coverage.

Misnomer

[Anonymous Coward]

"I've been a software engineer by profession for a few years and a programming enthusiast for much longer."

Programmers aren't engineers.

Re:Misnomer

[W2k]

Correction: Not all programmers are engineers, but there are still plenty of engineers who are programmers.

My take

[Pootie Tang]

Let me start by saying that there is no one size fits all
solution. You described the nature of your projects to a degree, but
not the nature of your employment.

Do you work for a company or are you a contractor? Is your
responsiblity just for your code or the application as a whole (which
is another way of asking how big the team is in most cases)? Are the
users of the software someone close to you, you can ask them "does this
do what you want" or is this going to go on the shelf in a store and
you have no idea who might wind up using it?

I think these kinds of things factor into the equation. For example a
number of people have suggested having dedicated testers. My guess is
if you had them, you wouldn't be asking slashdot. And if you don't,
it's probably not something you can make happen by yourself, though it
might be a good point to bring up to your coworkers/boss.

I disagree with the Unit Testing advice that is being given out
too. You should have unit tests, but it will only get you so far. Unit
tests are effective at guarding against bugs being introduced when
making changes, but far from sufficient to make sure there are no bugs
in the entire system. In my experience the people who most heavily
rely on unit tests are also the ones who produce the buggiest code. I
think some of the comments make them out to be more of a panacea than
they actually are, and worse yet can lead to a false sense of
security.

Automation is extremely helpful. However if your issue is that " there
is simply too much that could happen", it's hard to realistically
automate all those paths. Presumably a problem with the things like
CRUD operations are going to be tested in virtually any path through
the application, which is why I don't think unit tests are necessarily
what you are looking for. Still do automate everything you can, just
be aware of the limitations.

My advice is as follows:

1) Keep doing what you're doing. You won't be able to catch
everything, but if you stop trying to do mimic what you think an
end user will do, you're going to miss a lot of things.

2) Write code that is likely to be bug free. Easier said than done,
but it's extremely important. Comment things, use descriptive
names, clear separation of concerns. You already know this, but
don't lose sight of it.

3) Use a logging framework. Have an error reporter for client side
apps. Write assertion checks into your code. If you try to insert a
row into the database, check the return value to see how many rows
were inserted. If you expected 1 and got something else, make sure
it's logged so you can find it later. Make sure you log as much
about the data/application state as reasonable so you can tell what
circumstances cause that problem. Make sure it's logged in such a
way that you can distinguish between serious errors like this and
as-you-go debugging output.

4) Do code reviews with other programmers. Develop best
practices. Re-evaluate them periodically to make sure they are
still "best".

5) Have a short write, compile, run cycle. Test that code as soon as
you can after writing it. If you find one of those "hmmm... I think
this should work", try to verify as soon as you can before you
forget. Don't wait until the end of the day and then conclude
"guess it worked". Test corner cases while the corner cases are
still apparent to you. If getting just created code to run is a big
pain in the ass (which could mean o

Re:My take

[rhartness]

Thanks for the response! Actually, I'm following many of your suggestions already. To give you (and others) a little background I work for a company that has one IT supervisor and I am the sole, dedicated software engineer. We are a manufacturing company and the point of the project that I am currently working on is to build an advanced, in-house system that will be used by hundreds of people all across the globe, some in very remote locations for internal tracking. The idea for the system was the brain child of my boss who has loosely speced out the system and it's my job to make the software magic happen. I'm quite confident that I could make this a rather robust and [reasonably] bug free application by the release but when requests come up (on an almost weekly bases) to change the code and rework specific modules entirely at the whim of management, things start to get complicated. Code that might have been written four or five months ago might have been appropriate and efficient but now is pointless or problematic because of changes that have had to be made due to specification changes. I'm doing a fair job at 'managing' the situation but as this project grows, I am reaching my mental limits of recalling and remembering what each method in that application does and why I even wrote some of them in the first place. Sure, I comment my code and even try to give my methods meaningful handlers but some routines consist of thousands of lines of code and hundreds of method calls from initiation to finish. It gets confusing after a while when you hit a bug that is hidden in code that's been written months ago.

Re:My take

[jt2190]

...when requests come up (on an almost weekly bases) to change the code and rework specific modules entirely at the whim of management, things start to get complicated.

Willy-nilly changes are a huge risk to your success. Your management needs to focus on a goal, and let you reach it. It's like running twenty-five miles of a marathon and then having them move the finish line one you: Pretty soon, you drop dead from exhaustion. You can help them focus by making a list of all the stuff that they don't keep changing their minds about. Get that stuff built first. You can add window dressing later.

...as this project grows, I am reaching my mental limits of recalling and remembering what each method in that application does and why I even wrote some of them in the first place.

This is exactly why you want unit tests in place. The unit tests are your assumptions about how the code is supposed to behave, expressed in code. The ability to rerun hundreds or thousands of these tests very quickly against the code base, to make sure that things still work the same way as when you wrote them is a huge time saver.

By definition, unit tests don't test the assembly of the units, just the units themselves, so you'll still need to perform other types of testing to get complete coverage (funtional, integration, etc.) A good tester (i.e. someone who could write the program themselves but enjoys breaking code more than making it) can pay huge dividends here.

Re:My take

[Pootie Tang]

Given the additional information, I have a little more to say. It reinforces my impression that you are basically on the right track, there are just no silver bullets. However I think unit testing is probably more significant for you than I assumed at first. I think jt2190 expressed the value of unit tests very well.

I also agree with jt2190 about the significance of requirement changes. This is absolutely the area where you stand to make the most gains. If your boss does not have professional software development experience him/herself than he/she almost certainly does not appreciate how severe the consequences of changes are. Even if you tell them, they won't fully get it.

The fact that your boss is the brain child is a double edged sword, but mostly positive I think. Try to push more of the testing burden on your boss. After all, it's your boss who best understands how they want it to work. Explain that he/she can't just test the part that changed, but must as least give a cursory test to everything to check for side effects. That will help reinforce the importance of avoiding changes. Assuming your boss is as busy as most bosses are, it will also increase your chances of getting a dedicated tester hired (most likely from zero to slightly non-zero if you are the only dedicated developer).

When you get a new requirement or a requirement change, question it. Make sure you understand the underlying concern that is supposed to be solved by that requirement. This is an art, I'm sure you're getting better at it already. More importantly make sure THEY understand both the concern as well as the solution they are proposing. Specifying good requirements, just like good testing, is very hard and a lot of it comes from misidentifying the problem itself. If you have a good way of producing a mockup that allows them to explore the solution hands-on without needing to invest a lot of time to create it, it will go a long way. That's tricky as there are two closely related traps: the quick-and-dirty mockup evolves to production code without adequate planning or the quick-and-dirty mockup gets thrown out entirely despite the fact that some of it was valuable.

One thing I noticed you didn't say is that you are under pressure from your boss to reduce bugs. That leads to me to believe you realize how time consuming bug fixes can be and are self-motivated to find a solution. If so, that's a good sign.

One last thing that I should have mentioned initially as it applies to almost all situations, clean up that old code. It's inevitable you'll find crufty code as you go back to things. Sometimes it started that way for whatever reason (most often time pressure), sometimes it evolved that way. Cleaning it up (also known as refactoring) pays off 95% of the time. Unit tests are perfect when you want to change the code but not the behavior.

Open Source Testing Tools

[Anonymous Coward]

www.opensourcetesting.com

Winrunner/Loadrunner Equivalents

[Anonymous Coward]

Anyone know if there are open source equivalent applications to Winrnnner or Loadrunner?

A change and a read

[nightgeometry]

Someone above recommended Edward Kit's book. Definitely worth reading, imo.

But the biggest reason I see for developers not being so good at testing, is that they are of the mind set that they try to show how the SUT does work. Not really in the least bit helpful, that is what your unit tests should have been doing. No, the mind set is 'In what ways does this fail?'. All software fails, you just have to find out how. That is probably one of the reasons you think so much of the literature is of a philosophical bent, because it is. There are techniques that can help you find the places where it'll break (wikipedia - boundary value analysis, equivalence partitions and software testing may help), but ultimately it is just a form of scientific method. Your SUT is a hypothesis, the only way to show validity is by attempting all you can to disprove it.

You may want to have a quick read of James Bach's site, and may even see if you can get yourself along to one of his trainings.

GUI test automation (Mercury, Compuware et al) - they are potentially useful, but you need to be able to understand how to test before you can write good automated tests. Unfortunately this is often not the case, hence the oft heard saying "Test automation, enabling you to test badly, quicker." Test automation is not the same as manual testing, you need to understand how it is different for it to be of value (beyond unit testing - you probably know how to unit test, hence you probably have at least an initial idea of how to automtate unit tests).

Feel free to ask more, or disagree, or just curse me some :)

And just to own up to my own bias, I've been a tester for around 15 years.

Re:A change and a read

[Raenex]

But the biggest reason I see for developers not being so good at testing, is that they are of the mind set that they try to show how the SUT does work. Not really in the least bit helpful, that is what your unit tests should have been doing.

I have no idea what "SUT" stands for, but testing basic functionality is also important at higher levels than units. Integration testing seems to be the common phrase, though Wikipedia also mentions systems testing (not that I can discern a difference).

Yes.

[charleste]

More up-to-date examples et. Al.

Re:Production -still need a methodology

[hguorbray]

although I agree that a production environment can help make a good test effort it is not always practical or feasible and I think the poster actually needs to hear about best practices, methodologies, tools etc rather than just being told to get a production environment.

Also, many unit and functional tests don't need a full blown production environment either -at least not until integration testing...

If the production environment consists of $1 million worth of HW and dedicated outbound feeds to banks and inbound feeds from a stock market for instance it is unlikely that it can be duplicated exactly because you would have to get the bank to set up a bunch of dummy financial transaction servers, pay for another feed from the market, buy a bunch more oracle licenses, etc

I worked on one deathmarch project where the production environment for an online Supply chain management solution consisted of 150 servers! Since they couldn't afford another 150 servers for the test environment they gave us 75 servers for front-end and middleware and had us connect to the backend of the production environment....

Sometimes you have to stub out or simulate external systems and it is important then to know how those systems interface with your SIT (system under test) and how you can properly simulate that interface and important interface events and messages.

If you don't have access to intelligent testers then you at least need to get feedback from the system users so that you can build business case scenarios from which you can derive test cases to ensure full functional coverage.

-What's the speed of dark?

write a test plan, and/or hire a pro

[javaxman]

First, you should write a test plan. Actually, scratch that... first, you should write a design document. You have one of those, right? If not, you don't know what you're testing, and will eventually fail to cover some portion of your product, and that portion will likely break. Once you have a halfway decent design document, write a test plan that refers to your design document.

I'm going to second the notion I've seen in other posts that if your project really is big, there should be a programmer or analyst dedicated to testing it. You can hire someone who is less of a programmer and more of a tester if you want, but you'll get the best coverage if you can find someone with real programming chops who is able to design and implement tests. If you're doing your project right, that QA team will be every bit as involved in every portion of your project as you are.

Without knowing more about your particular project, it's very difficult to suggest appropriate tools. Different types of systems require different approaches to testing.

Verification?

[RAMMS EIN]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took. I don't know if there are any verification tools for the programming language you are using (the ones I used were all for toy languages), but that was my first thought.

Re:Verification?

[ChefAndCoder]

Formal verification is one of those last vestiges (especially in academia) of the idea that it is possible to prove that software programs are correct. It's true for very small focused programs which do not have to interact with anything i.e. self contained. In practice, the effort required to formally prove that your software is correct is impractical. For "real" software systems, the number of unknowns introduced via API's, O/S and Hardware Interfaces, even just user interaction makes such "proofs" grow exponentially in complexity. If you are a software engineer, then you should realize that you will NEVER have the certainty of knowing something will work guaranteed every time. Formal verification is, frankly, a pipe dream.

Re:Verification?

[Raenex]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took.

In the real world very little software is formally verified. It's very expensive and time-consuming, and the vast majority of programmers don't know how to do it. Most projects just cannot afford it.

You're making too big a deal out of his "software engineer" label. There's no such thing as a software engineer, in the sense that for software there's nothing like the standards that engineers have to meet in other disciplines.

Re:Verification?

[Anonymous Coward]

I find it amusing that you justify your opposition to formal methods with a bunch of handwaving arguments. That's cute.

Re:Verification?

[Anonymous Coward]

Well, he's opposed to formal methods. He may not have gone into much detail, but he provided some rationale. You, the formal methods supporter presumably, justified your position with "That's cute". Amusing indeed.

Coverage tool (white box testing)

[gustgr]

[url]http://portal.acm.org/citation.cfm?id=1138929 .1138952&coll=GUIDE&dl=&type=series&idx=1138929&pa rt=Proceedings&WantType=Proceedings&title=Internat ional%20Conference%20on%20Software%20Engineering&C FID=15151515&CFTOKEN=6184618[/url]

Ahem... try this

[plopez]

Go to wikipedia, look up software testing. Click aon some of the referenced links, then chase those links and you will find a wealth of information.

Just google +"software testing" you will find tons of links.

you might try these guys: http://www.softwareqatest.com/
They have some nice links as well.

Or would you rather start with a newsgroup? Go to google news groups and seach on software testing again. Lots of knowledgable and helpful folks hang out on those boards.

BTW, a colorful phrase for "trying to test everything a user might do" is "soap opera testing". Google it, the story behind the name is sort of fun.

TDD and Continuous Integration, also hire testers!

[eegreg]

TDD - test driven development - means that you write tests before you write the code. Good TDD is actually BDD - behavior driven development. Write unit tests that are specifications for behavior. Continuous integration means that unit tests are automatically ran whenever code is integrated. Obviously this is just a matter of automation. Look for a test framework in the language you code in. There are lots of frameworks out there now. One way to whittle them down is to pick one that supports mock objects (or functions), which is a way of quickly simulating interactions. For Ruby, rspec is awesome. For C, I am trying out Cgreen right now- looks promising.

Practical resource

[uijltje]

STAF will support you in automating tests.
The effort to implement something will approximately double, but you'll have a continuously regression tested system/project.

...also...

[LouGrossi]

If you have the option: I do all of the testing that I can, but I also write in that in the event of an exception write to a log (standard) and then email to me an exception report. The exception report typically contains the user that hit it, the page/file that hit it, and method that returned the exception. In the morning I drink coffee and read fix up the exceptions. Luckily, I haven't received any lately (fingers crossed).