Too Darned Big to Test?

gManZboy writes "In part 2 of its special report on Quality Assurance (part 1) Queue magazine is running an article from Keith Stobie, a test architect in Microsoft's XML Web Services group, about the challenges one faces in trying to test against large codebases."

Comments

Re:Shouldn't that be too bloated to test?

[gosand]

But this is exactly what happens in big software houses. The pressure to release ahead of your competition and stay ahead (or catch up with) the perceived feature curve is huge. Delays are bad -- delays equal lost sales. And once the product is done, unlike a bridge or a plane or a shuttle which will last 20 - 30 years or more as is, that software immediately starts getting new features and major modifications for "the next version".

This is not always the case. I just left a very large company for a smaller one, and I have been doing software testing for 11 years. I have worked for two very large companies in my career, and two small ones. In the large ones, I learned most of what good testing was about. I also learned most of what I know about the development process, and how it should be done. Unfortunately, at both of those companies, they talked a good game but didn't deliver very well.

When it comes to software projects, you have 4 factors:

Schedule

Cost

Quality

Features

The rule is, you get to optimize one of these, are constrained by one, and you have to accept the other two. Everyone always thinks that they can get around this somehow, but it never works out. Oh, and you have to make these choices when you start the project - if you change them mid-stream it changes the game.

NASA was used as an example. They are constrained by features and want to optimize quality. Therefore, it costs what it costs and you get it when you get it. Most big software houses are constrained by schedule and want to optimize features. That means they throw money at it and take whatever quality they get. Until they bitch about the quality. If only they really understood this. I presented this to my manager, and he said "But cost is free, because everyone is salaried and can just work overtime." He was serious. Do you wonder why I left?

We always thought we were constrained by schedule because every single release, some manager would say "This is the release date, and it is not moving!" It would move EVERY SINGLE RELEASE. For 4 years, we never hit a release date. Of course, we thought we did because we kept moving it during the cycle. Once, we delivered the release 1 year late - but it was on time according to our re-evaluation. Phbbbt. We did software for hospitals, and it wasn't that big of a deal if we missed our release date. These were huge inventory systems, and it took months for them to deploy. They had to be signed off by Beta sites before it could even be made available to everyone, and even then nobody just bought it off the shelf. We had to go in, install it in their test environments, train them on it, and set up transition dates. And we had to schedule it all within their budget constraints. So time to market wasn't nearly as big of an issue as it is in small companies, where if you don't deliver in a week or two, you can really hurt the company.

I guess my point to all of this is that there are good QA and testing practices, but they might not apply to all situations. The key is knowing when to apply what. If I tried to apply Quality Assurance to where I am now, it would be a total waste of effort. The same goes for testing methodology. (they are NOT even remotely the same things you know) Our build schedules at the big company were every 2 weeks. Where I am now, we do at least 4 releases of software in that time. But it is hosted software, so it is a totally different animal. I value my time at large companies, I learned how things work and don't work in the QA and software testing arenas. The good part is, there is still more out there to learn.