Can ISO 29119 Software Testing "Standard" Really Be a Standard?

New submitter yorgo writes The International Organization for Standardization (ISO) will soon publish part 4 of a 5 part series of software testing standards. According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation." However, many in the testing community are against it. Some wonder how the ISO/IEC/IEEE achieved consensus without their input. James Bach speculates that exclusion helped build consensus. Others, such as Iain McCowatt, argue that something as variable as software testing cannot be standardized, at all. And others believe that the motive behind the standards is not increased quality, but economic benefit, instead. Michael Bolton explains "rent-seeking" as he builds on James Christie's CAST 2014 presentation, "Standards – promoting quality or restricting competition?"

A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.

So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?

Comments

Standards

[Crizzam]

Are rules for some and suggestions for the rest of us. The IEEE can put a standard on cleaning the toilet. If your company wants to follow it to the letter, or just use it as another reference, that's your call. I think the organization of conceptually difficult concepts is a good thing, overall. What we do with that is a whole other thing.

Re:Standards

[JeffAtl]

It's not quite that simple - customers can require the certification. If a customer is sizable enough, the suppliers will have comply to stay in business.

For real world examples, look at the power that the state of Texas has wielded in text book requirements and the state of California with warning labels.

Can see it now:

[some old guy]

MBA CEO: I want our new product to be QA'd according to ISO 29119 before shipping.

Project Manager: Good idea, but that will add some time and overhead cost to my budget.

MBA CEO: Never mind, just ship it.

Re:Can see it now:

[UnderCoverPenguin]

MBA CEO: Never mind, just ship it.

More likely response: "Figure out how to get it done within the existing budget and schedule."

Re:Can see it now:

[tomhath]

Even more likely response: "Have the secretary fill out the paperwork and change our website to say we're ISO 29119 compliant".

Re:Can see it now:

[mariox19]

And if anything goes wrong, we'll crucify the QA people that we never gave enough testing time to in the first place.

Re:Can see it now:

[awebb]

Amen to that! No one gives a shit about QA or the 100 issues they find during testing. It's the 1 issue that slips out that causes a riot.

Which Michael Bolton?

[Type44Q]

Michael Bolton explains "rent-seeking"

The no-talent ass clown known for his God-awful "music" or the one who hates him??

Re:Which Michael Bolton?

[michaelbolton]

Ah. Another primitive attempt at what some of your fellow Earthlings call "humour". Keep practicing! ---Michael Bolton

Re:Which Michael Bolton?

[ESD]

As this particular thread is off-topic anyway: Thanks for your blog, Mr. Bolton! It's always an interesting read and I try to point new testers to it at least once to get some exposure to a usually unknown view on testing.

Wrong focus?

[QuietLagoon]

... According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation."...

If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.

Re:Wrong focus?

[TechyImmigrant]

This shit matters if you are selling things internationally. It makes it difficulty for country A to refuse imports of county B's software because it wasn't tested to their local testing standard. If it was tested to an ISO standard it'll do and the WTO will be on their ass if they try to claim otherwise.

That doesn't mean the specs are any good. They aren't. But there is a very real interaction with international trade regulations.

Re:Wrong focus?

[sinij]

You are incorrect. ISO standards are recognized by multiple countries. You can Google the full list.
 
The main advantage of any kind ISO is that it enables your software to get on government procurement lists. This doesn't impact small shops, they don't generally have resources or organizational maturity to sell to governments. There are multiple international, proprietary, and country-specific standards (e.g. ISO, FIPS, Common Criteria, PCI-DSS). International means you only have to go through certification once, not once per country that you were doing before the standard was adopted.

Re:Wrong focus?

[michaelbolton]

Lots of competent testers point out that compliance with ISO 29119 does not in the least establish a baseline level of competence. Your argument "even ineffective standards testing with associated increased overhead his better than a status quo" is basically equivalent to saying "friendly fire is better than not shooting" or "ineffective medicine with severe side effects is better than no medicine at all".

Shades of 2167

[Snotnose]

In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167

This sounds like the 21st century version of 2167.

Re:Shades of 2167

[luis_a_espinal]

In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167 This sounds like the 21st century version of 2167.

MIL SPEC 2167, iirc, deals with documentation and deliverables. The actual software development process was "guidelined" by some other MIL SPEC. With that said, those were supposed to act as guidelines for a waterfall methodology (which surprisingly, it can actually be used in some contexts, or subverted into a spiral/iterative process.)

I worked at a defense contractor not long ago, and alas, the software projects were horribly run. But I always saw that it was the execution, not the specs per say that was the culprit for each and every single snafu/fubar hybrid I encountered. That and management, and life-long-career jockeys from the punch-card era dictating how to write software, and department infighting.

It's just like CMM/CMMI - A CMMI level X or Y only indicates that, to some degree, an organization a) has a formal process, and b) follows such process.

It doesn't indicate that the process is good - it doesn't even guarantee that *it is not shit*.

What it does, is that it helps an organization guarantee that its constituent parts know what activities to do under what circumstances and tasks in a business lifecycle. And that helps an organization improve things (assuming said organization has the necessary technical wherewithal and culture.)

In private business and with defense contractors, it is the companies who fail to execute (let's think of how many companies ditch source control to become *agile*!) particular practices. Defense contractors have a lot of leeway in negotiating and tailoring the actual execution of a process. Typically, they do not do it because they suck (and for a lot of other political and financial motivation$.)

Automated test in is a minimum

[LostMyBeaver]

First of all. I HATE WRITING UNIT TESTS!!! Know what I hate more? When I get bit in the ass because something that did work before stopped.

Unit testing is step one in any decent software development and I will never enter into or manage another project without unit tests being a critical component of the project. I'll just hire a QA guy to unit test all my code... I don't want to do it haha.

Second, there is absolutely nothing which can't be automatically tested too. When you write code, GUI, Web, Command Line, message based, etc... An automated script to test the code is critical. There are tools for it.

Everything should be tested automatically... That even includes memory leaks when exiting. I would never hire someone even for a C#, Java, Python or even PHP position who doesn't write code which cleans up properly after itself (even if that means correct use of the garbage collector).

I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three.

So... Quit your bitching and write a test suite.

Re:Automated test in is a minimum

[omglolbah]

You would love the control system software we use at work... (world leading platform for the industry).

No revision control. You have 'current' revision. That is it.

Integrated code editor that has no syntax highlighting.

Patches to the system will break components that are not considered 'core'. Which forces updates of ALL components in the system. This has lead to bugs persisting at sites for years with no patch because nobody wants to fix bugs when it costs tens of millions of dollars to do so.

No automatic testing. Of anything. When we update a component everything has to be tested manually. Someone will sit for 2 weeks and check every state of GUI symbols for the whole HMI. Oh joy...

If you change ANYTHING in code, you can no longer connect to controllers to view live data. You need to do a download to the control with the code changes before live data can be observed. This means that as soon as you make changes, you lose the ability to view the old code running. There is no way to have both a 'online capable' version of the code and a changed codebase in the same control system. We operate two separate virtual environments and switch VLANs or just move a cat6 when testing...

This is for oil rig control systems. There is no automated testing of any kind, even for critical emergency shutdown systems. Every test is done manually.
The ESD systems are usually a complex matrix of casues and effects with hundreds of inputs, hundreds of outputs... This is all tested manually as the software does not support any reasonable simulation of the controller input/output systems.

Enjoy that little gem.

Re:Automated test in is a minimum

[swillden]

In a static typed language unit tests are pretty pointless.

Because static typing catches all bugs? That must be some statically-typed language that I've never seen. Unit tests are perhaps marginally less necessary than in dynamically-typed languages, but they're still necessary. Test-Driven Development is a life saver regardless of your toolset.

I can write you in 40 lines a function which you wont be able to automatically test. It only needs some nested loops and an 'if' cascades with loops inside.

There's nothing untestable about such a function. Basic code coverage tools will even identify any branches within the code that aren't taken, so you know to look for ways to write test cases that cover those branches. What's harder is ensuring coverage of all of the issues that could be provoked by different input data, even after you've covered all of the paths. With experience you learn to do that quite effectively, though.

Sure: you should refactor that into a lot of small functions, containing only a single loop or a single 'if'.

FTFY. Change it to "must" if I'm your code reviewer.

You lost quite some credibility with using terms or sentences like That even includes memory leaks when exiting. and "... even if that means correct use of the garbage collector ..." Unfortunately a exiting program can not leak memory

Sure it can. If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak. Again, there are good tools to help you find these leaks, like valgrind memcheck.

you don't use a garbage collector, it runs in the background. You can parametrize it perhaps ... but thats it. (and please don't tell me you are doing System.gc() in Java programs at "random" intervals)

And yet you can still have leaks in garbage-collected environments, and there are ways to test for them. It's a bit more complex than in non-GC'd environments, but it can -- and must! -- be tested if you want to have reliable software.

Re:Automated test in is a minimum

[Whatsisname]

That's not going to work, because you'll never be able to economically write a requirements document so complete that the behavior is so well defined that you can get meaningful test coverage from it.

To get that kind of completeness you'd have to code the entire software in MSWord, which is a terrible programming language, and without ever testing it along the way.

Testing needs to be a continuous process as part of software development, not something that happens parallel or afterwards.

Re:Automated test in is a minimum

[Just Some Guy]

I love testing. Honestly, if forced at gunpoint to give up testing or version control, I'd be hard pressed to pick. Testing means I can update an innocent-looking line of code without worrying whether it's going to break some arcane business logic that a customer depends on, and that I can gut and reimplement an API while having a reasonable expectation that consumers will keep working afterward. I'm a huge testing advocate.

But.

I have no desire whatsoever to conform to an ISO document about their idea of the right way to do things. Standards are great for ensuring interoperability, but how often do you care about that in a test suite (beyond the minimal "can I make its output look like JUnit so Jenkins can yell at someone when it breaks")?

I'll sit down next to you and help you crank out a comprehensive test suite, because while that's a tedious pain in the ass, it's far less of a pain in the ass than not having one. But I couldn't possibly care less whether the result of our labors is "ISO 29119 Certified (tm)!", probably by a $300 outside consultant, so that we can put an "ISO 29119 Certified (tm)!" logo on our website for PHB's to smile approvingly and longingly at and then forget about.

Re:Automated test in is a minimum

[Your.Master]

He's saying you can detect runtime memory leaks at exit time.

The memory won't be leaked anymore post-exit, and the leak doesn't matter at exit time, but that's when it is possible to deterministically detect memory leaks over the course of the software running.

Re:Automated test in is a minimum

[uncqual]

Does the average hobbyist dev have the skill set/ mind set of a tester? No.

FTFY.

If a developer doesn't have the mindset of a tester, perhaps they should find something else to do (maybe the local Walmart needs a greeter?) or be assigned support "on-call" more regularly so they can experience being waken at 3AM due to someone failing to design, implement, and test well. If someone doesn't have the mindset of a tester and their title is something like Senior Software Developer or higher, there is a serious title inflation problem in their workplace.

Design and implementation decisions, the classic "developer" roles, impact testing. Someone who doesn't grok and appreciate testing has no business playing a significant role in design. In my experience, one of the reasons that unit testing is sometimes quite hard (sometimes updating the unit tests takes much more effort than the corresponding modest change to the base code took) is that systems are rarely designed to be easily unit testable - commonly there's a "shadow" unit testing world of hacks layered on hacks relying on side effects of this subsystem or that subsystem that is cobbled together and labeled something like "the unit test environment".

When a developer commits code, they should be sure it works and take it personally (i.e., as a personal failing) and use it as a learning experience if the code fails either due to a design/coding error within the scope of the code (generally extending throughout the entire subsystem even if just one line in one module was changed) or misuse of some external service/API. Failures that arise from inter-subsystem interactions due to ambiguous specifications or not yet understood interactions are what testers should be finding (and the architects should usually be held accountable for those failures).

As well, you want developers to have a mindset of testing because they are then the most likely to develop and adapt tools that actually make testing a repeatable and low cost process.

Re:Automated test in is a minimum

[uncqual]

Asserts are fairly useless as a testing tool. Their excessive use can be extremely annoying as they clutter up the code and that clutter can increase the chances of introducing bugs.

However, responsible use of asserts can be quite useful in debugging - esp. when a customer is encountering a problem that is difficult to recreate (or, perhaps, nearly impossible from a practical standpoint because they won't give you access to their confidential data/environment). In those cases, running a "well asserted" debug version of the system on the customer site can sometimes help narrow the problem space substantially very quickly - not always of course, but the cost of running the test is often very small so it has a good ROI. Of course, if you're going to do this, you need to keep the asserts valid so systems test often needs to validate both the version WITH and the version WITHOUT the asserts enabled - else it's likely that the "well asserted" version has rotted and will result in a flurry of assert failures that actually don't represent a problem.

Of course if you're writing code that controls the United States nuclear weapon launch system, ignore the advice above.

Re:Automated test in is a minimum

[swillden]

Rofl. Acacademic half knowledge with complete wrongs

Dude. I've been a professional software developer for 25 years, and am a respected engineer at one of the biggest names in software. You use my software daily.

If there are any heap-allocated blocks remaining (not freed) at exit, the program has a memory leak No it has not. Where should the memory go to after the program has exited?

Well, obviously it'll all be returned to the system after exit. The point is to check AT EXIT. If there are any blocks still allocated, you've screwed up.

Technically you can test a 40 lines mess of loops and if cascades, practicaly you can't ... or how likely is it that you can prove me in a reasonable time that a certain branch in an if - else inside of a cascade of nested loops and if's is executed with meaningfull data in your test? Especially if I have written the function and you want to write the test?

I've done it many times. Just check to see which branches aren't being executed and work out what needs to be done to execute them.

Though it's much, much better to refactor the function first.

The rest I leave as it stands if you like to argue about how likely it is that a single compilation unit has a bug that is not dicovered by a functional user acceptance test ... unit tests without user acceptance tests or integration tests are pointless.

I've found thousands of bugs with unit tests that were not discovered by functional tests, integration tests, or user acceptance tests. In fact, unit tests are the most likely ones to find thing like subtle boundary condition errors which are hard to reproduce and are the source of nasty, rare bugs that are seen in production only once in a while.

The next thing is you tell me to test getters and setters ...

Typically those get exercised by integration tests... and it is a good idea to have coverage of them at some point in your automated suite, because the whole point of getters and setters is that they may someday become non-trivial. Writing tests for them at that point isn't as good as already having tests in place, because you want to verify that your new implementation doesn't have subtle differences in behavior.

But you will figure that soon enough when you have 100% code coverage and still have bugs and wonder why

No one is claiming that unit tests are sufficient, only that they're necessary.

Btw, I never was in a team that had a memory leak in a GCed language.

Then you haven't worked on many non-trivial systems.

Re:Automated test in is a minimum

[Your.Master]

No it is not :) how should that work?

The memory allocator can keep track of all the memory that is allocated (maybe you do this with a special build, maybe your default allocator does this).

Provided your heap isn't actually corrupt, (which tooling can also help detect with things like canary words and on special builds with extreme one-page-per-allocation policies, but is not totally detectable), your allocator can have a record of every allocation that does not have a corresponding free by the time of exit, no matter whether it's because of reference cycling or just a missing call to delete or what-have-you. It can even tag it with the allocation callstack and what-have-you.

It's also true that it won't detect obscene memory usage that isn't actually leaked, eg. suppose Firefox leaked on every tab close, but it managed to clean itself up after the whole window closed. That's not strictly a leak but it's just as bad in practice.

It's essentially like having a garbage collector that isn't there to collect garbage, but is there to identify garbage that has been left uncollected.

you programmed the whole application agnostic of memory allocation and then you want to travel all objects and set all references to null (and somehow trigger the GC) to have everything 'cleared'? (Sure, I can write a reflection based graph traversal that sets every reference it encounters to null. But what would be the point?)

No, I don't want it cleared, I want it detected so we can notify the owner of the leaking component that it was not cleared at the proper time.

We aren't detecting leaked memory in order to free it. We're detecting leaked memory in order to fix the bug where it leaked in the first place, possibly hours or days ago, and has been consuming your RAM for no good reason since that time.

Re:Automated test in is a minimum

[thegarbz]

Sounds like you have a poor choice of control system.

Our control system is the exact opposite:

Control graphics run locally on the workstations which means they can be tested away from the control room with live equipment and then push the completed graphics to other stations.
Loop simulation can be performed online for changes to control system parameters. Feed it a bit of historical data so it builds a model, then update the parameters and simulate, then push the result back out to the control processor.
Complete simulation systems available for both our ESD system and our control system. On the ESD system the checking process is still manual but that's exactly what the standards require, manual and comprehensive function check of every change, which is why we typically delay changes to the ESD system until maintenance periods so we can play without without causing unit upsets.

There's actually a lot of neat progress happening in control systems at the moment, it's a shame that in our industry the upgrade cycles are so slow. Especially in finite time installations like oil rigs. It's build once, and operate until the last drop is out. Process plants are a bit better but painfully slow. We are about to upgrade our last operator station from Solaris to XP, shame the vendor has already EOLed that product we are about to install.

Re:Automated test in is a minimum

[swillden]

I'm not interested in educating people who don't want to be educated. Just please tell me what software you write, so I can avoid it.

This sounds... familiar....

[QilessQi]

By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.

....Be the first on your block to collect all five! GOTTA CATCH 'EM ALL!

just like ISO 9000, that worked well!

[thebeastofbaystreet]

I mean, seriously, what kind of idiot thinks that have a standard for this will make any difference at all? Quality costs money and time end of story.

I see a bunch of whiners

[sirwired]

It seems as if their chief complaint is that they were not asked to provide input, and the personal communications with members of the committee didn't go anywhere. That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)... your organization (at least from the IEEE end, this is open to pretty much anybody that can muster up the nominal dues) signs up to be on the standards committee, you pay a nominal fee to be included in the working group, and Pow! Your organization is now a full voting member for the standard.

If you don't sign up for the working group, then it should be no surprise that your input is considered entirely optional and/or ignored entirely.

In the first article, the author describes a management course where a group was supposed to form a consensus. He complains that he disagreed with everyone else, wouldn't change his mind (because of his self-proclaimed "high-standards"), and was therefore excluded from the final output from the group, which then was reported to be a consensus. He disagreed that there was a consensus at all, since he didn't agree with it. That's not how "consensus" works; it does not mean that everybody will be satisfied with the outcome, or even want to be associated with it. He goes on to complain that the ISO process requires "consensus", but since he, and like-minded individuals, disagree with the standard, it should not be cleared as a standard.

Again, not how consensus works. In a consensus process, the majority approve of whatever the final output is, and the objections of the dissenters are noted and made available as part of the standards record. You can look on the website of pretty much any standards organization and access drafts, comments, meeting minutes, presentations, the whole works. This full record can help potential adopters of the standard decide if they want to utilize it or not.

Re:I see a bunch of whiners

[michaelbolton]

"After complaining about click-baiting laden titles, in your own writing at the link you provided, that's a pretty ironic statement." It's not a statement. It's a question, rhetorically delivered, that addresses a legitimate concern. Here's a fun game for the whole family: go to LinkedIn, and do a search for the string "29119". Note the preponderance of consultancies that offer services in interpreting and explaining 29119. Have a look at the constituency of the working group; note the overlap between those companies and those who are enthusiastic suppliers to the ISTQB certification mills. Have a look at the minutes of the meetings of the working group, and look for phrases like "marketing the standard" (rather than, say publicizing the standard). Are you really trying to claim that your business and yourself have no financial motivation in your actions? Or do you do all of your consulting gratis, merely on principle alone? That's another FAQ. “In one sense, it won’t make any difference to my business if 29119-1, 29119-2, and 29119-3 are left to stand, and if 29119-4 and 29119-5 move from draft to accepted. Rapid Software Testing is about actual testing skills—exploration, experimentation, critical thinking, scientific thinking, articulate reporting, and so forth. That doesn’t compete with 29119, in the same kind of way that a fish restaurant doesn’t compete with the companies that make canned tuna. We object to people manipulating the market and the ISO standards development process to suggest to the wider world that canned tuna is the only food fit for people to eat. I discuss that here: http://www.developsense.com/bl... “In another sense, 29119 could be fantastic for my business. It would offer me a way to extend the brand: how to do excellent, cost-effective testing that stands up to scrutiny in contexts where some bureaucrat, a long way away from the development project, was fooled into believing that 29119 was important. At the moment, I’m happy to refer that kind of business to colleagues of mine, but I suspect that it would be something of a gold mine for me. Yet still I oppose 29119, because what’s in my interest may not be in the interests of my clients and of society at large. “Let me be specific: There are existing standards for medical devices, for avionics, and the like. Those standards matter, and many of them are concise and well-written, and were created by genuine collaboration among interested parties. Testers who are working on medical devices or on avionics software have a limited number of minutes in the working day. As someone who flies a lot, and as someone who is likely to require the help of medical devices in the foreseeable future, I would prefer that those testers spend as many minutes as humanly possible actually investigating the software, rather than complying (authentically, pathetically, or maliciously) to an unnecessary standard for process modeling, documentation, and strategizing (a standard for developing a strategy—imagine that!). "You are free to ignore this standard" Yes, of course I am... until it creeps into regulation as the NIST points out in the second-last paragraph here: http://www.nist.gov/standardsg... ---Michael B.

Can it be a *useful* standard

[g01d4]

The Bolton-Christie argument, to me, boils down to: you can have too much of a good thing, e.g. documentation. This can impose unnecessary costs and defeat the purpose if, following the above example, onerous documentation doesn't get read. Too much of a standard means unnecessary cost goes out to the standards industry (rent seeking).

Re:Can it be a *useful* standard

[michaelbolton]

That's certainly an important point. There are others. To me, the issue not just the cost of preparing the documentation, but the degree to which compliance with the standard displaces the goal of actually testing the product or service. A moment that a tester spends on useless documentation is a moment in which she's not focused on identifying risks and finding problems that would cause loss, harm, or annoyance. http://www.developsense.com/bl...

On standards

[mr_mischief]

The best known standard quip about standards itself has multiple versions and attributions. How meta:

"The nice thing about standards is that you have so many to choose from." - Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254

"The nicest thing about standards is that there are so many of them to choose from." -- Ken Olsen

“The wonderful thing about standards is that there are so many of them to choose from.” -- Grace Murray Hopper

See also:

Obligatory (but who set that standard?): xkcd : Standards
Why are there so many plugs and sockets?

‘Mediocrity finds safety in standardization.’ -- Frederick Crane
‘It is not enough that X be standard, it should also be good.’ -- Rob Pike (Window Systems Should Be Transparent)
The two above can be found on the cat -v page on standards"
"Standards are like toothbrushes. Everybody wants one but nobody wants to use anybody else’s." -- Connie Morella

Re:Who cares

[Jane Q. Public]

I also suspect that this "standard" would be unworkable with some established de facto industry standards that aren't likely to change.