Slashdot Mirror
Can ISO 29119 Software Testing "Standard" Really Be a Standard?
New submitter yorgo writes The International Organization for Standardization (ISO) will soon publish part 4 of a 5 part series of software testing standards. According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation." However, many in the testing community are against it. Some wonder how the ISO/IEC/IEEE achieved consensus without their input. James Bach speculates that exclusion helped build consensus. Others, such as Iain McCowatt, argue that something as variable as software testing cannot be standardized, at all. And others believe that the motive behind the standards is not increased quality, but economic benefit, instead. Michael Bolton explains "rent-seeking" as he builds on James Christie's CAST 2014 presentation, "Standards – promoting quality or restricting competition?"
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
Considering ISO certification costs money and most companies refuse to spend money on stuff that isn't ISO 9001 then very few will probably get it. Most dedicated software houses I've encountered seem to prefer getting the TickIT certs instead of ISO certs for actual code certification but nobody bothers with the testing side of things.
Are rules for some and suggestions for the rest of us. The IEEE can put a standard on cleaning the toilet. If your company wants to follow it to the letter, or just use it as another reference, that's your call. I think the organization of conceptually difficult concepts is a good thing, overall. What we do with that is a whole other thing.
MBA CEO: I want our new product to be QA'd according to ISO 29119 before shipping.
Project Manager: Good idea, but that will add some time and overhead cost to my budget.
MBA CEO: Never mind, just ship it.
Scruting the inscrutable for over 50 years.
Michael Bolton explains "rent-seeking"
The no-talent ass clown known for his God-awful "music" or the one who hates him??
... According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation."...
If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.
In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167
This sounds like the 21st century version of 2167.
First of all. I HATE WRITING UNIT TESTS!!! Know what I hate more? When I get bit in the ass because something that did work before stopped.
Unit testing is step one in any decent software development and I will never enter into or manage another project without unit tests being a critical component of the project. I'll just hire a QA guy to unit test all my code... I don't want to do it haha.
Second, there is absolutely nothing which can't be automatically tested too. When you write code, GUI, Web, Command Line, message based, etc... An automated script to test the code is critical. There are tools for it.
Everything should be tested automatically... That even includes memory leaks when exiting. I would never hire someone even for a C#, Java, Python or even PHP position who doesn't write code which cleans up properly after itself (even if that means correct use of the garbage collector).
I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three.
So... Quit your bitching and write a test suite.
Koans and fables for the software engineer
While the idea of having a well vetted testing system in place that would allow customers to choose software that had been so vetted seems like a good idea, I have to wonder if it's doomed to the fate of SSL, at least outside of a few niche applications that mostly demand high levels of verification anyway.
With SSL, we all wanted the security; but everyone wanted it to be cheap, and wanted to avoid a monopoly over certificate authorityship. So, what did we get? A mass of CAs, many painfully shoddy, who will issue you a fancy looking cert for peanuts. Then we got "EV" certs, which are supposed to actually do what the original certs were supposed to do, only more expensive.
I mean, seriously, what kind of idiot thinks that have a standard for this will make any difference at all? Quality costs money and time end of story.
my blog of work misery - http://beastofbaystreet.com
The moment when someone with little real experience is tasked with designing a "testing methodology", they will look around the internet an choose an "industry standard". They have seriously tried to implement something like that at my current gig (due to gov regulations), and the results are hilarious. I wrote an application last week, this week I'm doing the design of the application, and I will hopefully have requirements by the end of the week. Yep, in that exact order...
Ohh, and they also forgot to have a real testing phase (as opposed to the validation phase), so now the Q&A people only do testing (in a clandestine way of course, since it's not part of the process) when they're not too busy documenting the process.
So the end result is a mix of a dysfunctional procedure and just plain sneaking things in under the radar to try to get anything done.
It seems as if their chief complaint is that they were not asked to provide input, and the personal communications with members of the committee didn't go anywhere. That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)... your organization (at least from the IEEE end, this is open to pretty much anybody that can muster up the nominal dues) signs up to be on the standards committee, you pay a nominal fee to be included in the working group, and Pow! Your organization is now a full voting member for the standard.
If you don't sign up for the working group, then it should be no surprise that your input is considered entirely optional and/or ignored entirely.
In the first article, the author describes a management course where a group was supposed to form a consensus. He complains that he disagreed with everyone else, wouldn't change his mind (because of his self-proclaimed "high-standards"), and was therefore excluded from the final output from the group, which then was reported to be a consensus. He disagreed that there was a consensus at all, since he didn't agree with it. That's not how "consensus" works; it does not mean that everybody will be satisfied with the outcome, or even want to be associated with it. He goes on to complain that the ISO process requires "consensus", but since he, and like-minded individuals, disagree with the standard, it should not be cleared as a standard.
Again, not how consensus works. In a consensus process, the majority approve of whatever the final output is, and the objections of the dissenters are noted and made available as part of the standards record. You can look on the website of pretty much any standards organization and access drafts, comments, meeting minutes, presentations, the whole works. This full record can help potential adopters of the standard decide if they want to utilize it or not.
ISO? Who are they?
“He’s not deformed, he’s just drunk!”
The Bolton-Christie argument, to me, boils down to: you can have too much of a good thing, e.g. documentation. This can impose unnecessary costs and defeat the purpose if, following the above example, onerous documentation doesn't get read. Too much of a standard means unnecessary cost goes out to the standards industry (rent seeking).
I guess the website's testing process failed to catch the edge case where someone wanted to navigate past the home page.
Whilst unit testing is a good thing in general it's actually hard to know how many tests to write. Whilst it is in theory possible to write tests for most things, it's completely infeasible to a priori write tests every for possible interaction with any reasonably sized bit of software. In practice everyone ships code with gaps in the test suite.
There are contexts in which we should be formally proving the correctness of code. This tends to be a niche thing tho' because it's hard and time-consuming.
You have to pay.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
The free dictionary (by Farlex) defines consensus: 1. An opinion or position reached by a group as a whole.
That's very democratic. Unfortunately, reality is not democratic.
Software testing is designed to unveil real vulnerabilities and errors in a complex system. Having a bunch of people hold up their hands and say, "Is this a problem?" is flatly ludicrous. In point of fact, it's the error that isn't noticed by the majority that constitutes the deepest problem. Remember the Columbia shuttle? A group of people got together and came to the concensus that the ice impact at launch was not a problem.
Testing, by it's very nature, is not subject to regimentation. It's a lot like "Job Descriptions" -- in real terms, establishing a job description is publishing a whole list of things that don't need to be addressed. Why does anyone think software testing will be different?
"Your piece of software has problems." "No, it doesn't. We fulfilled the standard for testing."
Giveth me a break.
Don't take life too seriously; it isn't permanent.
First 3 parts of 29119 cost $775, the total for all the parts when they are finished will probably be around $1200. This seems to be more about money than anything else.
You may have to extend it in some cases, but that's normal.
The Kruger Dunning explains most post on
The best known standard quip about standards itself has multiple versions and attributions. How meta:
"The nice thing about standards is that you have so many to choose from." - Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254
"The nicest thing about standards is that there are so many of them to choose from." -- Ken Olsen
“The wonderful thing about standards is that there are so many of them to choose from.” -- Grace Murray Hopper
See also:
Obligatory (but who set that standard?): xkcd : Standards
Why are there so many plugs and sockets?
‘Mediocrity finds safety in standardization.’ -- Frederick Crane
‘It is not enough that X be standard, it should also be good.’ -- Rob Pike (Window Systems Should Be Transparent)
The two above can be found on the cat -v page on standards"
"Standards are like toothbrushes. Everybody wants one but nobody wants to use anybody else’s." -- Connie Morella
Firstly, I'm not going to answer stupid leading questions. What is this, some kind of sound-bite-driven political debate?
If you don't like the way the standard is going, you form an organization of like-minded individuals and join the working group. Spread amongst a group of people, the costs are not that extreme, nor the commitment that dire.
And I don't the ability of education providers and consultants being able to advertise "We teach/use the XYZ Standard" as being some sort of nefarious plot. If you are looking for advice on something, being able to have a decent idea what you are going to learn about without extensive interrogation and negotiation can be quite useful for both parties.
The signing of an online petition is guaranteed to be utterly and completely ineffective... the costs of actual participation are so low, it's not entirely unjustified to ignore an online petition as anything other than an isolated group. There's a process to get heard on the issue, and internet petitions, and combative communications with those involved are not it.
I read that petition: "Because the people that signed this petition who couldn't be arsed to participate in the process disagree with the standard, consensus is not possible." How was THAT every going to go anywhere? "Consensus" is not reached (or not reached) by waiting until any schmoe with an axe is satisfied; consensus is reached when the committee votes on a standard and approves one. And yes, sometimes consensus cannot be reached, and the standard simply dies... that's a perfectly valid outcome too.
"It sometimes happens in a people amongst which various opinions prevail that the balance of the several parties is lost, and one of them obtains an irresistible preponderance, overpowers all obstacles, harasses its opponents, and appropriates all the resources of society to its own purposes. The vanquished citizens despair of success and they conceal their dissatisfaction in silence and in general apathy. The nation seems to be governed by a single principle, and the prevailing party assumes the credit of having restored peace and unanimity to the country. But this apparent unanimity is merely a cloak to alarming dissensions and perpetual opposition." - Alexis de Tocqueville, published 1831
*And I don't see how the ability*...
*ever*
*not reached (or failed to be reached)*
"Most of the people likely against it probably had or already have no intention of testing their shit software in the first place!" You are spectacularly wrong about that. Apparently you probably had or already have no intention of testing your theory; not even by the most cursory reading of any of the materials referred to at the beginning of this thread.
Most people don't understand the compliance. There's good and bad, but there's no going back once your industry (candle makers, software writers, barbers, whoever) adapts a standard it invariably becomes a tool of an authority.
Good: What I like about it is that our certifications increase accountability by encouraging recording mistakes. The "routine" of flagging mistakes and finding root causes and formalizing "corrective and preventative action" has been good and improved our company.
Bad: These standards are adapted by many companies in order to reduce competition, take away via consensus unique individual methods for doing things. They become almost like a "union", punishing individual innovation via auditors that view the world inside a "box". Uniqueness and innovation are an increased cost and risk to the third party auditor, and the auditor is ready to adapt the majority interpretation - which is usually to increase barrier of entry into the field of competiton.
As Morris Kleiner, the AFL-CIO chair in labor policy at the Humphrey School of Public Affairs at the University of Minnesota, put it "Occupational licensing has either no impact or even a negative impact on the quality of services provided to customers by members of the regulated occupation."
Gently reply
I agree. Please help us in a tiny way by signing the petition. http://www.ipetitions.com/peti...
First, I refused to answer those questions not because I bow down to suspect authority. (Where did THAT come from?) I refused to answer them because they are stupid. Of course the answer is "no", but that answer establishes nothing, because the questions imply conclusions that themselves are up for debate. If this is the way you and your like-minded compatriots engage in debate, no wonder you can't get anybody to take you seriously.
And I keep seeing this claim that the ISO process is set up to "suppress opposition and dissent"... but the only specifics ever mentioned is that the process takes a while and has meetings outside North America. Is that the best you can come up with?
To answer some of your points:
- Most technical standards organizations are private. IEEE, ANSI, SAE, IEC, UL, IIHS, IETF, W3C, ASTM, etc., are all private. (Not to mention tech-specific ones like the ones over Compact Flash, USB, Infiniband, etc.) Arguing that the ISO is illegitimate merely because it isn't a government agency is not likely to be persuasive. (And as a side note, the ISO was set up at the behest of the UN and is tightly coupled with them... it's not a UN agency like the ITU, but it might as well be.)
- I'm not aware of any standards organization (professional societies like the SAE and IEEE included) where standards are put out to vote by "a representative slice of practitioners"; they are all voted on by those that chose to participate in the standards process.
- Arguing that it's controlled by a bunch of money-grubbing consultants and trainers AND that it's rammed through by attrition is a contradiction. Stretching out the process indefinitely is exactly opposite to the goal of making money off the standard, since nobody makes money off a standard that doesn't exist.
- It's not deliberate attrition just because it takes longer than you'd like.
- Yes, the burden is on the ISO to demonstrate relevance of its standards. But once they have successfully done so to an organization and that organization comes to you for your services, objecting with nothing more than you "shouldn't have to defend yourself" is not likely to get you hired.
- If such a significant number of professional testers object to the contents of the standard, why could they not scrape up the funds necessary to participate in the standard? Even with meetings held in distant locales, on a per-person basis, it doesn't come out to much. (And could you not find any software testers in India, Japan, etc. that are like-minded to save on travel expenses?) If you want to be taken seriously, you gotta put your money where your mouth is...
- Again, consensus doesn't mean "everyone agrees that they can live with the content", it means that a majority (or super-majority, depending on the rules) approve of the content. If a single "no" vote could prevent a standard from being approved, we'd never have any standards at all (imagine if the Ethernet standard could have been completely halted by IBM signing up for the IEEE committee and voting "no" so it could push Token Ring instead.) This is not a difficult concept to understand, nor does it rise to "suppressing dissent".
- Like it or not, refusing to participate in the standards process does not bode well for arguing that there's "significant" objections to the standard, since those objectors could not be bothered to show up when it came to deciding on the content. (An online petition? Seriously? That's supposed to persuade anybody?)
That is a damn fine blog post. Very well said...