Slashdot Mirror
Can ISO 29119 Software Testing "Standard" Really Be a Standard?
New submitter yorgo writes The International Organization for Standardization (ISO) will soon publish part 4 of a 5 part series of software testing standards. According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation." However, many in the testing community are against it. Some wonder how the ISO/IEC/IEEE achieved consensus without their input. James Bach speculates that exclusion helped build consensus. Others, such as Iain McCowatt, argue that something as variable as software testing cannot be standardized, at all. And others believe that the motive behind the standards is not increased quality, but economic benefit, instead. Michael Bolton explains "rent-seeking" as he builds on James Christie's CAST 2014 presentation, "Standards – promoting quality or restricting competition?"
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
A comprehensive list of many other arguments, viewpoints, and information has been collected by Huib Schoots. Opponents of ISO 29119 have even started a petition aimed at suspending publication of the standard. Even so, this might be an losing battle. Gil Zilberfeld thinks that companies will take the path of least resistance and accept ISO 29119.
So, where do you stand? What constitutes a consensus? Can a standard be honored without consensus? Can an inherently sapient activity, such as testing, be standardized, at all? What is the real purpose of a standard? Will companies acquiesce and adopt the standard without question?
Considering ISO certification costs money and most companies refuse to spend money on stuff that isn't ISO 9001 then very few will probably get it. Most dedicated software houses I've encountered seem to prefer getting the TickIT certs instead of ISO certs for actual code certification but nobody bothers with the testing side of things.
Are rules for some and suggestions for the rest of us. The IEEE can put a standard on cleaning the toilet. If your company wants to follow it to the letter, or just use it as another reference, that's your call. I think the organization of conceptually difficult concepts is a good thing, overall. What we do with that is a whole other thing.
MBA CEO: I want our new product to be QA'd according to ISO 29119 before shipping.
Project Manager: Good idea, but that will add some time and overhead cost to my budget.
MBA CEO: Never mind, just ship it.
Scruting the inscrutable for over 50 years.
Michael Bolton explains "rent-seeking"
The no-talent ass clown known for his God-awful "music" or the one who hates him??
The job still sucks and everyone hates you.
... According to the website, "ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation."...
If I were to jump upon a standard for testing software, the fact that it is "internationally agreed" is way down on the requirements, yet it seems to be mentioned as the main feature here.
In the late 80s and early 90s I was involved in 2 projects run under MIL SPEC 2167, which was supposed to ensure product quality. Both were epic disasters. IMHO, 2167 pretty much guaranteed mediocre at best software, taking 3x longer to do, at a cost at least 6x of non-2167
This sounds like the 21st century version of 2167.
First of all. I HATE WRITING UNIT TESTS!!! Know what I hate more? When I get bit in the ass because something that did work before stopped.
Unit testing is step one in any decent software development and I will never enter into or manage another project without unit tests being a critical component of the project. I'll just hire a QA guy to unit test all my code... I don't want to do it haha.
Second, there is absolutely nothing which can't be automatically tested too. When you write code, GUI, Web, Command Line, message based, etc... An automated script to test the code is critical. There are tools for it.
Everything should be tested automatically... That even includes memory leaks when exiting. I would never hire someone even for a C#, Java, Python or even PHP position who doesn't write code which cleans up properly after itself (even if that means correct use of the garbage collector).
I have worked on several multi-million line commercial applications, some with 500 million+ users. I have never seen a piece of code which could not be properly tested using the right tools. That can even include small embedded systems where we would have to actually implement a QEMU module or three.
So... Quit your bitching and write a test suite.
Koans and fables for the software engineer
While the idea of having a well vetted testing system in place that would allow customers to choose software that had been so vetted seems like a good idea, I have to wonder if it's doomed to the fate of SSL, at least outside of a few niche applications that mostly demand high levels of verification anyway.
With SSL, we all wanted the security; but everyone wanted it to be cheap, and wanted to avoid a monopoly over certificate authorityship. So, what did we get? A mass of CAs, many painfully shoddy, who will issue you a fancy looking cert for peanuts. Then we got "EV" certs, which are supposed to actually do what the original certs were supposed to do, only more expensive.
The point of a standard is to pick a convention and build on it. Screw threads are standardized. Resistor color codes and values are standardized.
That's a good standard.
The part that makes me boil is where standards committees attempt to codify what a manager does by writing management standards. Most of them are overwrought variants of Plan, Do, Check, Act. DO WE REALLY NEED ANOTHER STANDARD LIKE THIS?
This kind of foolishness has gone way past the point of humor, past the point of ridicule, past the just plain stupid, and gone to just awfully sad.
Make It STOP!
THe link in the article points to: http://www.softwaretestingstandard.org/
THe links there only have tables of contents:
http://www.softwaretestingstandard.org/part1.php
Where is the actual content?
I mean, seriously, what kind of idiot thinks that have a standard for this will make any difference at all? Quality costs money and time end of story.
my blog of work misery - http://beastofbaystreet.com
Most of the people likely against it probably had or already have no intention of testing their shit software in the first place!
See: pretty much every website that runs PHP, most Microsoft products for some large examples
Google too. Goddamn Google.
A company so huge they can't even make non-corruptable file formats. That was solved decades ago.
I gave up after Picasa corrupted its database for a 4th time.
Google Chrome, regularly screws its own installations up. I've had it happen to me. I've had it happen to family. My friends have had it happen to them.
I've been programming from 9, and my friends mostly work in software and networking as well, and I can't say I have seen a worse program than Chrome, besides maybe Skype. (AND I STILL USE THEM)
Skype v5 had an almost instant BSOD if there was anything even remotely intense running while you answer a call. The videoframe wouldn't be loaded and it would try to draw to it... It was still there up until even Microsoft bought them. (oddly enough, the MSN Developers were actually the good devs in Microsoft, and the MSDN people for the most part)
What. The. Hell. Are. You. Doing. Developers. ?. That was totally accidental. OR WAS IT?!
There will almost certainly be a few legit reasons for people being against it, however.
One of them being "if you aren't certified, it must mean you are crap".
The moment when someone with little real experience is tasked with designing a "testing methodology", they will look around the internet an choose an "industry standard". They have seriously tried to implement something like that at my current gig (due to gov regulations), and the results are hilarious. I wrote an application last week, this week I'm doing the design of the application, and I will hopefully have requirements by the end of the week. Yep, in that exact order...
Ohh, and they also forgot to have a real testing phase (as opposed to the validation phase), so now the Q&A people only do testing (in a clandestine way of course, since it's not part of the process) when they're not too busy documenting the process.
So the end result is a mix of a dysfunctional procedure and just plain sneaking things in under the radar to try to get anything done.
It seems as if their chief complaint is that they were not asked to provide input, and the personal communications with members of the committee didn't go anywhere. That's not how the standards process works (I'm speaking from the IEEE perspective, anyway; don't know how ISO works)... your organization (at least from the IEEE end, this is open to pretty much anybody that can muster up the nominal dues) signs up to be on the standards committee, you pay a nominal fee to be included in the working group, and Pow! Your organization is now a full voting member for the standard.
If you don't sign up for the working group, then it should be no surprise that your input is considered entirely optional and/or ignored entirely.
In the first article, the author describes a management course where a group was supposed to form a consensus. He complains that he disagreed with everyone else, wouldn't change his mind (because of his self-proclaimed "high-standards"), and was therefore excluded from the final output from the group, which then was reported to be a consensus. He disagreed that there was a consensus at all, since he didn't agree with it. That's not how "consensus" works; it does not mean that everybody will be satisfied with the outcome, or even want to be associated with it. He goes on to complain that the ISO process requires "consensus", but since he, and like-minded individuals, disagree with the standard, it should not be cleared as a standard.
Again, not how consensus works. In a consensus process, the majority approve of whatever the final output is, and the objections of the dissenters are noted and made available as part of the standards record. You can look on the website of pretty much any standards organization and access drafts, comments, meeting minutes, presentations, the whole works. This full record can help potential adopters of the standard decide if they want to utilize it or not.
ISO? Who are they?
“He’s not deformed, he’s just drunk!”
The Bolton-Christie argument, to me, boils down to: you can have too much of a good thing, e.g. documentation. This can impose unnecessary costs and defeat the purpose if, following the above example, onerous documentation doesn't get read. Too much of a standard means unnecessary cost goes out to the standards industry (rent seeking).
Standards are written in ivory towers. MCA and BetaMax were some of the most impressive standards i never used. I'm not saying standards aren't useful, but they are not a substitute for "just works".
Companies make cheap crap because that is what most people want. Other people buy expensive crap. What they NEVER do is pay a lot of money for cheap crap. Standards help define these segments.
Whilst unit testing is a good thing in general it's actually hard to know how many tests to write. Whilst it is in theory possible to write tests for most things, it's completely infeasible to a priori write tests every for possible interaction with any reasonably sized bit of software. In practice everyone ships code with gaps in the test suite.
There are contexts in which we should be formally proving the correctness of code. This tends to be a niche thing tho' because it's hard and time-consuming.
The free dictionary (by Farlex) defines consensus: 1. An opinion or position reached by a group as a whole.
That's very democratic. Unfortunately, reality is not democratic.
Software testing is designed to unveil real vulnerabilities and errors in a complex system. Having a bunch of people hold up their hands and say, "Is this a problem?" is flatly ludicrous. In point of fact, it's the error that isn't noticed by the majority that constitutes the deepest problem. Remember the Columbia shuttle? A group of people got together and came to the concensus that the ice impact at launch was not a problem.
Testing, by it's very nature, is not subject to regimentation. It's a lot like "Job Descriptions" -- in real terms, establishing a job description is publishing a whole list of things that don't need to be addressed. Why does anyone think software testing will be different?
"Your piece of software has problems." "No, it doesn't. We fulfilled the standard for testing."
Giveth me a break.
Don't take life too seriously; it isn't permanent.
First 3 parts of 29119 cost $775, the total for all the parts when they are finished will probably be around $1200. This seems to be more about money than anything else.
Posting AC for obvious reasons...
When IEEE came calling to our org, they were repeatedly referred to "Captain Canada," a no-talent ass-clown PR guy in MSRC with an engineer title. He simply pointed to the SDL and told IEEE that "this is a solved problem, don't waste our time." At one point he was so rude to the IEEE coordinator he made her cry, and while that doesn't excuse IEEE's weak consensus-gathering, it does put some light on Microsoft's utterly tone-deaf and solid-bone-stupid response.
This mess is as much the community's fault (*our* fault) as IEEE's.
-AC
You may have to extend it in some cases, but that's normal.
The Kruger Dunning explains most post on
The best known standard quip about standards itself has multiple versions and attributions. How meta:
"The nice thing about standards is that you have so many to choose from." - Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254
"The nicest thing about standards is that there are so many of them to choose from." -- Ken Olsen
“The wonderful thing about standards is that there are so many of them to choose from.” -- Grace Murray Hopper
See also:
Obligatory (but who set that standard?): xkcd : Standards
Why are there so many plugs and sockets?
‘Mediocrity finds safety in standardization.’ -- Frederick Crane
‘It is not enough that X be standard, it should also be good.’ -- Rob Pike (Window Systems Should Be Transparent)
The two above can be found on the cat -v page on standards"
"Standards are like toothbrushes. Everybody wants one but nobody wants to use anybody else’s." -- Connie Morella
I am disinclined to acquiesce your request.
That can't come up with a standard without a consensus, and they never asked me nor the general public. It most likely is a way to control the software that is released into the wild, and I expect some OS's like Windows to perform the so called "standard test" and if specific hidden code (set of commands) is not included in it, then it will fail the test.
I won't go along with it under any circumstances, and I encourage the rest of you to ignore it as well.
Firstly, I'm not going to answer stupid leading questions. What is this, some kind of sound-bite-driven political debate?
If you don't like the way the standard is going, you form an organization of like-minded individuals and join the working group. Spread amongst a group of people, the costs are not that extreme, nor the commitment that dire.
And I don't the ability of education providers and consultants being able to advertise "We teach/use the XYZ Standard" as being some sort of nefarious plot. If you are looking for advice on something, being able to have a decent idea what you are going to learn about without extensive interrogation and negotiation can be quite useful for both parties.
The signing of an online petition is guaranteed to be utterly and completely ineffective... the costs of actual participation are so low, it's not entirely unjustified to ignore an online petition as anything other than an isolated group. There's a process to get heard on the issue, and internet petitions, and combative communications with those involved are not it.
I read that petition: "Because the people that signed this petition who couldn't be arsed to participate in the process disagree with the standard, consensus is not possible." How was THAT every going to go anywhere? "Consensus" is not reached (or not reached) by waiting until any schmoe with an axe is satisfied; consensus is reached when the committee votes on a standard and approves one. And yes, sometimes consensus cannot be reached, and the standard simply dies... that's a perfectly valid outcome too.
"It sometimes happens in a people amongst which various opinions prevail that the balance of the several parties is lost, and one of them obtains an irresistible preponderance, overpowers all obstacles, harasses its opponents, and appropriates all the resources of society to its own purposes. The vanquished citizens despair of success and they conceal their dissatisfaction in silence and in general apathy. The nation seems to be governed by a single principle, and the prevailing party assumes the credit of having restored peace and unanimity to the country. But this apparent unanimity is merely a cloak to alarming dissensions and perpetual opposition." - Alexis de Tocqueville, published 1831
*And I don't see how the ability*...
*ever*
*not reached (or failed to be reached)*
I've done lots of work under various DOD, DOE, FAA, ISO, IEEE, and many other publishers of standards.
Not once have I ever achieved "full compliance" with any but the most trivial standards. What we typically did was cherry-pick standards to highlight what was most useful in our specific situation, eliminate what was inapplicable (and try to find suitable replacements), and assess the rest for usefulness and effectiveness ("worth the effort").
This is called working to a "tailored" standard, and is very a common industry practice. For standards such as the FAA's DO-267, tailoring is required, because parts of the standards actually conflict (in a "choose path A or path B" sense).
I've found many standards to include some truly useful and insightful gems, though that's under 1% of the bulk. Still well worth finding and using. I'm especially a fan of the ISO-900x series, which has positively transformed every business I've been with that applied them properly (rather than wasting time and effort paying lip-service to them).
There are also "guidelines" that are just short of being standards that every software practitioner should be aware of, such as those from Carnegie Mellon's Software Engineering Institute for secure/mature software development. The MISRA guidelines are "the book" for embedded C programming.
Standards are not religious doctrine, to be slavishly adhered to upon fear of condemnation. They are the collected wisdom of groups of experienced practitioners and managers (perhaps groups that don't include you, but often still insightful).
Take the best, leave the rest. Move along.
Most people don't understand the compliance. There's good and bad, but there's no going back once your industry (candle makers, software writers, barbers, whoever) adapts a standard it invariably becomes a tool of an authority.
Good: What I like about it is that our certifications increase accountability by encouraging recording mistakes. The "routine" of flagging mistakes and finding root causes and formalizing "corrective and preventative action" has been good and improved our company.
Bad: These standards are adapted by many companies in order to reduce competition, take away via consensus unique individual methods for doing things. They become almost like a "union", punishing individual innovation via auditors that view the world inside a "box". Uniqueness and innovation are an increased cost and risk to the third party auditor, and the auditor is ready to adapt the majority interpretation - which is usually to increase barrier of entry into the field of competiton.
As Morris Kleiner, the AFL-CIO chair in labor policy at the Humphrey School of Public Affairs at the University of Minnesota, put it "Occupational licensing has either no impact or even a negative impact on the quality of services provided to customers by members of the regulated occupation."
Gently reply
(anonymous because i never retrieved my /. inlog on this comp)
I have been a professional tester. We do NOT do unit tests. Ever.
Unit tests are test designed to look at your code, look at where it should go right, sometimes also where it could go wrong, and test these situations. If a piece of software passes a unit test, all you know is that it did technically preform within the required parameters.
Like you hire a plumber to put in a bathtub, and maybe even make your bathroom look nice. He delivers a nice, shiny bathroom. With all taps working perfectly. Unit test passed.
So, the installed unit (bathtub) worked perfectly at delivery. Sign-off and pay. The plumber left.
You never inquired, when asking the plumber for his estimate, whether the drain would be connected. So he did not do that. It was not in the job-description. Apparently the wetness of the bathroom floor is just something you will have to learn as being part of the new situation. At least you have a tested bathtub!
To make sure that a piece of software works within the intended environment, there is a thing professionals call Unit Integration Testing. Who does it is in eternal debate. Programmers say I should, I say they should.
You piece of programming does not live in a world of its own. It is connected into a bigger environment. An environment of which 'some' parts might not be less properly documented. (Anyone in testing/QA knows what 'some' means: any or all)
Living in a nice house, your just renovated bathroom happens to be above your kitchen. Guess what. The substandard (with a passed unit-test) delivery by your plumber caused a leakage. The kitchen-tiles seem to develop a desired to jump off the wall and rest on the floor. It happens.
The unit (bathtub) within the bathroom (system) you had installed, suddenly makes problems for a totally different system within your living environment. Ouch.
Luckily, professional testers have developed ways and means to deal with this. System Integration. Nice short term, sounds simple. Has always been easy to sell. (Well... an easier sell then the need for testing professionally, even /.-ers do not seem to understand that.)
Every possible interaction between any present or proposed room within your house will have to be checked for any interactions which could possibly endanger the normal operations of the not-renovated rooms. (Just be happy that you installed only a bath, if it had been a toilet, this story would now get dirty.)
This includes smells which come through the long ago plastered over vent which was installed 30 years ago, but has not been in use for the last twenty years. Just one of the non-documented features of your house. Oh, and maybe there was a guy who drilled holes from every room to the next, to put in cat5, something like 15 years ago. But we had someone that out something like 8 years back. (remember what I said about less properly documented?)
It is also one of the hardest things you could ever do. Especially in older buildings/software.
Unless your house is totally enclosed and self sustaining, your whole integrated system will also have interaction with other systems. Like the water, power, fiber and sewage systems. To ensure that nothing endangers these, you will need to do the necessary Chain Integration Testing. Does your new bath draw so much water out of the environment that your neighbour can not water his lawn any more? Or does your drainage cause flooding somewhere else? Does your renovation increase the chance that burglars or other rats enter your home?
There is one more thing I hope you will try to understand about testing and professional testers.
For a professional tester it does not matter what kind of tools and intermediate goods you have used to create what you have made. If the end product contains flaws, it is not good.
A professional tester may not be blinded by developers use of potentially failure ridden intermediate goods. Whatever a deve
First, I refused to answer those questions not because I bow down to suspect authority. (Where did THAT come from?) I refused to answer them because they are stupid. Of course the answer is "no", but that answer establishes nothing, because the questions imply conclusions that themselves are up for debate. If this is the way you and your like-minded compatriots engage in debate, no wonder you can't get anybody to take you seriously.
And I keep seeing this claim that the ISO process is set up to "suppress opposition and dissent"... but the only specifics ever mentioned is that the process takes a while and has meetings outside North America. Is that the best you can come up with?
To answer some of your points:
- Most technical standards organizations are private. IEEE, ANSI, SAE, IEC, UL, IIHS, IETF, W3C, ASTM, etc., are all private. (Not to mention tech-specific ones like the ones over Compact Flash, USB, Infiniband, etc.) Arguing that the ISO is illegitimate merely because it isn't a government agency is not likely to be persuasive. (And as a side note, the ISO was set up at the behest of the UN and is tightly coupled with them... it's not a UN agency like the ITU, but it might as well be.)
- I'm not aware of any standards organization (professional societies like the SAE and IEEE included) where standards are put out to vote by "a representative slice of practitioners"; they are all voted on by those that chose to participate in the standards process.
- Arguing that it's controlled by a bunch of money-grubbing consultants and trainers AND that it's rammed through by attrition is a contradiction. Stretching out the process indefinitely is exactly opposite to the goal of making money off the standard, since nobody makes money off a standard that doesn't exist.
- It's not deliberate attrition just because it takes longer than you'd like.
- Yes, the burden is on the ISO to demonstrate relevance of its standards. But once they have successfully done so to an organization and that organization comes to you for your services, objecting with nothing more than you "shouldn't have to defend yourself" is not likely to get you hired.
- If such a significant number of professional testers object to the contents of the standard, why could they not scrape up the funds necessary to participate in the standard? Even with meetings held in distant locales, on a per-person basis, it doesn't come out to much. (And could you not find any software testers in India, Japan, etc. that are like-minded to save on travel expenses?) If you want to be taken seriously, you gotta put your money where your mouth is...
- Again, consensus doesn't mean "everyone agrees that they can live with the content", it means that a majority (or super-majority, depending on the rules) approve of the content. If a single "no" vote could prevent a standard from being approved, we'd never have any standards at all (imagine if the Ethernet standard could have been completely halted by IBM signing up for the IEEE committee and voting "no" so it could push Token Ring instead.) This is not a difficult concept to understand, nor does it rise to "suppressing dissent".
- Like it or not, refusing to participate in the standards process does not bode well for arguing that there's "significant" objections to the standard, since those objectors could not be bothered to show up when it came to deciding on the content. (An online petition? Seriously? That's supposed to persuade anybody?)
Here's what Moolya thinks of this 'standard'!
An open letter to the President of the International Organization for Standardization about ISO 29119 - http://moolya.com/blogs/2014/09/129/An-open-letter-to-the-President-of-the-International-Organization-for-Standardization-about-ISO-29119
by Pradeep Soundararajan