Slashdot Mirror
Standards for Bug Severities?
MythoBeast asks: "While attempting to determine just how far a (unnamed) company's software is from releasability, a group that I belong to discovered that they consider a piece of software releasable with numerous 'severity one' bugs. This seems outright heinous to us at first, but then the conversation devolved into an argument of what constituted a 'severity one' bug. Is there a standard for such things? Is there a standard for how many of these things are reasonable for 'releasable' software?"
Originally, I thought the purpose of testing was to prove there were no bugs in the system.
;)
As you know (now), the testers role is to bash the fuck out of your software, break it, and then tell you about it. Then the developer scratches his or her head for two or three days, sends the bug back as "User Understanding" and breaks something else in the process.
Then all the testers go to the pub at lunch time, and complain about how shit the code is, how we found a bug that should have been found by the developer in Unit Test, how some bastard from development has been messing up the application database, and how managment won't listen and the development/testing process is all fucked up.
Then we come back to work, and annoy the developers with almost infinate questions and problems.
This is the role of a Software Tester
(A bitter and twisted Software Tester)
This would be interesting except that it is simply a list of bugs. It doesn't explain the classifications.
We just started using Bugzilla at my workplace for bug tracking. The default high/low/medium severity was replaced in place of S1-S4.
...
S1 = Causes data loss or damage to system. Hangs at OS level. Fire, flood, famine, locusts
S2 = Hands outside of OS level. Prevents system from booting or restarting.
S3 = Annoyance bugs. Major look & feel issues of user interface.
S4 = Bad grammar or spelling, minor look & feel issues.
The guideline is being printed in huge letters & plastered all over the test lab so there's no confusion.
You may have to implement a parallel incentive program within DVT/QA. The toys will be different, of course. :)
;)
Hey, back up. Why wouldn't Testers want cool Geek toys? Look, can we get this clear please? Testers are just Geeks who don't want to code all day
Maybe you should go talk to your QA people some time, find out how much some of them really know.
At Microsoft, they have two measures for bugs: Severity and Priority. Different groups have slight variations on what each one of them means, but it generally comes out like this. Severity is how bad the effect is: Sev. 1 = BSOD Sev. 2 - 3 = General range of bugs Sev. 4 = Annoying blinking pixel, spelling error, etc. Sev. 5 = Change request > Priority is how important the fix is: Pri. 1 = Fix in hours (ASAP) Pri. 2 = Fix in Days (1-3 depending on workload) Pri. 3 = Fix in Weeks (1 or 2 - when able) The main difference is that some very serious bugs (Sev. 1) only occur in very unlikely situations (like simultaneously plugging in 3 USB devices while hitting the PC case with a running hairdryer. Don't try this at home.) Therefore it might have a lower priority than... say, a high visibility spelling mistake on the product's start up screen. Hope this helps clear up some confusion! Signed, A. Coward
I am a project manager for a software company, so I'm quite familiar with bug severities. I've worked at several software companies in the last ten years, and all of them have adhered to a 4-level system, though the exact definitions of the levels vary from company to company (or engineering group to engineering group within a company!)
:-)
The goal of a software company is to make money, and the way to make money is to satisfy the customers. So the most important thing is that the severity system reflects the users' satisfaction with the software. If there's some silly little glitch that drives the customers bats, it deserves a high-severity rating, even if the engineers think it's harmless. I currently use the following system, which I've refined over the years and which serves me well:
4 - A feature request, i.e., something outside the design spec of the software.
3 - The software ought to do it differently, but it's unlikely most users will run into it. There is no loss of functionality, or there is a workaround possible. There is no possibility the bug will cause loss of data.
2 - A serious bug...something lots of users will encounter, but that does not result in loss of data. There is a workaround possible.
1 - A serious bug causing loss of functionality. No workaround possible or causes loss of data.
A commercial release of a product should contain no severity 1 or 2 bugs. However, it's okay to ship a product with these bugs if the bugs and workarounds are described in the release notes. If you're embarrassed to write up a particular bug in the release notes, then fix it.
A beta release follows the same rules, except the release notes tend to be longer.
Just a point that i have not seen mentioend here yet, but i think that bug priority depends on context. A simple spelling error or typo in a business application might simply be seen as an unprofessional issue and be marked down to a "polish" priority, but a spelling error in a program that teaches spelling must be marked as a priority one bug.
> Is there a standard for how many of these things are reasonable for 'releasable' software?"
Yes. It is somthing on the line 'if it compiles, you can ship it'
Cheers,
--fred
Come on people. ALL software has bugs. Some of which should stop shipment of the software. But a "Priority 1" bug isn't necessarily a "stopper". And, when push comes to shove, management will downgrade the bug if their bonus is on the line. And all you purists who say there should be no bugs should try and maintain 10 million lines of code hacked on by 500 developers and see if you can keep it bug-free.
"It's easy for us technical heads to get on a high and mighty horse about quality and not shipping with bugs. But we tend to forget that what we're doing is NOT engineering (it's a lot closer to handicrafts in the days of guilds than it is to engineering the way it is done with bridges and aircraft). And we're not the people trying to run a business"
That is something technical people tend to forget.
OTOH, what people in the software industry tend to either not know, or know but avoid thinking about, is exactly how bad their reputation is in the rest of the business world. Let me illustrate.
For the last 10 years or so I have done software implementation and IT technical support for a variety of industrial products manufacturers. The last company I worked for had the following characteristics:
* Served heavy industrial customers with mechanical products
* Competitive market with fixed price, fixed delivery date (+ penalties!) contracts and annual price _decreases_
* Product had a software component
* Gross margin in the industry and its suppliers of about 1/4 that of software firms.
And oh yeah, if our product failed, there was about a 10% chance that the crew of four people operating the system would die a very, very unpleasant death.
Guess what? Our products didn't fail. Not the hardware, not the software. Of course, we weren't gods, so very rarely an accident would occur. But EVERY accident was reviewed in a very thorough and unpleasant manner by officers of the customer, who would come onto the shop floor and interrogate production line employees at length. And woe unto the supplier whose employees weren't (a) fully knowledgable about their job and product (b) fully truthful.
Now, in this environment, what opinion of the software "industry" do you think general management held? That is, the people responsible for the _business_? I am sure you can figure it out, but to be specific: a very low opinion. A typical question was, "Explain again why this product that I just paid $1.2M for, which figure represents 10% of our gross profit this year, crashes every time I press the print button?" These were not very nice conversations as you can imagine.
I won't go on, but I would suggest that the software developers reading this thread look up some of William Mossberg's columns in the Wall Street Journal and think a bit about what he has to say concerning software reliablity and complexity. From a business perspective.
sPh
Having worked in QA I can tell you that as the release date slips further past the scheduled date the more flexible the definition of severity becomes. Typically bugs that were severity one become severity n+1 bugs where n is the minimum level specified in the release criteria. To me it would nice to have a standard defition of severity but it much more important to have management with integrity.
Seems like a fairly simple sceme to me -
Hellooooooooooooo? The Perl Bugs Index doesn't actually define the different categories of bugs. Larry (or whomever) uses the force to determine which bugs fall into which categories. This is unfortunately common.
The question being asked by our poster is "How do you avoid doing this? What are good definitions for different categories of bugs?"
And so how does slashdot answer? "Oh, you do it the way Perl does, just use the force," which is exactly the answer not to give! And then for good measure, let's mod that post up to 5, Informative.
Come on folks. Think a little bit. This place is only as good as you make it. Before you post, think. Before you moderate, think.
End of sermon. Sorry. This just annoys me.
I have seen this in my company. There is no conspiracy against the customers. Usually I have detected two reasons for these kinds of semantic disputes:
(Sorry. Anonymous this time.)
If you are playing tetris on a computer controlling your life support system, and every time you get a green piece tetris pauses briefly and the CO2 filter shuts off so you suffocate Tetris has a Severity 4 or 5 bug.
Hoo boy, we used to get into this argument everytime we reached the "triage" portion of the project (typically after second beta or 3 months before ship, whichever came later).
Sev1 bugs: cause loss of data, hardware damage, or take down other programs (or the OS) which might cause loss of data.
Sev2 bugs: cause the program to crash without losing data, or otherwise make some feature non-functional
Sev3 bugs: impair the usefullness of some feature
Sev4 bugs: are cosmetic or may cause some confusion to the user (wrong status text, spelling errors, etc)
HOWEVER, there is also a priority scale. A Sev1 bug is not necessarily a Pri-1 bug, if it only occurs under extremely obscure conditions. Eg, if your program crashes, losing data, when a PC-Card is inserted, that's a Sev-1 bug. If it occurs any time any PC-card is inserted, then it's a Pri-1 bug. But if it's only one particular kind of PC-Card that exists in the wild in quantities of, say, a couple of hundred, then it's not a Pri-1 bug even though it's still a Sev-1 bug (you may still choose to fix the bug, but other Sev-1 bugs have priority since you can doc around this one). On the other hand, a low severity bug that's in your face every time you use the product is probably a fairly high-priority bug.
If you're running your business as a business, you will ship with bugs. If your product has a broad market, you will probably ship with a couple of Sev-1 bugs. (This sounds paradoxical, but consider -- the broader your market, the larger the number of permutations of hardware and software your product is going to inderact with, and the higher the chance some obscure situation will occur in which a Sev1 rears its head). It's a business decision to ship, so the number of bugs you ship with has to be based on more than just a technical analysis (Rule #1 of software development: shipping is a feature too).
It's easy for us technical heads to get on a high and mighty horse about quality and not shipping with bugs. But we tend to forget that what we're doing is NOT engineering (it's a lot closer to handicrafts in the days of guilds than it is to engineering the way it is done with bridges and aircraft). And we're not the people trying to run a business.
The corporate-wide standard in this case was badly broken, and was obviously written by someone who doenst understand software release.
QA is NOT there to find, log, and force a fix for every single bug in the system. That is an impossibiliy. Just like manufacturing QA, Software QA is a statistical processes.
The classic manufacturing QA example is the Bolt Building Machine. It makes bolts from metal bar at the rate of 10000 a day. It would be wonderful if manufacturing QA could take each one of the bolts produced by the machine and measure the eight or so critical bolt dimentions to ensure that each bolt was within tolerance. Unfortunatly, it takes about 15 minutes to measure a single bolt with that kind of precision, so it is not cost effective to measure each and every bolt. The next best way to ensure quality is to take a random sample of bolts and measure them, so that you can be statistically certain to an acceptable degree that the rest of the bolts are good.
Software QA works the same way. When a new release of software comes out, it is rarely run against the entire QA testplan again. This is especially true with patch releases where the development team believes that only a few minor things have changed. It just takes too long to run a full QA cycle against a build like that-- it is possible, but it isnt cost effective. Instead, the QA team runs a subset of the test plan against the build in an effort to sample the test space and come up with a statistical certainty that this release of the product isnt broken.
Bug classification is also a statistical problem equivilent to the classic 'expected value' problem. The standard five bug classifications are useful measures of the 'weight' of a problem. They answer the question, 'If this bug happens, how much does it hurt?' What the abovementioned corporate scheme failed to qualify was the 'How often it happens' part. Without knowing the frequency that the problem occurs, it is impossible to calculate the 'expected hurt' that the problem will generate.
Here is a degenerate case example. Say the program has code like this:
If a user directs a cosmic ray into the computer that corrupts the value of pointer between the time its validity is checked and the time it is dereferenced, the program will crash with severity 1. There is no workaround, because every time the user sends in this ray, the program fails. According to the abovementioned corporate policy, the product cant release until this problem is fixed, period.
What are the odds of a user here on earth actually sending in that cosmic ray at exactly the right time, let alone having a need to? Very close to zero. So close in fact, that it practically doesnt matter how severe the crash is. The 'expected hurt' from this problem is going to be very low, and the product should ship with that severity 1 bug going unfixed. If the software was going into space where cosmic rays are common, it might be prudent to recalculate the odds of the problem happening before shipping the code-- and maybe then the only practical solution is to ship with the bug but send up redundant systems.
QA is not there to ensure that the product has no bugs. QA is there to qualifiy risks, and to assure that the quality of the product falls within a certain tolerance. In a perfect universe, software would never have bugs, and bolts would never bend. Here in our solar system though, it all boils down to risk management.
A. Coward, (a QA Manager for a fortune 50)
The standard for how buggy a piece of software may be and still be releasable is the market. If people will buy your product in sufficient numbers for the project to be profitable, there were sufficiently few bugs at release time. This is also known as the Microsoft Lemma, as a reminder of the folks who lowered the bar for everyone.
How many bugs are they finding, and what severity are they? This speaks to the quality of the system design and the quality of the engineers working on the system. If there are few bugs found and fixed, you can be confident you have a good system going out. If there are a lot of bugs found and fixed, I would worry that there are a lot more left undiscovered.
the above is only true is the testing group is good. If they run one test case continiously they can miss a lot of bugs. (This has been known to happen, generally when many seemingly different cases all exercise the ame code path).
We have guildlines, that state something different, but i've noticed it comes down to a management decision "Is the pain of shipping with this bug outwieghed by the cost of fixed it?"
Early on everyone else's definition is what we use. Several months before shipment (and things are normally fairly good then) management does a bug review and decides what the priority is. At that point:
Sev 1: data corruption, or serious crash. encountered often and no work around. Affects major features. Cannot ship.
Sev 2: above, but we know exatly how to cause it, if you don't do that things work okay. Can ship to beta customers, but not to production use.
Sev 3: Not as serious, but a black eye. Fix if at all possibal. But work arounds exist, and in general we can live without. If it affects a minor feature we can turn off, then it is always this level. Anything shiped with this level or problem goes into the release notes. Before shipping management has to review each and decide to wait while it is fixed or ship now.
Sev 4: Minor. Deseriable to fix, but rare enough that we can just document it.
Note that for sev 3 and 4, there is a chance that if the feature in question isn't a major one we will disable it instead of fixing it. Also only major features can be sev 1 or 2, as we can't disable them.
This seems outright heinous to us at first
As it did to me at first, but think: a bug that causes a crash, if it happens sufficiently rarely, might not be worth fixing from an economic standpoint. While it still sometimes drives me nuts to ignore the bone-jarring crash-causing bugs to fix cosmetic flaws, if the crash happens rarely while the cosmetic flaw stares you in the face every time the program runs, it can make sense.
Make me aerodynamic in the evening air
Posted without attribution again, as usual. Sigh.
Umm.... no. Say I have this bug, X. If bug X triggers a BSOD when I'm surfing the Web, it is a level 4 or 5. If bug X triggers a BSOD when I'm trying to dock a space tourist, it's a level one. So, whether or not you have the intervening BSOD, you still have the problem the original poster described- the severity of bugs is completely situationally dependent. Of course, if your program is designed for an extremely specific task, then this type of listing is fine. But for any general purpose application (and more importantly, for any general purpose OS) this type of categorization is next to useless unless the categorization of the bug is closely tied to usage scenarios.
~luge
IAAL,BIANLY
Don't forget there is variation on how MS classifies bugs internally also. I've worked on 4 different teams @ MS and have seen some varations between sev1 and sev2 bugs across them. Several teams classify sev1 bugs as an OS crash caused by thier application. Sev2 bugs are crashes or major functionality problems within thier application that don't affect the OS. Sev3 and 4 are as the same as your list. I personally like this method better as I tend to find bugs that crash my OS to be far more annoying than a simple application crash. When you compare the possible problems caused by the crash and the time (reboot) required to resolve it to me it makes more sense.
Actually, I'd have no problem with this, except that I'd prefer that documentation on the existance of the bug be provided, along with known triggers and work-arounds. This is the problem I generally have with Microsoft; they will deny the problem, and if they do admit to it, bury the fix VERY deep inside of their "knowledgebase".
Show stoppers are usually bugs that crash the software when doing normal functions. Like if you open up a file and the software crashes, that would be a show stopper. In some cases it depends on your clients as well. If the client finds a bug they may deem it as necessary to be fixed. Most companies I have worked at usually work closely with clients. In many cases the software does not get FULLY tested till the client actually gets the software. Then when (not if they will find bugs) they tell us an dthen we make it a priority to fix them and send out patches to certain clients if not all. This would depend on the client contract.
"Low priority" bugs would be ones that do not impead functionality, but are more of annoyances. Take for instance I saw one bug that when you scroll the scrollbar the window would flicker. This is annoying, but does not prevent the user from using the software. Other less important bugs are bugs that are on reports. If a report does not show the correct data and you can get the data from the database or it is a known bug this can also be a low priority bug. The question usually is "Can the client work with the application with this bug or will it cause the client major problems or loss of money?" Take for instance the reports bug, a client could just not use the reports and create their own from sql.
Something you must remember is that some of the programs that I have worked on have been written by many programmers. They are thousands of files andhave client, server, and database code, in multiple languages. Some of these bugs could be caused by some of the software that we use. Bugs in windows API's, bugs in Sybase, or Oracle, bugs in the server OS, or even 'quirks' that many programmers do not know about.
Next, many times programmers get put in a job where they know the language but NOT the API and the API is not well documented, especially if this is a 'custom' system where the software company has developed programs for DB access on top of the 'db lib'. What about programmers that leave the company or get fired?
So who do you blame? The programmer that impemented something that he did not know the company API, the software company for not completely testing "EVERY" single case that could possibly come up?
You never know how a user will use software or what they may do or have installed on their system that could affect how your software will work.
I don't want a lot, I just want it all!
Flame away, I have a hose!
Only 'flamers' flame!
I've always thought that there are bugs in all software. Can the software to the basic tasks that it sould? For the "features" to occur is it something that people would do on a regular basis? By this I mean do you have to do 10 different things that would not be a normal occurence in the software for these things to activate.
As long as I can use the software in a reasonable manner, it's all good.
Don't you mean:
1. The computer knows all.
2. The computer is always right.
3. The computer loves you.
6. The computer sees everything.
4. The computer is in connttrrolll.
9. The computer wants all good things.
14. The computer will make things all better.
2. The computer cannot be questioned.
4. The computer is above you.
7. The computer makes you, thinks.
Apologies to the creators of Paranoid.
--
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
> some (the "Microsoft people") want everything there now, want it easy to maintain and setup just by clicking a few dialogs, don't want to fiddle with config files or memorize commands,
More or less.. no...
I honnestly don't know what Microsoft people want. What ever it is Microsoft is delivering.
Microsoft people admit you need to know what you are doing to make Windows work. You need to memoerize commands and fiddle with config files.. or at least one big config file.
Thats no diffrent than Linux.
However Windows is very diffrent from Linux.
Yeah Windows has it's problems and truth is those problems are NOT found in Linux.
The inverse is also true..
People put up with limitations to get something they value more.
If they could they'd go for something lacking the problems of Linux and Windows.
For now it's a choice.
If you want easy use Mac.
If power is all you want.. they all are..
Just a matter of picking the trade offs..
User friendlyness in trade for stability is NOT a tradeoff offered by Windows...
I don't actually exist.
BSOD is not a bug. BSOD is the result of some bug which could be classified according to the scales. Any number of problems, from severity 5 to severity 1 all trigger BSOD. (Note the BSOD is essentially an effort to convert severity 2 bugs to severity three, by giving the operator the option to continue. 'Course, it never works in real life that I've seen, but sometimes it works just well enough to save to a new file and try again.)
So by this model would all usability defects be classified as sev© 3 or sev© 4 ¥primarily sev© 4, I would imagine?
Nice© Tell that to your human factors friends©
This question comes up about every 3-4 months, and never gets 'resolved' (where resolution means a majority agree). Lots of good discussion, though.
you are working on a Saturday afternoon when the weather is sunny and warm...you're waiting for the QA team to go over bugs, you can't leave work til there closed...and you're reading /. to pass the time.
Boy does the truth hurt!
-- ladies and gentlemen we are floating in space!
If I could draw from petty cash to buy geeky toys, damn skippy I'm not gonna turn 'em over to no slack-ass programmer.
: )
He's always asking me how to install software and fix his computer anyway, so although his social skills are, uh, iffy, he's no geek.
Why yes, I AM a rocket scientist!
Ah. Good manager. Any idea where I could pick one of those up cheap? Usually, the manager has been the guy screaming "SHIP IT!" at the top of his lungs...
Your point is well taken, though. If everybody just did their damn job, the process would work fine!
Why yes, I AM a rocket scientist!
I've been in software testing for longer than I'd like to admit. I've never figured out how to classify bugs that are very annoying to the user (misspelled dialog box, say) and stupidly easy to fix. Unfortunately, most of the programmers I've worked with don't even LOOK at Sev2 and 3 bugs...only the showstoppers. Yes, they're bad programmers and they're breaking the process, but hey...I've got to get the job done, right?
Any suggestions, other than make EVERYTHING a Sev1 issue and making the whole process useless?
Why yes, I AM a rocket scientist!
If a user is playing Tetris on a computer controlling your life support system, there are two top priority bugs:
- The life support system has a bug in that it supports Tetris
- The user is exhibiting an ID10T bug in his own brain by playing Tetris on the life support computer.
Commercial grade platforms, including Linux and Unix (possibly excepting QNX), don't have bet-your-life reliability. Do you trust your life support to an OS that's been around for ten years and has an MTBF in years? For that sort of thing, I expect MTBF in decades or centuries! I will bet my money on something like Linux, but not my life.Bet-your-life systems tend to be very small, very limited in their functionality, isolated from the outside world, and tested up the yin-yang. If you are building a system that needs such systems, and want to add additional capabilities, the accepted practice is to put those capabilities on another computer that can't impact the bet-your-life system.
Take an airplane as an example. If it has a fly-by-wire system, that is a seperate computer. If there is a life support computer (do you need a computer to regulate pressure and the like?), that would be a separate computer. If it has an autopilot, that is a seperate computer. Other systems you can live without, such as navigation and targeting (on military jets), are again separate computers.
For that matter, there are airlines that want to provide net access in flight. This means that they have to have a small ISP on board their planes. These are certainly not going to get tied into the flight control computers. If they did, they can get infected with a virus in flight. By keeping the computers disconnected, any attacks on or failures of the ISP computers have no impact on flying the plane.
By doing this, engineers are able to protect the computing resources that you can't live without.
If you can play Tetris on your life-support system, then you lack the above isolation, and are betting your life on a system that may have bugs.
--The basis of all love is respect
as expounded by Watts S. Humphrey of SEI in "Managing the Software Process" (c) 1989, Addison-Wesley, p. 218:
"Severity 1: An error that prevents the accomplishment of an operational or mission-essential function, prevents the operator/user from performing a mission-essential function, or jeopardizes personnel safety.
Severity 2: An error that adversely affects the accomplishment of an operational or mission-essential function and for which no acceptable alternative workarounds are available.
Severity 3: An error that adversely affects the accomplishment of an operational or mission-essential function for which acceptable alternative workarounds are available.
Severity 4: An error that is an operator/user inconvenience and affects operational or mission-essential functions.
Severity 5: All other errors."
Any other questions?...
Severity vs. Priority
As several other posters have pointed out, it's important to make a distinction between severity and priority. I would also say that it helps to define severity as narrowly as possible, to refer only to the effect of a bug when it occurs. I've already seen some people who accept the severity/priority distinction make the mistake of letting other factors (e.g. availability of a workaround) be factored into severity when they really only affect priority. Priority is the "net net" of severity, frequency, relationship to core/extended functionality, availability of a workaround, and possibly even cost to fix.
Factors affecting severity
Even within a limited definition of severity, there are multiple factors to be considered - specifically scope, magnitude, and duration of damage. Scope is a matter of what was affected: the whole system, the program, access to particular features. Magnitude is a matter of how complete the failure is: did the affected component(s) fail totally, did it hiccup and then continue, did it suffer a performance degradation, etc. Lastly, duration is simply a question of transient vs. permanent errors. When you've characterized the bug according to these three criteria it's pretty easy to use a point system to sum it up into a single severity value (though it never hurts to record the details). Likewise, once you know severity you can use a point system incorporating that and other factors into a priority value. Obviously the assignment of point values can be a complex and contentious process, but it only needs to be done once and once it's done the assignment of both priorities and severities becomes so much clearer that it's worth the effort.
Other Points Just some random thoughts about bug classification, in no particular order.
Slashdot - News for Herds. Stuff that Splatters.
I disagree. If there are few bugs found and fixed, that could be a sign that the testing is inadequate. I once worked on a project where the customer expressed a concern about our bug counts being too low. This seemed counterintuitive at first, until they explained that in their (very broad) experience this more often indicated something about testing than about the product. Sure enough, we stepped up testing, found more bugs, and went on to deliver a much higher quality product than if we had sat around patting ourselves on the back for our low bug counts.
What's more important than the absolute bug count is the *trend* in bugs reported. Every non-trivial project can be expected to exhibit an increasing rate of bug reports as QA ramps up and as components become ready for testing, followed by a plateau, followed - hopefully - by a steady decrease as the bugs get nailed without introducing new ones. If the plateau goes on too long, or if the extrapolated decrease at the tail and does not reach an acceptable level by shipment time, there's a problem. If the rate starts to go back up at that late a stage, there's an even bigger problem: your programmers are getting desperate, and checking in "fixes" that introduce more new bugs. If you don't see a "hump" at all, it probably means your testers aren't doing their jobs (or possibly not getting the help they need from developers).
All of these patterns are readily apparent if you know to look for them. I'd say you're right that the purpose of QA is statistical, but I'd say it's a statistical measure of *process* as well as product quality, and if the process is broken you can't know anything about the product.
Slashdot - News for Herds. Stuff that Splatters.
I once wrote a shrink wrap program for a nitch market that was the best selling in the nitch.
I can tell you about a (true) Sev 2 (or maybe sev 1, as the main part of the program didn't work) bug that was LOW priority
Way back when, HP made a card that went into you PC that accelerated your Laser Printer. What it did was offload any graphics processing from the printer to the PC - If I remember right, it was called a LaserDirect card
Well anyway, we found a bug after we shipped - The the client was running a PS2 Model 80, in a certain video mode, with this HP card, NO report printing worked, AT ALL. When you consider the main purpose of the app was to print these reports, it was fairly bad
The thing is, in the first 6 months the app was out there, we found ONE customer who hit that bug. We had a simple answer - we gave them their money back, and said, "Sorry"
-- 73 de KG2V For the Children - RKBA! "You are what you do when it counts" - the Masso
that's why I cringe when I hear the term 'software engineer'. It doesn't mean much to me. I'm really glad civil engineers don't follow the same practices as 'software engineers'. Most software engineering in the commercial world has nothing to do with engineering. Military projects and things like air traffic control might be different (I hope they are!).
Consider this: Most _commercial software_ programmers write software to make them money, not to impress the world with perfect software.
If you can release software with bugs and still make more money than your competition, you've won. What I'm trying to say is, the motivation to fix bugs is to gain marketshare, not to "do the right thing".
All this because there's no standard software testing lab, and no software "building code" to comply to.
Hands in my pocket
Wow, its win98 glued to NT 4
with a 'computer manager' thing and a fancy graph for your cpu time
Juln
There is absolutely no way to consistently determine whether one bug is more severe than another because users needs are different. One bug may be catastrophic to one user while the same bug may be of no consequence to another.
This is theoretically true but practically false. Consider the equivalent statement about features: "There is absolutely no way to determine whether one efature is more important than another because user needs are different."
This may be true in the theoretical sense, in that you can never know for sure the complete needs of even one person, and aggregating the needs of thousands of users is even harder. But in practice designers and programmers build stuff every day that manages to more or less match the needs of their users.
And so it is with bugs.
sev 1: crashing bug, or data corruption
sev 2: feature broken, no workaround
sev 3: feature incomplete, or broken w/ workaround.
sev 4: cosmetic or UI flaw
This strikes me as pretty reasonable except for the "or UI" part. A bad or buggy UI can lead to user behaviors that destroy just as much data as a fandango on core does.
One of my favorite examples of this comes from the NeXT 1.0 OS release, which was a slick GUI on top of a BSD-derived Unix. As appropriate in a GUI OS, you could click on a floppy (mounted in the filesystem, of course) and choose "format" from a menu to erase the disk (after a little "are you sure you want to erase this disk" messsage.
But you would get the identical warning if you tried to format any other partition in the file system, even your root partition. And it was awfully easy to mistakenly select the directory that the floppy was mounted in rather than the floppy itself. In one sense, this is "just" a UI flaw. But it's certainly not a just cosmetic one; it should be classified as severity 1.
The GNATS bug-tracking software (oh, go to www.gnu.org and look on their software list) does the same thing. Severity is how badly the software screws up. Priority is how soon it needs to be fixed.
Take a look at how GCC manages their bug reports for an example of this in use.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
We track both severity and priority. Severity is graded into
1. Critical/Showstopper (weightage 3)
2. Serious (weightage 2)
3. Medium (weightage 1)
4. Low (no weightage)
5. Nice to have (which in many cases is actually a change request)
Priority is graded as
1. Immediate
2. High
3. Medium
4. Low
Severity is used to analyse the Quality of the software by assigning weightages and then calculating the overall average of all the bugs. If the severity index is more than 1.5 the software is still not stable.
Priority is used to attempt to fix bugs. The priority gets bumped up automatically if a bug has not been fixed for more than a week. So if the developement team hasnt fixed a medium prioruty bug for more a week it becomes High priority since it means developers are not fixing bugs or the project in some way is out of control. This can also mean that developers are too busy fixing high and immediate bugs and havent come around to medium bugs which indicates that the project is slightly out of control.
O this learning! What a thing it is - William Shakespeare
No category 1 or 2 defects should exist in released software and category 3 defects should only be permitted if the repairs will have a strong negative impact on the schedule. (I have never shipped a product with any but category 4 defects.)
The whole "valid" vs "invalid" inputs may confuse some people, so I should probably explain. I'm used to working in embedded systems where the system reads some measured values. Typically, those values will be represented to the system as a current, a voltage, or a contact closure. The "4 to 20" current loop is typical of the sort of thing I'm talking about. In that sort of system, the physical value is calculated from the current in the loop which varies from 4 mA to 20 mA. If the current is within that range, then the interpretation is easy. The question then becomes what happens if you get 0 mA or 22.5 mA? Neither of those is valid current as far as the system is concerned so it's not clear that the normal conversion to measured value should apply.
However, you will see such values in practice. The first is what happens when the wire gets cut, the second might be the result of an uncalibrated transmitter. A large part of the design of an automated control system has to do with figuring out what to do with those out of range values.
However, other sorts of programs get presented with invalid inputs, so the approach is not without merit in the more typical case. If, for example, you present a COBOL compiler with 6 MB of random data (as an acquaintance of mine once did) it shouldn't crash and should give you some kind of helpful error message. If it did the first, it's a category 1 defect, if it didn't do the second, then it's a category 3 defect.
How would you rate bugs like 'Misspelled company name'? I've seen plenty of bugs like that. They have no effect on how the program runs, but no way would you actually ship with them.
Or the "If we don't ship by x-mas, we go out of business, so we can let a crashing bug go in, because it only happens in rare cases"
The other problem with rankings is that anything 4 or 5 never ever gets fixed, so noone puts anything there. Then the real ratings are 1-3. Instant rating inflation.
"We only fix 1's and 2's" - and suprise, people only enter in 1's or 2's.
rbb
It just seems there isn't always time to make code that I'd want sold to me, or that's bug free. I don't know if that makes it acceptable to release buggy software however. Seems a bit of a grey area.
Um.... that post did *not* come from Michael.
"Well kids, you tried your best, and you failed. The lesson is, never try."
One would think that the open source 'Release often' group would understand that a well known & documented bug in a shipped product is still better than a fixed bug in something in the lab -- atleast thats how customers feel -- especially when they can expect updates. That said, our clients expect near perfection and they should for what they're requesting -- especially if they're willing to do hardcore testing on their purchasse and report back.
Rod Taylor
Yes, that's my standard too; but these days people don't want software right. They want it now. More and more I find myself lowering my standards in order to meet deadlines. Since I can't get away from that, it would be nice to have some indication of what can be expected from the software. "Okay, you got it now, but you also got this and this bugs..." At least I'd feel more honest.
It's irresponsible to release software that has at least one known Severity 1 or Severity 2 bug. The software is obviously not even complete, so why would you even toy with the idea of releasing it? And if those bugs aren't *known* at release time, then congrats, your testing team needs a good kick in the pants. "What testing team?" Oooh. That testing team. Yeah.
120 characters isn't enough to explain it.
I'd like to take small issue with this statement - I'd claim that software engineering, like any sort of creative process (and often like many forms of engineering), *IS* an art. If it weren't, you could take your specs & automate a process which could grind out the perfect solution w/o human intervention.
However, many so-called "artists" have a false idea of exactly how much discipline goes into a truly good piece of art. They believe that a "creative" person can bring forth the perfect solution for any problem full-blown from the murky depths of their unconscious minds. They point to tiny single-person projects that they've completed as "proof" of that software development model. Then they run into software projects where the details are too large to fit into one person's mind, and/or there is more than one person working on the project.
_THAT'S_ where the discipline and all of the software development "techniques" and methodologies are required - to help us mere humans keep the overall information flows cohesive enough to be useful. But after the problems have been broken down small enough for individuals to completely mentally assimilate, I believe that the results at those steps are still creative in nature, and can be classified as "art".
All the bugs that I submit are important. They are severity one.
All the bugs that you submit are useless. They are severity five.
The point? Get an objective standard, and don't take the submitter's priority too seriously. Otherwise everything will be "highest", at which point you're better off having no priority system at all- you'll spend time pretending to prioritize things and have no results.
I can explanate how to administrate your network. You must configurate and segmentate it, so it can computate.
As pointed out earlier just severity doesn't solve all of issues about what bugs have to be fixed before shipping. One suggestion was priority which is fine except that it begs the question, "Who gets to decide the priority?" What I've used as the second 'axis' is a property we called "visibility."
Imagine that your code crashes on alternate Thursdays when there is a full moon and the user leans on the space bar with their left elbow. Also imagine that your software misspells the Client's name on the opening screen. Which bug is more visible? Which bug should be fixed first? If all one cares about is severity it would be the first bug. Depending upon who is setting the priority, it could still be the first bug. But I'm going to claim that while in an ideal world the customer wants both fixed if they have to choose, they will choose the second.
So to get back to the first issue, "a standard for what constitutes a sev 1 bug," most of the other messages have a good suggestions. Pick your favorite and get the dev & test groups to buy in. Then work on the second axis, (visibility or priority or whatever makes sense in your environment.) Finally remember the roles, it's test's job to document as many defects as possible, it's dev's job to develop the code and fix whatever is possible within schedule and budget constraints and it's evil management's job to decide what those constraints are. Anything longer and potentially more profitable than "Hello, World" has bugs, the key is accurately knowing and assessing them so that a good decision about shipping can be made.
MAK
1. End World Hunger 2. Promote World Peace 3. Make Enough Money to Buy & Sell Each One of You
I tend to believe a product does not ship until it is finished, I put out beta versions, but a stable version is not going to go out until the product is rock solid stable, yea, we can miss bugs, beta should take care of that, any more we get done asap, on that version, not wait until the next stable version.
Take care - Robert Lazzurs, new Everybuddy maintainer
A bug (in my development experience, limited though it is) has both a severity and a frequency. A sev 1 bug that rarely happens might not be as important as a sev 3 bug that is a frequent annoyance. We always eliminated all known Sev 0 bugs (crash server; yikes) but the occasional Sev 1 bug that was very hard to kill AND deemed unlikely to affect the customer was left in so we could release now and fixed in the SP or point release. Yes, leaving bugs in sucks but sometimes not shipping until they are all dead sucks more.
Q:How many libertarians does it take to stop a Panzer division? A:None. Obviously market forces will take care of it.
They said: "We follow industry standard processes" I said: "Care to enumerate them?" They said: "I'm sorry, but those are trade secrets". No shit. Unfortunately, my client can't switch vendors... too much cost. Oh humm. We _do_ need industry standards.
How the did the above post get modded up to +4 interesting?
Logging bugs, and giving them "severities", is well and good. It can help in the same way that any effective software development tool helps, by enhancing communication. The moral is that bug tracking is useful only in the context of a team that can, and wishes to, use it effectively.
Wrong. Software engineering practices like tracking the severity and priority of bugs are aimed at creating higher quality software. Instead of developers spending time fixing bugs that are unimportant or not as annoying to the user, they have a way of categorizing and fixing bugs on a scale of importance.
Using bug count and severity as a measure of "releasability", however, is the fallacy of feeble-minded managers, who are afraid to make a decision without a number to back it up.
Yet people like you will bitch that Microsoft shouldn't release software with known severe bugs. I guess if they didn't track bugs they would be able to say they didn't know that there were any severe bugs with a clear conscience.
Software development (as practiced in all but epsilon of cases) is simply not a measurable process. You will only waste your time trying to quantify it.
Software development can be measured based on a large number of metrics. The fact that most people refuse to use software engineering practices that have been known for decades doesn't mean that the process is "immeasurable" it simply means our industry is full of people who'd rather subscribe to the myth that programming is an art instead of trying to make it an engineering discipline.
Releasability can only be determined by the judgement of the team working on the release (which may include developers, testers, release managers, beta testers, partners, etc). That is not to say that you should not draw upon the bug database for evidence upon which to base your judgement. But it requires intelligent interpretation, not counting up some totals.
So if the members of the team come up with the rules for what makes a bug a certain severity and what marks a bug as a certain priority exactly why can they not base releasability on these metrics? Quite frankly, waiting until the software "feels right" before releasing it is probably the most ridiculuos thing I've ever heard. On the other hand saying "we won't release until all the bugs that cause core dumps/segfaults/BSODs are fixed (severity one)" or that "we won't ship until we fix the annoying UI issues (priority one or two)" are quite reasonable even from a common sense point of view.
Some people consider this a failing of the software development process. I think they are too quick to condemn. The customer doesn't (usually) judge software by its bug count.
Interestingly most users of Windows 98 I know judged the software on its bug count (i.e. how many crashes needing reboots per day or other ridiculuos problems per day)
Most software is judged by an overall feel: if the software is compelling, many deficiencies will be overlooked. Further, it is difficult to guess ahead of time (even with beta testing) which bugs will really bite people and impede their use of the software.
If beta testers don't give you a feel for a large number of the bugs that will bite people on the ass then there are flaws in your beta-testing process.
Given the many interacting factors that determine the success of software, release decisions are naturally more art than science.
People like you are the major problem with the software industry. Those that believe that even though there have been increasing strides in the field of sooftware engineering, we should all still practice software development like it is a black art handed down from master to apprentice in a scene reminescent of guilds of the middle ages.
I know, I haven't presented any hard evidence. I'm arguing from experience in both free and closed software projects, and appealing to common sense. Most free software is released "when it's ready", without any metrics. Ponder on whether this is a strength or a weakness. And remember that when someone gives you number, the burden is on him to show that the number means something.
Do you really think Mozilla, GNOME, KDE, Apache, the Linux kernel, OpenBSD, etc are shipping "stable" releases of software without keeping an eye on bug severities and priorities? They may choose to ignore them for one reason or the other but they are keeping track.
--
Thats because QA has a inherent mistrust of developers ;D
Nothing personal, but if your developers can directly edit the Bug Report information after it's in the database, you can garuntee at least one will try it to avoid fixing a bug.
Syllable : It's an Operating System
You (and most of the people I've seen describe the "standard" at their companies in the comments section) must be in the retail software business, otherwise you wouldn't be asking this question.
I don't see how you figure that. Certainly all the products i've tested have not been retail software, and my experience still sits well with the experiences of the other people posting comments here.
QA is still QA, and customers are still customers. Retail or not.
Syllable : It's an Operating System
Yup, thats pretty close to the sort of scale most software houses seem to use, although in my current place we use a scale of 1-10, with 1 being most severe, and 10 being "Tester having a bad day, being pedantic"
:)
As a Software Tester myself, the four catagories you list make sense, as any good tester can imediatly class a bug into that scale. With the 1-10 scale I currently have to use, the diference between a 3 and a 4/5 bug is infitismal. So, don't let anyone try and tell you more catagories are better, they must certainly arn't
Syllable : It's an Operating System
But then there's the slight problem of schedule deadlines, timing and market share, isn't there? Hey, I care about quality of software, but from a business perspective, it's probably secondary. As long as it doesn't hurt business, a few (thousand) bugs isn't a big deal, right? Right?
This is what we use in my company too. (It's a pretty big software corp.)
Score 5, Insightful heh
Beg to differ.
We've always used two classifications, severity and sensitivity, with the latter being the "eye of the beholder" measure. You can have a severity one bug ("nobody goes home until it's fixed") that a customer may never see, but if they do it'll be a disaster. That would be high severity, low sensitivity.
You can also have cosmetic problems that don't affect the overall operation of the system, but they still piss the customers off something fierce--these would be low severity, high sensitivity.
Keeping these distinct helps prevent the severity from correlating too closely with how loudly the engineers are being cursed. It also helps prioritze things differently based on where you are in the release cycle.
Severity rates the effect of the bug and runs from 1 to 4. 1 being a bug that makes the software unusable (usually a bug that crashes the program) and 4 being something like a spelling mistake.
Priority rates how likely the bug is to affect a user and also is rated from 1 to 4. 1 means the user will always be affected and 4 means almost never.
Using this system I could see how it was possible to leave a severity 1 bug in the sorfware if its low priority 4.
"You can now flame me, I am full of love,"
Try looking around at the Software Engineering Institute site. They study such issues, and have standards of sorts. They're more oriented towards process than product, though.
I don't think that's an acceptable workaround for most users.
"Release early, release often" is only true for certain free software.
Not all are like that, and not all are of the same stability/quality.
I am not one of those who believe that "many eyes make bugs shallow". That is in the same category as the popular saying about monkeys, typewriters and Shakespeare.
That said, "Many eyes will find bugs of severity 1/priority 1" is probably true. But only trained eyes are likely to find _obscure_ bugs of severity 1 - e.g. remote root exploits. And there aren't many trained eyes with time to look at your code, so the trained eyes better be the developers themselves.
My point is, free software can often be better because there's less of a rush. Closed source software can be good too, or even better, if it's not rushed, and there are many examples of this.
Trouble is, often programmers are asked to ship the model/prototype.
Cheerio,
Link.
Because there is less urgency to release - developers release when they think it's ready, not when Marketing thinks it's ready.
However this can change when the developers start getting more pressure to release by a certain date.
Cheerio,
Link.
Of course the concept of a Free Software release is a little bit fuzzier than a commercial software release, as the software is generally available for use by anyone who's willing to download it well before the official release date. That means that the release is more generally a statement of confidence by the developers than anything else. As suce it's necessarily a bit murkier than a proprietary software release, as it's going to reflect a quantitative, rather than qualitative, difference in the number of users.
There's no point in questioning authority if you aren't going to listen to the answers.
1 - This is a drop-everything-to-fix-this-immediately kind of bug. A bug that makes it impossible to use the software at all.
2 - This level of bug is very serious and prevents some major part of the program from working. If we were doing a word processor, it might be cut-and-paste doesn't work.
3 - This level of bug displays a runtime error to the user but if they accept the error, the functionality is still there. Everything works, more or less.
4 - This is a spelling error or a size-of-textbox problem, something which the end user will notice but which obviously works properly.
Our standards say we are never to release with a level 1 or level 2 bug and, as much as possible, no level 3 bugs. Ideally, of course, we release with no bugs but that can be rather difficult.
Under no circumstances would we ever release with a level 1 bug because this simply means our app is broken.
--
Oceania has always been at war with Eastasia.
Using bug count and severity as a measure of "releasability", however, is the fallacy of feeble-minded managers, who are afraid to make a decision without a number to back it up. Software development (as practiced in all but epsilon of cases) is simply not a measurable process. You will only waste your time trying to quantify it.
Releasability can only be determined by the judgement of the team working on the release (which may include developers, testers, release managers, beta testers, partners, etc). That is not to say that you should not draw upon the bug database for evidence upon which to base your judgement. But it requires intelligent interpretation, not counting up some totals.
Some people consider this a failing of the software development process. I think they are too quick to condemn. The customer doesn't (usually) judge software by its bug count. Most software is judged by an overall feel: if the software is compelling, many deficiencies will be overlooked. Further, it is difficult to guess ahead of time (even with beta testing) which bugs will really bite people and impede their use of the software. Given the many interacting factors that determine the success of software, release decisions are naturally more art than science.
I know, I haven't presented any hard evidence. I'm arguing from experience in both free and closed software projects, and appealing to common sense. Most free software is released "when it's ready", without any metrics. Ponder on whether this is a strength or a weakness. And remember that when someone gives you number, the burden is on him to show that the number means something.
Finally, before someone else points it out, I know that software can be developed with sufficient care and process that it is measurable. But I challenge you to demonstrate that it can be cost effective in today's mainstream software markets. Until this is shown, I don't think that such methodologies merit consideration.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
The excess of moderator points hasn't subsided.
Wrong. Software engineering practices like tracking the severity and priority of bugs are aimed at creating higher quality software. Instead of developers spending time fixing bugs that are unimportant or not as annoying to the user, they have a way of categorizing and fixing bugs on a scale of importance.
You're saying the same thing I said. Bug tracking helps communication. People know where the nasty problems are, and nothing falls through the cracks.
Yet people like you will bitch that Microsoft shouldn't release software with known severe bugs. I guess if they didn't track bugs they would be able to say they didn't know that there were any severe bugs with a clear conscience.
That must be the most common fallacy on Slashdot: that everyone here thinks the same thing. I myself understand very well that Microsoft's business requires them to release software with serious bugs. I also know that I have the choice of software developed under a better (for me, at least) model.
Programming is certainly not engineering. Why? The requirements are nowhere near well enough defined. And with the pace of the field, it's not worth anyone's time to define them to "engineering" precision. This is simply a fact at this stage of the game.
So if the members of the team come up with the rules for what makes a bug a certain severity and what marks a bug as a certain priority exactly why can they not base releasability on these metrics?
For the same reason that "the law is an ass", "metrics are an ass". Even if developed by reasonable people with good intentions. See below.
Quite frankly, waiting until the software "feels right" before releasing it is probably the most ridiculuos thing I've ever heard.
Where did you get the idea that I want to release software based on good vibes and burning entrails? Look, I'm saying that you use all the evidence that you have available. But you use it more in the manner of a historian arguing a thesis, not a mathematicial constructing a proof. These are different modes of thinking, but each is valid in its domain. Or do you think history is new age bullshit?
On the other hand saying "we won't release until all the bugs that cause core dumps/segfaults/BSODs are fixed (severity one)" or that "we won't ship until we fix the annoying UI issues (priority one or two)" are quite reasonable even from a common sense point of view.
Depends what you mean. If you mean, "We reviewed the bug db and the beta tester feedback, and everyone agrees that we can release when the four known crashes are fixed, barring unforseen new issues", then I'm with you. If you mean, "Welcome to the new project, team; we're going to ship when the feature-set is implemented and there are no sev 1 bugs", you're fooling yourself. There are just too many possibilities, and no bug database can objectively capture the complex reality of a software project. Any hope that metrics will guide you is (pardon the slang) just wanking.
Interestingly most users of Windows 98 I know judged the software on its bug count (i.e. how many crashes needing reboots per day or other ridiculuos problems per day)
Really? They don't care about the zillions of applications they can run? What proportion of Windows users quit using it because of the bugs? If it's not high, then bug count is probably not a significant factor in the success of Windows.
Do you really think Mozilla, GNOME, KDE, Apache, the Linux kernel, OpenBSD, etc are shipping "stable" releases of software without keeping an eye on bug severities and priorities?
The projects I follow most closely are Linux and Debian. Debian explicitly eschews specific release criteria based on hard experience. Linus has been known to release kernels that don't compile.
They may choose to ignore them for one reason or the other but they are keeping track.
Hear my point: bug databases are good because they assist in keeping track. Releasing based on a formula is bad. "Ignoring bugs for one reason or another" represents the practice I advocate: look at the evidence (including bug database), apply intelligence, draw conclusion.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
Heh, I have always hated software that didn't even start up proberly, software that you buy, put in the disk/cd and when ya run it, boom, it crashs your system instantly.
::frowns:: )
Even worse are the installers then immediatly exit with a Divide by 0 error.
How in the world did they NOT catch that one? Didn't they even try to install their own software first???
Ugh, Black and White has had some simular issues. So far it seems to only work with 75% of the machines out there. The infamious level 5 bug should have been fixed by now. . . . hehe, it's not even that hard to make a work around for it, if Lionhead just went ahead and released at least a minor patch for that one little bug.
At least the Windows installer starts up most of the time, at least Microsoft got that right (notice that I said MOST of the time, heh. . . .
Need help treating your acne? Come here!
Seems like a fairly simple sceme to me -
I heard recently of a claim that Oracle using SunCluster caused data corruption when the primary node PANICed. Oracle were on to it immediately, investiagting to clear their name (which they did), since under *no* circumstances should data be corrupted.
Any software which does not live up to this kind of scrutiny should never be applied. An NFS link may die; a NIS server may disappear; the whole FS could disappear from under you; if your code cannot deal with *all* error conditions, you should be shot. fd=fopen() isn't enough; check for error conditions EVERYWHERE. This is something which Ada does right; okay, nobody uses Ada, but at least you have to check for everything. This *should* be done in every circumstance in every software project.
Unfortunately, this is very rarely, if ever, true.
#include <stddiscl.h>
Author, Shell Scripting : Expert Re
I used to work for a ISO-9000 compliant organization. They had strict quality control department. They has 3 levels of bug severity
Low - Small bugs. For example - incorrect labels for textboxes.
Medium - Allows you to complete your input operations but output results are incorrect.
Basically operations where data is not destroyed and recovery is possible by a simple fix to the program.
High - Program Crashes or security bugs.
I examined all kinds of bug tracking systems a few months ago, and Bugzilla just wasn't up to snuff. The major stumbling block was the lack of coherent, fine grained permissions in Bugzilla--it was difficult to create groups of users and let them view only certain groups of bugs, let alone prevent them from editing them. (I've been following Bugzilla since then, and see nothing indicating that this problem is being fixed, probably because it isn't a problem for the folks doing Bugzilla development).
As a developer, I had no problem with letting everyone edit bugs--after all, the trust has to start somewhere--but QA had a different viewpoint.
Dan
You give managers too much credit. I work for a huge company that depends upon software projects. The drive for management's push of sofwtare releases is profit, not politics.
"Population 1,656"
Everything works except "Open File".
Waltz, nymph, for quick jigs vex Bud.
Ship/NoShip rules have to be relative rather than absolute. The nature and severity bugs in the software the customer is using right now must be taken into account.
If you've discovered 4 "showstopper" bugs in the release 1.0 at the customer sites, and you've got a new patch that fixes 3 and adds 1, you usually have to ship the patch with 2 "showstopper" bugs in preference to the 4 baddies. Neither customers nor managers will boycott the buggy competition while you slowly grind out perfection. That "perfect" space shuttle software was awfully expensive to produce and took a very long time.
It's worse. You think you've got 2 bad ones in the patch, and you think you've got 4 bad ones in the field, but you could be wrong.
Bug severity isn't just program behavior. If you've fixed the software so there are more internal checks and better error messages, you may have to ship this diagnostic version to fix the bugs. Hopefully, you have prevented this embarrassment. But maybe not.
The decision to ship may be fed by quantitative factors, but the decision requires human judgement. But you knew that.
A more typical problem is that written specs and written quality standards usually do not exist. So how do you quantify the degree of conformance to the specification? Sometimes I can cut the Gordian knot by asking "Will somebody call us up to complain, or ask for support?". Suddenly, everybody agrees it's a bug (or it isn't). Users will complain about "features" they don't appreciate, but then, they should.
--
I18N == Intergalacticization
These are loose definitions but are used about this way in the software project I'm on... you can alter these somewhat, but they're close to correct.
Sev 1 bug:
This is a show stopper. You can't release software with this bug, the software just won't be usable.
Sev 2 bug: major functionality is broken. The software works in many ways, but you'd be embarrassed if you tried to present this (think win98 crashing on the day of its debut before hundreds- think of a billboard run by NT blue-screening in public.)
Sev 3 bug: can be handled by a workaround but really ought to be addressed. Can also be a documentation bug that needs to be fixed before docs can be shipped. (please don't ship docs after you've shipped the product.)
Sev 4 bug: works most of the time, but only under specific conditions. showed up in testing, can be addressed by a technote on the website.
Sev 5 bug: works most of the time, under most conditions, showed up in testing, can be addressed by a technote.
Any bug addressed by a technote must be fixed in the next release, or else. Any bug sev 1 or sev 2 bug must be fixed or you don't release. Sev three is one of those that you fix at the end of the release cycle, or handle with a technote and take care of it in a service release or next full release.
A host is a host from coast to coast, but no one uses a host that's close
- Crash Bugs - Crashing the OS, application, or another App. Includes Lockups
- Data Bugs - Data loss or corruption. This is Bad(tm)
- Functionality - Does the damn thing work?
- Compatibility - video and print driver issues for example, and many other more obvious issues
- Usability - Navigation, etc. This is an area intense debate.
- Design - Usually consistancy of design, Icons, etc. closely tied in to Usability. Did we do this right in the first Place?
- Content Bugs - typos, incorrect data, etc. Can include entire missing pages, etc. Note that legal issues for things like copyright notices can be important.
- Race Conditions - When things go wrong because you fire off widget 2 while widget 1 is still calculating. Can depend on system speed.
- Professional Polish - for lack of a better term - a subcategory of design bugs. Did you use the exact same size, color, and positioning of button etc everywhere in the app? Or did programmer one do it just a little different compared to programmers 2 and 3, resulting in a slightly off change in appearance or language in different areas of the app?
Priorities usually go from- Show Stopper -
- High -
- Medium -
- low -
Obviously something like the text color being dark blue instead of the spec of black doesn't hurt anything and would be a low level bug.I am sure there are plenty of QA sites out there, but I don't have the master page of links right at hand.
Check out the Vinny the Vampire comic strip
"It is a greater offense to steal men's labor, than their clothes"
Severity is in the eye of the beholder, and thus should always be classified in terms of its impact on the users of the software, as in
Severity 1: Essential functionality inoperative, no presvention or workaround possible.
Severity 2: Essential functionality inoperative, workaround or limited functionality use possible.
Severity 3: Reliability of essential functionality compromised, no workaround possible.
Severity 4: Reliability of essential functionality is compromised, workaround is possible.
Severity 5-8: repeat descriptions 1-4 for non-essential functionality.
When prioritizing and resourcing fixes for defects classified as described above, a separate independent assessment needs to be made as to the scope, complexity and risk of the fix(es) required, but I must emphasize that these are NOT related to the severity of the defect.
Also, I do not believe that error messages should attempt to characterize the severity the error condition they are reporting. It is to be left to the user to determine what constitutes essential vs. non-essential functionality, or whether a workaround is possible or not. EVERY error message should, however, directly contribute to giving the user/support enough information to correctly and consistently correct the problem. Sepcifically, every error message should report:
1) date/time
2) the identity of the software reporting the error
3) the exact operation that was attempted (report method names and arguments wherever possible)
4) the system-level completion status of that operation (report errno or its equivalent at a minimum)
5) what the software intends to do now, eg. use a default value, retry, ask the user for a better input value, abort, etc.)
Way to much otherwise well written software has been rendered useless by poorly thought out error handling and messages. Shame on us.
There are ways to decide:
Is the action being attempted in the requirements spec?
No required functionality that doesn't work is less than a severe error.
However, if you believe you can ship a 1.0, and noone will notice it, then you can delay it perhaps.
However, don't confuse _severity_ and _priority_.
FP.
--
Also FatPhil on SoylentNews, id 863
(I was a developer who moved into the testing area because my employers and peers found I had a knack for exposing weaknesses in a product. So let me draw on both sets of experiences to provide a "developer-oriented" answer to this question.)
The first goal of DVT/QA testing is to ensure that the product functions as the functional specification and documentation says it should.
That means DVT first tests each operation, each feature, and each option on each environment with a variety of recognized and accepted test data sets. Not only are DVT testers looking for proper operation with proper data, but also proper operation with bad data. In other words, do the error-catching routines actually catch errors? If there is a sequencing error, is it reported properly? Recovered from properly?In many DVT environments, the test conditions are required, to the best of the DVT system's ability, to test all of the code, using profiling and test coverage tools to indicate when each path has been checked.
Like development, design verification and test (DVT) and product quality assurance have as many levels of abstraction as your typical development effort. Properly structured testing looks at the major stuff first, options next, and then they go for the anomolies. Indeed, in the DVT efforts I have managed, the checkout of a software change required the "20-minute test", the "3-hour test", and the "24-hour test" before I would approve the change to be part of a release candidate subject to full regression testing. (From project to project, the actual length of time taken for each of these tests differed, but the names stuck.)
The second goal of DVT/QA testing is to find operational problems in-house before the customer finds them in the field.
This is the time when testers "bang on the product" to determine if there are any weaknesses or coupling effects. The best testing is when you have a good number of alpha testers that use the product in the manner it was intended to be used, followed by a good-sized group of beta testers. Beta test management is an art unto itself, with the most important function being the selection of the beta testers themselves. Some companies miss the boat by only soliciting one type of beta tester. The best beta programs court fringe users, those who would use the product in unusual ways, in order to ferret out more error conditions.Frankly, I find development easier to do than testing, but testing is to me the more satisfying activity...especially when my efforts expose some really funny cascades.
There is a third aspect to testing that doesn't get much play in companies: reproducing field problems. I have some war stories if someone serves up the beer...
This is probably a good time in this thread to throw out an allied problem: what happens when management doesn't like the results of the bug evaluation.
True story: a corporate-wide standard for bug determination was defined in a Fortune-50 corporation. In a project, though, the director of engineering decided that the standards "didn't reflect reality" -- or rather, that the numbers made him look bad. It also meant that there was no way he would be able to ship the product on time, because the number of ship-block bugs was just too high for his development staff to clear in a manner acceptible to him and to his superiors.
Now, the definition was pretty strict: Severity 1 bugs were features in the specification that were implemented and broken in significant ways, severity 2 were features in the specification that were implemented and met the operational requirements but not performance requirements, or were not implemented and marked in the specification as required for release, severity 3 were features that were implemented but broken in non-significant ways (cosmetics, usually) or features in the spec that were not implemented and marked as not required for current release, severity 4 were features that met spec but were identified as areas for improvement, and severity 5 were spec change requests. The rule is that the product MUST NOT ship with severity 1 bugs, SHOULD NOT ship with severity 2 bugs (release required a review and analysis of each S2 bug by corporate QA), MAY ship with severity 3 bugs, and MUST ship with priority 4 and 5 bugs.
So, Job One for this manager was to pull a Captain Kirk and change the rules. Severity 1 was redefined as "causing a system crash". Severity 2 was redefined to "causing an operation failure that does not result in a system crash". Severity 3 was everything else that used to be in S1 and S2, as well as what was originall in S3. That meant that missing or severely broken features identified as required for release obtained an S3 rating instead of the original S1 or S2 rating. Or, in other words, a broken product could now be shipped!
Of course, it didn't do very well in the market, but the manager in question still got his raise -- and I suspect that's all he cared about.
The root cause of this breakdown was that QA/DVT was an Engineering function, not a separate function. The guy saying "no" should be a peer to the guy saying "yes". (It would have helped if the original bug definitions would have been part of the specification, because the spec is king within this company -- but that didn't happen.)
...would seem to apply.
"In everything do to others as you would have them do to you..." -- Matthew 7:12, NRS TranslationA generally good philosophy, regardless of your adherence to ceratin world religions.
When releasing software, I ask myself: Would I be pleased if somebody gave/sold this program to me? It isn't so much a matter of numbering conventions of bug severities -- it's a matter of pride and responsibility.
--
Scott Robert Ladd
Master of Complexity
Destroyer of Order and Chaos
All about me
Fixing bugs has a cost associated with it (e.g. QA employee's salaries, test beds, test software, time required to perform tests.)
NOT fixing bugs has a cost associated with it (tech support staff, development staff's time and resources required to fix the bugs, testing the fixes, the cost of packaging and shipping fixes, loss of repeat sales, bad press.)
The question for a commercial software vendor, then, is this:
NOTE: How the costs are perceived seems to depend somewhat on the size of the operation and the nature of the product. Here are some examples from my experience:
I view testing and bug severity/priority/frequency assessments as a means to help management make an informed decision as to when software is ready to ship.
It is their responsibility to know the market, to know the business decisions as to when fixing bugs costs more than leaving them in. It is my job to use the best practices and tools that I know how to help provide them the information they need. In short: their expertise is in the business side; mine (as a QA professional) is in the technical side.
Lastly, 1*1000 = 1000*1; one nuclear bomb can ruin your whole day, but getting bit by a thousand mosquitos can ruin it, too... don't ignore the minor bugs! (Besides, what may appear to be cosmetic bugs can make it incredibly difficult to develop and maintain automated tests that depend on field prompts and error messages!)
There is absolutely no way to consistently determine whether one bug is more severe than another because users needs are different. One bug may be catastrophic to one user while the same bug may be of no consequence to another.
As one of my CS professors likes to say - Software Kills People. Therefore, you should treat all the bugs in your software accordingly.
An OS that crashes on a PC at home may not be that big of a deal to you, but if that OS is installed on a machine that's powering a space station and the lives of the crew depend on it, then you'll probably be a little more concerned about the bugs in that environment.
Despite the popular belief, it is possible to write software that is completely free of bugs (just look at the space shuttle launching code). It is a lot more difficult to do that, sure, but nevertheless quite doable.
Please please please mod this down.
1) We've all seen it.... a lot
2) It would have been better to post a link rather than pasting in all this text.
I also want the criteria for a standard for what alpha and beta versions are.
:-))
alpha Not all the features / business cases are part of the code. The developers are still adding them.. The code may be unstable, that is crashes for no reason. May contain bugs that trash files. This level should never reach testers who cannot recover their hard drives.
beta All features coded. The application runs OK. Outside testers (outside the development group) are asked to test it. After a while a public beta may be used where people outside the company may try it.
This is the traditional way of separating the two. Of course recently some companies have passed off alpha level as beta, the beta level as release. But that's another story
-----------
- - - - - - - - - - -
I am a programmer. I am paid to produce syntax not grammar. Deal with it.
Notice that there seems to be a huge number of mod points floating around? (I just got another 5 before I finished with the last batch.) Notice that just about everything is flying up to +5 and then back down?
Apparently Michael wasn't kidding:
Due to an as-yet-unexplained error in slashdot's database, over the past few days fewer moderator points than usual have been given out. We've fixed it, but will probably have to restart the webservers for changes to take effect - probably a lot of people will suddenly receive moderator points in the next hour or two.
Questions:
I'll be gracious and not use one of my other accounts to mod this up. ;-)
Unsettling MOTD at my ISP.
I don't know about that but it *did* come from his account. And distribution of mod points *has* gone berserk.
Unsettling MOTD at my ISP.
Most software companies are not in the retail software business (i.e. they do not sell boxed software to retail stores), but write software for specific (mostly corporate) customers. In that case, different rules apply. Your customer will have given you a list of requirements. If you do not meet the requirements, you have a bug. It might be something as simple as "I clicked the open button and it didn't open", but it would be still be a "critical" or "major" bug. Now, since you (should) have a good relationship with your customer, you can try and explain to the customer what would be involved in fixing the bug, and get them to agree to ship with this bug (they will probably ask you you to fix the bug in a later release).
Since you don't appear to have customers that tell you what is acceptable and not: GET THEM! Pull some people in off the street and have them test your software. If they run into bugs all the time and it prevents them from using the software for its intented purpose, you CANNOT ship. Every bug they (as average first-time users) run into should be fixed before release.
Um.... that post did *not* come from Michael
Unless you are claiming that micheal's account got hacked then that post came from him.
I always want my manager, his manager and as high up the management chain using the software... Nothing like having a VP of the company come down and saying, Hey this doesn't work... What they hell is going on to get a few more bugs fixed. I have seen managers use the excuse, it isn't ready for them to use it. Well if it isn't ready for our own company to use for free, then why are we trying to sell it to others. Nothing like a little EOOD (Eat our own Dogfood) to fix quality problems.
Some software I wrote recently stood inspection by the College of American Pathologists. They actually inspected the whole information system. But their means for criteria I thought applied to this discussion.
They have two types of problems that they can cite you for. They are called deficiencies. A "Type I" deficiency isn't terrible, but you still have to provide them with a solution to the cause of the deficiency. A "Type II" deficiency is more severe. If you are cited with a Type II, you have 30 days to fix it, or else they shut you down.
I think even something as simple as this method (where "Type I" bugs are allowable, but the software is not allowed to be released if it has any "Type II" bugs.) could easily be extended to the software industry (and not just the part that overlaps into medicine).
1."Show Stopper." The client absolutely refuses to sign-off on the project until this issue is solved.
2. "Mickey Mouse." Very minor error that can usually take a minute or two to fix, but wastes more of your time in the error reporting, documentation, and follow-ups.
3. "Ghost." Only the client sees it but can't replicate it or give you instructions so you can do that. The error is real (the client will send you a screen capture), but they can't helo you in recreating it.
Pedro
Pedro
----
The Insomniac Coder
Any bug is worthy of being a severity bug...as long as it can make your system crash, or cause some kind of major malfunction, its bad. MS would be promiement on this kind of program though, hehe
Slashdot Hypocrisy at work?
The glib answer is if you have to ask, you don't understand your software's use well enough.
It is also a failure in understanding processes. There are showstopper bugs... ('defects' is actually a better term for bugs), and those which are mere annoyances or happen in negligible circumstances, with only annoying results.
Complete List of Windows 95 Error Messages
.TMP files all over the place
WinErr: 001 Windows loaded - System in danger
WinErr: 002 No Error - Yet
WinErr: 003 Dynamic linking error - Your mistake is now in every file
WinErr: 004 Erroneous error - Nothing is wrong
WinErr: 005 Multitasking attempted - System confused
WinErr: 006 Malicious error - Desqview found on drive
WinErr: 007 System price error - Inadequate money spent on hardware
WinErr: 008 Broken window - Watch out for glass fragments
WinErr: 009 Horrible bug encountered - God knows what has happened
WinErr: 00A Promotional literature overflow - Mailbox full
WinErr: 00B Inadequate disk space - Free at least 50MB
WinErr: 00C Memory hog error - More Ram needed. More! More! More!
WinErr: 00D Window closed - Do not look outside
WinErr: 00E Window open - Do not look inside
WinErr: 00F Unexplained error - Please tell us how this happened
WinErr: 010 Reserved for future mistakes by our developers
WinErr: 013 Unexpected error - Huh ?
WinErr: 014 Keyboard locked - Try anything you can think of.
WinErr: 018 Unrecoverable error - System has been destroyed. Buy a new one. Old Windows licence is not valid anymore.
WinErr: 019 User error - Not our fault. Is Not! Is Not!
WinErr: 01A Operating system overwritten - Please reinstall all your software. We are terribly sorry.
WinErr: 01B Illegal error - You are not allowed to get this error. Next time you will get a penalty for that.
WinErr: 01C Uncertainty error - Uncertainty may be inadequate.
WinErr: 01D System crash - We are unable to figure out our own code.
WinErr: 01E Timing error - Please wait. And wait. And wait. And wait.
WinErr: 020 Error recording error codes - Additional errors will be lost.
WinErr: 042 Virus error - A virus has been activated in a dos-box. The virus, however, requires Windows. All tasks will automatically be closed and the virus will be activated again.
WinErr: 079 Mouse not found - A mouse driver has not been installed. Please click the left mouse button to continue.
WinErr: 103 Error buffer overflow - Too many errors encountered. Additional errors may not be displayed or recorded.
WinErr: 678 This will end your Windows session. Do you want to play another game?
WinErr: 683 Time out error - Operator fell asleep while waiting for the system to complete boot procedure.
WinErr: 625 Working Error - The system has been working perfectly for the past ten minutes
WinErr: 902 Screen Error - The system is working perfectly, I'm not lying, your monitor is wrong
WinErr: 72a Crucial Halt - Hang on, WHAT was that?
WinErr: 72b Memory Error - What? Tell me again.
WinErr: 39c Disk Error in drive a: - Although your disk is in perfect condition (I just formatted it), I don't like it any more
WinErr: 983 Hard Disk Error - The files on the hard disk were neatly arranged and fully optimised, so I had to mess them up and put a couple of hundred
WinErr: 294 BlackMail Error - $500 to Gates or your computer gets screwed
Winerr: 294b BlackMail Error - $500 to Gates or I'll show your wife the JPG's you just downloaded
vroom vroom
360 degrees of Karma
I have worked as a software developer professionally for 15 years.
'Crash bugs' am I all to familiar with, and I've heard the concept of 'Class A' bugs. It would be very nice to have some sort of (unofficial) standard for bug severity.
I also want the criteria for a standard for what alpha and beta versions are.
Don't tell me that Im offtopic here. The issues are related.
When the components arrived from Japan, a note was enclosed saying: "We don't know why you want the faulty components, but for your convenience they have been packed separately."
The Rational Unified Process (of which Micro$oft is on the designing committee) recommends that 95% of a piece of software function before its release. Hmm... That explains a lot.
Personally, I find that beta-testing finds all the show-stopping bugs, and if you have any organizational skills at all in your coding, you can, with little difficulty, fix any bug that is reproducible. Then again, I've seen a whole lot of disorganized code out there.
Music speeds up when you yawn, but does not change pitch.
One thing we do, in particular when evaluating which bugs to fix in the next release, is look at pain (severity in other posts), what problem this defect is causing to the users, and also look at the number of users having this problem (in the case of a bug that is found before it is shipped, this would be the likelihood that the defect will occur in the field).
We have shipped a product that bluescreened under somewhat obscure conditions. I think of our 10,000 customers, one managed to get it to happen. We knew about it beforehand, the dev and QA staff had a fit, but we shipped anyway (of course, two of the twenty developers is still on that project anymore, but that's a different story ;)
Some defects that seemed trivial, like stupid cosmetic errors in reporting, were fixed, because every single user saw them and complained about them (well not every single user)
Of course, we did try to track down the blue screen, but we never did. According to MS, blue screens (on NT/w2k) are never the fault of the app running in user mode anyway, so we were able to shift the blame to MS. The point is, another factor in what you fix is whether or not it is fixable by your dev staff (how long you are willing to spend trying to fix it before you give up).
So the three big factors are: How severe is the defect, how many people see it or are likely to see it, and how long do we think it will take (or how long have we spent trying) to fix it.
Personally I like the "pride" idea, but it doesn't sit well with management/sales. Where you see this is in private companies like id software, that doesn't have to maintain earnings to the street will keep their stock at a reasonable level, etc. In a public company, management needs to cut their losses and get product out the door - they can always fix bugs in the next version, but they can't make revenue from the current version until they ship it.
An interesting dilemma to be sure, especially when dealing with four or five year old applications that nobody really understands anymore...
PhilTheChill
there are some rules we would want to stick to at minimum
1. Software can not harm a user or through any inaction allow a user to come to harm
2. Software must obey orders give by said user unless it would conflict with first law
3. Software must preserve itself and OS unless conflicting with first or second law
Adapted from the Genius A.Asimov
"All I can tell the "lesser of two evils" folks is that if they keep voting for evil, they'll keep getting evil."-Lp.org
At my company, we categorize bugs several different ways. This is all from an IEEE standard, but I don't know which one.
The SEVERITY of the bug is entered when it is discovered. It is one of:
1 - Urgent, causes crash
2 - High - no workaround
3 - medium - workaround available
4 - low - inconvenience.
Other information, such as the steps to reproduce, who discovered it, , a unique ID, and date and version discovered are also recorded. Most important is the TYPE of bug, which is one of:
Code Problem
Configuration Problem
Data Problem
Design ISsue
Enhancement
UI Problem
Documentation Issue
Bugs are assigned to a developer to be fixed. If they are easily fixed, then they are immediately fixed. The unique ID is used in the CVS checkin for the changed files. If the bug is NOT easily fixed, then the developer gives a level of effort, so we can determine the 'cost' of fixing the bug.
Bugs are then assigned a PRIORITY in a 'Configuration Control Board' (CCB) meeting with me (the project manager), several developers, documentation writers, and/or our end users/customers/testers. This can be a small, informal meeting, or a large process-driven meeting. I prefere them small. The PRIORITY in one of:
1 - Urgent, resolve immediately
2 - High, resolve ASAP
3 - Medium, resolve pre-release
4 - Low, desired, but not urgent
5 - on hold, fix in future release.
Our customer has deemed that all severity 1 bugs are automatically priority 1, and all severity 2 are pri 2.
Severity 3 and below are given priorities based on the cost to fix, but if we have too many severity 3's in one 'use case', they collectively become a pri 2.
It's not nearly as much process as it seems, and it really streamlines our development.
We use a tool called TestTrack to track all this, but I would prefer to move towards Bugzilla, as my company has recently jumped on the open source bandwagon.
another comment I should have included in my other post...
As a developer, my relationship with our testing department has changed over the years...
Originally, I thought the purpose of testing was to prove there were no bugs in the system.
They ALWAYS discover bugs. I began to think that the purpose of testing was to discover ALL the bugs, and it was the engineer's job to fix them.
They NEVER discover ALL the bugs. I now think that the purpose of testing is statistical... How many bugs are they finding, and what severity are they? This speaks to the quality of the system design and the quality of the engineers working on the system. If there are few bugs found and fixed, you can be confident you have a good system going out. If there are a lot of bugs found and fixed, I would worry that there are a lot more left undiscovered.
It is a common practice to ship with known bugs, but we NEVER ship with a pri/sev 1 or 2, unless a political deadline forces it.
Sounds to me like the company whose product you are using found more bugs than they could reasonably fix in the time they allocated to this phase of their development cycle, and now they are being forced, for political reasons, to ship with those bugs. That would be a red flag to me that there will be other, undiscovered, severe bugs in their software.
We classify bugs on two axes: severity and priority. This is actually a variation on how MS classifies their bugs internally, and we find that it's great for answering the eternal developer questions of "how bad is the bug?", and "which one should I fix first?"
Severity has to do with how badly the bug affects users:
sev 1: crashing bug, or data corruption
sev 2: feature broken, no workaround
sev 3: feature incomplete, or broken w/ workaround.
sev 4: cosmetic or UI flaw
Priority has to do with how important fixing the bug is to shipping the product. There are no names for the priority levels, but in conjunction with severity levels you can make determinations on what to fix, what to cut, and in what order.
Using this system, a sev1/pri1 bug is clearly the first bug I'd fix. A sev1/pri1 bug gets fixed before a sev2/pri1, and very rarely, a sev2/pri1 bug will get fixed before a sev1/pri2. These last two are the corner cases: sev2/pri1 is typically a very ugly sev2 bug, and a common reason for marking a bug with this classification is to fix a beta or demo issue. A sev1/pri2 bug is also rare - it means that the product crashes or corrupts data after a very obscure repro case.
I've been using this system for years, and I have yet to find a clearer method.
Invisible Agent
Invisible Agent
This post is a mirror; when a monkey stares in, no hacker gazes out.
Get your company to pick a book on QA/Functional Testing. A good practical one that spends a lot of time discussiong bug reporting and problem tracking is: Kaner, Falk, Nguyen, Testing Computer Software, 2nd ed, 1993, Thomson Computer Press. Widely available, from your favorite tech. book store. Another, less practical, more eggheady book is: Beizer, Black-Box Testing, 1995, John Wiley & Sons. It's important to have a well-documented bug classification and tracking system, that everyone involved in the project, especially the non-techie (read marketing and management) types can understand.