Slashdot Mirror
How Do You Get Better Bug Reports From Users?
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.
We have a big internal web app, when it throws an error it will log that error as part of a graceful error handling process. (Page, scope vars, user info, time, etc) From there, the reporting system requires a unique bug tracking number (the error number logged above) and it checks for uniqueness and to see if it exists before it lets you open it. The only edge case it doesn't catch is when our tracking mechanism is also broken and these tend to be severe outages that are pretty easy to diagnose and reproduce. That is the only bug that can be reported with an email, all others are responded to with a "Please log in the application and we will route it appropriately"
It's not working can you fix it. kthxby
I've worked in support organizations for 15 years. In a commercial environment where you can afford the staff, having a tiered approach works best - you have a help desk to gather and refine the questions and answer the small stuff, then work your way up to the engineers that wrote the code. The tough part of that is having a skilled enough help desk to know when to skip the canned questions and just forward a request on once you have the right information.
For organizations without those resources, you need to rely on the user base to be the help desk. Give them as much concise information as possible and frame the bug submission so that any and all needed data is in the report. Then it's up to the developer to give good information back to the user.
As an example, I had a problem with my laptop's trackpad going wonky. Ubuntu made it really easy to compile information about my system and submit it as a bug report, then open it for me so I can add any additional text I wanted. The answer I got back asked me to try a different kernel, and included well-documented links and information on how to get and install it. Just saying something like "yeah, go grab something out of backports" doesn't help the user if they have no idea what you're talking about.
If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."
If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.
If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.
If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.
If your users are dumber than a pack of hammers, then I have no answers for you.
I am officially gone from
When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.
"Actually, I enjoyed this in the same vague, horrible way I enjoyed the A-Team" P. Opus
The sooner, the better. You need an agile bug response process.
In your bug tracking system, provide a bug template they need to fill in.
Encountered situation:
Expected situation:
Error messages, if any:
Steps to reproduce (please provide screenshots if appropriate):
If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.
Your users aren't code masters and never will be.
It doesn't involve users to be code masters, it just involves them engaging their brain a bit. I frequently get bug reports along the lines of "something broke last week, came up with some error (I don't remember what), but I rebooted it and its fine for now; please fix it so it doesn't happen again". You don't have to be a "code master" to figure out that reporting a bug and not actually tell me _what_ broke, what the error was or let me log into a system that is currently exhibiting the problem so I can look myself is not going to be condusive to me fixing things.
And this stuff happens again and again with the same customers... "one of our users is having a problem accessing some websites, please can you fix it?" - ok, so I have to go back and ask "which user" and "which websites", if I'm lucky the customer will give me this information, if I'm unlucky I get "I didn't ask". A week later I'll get an almost identical problem report from the same person about a different (but extremely similar) problem, and again none of the information I ask for _every_ time is included.
Also the great one that comes up occasionally is "this has been broken for a month and you haven't fixed it yet!".. well, if you'd actually told me that there was a problem I might've known to look into it, but since this is the first I've heard of it...
http://blog.nexusuk.org
On every page of the site that I did for this company, in the upper right corner, there is a button labelled "BUG!". Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in. The dialog box has a "Submit" button.
What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically.
And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page.
The BUG! button was invented - by me - as a reaction to Bugzilla, which seems to be designed to keep users from reporting problems. I can ignore pointless bug reports, but I want to hear everything.
... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.
If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
No matter what process you employ it will fail, what you need is someone who is willing to follow-up with the users and step in their shoes, I have realize users are always willing to report a bug as long as it makes them feel important and FOLLOWUP.
I believe it was Google who at some point had an option to highlight the portion of the page that showed the problem, as well as submit a comment about it. Being able to see exactly what this user is talking about would help quite a bit when I go to fix a bug.
--"Forget the nectar of the Gods, just give me some Mountain Dew."
Layer8-error
ID-Ten-T error (ID10T) ...
bickerdyke
You can reproduce the issue by getting their keystroke history. File an FOIA request with the NSA.
Hello everyone,
I have tried a bunch of ways. Trained the 'expert' users in the area on how to put in a better ticket. Sent tickets back to them because of lack of information. Judicious of cattle prods and a tack hammer... However, users will use what method is easiest to them, which tends to be:
In God we trust, all others require data.
- Record all their actions on a self overwriting one hour long file.
- Give them a "one button" way of reporting a bug. The button saves the user and the time, then waits for five minutes and then sends you the recorded actions file.
It's simple to develop and gives you a lot of information. It might be illegal in some countries.
You *owe* them a bug free product. If you did your job properly, this complaint would be moot.
OK. How you gonna get there, Chief? Certainly not by testing, since that would involve testing which means "using". Maybe not by end users but by someone "using" the software, whether you call them QA or whatever. So, what, you just write it bug-free in the first place? I don't know what you make, but I think it's safe to say that you should be paid more.
I am not a crackpot.
There are many bugs that would easily be detected by actually using the application on a daily basis.
Users do not work for you. When they do post bug reports, it is most likely in frustration.
Providing that the developers actually have the ability and resources to do that. Most (much?, some?) software isn't developed for use in the programmers' domain, and real-world deployments often have conditions that aren't fully replicated or anticipated even in even a good simulated environment. Your developers probably don't operate a retail POS or limestone quarry or credit card call center or whatever environment the product is used in.
Not that this is an excuse not to do your best to test it out before handing off to the end user, it's just that the conditions that make the bugs appear are sometimes the very conditions you don't anticipate and can't or don't know to test.
I am not a crackpot.
Your users aren't code masters and never will be.
It doesn't involve users to be code masters, it just involves them engaging their brain a bit. I frequently get bug reports along the lines of "something broke last week, came up with some error (I don't remember what), but I rebooted it and its fine for now; please fix it so it doesn't happen again". You don't have to be a "code master" to figure out that reporting a bug and not actually tell me _what_ broke, what the error was or let me log into a system that is currently exhibiting the problem so I can look myself is not going to be condusive to me fixing things.
And this stuff happens again and again with the same customers... "one of our users is having a problem accessing some websites, please can you fix it?" - ok, so I have to go back and ask "which user" and "which websites", if I'm lucky the customer will give me this information, if I'm unlucky I get "I didn't ask". A week later I'll get an almost identical problem report from the same person about a different (but extremely similar) problem, and again none of the information I ask for _every_ time is included.
Also the great one that comes up occasionally is "this has been broken for a month and you haven't fixed it yet!".. well, if you'd actually told me that there was a problem I might've known to look into it, but since this is the first I've heard of it...
Right. Because users are filing bug reports. They're calling for customer service/support. You get good bug reports by having good people answer the phone. I'll accept, "something broke," from a user. However, I find it inexcusable that a Helpdesk rep sends me a bug report that says the same thing or, worse, asks me to call the user for details. That being said, if you can't rely on Helpdesk or the user, then your software should be able to rollup everything you need to resolve the issue.
Requiring a user, after the fact, to recall an error message is futile. They simple have seen to many varied ones and their brain goes 'oh an error message' not 'oh a 504 error' or 'oh a invalid data type error'.
Believe it or not a user that doesn't remember the error message is not the worst kind of user.
I have some users that love translating text errors into numeric error themselves. Any time a page doesn't load, it's a 404. So that's what they report. "I'm trying to connect to thisdomaindoesntexist.com and I'm getting a 404."
I think the only chance is to create your own breed of users... May take some time ;-)
That is nothing. I love it when users are overly specific, because they are always specifically wrong.
In user speak "My Password won't work." is code for everything, including an unplugged computer.
Troll is not a replacement for I disagree.
Not to mention the 'horse-to-water' problem with a formal ticketing system. If you have a phone, and they know your extension, they'll just call you directly instead of, you know, actually doing anything. And unless you've got C-level buy-in on the ticketing system, telling the user to go open a ticket is usually a Career Limiting Event.
No, the way to enforce usage of a ticketing system is to not tell the users about any other way to contact you.
Never underestimate the power of stupid people in large groups.
I had just sent the below list of things required in a bug report to a group of users/testers this morning......
In general, the following items should be included when possible:
â Reproduction steps
â Expected behavior
â Observed behavior
â If available, any related steps or values that worked (for example when a bug âoesometimesâ happens, itâ(TM)s useful to know when it works and when it doesnâ(TM)t)
Then, the trick is to reject or de-prioritize any bug that doesn't conform to the above. Train the users that if they provide accurate information, they get quicker response. It works for Pavlov.
Maybe, the user didn't want to confuse you by sending in multiple problem reports at once. Or maybe the user's manager thought that would overload IT, or make it appear they were just complaining too much. :)
I had users who just ignored, or worked around errors, errors that had never been reported. Some were user errors (resolved by both program changes to prevent those errors and user training), some were UI errors that didn't impact the results, and some were program errors with actual consequences or impact on the data or utility of the software. I found out about these errors by watching the users, or as a side issue when they were having some other problem that they did report. I explained that reporting all errors ASAP gives the developers a broader view of the potential problem, and allows all errors to be fixed faster and more completely. I trained them to report every problem, no matter how small, and made a point of addressing those errors as quickly as practical to reinforce the behavior of reporting them. And, of course, I explained that unreported errors were very unlikely to be fixed, ever.
make imaginary.friends COUNT=100 VISIBLE=false
One place I worked at recently, the users had almost totally lost confidence in the ticket system. It required me to have comprehensive personal followup on tickets for a solid year before people started trusting the system again. Techs in charge of the queue were summarily deleting vague or "cannot reproduce" tickets with not so much as a response. Tickets were also silently closed whenever a tech though they had fixed the issue. (several tickets had been freshly ressubmitted over a dozen times because the tech didn't understand the user's problem and thought he had fixed it and the user was just too dumb to realize it) Tickets that had several dups in the system from the same user or few users were deleted en mass just to clear the clutter. ("if it's still a problem I'm sure they'll submit another ticket shortly") It was terrible. They had conditioned users to submit duplicates almost daily until they thought their problem had been fixed.
Overall a great comment, but I'm puzzled by "the user was just too dumb to realize it." Chances are the user realizes exactly what is going on in that situation. Techs are closing tickets without resolving issues due to either laziness or metrics which grade them on closing tickets and not on resolving issues.
With globalization and out-sourcing, walking over to the support team and standing over someone until your issue gets attention isn't an option. Then systems are put in place so submitting a ticket is the only avenue of communication for the user. When you make sure the only tool the user can access is a hammer, you can't complain when they treat everything like a nail.
The rest of your comment addresses issues within the support team and ticketing system that lead to duplicate tickets, so perhaps you meant dumb as in insane as in repeating the same action (creating a ticket) and expecting a different outcome.
To a large extent the issue of bug reports from users can be summed up as "garbage in, garbage out." You want more detail than "it doesn't work"? OK, then how much detail do you give users in your error messages? And I mean plain language messages end users can understand. Displaying the memory address of a misbehaving bit is very detailed, but not something you should expect the end user to remember or communicate.
For getting better context for issues or steps for recreation, how good are your instructions? Can the user include in a bug report, "I was performing procedure A and get the error on step 27"? If not, then don't expect that level of detail from the user. For one, most people don't have an active (and accurate) buffer of the last 5 or 10 things they did. What were you doing 3 steps before the error? Well, I didn't know an error was coming, and so I didn't know the thing I did 3 steps before was going to be important, and so I have a general idea of what I was trying to do, but I don't have the exact steps documented.
Also in many instances it is not practical for the user to try to recreate the issue. If it's an eCommerce system, I'm not going to repeatably submit an order trying to come up with a reproducible sequence to get an error and risk getting charged multiple times. There are many systems subject to regulatory and audit oversight where creating test records or dummy data is not acceptable. So, do give your users access to a non-production system where they can recreate their issue without affecting real data? If not, you're limiting their ability to provide that level of detail.
I'm not saying the issues are all on the technical side. I've been on the support side and seen the "it's not working" reports where the user doesn't even specify which application isn't working (let alone which feature or option). But there is a lot the technical side can do to provide the users with the tools to submit useful bug reports.
Users will do things the devellopers never dreamed of, because no develpoer can have the level of ignorance of the software that the user has.
-- hendrik