Slashdot Mirror
How Do You Get Better Bug Reports From Users?
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.
It's not working can you fix it. kthxby
I've worked in support organizations for 15 years. In a commercial environment where you can afford the staff, having a tiered approach works best - you have a help desk to gather and refine the questions and answer the small stuff, then work your way up to the engineers that wrote the code. The tough part of that is having a skilled enough help desk to know when to skip the canned questions and just forward a request on once you have the right information.
For organizations without those resources, you need to rely on the user base to be the help desk. Give them as much concise information as possible and frame the bug submission so that any and all needed data is in the report. Then it's up to the developer to give good information back to the user.
As an example, I had a problem with my laptop's trackpad going wonky. Ubuntu made it really easy to compile information about my system and submit it as a bug report, then open it for me so I can add any additional text I wanted. The answer I got back asked me to try a different kernel, and included well-documented links and information on how to get and install it. Just saying something like "yeah, go grab something out of backports" doesn't help the user if they have no idea what you're talking about.
If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."
If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.
If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.
If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.
If your users are dumber than a pack of hammers, then I have no answers for you.
I am officially gone from
When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.
"Actually, I enjoyed this in the same vague, horrible way I enjoyed the A-Team" P. Opus
The sooner, the better. You need an agile bug response process.
In your bug tracking system, provide a bug template they need to fill in.
Encountered situation:
Expected situation:
Error messages, if any:
Steps to reproduce (please provide screenshots if appropriate):
If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.
Your users aren't code masters and never will be.
It doesn't involve users to be code masters, it just involves them engaging their brain a bit. I frequently get bug reports along the lines of "something broke last week, came up with some error (I don't remember what), but I rebooted it and its fine for now; please fix it so it doesn't happen again". You don't have to be a "code master" to figure out that reporting a bug and not actually tell me _what_ broke, what the error was or let me log into a system that is currently exhibiting the problem so I can look myself is not going to be condusive to me fixing things.
And this stuff happens again and again with the same customers... "one of our users is having a problem accessing some websites, please can you fix it?" - ok, so I have to go back and ask "which user" and "which websites", if I'm lucky the customer will give me this information, if I'm unlucky I get "I didn't ask". A week later I'll get an almost identical problem report from the same person about a different (but extremely similar) problem, and again none of the information I ask for _every_ time is included.
Also the great one that comes up occasionally is "this has been broken for a month and you haven't fixed it yet!".. well, if you'd actually told me that there was a problem I might've known to look into it, but since this is the first I've heard of it...
http://blog.nexusuk.org
On every page of the site that I did for this company, in the upper right corner, there is a button labelled "BUG!". Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in. The dialog box has a "Submit" button.
What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically.
And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page.
The BUG! button was invented - by me - as a reaction to Bugzilla, which seems to be designed to keep users from reporting problems. I can ignore pointless bug reports, but I want to hear everything.
... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.
If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
You can reproduce the issue by getting their keystroke history. File an FOIA request with the NSA.
- Record all their actions on a self overwriting one hour long file.
- Give them a "one button" way of reporting a bug. The button saves the user and the time, then waits for five minutes and then sends you the recorded actions file.
It's simple to develop and gives you a lot of information. It might be illegal in some countries.
Your users aren't code masters and never will be.
It doesn't involve users to be code masters, it just involves them engaging their brain a bit. I frequently get bug reports along the lines of "something broke last week, came up with some error (I don't remember what), but I rebooted it and its fine for now; please fix it so it doesn't happen again". You don't have to be a "code master" to figure out that reporting a bug and not actually tell me _what_ broke, what the error was or let me log into a system that is currently exhibiting the problem so I can look myself is not going to be condusive to me fixing things.
And this stuff happens again and again with the same customers... "one of our users is having a problem accessing some websites, please can you fix it?" - ok, so I have to go back and ask "which user" and "which websites", if I'm lucky the customer will give me this information, if I'm unlucky I get "I didn't ask". A week later I'll get an almost identical problem report from the same person about a different (but extremely similar) problem, and again none of the information I ask for _every_ time is included.
Also the great one that comes up occasionally is "this has been broken for a month and you haven't fixed it yet!".. well, if you'd actually told me that there was a problem I might've known to look into it, but since this is the first I've heard of it...
Right. Because users are filing bug reports. They're calling for customer service/support. You get good bug reports by having good people answer the phone. I'll accept, "something broke," from a user. However, I find it inexcusable that a Helpdesk rep sends me a bug report that says the same thing or, worse, asks me to call the user for details. That being said, if you can't rely on Helpdesk or the user, then your software should be able to rollup everything you need to resolve the issue.
Requiring a user, after the fact, to recall an error message is futile. They simple have seen to many varied ones and their brain goes 'oh an error message' not 'oh a 504 error' or 'oh a invalid data type error'.
Believe it or not a user that doesn't remember the error message is not the worst kind of user.
I have some users that love translating text errors into numeric error themselves. Any time a page doesn't load, it's a 404. So that's what they report. "I'm trying to connect to thisdomaindoesntexist.com and I'm getting a 404."
That is nothing. I love it when users are overly specific, because they are always specifically wrong.
In user speak "My Password won't work." is code for everything, including an unplugged computer.
Troll is not a replacement for I disagree.
I had just sent the below list of things required in a bug report to a group of users/testers this morning......
In general, the following items should be included when possible:
â Reproduction steps
â Expected behavior
â Observed behavior
â If available, any related steps or values that worked (for example when a bug âoesometimesâ happens, itâ(TM)s useful to know when it works and when it doesnâ(TM)t)
Then, the trick is to reject or de-prioritize any bug that doesn't conform to the above. Train the users that if they provide accurate information, they get quicker response. It works for Pavlov.
Maybe, the user didn't want to confuse you by sending in multiple problem reports at once. Or maybe the user's manager thought that would overload IT, or make it appear they were just complaining too much. :)
I had users who just ignored, or worked around errors, errors that had never been reported. Some were user errors (resolved by both program changes to prevent those errors and user training), some were UI errors that didn't impact the results, and some were program errors with actual consequences or impact on the data or utility of the software. I found out about these errors by watching the users, or as a side issue when they were having some other problem that they did report. I explained that reporting all errors ASAP gives the developers a broader view of the potential problem, and allows all errors to be fixed faster and more completely. I trained them to report every problem, no matter how small, and made a point of addressing those errors as quickly as practical to reinforce the behavior of reporting them. And, of course, I explained that unreported errors were very unlikely to be fixed, ever.
make imaginary.friends COUNT=100 VISIBLE=false