Slashdot Mirror
How Do You Get Better Bug Reports From Users?
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.
We have a big internal web app, when it throws an error it will log that error as part of a graceful error handling process. (Page, scope vars, user info, time, etc) From there, the reporting system requires a unique bug tracking number (the error number logged above) and it checks for uniqueness and to see if it exists before it lets you open it. The only edge case it doesn't catch is when our tracking mechanism is also broken and these tend to be severe outages that are pretty easy to diagnose and reproduce. That is the only bug that can be reported with an email, all others are responded to with a "Please log in the application and we will route it appropriately"
If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."
If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.
If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.
If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.
If your users are dumber than a pack of hammers, then I have no answers for you.
I am officially gone from
In your bug tracking system, provide a bug template they need to fill in.
Encountered situation:
Expected situation:
Error messages, if any:
Steps to reproduce (please provide screenshots if appropriate):
If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.
... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.
If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.
Or make it copy-and-pastable so no human memory is required. Or log it and make sending the log easy enough for your users. A nice user message followed by a more detailed procedure unwind or whatever applies in your application can be mighty useful.
I am not a crackpot.
Your users aren't code masters and never will be.
It doesn't involve users to be code masters, it just involves them engaging their brain a bit. I frequently get bug reports along the lines of "something broke last week, came up with some error (I don't remember what), but I rebooted it and its fine for now; please fix it so it doesn't happen again". You don't have to be a "code master" to figure out that reporting a bug and not actually tell me _what_ broke, what the error was or let me log into a system that is currently exhibiting the problem so I can look myself is not going to be condusive to me fixing things.
And this stuff happens again and again with the same customers... "one of our users is having a problem accessing some websites, please can you fix it?" - ok, so I have to go back and ask "which user" and "which websites", if I'm lucky the customer will give me this information, if I'm unlucky I get "I didn't ask". A week later I'll get an almost identical problem report from the same person about a different (but extremely similar) problem, and again none of the information I ask for _every_ time is included.
Also the great one that comes up occasionally is "this has been broken for a month and you haven't fixed it yet!".. well, if you'd actually told me that there was a problem I might've known to look into it, but since this is the first I've heard of it...
Right. Because users are filing bug reports. They're calling for customer service/support. You get good bug reports by having good people answer the phone. I'll accept, "something broke," from a user. However, I find it inexcusable that a Helpdesk rep sends me a bug report that says the same thing or, worse, asks me to call the user for details. That being said, if you can't rely on Helpdesk or the user, then your software should be able to rollup everything you need to resolve the issue.
Requiring a user, after the fact, to recall an error message is futile. They simple have seen to many varied ones and their brain goes 'oh an error message' not 'oh a 504 error' or 'oh a invalid data type error'.
Believe it or not a user that doesn't remember the error message is not the worst kind of user.
I have some users that love translating text errors into numeric error themselves. Any time a page doesn't load, it's a 404. So that's what they report. "I'm trying to connect to thisdomaindoesntexist.com and I'm getting a 404."