Slashdot Mirror
Ask Slashdot: How To Get Non-Developers To Send Meaningful Bug Reports?
DemonGenius writes "I'm in the midst of a major rollout of one of our primary internal applications at work and we have a beta version available for all the staff to use. The problem here is most of the staff don't know how to send reports meaningful enough to get us devs started on solving their problems without constant back and forth correspondence that wastes both developer time and theirs. Some common examples are: screenshots of the YSOD that don't include the page URL, scaled screenshots that are unreadable, the complaint that wants to be a bug report but is still just a complaint, etc. From the user's perspective, they just send an email, but that email registers in our tracking system. Any thoughts on how to get the non-devs sending us descriptive and/or meaningful reports? Does anyone here have an efficient and user-friendly bug tracking system/policy/standard at their workplace? How does it work?"
Make your software send it...
You can not teach the world, so why try?
Either that, or send them all to reeducation camps.
Try not to stress about it, it's hopeless.
Sent from my PDP-11
A more pressing question is how to get developers to stop ignoring bug reports.
For Windows developers at least, there is one spectacular piece of the OS to make use of for bug reports. Mini-dumps. So awesome I wish Linux had an equivalent. Makes finding bugs much easier when you can load a mini-dump and break into it with a debugger. Best of all you don't have to host your own server to cache those dumps, Microsoft will do it for you as long as your copy of Visual Studio is legit.
Seriously, don't just keep blowing off the important user bugs for multiple release cycles. Once your bugs have been blown off for 6 months you stop submitting new ones.
So it is an ASP.NET app? Seriously, override application_error in the global.asax, and just send the emails with detailed error dumps and messages straight to you, in real time.
and find out what you think might be wrong, instead of what is wrong.
Engage the users. It's your problem, not theirs.
General users cannot and will not grasp the finer points of how you need the information. About all you can do is
1. Ask them to resend, but with specific directions
or 2. some sort of screen sharing application - "Ok, show me exactly what you were doing when..."
This is a great point. The distinction that actually matters is not so much whether the bug reports are submitted by developers or lusers, but whether they're submitted by idiots. The fact that someone's been hired as a developer reduces the odds of idiocy w.r.t. software, but it's neither necessary nor sufficient, and a certain degree of optimism is required to assert even the correlation. Likewise, it's plausible that lusers are more likely than developers to be idiots about software, but exceptions exist.
The real, general solution to this problem is, of course, to get rid of the idiots on both sides. The solution to this problem is left as an exercise for the reader.
I admittedly didn't read the full thread so far, and I know at least some of this was covered.
But have either a web site or a bug reporting facility in your app filled with a template that the user can fill out. Something like:
Brief description of problem:
Steps to reproduce the problem:
What you saw:
include crash logs, snapshots, etc
What you expected to see:
Severity: Crash/UI/Performance/etc..
Version of app/configuration info (filled in AUTOMATICALLY if possible).
and of course do NOT make them fill in every single field (if it's not reproducible, they won't have exact steps.. but you might be able to narrow down on the problem with multiple different bug reports).
Personally, I hate these templates in our bug system, and cmd-A, and type over them, since I already know what kind of info to put in.. but it is useful for regular users (or even developers, if there are special steps, like special debugging tools for your specific app/tools).
I've had success by educating a small group of business users to serve as triagers of bug reports - it's a lot easier to have them enforce your rules about what a validated bug looks like, particularly when you've armed them with a template to check against (example - without a screenshot including a URL, time of incident, browers/OS, the report gets sent right back at them) or have them validate the issue on their own before an engineer even looks at it. Eventually the business users get good enough to aggregate incidents together to locate true bugs and also can shoulder some of the prioritization burden. Remember that its in their rational best interest to care about software quality even if they can't speak about it in a sophisticated way.
We have also posted examples of good bug reports and a wall of shame for bad ones - if you can get incentives together (or punish those that resist reeducation) that helps too.
From the question:
It's an internal app so just have the app log everything that the user does in that app.
Then, when the user calls to say there is a problem, the dev team can pull the logs from that machine and recreate the exact sequence of events.
And don't worry about the logs becoming too large. If the dev cannot figure that out then there are larger problems there.
Also, have the app check the versions of the libraries and such in the OS.
Wouldn't this depend a lot on the type of app, and also on the type of bugs you want information about?
If it's a C++ app, then sure, having a built-in crash reporting mechanism shouldn't be that hard to build in. But what if it's a web app? Or some kind of low-level or embedded software?
Judging by the question, it's probably not anything embedded, but web apps are pretty common these days, and crashing isn't even a problem with those, instead it's other problems. Or what if it's some kind of defect in the UI, or some other imperfection that users might complain about? Those wouldn't be handled by an automated bug reporter.
Bingo. One of the main apps that I use has a 'Feedback' button that tech support has hijacked to have users send in bug reports. I'm sure the Marketing department is annoyed, but that just makes it better.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
A system that I've used in the past, that sometimes works, would be to have the users fill out a "checklist" of items/questions when submitting the email. Trouble Url? Error recieved? Complaint or problem? Etc etc. If you know the user base and this is provided up front, it may help with some of that. But that assumes the checklist is viable for them to fill out as well as making the questions answerable and usable from both their and the developers perspective.
-= My spoon is too big =- Errosion
Or they will just stop sending bug reports. If you are an ass to your users then they will weigh how much they care about the app working vs how much of a pain it is to deal with you. If the app is relatively unimportant to them or you are a very big ass then they will stop sending bug reports (and possibly stop using your app). In which case the dev loses.
YSOD? Maybe you need to be more clear to your users. I don't know what YSOD is and I work in the industry... Make sure your users understand what a bug report is, and how it helps to give as much information as possible. Avoid using terms they won't understand, and assume they don't know what you want. Some users will try to help if you tell them what you need, but give up if they feel like they have to figure it out on their own.
What a ridiculous comment. You are clearly not a software developer. I guess that's why you posted AC...
Even the best programmers make mistakes - but they also write unit tests to find them and functional tests to find issues with systems/integration. Even given that, a developer would still need to thorougly test (in various automated or manual ways) their work to make sure there are no bugs in a complex system - and the point of QA is both that they have expertise in that area, and it's not worth higher paid developers' time to go through all of those tests themselves.
Give them a "Report bug" clicky that brings up a form for them to fill out and which automatically extracts relevant information.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
You have to coach them. They don't really understand what you need.
When I get a email from someone about a bug, I go meet with them. I ask them all the questions I think may be relevant. What were they doing, how were they doing it. Were there any extra small steps or actions that jump out. Sometimes I explain why I'm asking certain questions and relate them back to previous bugs or issues.
I think what you need is someone to be the go between. Get a tester to receive those emails, recreate the issue, then file a bug report. Don't allow the end user to file bug reports directly into your system. It will mess with your tracking data. A high number of worthless bug reports closed quickly may look good in the reports but does not help anyone.
Im a gamer, not a grammer major. This post is full of spelling and grammer mistakes.
You haven't told us anything about what you've done to educate the staff about writing up defects. Have you done anything? Unless they also work as technical QAs, they almost certainly have no idea what is expected in a defect report.
I'm surprised that the primary problem doesn't have to do with listing ambiguous steps to reproduce a defect. For example, "I went to the page again" vs "I clicked the link again" vs "I closed the browser, reopened it, entered the URL, and then clicked the link on the page again."
Of the 3 things you mentioned (#1: screenshots of the YSOD that don't include the page URL, #2: scaled screenshots that are unreadable, #3: the complaint that wants to be a bug report but is still just a complaint), the first two sound like *you* are complaining -- if they're not technical people and they don't know how your system works or what you need to see in order to diagnose a problem, then the first two issues should have been expected.
How do you solve #1 and #2? If you're asking users to perform rudimentary QA, you're going to have to work with them. So, the solution to the first two problems is for you to realize you need to help out in these situations, explain what was wrong and what they can do to fix it next time. They don't know what you need in advance. Progress will be incremental but sure. And if they have to do all this in addition to their other tasks, then be nice while you're doing it.
How do you solve #3? First, if people are complaining then make sure you're not overreacting to a real usability issue and ignoring it because it's not in the format you wanted. Then mail a template with a description field, a steps to reproduce field, an expected results field, and an actual results field, with a description of what each one is. Have a meeting explaining how to use it. Give examples, especially highlighting the need for detailed and unambiguous steps to reproduce.
As bad as e-mail is for tracking issues, if this is a short term thing it may be the best approach. If you can see this going on, getting away from e-mail is a very good idea. Having recently been through a issue tracking system transition, getting everyone used to the new systems took some time even though it was handled very well (and not by me).
Sometimes that's because the difference between a defect and a feature request is only identified by the requirements document that was never given to QA in the first place. But yeah, it happens sometimes.
That's a cop-out....and completely unrealistic.
Finding a decent way to look through GB of logs to find something that happened a week ago is going to be MUCH more of a challenge than just arguing with any client to get the information you need. Not to mention, the only way you're going to trace any random malfunction back to any random source is if you have so much logging that half your computation time is spent managing your logging system and the needed disk writes....and let's not talk about the volume of log data THAT would produce.
Logging can, and should, only do so much. You can't, and won't, catch everything with it.
If the software is working the way its designers and architects intended for it to function, there's no bug.
absolutely correct, unless you consider usability to be a feature.
if you do, then every time a user is confused about your application, it's a bug. it may not be easy to fix. but if you don't consider it a bug, track it, prioritize it and potentially fix it, then you don't care about usability.
> If it's a C++ app, then sure, having a built-in crash reporting mechanism shouldn't be that hard to build in
That's precisely what I do. The default exception handling routine sends an email to me with the app, version, username, machine id, error description, call stack, and any useful data that that I saw fit to include while coding. It has saved me mountains of pain over the years, and also fuels my reputation as the all-seeing eye.
Dyolf Knip
I have seen several good ideas in this thread already. Here is another that may be useful.
Try to find a way to associate the outcome the user is hoping for with the behavior that you are trying to engender. The user is hoping for their problem to be resolved quickly. The behavior you want to engender is a complete, yet concise, description of the problem and the steps to reproduce it. Make the connection between the former and the latter in the user's mind, and you'll have a self-motivated convert.
When you get a clear and concise report, be sure to thank the submitter and help them realize that the clarity and brevity made it easier to solve the problem. They'll feel good about it, and you might even be one of the few on the tech side to have ever said, "thank you" to that particular end user. (we can be an ornery lot)
When you get an incomplete or vague report and have to ask the user for more information, be sure to explain that a more clear bug report will make it easier for you to solve their problem quickly in the future. Make sure it comes across as, "I want to solve your problem more quickly in the future", not, "You did it wrong."
When you get a rambling, nitpicky, or pure-complaint report, you'll have to use a bit more judgment to figure out how to guide the person for future reference, but you can usually find a way if you think about it.
Really it's just programming -- most end users don't exactly have a complicated UI. Be nice, help them understand how satisfying your needs will benefit them, they'll do it. A nice bonus is that they'll be better end users for every subsequent support person, and they will feel good about doing it.
Stop-Prism.org: Opt Out of Surveillance
Finding a decent way to look through GB of logs to find something that happened a week ago is going to be MUCH more of a challenge
I'm running a website with 2-3k concurrent users; using web server access logs for read-only users, and a more detailed custom logging system for any interactions that any user makes, only generates ~100GB of logs per month. In the grand scheme of things, this is a tiny amount (grep from the command line is still perfectly adequate). For someone who can dedicate a day or two to writing a log browser then I'd expect the search to be even better, with graphs and automated anomaly reports and such (As someone who actually has done this, I'm not sure whether to find your comment of unrealism worrying or complimenting :P).
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
Here's one I've been using, it worked well for my purposes:
http://www.codeproject.com/KB/debug/crash_report.aspx
Same. You can only get so many "I clicked X and an error popped up" bug reports with no information beyond that quote in the bug body before you look to a real solution.
You get information like "I clicked X"? The reports I get don't usually even tell me that much.
I tend to get "the internet doesn't work" (I write and manage proxy and email systems). This can mean various things:
1. The whole internet connection is down (why are you calling me? I'm not your ISP...)
2. I can't access my webmail
3. I can't send an email from my MUA
4. Someone on the other side of the planet sent me an email literally 2 seconds ago and I haven't received it!!!!
5. I can't access any web pages (any error messages that are appearing would be helpful... There's a big difference between "your account has been blocked" and "the proxy server has crashed")
6. *no one* on site can access any web pages.
7. I can't access a specific web page.
etc.
And inaccurate information also doesn't help - a reoccuring one I get is customers telling me that no one can access any web pages (maybe they think this makes it sound more urgent?) when actually it is one specific site that is down (usually because that third party website is actually down and therefore beyond my control, rather than a fault with the customer's systems).
The ones that bring a smile to my face are the customers who phone me up ranting about how I haven't responded to an email they sent complaining that their internet connection was broken.... (this email usually pops into my inbox shortly after their internet connection is restored and their MTA flushes its queue out onto the internet)
http://blog.nexusuk.org
Have your application send bugreports.
If your application is coded in Delphi there is MadExcept: http://madshi.net/madExceptDescription.htm
Whenever my apps crash, hang/lockup or throw an exception, MadExcept steps in and the user will simply see:
When they chose send bug report, it asks for their name and email adress (minimum user incentive IMHO), asks what they where trying to do, and then asks them if the shown screenshot (of the apps state) may be send along. And i allow the user to edit the screenshot before sending it.
Its free for non-commercial usage so you can test it yourself: it sells itself.
But asking the right questions like http://askmebetter.com/ helps too.
Hivemind harvest in progress..
Bug reports are often "when I do this, it doesn't do what it is supposed to do (or what I would like it to do)" rather than crashes or error messages.
> If it's a C++ app, then sure, having a built-in crash reporting mechanism shouldn't be that hard to build in
That's precisely what I do. The default exception handling routine sends an email to me with the app, version, username, machine id, error description, call stack, and any useful data that that I saw fit to include while coding. It has saved me mountains of pain over the years, and also fuels my reputation as the all-seeing eye.
I do much the same but include credit card information, mother's maiden name and social security number.
I've had enough of software folks looking for requirements and now good "bug reports". Users/Customers do not have requirements they have PROBLEMS. It is up to the developers to create tools that solve those problems. The user doesn't even know what's possible or easy or hard to implement, so they can't tell you what the requirements are. Same for bugs. The user can't tell you details about which internal parts of your code have a bug, they tell you (at best) what they were doing when something happened. They don't know a crash from accidentally closing a dialog box. The solution - especially for internally developed software - is to go visit the users and have a conversation. Talk to them. Understand how they view your software and you can then understand their explanation of what it did wrong. This will also give you insight into how to make it better. Stop expecting detailed bug reports and start understanding users.
We do this and it works great. We additionally send along some system data, such as last 5 actions, a URL (we're web based), and some other debug data.
It has saved us endless hours in finding the exact conditions needed to replicate a problem so we can fix it.