Slashdot Mirror
How Do You Get Better Bug Reports From Users?
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
Your users aren't code masters and never will be.
I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.
No. Well, not yet, but I don't expect that to change anytime soon.
We have a big internal web app, when it throws an error it will log that error as part of a graceful error handling process. (Page, scope vars, user info, time, etc) From there, the reporting system requires a unique bug tracking number (the error number logged above) and it checks for uniqueness and to see if it exists before it lets you open it. The only edge case it doesn't catch is when our tracking mechanism is also broken and these tend to be severe outages that are pretty easy to diagnose and reproduce. That is the only bug that can be reported with an email, all others are responded to with a "Please log in the application and we will route it appropriately"
Period ( . )
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Generally if we can ingrain that a screenshot and a username will be much more useful than any first-submission of a bug, we get things done faster. Most of the time we get the screenshot and find out PEBCAK...
I've worked in support organizations for 15 years. In a commercial environment where you can afford the staff, having a tiered approach works best - you have a help desk to gather and refine the questions and answer the small stuff, then work your way up to the engineers that wrote the code. The tough part of that is having a skilled enough help desk to know when to skip the canned questions and just forward a request on once you have the right information.
For organizations without those resources, you need to rely on the user base to be the help desk. Give them as much concise information as possible and frame the bug submission so that any and all needed data is in the report. Then it's up to the developer to give good information back to the user.
As an example, I had a problem with my laptop's trackpad going wonky. Ubuntu made it really easy to compile information about my system and submit it as a bug report, then open it for me so I can add any additional text I wanted. The answer I got back asked me to try a different kernel, and included well-documented links and information on how to get and install it. Just saying something like "yeah, go grab something out of backports" doesn't help the user if they have no idea what you're talking about.
If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."
If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.
If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.
If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.
If your users are dumber than a pack of hammers, then I have no answers for you.
I am officially gone from
When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.
"Actually, I enjoyed this in the same vague, horrible way I enjoyed the A-Team" P. Opus
The sooner, the better. You need an agile bug response process.
The easiest way to get goo bug reports is to have a good way to submit them. Have a form, have the form fields clearly labeled. Include documentation indicating what sort of information you need. Make it clear, short and simple. Have one person on the team whose job it is to follow up on bug reports, get more information and update the users on the progress of the bug. If users think a bug report is a black hole and nothing gets done they won't send good ones.
In your bug tracking system, provide a bug template they need to fill in.
Encountered situation:
Expected situation:
Error messages, if any:
Steps to reproduce (please provide screenshots if appropriate):
If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.
On every page of the site that I did for this company, in the upper right corner, there is a button labelled "BUG!". Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in. The dialog box has a "Submit" button.
What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically.
And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page.
The BUG! button was invented - by me - as a reaction to Bugzilla, which seems to be designed to keep users from reporting problems. I can ignore pointless bug reports, but I want to hear everything.
There are many bugs that would easily be detected by actually using the application on a daily basis.
Users do not work for you. When they do post bug reports, it is most likely in frustration.
Over years I've devised a system that works very well. I've used it mainly for web apps, but I'm sure it can be adopted. The key is:
A) Limited number of predefined fields
B) Extensive logging on backend.
ad A) I like to use following fields information: ...... ..... ..... to happen.
- User you're logged in as (login/email)
- Page that you've encountered error on (link).
- Screenshot (of whole screen! - add an flash app to take screenshots if possible).
- Operating system / Browser (sometimes needed, sometimes loggin on server is sufficient) best to have predefined list with radio buttons.
- Bug description in following format
After I did
happened
Instead I'd like for
ad B)
- Logging every visit, every page view with:
- Operating system, browser, browser dimensions.
- New relic.
- Airbrake.io (or alternative / equivalent for your language).
With those two things you should be able to track most of feature requests (otherwise known as bugs).
... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.
If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
No matter what process you employ it will fail, what you need is someone who is willing to follow-up with the users and step in their shoes, I have realize users are always willing to report a bug as long as it makes them feel important and FOLLOWUP.
I believe it was Google who at some point had an option to highlight the portion of the page that showed the problem, as well as submit a comment about it. Being able to see exactly what this user is talking about would help quite a bit when I go to fix a bug.
--"Forget the nectar of the Gods, just give me some Mountain Dew."
First stage is a php form I wrote which asks them very directed questions. They have to answer questions like "What was running on the computer when the bug occurred. Were you running the software in VM, Did you follow the setup directions properly, Did you download the newest version from the test server etc....
Second stage is they let the form take logs from the computer, I automated the process so the form takes what it needs. They literally hit start and go have coffee or something well it works away. Once that is done I get an email with the form data and the logs.
Third step is reproduction, I give the data to another user and asked them to reproduce the issue, if they can't then I visit the submitter if possible or ask for them to take a video of it occurring. If they can then I also grab the data and at that point I have what I need to fix the bug.
Once bugs are fixed both testers have to try and reproduce the issue and so far it's been very very rare that they can. The user can only answer the questions you ask them and I hate when QA tester ask bad question and complain about useless answers. Ask the right questions and you'll get the right data. You also have make sure developers are building great logging systems. A logging system should be good enough that with out the tester inputting to the issue you can see all the data you need! If you can't then the logging system isn't designed well enough and needs to be redone.
Encourage and make it easy for users to send feedback.
Listen to complains, no matter how rude.
Reward the users by acknowledging their reports and by actually fixing the bugs or by adding requested features when it makes sense.
Hello everyone,
I have tried a bunch of ways. Trained the 'expert' users in the area on how to put in a better ticket. Sent tickets back to them because of lack of information. Judicious of cattle prods and a tack hammer... However, users will use what method is easiest to them, which tends to be:
In God we trust, all others require data.
- Record all their actions on a self overwriting one hour long file.
- Give them a "one button" way of reporting a bug. The button saves the user and the time, then waits for five minutes and then sends you the recorded actions file.
It's simple to develop and gives you a lot of information. It might be illegal in some countries.
I got one trick for users that just cannot explain properly what their issue is. Nowadays everyone got phones that can record a video, make them record their screen (this skips the step of having them install a screen capture tool).
If your organization is small enough, send screenshots together with the report, or if you have some kind of desktop sharing ability (remote assistance, Lync), take your dev through the process of how to trigger the bug. Just had an issue yesterday where they couldn't reproduce it, so we fired up a screen sharing process, they had me load up the Firefox debug console, and I went through the steps... turned out to be my mouse having a higher dpi than what they were using, so instead of getting floats, my system was handing them doubles.
I suspect that one of these choices is incorrect. Correct.
posting to revert mod mistake
You *owe* them a bug free product. If you did your job properly, this complaint would be moot.
OK. How you gonna get there, Chief? Certainly not by testing, since that would involve testing which means "using". Maybe not by end users but by someone "using" the software, whether you call them QA or whatever. So, what, you just write it bug-free in the first place? I don't know what you make, but I think it's safe to say that you should be paid more.
I am not a crackpot.
Yup, maybe we should start paying for software with a small percentage of counterfeit money. You know. And ask them to file a report when that money doesn't "work". And maybe get around to fixing the problem. Eventually.
Seven puppies were harmed during the making of this post.
I've filed many bugs against certain open source projects only for the developers to respond with "How am I supposed to fix this? Why aren't you telling me anything?". I'm not sure why they think the first submission will contain everything they need to replicate the bug, but that seems to be the expectation.
That's a fair request. When the developer is secretly thinking of a bug report format and required additional information and users are required to guess what that is, they'll probably give up after one guess. After all, they didn't develop the product, so without telling them what's required to debug and how to report it, they'll probably never guess the secret. Refining the report with some back-and-forth would go a long way in cases where you're dealing with a user capable and willing to provide this extra info.
Of course, there will be users who just can't help that much due to their own limitations or the time they have available to debug your product, so their reports will probably never be all that helpful except in a statistical sense. That's OK. Not everyone is a programmer or QA specialist or whatever. You're probably not qualified to listen to a piano recital and give the performer constructive criticism either, even if you knew it didn't sound as pleasing as you would've liked.
I am not a crackpot.
Tibor will know what to do.
systemd is Roko's Basilisk.
Security bugs in chrome, firefox etc get actively hunted down because of the explicit cash rewards, it clearly works.
I recently "submitted a bug" in an IOS app for a restaurant (web contact form, no email, in-app submission etc), they said I'd get a "free stamp" (collect 5, get a free meal etc) if I sent them the crash logs. Not a big deal, (I'm a developer so not difficult for me to do) but even a small incentive encourages me to actually DO it.
If you/parent company can give away something like like loyalty-card points which relatively cost you nothing, give it away like it's halloween.
I put a button in my software. Click it and a form pops up with a "describe what you were doing when a crash or bug happened, or make a suggestion for a new feature" type message. It also, behind the scenes, captured the state of the program when the button was pressed. That worked well. I got a bunch of new feature requests, as some good hints as to what the user was trying to do when things went south. YMMV
And it's not working!
Actually yes... I get very good bug reports from users. Luckily I did tech support for about 15 years before I got into development. So I had a lot of experience dealing with people that basically had no idea what they were doing. Keep in mind, I work in a corporate environment where what I create is consumed by a limited set of about 5k people.
1. Relationships: Where possible I get to know, personally, the people that will use my software. I can't know them all because there's thousands. But I can find key people, Managers, Seniors, etc... take them out to lunch, make them feel comfortable talking to me, and explaining the need for tickets rather than random emails etc...
2. When a strange problem arrives or I suspect such a problem may exist in some new software, I give them tools. Specifically a button that says "REPORT BUG!" that takes screenshots, dumps the contents of variables and datasets. If available, previous steps taken. Also the log files... and sends it all directly to me. Usually adding such a thing is pretty simple depending on what you're working in. C# and VB have all kinds of pre-built stuff that make this sort of thing super easy.
3. I write documentation on how to properly submit a bug. I make it very clear to people that are not computer savvy what to do. Down to "How to save the file" and "How to take a screenshot" I make it so even my mom could follow the instructions. Generic instructions like this may take you a day or so to write, but then you can just shoot them out to anyone having trouble.
4. Restate the issue. This is the classic "My computer doesn't work! *Is it plugged in?* "I can't see back there, the power is out" You have to know exactly what the customers real issue is. Listen to the client, write down what you hear as the problem, then restate that problem back to them using entirely different language. "My computer doesn't work! *So if I understand you correctly, you're calling me because you've hooked up your computer correctly, you've tried to turn it on, but it just hangs there?* "No! I want to pay my electric bill!"
5. Once you've fixed the problem, confirm with the user that it is in fact fixed. There's nothing worse than fixing something, closing the ticket then having the user come back 3 weeks later asking when you're going to get around to finishing. *You contacted me because your report was giving you incorrect data. We've found that in the original requirements there was a typo and this app should have included an average next to the new sale button. We've added that and now it appears the way you expect it to?* "Yes! It looks great thanks! Want to go get some lunch?" *Sure thing Tammy*
Good luck.
I am a developer. I write good bug reports. But when using a product I'm not working on, if the devs or the process seem out of touch, I don't tend to. If there's a crash report, I won't invest much because the seems to be zero investment back.
Whenever a bug is submitted, a real response... not just an automated mailer... should be sent within a day. Get more details if needed, provide an ETA. Otherwise, we're spitting in the wind, and it doesn't seem worth the effort.
In one company where I worked, I was able to have the end users funnel all their bug reports through the company's internal education department. If the "bug" were actually a prevalent user error, the education department would note that and modify their internal courses to instruct the users on proper usage. If the"bug" were actually a programming problem, the bug reports we got from the education department were very helpful to resolving the problem.
I just held a meeting yesterday with my entire team to discuss this very topic. Generally, I explained that the onus of successful communication lies with the giver (not the receiver). That is, if I want you to understand me, I must communicate in such a way that is understandable by you. (If I speak gibberish, I can't get upset if/when you don't understand me) I explained that submitting a bug report is simply a form of communication, and the the bug report submitter is the "giver" of the communication. Thus, if they want to be understood, they must ensure that their communication is understandable by the receiver. And, since they don't know who the receiver might be, they must make is understandable to the "lowest common denominator" receiver. I said, "If you write a bug report that your grandmother can understand, there is less possibility that it will be misunderstood". I explained that, "if you write a bug report...with the intent that is will be read and used by SOMEONE ELSE...there is a good chance you will write a good bug report." That said, I also implemented a very simple, auto-populate "bug report template" that helps guide and remind users what to enter (ex: "Description of bug, What I did to cause the bug, What I thought was supposed to happen, What actually happened, etc.).
For small stuff, yes. But we're now in a world where software and hardware is complex. Even in an environment like Apple where they have tight control over the hardware, there's variations between operating systems and their hardware offering that make it difficult for a company to write a single app that does it all. Then look at the PC world where it's pretty much a free for all.
By this I mean that you should instrument the code with real, meaningful activity logging, not just some afterthought that grabs a stack trace and some state variables (although you'll want to have that data, too). If you instrument your code with logging that produces readily human-interpretable information about what's going on, the payback is huge, because it makes internal developers' lives easier, and it allows even first-level support folks to to a better job of triage and analysis. It's really important to make it meaningful to the human reader, not just "readable"--an XML representation full of hexadecimal doesn't cut it, it needs to include symbolic names.
Let the users see the logged data easily, if they ask for it, and maybe give them a single knob to turn that controls the level of logging. This will help technically sophisticated users give more useful reports, and it's really helpful in any sort of interactive problem resolution (OK, do X. Now read the last few log messages. Do any of them say BONK?).
It's really useful to include high-resolution time--both clock time and accumulated CPU time--in log messages. This is great for picking up weird performance problems, or tracking down timeouts that cause mysterious hangs. Depending on your architecture and implementation technology, other sorts of "ambient" data (memory usage, network statistics) can useful here, too.
There's a trade-off between logging by frameworks, mixins, macros, etc., and logging of specific events. The former approach gets comprehensive data, but it often can't provide enough contextual semantic information to be meaningful. The latter approach scatters logging ad-hoc throughout the code, so it's very hard to make any argument for comprehensiveness, but if done properly, it's spot-on for meaningful messages. Usually best to do some of each, and have good control knobs to select.
Logging can generate a lot of data, so it's important to be able to minimize that burden during routine operation (especially in deployed applications, where there should be a strict limit on the amount of space/time it takes up). But it's also useful (especially when it's configured to generate a lot of data) to have tools that allow efficient ad-hoc review and analysis--an XML tree view, maybe filtered with XSLT, can be easier than a giant text file.
In any complex system, logging is one of the very first things I recommend implementing. After the architecture is settled enough to know what will be some of the meaningful activities and objects to record, bolting in a high-efficiency, non-intrusive logging infrastructure is the very next step. Then comes business logic, user interface, and all the other stuff. Pays for itself many times over.
You must be incredibly lucky to have a test department with millions of machines with every conceivable variation in environment possible.
No colour or religion ever stopped the bullet from a gun
I think the only chance is to create your own breed of users... May take some time ;-)
Which is why, whenever I file a bug report, I make a point of asking what information I should provide that might help diagnose a problem. In case the developer is as clueless about responding to users as the typical user is about responding to developers.
-- hendrik
agree with the forms. You can't expect good answers if they're not replying to good questions.
the BARE minimum:
1. what were you trying to do?
2. what happened?
3. what were you actually expecting?
That's not necessarily a good list to give to a user, but this is what we tell our front line people to keep in mind when taking a call. There will be two basic limiting factors with online ticket forms. (1) people may not understand the questions and be intimidated enough to not be willing to use the system and (2) duplicate tickets. Make sure whatever you use can handle both issues.
If people aren't using your forms and you've done due diligence with making them aware of them and the need, and they're not getting used, don't blame the users because it's probably something you're doing wrong.
Probably the most frustrating experience for a user is to enter a ticket, and have it "closed" without their having believed it is fixed. Maybe it's fixed and they just didn't even look. Maybe it was changed to work properly and they don't know the new way to use it. Maybe they fixed the problem the customer described, but not what they MEANT. Maybe the fix didn't actually fix the problem at all, or didn't completely fix it and the customer's symptom remains. Maybe they've just been avoiding where the problem is for the last two weeks because they didn't realize it had been fixed on day 2.
Feedback and communication make tickets work.
- make sure you understand the problem. users often try to diagnose the problem and tell you what's wrong, instead of WHY they think it's wrong. here we say "don't check in diagnosis, check in SYMPTOMS." Users frequently mis-diagnose problems. "I can't send email" could stem from "I don't have internet". "My hard drive is dying" could stem from "my computer keeps suddenly shutting down". The entire process is completely dysfunctional if you don't understand their problem. Make sure your form gets symptoms, they're more important than problem descriptions.
- followup is costly, but you NEED to do it. So many ticket systems skip this it's sad. NOTHING frustrates a user more than to have their ticket closed without their believing their problem is fixed. "xxx is fixed. if we don't hear back from you in 24 hrs we will close the ticket" isn't the best solution, but should be considered a bare minimum. If the customer replies back that they don't feel it's fixed, don't just dump it back into the tier 1 queue. Escalate it, increase user interaction, get more information, get feedback. DO NOT rely exclusively on the information in their "it's still broken" reply message. It's time to make a phonecall for more effective live two-way communication.
- prioritize tickets. not just in order of submittal but in order of urgency. tickets that have been back-burnered for awhile should get a bump to their priority. More than one level's worth if needed. The person doing the bump should probably NOT be the one processing the tickets. It's been my experience that an objective fresh viewpoint is needed to make sure that unpleasant tickets don't languish on hold.
One place I worked at recently, the users had almost totally lost confidence in the ticket system. It required me to have comprehensive personal followup on tickets for a solid year before people started trusting the system again. Techs in charge of the queue were summarily deleting vague or "cannot reproduce" tickets with not so much as a response. Tickets were also silently closed whenever a tech though they had fixed the issue. (several tickets had been freshly ressubmitted over a dozen times because the tech didn't understand the user's problem and thought he had fixed it and the user was just too dumb to realize it) Tickets that had several dups in the system from the same user or few users were deleted en mass just to clear the clutter. ("if it's still a problem I'm sure they'll submit another ticket shortly") It was terrible. They had conditioned users
I work for the Department of Redundancy Department.
Force your users to create accounts at SuperDuperBugTrackingThingy.com to report anything. Then send them daily email updates from said tracking system. Also get your moderators to admonish any user that dares accidentally post an existing bug or posts in the wrong section of their clearly laid out bug tracking hierarchy. Or you could go full retard, like Google, and force people to use Google+ to do anything.
Just because you can't reasonably deliver doesn't mean that you don't owe your customers a bug free product. They didn't pay for a buggy product.
What you have to do is to start thinking of your customers like the one who pays your bills, not the one who causes you all of your problems.
It may or may not be a fair request, but it's only useful if the developer tells the user what information he needs and how to go about getting it.
-- hendrik
I've been on both sides of this issue.
As a user, I am often frustrated by the terse, non-informative error messages I get when something goes worng. "Cannot open file" is typical. Which file? In which directory? Why couldn't it be opened? What is the file for? What is supposed to be in it? What part of the program was trying to open the file?
The available on-line help is mostly not helpful in these cases, and FAQ's written by the developers, not the users are useless. Even finding out where the log is kept, much less actually seeing useful information in it? Good luck with that.
How can I as a user file an informative bug report with such scarce information?
So first suggestion: build into your programs meaningful error messages, along with enough information to accurately diagnose the problem. Include options to turn on debug output. When an error is detected, log complete information about it into the log file (the location and format of which should be documented). That way the user has a chance of making a good bug report. And be sure to ask for that information when the bug is filed, with specific directions about how to find and include it. "Append log file" isn't enough, you have to tell them where and how to find the log file.
As a developer, I know the frustration of vague bug reports, that leave out vital information. Simple things like which version of the program was being used, or the configuration settings. You have to make it clear to the reporter that you cannot solve the problem if you don't have enough information. One company I worked for actually had a bug state named "Not Enough Information." The only cure for this is to engage with the end user who reported the bug. And the biggest incentive for that person to engage with you, is the prospect of getting his bug fixed, soon.
So second suggestion: be very reactive to people who report bugs. Get back to them, tell them what's going on, ask questions, and let them test the fix. There is nothing that kills the enthusiasm for bug reporting like going through the process, waiting weeks to get a response, and then be told "Yeah, OK, it'll be fixed in the next version, due out in about 9 months. Thanksbye." And don't underestimate the effect of some user saying to his peers "I reported this bug, and they fixed it for me" to encourage others to report bugs, and make good bug reports.
Of course, there will always be lame bug reports, no matter how you do it. But you can at least encourage good ones, raising the signal to noise ratio.
If you can interact directly with your users, use Teamviewer or any screen sharing program. Watch them reproduce the error and you'll learn a lot.
This is closely related to the first point of this other comment so even if you can't interact with all your users (too many of them or too many layers inbetween) there will be someone who can. Again, a recorded screen sharing session can make its way to you.
PS: I know, making the user install the screen sharing program and properly start it can be a pain but many people know how to use Skype, Google Hangouts and other similar softwares nowadays. If their IT doesn't allow them, talk to their IT.
automated test can work or they can pass the test but still fail in ways that automated tests can see.
Dumping QA on to coders work load does not really work and you don't want the same people who code to be the testers any ways.
QA people need to be able to think out of the box and have full control over there PC systems for testing as well as well having more then 1 log in as well.
Also in power your help desk to be able to help and not be held back with call times and other BS.
We have a huge problem with vague or misleading error messages. Instead of messages like "A file could not be accessed", how about you tell me which file could not be accessed?
Competition Good, Monopoly Bad.
One place I worked at recently, the users had almost totally lost confidence in the ticket system. It required me to have comprehensive personal followup on tickets for a solid year before people started trusting the system again. Techs in charge of the queue were summarily deleting vague or "cannot reproduce" tickets with not so much as a response. Tickets were also silently closed whenever a tech though they had fixed the issue. (several tickets had been freshly ressubmitted over a dozen times because the tech didn't understand the user's problem and thought he had fixed it and the user was just too dumb to realize it) Tickets that had several dups in the system from the same user or few users were deleted en mass just to clear the clutter. ("if it's still a problem I'm sure they'll submit another ticket shortly") It was terrible. They had conditioned users to submit duplicates almost daily until they thought their problem had been fixed.
Overall a great comment, but I'm puzzled by "the user was just too dumb to realize it." Chances are the user realizes exactly what is going on in that situation. Techs are closing tickets without resolving issues due to either laziness or metrics which grade them on closing tickets and not on resolving issues.
With globalization and out-sourcing, walking over to the support team and standing over someone until your issue gets attention isn't an option. Then systems are put in place so submitting a ticket is the only avenue of communication for the user. When you make sure the only tool the user can access is a hammer, you can't complain when they treat everything like a nail.
The rest of your comment addresses issues within the support team and ticketing system that lead to duplicate tickets, so perhaps you meant dumb as in insane as in repeating the same action (creating a ticket) and expecting a different outcome.
To a large extent the issue of bug reports from users can be summed up as "garbage in, garbage out." You want more detail than "it doesn't work"? OK, then how much detail do you give users in your error messages? And I mean plain language messages end users can understand. Displaying the memory address of a misbehaving bit is very detailed, but not something you should expect the end user to remember or communicate.
For getting better context for issues or steps for recreation, how good are your instructions? Can the user include in a bug report, "I was performing procedure A and get the error on step 27"? If not, then don't expect that level of detail from the user. For one, most people don't have an active (and accurate) buffer of the last 5 or 10 things they did. What were you doing 3 steps before the error? Well, I didn't know an error was coming, and so I didn't know the thing I did 3 steps before was going to be important, and so I have a general idea of what I was trying to do, but I don't have the exact steps documented.
Also in many instances it is not practical for the user to try to recreate the issue. If it's an eCommerce system, I'm not going to repeatably submit an order trying to come up with a reproducible sequence to get an error and risk getting charged multiple times. There are many systems subject to regulatory and audit oversight where creating test records or dummy data is not acceptable. So, do give your users access to a non-production system where they can recreate their issue without affecting real data? If not, you're limiting their ability to provide that level of detail.
I'm not saying the issues are all on the technical side. I've been on the support side and seen the "it's not working" reports where the user doesn't even specify which application isn't working (let alone which feature or option). But there is a lot the technical side can do to provide the users with the tools to submit useful bug reports.
http://www.catb.org/esr/faqs/smart-questions.html
http://www.joelonsoftware.com/articles/fog0000000029.html
http://www.joelonsoftware.com/articles/fog0000000014.html
Just give them more interesting bugs to report. Link the error reporting to an easter egg for a fun game. Give them bonus points for originality. Before long, users will be lining up with reports on exactly how to reproduce the bug.
Oh, I'm sorry sir, I thought you were referring to me, Mr. Wensleydale.
I am a user, not a developer, so I have a little different perspective.
First, what does not work for me is a faceless drop box. Without feedback and a sense that the other end cares, it is hard to motivate to put in a bug request, let alone a good one. Submissions to nameless drop boxes become a rant to vent frustration.
For an external vendor, company A, whose design software I have now been using (suffering through) for 14 years I gave up submitting reports after 3-4 years. I never heard anything back, never saw my requests fulfilled within my attention span. In the last few years I moved to a company that uses the same software, but has managed to get a hold of them by the short and curlies. Now when I submit a request our guys get access to their bug tracker and make sure it gets resolved. I have 1 human point of contact at their end I send my reports to (CC'ing our main guy), he works with me to reproduce the problem and then submit it into the "real" system. The feedback portion is still lacking, but I have gotten dozens of long standing bugs and usability issues fixed.
Our internal software has a simple bug tracking system that is open for all of us to submit to. I almost always follow up with one human point of contact after submitting so he has a chance to see and verify it. Often he adds notes in the software groups own jargon. Later I get email traffic keeping me up to date on the resolution. The human contact plus feedback keeps me engaged and keeps me reminded of how to make a useful submission.
Writing a good bug report is not easy, and users should not be expected to know what information the developer will need to find and fix the bug. They only want to report that there is a problem, and that they'd like it fixed.
That said, you can guide them to give more useful information. I found that making a form to fill out with all the details broken out into separate pieces gave us more useful information. Want to know how to reproduce the bug? Make that an individual question on the report. Which component where you using, or what web page were you on? Make that a specific question. A real person can follow up if necessary to get other information, and then they can file the "official" bug report in the form that the developer can understand and use.
"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats
Shoot the users that submit bad bug reports. Within a few generations, only users that submit good bug reports will remain.
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
Pay them.
Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in.
"It's not working."
"What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically. And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page."
Science is all about firing a drunk pig out of a cannon just to see what happens.
Or FaceTime, so you can tell them what to do and show you the result.
...skip the user altogether. If you just have some data gathered and give a simple code, then the user can just provide the code. From QA? Demand a fiddler log/core dump. One time I worked at a company where this was standard... then the product team started to do QA as well (just for micromanagement). They refused to give fiddler logs. The problem was NOT fixable without their parameters... So... Though all my Senior-level associates agreed with me, the product team removed me from the project for not being able to do MAGIC. Need that info!
The first thing that would help, if you want good bug reports from users, is to give some indication that you actually PAY ATTENTION to anything users tell you.
I have never seen any hint that this is the case. The BEST I've ever got for a bug report is a "thank you for your interest in our product" reply written by a robot.
My first suggestion: Every developer should be forced to spend at least one day doing front-line support, so that they will begin to have some idea of what is actually broken.
http://www.geoffreylandis.com
Would you rather have a screenshot that's compressed to hell with JPEG? It's hard to get a 1920x1080 pixel screenshot through a mail system that requires an entire message to fit in 60 KiB including all attachments. Yes, I've had to work with such mail systems; at one time, Juno was this way.
I think it goes one step further: If there's an error, don't just print/log a message that says "12:34:56 X failed while reading Y". This might be somewhat useful, but also TELL ME WHY it failed in the first damn place! Over the years, it seems to me that these sorts of "failures at reporting failures" fall into three categories:
* The program just exited unexpectedly (but cleanly, i.e. not a segfault), but without logging so much as a "Sorry, I'm broken."
-- Why did it exit?
-- What function failed?
* The program received a piece of data of a type that was unexpected and can't be handled, but the program just issues a generic bad-data complaint.
-- What was the allowable range for the expected piece of data?
-- What was the actual data that was received?
-- From what source source was the offending data taken?
* Some ancillary process crashed before the main program threw the error, but the main program doesn't bother to mention this fact.
-- What was the error/crash message from that process?
-- Why couldn't your program continue gracefully after the other process crashed?
---- Why exactly did THAT program crash anyway (see above)?
I'm not talking about spitting out detailed crashdumps, backtraces, etc. or sorting out the cause of some random segfault. That's gdb (etc.) territory. I'm talking about printing, on the way down, a complete explanation of the error, the most likely reason it happened, and what the values of those (likely very few) variables were that were directly related to the process/function/line that actually failed (depending on how much granularity your language of choice offers). Even if your users can't make sense of those messages, YOU can (in theory anyway).
If you can't do that much in the error-handling part of your code, you're doing it wrong.
When a user is entering a ticket, the system MUST automatically try to find an existing ticket for the same problem. Use very loosely matching rules that can provide a small number of most likely duplicate tickets. Allow users to say "yeah, that. that's happening to me too". Don't just dump them out, add the user to the list of submitters for that ticket
Launchpad does this. When the user submits a title for a new issue report, the system searches for other issue reports that match that title, and the user can add himself to another bug's "me too" list. Bugzilla likewise has an approval voting mechanism that lets a registered user vote for issues that affect him.
Along that line, techs need to be able to EASILY break up tickets into separate tickets when a user submits several issues at once (it happens frequently) or when a fix is going to require a multi-pronged approach. (possibly from several techs with different expertise)
Bugzilla does this. A "tracking bug" is an issue report that is marked as depending on several other issue reports. If a user submits an issue with multiple facets, the developers can file individual reports for each facet and set the original submission as depending on those other facets. Or if project drivers want a release to include particular features and fixes, the drivers will file an issue report that depends on those features. This lets Bugzilla spit out reports about the tree of issue reports on which a particular release depends.
Yes! I can't believe nobody has posted this yet. It's the bible on how to write a bug report, written in plain English.
No kidding!!! What do you say at this point?
Everyone always wants step by step or screen shots. Why not use a tool that give you all of that. Camtasia Studio or similar software that captures a video of the screen. Train the users to capture a video of reproducing the bug. Then have them post the video in the bug report. Other than needing plenty of storage to handle a lot of videos, this is actually quite a good solution. Any bug that is submitted without a video is returned to the submitter requesting they duplicate it. Along with fixing the bug, you are going to watch what the users click and what they do and you might actually realize that as a developer you never really understood the user before.
Very often, I have absolutely minor bugs and problems that show up briefly, in passing. Not only is it hard to spend the time to replicate and figure out exactly how/why for someone that wants a "detailed report with reproducable steps" (ick!), I'm in the middle of ... you know, _DOING_ something. So I have to go back later?
Sigh.
What have I found to be the best way to demonstrate bugs? A full screen recording of what I'm doing. I'm not being silly here.
I'm in a beta test (NDA) for a new version of a program. When I'm playing with the beta software, I turn on the video recorder, and record EVERYTHING I'm doing. That company has actually thanked me for this approach.
While using OSX, I find plenty of little things, several times a day; by the time I've realized it, it's past and done, and I can't give you details on how/why. Do I report it as best I can? Used to. Gave up as "pointless/why bother".
I've seen people in this thread saying "Log everything, let the user hit "bug!" and all details are sent to you". I've seen people say "Record everything, snapshot everything, etc". And ... that's a good first step.
But is it enough? If I have a cursor that is supposed to change as I move around, and it only sometimes changes, sometimes does not, how do I log that? How do I report it? Reproducibility is pathetically poor.
How do you record and log your interaction with something else? In theory, the same events and responses that your program is / should be doing is what you'd be recording, right? So if the recording/logging works, your program does. Right?
If I have a problem with a USB drive and a loose connector, that leads to system disasters, and the view is "Won't fix the underlying cause because a disconnected drive has to be considered completely dirty for the worst possible potential situation", even if most such cases can be proven to be no where near that worst case, then what?
If I have a problem where a system program reaches "kill -9 does not work" about once a week, then what?
If I report a bug that is closed as "duplicate of bug #y", but I cannot follow bug #y, get no status updates, no feedback, etc, then why would I bother?
Make it easy for users to submit bugs.
Make it easy for users to tell you what happened.
Never put up a panel with text that cannot be copy/pasted.
Never put up a panel that won't fit on small screens if a lot of text is displayed.
If you are getting bad reports from people, start asking, not "what caused this bug?", but "What would help you provide better information next time?".
And make it so easy for people to report bugs that they can do it one-handed.
You will get bad reports. Don't try to prevent those.
That's not how you submit a bug for the /. mod system!
(There, now you're back on-topic.)
This post © Copyrite Duggeek, all rights reversed.
oh, I thought it was a feature =)
I hit "overrated" by accident on a decent post and thought the author deserved better treatment than that... afaik that's the only way to "undo"
Users won't report minor bugs, but bugs that destroy their data are typically reported. So, always make sure any bugs destroy data.
Go to Heaven for the climate, Hell for the company -- Mark Twain
I use a template asking how to reproduce the bug, and ask the user for the exact steps. If it can't be reproduced, its marked "invalid". If it can be reproduced, we check the requirements. If the behavior violate the requirements, the bug is against the code. If the behavior does not violate the requirements but is still in need of change, we update the requirements, and check the previous step. Pretty straight forward, although it takes some work.