Slashdot Mirror
How Do You Get Better Bug Reports From Users?
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
Your users aren't code masters and never will be.
Can't somebody else do that?
I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.
No. Well, not yet, but I don't expect that to change anytime soon.
We have a big internal web app, when it throws an error it will log that error as part of a graceful error handling process. (Page, scope vars, user info, time, etc) From there, the reporting system requires a unique bug tracking number (the error number logged above) and it checks for uniqueness and to see if it exists before it lets you open it. The only edge case it doesn't catch is when our tracking mechanism is also broken and these tend to be severe outages that are pretty easy to diagnose and reproduce. That is the only bug that can be reported with an email, all others are responded to with a "Please log in the application and we will route it appropriately"
Period ( . )
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
Generally if we can ingrain that a screenshot and a username will be much more useful than any first-submission of a bug, we get things done faster. Most of the time we get the screenshot and find out PEBCAK...
Developer and user relationship is one based on love, forget about streamlines and tickets open regarding the ticket tool itself, love is all a user needs.
I've worked in support organizations for 15 years. In a commercial environment where you can afford the staff, having a tiered approach works best - you have a help desk to gather and refine the questions and answer the small stuff, then work your way up to the engineers that wrote the code. The tough part of that is having a skilled enough help desk to know when to skip the canned questions and just forward a request on once you have the right information.
For organizations without those resources, you need to rely on the user base to be the help desk. Give them as much concise information as possible and frame the bug submission so that any and all needed data is in the report. Then it's up to the developer to give good information back to the user.
As an example, I had a problem with my laptop's trackpad going wonky. Ubuntu made it really easy to compile information about my system and submit it as a bug report, then open it for me so I can add any additional text I wanted. The answer I got back asked me to try a different kernel, and included well-documented links and information on how to get and install it. Just saying something like "yeah, go grab something out of backports" doesn't help the user if they have no idea what you're talking about.
If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."
If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.
If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.
If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.
If your users are dumber than a pack of hammers, then I have no answers for you.
I am officially gone from
You hire someone for fucking quality control you cheap bastard. USERS ARE NOT THE ONES WHO SHOULD BE MAKING BUG REPORTS. The software industry has been lazy for the past 25 years. Fix your fucking code before you release it, not after.
When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.
"Actually, I enjoyed this in the same vague, horrible way I enjoyed the A-Team" P. Opus
The sooner, the better. You need an agile bug response process.
The easiest way to get goo bug reports is to have a good way to submit them. Have a form, have the form fields clearly labeled. Include documentation indicating what sort of information you need. Make it clear, short and simple. Have one person on the team whose job it is to follow up on bug reports, get more information and update the users on the progress of the bug. If users think a bug report is a black hole and nothing gets done they won't send good ones.
In your bug tracking system, provide a bug template they need to fill in.
Encountered situation:
Expected situation:
Error messages, if any:
Steps to reproduce (please provide screenshots if appropriate):
If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.
This kind of idea of dedicated forms works fine and reduces the clutter:
http://www.linksceem.eu/ls2/user-resources/user-support/access-issues-login.html
(in fact, if you force users to pass through the process first time before entering the system,
magically things work smooth)
Direct all bug reports to /dev/null
It's the responsibility of the programers and developers to test their software, not pawn this responsibility off on the users. It's all about $.
On every page of the site that I did for this company, in the upper right corner, there is a button labelled "BUG!". Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in. The dialog box has a "Submit" button.
What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically.
And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page.
The BUG! button was invented - by me - as a reaction to Bugzilla, which seems to be designed to keep users from reporting problems. I can ignore pointless bug reports, but I want to hear everything.
I've filed many bugs against certain open source projects only for the developers to respond with "How am I supposed to fix this? Why aren't you telling me anything?". I'm not sure why they think the first submission will contain everything they need to replicate the bug, but that seems to be the expectation.
There are many bugs that would easily be detected by actually using the application on a daily basis.
Users do not work for you. When they do post bug reports, it is most likely in frustration.
Over years I've devised a system that works very well. I've used it mainly for web apps, but I'm sure it can be adopted. The key is:
A) Limited number of predefined fields
B) Extensive logging on backend.
ad A) I like to use following fields information: ...... ..... ..... to happen.
- User you're logged in as (login/email)
- Page that you've encountered error on (link).
- Screenshot (of whole screen! - add an flash app to take screenshots if possible).
- Operating system / Browser (sometimes needed, sometimes loggin on server is sufficient) best to have predefined list with radio buttons.
- Bug description in following format
After I did
happened
Instead I'd like for
ad B)
- Logging every visit, every page view with:
- Operating system, browser, browser dimensions.
- New relic.
- Airbrake.io (or alternative / equivalent for your language).
With those two things you should be able to track most of feature requests (otherwise known as bugs).
... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.
If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
No matter what process you employ it will fail, what you need is someone who is willing to follow-up with the users and step in their shoes, I have realize users are always willing to report a bug as long as it makes them feel important and FOLLOWUP.
I believe it was Google who at some point had an option to highlight the portion of the page that showed the problem, as well as submit a comment about it. Being able to see exactly what this user is talking about would help quite a bit when I go to fix a bug.
--"Forget the nectar of the Gods, just give me some Mountain Dew."
First stage is a php form I wrote which asks them very directed questions. They have to answer questions like "What was running on the computer when the bug occurred. Were you running the software in VM, Did you follow the setup directions properly, Did you download the newest version from the test server etc....
Second stage is they let the form take logs from the computer, I automated the process so the form takes what it needs. They literally hit start and go have coffee or something well it works away. Once that is done I get an email with the form data and the logs.
Third step is reproduction, I give the data to another user and asked them to reproduce the issue, if they can't then I visit the submitter if possible or ask for them to take a video of it occurring. If they can then I also grab the data and at that point I have what I need to fix the bug.
Once bugs are fixed both testers have to try and reproduce the issue and so far it's been very very rare that they can. The user can only answer the questions you ask them and I hate when QA tester ask bad question and complain about useless answers. Ask the right questions and you'll get the right data. You also have make sure developers are building great logging systems. A logging system should be good enough that with out the tester inputting to the issue you can see all the data you need! If you can't then the logging system isn't designed well enough and needs to be redone.
Encourage and make it easy for users to send feedback.
Listen to complains, no matter how rude.
Reward the users by acknowledging their reports and by actually fixing the bugs or by adding requested features when it makes sense.
Hello everyone,
I have tried a bunch of ways. Trained the 'expert' users in the area on how to put in a better ticket. Sent tickets back to them because of lack of information. Judicious of cattle prods and a tack hammer... However, users will use what method is easiest to them, which tends to be:
In God we trust, all others require data.
- Record all their actions on a self overwriting one hour long file.
- Give them a "one button" way of reporting a bug. The button saves the user and the time, then waits for five minutes and then sends you the recorded actions file.
It's simple to develop and gives you a lot of information. It might be illegal in some countries.
I got one trick for users that just cannot explain properly what their issue is. Nowadays everyone got phones that can record a video, make them record their screen (this skips the step of having them install a screen capture tool).
If your organization is small enough, send screenshots together with the report, or if you have some kind of desktop sharing ability (remote assistance, Lync), take your dev through the process of how to trigger the bug. Just had an issue yesterday where they couldn't reproduce it, so we fired up a screen sharing process, they had me load up the Firefox debug console, and I went through the steps... turned out to be my mouse having a higher dpi than what they were using, so instead of getting floats, my system was handing them doubles.
I suspect that one of these choices is incorrect. Correct.
posting to revert mod mistake
For compiler structured error handling & piping those to a log since they are "more scary"/"less intelligible" to non-coders! Then, also providing users a friendlier message to contact me via email, pushing the "E.Message" (from a variable E:Exception type), along with the notice to mail me that message OR the contents of %AppDir%/APKErrLog.txt which contains the actual structured err handler's data dump!
(Accessible since I override the "controlbox" in the upper left-hand side corner of EVERY Window/hWnd, & that simply spawns notepad.exe so they can copy/paste the errtrap log's data into said email to contact me with it).
It's worked out very well in this app so far the past year now (helped me take it from version 5.0++ to present 9.0++ in fact):
---
APK Hosts File Engine 9.0++ 32/64-bit:
http://start64.com/index.php?option=com_content&view=article&id=5851:apk-hosts-file-engine-64bit-version&catid=26:64bit-security-software&Itemid=74
---
* Recently, I tested that ware above with a neighbor's kids - and yes: They beat the heck out of it, operating it way, Way, WAY "out-of-order" (vs. the order I intended for it to operate in, and it's clearly shown how that's done, including "wizardy" 'do this next' type stuff), but they didn't "follow directions" anyhow, & busted it a couple times! Well, in the end after that - that made me realize I had to account for that, too! In the end? The app came out much better than it was before version 9.0++!
APK
P.S.=> It's a working system, & that I received much data + great feedback for improvements from many testers the past year++ with! Yes - you can *try* to find all of your "bugs" or "useability issues" yourself, but good luck to that, especially when YOU designed the app - you have "set ways"/"set patterns" that "get in your way" imo, believe-it-or-not, when users, don't.
... apk
You *owe* them a bug free product. If you did your job properly, this complaint would be moot.
OK. How you gonna get there, Chief? Certainly not by testing, since that would involve testing which means "using". Maybe not by end users but by someone "using" the software, whether you call them QA or whatever. So, what, you just write it bug-free in the first place? I don't know what you make, but I think it's safe to say that you should be paid more.
I am not a crackpot.
Security bugs in chrome, firefox etc get actively hunted down because of the explicit cash rewards, it clearly works.
I recently "submitted a bug" in an IOS app for a restaurant (web contact form, no email, in-app submission etc), they said I'd get a "free stamp" (collect 5, get a free meal etc) if I sent them the crash logs. Not a big deal, (I'm a developer so not difficult for me to do) but even a small incentive encourages me to actually DO it.
If you/parent company can give away something like like loyalty-card points which relatively cost you nothing, give it away like it's halloween.
That's your main issue... get rid of the users, but fix will be 100% easier!
Writing good, useful bug reports is surprisingly hard. Even for people who consider themselves dyed-in-the-wool developers. They may have an idea of where the problem is and cut to the chase... only to find their assumptions and hunches were wrong. Users have a much more limited view and so often don't even stop and think what may be needed. Even as a seasoned troubleshooter it can be quite hard to drop the right information on a sysadmin's plate. And it can get immeasurably worse if the phone screen turns out to be too dense to penetrate. Or if the responsible fixer (dev, techie, admin, what have you) refuses to cooperate in some way. Sometimes that's justified, but not always, certainly not.
Plenty of automated solutionising solutions (you know the things, the "we haven't sent your request yet, please look at these umpteen useless links first) or completely railroaded form systems seem to work, but in reality often don't, and if you don't fall in one of the neatly compartimentalised problem boxes, you're sorely out of luck. And the back-end will never notice.
That cuts down on getting requests, sure, but it doesn't help the users and so doesn't make them any happier. Just count the number of clicks from start to finish navigating, say, google's problem resolution thing, trying to get to some way to submit a problem that is not in the standard answer boxes. The last few times I tried I got stuck in infinite loops.
There are a couple of instances where the bug and request handling works reasonably well. I've even seen functioning helpdesks. Usually because the people staffing them are a few notches above the regular cut, and they think really hard about reaching out and finding problems instead of making the annoying customer go away.
I put a button in my software. Click it and a form pops up with a "describe what you were doing when a crash or bug happened, or make a suggestion for a new feature" type message. It also, behind the scenes, captured the state of the program when the button was pressed. That worked well. I got a bunch of new feature requests, as some good hints as to what the user was trying to do when things went south. YMMV
And it's not working!
Actually yes... I get very good bug reports from users. Luckily I did tech support for about 15 years before I got into development. So I had a lot of experience dealing with people that basically had no idea what they were doing. Keep in mind, I work in a corporate environment where what I create is consumed by a limited set of about 5k people.
1. Relationships: Where possible I get to know, personally, the people that will use my software. I can't know them all because there's thousands. But I can find key people, Managers, Seniors, etc... take them out to lunch, make them feel comfortable talking to me, and explaining the need for tickets rather than random emails etc...
2. When a strange problem arrives or I suspect such a problem may exist in some new software, I give them tools. Specifically a button that says "REPORT BUG!" that takes screenshots, dumps the contents of variables and datasets. If available, previous steps taken. Also the log files... and sends it all directly to me. Usually adding such a thing is pretty simple depending on what you're working in. C# and VB have all kinds of pre-built stuff that make this sort of thing super easy.
3. I write documentation on how to properly submit a bug. I make it very clear to people that are not computer savvy what to do. Down to "How to save the file" and "How to take a screenshot" I make it so even my mom could follow the instructions. Generic instructions like this may take you a day or so to write, but then you can just shoot them out to anyone having trouble.
4. Restate the issue. This is the classic "My computer doesn't work! *Is it plugged in?* "I can't see back there, the power is out" You have to know exactly what the customers real issue is. Listen to the client, write down what you hear as the problem, then restate that problem back to them using entirely different language. "My computer doesn't work! *So if I understand you correctly, you're calling me because you've hooked up your computer correctly, you've tried to turn it on, but it just hangs there?* "No! I want to pay my electric bill!"
5. Once you've fixed the problem, confirm with the user that it is in fact fixed. There's nothing worse than fixing something, closing the ticket then having the user come back 3 weeks later asking when you're going to get around to finishing. *You contacted me because your report was giving you incorrect data. We've found that in the original requirements there was a typo and this app should have included an average next to the new sale button. We've added that and now it appears the way you expect it to?* "Yes! It looks great thanks! Want to go get some lunch?" *Sure thing Tammy*
Good luck.
I am a developer. I write good bug reports. But when using a product I'm not working on, if the devs or the process seem out of touch, I don't tend to. If there's a crash report, I won't invest much because the seems to be zero investment back.
Whenever a bug is submitted, a real response... not just an automated mailer... should be sent within a day. Get more details if needed, provide an ETA. Otherwise, we're spitting in the wind, and it doesn't seem worth the effort.
In one company where I worked, I was able to have the end users funnel all their bug reports through the company's internal education department. If the "bug" were actually a prevalent user error, the education department would note that and modify their internal courses to instruct the users on proper usage. If the"bug" were actually a programming problem, the bug reports we got from the education department were very helpful to resolving the problem.
I just held a meeting yesterday with my entire team to discuss this very topic. Generally, I explained that the onus of successful communication lies with the giver (not the receiver). That is, if I want you to understand me, I must communicate in such a way that is understandable by you. (If I speak gibberish, I can't get upset if/when you don't understand me) I explained that submitting a bug report is simply a form of communication, and the the bug report submitter is the "giver" of the communication. Thus, if they want to be understood, they must ensure that their communication is understandable by the receiver. And, since they don't know who the receiver might be, they must make is understandable to the "lowest common denominator" receiver. I said, "If you write a bug report that your grandmother can understand, there is less possibility that it will be misunderstood". I explained that, "if you write a bug report...with the intent that is will be read and used by SOMEONE ELSE...there is a good chance you will write a good bug report." That said, I also implemented a very simple, auto-populate "bug report template" that helps guide and remind users what to enter (ex: "Description of bug, What I did to cause the bug, What I thought was supposed to happen, What actually happened, etc.).
For small stuff, yes. But we're now in a world where software and hardware is complex. Even in an environment like Apple where they have tight control over the hardware, there's variations between operating systems and their hardware offering that make it difficult for a company to write a single app that does it all. Then look at the PC world where it's pretty much a free for all.
By this I mean that you should instrument the code with real, meaningful activity logging, not just some afterthought that grabs a stack trace and some state variables (although you'll want to have that data, too). If you instrument your code with logging that produces readily human-interpretable information about what's going on, the payback is huge, because it makes internal developers' lives easier, and it allows even first-level support folks to to a better job of triage and analysis. It's really important to make it meaningful to the human reader, not just "readable"--an XML representation full of hexadecimal doesn't cut it, it needs to include symbolic names.
Let the users see the logged data easily, if they ask for it, and maybe give them a single knob to turn that controls the level of logging. This will help technically sophisticated users give more useful reports, and it's really helpful in any sort of interactive problem resolution (OK, do X. Now read the last few log messages. Do any of them say BONK?).
It's really useful to include high-resolution time--both clock time and accumulated CPU time--in log messages. This is great for picking up weird performance problems, or tracking down timeouts that cause mysterious hangs. Depending on your architecture and implementation technology, other sorts of "ambient" data (memory usage, network statistics) can useful here, too.
There's a trade-off between logging by frameworks, mixins, macros, etc., and logging of specific events. The former approach gets comprehensive data, but it often can't provide enough contextual semantic information to be meaningful. The latter approach scatters logging ad-hoc throughout the code, so it's very hard to make any argument for comprehensiveness, but if done properly, it's spot-on for meaningful messages. Usually best to do some of each, and have good control knobs to select.
Logging can generate a lot of data, so it's important to be able to minimize that burden during routine operation (especially in deployed applications, where there should be a strict limit on the amount of space/time it takes up). But it's also useful (especially when it's configured to generate a lot of data) to have tools that allow efficient ad-hoc review and analysis--an XML tree view, maybe filtered with XSLT, can be easier than a giant text file.
In any complex system, logging is one of the very first things I recommend implementing. After the architecture is settled enough to know what will be some of the meaningful activities and objects to record, bolting in a high-efficiency, non-intrusive logging infrastructure is the very next step. Then comes business logic, user interface, and all the other stuff. Pays for itself many times over.
I think the only chance is to create your own breed of users... May take some time ;-)
Give them a simple way to take screenshots of the problem step by step.
Force your users to create accounts at SuperDuperBugTrackingThingy.com to report anything. Then send them daily email updates from said tracking system. Also get your moderators to admonish any user that dares accidentally post an existing bug or posts in the wrong section of their clearly laid out bug tracking hierarchy. Or you could go full retard, like Google, and force people to use Google+ to do anything.
Just because you can't reasonably deliver doesn't mean that you don't owe your customers a bug free product. They didn't pay for a buggy product.
What you have to do is to start thinking of your customers like the one who pays your bills, not the one who causes you all of your problems.
My biggest beef is cryptic error messages. Fix that.
Water-board them.
I've been on both sides of this issue.
As a user, I am often frustrated by the terse, non-informative error messages I get when something goes worng. "Cannot open file" is typical. Which file? In which directory? Why couldn't it be opened? What is the file for? What is supposed to be in it? What part of the program was trying to open the file?
The available on-line help is mostly not helpful in these cases, and FAQ's written by the developers, not the users are useless. Even finding out where the log is kept, much less actually seeing useful information in it? Good luck with that.
How can I as a user file an informative bug report with such scarce information?
So first suggestion: build into your programs meaningful error messages, along with enough information to accurately diagnose the problem. Include options to turn on debug output. When an error is detected, log complete information about it into the log file (the location and format of which should be documented). That way the user has a chance of making a good bug report. And be sure to ask for that information when the bug is filed, with specific directions about how to find and include it. "Append log file" isn't enough, you have to tell them where and how to find the log file.
As a developer, I know the frustration of vague bug reports, that leave out vital information. Simple things like which version of the program was being used, or the configuration settings. You have to make it clear to the reporter that you cannot solve the problem if you don't have enough information. One company I worked for actually had a bug state named "Not Enough Information." The only cure for this is to engage with the end user who reported the bug. And the biggest incentive for that person to engage with you, is the prospect of getting his bug fixed, soon.
So second suggestion: be very reactive to people who report bugs. Get back to them, tell them what's going on, ask questions, and let them test the fix. There is nothing that kills the enthusiasm for bug reporting like going through the process, waiting weeks to get a response, and then be told "Yeah, OK, it'll be fixed in the next version, due out in about 9 months. Thanksbye." And don't underestimate the effect of some user saying to his peers "I reported this bug, and they fixed it for me" to encourage others to report bugs, and make good bug reports.
Of course, there will always be lame bug reports, no matter how you do it. But you can at least encourage good ones, raising the signal to noise ratio.
"OK. How you gonna get there, Chief?"
Well usually it involves building a good QA team of professionals and working close with them so they can pull me in if there's a bug and show me right there and then.
" Certainly not by testing, since that would involve testing which means "using""
Yep, 'using' by expert QA who know the task and know how to report bugs. Which is why we hire them.
" So, what, you just write it bug-free in the first place? I don't know what you make, but I think it's safe to say that you should be paid more."
I make software in a professional industry alongside professional QA people. I'm paid the going rate for delivering successful software that does its job and doesn't crash. The customer pays the money (about $50k/unit at last check), and in return they get something that does what it says on the box. That's why they pay the money and don't take they $50k and buy the $30k buggy competitors.
But hey, yeh , damn users don't know how lucky they are to get buggy software, they should try harder and not be so *incompetent* with *their* bug reports. Yeh *that's* really the problem here. (sarcasm just in case you think this is real life).
Geez, get rid of that attitude, it's just software, and you should be able to deliver software like a pro.
If you can interact directly with your users, use Teamviewer or any screen sharing program. Watch them reproduce the error and you'll learn a lot.
This is closely related to the first point of this other comment so even if you can't interact with all your users (too many of them or too many layers inbetween) there will be someone who can. Again, a recorded screen sharing session can make its way to you.
PS: I know, making the user install the screen sharing program and properly start it can be a pain but many people know how to use Skype, Google Hangouts and other similar softwares nowadays. If their IT doesn't allow them, talk to their IT.
automated test can work or they can pass the test but still fail in ways that automated tests can see.
Dumping QA on to coders work load does not really work and you don't want the same people who code to be the testers any ways.
QA people need to be able to think out of the box and have full control over there PC systems for testing as well as well having more then 1 log in as well.
Also in power your help desk to be able to help and not be held back with call times and other BS.
We have a huge problem with vague or misleading error messages. Instead of messages like "A file could not be accessed", how about you tell me which file could not be accessed?
Competition Good, Monopoly Bad.
Everyone wants them stored in the attic staring like a mad cousin no-one wants to see. Getting the tech writers to interact with the customers and helping them tease their problems from a vague feeling into black and white not only improves the doc but gives you fantastic, well ordered and structured feedback.
However, product managers typically never do this because well formed customer feedback translated into actionable prose is often the last thing product managers want to hear.
There are many good ideas here, but I did not see anything about logging failures. If you can trap and log errors, that can help a great deal. Include things like the Java stack trace. Ideally, that log/error data can go back to your, such as in the Android/iOS worlds via Google Play or Apple app store. This is part of a solution, along with much of the other suggestions in this topic.
http://www.catb.org/esr/faqs/smart-questions.html
http://www.joelonsoftware.com/articles/fog0000000029.html
http://www.joelonsoftware.com/articles/fog0000000014.html
Just give them more interesting bugs to report. Link the error reporting to an easter egg for a fun game. Give them bonus points for originality. Before long, users will be lining up with reports on exactly how to reproduce the bug.
Oh, I'm sorry sir, I thought you were referring to me, Mr. Wensleydale.
I found that politely asking for more information in email gets ignored 90% of the time.
I found that rudely closing with "You haven't provided enough information for us to track down the problem. Provide the following if you actually want us to do something," only gets ignored 80% of the time.
And then play favorites with the folks that actually provide information.
*AND* if you suspect that something that's poorly reported is a bigger issue, mention it to one of the guys that actually provides information.
I am a user, not a developer, so I have a little different perspective.
First, what does not work for me is a faceless drop box. Without feedback and a sense that the other end cares, it is hard to motivate to put in a bug request, let alone a good one. Submissions to nameless drop boxes become a rant to vent frustration.
For an external vendor, company A, whose design software I have now been using (suffering through) for 14 years I gave up submitting reports after 3-4 years. I never heard anything back, never saw my requests fulfilled within my attention span. In the last few years I moved to a company that uses the same software, but has managed to get a hold of them by the short and curlies. Now when I submit a request our guys get access to their bug tracker and make sure it gets resolved. I have 1 human point of contact at their end I send my reports to (CC'ing our main guy), he works with me to reproduce the problem and then submit it into the "real" system. The feedback portion is still lacking, but I have gotten dozens of long standing bugs and usability issues fixed.
Our internal software has a simple bug tracking system that is open for all of us to submit to. I almost always follow up with one human point of contact after submitting so he has a chance to see and verify it. Often he adds notes in the software groups own jargon. Later I get email traffic keeping me up to date on the resolution. The human contact plus feedback keeps me engaged and keeps me reminded of how to make a useful submission.
Writing a good bug report is not easy, and users should not be expected to know what information the developer will need to find and fix the bug. They only want to report that there is a problem, and that they'd like it fixed.
That said, you can guide them to give more useful information. I found that making a form to fill out with all the details broken out into separate pieces gave us more useful information. Want to know how to reproduce the bug? Make that an individual question on the report. Which component where you using, or what web page were you on? Make that a specific question. A real person can follow up if necessary to get other information, and then they can file the "official" bug report in the form that the developer can understand and use.
"Education is not the filling of a pail, but the lighting of a fire." -- William Butler Yeats
Shoot the users that submit bad bug reports. Within a few generations, only users that submit good bug reports will remain.
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
Stop being a whiny biatch about 'unimportant' or 'unhelpful' bugs and realize that you did it wrong to start with?
That guy who answers all of the Internet Explorer WebDriver responses with "I'm just going to close your bug if you don't mail me the entire contents of your computer" comes to mind . . .
Yeah, I'll go with "listen and fix".
Pay them.
Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in.
"It's not working."
"What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically. And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page."
Science is all about firing a drunk pig out of a cannon just to see what happens.
Or FaceTime, so you can tell them what to do and show you the result.
...skip the user altogether. If you just have some data gathered and give a simple code, then the user can just provide the code. From QA? Demand a fiddler log/core dump. One time I worked at a company where this was standard... then the product team started to do QA as well (just for micromanagement). They refused to give fiddler logs. The problem was NOT fixable without their parameters... So... Though all my Senior-level associates agreed with me, the product team removed me from the project for not being able to do MAGIC. Need that info!
If you want good bug reports, don't drive away those users who are good at making usable bug reports. If you're going to let bugs languish for over a decade despite overwhelming user desire to see them fixed, if you're going to let ego-centric dicks override the clear wishes of the user base and for no technical reason, and if you're going to ignore patch contributions or make the process so difficult that few want to do it, then you're going to discourage the good bug writers and your project gets what it deserves. I'm looking at you Mozilla.
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html should be mandatory reading for (would-be) bug reporters.
The coder's perception of the world is this:
"I write code that gets compiled into programs that users run. Users do things that crashes my code and I need to know what it is. Users should tell me what is going wrong so I can fix it."
The User's perception of the world is this:
"I am trying to do my !@#$ing job! Why can't the !@#$ing coders error test this cr@p before I use it? It isn't my job to fix their !@#$ing mistakes, that's THEIR JOB!"
You see the problem here, right?
That bundles up the logs, thread dump, etc. Like OSX does.
Don't fucking insult them. (Can you hear me, Mozilla.)
The first thing that would help, if you want good bug reports from users, is to give some indication that you actually PAY ATTENTION to anything users tell you.
I have never seen any hint that this is the case. The BEST I've ever got for a bug report is a "thank you for your interest in our product" reply written by a robot.
My first suggestion: Every developer should be forced to spend at least one day doing front-line support, so that they will begin to have some idea of what is actually broken.
http://www.geoffreylandis.com
Would you rather have a screenshot that's compressed to hell with JPEG? It's hard to get a 1920x1080 pixel screenshot through a mail system that requires an entire message to fit in 60 KiB including all attachments. Yes, I've had to work with such mail systems; at one time, Juno was this way.
The only useful bug reports I've ever gotten from end-users was when we added the ability to get a stack trace to the GUI and asked customers to email that when we were on the phone with them understanding exactly the scenario which caused a problem.
We were lucky - the code was extremely well written with only 1 bug reported per month from users. These were picky users, BTW - NASA flight controllers.
The stack trace enabled me to determine 90% of the time what code they'd hit and why there was any issue at all. I'd pass it onto the developer and it would be corrected in 24 hrs - tops. Getting the corrected code back into production could take between 1 day and 2 months depending on launch schedules. I was the slowest dev to fix, but I had 3 different roles on the team with conflicting priorities, development was usually not time sensitive, so it was my lowest priority task most of the time.
I've worked with about 10 different software dev teams over the years and this one **by far** was the most excellent and produced extremely high quality code with relatively tiny overhead. We didn't have any QA, no formal testing, and only developers on the team. Every dev was extremely sensitive to any issue with their code - it was a point of honor to deliver bug free code. I've found no better motivation in 25+ yrs of development. Memory leaks were considered MAJOR errors by everyone.
When a user is entering a ticket, the system MUST automatically try to find an existing ticket for the same problem. Use very loosely matching rules that can provide a small number of most likely duplicate tickets. Allow users to say "yeah, that. that's happening to me too". Don't just dump them out, add the user to the list of submitters for that ticket
Launchpad does this. When the user submits a title for a new issue report, the system searches for other issue reports that match that title, and the user can add himself to another bug's "me too" list. Bugzilla likewise has an approval voting mechanism that lets a registered user vote for issues that affect him.
Along that line, techs need to be able to EASILY break up tickets into separate tickets when a user submits several issues at once (it happens frequently) or when a fix is going to require a multi-pronged approach. (possibly from several techs with different expertise)
Bugzilla does this. A "tracking bug" is an issue report that is marked as depending on several other issue reports. If a user submits an issue with multiple facets, the developers can file individual reports for each facet and set the original submission as depending on those other facets. Or if project drivers want a release to include particular features and fixes, the drivers will file an issue report that depends on those features. This lets Bugzilla spit out reports about the tree of issue reports on which a particular release depends.
Everyone always wants step by step or screen shots. Why not use a tool that give you all of that. Camtasia Studio or similar software that captures a video of the screen. Train the users to capture a video of reproducing the bug. Then have them post the video in the bug report. Other than needing plenty of storage to handle a lot of videos, this is actually quite a good solution. Any bug that is submitted without a video is returned to the submitter requesting they duplicate it. Along with fixing the bug, you are going to watch what the users click and what they do and you might actually realize that as a developer you never really understood the user before.
Very often, I have absolutely minor bugs and problems that show up briefly, in passing. Not only is it hard to spend the time to replicate and figure out exactly how/why for someone that wants a "detailed report with reproducable steps" (ick!), I'm in the middle of ... you know, _DOING_ something. So I have to go back later?
Sigh.
What have I found to be the best way to demonstrate bugs? A full screen recording of what I'm doing. I'm not being silly here.
I'm in a beta test (NDA) for a new version of a program. When I'm playing with the beta software, I turn on the video recorder, and record EVERYTHING I'm doing. That company has actually thanked me for this approach.
While using OSX, I find plenty of little things, several times a day; by the time I've realized it, it's past and done, and I can't give you details on how/why. Do I report it as best I can? Used to. Gave up as "pointless/why bother".
I've seen people in this thread saying "Log everything, let the user hit "bug!" and all details are sent to you". I've seen people say "Record everything, snapshot everything, etc". And ... that's a good first step.
But is it enough? If I have a cursor that is supposed to change as I move around, and it only sometimes changes, sometimes does not, how do I log that? How do I report it? Reproducibility is pathetically poor.
How do you record and log your interaction with something else? In theory, the same events and responses that your program is / should be doing is what you'd be recording, right? So if the recording/logging works, your program does. Right?
If I have a problem with a USB drive and a loose connector, that leads to system disasters, and the view is "Won't fix the underlying cause because a disconnected drive has to be considered completely dirty for the worst possible potential situation", even if most such cases can be proven to be no where near that worst case, then what?
If I have a problem where a system program reaches "kill -9 does not work" about once a week, then what?
If I report a bug that is closed as "duplicate of bug #y", but I cannot follow bug #y, get no status updates, no feedback, etc, then why would I bother?
Make it easy for users to submit bugs.
Make it easy for users to tell you what happened.
Never put up a panel with text that cannot be copy/pasted.
Never put up a panel that won't fit on small screens if a lot of text is displayed.
If you are getting bad reports from people, start asking, not "what caused this bug?", but "What would help you provide better information next time?".
And make it so easy for people to report bugs that they can do it one-handed.
You will get bad reports. Don't try to prevent those.
That's not how you submit a bug for the /. mod system!
(There, now you're back on-topic.)
This post © Copyrite Duggeek, all rights reversed.
oh, I thought it was a feature =)
I hit "overrated" by accident on a decent post and thought the author deserved better treatment than that... afaik that's the only way to "undo"
Users won't report minor bugs, but bugs that destroy their data are typically reported. So, always make sure any bugs destroy data.
Go to Heaven for the climate, Hell for the company -- Mark Twain
I use a template asking how to reproduce the bug, and ask the user for the exact steps. If it can't be reproduced, its marked "invalid". If it can be reproduced, we check the requirements. If the behavior violate the requirements, the bug is against the code. If the behavior does not violate the requirements but is still in need of change, we update the requirements, and check the previous step. Pretty straight forward, although it takes some work.