Slashdot Mirror
Ask Slashdot: How To Get Non-Developers To Send Meaningful Bug Reports?
DemonGenius writes "I'm in the midst of a major rollout of one of our primary internal applications at work and we have a beta version available for all the staff to use. The problem here is most of the staff don't know how to send reports meaningful enough to get us devs started on solving their problems without constant back and forth correspondence that wastes both developer time and theirs. Some common examples are: screenshots of the YSOD that don't include the page URL, scaled screenshots that are unreadable, the complaint that wants to be a bug report but is still just a complaint, etc. From the user's perspective, they just send an email, but that email registers in our tracking system. Any thoughts on how to get the non-devs sending us descriptive and/or meaningful reports? Does anyone here have an efficient and user-friendly bug tracking system/policy/standard at their workplace? How does it work?"
Make your software send it...
You can not teach the world, so why try?
Either that, or send them all to reeducation camps.
Try not to stress about it, it's hopeless.
Sent from my PDP-11
Electro-shock. 'Nuff said.
A more pressing question is how to get developers to stop ignoring bug reports.
Give people a small, but meaningful bonus when they send in a meaningful bug report. If not, just disregard it and send them a canned form on which they will have to write all their bug reports.
For Windows developers at least, there is one spectacular piece of the OS to make use of for bug reports. Mini-dumps. So awesome I wish Linux had an equivalent. Makes finding bugs much easier when you can load a mini-dump and break into it with a debugger. Best of all you don't have to host your own server to cache those dumps, Microsoft will do it for you as long as your copy of Visual Studio is legit.
Send them a bug report template. Mark in red the things that are very important for your particular application (e.g. you mentioned URLs).
You can then help you help them help you with little questions like 'can work around' or 'can be reproduced'.
I was thinking more along the lines of having a meeting for each group, then referring people to the rules if they don't submit enough information.
But yeah, lazy and pointless.
Seriously, don't just keep blowing off the important user bugs for multiple release cycles. Once your bugs have been blown off for 6 months you stop submitting new ones.
Was this really a question?
Don't most of the apps you use have some means of reporting problems back to the developers when they crash or have errors?
So it is an ASP.NET app? Seriously, override application_error in the global.asax, and just send the emails with detailed error dumps and messages straight to you, in real time.
and find out what you think might be wrong, instead of what is wrong.
Engage the users. It's your problem, not theirs.
If they did Unity, Gnome 3 and Firefox wouldn't have the massive amounts of negative feedback they have now. In fact, Firefox development has gotten so bad they are currently having issues compiling
General users cannot and will not grasp the finer points of how you need the information. About all you can do is
1. Ask them to resend, but with specific directions
or 2. some sort of screen sharing application - "Ok, show me exactly what you were doing when..."
...ed and meaningless "article". This is probably the laziest and most pointless story I've seen on /. Try debugging in alpha and beta versions more before releasing into the wild.
So fast to jump the trigger. Letting aside that you don't debug, but you fix the bugs discovered by testing (well, you may use a debugger, but that's not the only mean available), consider this:
Recession; meaning: short of money; meaning: less development resources available or available for shorter time.
Consequence: do what you can within the bounds of "Cheap, good and quick: pick any two" - it is going to be way more expensive on long term, but... try teaching this to CEO-es under pressure to show results to shareholders.
Questions raise, answers kill. Raise questions to stay alive.
This is a great point. The distinction that actually matters is not so much whether the bug reports are submitted by developers or lusers, but whether they're submitted by idiots. The fact that someone's been hired as a developer reduces the odds of idiocy w.r.t. software, but it's neither necessary nor sufficient, and a certain degree of optimism is required to assert even the correlation. Likewise, it's plausible that lusers are more likely than developers to be idiots about software, but exceptions exist.
The real, general solution to this problem is, of course, to get rid of the idiots on both sides. The solution to this problem is left as an exercise for the reader.
I admittedly didn't read the full thread so far, and I know at least some of this was covered.
But have either a web site or a bug reporting facility in your app filled with a template that the user can fill out. Something like:
Brief description of problem:
Steps to reproduce the problem:
What you saw:
include crash logs, snapshots, etc
What you expected to see:
Severity: Crash/UI/Performance/etc..
Version of app/configuration info (filled in AUTOMATICALLY if possible).
and of course do NOT make them fill in every single field (if it's not reproducible, they won't have exact steps.. but you might be able to narrow down on the problem with multiple different bug reports).
Personally, I hate these templates in our bug system, and cmd-A, and type over them, since I already know what kind of info to put in.. but it is useful for regular users (or even developers, if there are special steps, like special debugging tools for your specific app/tools).
Look at Firefox. Not only have they ignored the memory leak problem since inception, they not only dismiss it as fiction, but actively brand the users as liars. Is your company like that? If so, don't bother.
Maybe if you didn't use obscure acronyms then your users would be less intimidated to write up a meaningful report without their situation being discussed on /.
I've had success by educating a small group of business users to serve as triagers of bug reports - it's a lot easier to have them enforce your rules about what a validated bug looks like, particularly when you've armed them with a template to check against (example - without a screenshot including a URL, time of incident, browers/OS, the report gets sent right back at them) or have them validate the issue on their own before an engineer even looks at it. Eventually the business users get good enough to aggregate incidents together to locate true bugs and also can shoulder some of the prioritization burden. Remember that its in their rational best interest to care about software quality even if they can't speak about it in a sophisticated way.
We have also posted examples of good bug reports and a wall of shame for bad ones - if you can get incentives together (or punish those that resist reeducation) that helps too.
From the question:
It's an internal app so just have the app log everything that the user does in that app.
Then, when the user calls to say there is a problem, the dev team can pull the logs from that machine and recreate the exact sequence of events.
And don't worry about the logs becoming too large. If the dev cannot figure that out then there are larger problems there.
Also, have the app check the versions of the libraries and such in the OS.
Since I've seen QA people do stuff like that. (And not put down the steps they did to produce the bug in the first place. Hell, I've seen QA not get the difference between a bug and a feature request.)
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
Give up while you're ahead. Getting useful diagnostic information from someone trained in the art of programming can be a trial in of itself; from someone not trained in the art, it's all but impossible.
If anything, make your software grab all relevant information.
I am John Hurt.
The best thing you can do is prefill a form with something like:
1. What are the steps to reproduce the problem?
a.
b.
c.
2. What happened?
3. What did you expect to happen?
4. Are there any workarounds to the problem?
Bingo. One of the main apps that I use has a 'Feedback' button that tech support has hijacked to have users send in bug reports. I'm sure the Marketing department is annoyed, but that just makes it better.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
"The computer is acting funny."
Which computer? Do you think it's a comedian? Is it here to amuse you?
A system that I've used in the past, that sometimes works, would be to have the users fill out a "checklist" of items/questions when submitting the email. Trouble Url? Error recieved? Complaint or problem? Etc etc. If you know the user base and this is provided up front, it may help with some of that. But that assumes the checklist is viable for them to fill out as well as making the questions answerable and usable from both their and the developers perspective.
-= My spoon is too big =- Errosion
Or they will just stop sending bug reports. If you are an ass to your users then they will weigh how much they care about the app working vs how much of a pain it is to deal with you. If the app is relatively unimportant to them or you are a very big ass then they will stop sending bug reports (and possibly stop using your app). In which case the dev loses.
It sounds like you are just depending on people to email their issues straight to a developer. I would put up a web portal for bug reporting that includes pre-defined sections and issue types. For instance: "Main page >> graphical issue >> unreadable graphic" or "Product specific page >> functionality >> button issue". Then have a "what were you trying to do" and a "what did you do" fill in section.
YSOD? Maybe you need to be more clear to your users. I don't know what YSOD is and I work in the industry... Make sure your users understand what a bug report is, and how it helps to give as much information as possible. Avoid using terms they won't understand, and assume they don't know what you want. Some users will try to help if you tell them what you need, but give up if they feel like they have to figure it out on their own.
We use JIRA and follow a general template (description of problem, steps to reproduce, users impacted/business impact, severity). Tickets submitted under a few different main project areas go to the respective SME for triage (follow-up, prioritization, reassignment to engineering). Speaking as one of the non-devs but with some prior QA experience, it seems to work just fine.
What a ridiculous comment. You are clearly not a software developer. I guess that's why you posted AC...
Even the best programmers make mistakes - but they also write unit tests to find them and functional tests to find issues with systems/integration. Even given that, a developer would still need to thorougly test (in various automated or manual ways) their work to make sure there are no bugs in a complex system - and the point of QA is both that they have expertise in that area, and it's not worth higher paid developers' time to go through all of those tests themselves.
A thingy that taught me how to send bug reports was the template in Bugzilla. Just a pre-filled piece of text in the submission form:
Summary:
Description:
Steps to reproduce:
1.
2.
3...
Expected Results:
Actual Results:
Additional informations:
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
Give them a "Report bug" clicky that brings up a form for them to fill out and which automatically extracts relevant information.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
The problem here is most of the staff don't know how to send reports meaningful enough to get us devs started on solving their problems without constant back and forth correspondence that wastes both developer time and theirs.
It sounds like the actual problem is that the software does not meet the needs of the users (and presumably the organization). This is where you ask "why" recursively at least 5 levels deep to figure out what is actually going on before you assume that the problem is a lack of proper bug reporting. It may turn out that something more fundamental needs to be fixed first which makes the bug reporting issue irrelevant.
Give them a checklist of information you want. When they give you a bad bug report tell them exactly information you want. You want them be specific, so be specific about what you want.
Democracy Now! - your daily, uncensored, corporate-free
You have to coach them. They don't really understand what you need.
When I get a email from someone about a bug, I go meet with them. I ask them all the questions I think may be relevant. What were they doing, how were they doing it. Were there any extra small steps or actions that jump out. Sometimes I explain why I'm asking certain questions and relate them back to previous bugs or issues.
I think what you need is someone to be the go between. Get a tester to receive those emails, recreate the issue, then file a bug report. Don't allow the end user to file bug reports directly into your system. It will mess with your tracking data. A high number of worthless bug reports closed quickly may look good in the reports but does not help anyone.
Im a gamer, not a grammer major. This post is full of spelling and grammer mistakes.
.
If it is an open source product, it is not so easy....
- some projects require you to appeal to the egos of the developers. This is a losing proposition, as their egos are far larger than you can imagine.(the Linux kernel comes to mind)
- some projects require you to navigate a maze of problem reporting pages. These projects have lost their compass.
- some projects require you to show technical or use case reasons why what you suspect may be a bug is really a bug. There is hope here.
FYI: YSOD = yellow screen of d
It's a common occurrence when the back end server sends out the YSOD instead of the content you intended for it to send when you are running buggy ASP.NET code (ASP.NET is a Microsoft web application framework based on Active Server Pages. See http://en.wikipedia.org/wiki/ASP.NET ).
Generally, you would hire someone who understood overriding the application_error, as was previously suggested by an AC poster in this posting: http://ask.slashdot.org/comments.pl?sid=2572922&cid=38365514 . In other words, you'd hire someone who could fix the problem, rather than asking Slashdot.
They'd probably also be clueful enough to know that the place to ask for help with Microsoft products is Microsoft, and that they are unlikely to get a lot of love from the Slashdot crowd, who typically avoid work-for-hire coding models that make it difficult to reuse their code later on if they go to work for a different employer. Very few clueful people like to have to solve a given problem more than once.
-- Terry
You haven't told us anything about what you've done to educate the staff about writing up defects. Have you done anything? Unless they also work as technical QAs, they almost certainly have no idea what is expected in a defect report.
I'm surprised that the primary problem doesn't have to do with listing ambiguous steps to reproduce a defect. For example, "I went to the page again" vs "I clicked the link again" vs "I closed the browser, reopened it, entered the URL, and then clicked the link on the page again."
Of the 3 things you mentioned (#1: screenshots of the YSOD that don't include the page URL, #2: scaled screenshots that are unreadable, #3: the complaint that wants to be a bug report but is still just a complaint), the first two sound like *you* are complaining -- if they're not technical people and they don't know how your system works or what you need to see in order to diagnose a problem, then the first two issues should have been expected.
How do you solve #1 and #2? If you're asking users to perform rudimentary QA, you're going to have to work with them. So, the solution to the first two problems is for you to realize you need to help out in these situations, explain what was wrong and what they can do to fix it next time. They don't know what you need in advance. Progress will be incremental but sure. And if they have to do all this in addition to their other tasks, then be nice while you're doing it.
How do you solve #3? First, if people are complaining then make sure you're not overreacting to a real usability issue and ignoring it because it's not in the format you wanted. Then mail a template with a description field, a steps to reproduce field, an expected results field, and an actual results field, with a description of what each one is. Have a meeting explaining how to use it. Give examples, especially highlighting the need for detailed and unambiguous steps to reproduce.
As bad as e-mail is for tracking issues, if this is a short term thing it may be the best approach. If you can see this going on, getting away from e-mail is a very good idea. Having recently been through a issue tracking system transition, getting everyone used to the new systems took some time even though it was handled very well (and not by me).
Even better set up something that will automatically configure my system with all the debugging tools. Give me scripts that allow me to easily compile and run older versions to find where a regression occurs.
Unscrew gdb so I don't have to know arcane crap just to get it to give the backtrace you need.
Once I have that and some minimal instructions on how to generate the reports I will.
Don't have a pissy fit when I ask you how to do stuff that you need done.
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
That's a cop-out....and completely unrealistic.
Finding a decent way to look through GB of logs to find something that happened a week ago is going to be MUCH more of a challenge than just arguing with any client to get the information you need. Not to mention, the only way you're going to trace any random malfunction back to any random source is if you have so much logging that half your computation time is spent managing your logging system and the needed disk writes....and let's not talk about the volume of log data THAT would produce.
Logging can, and should, only do so much. You can't, and won't, catch everything with it.
Programming is black magic voodoo to just about anyone who isn't a programmer and they think that we're wizards with keyboards who can solve any bug with the most vague instructions.
Ask them for precise, numbered, step-by-step instructions on how to get to the point where you will see the exact problem they are seeing (don't bother shortening that phrase to "replicating the problem", they won't know what you're asking). You will likely need to ask them several times. Make sure they know that you need to be able to SEE the problem in order to fix it. Heck, tell them it's otherwise like going to a mechanic and saying, "my car is broken, fix it", if you need to.
Yes, you are tilting at windmills to some extent.
if they are on windows 7 and can reproduce the problem, see if you can get them to use Problem Steps Recorder
Just have them click on the start button and type PSR. You will get a macro recorder like window up to start recording. it tracks what they do, takes screen shots, lets them add comments, and builds a ziped file for mailing when they are done. Takes some training but can be alot less work for the user and very helpfull for you.
Good luck
picture tells a thousand words.
Develop meaningful bugs.
The subject sounds offtopic, but seriously, a clear, natural, everyday usage of the scientific method to solve problems would make your average user be capable of sending meaningful bug reports,because they would understand how problems are solved in the most general sense. They would realize that in order to diagnose where a problem is coming from, all of the possible sources of that problem must be independently tested in a controlled way. Doing that by nature makes for naturally good bug reports.
that's the best way to communicate to the user, that took time out of their non-programming day in order to do you the favor of telling you how your application doesn't work, that you don't really care about them, the problem they were having, or if they are able to use your program.
you don't work on the TortoiseSVN project, do you?
It sounds like you need multiple levels of support so the only ones that enter bug reports into your bug tracker have been trained and you are not talking to users directly. Pretty standard stuff, devs talk to level 3, level 3 to level 2 and level 2 to those that answer the phones, emails, forms etc. all coordinated with trouble ticket systems and support contracts to prevent the silly cup holder calls.
The "Send Feedback" button on plus.google.com (and some of their other sites) is really nice. I'd encourage you to try something like that.
So, really, two things come to mind:
1.) In whatever bug reporting mechanism you have (which should be easy to access in a 'help' menu or a hyperlink that's easy to reach or whatever, provide directions that give people an idea of what a bug report should contain. How long should it be? What type of information is most relevant? Etc. If this is a company-internal application, generally your users will have more incentive to help you than many other applications you could be using, so they'll maybe even read the instructions.
2.) Yes, you need a tracking system for every bug report that comes in... but that tracking system probably shouldn't, if you're accepting reports from the "general public", be the same as your resolution system. It seems like half your problem, from your post, is a process issue where "Bad" bug reports are mucking up your bug tracking/resolution system. This should be resolved by having someone vet the bugs, do the followup, de-dupe etc. before it gets into the primary resolution center. Granted somebody has to do this, but it can be a job for a helpdesk guy, a manager, or even as a way to teach a new developer to your team about the system. (yes a manager . . . if the number of reports isn't too high, when I'm leading a software effort that's in deployment, I find reading the raw reports can be valuable to give me an idea both of what's going wrong, and how my system is being used, and if I'm doing that anyhow, I have some time to do the basic deduping)
it's amazing how often i see this happen with open-source software: some guy releases his pet project that he's been working on in his mom's basement for 2 years, he thinks it's the greatest thing on earth and as soon as someone comes along and leaves some comment that isn't just gushing praise they start behaving like a defensive six-year-old.
if you're getting YSOD's you should NuGet Elmah
Only sell your software to developers.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
When your body has a bug to report (pain), you call the doctor. What does the doctor do?
- Ask where it hurts.
- Ask when the problem started.
- Ask about other symptoms that might be occurring.
- Run some diagnostic tests.
The fact is, you're never going to get meaningful bug reports from non-technical people, any more than patients are going to report detailed symptoms and vital signs, complete with test results, to their doctor. That's the doctor's job, and diagnosing software problems is your job.
I run bug/task reporting tools at a marginally large company. We use Atlassian Jira for that, and one of the processes that has worked well to capture data from non QA type folks is a bug type called "simplified bug" that simply has space to comment, add screen shots, and feed in minimal data (what release is it, platform, and maybe two others).
Any bugs filed from simplified that appear coherent enough are modified into a full regular bug, with some followup. It's a bit more labor intensive, but the qa team has a few people watching the simplified bug list, and we get a lot of useful data from it.
What a ridiculous comment.
What is ridiculous about it?
.
(note - I am not posting AC)
Why do even the best programmers call their mistakes bugs and not their mistakes? Why are even the best programmers avoiding taking responsibility for their errors
Why do you try to rationalize programmers' errors as being the result of ~there is too much to test~? Don't you think that programmers who really care about the quality of the code they write might object to your kicking the can down the road and blaming the problem on QA?
Why are you avoiding the responsibilty for the errors you as a programmer create?
Always assume the bug reporter knows NOTHING...reporting everything inside the app you can & reporting the crash back to you directly. If you're excepting anything past "I clicked on this & it said error XYZ" LOL... that doesn't' scale past 10 people.
For bug reports, I wrote a tool that takes a screenshot, dumps the contents of every control in every open window as well as a bunch of crap from the registry, packages it all up in an email that it pops open, with all sorts of stuff pre-filled out and the cursor ends up at the end of a line that says "The problem I experienced was: " They type a couple of lines and click send. Done.
They have complaints that aren't bugs? No they don't. Their complaints ARE bugs... just not software bugs. They are bugs in process, and not your department. Forward them on to whomever designing the process they have to follow, CC the person that sent it to you explaining what you're doing. Or is it that they have a feature request? Tell them so, pass it on to whomever is in charge of such things. If 90% of what you get is not your responsibility, and you just ignore it, you'll be seen as a black hole and your users will not waste the effort to send you anything... as a result you'll lose 100% of the actual data that you need. If you properly handle all of the users requests, and make the process transparent to the user, they will learn... and start sending the appropriate info to the right place the first time.
The application I maintain interfaces with nearly every software package and department in our rather large company. There are probably thousands of software products it touches. I get this sort of thing all day long. I've had the best success when I've found a "Senior" or other, more technically apt person in each department and designated them as "bug reporter" or some such. Their team takes issues to them, they bring them to me. I train them on what to send to me, and where to send it if it's not me. We learn together. Often I don't even know how they are using the software, so they can help me in return.
Long story short, you catch more flys with honey.
In Windows 7, Microsoft created a wonderful tool to help you. It's called Problem Steps Recorder. Basically the user runs it and it snaps screenshots while the user reproduces the problem. There's a comment box allowing the user to type in comments like what they typed, what they expect, etc.
The best part? It emits output as HTML (!!!). Just send the zip file to the devs (it offers a handy email option), and it gives them repro steps and everything.
Wonderful utility. Even more wonderful is it can be used as more than just recording problem steps - you can record tutorials in it and post them online and stuff.
Create a page document shows guidelines on how to submit bug reports, and send an email to all users that says that this is where the guidelines for filing bug reports live.
When you get a report that doesn't help, close the report with a referral to said document. This, of course, presumes that your users get email reports on the status of bug reports. If they complain that nothing is getting done, refer them to the page again, with a statement that you require more information for the bug report to be useful.
This said, be clear on what, say, YSOD is. TLAs and ETLAs are meaningless outside of context.
This sig no verb.
First of all, if it's a plugin that causes it, why has that happened for at least the last 3 years and nobody found the plugin that did it and blacklisted it? Second of all, why should that plugin, or other cause of leak crash all my tabs? Chrome crashes only the tab of the offending page/bug.
Replication is easy, windows XP or higher, any firefox that is currently actively developed on and a recent flash plugin. Open about 100 tabs of various sites, taking care to make these a representation of the (type of) sites the average user visits and start browsing in them. I can guarantee you that you'll have a bigger than 1.5GB browser that will crash in less than a few hours.
Sure, you can blame flash like Apple did, but not being able to fix it for 3 consecutive years? There was a period when the development version of FF would simply dump page history once you browsed away from a page. That made it several times better and faster by itself. Unfortunately, they either took that out, or the leaks have gone so bad that even dumping page history from memory doesn't help anymore.
I was promised a flying car. Where is my flying car?
The saddest thing is when the staff who fails to produce a decent bug report is QA. I have rarely seen QA actually have the ability to capture data and think critically about how to get to a resolution. Therefore, I have very little expectation that anyone else would, either. Just ignore the reports until another avenue for fixing it reveals itself. Your responsibility to stay sane and respectful overrides the responsibility to improve software quality.
I have seen several good ideas in this thread already. Here is another that may be useful.
Try to find a way to associate the outcome the user is hoping for with the behavior that you are trying to engender. The user is hoping for their problem to be resolved quickly. The behavior you want to engender is a complete, yet concise, description of the problem and the steps to reproduce it. Make the connection between the former and the latter in the user's mind, and you'll have a self-motivated convert.
When you get a clear and concise report, be sure to thank the submitter and help them realize that the clarity and brevity made it easier to solve the problem. They'll feel good about it, and you might even be one of the few on the tech side to have ever said, "thank you" to that particular end user. (we can be an ornery lot)
When you get an incomplete or vague report and have to ask the user for more information, be sure to explain that a more clear bug report will make it easier for you to solve their problem quickly in the future. Make sure it comes across as, "I want to solve your problem more quickly in the future", not, "You did it wrong."
When you get a rambling, nitpicky, or pure-complaint report, you'll have to use a bit more judgment to figure out how to guide the person for future reference, but you can usually find a way if you think about it.
Really it's just programming -- most end users don't exactly have a complicated UI. Be nice, help them understand how satisfying your needs will benefit them, they'll do it. A nice bonus is that they'll be better end users for every subsequent support person, and they will feel good about doing it.
Stop-Prism.org: Opt Out of Surveillance
You are lucky enough to have users that actually report problems, rather than just complaining that "the new version is crap" at the coffee machine. Setup a bug triage task, assign it to a different developer each day (because it's boring). Precisely mark the bugs that are not submitted with enough information. Every Monday, Identify the top 5 users based on the number of bug reports they send, and the ratio of their bug reports that miss information. Set up a meeting with each one of these users. Sit next to each one of your top 5 users, and write the mails describing the issues with them (mark the previous one as duplicate of the new issue). These users are willing to give you information, educate them on the information you need. Then do it again the following Monday; your previous week's top 5 users shouldn't be in the top 5 again, since you wrote their bug reports with them. Be transparent to your management on this new task you are setting up, and its cost. They may be willing to help you.
Have the user give a step-by-step list of how to achieve the bug in question, along with screenshots, and system information. The problem with bug reports is that if a bug can't be duplicated reliably, then statistically speaking, it's probably not a bug in the software. While it is up to the software developer(s) to make sure that the compiled code executes correctly on a system, it's also up to the user to give a meaningful bug report. If the user is unable, or sometimes unwilling, to work with and/or provide information to the dev(s) to figure out what's causing the error, then there's really nothing that can be done.
This is why there's alpha releases and beta testing. From personal experience, I try to have testers that at least have some clue of what's going on, and during the alpha phase, provide them with full source code to the product that's going to be released. Once the features have been reworked and simplified to an acceptable level, then the source code is no longer provided, and the user(s) are then required to test the actual program itself. If any flaws crop up at that point, both parties have a general idea of where the problem might be occurring. However, most companies don't really follow this approach as much as they should.
In summary...
Users: Try not to act like retard, and make sure the information is clear and concise. Telling devs the equivalent of "ur shit sux lololololol bc teh app crashed" doesn't help at all. In fact, that alone is the biggest reason why most "complaints" go unanswered. You have a brain, so use it.
Devs: Take the proper route and prevent bugs before they have a chance to crop up. Use less lines of code, stress test constantly, and simplify every aspect of an application as much as possible. It'll save you a lot of headache down the road. Most of you either have degrees or have the equivalent experience thereof, so start acting like it.
Finding a decent way to look through GB of logs to find something that happened a week ago is going to be MUCH more of a challenge
I'm running a website with 2-3k concurrent users; using web server access logs for read-only users, and a more detailed custom logging system for any interactions that any user makes, only generates ~100GB of logs per month. In the grand scheme of things, this is a tiny amount (grep from the command line is still perfectly adequate). For someone who can dedicate a day or two to writing a log browser then I'd expect the search to be even better, with graphs and automated anomaly reports and such (As someone who actually has done this, I'm not sure whether to find your comment of unrealism worrying or complimenting :P).
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
How about talking to the users and actually have them show you what is wrong?
It does not even have to be the developers. If your company is so big that the users are too many to talk directly to the developers, then surely there is a helpdesk that they can call and who can help them file the bug report. With a little screen sharing whoever the user talks to can do the needed screen dumps.
It seems that IT nowadays are so afraid of actually talking to a user that is is funny. It is supposed to save money, but taken to the extremes as described here it is just silly.
I have worked on the development team as one of the primary developers of a product which had in a period of 5 years roughly 2.5 million bug reports with approximately 10 million users. In modern times, that's not really that much anymore, hell the guy writing MineCraft would laugh at such small numbers.
.NET so that when a YSOD occurs, you can gather the trace log, maybe take an automatic screen shot, etc... adding a key logger to your app (not a malicious one) so that all events are logged (pretty easy by overloading the application event handler) for the past minute is useful too. This should allow you to know what was clicked and in what order.
In any case here are some of the things to consider.
1) The majority of people submitting bugs are thoroughly incapable of writing anything meaningful.
2) The job of filtering through bugs (even on a small project for a hundred or so users) is a lot of work and you need dedicated staff for this. College students, summer interns, temp workers, anything. You need to have someone actually sitting, reading the bug reports, marking them as duplicates, reproducing them and then documenting them with more meanuing than a user can write. That person/people will be responsible for contacting the users to find out more about the bug if necessary.
3) An open line of dialog between the developers and the users will NOT be productive. It will make the users feel special, but users have a terrible habit of thinking that they can start coming to the developer all the time with every problem they have.
4) $10 an hour for a bug hunting monkey will save you hundreds of dollars for that same hour of well paid developer time. In addition, it will help a young computer wanna-be gain meaningful experience to put on their resume/cv.
5) Bugs need to be prioritized and assigned. You should make sure that things like crashes in the billing system which could cause invoices to get lost are fixed before (and by the right person) bugs in the inventory system database. (For example). While inventory is a critical component of a business, if a bill gets lost, instead of a delay in a shipment, you'll lose the payment altogether.
6) As others have mentioned... automatic crash reporting is necessary. It's usually not that hard to implement in most languages. I'm sure there's a function for it in
Large scale deployment of applications is utterly unmanageable without a proper support system for the developers. An extra employee is a big expense, but if the management decided to develop the project internally instead of farming it out, they volunteered to take on the additional expenses of development. Get the bug boys/girls in and get them sorting. It's a shitty clerical job, but it will save the company tons of money in the long run.
First you seek key users. These will be the people who should be testing your software. See that they are a good mix of people of technical knowledge and level of command, depending on who must work with it.
That key group must be already availble to you at the beginning, naming demands. This will mean actual meetings. In the beginning you will hear all the complaints about the current sftware, so you have to dilute what all want. Show mockups to them at later meetings. That way they will see that you listen to them.
Once they notice that AND they are sure it will be an improvement, they will WANT to betatest for you. Not only that, they will be your advocates who will make people also WANT to use it.
As this is a small group of people it is easy to explain them how to file bug reports. Next you do a tryout on one department where your most energetic test member is located who also. Ask (not tell) them to do the test. Still small enough to do an explanation on how bugreports should be done and what you are looking for.
Once that is done, and you are ready for the whole rollout, have a webpage set up that is easy to use. SOme simple technical questions and some fields with likes and dislikes.
Ask the test members to see that as many people have answerd and publish the result and also do something if major issues arise. Especially in the beginning. This will make more people aware that you listen and will be more willing to help with the next project.
The downside is that suddenly YOU are the one that requests meetings and have social contact.
Don't fight for your country, if your country does not fight for you.
Hi, you could potentially get corporate to spring for commercial software that can help with alleviating these issues (or build your own)
:). Feel free to PM me for some suggestions.
One approach could be monitoring approach, where Network Traffic gets analyzed and sessionized and you get alerted on events. Session data should allow the identification of users (that way you should be able to "look up" the user's session). The session page-flow could then give you URLs and POST/GET/Cookie/Header data. In conjunction you could utilize server-based deep dive diagnostics that provide contextual code-level execution for root cause analysis. Actually, this gets you half-way to have decent Application Performance Management (APM). Integrated into Business Service Management (BSM) it's fairly hot.
Another approach is an agent-based solution that allows the end user to interact with the agent to send you the last n amount of minutes with a high-level description (simultaneously recording the application on the server-tiers so that you later can do root cause analysis). This is an efficient approach, especially for internal users. This approach can also be leveraged during testing cycles by the Quality Assurance team.
I have some examples of commercial vendors, but am hesitant to post names to avoid being flamed
Good luck.
You missed punctuation at the end of your third sentence. While I understood it, a computer might not. That's a mistake or bug, whatever you want to call it. Some feel secure enough not to care.
Anyway, I don't think most programmers avoid responsibility, they just have QA report their mistake/bug and then they fix it. Big f-ing deal. It's not "too much to test", it's tested by the testers. It's not "kicking the can", it's letting those who specialize in testing do their job.
Again, you clearly are not a real developer (or if you are not the OP, you are not a developer *either* :) If you were, you would have gone through at least one marathon bug fixing session before an important release (and of course often the bugs found & fixed weren't even in code you wrote). And even if not, if you are somehow claiming "good" programmers don't make mistakes, there are by your definition no "good" programmers so you should probably stop using your computer right now.
Programmers who care about the quality of code they write also care about the quality of the QA department who tests it. I have made literally hundreds of mistakes (or more?) in my career resulting in many bug reports that I have fixed. And as a result of that, I have developed software being happily used by millions of people every day. I seriously doubt you can say the same.
What worked best for me was to identify and develop personal relationships with my most insightful users (and just monitor the others for useful comments). That's something all these software gimmicks can't do.
"The ability to delude yourself may be an important survival tool" - Jane Wagner -
You're lucky because it's an internal application. You should know your users or have easy access to them. Go talking with them when you get bug reports. If they reported a bug they care about it and they'll be pleased that you care about them. They'll show how they got it and you'll teach them how to report it properly when something else happens. Then be sure to fix those bugs or explain them face to face why you're not going to fix them. The keyword is cooperation.
Have your application send bugreports.
If your application is coded in Delphi there is MadExcept: http://madshi.net/madExceptDescription.htm
Whenever my apps crash, hang/lockup or throw an exception, MadExcept steps in and the user will simply see:
When they chose send bug report, it asks for their name and email adress (minimum user incentive IMHO), asks what they where trying to do, and then asks them if the shown screenshot (of the apps state) may be send along. And i allow the user to edit the screenshot before sending it.
Its free for non-commercial usage so you can test it yourself: it sells itself.
But asking the right questions like http://askmebetter.com/ helps too.
Hivemind harvest in progress..
Simon Tatham has a good writeup about bug reporting. It's a great article, well worth the read.
http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
But, if you won't read it, here are some one/two-sentence points from that article:
Or Simon's summary:
Describe everything in detail. State what you saw, and also state what you expected to see. Write down the error messages, especially if they have numbers in.
Ask me about repetitive DNA
When I've been working on an in-house application, *the* most effective way of understanding the bug is for the developer (me) to go and sit with the user while they demonstrate the problem. This helps with developer/user education too: "You're doing *what*? I never thought of that! It's much easier if you do this way..." Obviously, this only works if you're in roughly the same place, but spending a couple of minutes to go over there can save many times that in trying to decipher an unclear testcase, or weird enhancement request. Your users will appreciate the attention too...
1. Make a simple clear web page that (in not too many words) shows people what is expected.
2. If they do a good job, reward them by promptly fixing the bug for them.
3. If they do a bad job, bounce the ticket back to them with a link to the "how to raise a bug" page.
Believing something doesn't make it true. Not believing something doesn't make it false.
Make them run some sort of diagnostic utility before they reproduce the bug. It captures all their activity, running environment data, and reports it, giving you perfect diagnostic data.
Oh wait...
Not to be a jerk here, but the best way to get bug reports is to make it clear that you'll respond to them. There is nothing more frustrating to a user than taking the time to file a report of any kind (either by using the software's automated thing - Opera web browser does this) or manually (I've filed several bugs against KDE PIM software) only to find it unaddressed. If on your website you make it obvious that you're receiving and dealing with issues, users will be more tempted to write you, and thus more inclined to do so in a helpful/thoughtful way. I've had great success with softmaker.de - every email I've ever sent with a bug report has been responded to with a solution, usually very quickly.
If this were Usenet, I'd killfile the lot of you.
You can tell them og write it to them but don't expect it to work.
Best I can think of is to write a bug report to their bug report. So when your get a useless bug report, write back and explain what is wrong with the report and how it should have looked. Explain to them that your crystal ball is broken, so you don't know what the "it" in "it doesn't work" is. Asked them read the bug report before they send it and then stop and think if it even makes any sense....be nice but firm about what you require to fix the bug and let them know that it will make the process go much faster....but guess what...none of this will work.
If you want to make it stop - get another job. ;-)
For the record - no, I am not bitter (yet) but I just sent a new site into test and I am just waiting for the first useless bug report that ignores all the instructions I gave.
People seem to forget that extracting meaningfull information from end users is a highly specialist skill. Developers are from my experience clever folks, however frequently they lack the ability to learn the most important language in IT, EUS - End User Speak. To my mind the most undervalued skill in the industry.
In my last job I was the first ever IT person to be based in the branch office. The users perception of the software and systems was very negative. The main IT function was based 300 miles away and only did a token visit to the site once a month. Being based on site and actually sitting with the users I could see just what impact the systems were having; when you end up staying past midnight, helping a user in tears becuase they could loose their job if the client didn't get the expected reports. You then start to appreciate the issues a hell of a lot more and why they have issues in sending developers good information to work from.
You have three options:
1. Go sit with the users or pay somebody else to do it for you and act as an interpreter
2. Automate error reporting. This should involve no more user intervention than clicking on a confirmation and should send them back a case reference number. This must then feedback to the user to make them feel involved in the process.
3. Educate some users and appoint bug monkeys on teams having repeated issues. There is usually one person on a team that thinks they know IT and likes to have a badge. Play on it, give them an hour or so of your time and get them to report the bugs for users on the team.
Support staff don't tend to be that well paid, we don't tend to get much love from users or developers. What you're seing now is what we can do for you and what we protect you from.
Perfection in a damaged box.
Any thoughts on how to get the non-devs sending us descriptive and/or meaningful reports? Does anyone here have an efficient and user-friendly bug tracking system/policy/standard at their workplace? No.
Just the washing instructions on life's rich tapestry
lusers do not think like us.
Don't bother
poor choice of wording - debugging - I meant exactly what you said. Fix more bugs during testing.
Of course it's not, but the question raised it how can a programmer get more reliable information on their broken code? Don't rush the software through testing and keep it in the hands of more knowledgeable testers before handing it over to users. I didn't say it was possible, I'm just pointing out the obvious.
I included a short section in the user documentation for some software used to access a data archive with up to seven items of information that would be helpful to quickly pinpoint problems. Over the years this software was in use, I think I only had one user actually send the requested information. Most couldn't be bothered to actually read the documentation, and neither could the help desk personnel, who failed to elicit the information before passing it on to us.
I disagree. I implemented this some time back on a financial application. We had the same problem where users didn't give us accurate information to find bugs, or fix their transactions. We decided that we would capture everything when an error was detected. User ID, transaction ID, time, error message, plus a few other fields were all recorded in a database. When the user called, we had several ways to find their error (I caught several users lying). We could also generate reports showing the frequency of an error, or category of errors. I agree that you can capture too much, but with some careful thought you can capture decent info to help augment the story from your user.
I have been educating users for years on how to submit even vaguely reasonable problem reports. It's an uphill battle.
Everything in our system has a ticket id.
More than 90% of all problem reports never mention a ticket id.
Some of the worst offenders have been other developers!
If peer developers cannot do it, what makes anyone think that the regular user community will step up?
Regard it as an exercise in patience and endlessly request the ticket id that was missing from the original problem report.
Reminds me of my days in the early seventies fixing tvs. Half the trouble reports would be "All black! It's all black!" even when what was actually wrong was that the sound was out.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Although the company I work at is small enough that I can just walk over to the user's desk and ask them to walk me through the problem. It helps if as they do this, you point out what information will help you fix the problem, so maybe next time they'll do better.
I always asked that any user take screen shots, and remember what they were doing at the time of the bug...what record they were on, what buttons they pressed in sequence, if they can usually you can reproduce it, although including an internal error log with the call stack and all subsequent info is much quicker as well.
Of course the worst type of bugs are those that turn up frequently enough to be a massive PITA but which there is no known way to reproduce. Such bugs tend to leave both users and developers MASSIVELY frustrated.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
We have the same problem where I work. When I use to have to gather logs I had to run 7 tools and enable flags. The development team then went and wrote one application with a start button that gathered everything for us. If you want good bug logs then provide the users with good bug tools. The less they can do to get the log the better the log will be. In our case this tool grabs everything that the device can give you and zip's it in an easy to use file.
I've had enough of software folks looking for requirements and now good "bug reports". Users/Customers do not have requirements they have PROBLEMS. It is up to the developers to create tools that solve those problems. The user doesn't even know what's possible or easy or hard to implement, so they can't tell you what the requirements are. Same for bugs. The user can't tell you details about which internal parts of your code have a bug, they tell you (at best) what they were doing when something happened. They don't know a crash from accidentally closing a dialog box. The solution - especially for internally developed software - is to go visit the users and have a conversation. Talk to them. Understand how they view your software and you can then understand their explanation of what it did wrong. This will also give you insight into how to make it better. Stop expecting detailed bug reports and start understanding users.
As a lot of people already suggested, try to automatically collect as much data as possible. But of course, that is usually not enough. For the data that has to come from the user, just make the form as specific as possible. So, for example, this is useless:
A user who has never submitted a bug report would have no idea what to enter into a form like that. This is better, but still not nearly good enough:
You would probably get an answer like "I was trying to create a photo album." instead of step-by-step instructions to reproduce. Try something like this:
...etc. Try to think of any relevant information that you're not able to collect automatically.
Your users will probably forgive you even if the form is long, if you don't make them think too much. Depending on how clever your users are, you might want to add some kind of motivational text (e.g. "the more information you provide, the better chance that we will be able to reproduce and fix the bug"), but there's a good chance that a user who submits a bug report already knows why they are doing it, so it would just feel patronizing.
Splunk is great but it should be noted it comes with a extremely high cost.
Got Code?
Use Joel Spolsky's guide to reporting errors:
1. Where exactly were you in the application (screen shot).
2. What happened when you pressed the bad button? (screen shot)
3. (most important) What did you expect to happen when you pressed the button?
That will generate good bug info. Each report still needs to go thru some sort of approval committee tho before being labeled a bug or not and getting 'fixed.'
After reading only the title, I was too shocked already to read the rest. :)
Developers, lowly-intelligent and lowly-educated as almost all of them are, are themselves amongst the people that deliver the worst imaginable bug reports. Give me a normal "user" anyday; their lack of developer arrogance makes them a functional partner in getting the report right. Developers on the other hand tend to mistakenly concentrate on technicalities, their own warped view of "what users want", or even already on what they "know is the solution".
Have a nice day!
We do this and it works great. We additionally send along some system data, such as last 5 actions, a URL (we're web based), and some other debug data.
It has saved us endless hours in finding the exact conditions needed to replicate a problem so we can fix it.
Web apps make it easy to trap and log exceptions in your framework, which you can then use to display your error form. Send a URL, browser data, and last 5 pages along with post/get data for them.
Hire actual QA to report and process bugs. Put at least one of the QA people on the project of processing user emails into bug/enhancement/etc.
User testing is great - it's a wonderful way to observe how end-users might interact with the product. But it isn't the same as QA - professional, systematic defect testing, reporting, analysis.
I'm a developer. Most of the time QA annoys me with issues like "if you enter more than 256 characters into the name field, what should happen?" and "If I'm in IE6 and I use the back button then repeatedly click submit, I get an error." These are usually things no user will ever encounter, but they're also things that should be tracked and checked, at least to be sure they don't corrupt the database or hang the system if the one user in 10,000 (usually a hacker "having fun") does try them.
Have your application link to a web form with the following fields:
Summary
Reproduction Steps
Result
Expected Result
Clearly explain the fields with examples.
All I ask is that if someone submits a bug report to tell me exactly what I need to do to reproduce the bug, and what you want changed.
A good bug report should have enough information in it so a complete stranger can follow a step by step procedure to produce the "bug" and what I need to be looking for. I figure that if I can't reproduce the bug, it doesn't exist (and therefore I don't need to fix it).
Joel has a interesting article on this.
I looked at the website, and quite frankly couldn't work out what it does. Marketing failure right there.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
Sigh. Bug != careless mistake. Bug = software not working as intended. Sometimes it's a silly mistake, sometimes it's an architectural flaw, and sometimes it's not even your code.
Even if a developer's code is 100% correct, there can still be bugs in an application due to the libraries used, the operating system, the compiler, even the hardware. I have worked around issues with all of those in application code before - in the end it doesn't matter if it was your fault or not, hopefully you have good QA that finds even the obscure, intermittent problems, and you make sure it doesn't happen any more.
moderated by ms-astroturf. They don't like people making fun of their business models [ now, that is a troll ].