Slashdot Mirror
Debugging Expert Wins ACM Dissertation Award
An anonymous reader writes "The Association for Computing Machinery (ACM) is reporting that Ben Liblit has been awarded the 2005 Doctoral Dissertation Award for his study on understanding and fixing software 'bugs' in the real world. From the article: 'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers. His approach uses sparse random sampling rather than complete data collection for gathering information from the experiences of large numbers of software end users. It also simultaneously ensures that the observed data is an unbiased, representative subset of the complete program behavior across all runs.' Slashdot broke the story on this research back in 2003. Apparently the project is still going strong."
No, not the wild-eyed madcap scientist from Back to the Future. Doctor Watson is an OS service present in Windows that monitors the running process list for terminal assertions. When a program hits an exception that it can't handle, it terminates immediately and Doctor Watson is on the scene to read the last gasps of the process before its bits get blasted. Microsoft even came up with a way to harness this to allow users to send real-time feedback to Microsoft HQ whenever a crash occurred in a program. No one I know ever sends that data back, but I'm sure someone must have once.
The current idea seems to be tracking the same termination events in the same way as Doctor Watson and sending the relevant data back to UWisc without informing the user. It sounds like a good idea, but I doubt it is in Liblit's power to fix Windows OS bugs.
http://findbugs.sourceforge.net/
This research has been a wonderful collaborative effort, and many people deserve to share the credit. To quote from part of the Acknowledgements section of my dissertation:
So thanks, Slashdot, for helping me find those users (or helping them find me). The exposure was invaluable. And thanks, open source community, for your participation. I've benefitted greatly from standing on your massed shoulders. This could not have happened without you.
Ben Liblit is now an assistant professor at the University of Wisconsin-Madison. He joins a fantastic Computer Science department. Good luck Ben!
So somebody went and formalized the theory of "the users are the beta testers"...
*tears out own hair and screams*
Shouldn't that be "leveraging out own hair and screaming"?
READY.
PRINT ""+-0
The installation of CBI is implicit consent to such monitoring, of course, and I didn't mean to imply that there was no consent involved at all.
However, asking us to read 170-odd pages of your dissertation is a little much. Would it be possible to describe the data collection system, how reports are generated and if the reports are sent automatically or as in the case of Dr. Watson sent with user approval. Also, what types of bugs you found using your statistical methods, as well as what types of bugs you think would be difficult to find using such methods.
A quick comparison to related mainstream debugging techniques would be useful to give us out here in the trenches a firmer grip on the techniques you describe.
And finally, if you wouldn't mind, could you describe a real-world scenario where a generalized product (codename: CBIMax) would be marketable. If such a general product is impossible, is it because each product is different and the methods you describe would need to be revised each time? What is the maximum level of abstraction of these techniques from specific scenarios that is achievable yet still retaining enough so as to not require largescale retooling for each project?
Thanks!
tr.v. leveraged, leveraging, leverages 1.
..." does not make sense. The overwhelming numbers are there they do not need improvement. What you may want to say is sthng like:
"for using the key strengh of user communities as a leverage to blah blah blah"
...
a. To provide (a company) with leverage.
b. To supplement (money, for example) with leverage.
2. To improve or enhance: "It makes more sense to be able to leverage what we [public radio stations] do in a more effective way to our listeners" Delano Lewis.
So listen and listen good all you academic paper writers: unless what you really mean by "leverage" is "improve", don't use it.
"Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers." WRONG. "improving the key strength of communities -their overwhelming
Just because a word sounds good it does not mean it should be used as a wild card
I know that in one particular http://www.kingdomofloathing.com/game, they tend to follow this approach. Once a new feature is created, and debugged enough so that it's stable and doesn't break anything, the feature is released to the general populace. After all, once all of the important bugs are found, a thousand users will find the minor bugs through general usage faster than a small dedicated team of testers. Also, the time the testers save by not having to verify every single minor detail can be used to work on new material.
Add into the equation that without some elaborate software (such as Mercury LoadRunner, or an open-source equivalent), it's hard to simulate the effect the entire population will have when they start hammering on the server. It can also help track down extremely low-occurance bugs, because with enough people working on it, those one-in-a-million cases will eventually come up.
Kinda reminds me of infinite monkeys eventually producing the works of Shakespeare.
The root word is lever and the basic idea is that you use something under your control to effect control over what would normally be outside your control. Like a very long handle on a pipe wrench.
The money aspect you refer to has to do with debt financing whereby you manage to use your equity to finance something larger than your equity. I don't think the article is referring to corporate finance.
In a perfect world you would use a few people who would recognize and fix the bugs. These people would never talk to the users. They would have no need to and neither would gain from the experience.
In the world that I exist in, users are the ones who spot the bugs, specifically the circumstances under which the bugs exhibit themselves. I use my user's eyes to leverage {user's eyes, my skills}.
If all you mean is "improve", you would not use a word which essentially demands a discrepancy in the metrics between cause and effect.
b. To supplement (money, for example) with leverage.
If you add money to an account because of a margin call, does this increase or decrease your leverage? That is a horrible excuse for a definition.
I told people we were switching to new software (Mailman)- and that if they got an error message or similar, to flip a quarter X times (I forget how many) and ONLY email me if they got all heads. I didn't want to get a couple dozen reports of the same problem, and I figured that if there were any problems, they'd affect a large set of the 1000+ users of the list.
It worked brilliantly.
Please help metamoderate.