Slashdot Mirror
Debugging Expert Wins ACM Dissertation Award
An anonymous reader writes "The Association for Computing Machinery (ACM) is reporting that Ben Liblit has been awarded the 2005 Doctoral Dissertation Award for his study on understanding and fixing software 'bugs' in the real world. From the article: 'Liblit's dissertation proposes a method for leveraging the key strength of user communities - their overwhelming numbers. His approach uses sparse random sampling rather than complete data collection for gathering information from the experiences of large numbers of software end users. It also simultaneously ensures that the observed data is an unbiased, representative subset of the complete program behavior across all runs.' Slashdot broke the story on this research back in 2003. Apparently the project is still going strong."
It would have been interesting to know the difference in number and type of bugs between a closed source product and an equivalent open source product... let's say the MS Office suite versus OpenOffice.org suite.
The installation of CBI is implicit consent to such monitoring, of course, and I didn't mean to imply that there was no consent involved at all.
However, asking us to read 170-odd pages of your dissertation is a little much. Would it be possible to describe the data collection system, how reports are generated and if the reports are sent automatically or as in the case of Dr. Watson sent with user approval. Also, what types of bugs you found using your statistical methods, as well as what types of bugs you think would be difficult to find using such methods.
A quick comparison to related mainstream debugging techniques would be useful to give us out here in the trenches a firmer grip on the techniques you describe.
And finally, if you wouldn't mind, could you describe a real-world scenario where a generalized product (codename: CBIMax) would be marketable. If such a general product is impossible, is it because each product is different and the methods you describe would need to be revised each time? What is the maximum level of abstraction of these techniques from specific scenarios that is achievable yet still retaining enough so as to not require largescale retooling for each project?
Thanks!
Plenty of users do. There's a great blog posting by Raymond Chen called There's an awful lot of overclocking out there where he talks about investigating some of these "Watson" crashes.
The crashes were impossible - instructions like
Turns out unscrupulous vendors were selling overclocked computers without informing buyers. Pretty cool article.
I know that in one particular http://www.kingdomofloathing.com/game, they tend to follow this approach. Once a new feature is created, and debugged enough so that it's stable and doesn't break anything, the feature is released to the general populace. After all, once all of the important bugs are found, a thousand users will find the minor bugs through general usage faster than a small dedicated team of testers. Also, the time the testers save by not having to verify every single minor detail can be used to work on new material.
Add into the equation that without some elaborate software (such as Mercury LoadRunner, or an open-source equivalent), it's hard to simulate the effect the entire population will have when they start hammering on the server. It can also help track down extremely low-occurance bugs, because with enough people working on it, those one-in-a-million cases will eventually come up.
Kinda reminds me of infinite monkeys eventually producing the works of Shakespeare.
My main issue with the good Doctor is that most of the time, when I had a program that crashed and invoked him, it wasn't a Microsoft product. Typically, it's because I was working on a programming assignment and it 'burped.' So unless Microsoft was willing to help me debug my homework, I didn't see much point in sending the data on to Redmond.
Not that I mind sending back data when it can be useful; if someone is going to look at the error logs, memory, etc., and try and make it so that it won't crash again, I'm all for it. I just pity the poor person who accidentally leaves a major bug in the code, and swamps the system with error reports.
The root word is lever and the basic idea is that you use something under your control to effect control over what would normally be outside your control. Like a very long handle on a pipe wrench.
The money aspect you refer to has to do with debt financing whereby you manage to use your equity to finance something larger than your equity. I don't think the article is referring to corporate finance.
In a perfect world you would use a few people who would recognize and fix the bugs. These people would never talk to the users. They would have no need to and neither would gain from the experience.
In the world that I exist in, users are the ones who spot the bugs, specifically the circumstances under which the bugs exhibit themselves. I use my user's eyes to leverage {user's eyes, my skills}.
If all you mean is "improve", you would not use a word which essentially demands a discrepancy in the metrics between cause and effect.
b. To supplement (money, for example) with leverage.
If you add money to an account because of a margin call, does this increase or decrease your leverage? That is a horrible excuse for a definition.
Automated dump analysis is an old idea in the mainframe world, but almost unknown outside it. The microprocessor world grew up with interactive debuggers and an early user-as-programmer assumption. This hasn't translated well to the modern software world.
In the mainframe world, there have even been mainframes that recorded the last 64 or so branches using dedicated hardware, so that after a crash, the control path could be recovered.
What does the Mozilla project do with the data from their "quality feedback agent", anyway?