Still haven't figured out the financial incentives.:)
If you mean incentives from the users' perspective, I like to pitch it this way. My statistical methods naturally tend to "learn" the most, most quickly, about the failures that happen most often. So the more a user participates, the more the developers' attention will be swayed to the bugs that user cares about. Thus, users can help steer bug triage.
Open source bug trackers can work the same way. When you report bugs in some project's Bugzilla system, you're helping that project see what they need to work on. At the same time, you're being selfish by drawing the developers' attention to your issues. It's a sort of enlightned self-interest.
However, asking us to read 170-odd pages of your dissertation is a little much.
Hey, it's a real page-turner. Well, it has pages and they turn, at least.
The other questions you ask are all good ones, but a bit much to address in a Slashdot comment. Please see the project home page for more information. The "Learn More" page may answer some of your questions, and there are additional drill-down pages from there with even more technical material on selected topics.
Please understand that I don't mean to brush off your insightful questions. They are just questions for which satisfactory answers are hard to give in a sentence or two.
sending the relevant data back to UWisc without informing the user.
Informed participation is a really big deal for me. No user should ever find themselves participating in the Cooperative Bug Isolation Project without their knowledge. Opt-in is explicit and revokable, and if the opt-in system runs into trouble of any kind, the fallback position is no data reporting at all.
The whole thing collapses if users don't trust me. So I've taken every measure I can think of to ensure that they can. Please see the relevant project page for more details about privacy matters.
It sounds like a good idea, but I doubt it is in Liblit's power to fix Windows OS bugs.
Working on it! Check back in with me in a few years... maybe less.:-)
This research has been a wonderful collaborative effort, and many people deserve to share the credit. To quote from part of the Acknowledgements section of my dissertation:
I am indebted to the many members of the open source community who have supported our work. My thanks go out to the many anonymous users of our public deployment, and to the developers of the open source projects used in our public deployment and case studies.
So thanks, Slashdot, for helping me find those users (or helping them find me). The exposure was invaluable. And thanks, open source community, for your participation. I've benefitted greatly from standing on your massed shoulders. This could not have happened without you.
Does this only send back reports when gnumeric crashes or is it sending back reports on a constant basis?
It sends back a report when the application exits. Crashes include some extra crash information, but successful exits upload a report too. (That way we can look for the differences between good runs and bad.) See the web site for more details on exactly what's in those reports.
There is no continuous reporting while the program runs: it's just one blob sent when the program exits.
How much bandwidth does it use? Can I just leave it on knowing it is trivial usage?
We've worked hard to keep the reports small. The median size for Gnumeric reports received thus far is just over 8KB: about one second across a 56K modem. The single biggest Gnumeric report we've seen is still under 16KB: about two seconds across a 56K modem.
I guess they don't want the thousands of gentoo users to participate.
Hey now, don't take it personally. That's not what it's about. It's just a matter of prioritizing our limited resources. I would love to add Gentoo to our list, and in fact foser has expressed an interest in teaming up. We just need to find the time to get all the pieces put together.
We would like to support more distros; it's just a matter of prioritizing our limited resources. If there is a specific alternate distro that you want to see, please let us know. I can't make any promises, but we have every motivation to be responsive to the interests of potential users.
The instrumentation is injected early during compilation, at roughly the source code level. We're looking at things like variables, function returns, branches, etc.: source-level constructs. Thus, the data is not architecture specific.
Of course, if the bug is architecture specific, then identifying the vulnerable architecture is part of what we'd want the system to produce as analysis results.
By design the system learns the most about the most frequent failures. So yes, that probably means that we will find cross-platform bugs before x86-specific bugs, and x86-specific bugs before PPC-specific bugs. From a practical "bug triage" standpoint, that's exactly the behavior you want. If there are more bugs than engineers, you need to pick your priorities.
If you're a raving ignoramus, you're a raving ignoramus who asks good questions.:-)
In our current deployment, we only learn about problems that crash the application. These are important bugs, but they are certainly not the only bugs.
We are considering ways to let users manually report non-crash misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.
Or to put things into academic terms, "FORMAT MY *$^&$* TABLE CORRECTLY" is future work.:-)
In the future we may add the ability for users to manually report non-crash application misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.
But in the current public deployment you are quite right that we only pick up on crash bugs. Consider that the more general project name gives us something to aspire to.:-)
How can they tell which bugs are the culprits driving the users crazy every day?
It's a numbers game. We're looking for statistical trends in large numbers of runs. That means we will learn the most, the most quickly, about the bugs that are happening most often to the most users. Bug triage falls out as a natural consequence of the sparse sampling and statistical modeling approach.
That said, you suggestion about measuring head-into-keyboard smacks isn't half bad. There are some groups here at Berkeley that work on haptic (touch-based) interfaces; perhaps I should pass that suggestion along to them.:-)
Any chance of making the Gnumeric GUI a little less, um, sluggish?
That depends on where the sluggishness is coming from. If the Bug Isolation Project builds are sluggish but standard Gnumeric binaries are not, then that is something I need to look at. Our instrumentation may be taking up more than its fair share of your time. On the other hand, if you see the same problems in regular Gnumeric, then your best option would be to file bug reports or contact the developers directly. They will be able to help you out more directly than I can.
When entering data the toolbars enable/disable with an annoying lag (instantaneous with Excel).
This may be intentional, or at least an intentional temporary hack. See this message from the Gnumeric mailing list archives, where Jody says "We're talking with [the gtk developers] to improve performance here, but in the mean time we've put the desensitisation on a delay to avoid pointless flicker when doing data entry quickly."
And F9-ing a (sub)formula in the formula bar is a big missing item
Sounds like a perfect item to suggest in either a bug report or on the mailing list. The Gnumeric developers (of whom I am not one) are generally pretty responsive to MS Excel feature parity requests.
Trust me, you don't want to see the names we came up with before this one. "Cooperative Bug Isolation Project" may not be catchy or clever, but it could have been a lot worse.
I guess it is on a higher level than a core dump. Perhaps showing the programmer what buttons had been clicked recently and things like that?
It's not higher level so much as continuous instead of only-after-you-crash. The specific behaviors we measure are things like which way branches went, or whether functions returned negative/zero/positive values, or the relative sizes of two variables we guess might be related.
It's very low-level, down at the scale of individual source program statements and variables. And it's sampled randomly throughout program execution, instead of only being observed as a single snapshot in a post-mortem core file.
The short version is that we are looking for differences in program behavior between runs that succeed and runs that crash. If there's something the program only seems to do when it crashes, then that may lead us to the bug.
The approach combines techniques from program analysis and statistical machine learning. The program analysis side addresses how to insert instrumenation that can be sampled in a sparse, random, but fair way with very little overhead. The statistical machine learning side addresses how to find the few instrumentation "needles" in the big "haystack" of noise that are strongly predictive of failure.
The goal is similar to that of Mozilla's Talkback system: find and fix the bugs that happen most often to the most people. In fact we have met with the Mozilla Talkback team on several occasions to brainstorm and exchange perspectives. A key difference is that Talkback can only observe program state at the moment of failure, whereas we collect measurements continuously as the program runs.
That potentially gives us the ability to discover deeper bugs, including things like memory corruption where the point of failure can be distant from the point where the bad behavior really happened. In one of our papers we showed how we can isolate a previously unreported buffer overrun in bc by detecting an unusual loop iteration count correlated with program failure.
More detail that that will not, I'm afraid, fit in a reasonable Slashdot comment. See the project pages, and especially the "learn more" area for more info, including links to papers of varying length and technical depth.
If you want to take Gnumeric 1.2.0 for a spin, consider participating in The Cooperative Bug Isolation Project, a research project being conducted at UC Berkeley. We have prebuilt Red Hat 9 packages of Gnumeric and several other popular applications. These binaries are built with extra feedback instrumentation that lets us understand how the software is working (or failing to work) in the hands of real users.
Even if you have never written a line of code in your life you can help make the software better for everyone simply by using our special bug-hunting feedback packages.
Do you have a neural network that controls the light or the temperature in your home?
Yes, I do. I call it a "brain", and it works great. Some might fault its need for constant maintenance and periodic down time, but it genuinely does an unbeatable job of setting the light and temperature exactly as I want them. It's also nice and expandable, capable of taking on all manner of additional functions not in its original design. I've had my "brain" for a long time now, and although it may not be a speedy as some of the newer models, it's always worked like a champ. I guess you could say I'm kind of attached to it.
I've always thought it would be cute for linkers to be aware of software licenses, and of license incompatibilities. License identification would be embedded directly within object files and libraries, presumably put there by license-aware compilers. If you tried to link GPL-incompatible application code with a GPL-covered library, the linker would report an error.
Yes kids, that's why I'm at Berkeley. And if you patent my idea and make millions off of it, I'll sue you silly.:-)
Slashdot Mirror
If you mean incentives from the users' perspective, I like to pitch it this way. My statistical methods naturally tend to "learn" the most, most quickly, about the failures that happen most often. So the more a user participates, the more the developers' attention will be swayed to the bugs that user cares about. Thus, users can help steer bug triage.
Open source bug trackers can work the same way. When you report bugs in some project's Bugzilla system, you're helping that project see what they need to work on. At the same time, you're being selfish by drawing the developers' attention to your issues. It's a sort of enlightned self-interest.
Liblit. There's an off-by-one-error in your count of L's. (It doesn't matter what they say about you, as long as they spell your name right.)
Yeah, that's me along with a truly fantastic team of collaborators. And there's more where that came from.
(Hi Calyxa!)
Hey, it's a real page-turner. Well, it has pages and they turn, at least.
The other questions you ask are all good ones, but a bit much to address in a Slashdot comment. Please see the project home page for more information. The "Learn More" page may answer some of your questions, and there are additional drill-down pages from there with even more technical material on selected topics.
Please understand that I don't mean to brush off your insightful questions. They are just questions for which satisfactory answers are hard to give in a sentence or two.
Yes, exactly. The users are beta testers; we may as well admit it. I want to make them better beta testers. :-)
Informed participation is a really big deal for me. No user should ever find themselves participating in the Cooperative Bug Isolation Project without their knowledge. Opt-in is explicit and revokable, and if the opt-in system runs into trouble of any kind, the fallback position is no data reporting at all.
The whole thing collapses if users don't trust me. So I've taken every measure I can think of to ensure that they can. Please see the relevant project page for more details about privacy matters.
Working on it! Check back in with me in a few years ... maybe less. :-)
This research has been a wonderful collaborative effort, and many people deserve to share the credit. To quote from part of the Acknowledgements section of my dissertation:
So thanks, Slashdot, for helping me find those users (or helping them find me). The exposure was invaluable. And thanks, open source community, for your participation. I've benefitted greatly from standing on your massed shoulders. This could not have happened without you.
My keyboard back.
Look, I said I was sorry!
As a proud leftie, I am deeply offended by your use of the word "gauche."
No, not really. You have the right to use "gauche" if you want to.
Wait, now I'm deeply offended by my own use of the word "right"! Why can't you have the left to use "gauche"?
Aagh!
All good questions, Harry8.
It sends back a report when the application exits. Crashes include some extra crash information, but successful exits upload a report too. (That way we can look for the differences between good runs and bad.) See the web site for more details on exactly what's in those reports.
There is no continuous reporting while the program runs: it's just one blob sent when the program exits.
We've worked hard to keep the reports small. The median size for Gnumeric reports received thus far is just over 8KB: about one second across a 56K modem. The single biggest Gnumeric report we've seen is still under 16KB: about two seconds across a 56K modem.
Hey now, don't take it personally. That's not what it's about. It's just a matter of prioritizing our limited resources. I would love to add Gentoo to our list, and in fact foser has expressed an interest in teaming up. We just need to find the time to get all the pieces put together.
We would like to support more distros; it's just a matter of prioritizing our limited resources. If there is a specific alternate distro that you want to see, please let us know. I can't make any promises, but we have every motivation to be responsive to the interests of potential users.
The instrumentation is injected early during compilation, at roughly the source code level. We're looking at things like variables, function returns, branches, etc.: source-level constructs. Thus, the data is not architecture specific.
Of course, if the bug is architecture specific, then identifying the vulnerable architecture is part of what we'd want the system to produce as analysis results.
By design the system learns the most about the most frequent failures. So yes, that probably means that we will find cross-platform bugs before x86-specific bugs, and x86-specific bugs before PPC-specific bugs. From a practical "bug triage" standpoint, that's exactly the behavior you want. If there are more bugs than engineers, you need to pick your priorities.
If you're a raving ignoramus, you're a raving ignoramus who asks good questions. :-)
:-)
In our current deployment, we only learn about problems that crash the application. These are important bugs, but they are certainly not the only bugs.
We are considering ways to let users manually report non-crash misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.
Or to put things into academic terms, "FORMAT MY *$^&$* TABLE CORRECTLY" is future work.
In the future we may add the ability for users to manually report non-crash application misbehavior. I know that this is something that The GIMP's developers are very interested in, and we are already doing controlled experiments along similar lines. The underlying statistical debugging techniques still apply.
:-)
But in the current public deployment you are quite right that we only pick up on crash bugs. Consider that the more general project name gives us something to aspire to.
cybermace5 asked:
It's a numbers game. We're looking for statistical trends in large numbers of runs. That means we will learn the most, the most quickly, about the bugs that are happening most often to the most users. Bug triage falls out as a natural consequence of the sparse sampling and statistical modeling approach.
That said, you suggestion about measuring head-into-keyboard smacks isn't half bad. There are some groups here at Berkeley that work on haptic (touch-based) interfaces; perhaps I should pass that suggestion along to them. :-)
That depends on where the sluggishness is coming from. If the Bug Isolation Project builds are sluggish but standard Gnumeric binaries are not, then that is something I need to look at. Our instrumentation may be taking up more than its fair share of your time. On the other hand, if you see the same problems in regular Gnumeric, then your best option would be to file bug reports or contact the developers directly. They will be able to help you out more directly than I can.
This may be intentional, or at least an intentional temporary hack. See this message from the Gnumeric mailing list archives, where Jody says "We're talking with [the gtk developers] to improve performance here, but in the mean time we've put the desensitisation on a delay to avoid pointless flicker when doing data entry quickly."Sounds like a perfect item to suggest in either a bug report or on the mailing list. The Gnumeric developers (of whom I am not one) are generally pretty responsive to MS Excel feature parity requests.
Har har. :-)
Trust me, you don't want to see the names we came up with before this one. "Cooperative Bug Isolation Project" may not be catchy or clever, but it could have been a lot worse.
On the other hand, do we have a really spiffy logo.
It's not higher level so much as continuous instead of only-after-you-crash. The specific behaviors we measure are things like which way branches went, or whether functions returned negative/zero/positive values, or the relative sizes of two variables we guess might be related.
It's very low-level, down at the scale of individual source program statements and variables. And it's sampled randomly throughout program execution, instead of only being observed as a single snapshot in a post-mortem core file.
See the techie details on instrumentation schemes page for more information on the kind of measurements we take.
The short version is that we are looking for differences in program behavior between runs that succeed and runs that crash. If there's something the program only seems to do when it crashes, then that may lead us to the bug.
The approach combines techniques from program analysis and statistical machine learning. The program analysis side addresses how to insert instrumenation that can be sampled in a sparse, random, but fair way with very little overhead. The statistical machine learning side addresses how to find the few instrumentation "needles" in the big "haystack" of noise that are strongly predictive of failure.
The goal is similar to that of Mozilla's Talkback system: find and fix the bugs that happen most often to the most people. In fact we have met with the Mozilla Talkback team on several occasions to brainstorm and exchange perspectives. A key difference is that Talkback can only observe program state at the moment of failure, whereas we collect measurements continuously as the program runs.
That potentially gives us the ability to discover deeper bugs, including things like memory corruption where the point of failure can be distant from the point where the bad behavior really happened. In one of our papers we showed how we can isolate a previously unreported buffer overrun in bc by detecting an unusual loop iteration count correlated with program failure.
More detail that that will not, I'm afraid, fit in a reasonable Slashdot comment. See the project pages, and especially the "learn more" area for more info, including links to papers of varying length and technical depth.
If you want to take Gnumeric 1.2.0 for a spin, consider participating in The Cooperative Bug Isolation Project, a research project being conducted at UC Berkeley. We have prebuilt Red Hat 9 packages of Gnumeric and several other popular applications. These binaries are built with extra feedback instrumentation that lets us understand how the software is working (or failing to work) in the hands of real users.
Even if you have never written a line of code in your life you can help make the software better for everyone simply by using our special bug-hunting feedback packages.
Read more about it or download and install today!
When my internal-combusion engine is not working quite right, it runs noisy or pings. A malfunctioning electro-pneumatic car would, what, fart?
I'm sorry. I refuse to drive a farting car. It's as simple as that.
Yes, I do. I call it a "brain", and it works great. Some might fault its need for constant maintenance and periodic down time, but it genuinely does an unbeatable job of setting the light and temperature exactly as I want them. It's also nice and expandable, capable of taking on all manner of additional functions not in its original design. I've had my "brain" for a long time now, and although it may not be a speedy as some of the newer models, it's always worked like a champ. I guess you could say I'm kind of attached to it.
cat /dev/mouse > /dev/audio
This doesn't work so well under Linux, actually, though it was really funky under SunOS 4 (back in the pre-Solaris days).
I've always thought it would be cute for linkers to be aware of software licenses, and of license incompatibilities. License identification would be embedded directly within object files and libraries, presumably put there by license-aware compilers. If you tried to link GPL-incompatible application code with a GPL-covered library, the linker would report an error.
Yes kids, that's why I'm at Berkeley. And if you patent my idea and make millions off of it, I'll sue you silly. :-)