Our sandbox crashed around 4AM, while all maintainers were sleeping. Thus, all 45 later submissions got these "internal errors". I have restarted Igor to recompute all failed submissions, including the one above; just try again later.
I'm sorry for the inconvenience, and have made sure this problem won't happen again. But remember that this is a research project, and we can't sit besides the machine 24/7.
I totally agree with your observation. But the program doesn't have to crash for Igor to work. If the program simply outputs, say, "fail" instead of "pass" (as in the working run), AskIgor will narrow down the cause-effect chain that leads to the "fail".
Your two runs must result in a different output or a different exit status. AskIgor will determine the cause-effect chain that leads to this difference.
Igor initially guesses which part of the difference constitutes "pass" vs. "fail"; this guess is adequate in almost all cases. If you want to be 100% sure, make your program (or a wrapper) output just "pass" or "fail".
I have updated the usage guide accordingly. Thanks!
"99% of the effort in my experience has been figuring out what causes something to fail, not finding the failure itself."
That's what delta debugging is about - finding failure causes automatically. These causes can be in the input, in the configuration, in the program changes - and eventually in the program state, as determined by AskIgor.
We're talking about two different issues here - reproducing the bug and actually understanding how it came to be. Most bugs are easy to reproduce and easy to understand, just by re-invoking them and looking at the stack trace.
Some bugs, though, are hard to reproduce, which means that you should set up appropriate isolation and monitoring layers in your program. The recent issue of Communications of the ACM, for instance, has a nice article on record/replay of program executions, which solves all problems of reproducing runs - in principle, that is, because such tools are not readily available yet.
Now there's some bugs that are hard to understand, especially those where there's a long delay between the defect being executed and the experienced failure. Think about a compiler that generates bad code. This is not an issue of reproduction; nor can the problem be solved by examining the stack trace. You have to work your way backwards all through the execution to understand the cause-effect chain. This is where AskIgor can really help you.
And, yeah, I got 12 years of C/C++ programming and debugging experience, too:-)
I just removed the Google redirection. It turns out our server still easily handles the/. load while running the Igor daemons and its Amanda backup at the same time. I can happily sleep now.
The core technology behind Igor is
delta debugging, a general method to determine the difference between a working and a failing scenario.
While not applying AskIgor (being a huge mix of Python, PHP, and SQL) on itself, we have applied delta debugging on various parts of AskIgor to detect failure-inducing code changes, and especially failure-inducing GDB commands.
We're currently working on Eclipse plugins written in Java, and working on Java - and these will work 100% on themselves.
We're running AskIgor as a Web service (instead of a standalone program) for two reasons:
We get a set of programs for regression testing.
We get some ratings about the quality of our work.
Although we did our best to give AskIgor a decent interface, it's still a research prototype, and there's many possible ways to go. Your submissions help us making these decisions, and guide our future development.
Most people coming from/. just have a look at the main site. By redirecting, we reduce traffic by about 50% and Apache load by 75%. If you want to go beyond (and maybe submit some buggy program), you're welcome!
Slashdot Mirror
I'm sorry for the inconvenience, and have made sure this problem won't happen again. But remember that this is a research project, and we can't sit besides the machine 24/7.
I totally agree with your observation. But the program doesn't have to crash for Igor to work. If the program simply outputs, say, "fail" instead of "pass" (as in the working run), AskIgor will narrow down the cause-effect chain that leads to the "fail".
Igor initially guesses which part of the difference constitutes "pass" vs. "fail"; this guess is adequate in almost all cases. If you want to be 100% sure, make your program (or a wrapper) output just "pass" or "fail".
I have updated the usage guide accordingly. Thanks!
That's what delta debugging is about - finding failure causes automatically. These causes can be in the input, in the configuration, in the program changes - and eventually in the program state, as determined by AskIgor.
We are well-aware that this is a conflict, and we're currently working on this. Sorry, no further comment on this matter.
Some bugs, though, are hard to reproduce, which means that you should set up appropriate isolation and monitoring layers in your program. The recent issue of Communications of the ACM, for instance, has a nice article on record/replay of program executions, which solves all problems of reproducing runs - in principle, that is, because such tools are not readily available yet.
Now there's some bugs that are hard to understand, especially those where there's a long delay between the defect being executed and the experienced failure. Think about a compiler that generates bad code. This is not an issue of reproduction; nor can the problem be solved by examining the stack trace. You have to work your way backwards all through the execution to understand the cause-effect chain. This is where AskIgor can really help you.
And, yeah, I got 12 years of C/C++ programming and debugging experience, too :-)
I just removed the Google redirection. It turns out our server still easily handles the /. load while running the Igor daemons and its Amanda backup at the same time. I can happily sleep now.
While not applying AskIgor (being a huge mix of Python, PHP, and SQL) on itself, we have applied delta debugging on various parts of AskIgor to detect failure-inducing code changes, and especially failure-inducing GDB commands.
We're currently working on Eclipse plugins written in Java, and working on Java - and these will work 100% on themselves.
We're redirecting only if you come from /.
- We get a set of programs for regression testing.
- We get some ratings about the quality of our work.
Although we did our best to give AskIgor a decent interface, it's still a research prototype, and there's many possible ways to go. Your submissions help us making these decisions, and guide our future development.Most people coming from /. just have a look at the main site. By redirecting, we reduce traffic by about 50% and Apache load by 75%. If you want to go beyond (and maybe submit some buggy program), you're welcome!
No problem. Igor runs your program in a sandbox whose contents get restored with each new submission.