Slashdot Mirror


Mountain Biking Helps Squash Bugs

Dr.Milius writes "Henning Brauer of the OpenBSD project recently made an interesting post to the openbsd-tech mailing list about how a mountain bike ride helped him relate two baffling bugs in their new BGP and NTP daemons. It turns out they were both off-by-one errors that were easy to fix but notoriously difficult to spot. Always great when the experts show us how it's done."

82 comments

  1. RTFB by Anonymous Coward · · Score: 5, Funny

    Ride the fucking bicycle.

    1. Re:RTFB by Tagren · · Score: 0

      Innovation.

  2. works for anything by Anonymous Coward · · Score: 3, Insightful

    if you can't figure it out, stop trying so damn hard and just do something else.

    1. Re:works for anything by nusratt · · Score: 1

      "if you can't figure it out, stop trying so damn hard and just do something else"

      Ditto.
      This is news?
      EVERY half-decent dev or tech has these kinds of down-time epiphanies.
      (For me, it's the shower and commuting.)

    2. Re:works for anything by GoksinAkdeniz · · Score: 1

      I do agree! Better step aside and rest than drowning a sea of problems! :)

  3. road bike.. by MoOsEb0y · · Score: 5, Funny

    I wonder what effect a road bike would have upon bugs? I mean, one could go faster in the flats where bugs are likely to reside, but at the same time, wider tires would allow one to run over more ants. Oh, the dillemmas!

    1. Re:road bike.. by JMS-Web · · Score: 1

      wider tires would allow one to run over more ants. Oh, the dillemmas! And running over ANT is good? I thought it was all about the Bugs.

      --
      Fave site: www.PatriotsInsider.com
  4. This is news? by agent+dero · · Score: 4, Informative

    This is something well known to work. It's not the bike riding, it's the act of "taking a break from the problem." Think back to the origins of "Eureka!"

    I, for example, will often go grab a Coke, talk to people, etc, and somewhere along the line, by _not_ focussing so hard on the problem, I come up with the answer.

    --
    Error 407 - No creative sig found
    1. Re:This is news? by ezzzD55J · · Score: 1, Flamebait
      Indeed.. Experts? Show us how it's done? The guy may very well be a guru, but this story isn't evidence of it.. Although it's always easy to comment from the sidelines (and with 20/20 hindsight), the guy says it himself:
      when maintaining counters for list/queue/... entries, don't fuck with either the counter or the list directly anywhere; use wrapper functions that take care for both (not using a counter/list pair is not an option in many, including these two, cases). Not that this is really news, but very well worth recalling...
      No kidding it's not news not to fuck with abstractions, that's the fucking point!
    2. Re:This is news? by Anonymous Coward · · Score: 4, Insightful

      Many of the problems I deal with, I get that Eureka moment while on the toilet.

    3. Re:This is news? by bhima · · Score: 1

      I find that I have to get much further from the problem, in fact it happens to me so often that I hike with an ultra-portable, just in case.

      --
      Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
    4. Re:This is news? by c13v3rm0nk3y · · Score: 4, Funny

      It's not the bike riding, it's the act of "taking a break from the problem."

      I'll second that. I do some of my best coding in the bathtub, sans the dangerous electronics, of course.

      --
      -- clvrmnky
    5. Re:This is news? by ezzzD55J · · Score: 0, Flamebait

      Flamebait? Overrated? May the metamods roast you moderators for this.

    6. Re:This is news? by bigbadbuccidaddy · · Score: 1

      That is just because those are the most common breaks you take from work. I work really well when I'm hungover and have to take more frequent bathroom breaks.

  5. not difficult to spot at all by jeif1k · · Score: 5, Insightful

    It is ironic that people, on the one hand, fool themselves into thinking that these things don't happen to them, and on the other hand, have to go off bug hunting for days to find "baffling bugs". How much more productively could that development time have been spent?

    Face it, memory corruption bugs, off-by-one-errors, and all that, happen to even the most experienced and careful programmers. The way to deal with them is to use programming languages that detect them reliably. In different words, we need to retire C or fix C.

    And, no, "retiring C" doesn't mean switching to Java or C#. Instead, it means switching to a systems programming language that has error checking by default but still gives you all the low-level features you need when you need them. There have been a number of such languages over the years, but, unfortunately, they were all killed by C, not because C was better, but because C shipped with UNIX.

    1. Re:not difficult to spot at all by Homology · · Score: 4, Informative
      The OpenBSD developers does not fool themselves into thinking that they don't make mistakes. Several of the techniques they use, like privilege revocation and privilege separation is to lessen the impact of programming mistakes, including their own. Theo de Raadt recently gave a talk on Exploit Mitigation Techniques

      As for not using C, I've read that Theo de Raadt likes the compiler and language that is used in Plan 9. Can't use it due to license problems, though.

    2. Re:not difficult to spot at all by jeif1k · · Score: 1

      As for not using C, I've read that Theo de Raadt likes the compiler and language that is used in Plan 9. Can't use it due to license problems, though.

      You mean Alef? I don't think there are licensing problems with the language, and writing a compiler for it wouldn't be hard. I'm not convinced, though, that Alef is a good systems programming language.

      But safer systems programming languages are a dime a dozen, people just have to use them.

    3. Re:not difficult to spot at all by Homology · · Score: 4, Informative

      Most likely the C-dialect compiler that Plan 9 uses. But the AT&T license is too restrictive for Plan 9 code to be imported into BSD, as far as I know.

    4. Re:not difficult to spot at all by ufnoise · · Score: 3, Informative
      Face it, memory corruption bugs, off-by-one-errors, and all that, happen to even the most experienced and careful programmers. The way to deal with them is to use programming languages that detect them reliably. In different words, we need to retire C or fix C.


      There are debugging tools out there which reliably find these bugs in C/C++ and FORTRAN. For example:
      purify (not free)
      valgrind (free)

      Tools such as these help to find problems, while avoiding the inefficiency of array bounds checking on each access.

    5. Re:not difficult to spot at all by Xenophon+Fenderson, · · Score: 5, Insightful

      How is bounds checking inefficient if it protects you from nasty off-by-one errors and evil buffer overflows? Or are you one of those short-sighted programmers who believes cheap CPU clock time is more important than very expensive human time (i.e. time wasted dealing with crappy programming languages and debugging stupid bounds problems, or worse, money lost because a critical bit of information was destroyed, altered, or disclosed due to a buffer overflow vulnerability)?

      And yes, those were both rhetorical questions. I am sick and tired of crappy, bug-ridden software that doesn't do proper input and bounds checking. As far as I'm concerned, it's the programmers fault, either because she didn't write the bounds checks in manually or because he's not using a modern programming language (where "modern" == "not a glorified assembler"). You all aren't Mel, who can write practically perfect programs while sleepwalking, so don't give me a load of crap about how bloated your program would be if it was actually written properly. As a computer security guy, I am sick and tired of cleaning up after the exploits you all apparently work so hard to code into your software.

      </rant>

      --
      I'm proud of my Northern Tibetian Heritage
    6. Re:not difficult to spot at all by Anonymous Coward · · Score: 0

      You can run all your programs under Valgrind if it concerns you that much. I can't even recall the last time I've had a memory leak/overrun, thanks to Valgrind and strlcat/cpy().

    7. Re:not difficult to spot at all by ufnoise · · Score: 1

      You are right. It is the programmer's fault. But it is also the programmer's fault if the program is so inefficient, it is useless. You would then be complaining about how slow the program is, rather than about the exploit. I don't know who Mel is, but I assume he has debugging tools which would allow him to diagnose the errors found in this article. In addition, there is no reason array bounds checking could not be performed in debug versions of the code, but left out of the final optimized version. As far as all this stuff about security, it did not appear that the off by 1 error would have resulted in a security exploit, but a crash. The security exploits you worry about have been reduced with the advent of additional functions in the C standard library which make sure that strings cannot overrun their buffer.

    8. Re:not difficult to spot at all by Profound · · Score: 2, Informative

      You can use use C++ STL vectors with at() when you want bounds checking or use a direct index when you don't. Thus you can keep the backwards compatability with C and have the choice of sacrificing safety for speed when you need it.

    9. Re:not difficult to spot at all by setagllib · · Score: 2, Informative

      Mel: http://www.catb.org/jargon/html/story-of-mel.html

      No debugging tools there, if any tools at all.

      --
      Sam ty sig.
    10. Re:not difficult to spot at all by tedu · · Score: 4, Insightful

      have you used purify/valgrind? as far as "avoiding the inefficiency of array bounds checking on each access" they pretty much suck. performance is nowhere close to what could be considered "production" level.

    11. Re:not difficult to spot at all by tedu · · Score: 4, Insightful

      no amount of "error checking" would have helped. this wasn't a buffer overflow off by one. it was ref count.

      yeah, garbage collection would help, but C with GC is not a "systems programming language with error checking".

    12. Re:not difficult to spot at all by ufnoise · · Score: 2, Informative

      Yes I have. I only use them when I am debugging. Then I don't need to implement array bounds checking on the "production" version of the code.

    13. Re:not difficult to spot at all by jeif1k · · Score: 2, Interesting

      The C dialect in Plan 9 is not very different from ANSI C; it doesn't address safety or security issues. Alef, on the other hand, looks pretty similar to C, but I believe is considerably safer than C.

    14. Re:not difficult to spot at all by jeif1k · · Score: 1

      There are debugging tools out there which reliably find these bugs in C/C++ and FORTRAN.

      No, they don't find them "reliably", they only find them if you happen to have a test case that triggers the bug and you run the software under valgrind or purify when you execute the test case. For security problems arising from buffer overflows, the probably is usually that developers don't even think of the case.

      Tools such as these help to find problems, while avoiding the inefficiency of array bounds checking on each access.

      The "inefficiency" of array bounds checking is a few percent in real code, hardly a big cost. But it's wrong to look at that cost by itself anyway--the question is: if you don't have array bounds checks in the compiler, what are you going to do instead? And pretty much everything you are going to do instead is going to be much more costly. Using languages in which unsafe array access are the default is penny wise and pound foolish.

      And in really performance critical code, you can selectively turn it off in most languages.

    15. Re:not difficult to spot at all by Anonymous Coward · · Score: 0

      You can use use C++ STL vectors with at() when you want bounds checking or use a direct index when you don't. Thus you can keep the backwards compatability with C and have the choice of sacrificing safety for speed when you need it.

      That's the ticket. C++ offers (almost) everything that C does, plus some high-level stuff that keeps you from having to write "glorified assembler" all the time, so let's start using it!

    16. Re:not difficult to spot at all by gokeln · · Score: 1

      The biggest problem I have with the STL is that many debuggers don't have nice support for it. Thus, it's very difficult to observe what's stored in that nice vector or tree, when you hit the breakpoint. A few newer ones are adding automatic display of STL structures, which will be very nice, but for now, I'm stuck with VC++6, which is horrendous WRT these kinds of data structures.

      I'd certainly welcome any ideas people have on how they actually debug code using the STL.

      --

      There's no time to stop for gas, we're already late.
    17. Re:not difficult to spot at all by chaos_echo · · Score: 1
      ... it means switching to a systems programming language that has error checking by default but still gives you all the low-level features you need when you need them. There have been a number of such languages over the years, but, unfortunately, they were all killed by C, not because C was better, but because C shipped with UNIX.

      I don't think you understand the "low-level" appeal of C, error checking and other useful security features like strong typing are not low-level. The ability to break abstractions and treat memory as memory instead of data is both the cause of C's problems and the force behind C's use. The problem isn't C being included in Unix (dozens of languages are included in most modern Unix environments), the problem is that C is genuinely useful for the same reasons that it is hard to secure.

    18. Re:not difficult to spot at all by jeif1k · · Score: 1

      Same problem: C is a language that makes it difficult to write correct code, for numerous reasons.

      A systems programming language with error checking needs to handle both automatic storage management and array bounds checking (plus a few other compile-time and runtime tests that are missing from C). Users should also be able to turn off either/both selectively and explicitly.

    19. Re:not difficult to spot at all by jeif1k · · Score: 2

      Yes, if you use C++ with a decent class library and good programmers, you get good runtime safety. The problem with C++ is that if you have a large project with lots of ex-C programmers, they sneak in unsafe code everywhere and usually don't even understand that they are doing so.

      Nevertheless, it would probably be a big step forward if Linux and BSD allowed C++ code in the kernel.

    20. Re:not difficult to spot at all by Mr.+Slippery · · Score: 1
      The way to deal with them is to use programming languages that detect them reliably. In different words, we need to retire C or fix C.

      The best check for such errors is other eyeballs.

      It's not the language. It's development procedures. Code reviews will catch these things, and other bugs besides (as well as making people write neater, more easily understandable code, and giving coders and opportunity to learn from each other), but reviews take development time - never mind all the time they save in debugging and maintenance, management cares only about that ship date.

      --
      Tom Swiss | the infamous tms | my blog
      You cannot wash away blood with blood
    21. Re:not difficult to spot at all by Anonymous Coward · · Score: 0

      Valgrind won't spot all overruns/underruns -- only those on the heap, not those on the stack or in static memory.

    22. Re:not difficult to spot at all by insulanus_hailstorm · · Score: 1
      have you used purify/valgrind? as far as "avoiding the inefficiency of array bounds checking on each access" they pretty much suck. performance is nowhere close to what could be considered "production" level.

      This is only because the C runtime does not help in this regard. This can be done very efficiently in other environments.

  6. Not new... by HawkingMattress · · Score: 2, Insightful

    Every developper knows that...
    It happened to each of us, no need to be an expert or something: the bug drives you nuts, and you end up saying "fsck that" and go out, totally out of ideas. Then you smoke your cigarette of whatever, or go for a walk. And because you've sort of given up, you start to look at the problem in a different way, probably also because you're not in front of your screen and your brain is more "free", you realize that there are some obvious things that you didn't checked... And 5 minuts later, there is no more bug. That's why i go out to smoke a cig every hour (well not really, but still, it's a nice excuse ;)

    Besides, if you look at the well known "eureka" moments, it seems they all happened when the person was relaxing. Maybe we should be forced to relax each time we're facing an intellectual difficulty, really...

    1. Re:Not new... by lphuberdeau · · Score: 1

      I guess everyone has it's own way of taking... fresh air. Depending on the time of the day, I do different things.

      If it's the second half of the afternoon,
      I just go back home.

      During the evening,
      Off for a beer.

      Any other moment,
      Walk, coffee or walk to coffee...

      And yes, it does work. I usually find my bugs within 10 minutes after sitting back to my desk, which obviously includes reading slashdot.

      --
      Qui ne va pas à la chasse n'a pas de gibier
      PHP Queb
    2. Re:Not new... by mattgorle · · Score: 1

      Very well, I order you to relax!

      --
      Slackware user since 1997.
    3. Re:Not new... by vrt3 · · Score: 1

      Another thing that helps is explaining the problem to someone else. It also makes you look at the bug from another point of view, and very often you find the solution without the other person even saying a word.

      --
      This sig under construction. Please check back later.
    4. Re:Not new... by bondjamesbond · · Score: 2, Funny

      ...and having sex works well - if you can get it, that is.

    5. Re:Not new... by Gauchito · · Score: 1

      Something that I have found to be very, very useful is describing the problem to somebody else. Just having to reorder your thoughts about the issue in order to get the message across puts everything in a different light. Plus, you consider things that your brain usually just skips over without a second thought. I can't count how many times I've been telling a coworker about a problem and halfway through the explanation it hits me ("Well, then when we're iterating through the list the code... Oh, crap, found it.").

    6. Re:Not new... by fgb · · Score: 1

      Usually, if I'm working on a problem late at night, I find it more productive to just go to sleep. You wouldn't believe the number of times I've woken up in the morning with the solution being blindingly obvious to me.

    7. Re:Not new... by HawkingMattress · · Score: 1

      Yes me too. The only problem is that even if i know this, usually when i'm coding at night i'm "zoning", in a state which pushes me to go on. So i'll search a solution for several hours generally for nothing, and i'll discover in the morning that I was doing stupid obscure things while the solution was indeed obvious...
      I'm now trying to restrain myself from doing this, as sometimes you can furthermore do stupid things which will cause you to spend even more time later on, solving problems which should have never been here. Never do too much of one thing, they say... so true.

  7. Yup. by Anonymous Coward · · Score: 0

    Sometimes cracking your head open works, sometimes a quick ice cream run works, and sometimes a 6 month (year) hiatus works. Either way, it eventually works.

  8. taking a crap by endx7 · · Score: 4, Funny

    Hmm. Well, I did figure out how I wanted to implement a portion of my code when I was taking a crap recently.

    1. Re:taking a crap by r_j_howell · · Score: 4, Funny

      I had a good friend at school remark to me that ALL his best programming ideas came to him in the bathroom. Mine generaly don't. But you couldn't tell that from looking at some of the stuff I've churned out.

    2. Re:taking a crap by Anonymous Coward · · Score: 0

      core dump, anyone?

    3. Re:taking a crap by Anonymous Coward · · Score: 0

      As opposed to "memory" leak?

  9. Bake Cookies! by maskedbishounen · · Score: 4, Interesting

    On the food aspect, whenever I run into coding problems, I go bake cookies. Helps if you have a near by oven and supply of dough, of course, but it gives you a good 30 minutes to stop and focus on something else. I usually end up pacing around while they bake, looking out the windows, looking around the kitchen, and more oft than not have a viable solution in addition to some fresh cookies when I go back to coding.

    --
    "An infinite number of monkeys typing into GNU emacs would never make a good program."
    1. Re:Bake Cookies! by drinkypoo · · Score: 1

      This also provides the side benefit that you can tell how good someone's code is the first time by how fat they are...

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  10. *yawn* by Estanislao+Mart�nez · · Score: 2, Insightful

    Extremely commonplace sort of phenomenon. You work on some problem really hard, then at some point where you're not working on it, the solution comes in a flash. Happened to me last week with a mysterious bug.

  11. Would a more modern language madea difference? by CaptainPinko · · Score: 1
    It turns out they were both off-by-one errors that were easy to fix but notoriously difficult to spot

    I was wondering if this is the kind of thing that most likely would ahve been avoided in a more "modern" language, like pointer arithmetic problems could be avoided by not using C? Not a trool -really- I was just wondering. I RTFA but can't tell from reading it...

    --
    Your CPU is not doing anything else, at least do something.
    1. Re:Would a more modern language madea difference? by Anonymous Coward · · Score: 1, Insightful

      It was not a "pointer arithmetic" problem, it was a counter problem. You don't even need a computer for that to happen, try counting the runners passing the finish line at a marathon... Miss one, and you'll be waiting forever for the last guy to come in.

    2. Re:Would a more modern language madea difference? by NickHolland · · Score: 1, Insightful

      Curiously, the people who know OSs the best have been sticking with C. The people talking about "modern languages" have been mostly doing just that: talking. The fact that nothing has been done beyond "talk" says more then all the talk.

      Someone who believes this can be done has to prove it by reimplementing an existing OS in one of these "modern languages", and demonstrate that it is maintainable, performs sufficiently, and actually works better than a C implemented OS. Implementing a "whole new" OS isn't going to work, as the barriers to entry in this business are just too high -- you would have to implement an entire software library as well. And note: having to have a multi-processor P4 or AMD64 to do what is currently done with a Pentium 90 isn't going to fly...

      I really doubt any programming language can make programing "intrinsically safe". In fact, if past experience is any guide, it would probably lower the bar to entry into OS programming, and probably end up with a net REDUCTION in quality. All it takes to prove me wrong is ...well, do it. :)

    3. Re:Would a more modern language madea difference? by Ashtead · · Score: 1

      Off-by-one errors are possible in almost every language. Wherever there is some kind of list or set of objects, however expressed, there is the potential for such an error. The actual implementation language doesn't matter here.

      --
      SIGBUS @ NO-07.308
    4. Re:Would a more modern language madea difference? by norkakn · · Score: 1

      System programming is done with C because it is a pretty lcean abstraction away from the hardware. Assembly is the cleanest, but C maps pretty well onto assembly that maps pretty well onto what the processor actaully implements.

      This is important because the programmer has some idea of what the program is actaully doing, and a good programmer has a great idea of what the program is actually doing.

      Anything that is pretty close to the hardware should be done in C or soemthing similar. Computers have been fucked up enough by people who don't actually realize how computers work and just right code that works rather than being sleek or taking advantage of the tasks that computers are good at.

      As a hardware person, any time someone talks about using higher level languages for system programming, it makes me want to go postal.

  12. hard-to-find bugs are often the easiest to fix by cpeterso · · Score: 2, Interesting


    I often find that the bugs that are most difficult to find are the easiest to fix. They are often some tiny corner case in one line of code that someone never thought of.

    In the last product I worked on, we had a killer crash bug that different developers spent WEEKS investigating it, giving up, and then "hot potatoing" the bug to another developer. About two months later, I finally fixed the bug. A BSTR allocated using SysAllocString() should have been freed using SysFreeString(), but it was being "freed" using COM's CoTaskMemFree(). This would corrupt COM's heap causing random COM crashes in unrelated code much later! :(

    1. Re:hard-to-find bugs are often the easiest to fix by Detritus · · Score: 2, Interesting

      This class of bug, and some others, can be prevented if the free routine checks the to-be-freed memory chunk for validity. I've done this in some embedded systems by setting a hidden magic number in each memory chunk that is allocated. The free routine checks for the magic number before it touches it. If the magic number isn't there, it generates an alarm message and returns an error code to the caller. Unique magic numbers can be used for each class of buffer, and to indicate whether the buffer is free or in-use.

      --
      Mea navis aericumbens anguillis abundat
    2. Re:hard-to-find bugs are often the easiest to fix by darweidu · · Score: 1

      If you're on windows using MS tools, you can get your compiler to do this automatically. Why reinvent the wheel?

      http://msdn.microsoft.com/library/default.asp?url= /library/en-us/vsdebug/html/vchowRuntimeChecksAPI. asp Only in debug mode, of course.

    3. Re:hard-to-find bugs are often the easiest to fix by Detritus · · Score: 1
      Not everyone is using MS tools.

      Only in debug mode, of course.

      I've never understood this philosophy. Let's put lots of error checks and consistency checks into the system, then rip them all out when we ship the software or transfer it to operations.

      If it doesn't have to be correct, we can make it run as fast as you like.

      --
      Mea navis aericumbens anguillis abundat
  13. Valgrind and/or GCC boundschecking by Bluelive · · Score: 1

    The tools are there, why not use them. Or just use java.

  14. Re:I found the first reply most revealing by Anonymous Coward · · Score: 0

    Hey, that IS the first reply. Now that is funny!

  15. Yes! by agentk · · Score: 2, Informative
    First, when maintaining counters for list/queue/... entries, don't fuck with either the counter or the list directly anywhere; use wrapper functions that take care for both (not using a counter/list pair is not an option in many, including these two, cases).

    Yes! This can make tweaking your overall algorithm or approach so much easier as well, if these wrapper functions are there, and well defined in their actions. You can never have *too short* a function!

    --

    VOS/Interreality project: www.interreality.org

  16. Just taking a break is helpful, but ... by Dr.+Mu · · Score: 1

    The most productive breaks for me are those that force oxygenated blood to my brain, e.g. running, hiking, biking, etc. Of those, the best are the ones that don't require concentration but allow the mind to wander freely.

  17. Hummm.... by utlemming · · Score: 2, Funny

    I guess that means that there is actually a Bikeshed. Who would have figured? And I thought that it was just proverbial...

    --
    The views expressed are mine own and do not express the views of my employer.
  18. really stupid by BillyBurrito · · Score: 0, Troll

    Can we get original?

    1. Re:really stupid by sgant · · Score: 2, Funny

      sheesh, lighten up...

      --

      "Leo Fender was in a 'state of grace' when he designed the Stratocaster." -- Paul Reed Smith
    2. Re:really stupid by BillyBurrito · · Score: 2, Insightful

      ok

  19. You are a moron. by Anonymous Coward · · Score: 0

    Neither of those would have helped dipshit, learn to read or shut that big gaping hole that stupidity constantly pours out of. It was a ref count off by one, no magic compiler or dev "safety" tools would have caught it.

  20. bounds checking gcc... by bani · · Score: 1

    ...would have found this immediately.

    use it.

  21. Off by one error by pcjunky · · Score: 1

    I beleive that the hackers dictionary calls these Obi Wan Errors.

  22. Bugs? by Anonymous Coward · · Score: 0

    What about the bugs between your teeth on such a bike ride?