Slashdot Mirror


Debugging the FreeBSD Kernel Transparently

An anonymous reader writes "To debug the FreeBSD kernel core dump efficiently, it is best to set up a remote debugging session between a development machine and the target machine, and remotely debug the kernel using serial communication. This article shows you how you can debug as many kernel images as you want; it becomes transparent to users once debugging starts, and your system's performance is not affected at all."

55 comments

  1. Surely... by BluhDeBluh · · Score: 4, Funny

    If this news day gets any slower it's just going to just stop.

    1. Re:Surely... by MattPat · · Score: 1, Interesting

      Please, enlighten me as to how something can be "1, Redundant" if it's the first post?

    2. Re:Surely... by mrbluze · · Score: 1

      I think people will see through this transparent debugging process for what it really is... A thinly veiled attempt to strip the code naked, make the kernel look sexy, and get something posted on slashdot!

      --
      Do it yourself, because no one else will do it yourself. [beta blockade 10-17 Feb]
    3. Re:Surely... by pembo13 · · Score: 1

      I've wondered that myself.

      --
      "Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
    4. Re:Surely... by __aaclcg7560 · · Score: 0, Offtopic

      Yup. BTW, What time is is?

    5. Re:Surely... by Brandybuck · · Score: 1

      I've got no idea what time it is. I'm lucky to be posting at all, what with all the traffic signals not working and ATM machines crashed and people rioting in the streets...

      --
      Don't blame me, I didn't vote for either of them!
    6. Re:Surely... by lord_sarpedon · · Score: 1

      You might say that we've reached the break(ing)point...

      --
      "Strangers have the best candy" -Me
    7. Re:Surely... by Anonymous Coward · · Score: 0

      Perhaps marking all first posts "Offtopic" is going out of style.

  2. I was just about to post this story.... by concernedadmin · · Score: 3, Informative

    Hehe, good to see others saw this as an influential story of the day.
    However, I have a question that perhaps others can answer.
    What does the diagram actually work out?
    Listing 1. 25-Pin NULL modem cabling

    2 3 Transmit Data
    3 2 Receive Data
    4 5 Request to Send
    5 4 Clear to Send
    6 20 Data Set Ready and Carrier Detect
    7 7 Signal Ground
    20 6 Data Terminal Ready

    What does that mean? I thought serial -> serial is merely a connection. Do I have to solder something?

    Regards

    1. Re:I was just about to post this story.... by ampmouse · · Score: 4, Informative

      What does the diagram actually work out?
      This is a Null modem adapter. It allows you to connect to terminals, such as computers, together without a modem. It is the serial equivalent of a ethernet crossover cable.
    2. Re:I was just about to post this story.... by Spaceball_3000 · · Score: 2, Informative

      Normally from PC to modem the modem doesn't have wires crossed just a straight through cable, as the device automatically will swap certain pins like pins 2 and 3 in the device itself. But if your connecting from one computer to another using an straight through cable, the same pin like receive will connect to the same pin on the other PC (which is the receive), thus this will not work; as for normal operation you need to connect the Transmit to Receive from each PCs. Hope this helps.

    3. Re:I was just about to post this story.... by mattjb0010 · · Score: 1

      What does that mean?

      You were going to submit the article without RTFA? From the article, comes this link, which explains things rather well (IMHO).

      Do I have to solder something?

      No, you should be able to buy a null modem cable.

    4. Re:I was just about to post this story.... by Alwin+Henseler · · Score: 2, Funny

      Do I have to solder something?

      You must be in the wrong place here. A true Slashdot user would ask: "Please, can I solder something to make this work?"

    5. Re: I was just about to post this story.... by Dolda2000 · · Score: 2, Interesting

      I thought serial -> serial is merely a connection. Do I have to solder something?
      You seem to think in strange ways. Indeed, serial -> serial is a connection, but surely, you realize that you have to make that connection with something, or did you think it was wireless?

      Either way, to actually answer the question, you don't actually have to solder anything -- you should be able to buy a null modem cable anywhere. However, considering this is Slashdot, don't you think it would be easier to solder it yourself than going out to it? You wouldn't even have to leave your house. :)

      What I don't understand is why they provide the listing for a 25 pin cable. Do any computers these days have anything but 9 pin RS-232 ports?

    6. Re:I was just about to post this story.... by King+tweak · · Score: 1

      No you already have to many whirs. twee will do. 1,2,7 xon baby...

    7. Re:I was just about to post this story.... by King+tweak · · Score: 1

      oopy 2,3 and the mighty 7 oh my

    8. Re:I was just about to post this story.... by dioxide · · Score: 1

      it's the pinout of the null modem cable. you can't use a straight through serial cable, the tx/rx lines need to be reversed so that computer a's transmit ends up at computer b's receive.

      you can either cut a straight through cable and resolder, buy the db9 or db25 ends and make your own cable (can be soldered, or solderless), or just go buy a null modem cable.

    9. Re:I was just about to post this story.... by DeadCatX2 · · Score: 1

      tx
      rx
      gnd
      ftw

      --
      :(){ :|:& };:
  3. gdb remote is useful by EmbeddedJanitor · · Score: 2, Interesting
    This is a very cool feature in gdb that is very commonly used to debug various embedded systems etc. Sometimes gdb remote uses a serial port or tcp from the host to a stub running in the target (as is the case with Redboot).

    Also, gdb remote is often used to talk to a server that mimics a gdb remote stub and then turns the commands into some other connection into the target (eg. a JTAG debugger).

    All up, this makes debugging embedded systems a lot simpler than it would otherwise be.

    --
    Engineering is the art of compromise.
  4. Stop pissing about by DrSkwid · · Score: 2, Interesting

    Get yourselves a system, with a proper debugger

    --
    There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
    1. Re:Stop pissing about by Anonymous Coward · · Score: 0

      Get yourselves a system, with a proper debugger


      Firstly, "Acid is an unusual source-level symbolic debugger for Plan 9" .. which of course means that it's atypical and not necessarily proper.

      Secondly, this link alone pretty much qualifies you as a crackpot.

      Now, log off and go take your haldol.
    2. Re:Stop pissing about by Anonymous Coward · · Score: 0

      Did you even bother to read the Acid paper? I glanced over it, and I don't see why it would be "not proper". It's programmable, which means it does as little or much as you want it to.

      Milk link? Well, whatever.

  5. debugging kernels by digitalderbs · · Score: 2, Funny

    This article shows you how you can debug as many kernel images as you want
    including zero.
  6. Test; please ignore. by Anonymous Coward · · Score: 0
  7. Computers these days by DeadCatX2 · · Score: 1

    We just bought a few brand new PCs recently, and they didn't come with RS-232 ports at all.

    Laptops almost never have serial or parallel connectors.

    But, you are correct, 25-pin serial is rather rare...

    --
    :(){ :|:& };:
    1. Re:Computers these days by King+tweak · · Score: 1

      Humm. Sun boxens. And if you lucky you don't get an OK India friends OK

    2. Re:Computers these days by Anonymous Coward · · Score: 0

      Suns newest x86 boxes don't have a usuable serial port, it is needed for remote console access via the network. It is like this on x4500's, x4100's, x4200's, and I am pretty sure x8400's.

  8. Unfortunately by edderly · · Score: 3, Funny

    Jayesh V. Rane works as a Systems Software Engineer for IBM and has five years of experience in product development, mostly working on network programming and storage management software. He has worked extensively on FreeBSD and AIX operating systems. Jayesh is currently working with the IBM India Systems and Technology Lab in Pune, India. You can contact him at jayesh.rane@in.ibm.com. Unfortunately Jayesh doesn't understand that compiling your kernel with -g will in fact affect the systems performance and is currently looking for a job where he can pretend to be a kernel engineer. Contact Jayesh at jayesh.rane@aol.com

    1. Re:Unfortunately by Anonymous Coward · · Score: 0

      Where is Rob Enderle when you need him?

  9. Re:Who cares? by King+tweak · · Score: 1

    Go away to another land and find your fortune cookie u kinky kernel.

  10. Good luck by Anonymous Coward · · Score: 0

    Yeah, good luck trying to debug with all the blobs loaded into the kernel.

  11. Re:I've been wondering... by Anonymous Coward · · Score: 0

    Well, that isn't hard to figure out. I guess it is true that beautiful people are really fucking dumb.

    For every Mac user there are about 10,000 PC users. Of that 10,000 about 75 are beautiful, so beautiful PC users outnumber you trendoid Mac dorks by a far margin.

    Do you understand, honey ?

  12. Developer Laments: What Killed FreeBSD by Anonymous Coward · · Score: 0, Troll

    The End of FreeBSD

    [ed. note: in the following text, former FreeBSD developer Mike Smith gives his reasons for abandoning FreeBSD]

    When I stood for election to the FreeBSD core team nearly two years ago, many of you will recall that it was after a long series of debates during which I maintained that too much organisation, too many rules and too much formality would be a bad thing for the project.

    Today, as I read the latest discussions on the future of the FreeBSD project, I see the same problem; a few new faces and many of the old going over the same tired arguments and suggesting variations on the same worthless schemes. Frankly I'm sick of it.

    FreeBSD used to be fun. It used to be about doing things the right way. It used to be something that you could sink your teeth into when the mundane chores of programming for a living got you down. It was something cool and exciting; a way to spend your spare time on an endeavour you loved that was at the same time wholesome and worthwhile.

    It's not anymore. It's about bylaws and committees and reports and milestones, telling others what to do and doing what you're told. It's about who can rant the longest or shout the loudest or mislead the most people into a bloc in order to legitimise doing what they think is best. Individuals notwithstanding, the project as a whole has lost track of where it's going, and has instead become obsessed with process and mechanics.

    So I'm leaving core. I don't want to feel like I should be "doing something" about a project that has lost interest in having something done for it. I don't have the energy to fight what has clearly become a losing battle; I have a life to live and a job to keep, and I won't achieve any of the goals I personally consider worthwhile if I remain obligated to care for the project.

    Discussion

    I'm sure that I've offended some people already; I'm sure that by the time I'm done here, I'll have offended more. If you feel a need to play to the crowd in your replies rather than make a sincere effort to address the problems I'm discussing here, please do us the courtesy of playing your politics openly.

    From a technical perspective, the project faces a set of challenges that significantly outstrips our ability to deliver. Some of the resources that we need to address these challenges are tied up in the fruitless metadiscussions that have raged since we made the mistake of electing officers. Others have left in disgust, or been driven out by the culture of abuse and distraction that has grown up since then. More may well remain available to recruitment, but while the project is busy infighting our chances for successful outreach are sorely diminished.

    There's no simple solution to this. For the project to move forward, one or the other of the warring philosophies must win out; either the project returns to its laid-back roots and gets on with the work, or it transforms into a super-organised engineering project and executes a brilliant plan to deliver what, ultimately, we all know we want.

    Whatever path is chosen, whatever balance is struck, the choosing and the striking are the important parts. The current indecision and endless conflict are incompatible with any sort of progress.

    Trying to dissect the above is far beyond the scope of any parting shot, no matter how distended. All I can really ask of you all is to let go of the minutiae for a moment and take a look at the big picture. What is the ultimate goal here? How can we get there with as little overhead as possible? How would you like to be treated by your fellow travellers?

    Shouts

    To the Slashdot "BSD is dying" crowd - big deal. Death is part of the cycle; take a look at your soft, pallid bodies and consider that right this very moment, parts of you are dying. See? It's not so bad.

    To the bulk of the FreeBSD committerbase and the developer community at large - k

  13. Wyse terminals work well for this. by Anonymous Coward · · Score: 3, Interesting

    About 5 years ago I used an old Wyse terminal I got at a junkyard for exactly this job using a homemade cable, it worked beautifully. Once complete I then used the terminal to control and update the server from the living room couch.

    I also used the same terminal and SoftIce to *ahem* "debug" several windows applications.

    Wyse terminals (at least the older ones) are excellent gear, very sturdy, nice keyboards, though the monochrome monitor was a bit burnt in on the one I used. An industrial strength terminal for a rock solid OS. A match made in heaven. FreeBSD is an excellent OS.

    1. Re:Wyse terminals work well for this. by archen · · Score: 1

      Is it possible to use wyse terminal keyboards with a regular PC? I've got a lot of them laying around and really like them but nothing to hook them up to really (only two serial terminals).

    2. Re:Wyse terminals work well for this. by Anonymous Coward · · Score: 0

      I don't know, I only used the whole terminal on a db25 serial port on my ASUS-PIII system. I seem to recall the one I used had a RJ9 style connector. You could use this info to adapt a cable for the RJ9 style Wyse keyboard connector to try.:
      http://www.wyse.com/service/support/kbase/wits/802 19.txt
      http://en.wikipedia.org/wiki/PS/2_connector
      USB would be another matter...
      Hours of fun! I'm betting they probably won't work, and I certainly would not try it on a new PC that I cared about. Perhaps some of the older ones with RJ9 connectors might work, but the newer DIN connector ones certainly won't.

      Here is the knowledgebase on the old Wyse terminals:
      http://www.wyse.com/service/support/kbase/wyseterm .asp
      The info on all the old keyboard models is available there.

      I did love that keyboard though, much like the old IBM ones. My standard keyboard now is the base Keytronics model, it sets the standard in price, durability and feel in my opinion. Check out the environmental and mechanical data, pretty amazing. I have abused mine horribly for the last 8 years without a single failure. All these new cheaply made keyboards with volume wheels and programmable buttons that require windows drivers are just plain retarded as far as I'm concerned.

  14. Re:Who cares? by reset_button · · Score: 0, Troll

    Why was this modded as troll? Netcraft has confirmed - BSD is dying!

  15. Transparent to users? by istartedi · · Score: 1

    Something tells me that if the kernel has dumped core and dropped down into a debugger, the users are going to notice something. Just a hunch.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
    1. Re:Transparent to users? by flithm · · Score: 1

      A core dump does not necessarily signify a crash, or dropping "down" into a debugger or anything. In fact you can actually force pretty much any app to generate a core dump. Usually all you need to do is send a SIG_QUIT signal to the app. However, this depends on your security settings and so forth, since a core dump of a running program might reveal sensitive information.

      At any rate, FreeBSD provides a mechanism for allowing core dumps of the kernel to happen whenever you (ie someone who's authorized) wish. You tell the kernel to core dump, and then attach a debugger to the image and the kernel keeps on going. Users really will never notice!

  16. It was a dark day for Linux by Chatz · · Score: 2, Interesting

    It was a dark day for Linux when Linus said he didn't believe in kernel debuggers.

    Last time I had a good look at what was on offer while developing a filesystem for multiple operating systems, I compared the tools for Linux to Windows, MacOSX, IRIX and Solaris.

    Windows was clearly the best for remote kernel debugging with windbg and I don't see that changing anytime soon. A fully fledged debugger, automatic download of kernel symbols from Microsoft and your own repository, reliable capture of dumps etc. That and the API documentation made the black box nature of the O/S a much smaller issue than the open source community would like us to believe. I was always so much more confident if we had a crash due to our software at a customer site that with Windows we had the best chance of capturing and hopefully identifying the problem.

    For live debugging IRIX was the best with icrash. Its port to Linux, lcrash, was not as good and reliable at the time, maybe its better now but the lack of symbol files it need on Linux always made it frustrating. And now IRIX is dead.

    I've heard some really good things about dtrace, so Solaris and Mac look to have the best live debugging and tracing tool there is today...I see some progress on Linux ports which is great to see.

    But when it comes to kernel debugging on Linux, the picture is still bleak.

    --
    There is folly and foolishness on the one side, and daring and calculation on the other. - Admiral Pellew, Hornblower
    1. Re:It was a dark day for Linux by ThePhilips · · Score: 1

      But when it comes to kernel debugging on Linux, the picture is still bleak.

      I recently was customizing for myself frame buffer driver. I have overhauled about 20k of kernel code - and after several compilation errors it worked perfectly fine. As in "just works."

      On Linux side I can tell only one thing: if you need a debugger for kernel, then you are better off doing your stuff in user space. 2.6 is heck a lot cleaner and polished than any linux kernel I have worked before - 2.0/2.1/2.2/2.4. Most stuff is so frigging straight forward that kernel framework basically leaves you ... little place for error. And especially if you do programming by good ol' "copy'n'pasting" method and adopt code from other part of kernel it is damn obvious how stuff works and how to change it - just from its look. Room for error is very very tiny - in many cases non-existent.

      As an advice, try to program w/o debuggers for some time. Start trusting your hunch feeling - what is right and what is wrong. (K.I.S.S. helps here too: "looks complicated" == "very likely buggy"). Do more code audit/review so that your eyes/mind would be able to spot errors sooner than later. In the end it would improve your coding speed: because that would train your imagination to program not in text editor/ide - but firstly in your brain. To amend algorithm in best editor you need to type/click what takes time - amending algorithm in your brain takes no keyboard/mouse effort what is much much quicker.

      Most people I know - who use debuggers - use hardware jtag port debuggers. If you are digging something more in hardware than in software (e.g. driver for very very complicated piece of hardware) than jtag is must have anyway. But even then it is possible to go w/o debugger - modularized Linux (with module dependencies) makes that a bliss.

      Windows was clearly the best for remote kernel debugging with windbg and I don't see that changing anytime soon. A fully fledged debugger, automatic download of kernel symbols from Microsoft and your own repository, reliable capture of dumps etc. That and the API documentation made the black box nature of the O/S a much smaller issue than the open source community would like us to believe. I was always so much more confident if we had a crash due to our software at a customer site that with Windows we had the best chance of capturing and hopefully identifying the problem.

      This is not *nix way. You need to always remember: inside of every big problem there is little one trying to break out. *nix way is to "divide and conquer". If instead of single monster application you would have set of executables each running in separate memory space - error localization would be improved drastically.

      I did once for my friends dumb conversion of their (Windows) application into such one. First I have split application into N libraries. Over some time interfaces had settled down and I have processed to second step: I have compiled every library into application automatically with script generating RPC-like stubs for every function. RPC performance sucked big time. But that way we were able to localize several hard to spot bugs - including nasty one reported by our customers which kick-started the effort. In our case, when system is in production cycle you cannot access it with debugger anyway. Reproducing customer's environment is impossible for it depends on large unique hardware installation. But with simple trick we were able to make system much much more manageable and accessible to testing. All that WITHOUT DEBUGGERS. (I'd say all thanks to clever M$' compiler and linker which allowed that feat. Though of course method I have learned from *nix (-; )

      --
      All hope abandon ye who enter here.
    2. Re:It was a dark day for Linux by nuzak · · Score: 1

      > On Linux side I can tell only one thing: if you need a debugger for kernel, then you are better off doing your stuff in user space.

      Yeah, it's too bad Linus doesn't believe in microkernels either. This is the same guy who thought the OOM killer was a great idea.

      --
      Done with slashdot, done with nerds, getting a life.
  17. DDB by cachimaster · · Score: 0

    Why it can't be like ddb, the OpenBSD debugger. You press Ctrl+Alt+Esc and Bam! you are debugging the kernel, with full symbols and breakpoints. Very cool kernel debugger.

    1. Re:DDB by Eivind+Eklund · · Score: 1
      DDB is in FreeBSD, too, since always. In fact, DDB was ported from Mach to 386BSD, which predates the FreeBSD/NetBSD split / diverge, which again predates the NetBSD/OpenBSD split.

      Eivind.

      --
      Doubting the existence of evolution is like doubting the existence of China: It just shows that you're uninformed.
  18. Which system? by WereRaven · · Score: 1

    The "system performance" he is referring to is on your development machine, i.e. where you are running kgdb. Of course compiling your kernel with -g will affect performance on the target machine! By not running a development kernel on your development machine you are not affecting your system's performance. I don't know about you but it really affects my system's performance when the kernel panics.

  19. Are you sure about that? by EdSchouten · · Score: 0

    As far as I know, it shouldn't make a difference. The only thing -g does, is adding debugging information (DWARF?) to the kernel's binary, stored in a seperate section. If you'd load that kernel, it would cost a lot more memory, but wouldn't slow down your system. The FreeBSD people aren't stupid, so what they do: they strip the debugging info before they install it. You load the kernel without the debugging information, but use the other one with kgdb.

  20. Re:I've been wondering... by Anonymous Coward · · Score: 0

    You forgot the part about how it's more like there are about 5 mac users for every 100 pc users which would mean the ratio is 1 to 20, which means 500 to 10000. If we now assume that your comment about 75 of every 10000 pc users being beautiful then that means the ratio of mac users to beautiful pc users is 500 to 75 or 100 to 15. You lose.

  21. quantum by null-sRc · · Score: 1

    maybe the mere act of viewing the event will change it... is debugging ultimately futile?

    --
    -judging another only defines yourself
    1. Re:quantum by nuzak · · Score: 1

      This is sometimes known as a Heisenbug (ah yes the Jargon File did have some nice entries back in the old days). Multithreaded or just plain time-sensitive code (like that found in device drivers) is notorious for this sort of thing.

      --
      Done with slashdot, done with nerds, getting a life.