Slashdot Mirror


Debugging the FreeBSD Kernel Transparently

An anonymous reader writes "To debug the FreeBSD kernel core dump efficiently, it is best to set up a remote debugging session between a development machine and the target machine, and remotely debug the kernel using serial communication. This article shows you how you can debug as many kernel images as you want; it becomes transparent to users once debugging starts, and your system's performance is not affected at all."

34 of 55 comments (clear)

  1. Surely... by BluhDeBluh · · Score: 4, Funny

    If this news day gets any slower it's just going to just stop.

    1. Re:Surely... by MattPat · · Score: 1, Interesting

      Please, enlighten me as to how something can be "1, Redundant" if it's the first post?

    2. Re:Surely... by mrbluze · · Score: 1

      I think people will see through this transparent debugging process for what it really is... A thinly veiled attempt to strip the code naked, make the kernel look sexy, and get something posted on slashdot!

      --
      Do it yourself, because no one else will do it yourself. [beta blockade 10-17 Feb]
    3. Re:Surely... by pembo13 · · Score: 1

      I've wondered that myself.

      --
      "Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
    4. Re:Surely... by Brandybuck · · Score: 1

      I've got no idea what time it is. I'm lucky to be posting at all, what with all the traffic signals not working and ATM machines crashed and people rioting in the streets...

      --
      Don't blame me, I didn't vote for either of them!
    5. Re:Surely... by lord_sarpedon · · Score: 1

      You might say that we've reached the break(ing)point...

      --
      "Strangers have the best candy" -Me
  2. I was just about to post this story.... by concernedadmin · · Score: 3, Informative

    Hehe, good to see others saw this as an influential story of the day.
    However, I have a question that perhaps others can answer.
    What does the diagram actually work out?
    Listing 1. 25-Pin NULL modem cabling

    2 3 Transmit Data
    3 2 Receive Data
    4 5 Request to Send
    5 4 Clear to Send
    6 20 Data Set Ready and Carrier Detect
    7 7 Signal Ground
    20 6 Data Terminal Ready

    What does that mean? I thought serial -> serial is merely a connection. Do I have to solder something?

    Regards

    1. Re:I was just about to post this story.... by ampmouse · · Score: 4, Informative

      What does the diagram actually work out?
      This is a Null modem adapter. It allows you to connect to terminals, such as computers, together without a modem. It is the serial equivalent of a ethernet crossover cable.
    2. Re:I was just about to post this story.... by Spaceball_3000 · · Score: 2, Informative

      Normally from PC to modem the modem doesn't have wires crossed just a straight through cable, as the device automatically will swap certain pins like pins 2 and 3 in the device itself. But if your connecting from one computer to another using an straight through cable, the same pin like receive will connect to the same pin on the other PC (which is the receive), thus this will not work; as for normal operation you need to connect the Transmit to Receive from each PCs. Hope this helps.

    3. Re:I was just about to post this story.... by mattjb0010 · · Score: 1

      What does that mean?

      You were going to submit the article without RTFA? From the article, comes this link, which explains things rather well (IMHO).

      Do I have to solder something?

      No, you should be able to buy a null modem cable.

    4. Re:I was just about to post this story.... by Alwin+Henseler · · Score: 2, Funny

      Do I have to solder something?

      You must be in the wrong place here. A true Slashdot user would ask: "Please, can I solder something to make this work?"

    5. Re: I was just about to post this story.... by Dolda2000 · · Score: 2, Interesting

      I thought serial -> serial is merely a connection. Do I have to solder something?
      You seem to think in strange ways. Indeed, serial -> serial is a connection, but surely, you realize that you have to make that connection with something, or did you think it was wireless?

      Either way, to actually answer the question, you don't actually have to solder anything -- you should be able to buy a null modem cable anywhere. However, considering this is Slashdot, don't you think it would be easier to solder it yourself than going out to it? You wouldn't even have to leave your house. :)

      What I don't understand is why they provide the listing for a 25 pin cable. Do any computers these days have anything but 9 pin RS-232 ports?

    6. Re:I was just about to post this story.... by King+tweak · · Score: 1

      No you already have to many whirs. twee will do. 1,2,7 xon baby...

    7. Re:I was just about to post this story.... by King+tweak · · Score: 1

      oopy 2,3 and the mighty 7 oh my

    8. Re:I was just about to post this story.... by dioxide · · Score: 1

      it's the pinout of the null modem cable. you can't use a straight through serial cable, the tx/rx lines need to be reversed so that computer a's transmit ends up at computer b's receive.

      you can either cut a straight through cable and resolder, buy the db9 or db25 ends and make your own cable (can be soldered, or solderless), or just go buy a null modem cable.

    9. Re:I was just about to post this story.... by DeadCatX2 · · Score: 1

      tx
      rx
      gnd
      ftw

      --
      :(){ :|:& };:
  3. gdb remote is useful by EmbeddedJanitor · · Score: 2, Interesting
    This is a very cool feature in gdb that is very commonly used to debug various embedded systems etc. Sometimes gdb remote uses a serial port or tcp from the host to a stub running in the target (as is the case with Redboot).

    Also, gdb remote is often used to talk to a server that mimics a gdb remote stub and then turns the commands into some other connection into the target (eg. a JTAG debugger).

    All up, this makes debugging embedded systems a lot simpler than it would otherwise be.

    --
    Engineering is the art of compromise.
  4. Stop pissing about by DrSkwid · · Score: 2, Interesting

    Get yourselves a system, with a proper debugger

    --
    There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
  5. debugging kernels by digitalderbs · · Score: 2, Funny

    This article shows you how you can debug as many kernel images as you want
    including zero.
  6. Computers these days by DeadCatX2 · · Score: 1

    We just bought a few brand new PCs recently, and they didn't come with RS-232 ports at all.

    Laptops almost never have serial or parallel connectors.

    But, you are correct, 25-pin serial is rather rare...

    --
    :(){ :|:& };:
    1. Re:Computers these days by King+tweak · · Score: 1

      Humm. Sun boxens. And if you lucky you don't get an OK India friends OK

  7. Unfortunately by edderly · · Score: 3, Funny

    Jayesh V. Rane works as a Systems Software Engineer for IBM and has five years of experience in product development, mostly working on network programming and storage management software. He has worked extensively on FreeBSD and AIX operating systems. Jayesh is currently working with the IBM India Systems and Technology Lab in Pune, India. You can contact him at jayesh.rane@in.ibm.com. Unfortunately Jayesh doesn't understand that compiling your kernel with -g will in fact affect the systems performance and is currently looking for a job where he can pretend to be a kernel engineer. Contact Jayesh at jayesh.rane@aol.com

  8. Re:Who cares? by King+tweak · · Score: 1

    Go away to another land and find your fortune cookie u kinky kernel.

  9. Wyse terminals work well for this. by Anonymous Coward · · Score: 3, Interesting

    About 5 years ago I used an old Wyse terminal I got at a junkyard for exactly this job using a homemade cable, it worked beautifully. Once complete I then used the terminal to control and update the server from the living room couch.

    I also used the same terminal and SoftIce to *ahem* "debug" several windows applications.

    Wyse terminals (at least the older ones) are excellent gear, very sturdy, nice keyboards, though the monochrome monitor was a bit burnt in on the one I used. An industrial strength terminal for a rock solid OS. A match made in heaven. FreeBSD is an excellent OS.

    1. Re:Wyse terminals work well for this. by archen · · Score: 1

      Is it possible to use wyse terminal keyboards with a regular PC? I've got a lot of them laying around and really like them but nothing to hook them up to really (only two serial terminals).

  10. Transparent to users? by istartedi · · Score: 1

    Something tells me that if the kernel has dumped core and dropped down into a debugger, the users are going to notice something. Just a hunch.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
    1. Re:Transparent to users? by flithm · · Score: 1

      A core dump does not necessarily signify a crash, or dropping "down" into a debugger or anything. In fact you can actually force pretty much any app to generate a core dump. Usually all you need to do is send a SIG_QUIT signal to the app. However, this depends on your security settings and so forth, since a core dump of a running program might reveal sensitive information.

      At any rate, FreeBSD provides a mechanism for allowing core dumps of the kernel to happen whenever you (ie someone who's authorized) wish. You tell the kernel to core dump, and then attach a debugger to the image and the kernel keeps on going. Users really will never notice!

  11. It was a dark day for Linux by Chatz · · Score: 2, Interesting

    It was a dark day for Linux when Linus said he didn't believe in kernel debuggers.

    Last time I had a good look at what was on offer while developing a filesystem for multiple operating systems, I compared the tools for Linux to Windows, MacOSX, IRIX and Solaris.

    Windows was clearly the best for remote kernel debugging with windbg and I don't see that changing anytime soon. A fully fledged debugger, automatic download of kernel symbols from Microsoft and your own repository, reliable capture of dumps etc. That and the API documentation made the black box nature of the O/S a much smaller issue than the open source community would like us to believe. I was always so much more confident if we had a crash due to our software at a customer site that with Windows we had the best chance of capturing and hopefully identifying the problem.

    For live debugging IRIX was the best with icrash. Its port to Linux, lcrash, was not as good and reliable at the time, maybe its better now but the lack of symbol files it need on Linux always made it frustrating. And now IRIX is dead.

    I've heard some really good things about dtrace, so Solaris and Mac look to have the best live debugging and tracing tool there is today...I see some progress on Linux ports which is great to see.

    But when it comes to kernel debugging on Linux, the picture is still bleak.

    --
    There is folly and foolishness on the one side, and daring and calculation on the other. - Admiral Pellew, Hornblower
    1. Re:It was a dark day for Linux by ThePhilips · · Score: 1

      But when it comes to kernel debugging on Linux, the picture is still bleak.

      I recently was customizing for myself frame buffer driver. I have overhauled about 20k of kernel code - and after several compilation errors it worked perfectly fine. As in "just works."

      On Linux side I can tell only one thing: if you need a debugger for kernel, then you are better off doing your stuff in user space. 2.6 is heck a lot cleaner and polished than any linux kernel I have worked before - 2.0/2.1/2.2/2.4. Most stuff is so frigging straight forward that kernel framework basically leaves you ... little place for error. And especially if you do programming by good ol' "copy'n'pasting" method and adopt code from other part of kernel it is damn obvious how stuff works and how to change it - just from its look. Room for error is very very tiny - in many cases non-existent.

      As an advice, try to program w/o debuggers for some time. Start trusting your hunch feeling - what is right and what is wrong. (K.I.S.S. helps here too: "looks complicated" == "very likely buggy"). Do more code audit/review so that your eyes/mind would be able to spot errors sooner than later. In the end it would improve your coding speed: because that would train your imagination to program not in text editor/ide - but firstly in your brain. To amend algorithm in best editor you need to type/click what takes time - amending algorithm in your brain takes no keyboard/mouse effort what is much much quicker.

      Most people I know - who use debuggers - use hardware jtag port debuggers. If you are digging something more in hardware than in software (e.g. driver for very very complicated piece of hardware) than jtag is must have anyway. But even then it is possible to go w/o debugger - modularized Linux (with module dependencies) makes that a bliss.

      Windows was clearly the best for remote kernel debugging with windbg and I don't see that changing anytime soon. A fully fledged debugger, automatic download of kernel symbols from Microsoft and your own repository, reliable capture of dumps etc. That and the API documentation made the black box nature of the O/S a much smaller issue than the open source community would like us to believe. I was always so much more confident if we had a crash due to our software at a customer site that with Windows we had the best chance of capturing and hopefully identifying the problem.

      This is not *nix way. You need to always remember: inside of every big problem there is little one trying to break out. *nix way is to "divide and conquer". If instead of single monster application you would have set of executables each running in separate memory space - error localization would be improved drastically.

      I did once for my friends dumb conversion of their (Windows) application into such one. First I have split application into N libraries. Over some time interfaces had settled down and I have processed to second step: I have compiled every library into application automatically with script generating RPC-like stubs for every function. RPC performance sucked big time. But that way we were able to localize several hard to spot bugs - including nasty one reported by our customers which kick-started the effort. In our case, when system is in production cycle you cannot access it with debugger anyway. Reproducing customer's environment is impossible for it depends on large unique hardware installation. But with simple trick we were able to make system much much more manageable and accessible to testing. All that WITHOUT DEBUGGERS. (I'd say all thanks to clever M$' compiler and linker which allowed that feat. Though of course method I have learned from *nix (-; )

      --
      All hope abandon ye who enter here.
    2. Re:It was a dark day for Linux by nuzak · · Score: 1

      > On Linux side I can tell only one thing: if you need a debugger for kernel, then you are better off doing your stuff in user space.

      Yeah, it's too bad Linus doesn't believe in microkernels either. This is the same guy who thought the OOM killer was a great idea.

      --
      Done with slashdot, done with nerds, getting a life.
  12. Which system? by WereRaven · · Score: 1

    The "system performance" he is referring to is on your development machine, i.e. where you are running kgdb. Of course compiling your kernel with -g will affect performance on the target machine! By not running a development kernel on your development machine you are not affecting your system's performance. I don't know about you but it really affects my system's performance when the kernel panics.

  13. Re:DDB by Eivind+Eklund · · Score: 1
    DDB is in FreeBSD, too, since always. In fact, DDB was ported from Mach to 386BSD, which predates the FreeBSD/NetBSD split / diverge, which again predates the NetBSD/OpenBSD split.

    Eivind.

    --
    Doubting the existence of evolution is like doubting the existence of China: It just shows that you're uninformed.
  14. quantum by null-sRc · · Score: 1

    maybe the mere act of viewing the event will change it... is debugging ultimately futile?

    --
    -judging another only defines yourself
    1. Re:quantum by nuzak · · Score: 1

      This is sometimes known as a Heisenbug (ah yes the Jargon File did have some nice entries back in the old days). Multithreaded or just plain time-sensitive code (like that found in device drivers) is notorious for this sort of thing.

      --
      Done with slashdot, done with nerds, getting a life.