Kernel Hacker Keith Owens On kbuild 2.5, XFS, More

Jeremy Andrews writes: "Kerneltrap interviews Keith Owens this week, an experienced kernel hacker who has long contributed to the Linux kernel. His contributions include updating ksymoops and modutils, both of which he maintains. He also works on kbuild 2.5. Earlier, he built the original Integrated Kernel Debugging patch. He's also working on kdb and XFS. Check out the interview." Lots of good information in here about things to expect in 2.5.

Comments

This is so cool

[Anton Anatopopov]

I have often wanted to have a go at kernel programming. I want to try and write some device drivers, but I am always too scared of this 'black art'. Its good to see someone taking time out to make it a bit more comprehensible for 'the rest of us'.

I'm wondering, do kernel developers use tools like vmware/plex86 to debug their running kernels ? It seems like we've come a long way since debugging with strategically placed printfs

Re:This is so cool

[rasmus_a]

> I have often wanted to have a go at kernel programming. I want to try and write some device drivers, but I am always
>too scared of this 'black art'. Its good to see someone taking time out to make it a bit more comprehensible for 'the rest of us'.

Try www.kernelnewbies.org. Esp. look in the book section for Linux Device Drivers v.2 and the online version.

Wrt. debugging, prinks are still used alongside everything else. I do not think that debugging is done with vmware or plex86 yet but there is a port of the kernel to userland (User-Mode Linux) which is used by some.

Rasmus

Re:This is so cool

[SurfsUp]

I'm wondering, do kernel developers use tools like vmware/plex86 to debug their running kernels ? It seems like we've come a long way since debugging with strategically placed printfs

Vmware or plex86 could possibly be of some use, except that they're no good for debugging device drivers since the real devices are hidden behind a virtualization layer. For non-device driver work User Mode Linux is a more lightweight solution, and tracks the latest development kernels much more closely. An increasing number of the core developers are using User Mode Linux regularly.

For heavy debugging work on live kernels, kgdb is the perferred solution, with a serial cable link to a test machine. It takes a little more work to set this up and you need two machines. Kdb is a simpler debugger that can be patched into the kernel, useful for tracking down elusive kernel problems. It's included by default in SGI's XFS patch and pre-patched kernels.

There are some great tools available including LTT, the Linux Trace Toolkit and various lock-monitoring patches. Unfortunately, most driver development is still being done by the printk/reboot method. If this is your preferred method, make sure you install a journalling filesystem unless you like spending most of your time watching fsck work.

--
Daniel

It's a shame about Linus's opinion on kdb

[wowbagger]

While I can certainly understand Linus wanting to encourage would-be kernel developers to learn "a gram of analysis is worth a kilo of debugging", I do wish he would consider one area in which a kernel debugger is invaluable - hardware integration.

"In theory, there is no difference between theory and practice. In practice, there is." In hardware development, there is the theory of what the hardware documentation says the chip will do, and then there is the practice of what it actually does. DMA's don't, interrupts stick, registers report old data. Obviously, you START by writing a user space app that pokes at the hardware (and this is one area in which Linux is head and shoulders above WinNT - there is NO way for a user space app to access hardware in NT, while in Linux you simply have to be root), but when you finally need to hook interrupts, allocate DMA buffers, etc., you need a debugger that can look at these events.

Also, when porting to other CPUs, you sometimes need to see what is going on at the hardware level, and how it affects the drivers in the kernel.

Yes, allowing debugging without analysis is bad. But throwing us back to the stone knives and bear skins era just to encourage hardier folks is an overreaction. Sure, make a KDB kernel bitch and moan during startup. Make it only allow root access, not normal user access. Force all file systems to run in full sync mode. But please don't make debugging buggy hardware any harder than it needs to be.

(Now, if only AMD would add a JTAG debugger to the Athlon chip, I'd be a happy man.)

Global Makefile!

[swillden]

From the interview:

...kbuild 2.5 builds a global Makefile from Makefile.in fragments in each directory then compiles and links only the sections of code that absolutely need to be recompiled

This is excellent, and I hope more open source projects start to go this way. It's been known for a while that recursive make is a bad idea because it's inaccurate. Naive recursive makefile structures tend to miss stuff that needs to be built/installed and fixing that problem (usually with ugly hacks like make dep) generally results in building stuff that doesn't need to be built.

What Keith describes is a nice solution that provides the benefits of recursive make without the problems: Use per-directory makefile fragments which can be maintained locally, but automatically generate a complete, tree-wide makefile that is actually used for the build.

There are tools other than make that provide more elegant solutions, but given that they never seem to catch on, I'm happy to see that someone is applying the tool we have (make) correctly, for once.

I'm looking forward to this one.