Breaking Into The World Of Kernel Hacking?

crow_t_robot asks: "In the past couple of months I have become increasingly interested in kernel programming and have finally decided to take the leap and 'get my hands dirty.' I have searched around the web and read a few docs and FAQs on getting started with the kernel but I was wondering what kind of personal experiences those in the Slashdot crowd have had that are so bold as to start goofing with the kernel code. For those that have become competent kernel programmers, how did you 'break in' and what advice would you give beginners?"

Comments

very easy... 10 steps to kernel coding:

[Cheetahfeathers]

1. Learn to code.
2. Learn to code in C.
3. Figure out what YOU want to add to the source.
4. Read the kernel source.
5. Understand what you read.
6. Make changes/additions to the source, per step #3.
7. Test out the changes/additions on your own system.
8. Make it work for you.
9. Send in your contribution.
10. Have it accepted/rejected.

Device Drivers

[Eagle7]

Buy the O'Reilly book on device drivers mentioned above, and pick a driver you use. Try to understand it, and then tweak it a bit. Since they can be loaded and unloaded, device drivers are a little easier/quicker to play with. And there is a good book on them. ;)

Working on it during part of an 8 hour work day, in about 1 month I was able to hack tab support into the s390 vm console driver with nothing more than reading code, searching the net, and using that book. And that was probably a little on the slow side. (see here http://www.eagle7.org/ibmlinux.html

And like Alan Cox said in the interview posted to /. a bit ago - Have Fun!

Re:Device Drivers

[LordNimon]

I agree with Eagle7. Start with a very simple driver, one that doesn't actually talk to hardware. Perhaps one that displays some krenel data whenever called.

From there, you can decide to either move into the kernel itself, or stick with device drivers. If you want to practice on a device driver, stick with a simple ISA device. If you can find the hardware, a driver that displays some text on a Monochrome Display Adapter (MDA) is an easy one.

Writing Kernel Modules

[PlazMatiC]

This Linux Journal article gives a really quick introduction into writing kernel modules.

It doesn't go into all that much detail, but I found it a good starting point for messing around on my linux machine at home.

Hth =)

Make your own system call

[BetaJim]

I think about the most easy kernel hack is to create is your own system call. Last year in OS this is what we did.

The assignment was to add a system call that would return the number of processes created and killed up to that point. The only difficult thing was to grok the system call table. Please look at this option as a good introduction to the kernel.

Just Do IT....

[CDWert]

Its not like youre going to break anything, real, or permanent, this is all hocus pocus digital alchemy anyhow. If you kernel flies or not, it information, either on what you did right or did wrong. Have fun break some shit. Its no fun it works right all the time anyhow.

I also highly suggest the IRC channel
#kernelnewbies
I went there today for the first time looking for Rik van Riel I had some qustions about the rmap 11b patch and Guess what he was there and told me the 11c is coming today , REAL time info from people that really know their stuff.....

Kernel hacking can be fun if youre not in a rush IMHO, doing it against a dealine can be ....uncomfortable to say the least.

Im assuming you know huw to extract the source tree and apply a patch , if not start there. Im also assuming you know C if you dont ....what are you going to hack, or just patch and compile, Look through the tree for stuff you know, see how they do it , see if you think you can do better and try it, start small....... Me, ive never personally done anything that need make it into the tree on a permanent basis. but hey its been 8 years of all kinds of fun.....

One last point DO IT OFTEN !, Dont let yourself get rusty, Set a goal, a kernel hack a month, or at least a patch and compile a week if youre not cutting code....

Kernel trashing :)

[jd]

I've found that the kernel source code varies in quality between Ultra Brilliant to Infra Crud. (Sadly, a lot of the networking code seems to fall in the latter category.)

Probably the best place to start is to find some definite itch you want to scratch, and so badly that you won't stop until it's done. For me, that was getting use[ful|less] features into the kernel. As many as I could cram in, without the hard disk exploding. And then fixing all the incompatibilities, as best I could. (Phew!! There are generally a lot!! That's why many of the FOLK patches I produce are unstable.)

What you do next depends on the "itch":

• Fixing broken/cruddy code: Generally, it's better to rip it out and start again. Find out what goes in, what comes out, and what is supposed to happen. Software Engineers tend to talk of "black boxes", where one piece of code doesn't need to know anything about other pieces of code. It just needs to know what gets put where. You can use this to test your code outside the kernel. Simply use the kernel headers for the structures, rig up some mock values, and see if your code does what it's supposed to. If it does, then all that's left is cutting & pasting the code into the kernel. Oh, and producing a diff file for the rest of us.
  
• Adding new drivers: There should be skeleton drivers in the kernel. Simply fill out one of those. If there aren't, find a driver that's similar to what you want and replace the stuff you don't need, as above.
  
• Adding new features: Ignore the rest of the kernel, completely, at the start. Since it's a new feature, rather than an extension of something already there, you're going to be mostly writing new code and working on new structures. The only time the kernel matters is when you want to get data in or out of something thats already there, or when you want to link into some existing capability (such as kernel modules). Then, only pay attention to the API in the headers. The implementation is likely to be replaced three times before the weekend, so spending too much time on it is stupid.
  

XINU is golden for learning about internals of OSs

[edrugtrader]

XINU = XINU is not unix.

it is a very simple OS with multithreading and a bunch of other stuff.

the full source is available and only like 8000 lines or something. it steps you through it in the book, and it is EXTREMELY easy to read.

this was what was used to teach my OS classes in college. you can actually get in and hack the thing away and know what you are changing right from the start.

Starter projects...

[stripes]

A good starter project is a device driver for something simple. Even easier is if you find a device Net/Free/OpenBSD supports and Linux doesn't, port the driver, that way there is somewhat less code to write.

That could be harder for you then me since Linux has so many drivers now (I started with 386BSD 0.0, so there were maybe 15 supported devices...my friend and I ported the MACH SoundBlaster driver). If all else fails, write a driver for something that already has a driver.

After that you can wonder off into the kernel proper and do some "real" stuff. I did IP traffic shaping, but someone seems to have done that to Linux already...

I was forced to do it

[BurpingWeezer]

Well, I wasn't forced to do it, but we had a class on it at my school, the University of Alberta. The class was officially called system and network management but was unofficially called "kernel hacking". Why? Well, the goal of the class was to put to practical use all of the OS concepts we had learned in previous years. Kernel hacking is perfect for that.

We were given a problem to solve: build a kernel patch to any version of the linux kernel that allowed a third party who is writing is driver to have the driver interact with a software emulator instead of real hardware. The end goal was to enable people to write drivers based only on the hardware specs and an emulator. The emulator would instruct the kernel to route certain hardware calls to it. After the driver was written the real hardware could be dropped in place and the driver would not have to be re-written, it could be used out of the box. Essentially it was hardware abstraction without knowing what hardware to abstract before hand.

Personally, if you are just looking at getting into it just find yourself a problem and solve it. It doesn't matter what problem it is, any problem, useful to the world or not. Remember that the end result of this step is to learn, not solve 2.4 VM problems, or to build a better SMP design. (That's the next step!)

Dual keyboards with seperate keyboard mappings!

[karlm]

I did my first real kernel programming (besides a little "hello, woeld!! This is the kernel speaking" module). Last week. My keyboard died a few months ago and I didn't have time to mess with it, so I went out and baught and IBM USB/PS2 keyboard (Rapid Acess Pro) and hooked it upto my machine as a PS/2 keyboard.

When I had time, I went and fixed the old keyboard, and rearranged the keys as a Devorak keyboard while I was at it. Unfortunately, the USB-PS/2 dongle that came with my IBM keyboard doesn't work with my Dell PS2 keyboard. However, I currently have the IBM keyboard hooked up to my USB port and my old Dell keyboard (with rearranged keys) as my PS/2 keyboard.
Luckily, keybdev.c (both the one in drivers/input and the one in drivers/usb) is astonishingly short. There's all of about 5 functions in there and most of the complexity is hidden in other modules.
The USB keyboard driver interfaces the PS/2 subsystem IIRC (don't know where I read this, maybe the hid.c documentation on linux-usb) so you can't have seperate keyboard mappings, unless you munge the keycodes inside the USBdriver. As long as you don't lock up your USB keyboard driver or have a buffer overun, you should always be able to restart.
I have LILO (GRUB, actually) setup to boot me into either a 2.2.20 kernel or a 2.4.17 kernel. That way, I can ensure that my hacked module won't be loaded by hotplug if I screw it up.

The Steps I took
Downloads I downloaded the 2.4.17 source code and used apt-get to install kernel-package (makes Debian kernel packages, so your nightly apt-get upgrades won't destroy your work).
Renaming I went into /usr/src/linux (after untarring the 2.4.17 source) and changed the line near the to from "EXTRAVERSION = " to "EXTRAVERSION = -maxmodular" this changes the name of the kernel from vmlinuz-2.4.17 to vmlinuz-2.4.17-maxmodular and makesa seperate directory in /lib/modules for your hacked modules. (I called it maxmodular because I even made the "misc binary support", "a.out binary support" and "floppy drive support" as modules.) Potential Gottcha: IDE support is no enabled by default, and comes after the IDE-parport questions in the config menu, so I must have recompiled 10 times trying to figure out why it couldn't mount the root partition.
Snooping arroundI poked around the drivers/usb/ kernel source and documentation some to try and figure out what needed to be hacked. I incorreclty identified drivers/usb/keybdev.c and after none of my printk's worked, I tried drivers/input/keybdev.c (this does not affect the PS/2 keyboard).
Printk() is your friend. Add printk()s (printk() works just like printf(). Make sure to do your kernel module insertion from a non-X virtual terminal to minimize the time it takes prints to get to you in the case of an impending crash.) to the beginning of every function in the driver, letting you know that it's being called (and optionally the arguments). Then recompile and load your module. This gives you a feel for how the driver works and confirms that you have the right driver.
Trim prink()s to only print out the information you need. In my case, I got rid of the printks from everything but the keyboard event handling function and the emulate_raw function kalled by the event handler. I printed out every keycode that was pressed (but not it's release) so that I could write down the keycode for each key I needed to remap. (And some of the neat colorful buttons IBM added above the funtion keys. I later used these to turn on and off the devorak remapping and turn on and off the printks in my module.

After that, what you do is pretty specific to your module. For keyboards, they send a 16-number, called a scancode that is one of the arguments to the keyboard event hadler function. All of the normal keys have scancodes less than 256, so I just made an unsigned char[256] usb_key_remap_table and did something like
if (code < 256 && code >= 0) {code = usb_key_remap_table[code];}
The more astute amoung you will remeber that I'm only remapping the USB keyboard, which is my good keyboard and therefore unmodified. Oh well, I figure having the incorrect letters written on the keys discourages me from looking down.

Anyway, the USB subsystem seems very readable for the higher level drvers and USB keeps getting more and more devices, so I'd recomed starting with a USB subsystem... too bad RadioShack is out of USB cue:cats :-D