Best Advanced Linux Kernel Training?

hdxia writes "Can anybody recommend a good Linux kernel training course? I have had some Linux kernel hacking experience, but would like to further harden and improve my understanding of the kernel. I expect the course would be advanced. You may say that the best method would be to dig into the kernel myself, but I really want to have a chance to discuss and learn all aspects of the kernel with an experienced instructor."

Comments

obligatory?

[feld]

RTFM

there, now everybody should feel at home

seriously though, it would be cool if this was offered at major universities. but you'd need knowledgable instructors and they'd be hard to come by.

Re:obligatory?

[cide1]

Kernel hacking is taught at major universities, but under the title "Operating Systems". These classes teach scheduling, device drivers, file systems, memory management, networking, etc. The goal of a university is to give you the background knowledge necessary to understand specific implementations. The specifics of the linux kernel change rapidly, but the concepts are 40 years old. No instructional course has a chance of keeping up, the kernel documentation cant even keep up. It takes a problem solver, and an inquizitive mind to be able to apply the background knowledge to specific instances.

Re:obligatory?

[jc42]

the computer doesn't want any beer, no matter how much you think it does. NEVER, EVER feed your computer beer.

OTOH, when it asks for a cookie, you often find that you have to go along with the request, or the thing you tried to do just won't work quite right.

Re:obligatory?

[c_sd_m]

But if you give your computer a cookie, it's going to ask for a glass of milk...

Re:obligatory?

[EvanED]

Kernel hacking is taught at major universities, but under the title "Operating Systems".

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java. We did no kernel programming in any class I've taken so far, except for the graduate OS class at my graduate institution, and that was only because that was the project I happened to pick.

You learn OS theory, sure, and you almost need an understanding of a lot of that to know all of what's going on around the Linux kernel, but you're not going to learn about the Linux kernel in an OS class.

Re:obligatory?

[ozmanjusri]

Kernel hacking is taught at major universities

Yes, I understand there's a professor Tannenbaum at the Vrije University in the Netherlands who will discuss the Linux kernel in depth, if you ask him.

Re:obligatory?

[Yaztromo]

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java.

I'm currently the 3rd year Undergraduate Operating Systems instructor at a big University, and while I'd love to have my students do their work with a real kernel (and preferably an Open Source kernel, like Linux, BSD, or Xnu), it just isn't feasible. First off, the student's C is somewhat weak -- by the time they get to me, they've spent most of their time working in Java. Giving them 2 - 3 week long assignments hacking the Linux kernel would absolutely brutalize them. In my case, I have have a lab issue -- the department hasn't assigned a lab to the course, so I don't have a common system they can do their work on. And even if I did, the IT department probably wouldn't be too fond of them having the ability to recompile and load their own kernels (although this could be mitigated by having them run entirely within a VM -- if we had an assigned lab for the course). And finally, the burden on the grad student marker to be able to mark such assignments would be rough.

The only way I can think to make such a course work (at least where I'm teaching) would be to ask the students to study and explain how various OS subsystems work. I'm all for doing such a thing, but my department wants the students to do programming assignments (note that I'm just a lowly Instructor -- I'm not a tenured Professor).

In the end, however, I don't think that it's realistic for me to expect my students to be able to write an OS once they get out, as few (if any) ever will. My stated goals for them are to have them be able to understand how OS's work, so that they can a) write code that interfaces correctly with the system (API/system calls, IPC, memory management, etc.), and b) be able to compare and contrast different aspects of different OS's, and recommend the best OS for a given task.

Now ideally, my course would then segue to a more advanced kernel-hacking course for those who are sufficiently motivated to take it. However, I doubt many Universities have a suitable practical kernel hacking Instructor/Professor on staff. I'd love to be able to teach such a course, but my practical Linux kernel experience doesn't really extend beyond make clean;make menuconfig;make dep;make;make modules;make install.

Such a course would be cool -- I just imagine many Universities lack the expertise in house to offer such a course.

Yaz.

Re:obligatory?

[nacturation]

But if you give your computer a cookie, it's going to ask for a glass of milk... And that explains why some people look at breasts all day on their computers?

Re:obligatory?

[yada21]

Back when it was all diallup, the interweb was a series of straws.

Re:obligatory?

[Yaztromo]

I fear for the future of humanity if all Universities are doing is churning out mediocre Java developers and then cutting out essential parts of the course because it doesn't fit the Javarific world they've chosen for themselves. Tannenbaum would be spinning on his bicycle.

I don't disagree at all. By the time they get to me, they will have had a little bit of C experience, but it's just a perhaps month-long module in a course on general Unix development (which also includes Unix commands, Perl, and CVS). I am having the tutorial leader teach them various C stuff in their weekly tutorial sessions, which is a start I suppose.

And while we're complaining, I'll also note that I'm introducing a paper to the course this term, as I've found with some previous experience at this University that the report writing abilities of the undergrad students here are pretty weak. I can't exactly blame the students however, as by-and-large most of their courses in Computer Science are strictly programming courses.

I imagine we can find problems with the curriculum at pretty much any University, but the one I'm at needs to do more advanced low-level programming and more report writing at the lower year levels IMHO.

Yaz.

Re:obligatory?

[cecille]

I had the complete opposite experience at my university. Our instructor was fantastic and INCREDIBLY knowledgeable - he actually wrote part of the original BSD kernel back in the day. We were lucky because they did set up computers for us to play with - dual boot windows and freeBSD. Students were given the root password for the freeBSD, with the full knowledge that the installs WOULD be messed up and and have to be repaired pretty frequently. We didn't have individual accounts on freeBSD (used linux in CS and hpux on central), so we were warned about leaving files around etc. It worked out well actually. Assignments were fairly simple, usually only a few lines of code, stuff like replacing the missing parts of drivers, setting up simple things etc. They'd take maybe 30 mins if you knew what you were doing, much MUCH longer if you didn't quite get it. Overall very informative.

Anyway, there are a lot of universities that DO offer this kind of thing - that might be a good place to go. A lot of times they will let you take courses on their own if you jump through some hoops.

Re:obligatory?

[quarkie68]

I am a bit surprised with what I read, as I happen to know many academic institutions (UK based) that do teach the inner parts of the Linux kernel to final year undergrads. Some courses are a little bit out of date (teaching 2.4 kernel stuff) but one could get the basic idea and then find references (see latter paragraphs) to jump from 2.4 to 2.6 . In fact, I graduated from one of them 7 years ago and I do remember the long nights behind a source editor looking at kernel structures and making the kernel panic in all sorts of ways :-P .

However, I would agree that the C programming skills are in a downturn, something reflected by the fact that most courses are geared towards application programming and systems programming (not the digital electronics and the C to interface to microcontrollers) seems to be heading towards a new area51 tag :-) .

Anyways, to respond to the original question, I would say, from my own humble experience, that kernel programming is not something that you could digest easily, even in a semester course, especially in your final year (I am not trying to put you off, read on). Not only because, as correctly said, there is relative lack of intermediate C programming skill in the majority of uni courses. Even if you were an intermediate C programmer, there is too much to grasp, from OS theory, down to programmming style and the way things are done in Linux (there is a different C programming style for applications, and a different one for kernel system calls). So, what to do to overcome this issue:

1)Make sure you get your hands on:

i) Linux Kernel Development (2nd Edition) by Robert Love

ii)O Reilly Practical C Programming, the 3rd Edition.

iii)A 2.6 based linux kernel distro.

2)Start with Robert Love's book. Read chapters 1-4, up to the point you understand the theory behind system calls.

3)Make sure you then read systematically the C programming book, especially the chapters about pointers, structures and linked lists (advanced pointers). At the same time, make sure you get frequently into the process of compiling your own kernel.

At this point, continue with the rest of the chapters of Robert Love's group. THEN it is the right time, when you have an idea, to look for a course, and you will get a lot more from it, having done your reading. Make sure to get in touch with your local LUG (Linux User Group). There must be somebody there that shares your interests that can inform you about courses or assist you and others to go deeper into the kernel.

GM

Re:obligatory?

[the_ed_dawg]

The replies to this comment seem to be along the lines of "they don't teach that at my university, so you're wrong." As a clarification to the parent, it should be noted that the CS 503 course taught at Purdue (cide1's and my university) has a lab where you have to implement a scheduler, a full virtual memory subsystem with demand paging, and a file system in the Xinu operating system on real hardware. If you break something, the only feedback you get is three beeps and a new boot prompt about 30 seconds later. No debugger. Non-deterministic execution. Minimal crash information. Anger, hate, and suffering. It's remarkably similar to my experience with Linux kernel hacking.

The CS 503 course provides a (relative to Linux) small implementation effort that provides a sample of what programming in a major OS kernel is like. From that perspective, cide1's comment is dead on. You can't expect a college course to give you the full story. At some point, you're going to have to teach yourself. Personally, I am content that I can read "Understanding the Linux Kernel" and "Linux Kernel Development" and understand all the different concepts without having to refer to another book.

Re:obligatory?

[akar_naveen]

Prof. Raphael Finkel (about)has been offering a "Linux Internals" course for a few years now. He started it with the 2.4 kernel with Understanding the Linux Kernel (2nd Edition) by Bovet & Cesati as the text book; he's even updated the course to the 2.6 kernel and uses both Robert Love's Linux Kernel Development (2nd edition) and Bovet & Cesati(3rd Edition) as references. Check out the syllabus here:

http://www.cs.uky.edu/~raphael/courses/CS585.html

I think this is one of the best Linux kernel training courses you can get anywhere.

Re:obligatory?

[Wooloomooloo]

If you mean "novelty" as in "the palm of the hand", you might have a point.

Sorry to say

[darthgnu]

Sorry to say, but experience is very often the best teacher. Experience being mistakes with time.

Hire someone

[bscott]

Truly "advanced" courses are hard to come by, because of limited appeal and other factors. The further you go into something the more inevitable specialization becomes, so it's not just a matter of offering one course for the tenth-of-a-percent of the market who's even heard of the Linux Kernel, but in all likelihood offering half a dozen ranging from security to device drivers to assembler optimization and so on. (and I'm just guessing here, knowing next to nothing about the kernel myself)

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...) for some lessons. Keep in mind that a good programmer is not the same as a good teacher, but if you find someone who can explain things the way you need to hear them then you won't need that many lessons to make a lot of progress - the cost could very well end up in the same league as a commercially-vended course.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Re:Hire someone

[acidrain]

Keep in mind that a good programmer is not the same as a good teacher

When it comes to this level of specialization you take what you can get, and be happy about it. There is a reason many Universities have brilliant professors doing incredible research who also happen to be poor teachers. (At least in the sciences.)

Re:Hire someone

[bfields]

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...)

The gender ratio is pretty extreme, but it's not 100%--there *are* expert female kernel hackers.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Personal tutoring is a pretty expensive way to get an education, especially if it's in a fast-moving field whose experts are in demand for other work.

Off the top of my head:

• Volunteer, and find a problem to work on.
• Find someone to hire you to do kernel work, or some other way to work with people doing what you want to do.
• Have you considered grad school? There's places where you could get a degree taking operating systems classes and hacking the kernel for your dissertation. And when you're done you'll probably have an easier time with the previous item.
• Local user groups and universities might be good places to meet up with people who share your interests. Maybe start a study group to learn some kernel subsystem together?
• Conferences: OLS and linux.conf.au are fun. The OLS papers at least are on line if you can't go.
• Mailing lists, irc channels--look at kernelnewbies, etc.
• Books: Linux Device Drivers, Understanding the Linux Kernel, Robert Love's book.
• Read the code!

And if people have told you that "the best method would be to dig into the kernel myself",... actually, in the end, it's probably the *only* way. There's a certain point in your study of any field where you just run out of "courses". That's good. It means you're ready to do real work, because you're at the point where people are still busy doing the work and figuring stuff out, and haven't yet figured out how to break it down into manageable chunks and explain it in a logical order, which is a great deal of work in and of itself.

Re:Hire someone

[azhrei_fje]

I've been teaching Linux Kernel Internals and Linux Device Driver courses (among others) since 1995 and other Un*x topics prior to that.

Many of the big names "outsource" their very technical training to third parties unless the topic is something related to an internal project (I won't name those companies here, but suffice to say that their names are abbreviations and usually three characters or less in length :)).

These training classes will cost the company $1k-$2k per student, depending on the exact nature of the course. This covers a 5-day class, 7-8 hours per day, with labs as practical for each topic. Obviously, in 5 days you're not going to get a lot of depth in any single topic, but a good instructor will be able to answer off-the-cuff questions during breaks between topics. I know that in a 10-15 minute break, I often get about 3 minutes to make a run to a restroom and spend the rest of the time going over details with one or more students. The most common areas of questions include processor scheduling (big changes in this area right now), virtual memory implementation details (especially the slab allocator and zoned memory concepts), and the block layer API. We don't get heavy into implementation details, but the student is expected to have at least some C background so that they can accomplish the lab exercises.

Some customers require detailed knowledge about specific subsystems and I will add a "chalk talk" ("dry erase talk"?) as time permits to cover those areas as much as possible. For example, a company that makes video poker machines running Linux might want details on hacking the interrupt handlers, while a company that builds disk storage units might want to talk about how best to support a custom RAID controller. Those types of things come up primarily in the Linux Device Driver course; the Internals course typically comes first in the curriculum and can be applied by application and system programmers to the code they write.

Most individuals are not going to be able to afford the associated costs, however. There are some training companies that offer "public" courses: I do those classes as well as on-site classes, but public classes for Internals don't often happen because there's not enough interest to fill the classroom with warm bodies. Send me a message if you're looking for such classes and I'll give you a list of vendors. If you opt to go the less expensive route, I suggest you get Robert Love's book on the Linux Kernel; overall it's a great book, but it does lack depth in one or two areas. After you understand what he covers there, then download the free book on the Linux VMM from

Sorry, I'm rambling. I'll now return you to your regularly scheduled pr0n viewing. :)

Re:Hire someone

[bfields]

"The gender ratio is pretty extreme, but it's not 100%--there *are* expert female kernel hackers." Like who? ;)

I don't know how fair it is to single people out--they probably just want to get on with their work without being made examples of. But Val Henson (chunkfs), Mingming Cao (ext3/ext4), and Suparna Bhattacharya (aio, various fs hacking) are three that come to mind immediately.

And the fact that you may not know their names if you aren't a kernel developer doesn't mean much. I don't think most people realize how large the Linux kernel project is at this point....

Re:Harden ones understanding?

[Frogbert]

Sounds wacky.. or tacky .. As opposed to your sig?

Umm, we're programmers

[QuantumG]

We don't get training. Unless it is in how to use revision control systems or avoid sexual harassment lawsuits.

Compare this with, say, a DBA or a network engineer. They get training which is actually relevant to their job.

Re:Umm, we're programmers

[QuantumG]

Yep, we're just like artisans. Toiling away without any standards. No codification of best practices. We even have our little cults: extreme programming, object and aspect oriented programming, spiral and waterfall methodologies, not to mention the plethora of programming languages. When programmers get old we don't put them into teaching positions to pass on their arcane knowledge, we make them managers and prohibit them from even looking at the code.

It's not surprising the software is so crumby, it's only surprising that it works at all.

Reads like a mailinglist posting by some H1B

[Gothmolly]

Hello all, please give me, a foreign consultant with no work experience but an H1B visa, all your knowledge and a step by step instruction on how to do your job, so that I can displace you in yours, since your boss only looks at buzzwords on a resume anyways.
I await your full attention to this matter,
Samir Nagheenanajar

Re:Reads like a mailinglist posting by some H1B

[unlametheweak]

When I did tech support I worked with a lot of East Indians who would probably find that snippet quite amusing. I never did find humour offensive, but I realize that humour (along with any other art form) will always insight the Politically Correct.

It would be nice if I could feel free to express myself and give a bit of humour to the world without worrying about offending somebody. Unfortunately there will always be the intolerant and Politically Correct among us who will project there own anxieties on other people.

Say anything good, bad or neutral that involves a "race", and that statement can be said to be "racial", but "racist" implies intolerance and dis-respect, and this is something I just don't see. Perhaps this is because I view ethnic and phenomic differences as trivial, and yet view humour and the creative use of words and ideas as a very important part of my personality. The sad thing is, is that I don't use humour a lot because I know there will always be some people who just don't "get it". So I just give up :(

Just start implementing

[Anonymous Coward]

Just start implementing in the stable kernel and as each of your inputs is rejected, you'll learn. I hope you've had - at least - an operating systems design class (or equiv experience), and you don't try to implement something in kernel space that should be in user space.

BTW, I'm not qualified beyond hacking the IP stack a few years ago with a search/replace, use your imagination for what text was removed/replaced. all this, Just for Fun http://www.amazon.com/Just-Fun-Story-Accidental-Re volutionary/dp/0066620732 It appears someone else had more time for fun than I.

No, experience is the worst teacher

[schon]

experience is very often the best teacher Experience is the worst teacher, because it's always "test first, lesson afterwards."

Re:No, experience is the worst teacher

[misleb]

Experience is the worst teacher, because it's always "test first, lesson afterwards."

That just isn't true! Experience should include a lot of independent research and planning. It isn't like you're just blindly trying things to see if they work or not. The only real difference between school learning and experience based learning is that there are real things at stake when learning by experience. And that can be a great motivator.

-matthew

Re:No, experience is the worst teacher

[misleb]

It motivates you to go for the quick and easy solution instead of the right solution. And as the gp said, in the end that will cause you to go with the "test first, lesson afterwards (if at all)"

It depends on the environment, how much pressure you're under, and how much you know going into the situation. What you say is true if you throw someone who doesn't know much into a high pressure situation with high stakes. They're going to find the easiest and quickest solution possible. No question about that. But if a person is consistently pushed only a little bit beyond their capabilities with reasonable demands and stakes, I'm confident that it can be a constructive learning experience. This is even employed in school. You're given reasonable tasks/projects to gain experience. But eventually you outgrow the the kind of experience that a school can offer. Eventually you need to go into the real world. You might start as an intern, or a junior programmer, for example, until you learn from enough experience to move on....

Even in our young industry there is a wealth of knowledge out there on how to best do things. To throw it out in the guise of "I'll experience it all myself" is just plain wasteful.

A solid education is important as a foundation. I'd never dispute that.

-matthew

Teaching and doing

[lawpoop]

Well, good luck. Have you ever heard the saying, "Those who can, do; those who can't, teach" ? I don't agree with it , but there is a kernel of truth. You've probably had a professor who was a genius, and and expert in their field, but couldn't teach worth a damn. You've probably also learned from someone who was a good teacher, but didn't know their stuff, or didn't have the resources to teach it properly.

Finding someone who is an expert in the linux kernel, *and* who can teach, and has the time and willingness to teach you one-on-one, will be a rare find indeed. ( Are you willing to pay them what they're worth for their background and ability? )

That person has probably already written a book.

Re:Harden ones understanding?

[mr_stinky_britches]

As opposed to your sig?

Precisely!

--
Wi-Fizzle Fo' Shizzle Dizzle

Teach

[ziah]

Teaching is the best way to learn. I was a computer science tutor up at Berkeley and I learned FAR MORE from tutoring than I did from the classes. Find someone who is interested in the linux kernel and teach them. If they're smart they'll ask you questions you've never thought about asking, which in turn will end up solidifying your knowledge.

RTFML

[aero6dof]

Read the LKML archive?

rule number 1: get the name right

[Karma Sucks]

First of all, it's GNU/Linux kernel not Linux kernel... I remember reading something about this and why getting the name right is important but can't find the link anymore. Cheers!

Re:rule number 1: get the name right

[dhasenan]

Whoosh.

Re:rule number 1: get the name right

[dbIII]

Many people will not spot the joke so I will point out that linux is not a gnu project.

There was a move a few years ago to get distributions of linux, gnu tools, X windows and other bits renamed to LiGnuX to draw attention the the gnu project. This wasn't taken seriously so later there was a move to put a gnu prefix on before linux in the name of distributions - only Debian took it up as far as I know. Newbies that missed the point also started putting gnu in front of any reference to a distribution containing linux and then any reference to linux at all.

The best advanced kernel course I have found

[12AU7A]

The very best course I have found is a ~32 hour DVD course on the FreeBSD kernel internals and: Advanced FreeBSD Kernel Code Walkthrough Videos I've never found anything more thorough.

Re:Getting Training vs. Getting Good

[ezratrumpet]

Failure comes wrapped in many packages. I'd hate to think my desire to have someone who could handle setbacks and failures would cause me to overlook a person who had failed in other careers, been kicked to the curb in other disciplines, and finally found their place in the sun, so to speak.

Every sorcerer was once an apprentice. Every wizard was once a n00b. Even Linus, Bill, and Steve.

Some classes on the left coast

[rlh100]

Out here on the left coast the extension programs at the various University Of California campuses have some Linux Kernel classes. These tend to be developed and taught by engineers in the industry with a real working knowledge of the subject.
UC Santa Cruz Extension, http://www.ucsc-extension.edu/ has an "Linux Kernel Architecture and Programming" which looks like an intro course. You can take it online or as two Saturdays. There is also a Linux device drivers class which a once a week class and an Advanced Device Drivers class which is 4 Saturdays.
I checked the other campuses but they all seem to be summer schedule with a limited set of classes.
Red Hat also has a one week Kernel internals class which is a "hands on" which to me means a trade off of less information for some finger programing of the brain.
All of these courses seem to have an introductory flavor to them. But I suspect that you will learn a lot about all of the various areas of the kernel and how the different parts hang together. My experience as kernel hacker is that I have learned a lot about the parts I am interested in, but that there are many big areas of the kernel that I only have a superficial understanding of.

Hope this helps
RLH

Not trying to be an asshole, but...

[cdw38]

...this made the front page? The course is called "Operating Systems," and it's taught at major universities, as someone previously mentioned. In many of these courses, the semester-long project is to develop your own distribution of Linux (as a class or group).

Red Hat Courses

[Erisian Pope]

I may be a little biased since I've taught the courses... but...

If what you're looking for is the typical corporate 1 week training course, then IMHO you can't do much better. But! Let's be honest here, what can you get out of a one week course? If it's a serious course, like these are, you're going to get so much information that you'll have trouble staying afloat. You're not going to master these skills in a week. Depending on your programming skills and operating systems background you're looking at months to years before you're comfortable kernel hacking. What these courses can give you is a lot of solid coding examples to build your skills. But don't take my word for it, the code samples are available at ftp://axian.com/pub/RHD_SOLUTIONS. Oh, and of course the Red Hat info: http://www.redhat.com/training/developer/courses/

If you're not in the typical corporate time crunch mode, I'd definitely recommend college courses. Get some general background in OS design while you're at it if you don't already have it. Oh, and LUGs, lots of lugs have groups doing kernel hacking, not a bad place to start there either, plus LUGs don't cost anything!

Operating Systems understanding + books + practice

[edis]

I doubt your target for course is correct. Having been trough after-studies Operating Systems course like 20 years ago, centering on RSX-11, I still feel solid in concepts of today's OSes - Linux, too.

For good professional background, I would recommend you to get also good books. On OS principles themselves, and on Linux Kernel. I have been printing and binding myself freely distributable online David Rusling book "The Linux Kernel" (that over 10 years ago). It mentions even Alpha processor view of things - very good for broader understanding. Love this book, very special. Also in my library sits book of Linux kernel anatomy: too lazy to go upstairs for exact title, but it must have been published by SAMS, and is book analyzing specifically code, that makes Linux Kernel, there are sources of very first versions of kernel included too, which should help to understand evolution of kernel as final touch. And plenty of code and discussion of it. Author might be not available for classes, but book is serious alternative for such.

Good luck! OS things deserve your attention.

Re:Operating Systems understanding + books + pract

[dgp]

i second the good books notion and thanks for the reference to Rusling's book - I haven't heard of that one.

I can say that Robert Love's book, Linux Kernel Development (2004) is very well written and easy to read and understand. (print only)

There is also Greg Kroah-Harman's Linux Kernel in a nutshell at http://www.kroah.com/lkn/ - free PDF download.