Best Advanced Linux Kernel Training?

hdxia writes "Can anybody recommend a good Linux kernel training course? I have had some Linux kernel hacking experience, but would like to further harden and improve my understanding of the kernel. I expect the course would be advanced. You may say that the best method would be to dig into the kernel myself, but I really want to have a chance to discuss and learn all aspects of the kernel with an experienced instructor."

Comments

Sorry to say

[darthgnu]

Sorry to say, but experience is very often the best teacher. Experience being mistakes with time.

Hire someone

[bscott]

Truly "advanced" courses are hard to come by, because of limited appeal and other factors. The further you go into something the more inevitable specialization becomes, so it's not just a matter of offering one course for the tenth-of-a-percent of the market who's even heard of the Linux Kernel, but in all likelihood offering half a dozen ranging from security to device drivers to assembler optimization and so on. (and I'm just guessing here, knowing next to nothing about the kernel myself)

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...) for some lessons. Keep in mind that a good programmer is not the same as a good teacher, but if you find someone who can explain things the way you need to hear them then you won't need that many lessons to make a lot of progress - the cost could very well end up in the same league as a commercially-vended course.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Re:Hire someone

[acidrain]

Keep in mind that a good programmer is not the same as a good teacher

When it comes to this level of specialization you take what you can get, and be happy about it. There is a reason many Universities have brilliant professors doing incredible research who also happen to be poor teachers. (At least in the sciences.)

Re:Hire someone

[bfields]

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...)

The gender ratio is pretty extreme, but it's not 100%--there *are* expert female kernel hackers.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Personal tutoring is a pretty expensive way to get an education, especially if it's in a fast-moving field whose experts are in demand for other work.

Off the top of my head:

• Volunteer, and find a problem to work on.
• Find someone to hire you to do kernel work, or some other way to work with people doing what you want to do.
• Have you considered grad school? There's places where you could get a degree taking operating systems classes and hacking the kernel for your dissertation. And when you're done you'll probably have an easier time with the previous item.
• Local user groups and universities might be good places to meet up with people who share your interests. Maybe start a study group to learn some kernel subsystem together?
• Conferences: OLS and linux.conf.au are fun. The OLS papers at least are on line if you can't go.
• Mailing lists, irc channels--look at kernelnewbies, etc.
• Books: Linux Device Drivers, Understanding the Linux Kernel, Robert Love's book.
• Read the code!

And if people have told you that "the best method would be to dig into the kernel myself",... actually, in the end, it's probably the *only* way. There's a certain point in your study of any field where you just run out of "courses". That's good. It means you're ready to do real work, because you're at the point where people are still busy doing the work and figuring stuff out, and haven't yet figured out how to break it down into manageable chunks and explain it in a logical order, which is a great deal of work in and of itself.

Re:Hire someone

[azhrei_fje]

I've been teaching Linux Kernel Internals and Linux Device Driver courses (among others) since 1995 and other Un*x topics prior to that.

Many of the big names "outsource" their very technical training to third parties unless the topic is something related to an internal project (I won't name those companies here, but suffice to say that their names are abbreviations and usually three characters or less in length :)).

These training classes will cost the company $1k-$2k per student, depending on the exact nature of the course. This covers a 5-day class, 7-8 hours per day, with labs as practical for each topic. Obviously, in 5 days you're not going to get a lot of depth in any single topic, but a good instructor will be able to answer off-the-cuff questions during breaks between topics. I know that in a 10-15 minute break, I often get about 3 minutes to make a run to a restroom and spend the rest of the time going over details with one or more students. The most common areas of questions include processor scheduling (big changes in this area right now), virtual memory implementation details (especially the slab allocator and zoned memory concepts), and the block layer API. We don't get heavy into implementation details, but the student is expected to have at least some C background so that they can accomplish the lab exercises.

Some customers require detailed knowledge about specific subsystems and I will add a "chalk talk" ("dry erase talk"?) as time permits to cover those areas as much as possible. For example, a company that makes video poker machines running Linux might want details on hacking the interrupt handlers, while a company that builds disk storage units might want to talk about how best to support a custom RAID controller. Those types of things come up primarily in the Linux Device Driver course; the Internals course typically comes first in the curriculum and can be applied by application and system programmers to the code they write.

Most individuals are not going to be able to afford the associated costs, however. There are some training companies that offer "public" courses: I do those classes as well as on-site classes, but public classes for Internals don't often happen because there's not enough interest to fill the classroom with warm bodies. Send me a message if you're looking for such classes and I'll give you a list of vendors. If you opt to go the less expensive route, I suggest you get Robert Love's book on the Linux Kernel; overall it's a great book, but it does lack depth in one or two areas. After you understand what he covers there, then download the free book on the Linux VMM from

Sorry, I'm rambling. I'll now return you to your regularly scheduled pr0n viewing. :)

Re:obligatory?

[cide1]

Kernel hacking is taught at major universities, but under the title "Operating Systems". These classes teach scheduling, device drivers, file systems, memory management, networking, etc. The goal of a university is to give you the background knowledge necessary to understand specific implementations. The specifics of the linux kernel change rapidly, but the concepts are 40 years old. No instructional course has a chance of keeping up, the kernel documentation cant even keep up. It takes a problem solver, and an inquizitive mind to be able to apply the background knowledge to specific instances.

Re:Umm, we're programmers

[QuantumG]

Yep, we're just like artisans. Toiling away without any standards. No codification of best practices. We even have our little cults: extreme programming, object and aspect oriented programming, spiral and waterfall methodologies, not to mention the plethora of programming languages. When programmers get old we don't put them into teaching positions to pass on their arcane knowledge, we make them managers and prohibit them from even looking at the code.

It's not surprising the software is so crumby, it's only surprising that it works at all.

No, experience is the worst teacher

[schon]

experience is very often the best teacher Experience is the worst teacher, because it's always "test first, lesson afterwards."

Re:No, experience is the worst teacher

[misleb]

Experience is the worst teacher, because it's always "test first, lesson afterwards."

That just isn't true! Experience should include a lot of independent research and planning. It isn't like you're just blindly trying things to see if they work or not. The only real difference between school learning and experience based learning is that there are real things at stake when learning by experience. And that can be a great motivator.

-matthew

Re:No, experience is the worst teacher

[misleb]

It motivates you to go for the quick and easy solution instead of the right solution. And as the gp said, in the end that will cause you to go with the "test first, lesson afterwards (if at all)"

It depends on the environment, how much pressure you're under, and how much you know going into the situation. What you say is true if you throw someone who doesn't know much into a high pressure situation with high stakes. They're going to find the easiest and quickest solution possible. No question about that. But if a person is consistently pushed only a little bit beyond their capabilities with reasonable demands and stakes, I'm confident that it can be a constructive learning experience. This is even employed in school. You're given reasonable tasks/projects to gain experience. But eventually you outgrow the the kind of experience that a school can offer. Eventually you need to go into the real world. You might start as an intern, or a junior programmer, for example, until you learn from enough experience to move on....

Even in our young industry there is a wealth of knowledge out there on how to best do things. To throw it out in the guise of "I'll experience it all myself" is just plain wasteful.

A solid education is important as a foundation. I'd never dispute that.

-matthew

Teaching and doing

[lawpoop]

Well, good luck. Have you ever heard the saying, "Those who can, do; those who can't, teach" ? I don't agree with it , but there is a kernel of truth. You've probably had a professor who was a genius, and and expert in their field, but couldn't teach worth a damn. You've probably also learned from someone who was a good teacher, but didn't know their stuff, or didn't have the resources to teach it properly.

Finding someone who is an expert in the linux kernel, *and* who can teach, and has the time and willingness to teach you one-on-one, will be a rare find indeed. ( Are you willing to pay them what they're worth for their background and ability? )

That person has probably already written a book.

Teach

[ziah]

Teaching is the best way to learn. I was a computer science tutor up at Berkeley and I learned FAR MORE from tutoring than I did from the classes. Find someone who is interested in the linux kernel and teach them. If they're smart they'll ask you questions you've never thought about asking, which in turn will end up solidifying your knowledge.

RTFML

[aero6dof]

Read the LKML archive?

Re:Getting Training vs. Getting Good

[ezratrumpet]

Failure comes wrapped in many packages. I'd hate to think my desire to have someone who could handle setbacks and failures would cause me to overlook a person who had failed in other careers, been kicked to the curb in other disciplines, and finally found their place in the sun, so to speak.

Every sorcerer was once an apprentice. Every wizard was once a n00b. Even Linus, Bill, and Steve.

Re:obligatory?

[c_sd_m]

But if you give your computer a cookie, it's going to ask for a glass of milk...

Not trying to be an asshole, but...

[cdw38]

...this made the front page? The course is called "Operating Systems," and it's taught at major universities, as someone previously mentioned. In many of these courses, the semester-long project is to develop your own distribution of Linux (as a class or group).

Re:Reads like a mailinglist posting by some H1B

[unlametheweak]

When I did tech support I worked with a lot of East Indians who would probably find that snippet quite amusing. I never did find humour offensive, but I realize that humour (along with any other art form) will always insight the Politically Correct.

It would be nice if I could feel free to express myself and give a bit of humour to the world without worrying about offending somebody. Unfortunately there will always be the intolerant and Politically Correct among us who will project there own anxieties on other people.

Say anything good, bad or neutral that involves a "race", and that statement can be said to be "racial", but "racist" implies intolerance and dis-respect, and this is something I just don't see. Perhaps this is because I view ethnic and phenomic differences as trivial, and yet view humour and the creative use of words and ideas as a very important part of my personality. The sad thing is, is that I don't use humour a lot because I know there will always be some people who just don't "get it". So I just give up :(

Re:Operating Systems understanding + books + pract

[dgp]

i second the good books notion and thanks for the reference to Rusling's book - I haven't heard of that one.

I can say that Robert Love's book, Linux Kernel Development (2004) is very well written and easy to read and understand. (print only)

There is also Greg Kroah-Harman's Linux Kernel in a nutshell at http://www.kroah.com/lkn/ - free PDF download.

Re:obligatory?

[the_ed_dawg]

The replies to this comment seem to be along the lines of "they don't teach that at my university, so you're wrong." As a clarification to the parent, it should be noted that the CS 503 course taught at Purdue (cide1's and my university) has a lab where you have to implement a scheduler, a full virtual memory subsystem with demand paging, and a file system in the Xinu operating system on real hardware. If you break something, the only feedback you get is three beeps and a new boot prompt about 30 seconds later. No debugger. Non-deterministic execution. Minimal crash information. Anger, hate, and suffering. It's remarkably similar to my experience with Linux kernel hacking.

The CS 503 course provides a (relative to Linux) small implementation effort that provides a sample of what programming in a major OS kernel is like. From that perspective, cide1's comment is dead on. You can't expect a college course to give you the full story. At some point, you're going to have to teach yourself. Personally, I am content that I can read "Understanding the Linux Kernel" and "Linux Kernel Development" and understand all the different concepts without having to refer to another book.