Best Advanced Linux Kernel Training?

hdxia writes "Can anybody recommend a good Linux kernel training course? I have had some Linux kernel hacking experience, but would like to further harden and improve my understanding of the kernel. I expect the course would be advanced. You may say that the best method would be to dig into the kernel myself, but I really want to have a chance to discuss and learn all aspects of the kernel with an experienced instructor."

Comments

Sorry to say

[darthgnu]

Sorry to say, but experience is very often the best teacher. Experience being mistakes with time.

Hire someone

[bscott]

Truly "advanced" courses are hard to come by, because of limited appeal and other factors. The further you go into something the more inevitable specialization becomes, so it's not just a matter of offering one course for the tenth-of-a-percent of the market who's even heard of the Linux Kernel, but in all likelihood offering half a dozen ranging from security to device drivers to assembler optimization and so on. (and I'm just guessing here, knowing next to nothing about the kernel myself)

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...) for some lessons. Keep in mind that a good programmer is not the same as a good teacher, but if you find someone who can explain things the way you need to hear them then you won't need that many lessons to make a lot of progress - the cost could very well end up in the same league as a commercially-vended course.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Re:Hire someone

[acidrain]

Keep in mind that a good programmer is not the same as a good teacher

When it comes to this level of specialization you take what you can get, and be happy about it. There is a reason many Universities have brilliant professors doing incredible research who also happen to be poor teachers. (At least in the sciences.)

Re:Hire someone

[bfields]

My suggestion would be to find someone who's pretty savvy in the area you're aimed at, and hire him or her (OK, let's face it, "him"...)

The gender ratio is pretty extreme, but it's not 100%--there *are* expert female kernel hackers.

I'm just guessing that finding a kernel guru willing to give up a month of Saturday afternoons at $300 a session will be easier than finding "Linux Kernel for Experts" at the downtown Learning Annex.

Personal tutoring is a pretty expensive way to get an education, especially if it's in a fast-moving field whose experts are in demand for other work.

Off the top of my head:

• Volunteer, and find a problem to work on.
• Find someone to hire you to do kernel work, or some other way to work with people doing what you want to do.
• Have you considered grad school? There's places where you could get a degree taking operating systems classes and hacking the kernel for your dissertation. And when you're done you'll probably have an easier time with the previous item.
• Local user groups and universities might be good places to meet up with people who share your interests. Maybe start a study group to learn some kernel subsystem together?
• Conferences: OLS and linux.conf.au are fun. The OLS papers at least are on line if you can't go.
• Mailing lists, irc channels--look at kernelnewbies, etc.
• Books: Linux Device Drivers, Understanding the Linux Kernel, Robert Love's book.
• Read the code!

And if people have told you that "the best method would be to dig into the kernel myself",... actually, in the end, it's probably the *only* way. There's a certain point in your study of any field where you just run out of "courses". That's good. It means you're ready to do real work, because you're at the point where people are still busy doing the work and figuring stuff out, and haven't yet figured out how to break it down into manageable chunks and explain it in a logical order, which is a great deal of work in and of itself.

Re:Hire someone

[azhrei_fje]

I've been teaching Linux Kernel Internals and Linux Device Driver courses (among others) since 1995 and other Un*x topics prior to that.

Many of the big names "outsource" their very technical training to third parties unless the topic is something related to an internal project (I won't name those companies here, but suffice to say that their names are abbreviations and usually three characters or less in length :)).

These training classes will cost the company $1k-$2k per student, depending on the exact nature of the course. This covers a 5-day class, 7-8 hours per day, with labs as practical for each topic. Obviously, in 5 days you're not going to get a lot of depth in any single topic, but a good instructor will be able to answer off-the-cuff questions during breaks between topics. I know that in a 10-15 minute break, I often get about 3 minutes to make a run to a restroom and spend the rest of the time going over details with one or more students. The most common areas of questions include processor scheduling (big changes in this area right now), virtual memory implementation details (especially the slab allocator and zoned memory concepts), and the block layer API. We don't get heavy into implementation details, but the student is expected to have at least some C background so that they can accomplish the lab exercises.

Some customers require detailed knowledge about specific subsystems and I will add a "chalk talk" ("dry erase talk"?) as time permits to cover those areas as much as possible. For example, a company that makes video poker machines running Linux might want details on hacking the interrupt handlers, while a company that builds disk storage units might want to talk about how best to support a custom RAID controller. Those types of things come up primarily in the Linux Device Driver course; the Internals course typically comes first in the curriculum and can be applied by application and system programmers to the code they write.

Most individuals are not going to be able to afford the associated costs, however. There are some training companies that offer "public" courses: I do those classes as well as on-site classes, but public classes for Internals don't often happen because there's not enough interest to fill the classroom with warm bodies. Send me a message if you're looking for such classes and I'll give you a list of vendors. If you opt to go the less expensive route, I suggest you get Robert Love's book on the Linux Kernel; overall it's a great book, but it does lack depth in one or two areas. After you understand what he covers there, then download the free book on the Linux VMM from

Sorry, I'm rambling. I'll now return you to your regularly scheduled pr0n viewing. :)

Re:Harden ones understanding?

[Frogbert]

Sounds wacky.. or tacky .. As opposed to your sig?

Umm, we're programmers

[QuantumG]

We don't get training. Unless it is in how to use revision control systems or avoid sexual harassment lawsuits.

Compare this with, say, a DBA or a network engineer. They get training which is actually relevant to their job.

Re:Umm, we're programmers

[QuantumG]

Yep, we're just like artisans. Toiling away without any standards. No codification of best practices. We even have our little cults: extreme programming, object and aspect oriented programming, spiral and waterfall methodologies, not to mention the plethora of programming languages. When programmers get old we don't put them into teaching positions to pass on their arcane knowledge, we make them managers and prohibit them from even looking at the code.

It's not surprising the software is so crumby, it's only surprising that it works at all.

Reads like a mailinglist posting by some H1B

[Gothmolly]

Hello all, please give me, a foreign consultant with no work experience but an H1B visa, all your knowledge and a step by step instruction on how to do your job, so that I can displace you in yours, since your boss only looks at buzzwords on a resume anyways.
I await your full attention to this matter,
Samir Nagheenanajar

Re:Reads like a mailinglist posting by some H1B

[unlametheweak]

When I did tech support I worked with a lot of East Indians who would probably find that snippet quite amusing. I never did find humour offensive, but I realize that humour (along with any other art form) will always insight the Politically Correct.

It would be nice if I could feel free to express myself and give a bit of humour to the world without worrying about offending somebody. Unfortunately there will always be the intolerant and Politically Correct among us who will project there own anxieties on other people.

Say anything good, bad or neutral that involves a "race", and that statement can be said to be "racial", but "racist" implies intolerance and dis-respect, and this is something I just don't see. Perhaps this is because I view ethnic and phenomic differences as trivial, and yet view humour and the creative use of words and ideas as a very important part of my personality. The sad thing is, is that I don't use humour a lot because I know there will always be some people who just don't "get it". So I just give up :(

Re:obligatory?

[cide1]

Kernel hacking is taught at major universities, but under the title "Operating Systems". These classes teach scheduling, device drivers, file systems, memory management, networking, etc. The goal of a university is to give you the background knowledge necessary to understand specific implementations. The specifics of the linux kernel change rapidly, but the concepts are 40 years old. No instructional course has a chance of keeping up, the kernel documentation cant even keep up. It takes a problem solver, and an inquizitive mind to be able to apply the background knowledge to specific instances.

No, experience is the worst teacher

[schon]

experience is very often the best teacher Experience is the worst teacher, because it's always "test first, lesson afterwards."

Re:No, experience is the worst teacher

[misleb]

Experience is the worst teacher, because it's always "test first, lesson afterwards."

That just isn't true! Experience should include a lot of independent research and planning. It isn't like you're just blindly trying things to see if they work or not. The only real difference between school learning and experience based learning is that there are real things at stake when learning by experience. And that can be a great motivator.

-matthew

Re:No, experience is the worst teacher

[misleb]

It motivates you to go for the quick and easy solution instead of the right solution. And as the gp said, in the end that will cause you to go with the "test first, lesson afterwards (if at all)"

It depends on the environment, how much pressure you're under, and how much you know going into the situation. What you say is true if you throw someone who doesn't know much into a high pressure situation with high stakes. They're going to find the easiest and quickest solution possible. No question about that. But if a person is consistently pushed only a little bit beyond their capabilities with reasonable demands and stakes, I'm confident that it can be a constructive learning experience. This is even employed in school. You're given reasonable tasks/projects to gain experience. But eventually you outgrow the the kind of experience that a school can offer. Eventually you need to go into the real world. You might start as an intern, or a junior programmer, for example, until you learn from enough experience to move on....

Even in our young industry there is a wealth of knowledge out there on how to best do things. To throw it out in the guise of "I'll experience it all myself" is just plain wasteful.

A solid education is important as a foundation. I'd never dispute that.

-matthew

Teach

[ziah]

Teaching is the best way to learn. I was a computer science tutor up at Berkeley and I learned FAR MORE from tutoring than I did from the classes. Find someone who is interested in the linux kernel and teach them. If they're smart they'll ask you questions you've never thought about asking, which in turn will end up solidifying your knowledge.

The best advanced kernel course I have found

[12AU7A]

The very best course I have found is a ~32 hour DVD course on the FreeBSD kernel internals and: Advanced FreeBSD Kernel Code Walkthrough Videos I've never found anything more thorough.

Re:rule number 1: get the name right

[dhasenan]

Whoosh.

Some classes on the left coast

[rlh100]

Out here on the left coast the extension programs at the various University Of California campuses have some Linux Kernel classes. These tend to be developed and taught by engineers in the industry with a real working knowledge of the subject.
UC Santa Cruz Extension, http://www.ucsc-extension.edu/ has an "Linux Kernel Architecture and Programming" which looks like an intro course. You can take it online or as two Saturdays. There is also a Linux device drivers class which a once a week class and an Advanced Device Drivers class which is 4 Saturdays.
I checked the other campuses but they all seem to be summer schedule with a limited set of classes.
Red Hat also has a one week Kernel internals class which is a "hands on" which to me means a trade off of less information for some finger programing of the brain.
All of these courses seem to have an introductory flavor to them. But I suspect that you will learn a lot about all of the various areas of the kernel and how the different parts hang together. My experience as kernel hacker is that I have learned a lot about the parts I am interested in, but that there are many big areas of the kernel that I only have a superficial understanding of.

Hope this helps
RLH

Re:obligatory?

[EvanED]

Kernel hacking is taught at major universities, but under the title "Operating Systems".

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java. We did no kernel programming in any class I've taken so far, except for the graduate OS class at my graduate institution, and that was only because that was the project I happened to pick.

You learn OS theory, sure, and you almost need an understanding of a lot of that to know all of what's going on around the Linux kernel, but you're not going to learn about the Linux kernel in an OS class.

Red Hat Courses

[Erisian Pope]

I may be a little biased since I've taught the courses... but...

If what you're looking for is the typical corporate 1 week training course, then IMHO you can't do much better. But! Let's be honest here, what can you get out of a one week course? If it's a serious course, like these are, you're going to get so much information that you'll have trouble staying afloat. You're not going to master these skills in a week. Depending on your programming skills and operating systems background you're looking at months to years before you're comfortable kernel hacking. What these courses can give you is a lot of solid coding examples to build your skills. But don't take my word for it, the code samples are available at ftp://axian.com/pub/RHD_SOLUTIONS. Oh, and of course the Red Hat info: http://www.redhat.com/training/developer/courses/

If you're not in the typical corporate time crunch mode, I'd definitely recommend college courses. Get some general background in OS design while you're at it if you don't already have it. Oh, and LUGs, lots of lugs have groups doing kernel hacking, not a bad place to start there either, plus LUGs don't cost anything!

Operating Systems understanding + books + practice

[edis]

I doubt your target for course is correct. Having been trough after-studies Operating Systems course like 20 years ago, centering on RSX-11, I still feel solid in concepts of today's OSes - Linux, too.

For good professional background, I would recommend you to get also good books. On OS principles themselves, and on Linux Kernel. I have been printing and binding myself freely distributable online David Rusling book "The Linux Kernel" (that over 10 years ago). It mentions even Alpha processor view of things - very good for broader understanding. Love this book, very special. Also in my library sits book of Linux kernel anatomy: too lazy to go upstairs for exact title, but it must have been published by SAMS, and is book analyzing specifically code, that makes Linux Kernel, there are sources of very first versions of kernel included too, which should help to understand evolution of kernel as final touch. And plenty of code and discussion of it. Author might be not available for classes, but book is serious alternative for such.

Good luck! OS things deserve your attention.

Re:obligatory?

[ozmanjusri]

Kernel hacking is taught at major universities

Yes, I understand there's a professor Tannenbaum at the Vrije University in the Netherlands who will discuss the Linux kernel in depth, if you ask him.

Re:obligatory?

[Yaztromo]

We did no kernel programming in either the undergrad or graduate OS class at my undergraduate institution (I took both). It was all fairly low-level C systems programming. The undergrad OS class at my graduate institution (a top ten CS school) is the same way -- in fact, the OS class there is sometimes taught in Java.

I'm currently the 3rd year Undergraduate Operating Systems instructor at a big University, and while I'd love to have my students do their work with a real kernel (and preferably an Open Source kernel, like Linux, BSD, or Xnu), it just isn't feasible. First off, the student's C is somewhat weak -- by the time they get to me, they've spent most of their time working in Java. Giving them 2 - 3 week long assignments hacking the Linux kernel would absolutely brutalize them. In my case, I have have a lab issue -- the department hasn't assigned a lab to the course, so I don't have a common system they can do their work on. And even if I did, the IT department probably wouldn't be too fond of them having the ability to recompile and load their own kernels (although this could be mitigated by having them run entirely within a VM -- if we had an assigned lab for the course). And finally, the burden on the grad student marker to be able to mark such assignments would be rough.

The only way I can think to make such a course work (at least where I'm teaching) would be to ask the students to study and explain how various OS subsystems work. I'm all for doing such a thing, but my department wants the students to do programming assignments (note that I'm just a lowly Instructor -- I'm not a tenured Professor).

In the end, however, I don't think that it's realistic for me to expect my students to be able to write an OS once they get out, as few (if any) ever will. My stated goals for them are to have them be able to understand how OS's work, so that they can a) write code that interfaces correctly with the system (API/system calls, IPC, memory management, etc.), and b) be able to compare and contrast different aspects of different OS's, and recommend the best OS for a given task.

Now ideally, my course would then segue to a more advanced kernel-hacking course for those who are sufficiently motivated to take it. However, I doubt many Universities have a suitable practical kernel hacking Instructor/Professor on staff. I'd love to be able to teach such a course, but my practical Linux kernel experience doesn't really extend beyond make clean;make menuconfig;make dep;make;make modules;make install.

Such a course would be cool -- I just imagine many Universities lack the expertise in house to offer such a course.

Yaz.

Re:obligatory?

[yada21]

Back when it was all diallup, the interweb was a series of straws.