Behind the Scenes in Kernel Development

An anonymous reader writes "Some interesting changes took place in the way the Linux kernel is developed and tested. In many ways, the methods used to develop the Linux kernel are much the same today as they were 3 years ago. However, several key changes have improved overall stability as well as quality. This article takes a look behind the scenes at the tools, tests, and techniques -- from revision control and regression testing to bugtracking and list keeping -- that helped make 2.6 a better kernel than any that have come before it." We might as well mention here (again) that a couple of new kernels are out: leif.singer writes "2.6.3 and 2.4.25 are out, fixing another vulnerability in do_mremap()."

Comments

Kernel development interests me terribly

[ObviousGuy]

I wish I could wrap my head around even the smallest part of the kernel. There is so much code in there and aside from main(), it is hard to find a good place to start studying.

Would these tests be a good starting place?

Re:Kernel development interests me terribly

[deadlinegrunt]

Find a particular functionality of the kernel that really interest you; read any documentation you can find about it; then grep the src till you see the relevant sections of code and start perusing with your $(EDITOR)

Much time is spent teaching people how to write code but never really reading it. This is a perfect example of how to do it and why you would.

Re:Kernel development interests me terribly

[millahtime]

"Much time is spent teaching people how to write code but never really reading it. This is a perfect example of how to do it and why you would."

Reading code can be a huge help in becomeing a better coder. You see how other coders do things. Learning from their bad on what not to do and seeing new good methods you may not have come up with on your own.

Re:Kernel development interests me terribly

[Rosco P. Coltrane]

I wish I could wrap my head around even the smallest part of the kernel. There is so much code in there and aside from main(), it is hard to find a good place to start studying.

You could contribute some work to SCO: I hear they're very interested in having someone sprinkle several "printk("(c) SCO\n");" lines here and there in init/main.c, since they can't do it themselves, having no technical department, being a law firm and all...

Re:Kernel development interests me terribly

[tcopeland]

Code Reading by Diomidis Spinellis contains a bunch of ideas on ways to comprehend large codebases more easily.

He talks about browsing code, package structures, adding features or fixing bugs in a large codebase, and so on. It's a good read - well worth the money.

Re:Kernel development interests me terribly

[po8]

Does the fact that Diomidis Spinellis has repeated won the International Obfuscated C Code Contest (IOCCC) make him more or less qualified to write such a book :-)? Check out his "best abuse of the rules" entry from 1988 that is my all-time favorite. BTW, the contest is currently open.

Kernel quality

[Rosco P. Coltrane]

However, several key changes have improved overall stability as well as quality.

I have a suggestion : how about not calling development kernels with an even version number?

- 2.6.0-beta-something kernels were bad (okay fair enough, it was beta, and Linus admitted having called a 2.5.x kernel 2.6 in order to lure early adopters and get them to test it).

- 2.6.0, 2.6.1 and 2.6.2 were unstable for me, with doozies such as oopses while rmmoding and random crashes using ide-scsi (yes I know it's deprecated, but some of us need it).

I now run 2.6.3-rc3 and it's the first time it seems stable enough to be called a 2.6 kernel. There are some problems left, but overall it's getting decent. But then why are the others "2.6" kernel called 2.6 at all? they were really 2.5 kernels imho.

This has happened before, with the beginning of the 2.4 serie. I only felt it was getting good enough at version 2.4.6 and above (I'm not counting the failed 2.4.11 release). When 2.4.0 went out, I thought it meant it was ready for prime time, like 2.2.0 was, or at least was more, but no it was crap. I was slightly annoyed with Linus then, but I thought he had been pressured by commercial Linux shops and that he wouldn't do it again. But no, he did it again with 2.6.

It's really quite annoying, because those who follow Linux know the first "stable" kernels aren't stable at all, therefore avoid it, therefore defeat the point of testing it for Linus, but beginners think "cool, a new stable kernel", try it and are disappointed, giving a bad name to an otherwise great kernel. Too bad ...

Re:Kernel quality

[Psiren]

This is a chicken and egg situation. Unless there is widespread testing of a kernel, some bugs won't be found. But not everyone wants to risk running a development kernel, so the only way to get them to test is to bend the truth slightly, and call a beta version the new stable kernel. At the end of the day, the number just reflects the developers opinion on the stability of the thing as a whole. They could make no changes to 2.6.3 and release it as 2.7.0, but that wouldn't make it any less unstable.

Re:Kernel quality

[Anonymous Coward]

and random crashes using ide-scsi (yes I know it's deprecated, but some of us need it).

disable ide-scsi and use latest cdrecord with

dev=ATAPI:x,x,x

instead of

dev=x,x,x

and everything is cool. Don't forget to check lilo.conf. Stop using cdrdao. Ignore xcdroast's "This would be faster is you had ide-scsi enabled" dialogs.

This should apply to almost every kind of ide-scsi use.

Re:Kernel quality

[adrianbaugh]

That's exactly what Linus does: there is such a series of release candidates (first introduced prior to 2.4). You can argue that it isn't long enough, but there's an obvious counterargument that if you wait forever nothing will ever get released.
I can't think of a x.y.0 release of any software project that's been properly stable. It's not just linux, it's the way the world is. You could argue that software never ever becomes perfectly stable: marking a series as "stable" is really just shorthand for good enough that further development is largely maintenance, therefore we expect the structure and codebase to remain stable", not some guarantee that they'll never go wrong. It's more a development term than a performance or reliability term, though the stability of development generally arises from the performance and reliability being sufficient to obviate the need for large changes to the code.

Even quite late in stable kernel release cycles there's occasionally a shocker - anyone remember 2.0.33?

If you don't like those kernels, just stick with 2.4 until a distribution ships with 2.6.8 or so. For what it's worth 2.6.(1+) has been fine for me.

Nobody's lying to you - there has to be some cut-off where a kernel series is declared stable, and by and large I think Linus judges it pretty well.

Post: -1, Redundant

[HoldmyCauls]

"2.6.3 and 2.4.25 are out, fixing another vulnerability in do_munmap()."

The announcement for 2.6.3 and 2.4.25 was yesterday, and the vulnerability to which the link in the text above refers was with mremap, not munmap; there's also another vulnerability with mremap mentioned yesterday as an *update* to the kernel announcement.

SCO

[rauhest]

Without RTFA (of course), I tried to find any reference to "sco".

The only match was "a misconfigured system". :)

ACPI cure for 2.4.25 HOW-TO

[Quietti]

Was here in yesterday's thread about 2.4.25 and 2.6.3 releases.

Bitkeeper

[jurgen]

I sent this to the author of the article...

[the author] wrote:

The lack of formal revision control and source code management led many to suggest the use of a product called BitKeeper.

I grant that sometimes you have to simplify history to avoid digressing in an article, but this is a bit too inaccurate to let stand.

Bitkeeper wasn't suggested by anyone; it didn't have to be. It was developed from the ground up to Linus' requirements. Larry McVoy had a discussion about source control with Linus years ago, in which Linus said "none of the products are good enough" and Larry said, "ok, I'm going to write one that is". Apparently he had this on his mind anyway, and so he started Bitmover Co. As bitkeeper became a usable product Larry continued to take Linus feedback and improve it until it was good enough for Linus to use... at which point Linus started using it.

This is still a simplification of course, but it's closer... and as you can see, there were no third party suggestions involved.

Beautiful

[maximilln]

I don't like to start new threads but I didn't see this: A general "Thank you for your time, effort, and a job well done" to all of the kernel hackers out there. They're fixing kernel level bugs that are almost at the hardware level while M$ is still patching their web browser. I don't think there's any doubt which system is ultimately more secure.

Can anyone take a guess how many low-level memory exploits are in Windows XP, 2k, or others? Perhaps it's irrelevant. Who needs to crack the low mem when there are so many ways into the system at the document level?