What Happens To -AC (And Other) Kernel Mods?

RedLeg wrote with this poser: "So, looking at the changelog for the 2.4.9 kernel release, I see a few '- Alan Cox: driver merges' entries. Intelligent consumers of (or those of us who modify them for our own uses) RedHat Kernel src.RPMs look at the patches in the RH kernel builds. Alan's (and other persistent RH) patches don't seem to be integrated into Linus' 'mainstream' kernel trees on any kind of a predictable basis, and this frequently causes projects like freeswan to have difficulty merging their patches (not intended for kernel inclusion) with kernels that appear 'in the wild' like the kernel RPMs from RedHat. Often, kernel patches for obviously older kernel versions continue to be applied (in the RPMs) to newer kernel versions. Alan is a RedHat-er, so he obviously has an inside track to RedHat kernel builds, but he's also Linus' Right-Hand man, but his patches are not (apparently) consistently making it into the 'mainstream' kernel. What am I missing?" Who better to answer this question than Alan Cox? Alan was kind enough to write an explanation of the (still complicated) process of merging -- and it's not as simple as who works for what distro maker ;)

Note: Here's what Alan passed on in response to this question. As usual, things aren't quite as simple as they first appear. -T.

Alan Cox: Probably the first thing to explain is the Red Hat kernel. That actually isn't something I am responsible for. Arjan van de Ven is the keeper of the distribution kernel, and has the unenviable task of getting a kernel together that will actually pass all the brutal QA testing. Arjan is perfectly entitled to (and sometimes does) throw out bits of -ac changes.

You'll see Red Hat patches being merged into -ac and Linus trees when appropriate, often from Arjan or Pete Zaitcev. Many of the other patches in the RH tree are considered "fixups" - they are workarounds for problems but not generalised or clean enough to feed into the main tree without further work. Others are RH specific patches for things like packaging.

With the -ac tree I try and do rapid rolling releases, sucking in new code to test it and also its interactions with other new code. By doing releases every few days I get a high number of people testing and reporting bugs before there are too many possible causes. This is how Linus trees used to work long ago, and I still think its the better technique.

At regular intervals I take stuff from the -ac tree and feed it to Linus. Sometimes Linus doesn't want to take other changes in case they confuse other things being done, sometimes they just vanish and fairly often they get applied.

I'm actually limited in the rate I can forward patches because I need to feed Linus blocks that are debuggable. Thus I don't want to feed Linus both file system and disk driver changes at once or I won't know which to blame if there are corruption reports.

I also don't feed Linus code that has active maintainers unless the maintainer has asked me to do so. Thus the USB diverges quite a lot because Johannes Erdfelt has chosen not to feed chunks of the USB and input changes on. Similarly, the user-mode-linux port in -ac has not been fed on to Linus because Jeff Dike wishes to improve it further before submitting it.

I have been concentrating on getting the driver code and some architectures synchronized with Linus, and that is now mostly done. The next big challenge is getting all the file system work on to Linus, and Al Viro has begun that and fed Linus the first blocks of the superblock handling cleanup.

Finally we have changes that are down to fundamental disagreements, perhaps in part stemming from the fact my background is real production systems rather than OS design work. Linus decided to update the 3D support without keeping back compatibility - I kept both. Linus I suspect will never accept a patch to do that. Secondly he decided that he didn't wish to allocate new device major numbers but look for a saner solution over time. Laudible, but not in the middle of a stable release. The -ac tree has drivers allocated "non-Linus" major numbers that are recognized by LANANA and thus common across vendors. These drivers like the HPT370 and Promise IDE raid will thus always be part of the -ac tree only.

The -ac tree also tries hard to avoid any incompatibilities. Having applications that require -ac or Linus trees is simply not an acceptable situation. The only specific exception for that right now for 2.4.x is deep at the system level and is for quota tools. That one was unavoidable to get 32bit uid quota working.

Comments

Re:Post-Mortem debugging of multithreaded processe

[scorpioX]

Max OS X has this feature as well. If you set CRASHDEBUG=-YES- in /etc/hostconfig, you will get a dump of all thread stacks and the CPU(s) registers when a process bombs out. Very handy. I believe that HP-UX also has this feature. Surprising that Linux doesn't.

from the cyfrifiadurol dept...

[JJGreenaway]

In case anyones wondering 'cyfrifiadurol' isn't a typo. It's Welsh roughly meaning 'to do with computers'.

And before anyone says it, yes, computers have reached Wales now...

Re:from the cyfrifiadurol dept...

[Anonymous Coward]

Argh... you "GUESS that the Y *is* a vowel"? Guess again.

I'm not sure what you mean here, but it's obvious that you're a real moron. Some letters have different pronunciations in different languages, and 'y' is one of those. In English it is either vowel or consonant depending on the context, which is actually an orthographic mistake in my opinion. This usage comes from Middle French, though. (French has dropped the 'y' in most places, compare old "roy" with modern "roi.") In Latin, as far as I understand it was always a vowel which was used to represent the 'u' sound like in French (like the 'ue'/u+umlaut sound in German). (That's a front high rounded vowel, for the phonetically informed.) This came from Greek and was used only in writing Greek words, since no native Latin word uses the sound. The same goes for old English. In a popular transcription system for Russian, 'y' represents the letter whick looks roughly like Ib in Russian (high middle unrounded vowel?). In many newer writing systems, such as Wolof (to take one that I speak), 'y' is always a consonant, a palatal glide like in "yes."

As a more logical alternative, many languages, such as Dutch, German and Polish, have chosen to use 'j' for the sound equivalent to the English consonant 'y'. This is really more sensible, because 'j' is derived from 'i', which in Latin was used for both a consonant and a vowel, like 'y' in modern English. The practice orginated among scribes wishing to make Latin texts more clear, so they added a little curl at the bottom of the i's that were consonants. People in Central Europe picked this up when developing writing systems for their own vernacular langauges.

So, although I can't say I know any Welsh, I do know a thing or two about languages and alphabets. Why don't you think before you open your fool mouth next time?

Re:Wait a minute

[Alan Cox]

You make an assumption that the right way to test code is in big lumps. That is somethiny any engineer will tell you is bogus.

You test continually, you test each changeset, and then every so often you run a several day shakedown test.

You are right that you can't QA a kernel to vendor production grade in two weeks. Some of the RH test runs take several days per run for example.

Re:Promise IDE RAID

[Tony Hoyle]

The Promise RAID *is* software RAID. All the kernel can give you is access to the extra IDE ports (which is does).

Re:Promise IDE RAID

[Sunthalazar]

I don't know specifically about the Promise IDE RAID, but I do know about the HPT370 RAID. They are actually included together.
All I can really say is that it's not quite perfect. I'm using an on-board HPT370 [it's part of my Abit VP-6], and under Win2K I don't think I've had any crashes [well other than some reproducable ones that are things that I've done]
But I've used the -ac patches in 2.4.6-2.4.8 and so far there is still a couple of times when my machine will just lock up. It seems to be related to disk access, but it also only happens when I'm running X. Without X, I haven't had any problems [although I can't run Mozilla or XMMS, etc without X]
In general, though, it recognizes and runs fine. I haven't had any general data inconsistency [I run ReiserFS on the RAID partitions]
Again, this is for the HPT370, not the Promise IDE RAID, but since they are in the same kernel patch, I figured their results would be similar.

Re:Post-Mortem debugging of multithreaded processe

[n0ano]

The thread core dump patch was originally put into Alan's tree around the 2.4.3 time frame. It was quite correctly labeled experimental at the time (it took a few iterations to get it right.) The intent is to merge it into Linus' tree at some time, it just hasn't gotten there yet.

In the mean time, if you're desperate, I can give you a patch that provides this capability to any Linus tree.

Re:Athlon CPU support should (EXPERIMENTAL)

[Anonymous Coward]

Athlon CPU support is rock solid. Certain combinations of VIA chipset motherboards and AMD Athlon processors seem to leave a lot to be desired

Avoiding the streaming memory fetches will help on most of those boards (ie optimise for PII or K6)

Re:Post-Mortem debugging of multithreaded processe

[Mark Kettenis]

As a member of the GDB team (maintainer of the Linux/i386 port and co-maintainer of the threads support in GDB) I'm not aware of any coordination between the kernel folks an GDB at all. On top of that I'm not inclined to add support for this to GDB until it ends up in Linus' kernel. Anyway, the one-core-file-per-pid approach seems wrong to me. It's a waste of disk space since you're duplicating the VM for every pid. And isn't well suited to how GDB deals with multi-threaded core files on other platforms. A better approach would be to add an additional note with the register contents for each LWP to the same core file.