Is the Stable Linux Kernel Moving Too Fast?

darthcamaro writes "Yesterday the stable Linux 3.10 kernel was updated twice — an error was made, forcing a quick re-issue. 'What happened was that a patch that was reported to be broken during the RC [release candidate] review process, went into the release, because I mistakenly didn't pull it out in time,' Greg Kroah-Hartman said. The whole incident however is now sparking debate on the Linux Kernel Mailing List about the speed of stable Linux kernel releases. Are they moving too fast?"

Comments

TDD

[Anonymous Coward]

TDD, BDD anyone ?

Re:TDD

[Nemesisghost]

Neither will guarantee defectless & errorless code. They help, but you still need to do more. If you are relying only on either, then your process is broken.

Re:TDD

[greg1104]

"the fun thing about a kernel is that there is no good way to test it except to run it" --Greg Kroah Hartman

I work on PostgreSQL, and nothing goes out until it's been validated on the entire buildfarm. It's hard to have such a thing for the Linux kernel though, because it's so easy for a bug to break test machines. You need to catch when machines are responding, do a hardware reset, and then rollback to a known good kernel instead. It's much harder than most software testing to automate.

Re:TDD

[malacandrian]

Doesn't check it works perfectly on all hardware, but a good chunk of the testing could be done on VMs running monkey-scripts which are simply disposed of & a new image spun up if they break. That said, automating hardware resets is hardly an unsolved problem.

Re:TDD

[greg1104]

The logs of the machines that break are the most important part of the test results here. You can't just throw them away when a VM dies without losing most of the testing value. If you're lucky, a busted kernel will stay alive long enough to create a crash dump when running on dedicated hardware. That's less likely to happen on VMs.

It's also possible to automate hardware resets with IPMI or similar lights-out management code. All of that takes special hardware though, and test code like this is painful to build.

Re:TDD

[cheater512]

VMs create a single 'hardware' platform. You aren't testing more than a fraction of the kernel.

And your automated resets will let you boot a new kernel easily (so does 'reboot' anyway), but if it does break you can sit there resetting over and over again in to the same broken kernel. Yay!

Re:TDD

[hobarrera]

You'd need VMs to emulate all sorts of hardware and architectures. We don't have all that yet. We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).

Re:TDD

[ultranova]

We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).

Which rises question of just why are they part of the kernel? Why does a mouse driver need to run at Ring 0?

Re:TDD

[The_Wilschon]

For the purpose of release testing, though, the only thing you care about is whether or not there was a crash. If there was a crash, don't release. Back out the busted patch and release the working version. Then you can spend your time debugging the busted patch, which requires the logs and all.

Re:TDD

[Anonymous Coward]

Wow really? There are hundreds of thousands of permutations of devices? Is he implying that each and every driver is not operating through some API but instead, everyone is freely messing around in kernel space? Laughable.

Re:TDD

[MikeBabcock]

You need to do some more reading on how Linux works.

Re:TDD

[serviscope_minor]

You need to do some more reading on how Linux works.

So do you.

Linux is sort of a hybrid kernel now. Some hardware drivers are in ring 0. Quite a lot are no longer. libUSB for example allows userspace USB drivers. They work great. FUSE allows for user space filesystems which work great where absolute top performance is not necessary (eg sshfs, ftpfs etc).

A good fraction of the bluetooth stack, for example anything above L2CAP (the Bluetooth world equivalent of the IP stack's SCTP) , such as ATT and GATT is all userspace (and the non kernel side sucks donkey balls by the way). That means I could (if it didn't suck massive donkey balls) control all the various profiles with the majority of the code in userspace.

All the printer drivers are mostly in userspace (yay).

The graphics (X11) is largely in userspace for now...

Sound has a large userspace component and all the complex stuff like routing, mixing, and figuring out what to send where is in userspace (pulse or Jack).

The Linux kernel as-is is more than capable of running a mouse driver in userspace.

Re: TDD

[nbritton]

I've always felt that a rolling release, as exemplified by FreeBSD, naturally stablizes things. They have two branches, STABLE and HEAD (a/k/a CURRENT). New code is commited to HEAD and only once it's stabilized is it MFC'd (Merged From Current). Once in the STABLE branch it's feature frozen, and receives continuous refinement and debugging for the life of the code. There has always been a saying that "FreeBSD works like running water", and I attribute this statement to the development process.

In one of Greg's presentations he stated that Kernel development is the largest and fastest moving software development project in the world, by an order of magnitude. I don't have the answers for what needs to be done, but it speaks volumes that Enterprise Linux distributions choose to use a kernel from 2009. Something needs to be done.

Re:TDD

[zidium]

What's a decent alternative? FreeBSD?

Re:TDD

[Anonymous Coward]

All the printer drivers are mostly in userspace (yay).

That's an oxymoron. Try this instead: "Printer drivers are mostly in userspace."

Re:TDD

[lord_rob the only on]

Hmmm :)

Nothing like a good troll to read in the afternoon. I love reading Slashdot at -1 :)

Re:TDD

[ultranova]

You need to do some more reading on how Linux works.

Not very well in this regard, according to the grandparent, which gets us back to my question: why do it this way?

But thanks for your +5 Informative comment anyway. I certainly know more having read it.

Re:TDD

[tempest69]

The All clears an ambiguity. This means the poster means that all the print drivers have most of their stuff in userspace. Rather than most of the print drivers are in userspace.
I don't follow printer drivers closely enough to know if the poster was right. I try and follow the nuance language, it is abused often.

Re:TDD

[nebosuke]

Your parent's statement is not an oxymoron.

If every single print driver has components running in both ring 0 and userspace, but the preponderance of components (by number or 'size') of every single print driver is in userspace, then it is more precise to say that "all print drivers are mostly in userspace" as opposed to "printer drivers are mostly in userspace". The latter is semantically a superset of the former, as it could either mean the same thing, or also describe a situation where some printer drivers are completely implemented in ring 0, but the majority are completely or mostly implemented in userspace such that the preponderance of the set of all printer driver implementations resides in userspace.

In other words, your parent's statement is more precise about not only the aggregate population of printer drivers, but the distribution within the population. Whether that statement is actually correct or even the real intent of your parent poster's statement is another question :).

Re:TDD

[MikeBabcock]

Good job on not paying attention to the last ten years of Linux development.

So the stuff that isn't a hardware driver attached to the Linux kernel isn't in the kernel ... good job on deducing that.

Printers aren't in the kernel because (get this) they aren't part of the hardware. The Bluetooth stack isn't hardware, the hardware is hardware. That stuff's in the kernel. The sound system? The hardware support is (again) in the kernel, the stuff that's at a higher level isn't.

There are some very specific edge cases, but for the most part, all actual hardware support in Linux is in the kernel because that's where it belongs in a monolithic design.

Re:TDD

[MikeBabcock]

As though I gave myself +5 ... jeez.

The post I responded to was out of touch with how Linux is designed. If you want to know more, go do some reading. Its not hard.

No

[Stumbles]

its moving along just fine. People make mistakes, get over it, its not the end of the world. Considering its current release speed, the amount of changes made over the long term the Linux kernel folks have as good or better track record than most other software houses.

Re:No

[Anonymous Coward]

Seems any slip in the Linux world is heavily scrutinized by their own community. Feels a bit like self-sabotage.

Re:No

[Stumbles]

Well everyone has bellybuttons but that doesn't mean we all should gaze upon the great naval; at least for to long else your eyes will burn out.

Re:No

[CAIMLAS]

Really?

The current process resulted in us having CFQ + EXT3 as the default for a long time (some distros still have this). This basically means any sort of interactive performance is worse than horrible. The only reason we're beyond it now is because EXT3 is on its way out with EXT4 being preferred.

IIRC, wasn't CFQ one of the first major infrastructural things put into 'stable' after this 'rapid release' approach was adopted?

Also, udev.

I'm sure there are other examples... and maybe I'm projecting a bit.

Re:No

[Shark]

All in all, I think it's a nice problem to have. Compare that to the kernel being stagnant, it's great that being able to include submitted code safely and fast enough becomes an issue to look into. I doubt MS or any of the other big software companies have issues where features and improvements are being *produced* so fast that QA is unable to keep up. I suspect that they have more of an issue with them being *requested* faster than the company can provide.

Re:No

[vilanye]

That is the way it should be.

That's how Linux can thrive dispite the chaos of getting patches from 1000's of independant programmers.

Re:No

[Anonymous Coward]

>> Feels a bit like self-sabotage.

Feels a bit like it is working as intended.

Re:No

[msobkow]

Timely releases of the Linux kernel don't hurt anything anyhow, because most of the packagers and distributors don't release the updates until they've had further testing. In fact, most distros lock in at a particular kernel release, and backport patches to *that* release rather than updating the whole kernel.

So there are really three levels: dev, stable, and production.

Re:No

[Anonymous Coward]

People make mistakes, get over it, its not the end of the world.

Linus ripped one of the kernel devs for making almost the exact same statement.

Re:No

[cheater512]

CFQ only comes in to play when accessing new uncached data from disk when disk is at medium-high usage (at low usage any read/write queues are empty).
I'm struggling to figure out any interactive programs that fit that description. Web/email/documents/games etc... don't touch the disk in any significant way and the cache handles most disk accesses for them anyway.

Re:No

[Anonymous Coward]

People make mistakes... and airplanes fall from the sky, cars crash, and infotainment systems BSOD.

Linux is used in those environments of the above examples, and though folks will hate it, the kernel development for the sake of development is moving along fine. For power users that have no need for kernel hacking, this is very bad, cause all marketing, university professors, and customers only see what's in the latest release (i.e. new features), and dev teams are directed by them in the end... hence creating heartache as you update to the latest kernel and breaks all you apps, video drivers, and even booting (typically). You can't have them in mission critical systems (or just important systems) as they are released.

In the end this is good as it will force mission critical developers that use linux to either:
a. support older, more stable versions that are feature rich (a compromise).
b. drop Linux

Number of changes or fixes doesn't determine track record alone, usability, compatibility, design, and robustness does as well.

Re: No

[coffbr01]

I prefer to think that an immediate response from the community is a sign of an active and concerned user base. It might be worse if the community insisted on staying on LTS-type releases.

Re:No

[jhol13]

Linux (the kernel) is accumulating new security holes at least at the same speed as they are fixed.
Proof: Ubuntu kernel security hole fix rate has been constant for years.

(actually I have not counted the actual number of holes, only the actual number of kernel security patches - these two should correlate though).

Re:No

[Runaway1956]

Self flagellation is preferred over allowing the greater world to whip you mercilessly.

Re:No

[Anonymous Coward]

CFQ under ubuntu 12.04 lts produces studders and freezing of applications with only one or two VM running.

BFQ is a million times better. Thank goodness for the liquorix kernel.

Re:No

[tibman]

How does the Kernel drive what your disks are formatted to? Your disks are formatted to ext3 or 4 long before you config/compile the Kernel.

Re: No

[TCM]

As if you can't have both. That's what branches are for! Duh.

But of course you'd have to understand branches and their use first. Why call it stable if it's just another testbed where everything is shipped regardless? If "stable" is moving so fast that bugs that are already recognized cannot be pulled back in time, something is just wrong.

Re:No

[dAzED1]

yeah, I'm a bit confused by his comment too. Is he suggesting the ext4 support is not available? Because umm...wtf does the kernel (otherwise) have to do with what formatting is "default?"

Re:No

[gweihir]

Indeed. Also anybody with some experience in rolling their own kernels with sources from kernel.org knows not to use any x.y.0 releases when they want stability. Give a specific release some time to mature before using it, say 2-3 weeks. You may also have to read changelogs to find out what things were patched. There are also nice, pretty current "longterm" kernels.

That said, I have been on some current releases for some 24/7 machines and did not encounter any issues. I would say that the Linux kernel folks are doing a fine job. I can only guess that the story mistakenly assumes a freshly released kernel version is somehow "perfect". Rolling your own kernels from kernel.org is not a beginners game. Get over it.

Re:No

[MikeBabcock]

The current process didn't result in that at all. Your distro chose that default and could've changed it any time they liked.

Get involved with your distro if you care so much and help them choose sane defaults.

Re:No

[tlhIngan]

CFQ only comes in to play when accessing new uncached data from disk when disk is at medium-high usage (at low usage any read/write queues are empty).
I'm struggling to figure out any interactive programs that fit that description. Web/email/documents/games etc... don't touch the disk in any significant way and the cache handles most disk accesses for them anyway.

Until you start swapping. And most users tend to run a ton of programs at once (or load up dozens of tabs in Firefox) so the kernel starts to swap to disk. Yes, they probably should upgrade their RAM, but that takes money and people don't like spending it.

So now you have a system under memory pressure trying to save some memory for disk cache, and on the edge of thrashing. All you need is something like a movie or music playback to blip because its buffers ran out.

and yes, it also appears some webmasters code their websites assumiing their users have machines with quad cores and 16GB of RAM.

All in all, I think it's a nice problem to have. Compare that to the kernel being stagnant, it's great that being able to include submitted code safely and fast enough becomes an issue to look into. I doubt MS or any of the other big software companies have issues where features and improvements are being *produced* so fast that QA is unable to keep up. I suspect that they have more of an issue with them being *requested* faster than the company can provide.

Until you're stuck on the other side - Android tracks the latest kernel, so what happens is OEMs don't like to upgrade the OS on the phone because even going between versions means a kernel upgrade and porting drivers to the new kernel (a major annoyance when APIs change, too).

Re:No

[Anonymous Coward]

Tell that to people who have a wireless logitech keyboard/mouse. There is a reason why I re-installed windows. Any kernel release shouldn't break a fucking keyboard ffs. If the 3.10 is supposed to be a LTS kernel, that's fucking gold.

Re:No

[cheater512]

Ok that may be a scenario where CFQ doesn't perform adequately, however I'd say that is a pretty poor example.
You cannot reasonably expect a fluid desktop environment with moderate swapping. It is never going to happen.

Re:No

[Anonymous Coward]

Constant security hole fix rate only proves that the resources for making security hole fixes have been constant.

Time period sampled is too short to conclude anything about bug density.

Re:No

[Tripkipke]

I have, what's the problem with it?

Re:No

[semi-extrinsic]

and yes, it also appears some webmasters code their websites assumiing their users have machines with quad cores and 16GB of RAM.

This. I had to throw more RAM into my wife's laptop because a significant portion of the interior design blogs she reads are coded by seriously inept people.

Re:No

[Anonymous Coward]

https://bugzilla.kernel.org/show_bug.cgi?id=60507 and still not fixed or 1month to mark that one as duplicate https://bugzilla.kernel.org/show_bug.cgi?id=60572, lovely

Re:No

[Error27]

I work in kernel security and I would say we have improved. You can't just tell people "don't make mistakes" and expect security to improve the only way you can improve is by improving the process.

1) We've added a few exploit prevention techniques like hiding kernel pointers.
2) Our fuzz testers have improved.
3) Our static checkers have improved.

But we're not perfect.

For example, we earlier this year we merged user namespaces. Obviously this is tricky code which deals with security. People had been working on it since 2007, but even after five years we all knew there were going to be some security bugs which we had missed. Code has bugs. That's life. But user namespace is a valuable feature and we had done everything we knew how to do.

Actually, in some ways, user namespaces will improve security overall because we can use it to remove a setuid binary from the Chrome browser.

Btw, you can't just look at CVE count. If could be that the bug is old but it was only found recently because of the improved tools. Also two years ago we probably wouldn't have issued a CVE for info leaks like CVE-2013-2148.

Re:No

[jhol13]

Although you are more qualified than me (I gave up following security actively over ten years ago), I beg to differ. No offence.

Seems like you have invented a very complicated version of jail. Typical Linux attitude, both "NIH" and "it is a superset (i.e. can do more), therefore it is better". In security both have been proven to be bad ideas quite a few times.

[quote] Code has bugs. That's life [/quote]

I do not want to talk that much about whether some code has bugs or not, rather about attitude. That sentence alone has bad attitude, but Linux's (Linus's) most cherished attitude is "change is everything", i.e. any reason whatsoever is good enough reason to make a backward incompatible change. In theory this has no effect in security, but in practice the effect is big: you really cannot test all or even most changes to all device drivers. This is why I do not believe you have improved. This is why I think companies like Vupen prosper - they get it too easy.

As said, I do not count CVE's (there are far too many to count), I count necessary reboots. Yes, even worse as a statistical tool, but as said, I stopped following security. Reboots occur at a slowly increasing rate, about ten per year. This year has had more. My computer now is more in "please reboot" state than in "ok" state. The security updates are pretty much the only reason to reboot - for me.

Re:No

[jhol13]

One thing: I do not and have never claimed that some-other-OS is better (you won't get me in a OS flame war).
I claim Linux could do hugely better.

Re:No

[CAIMLAS]

It used to not be a problem, that's the thing. Before all the modern schedulers came to be (so, back in the 2.4 days) it was entirely possible to stream a video over the network without stuttering - while running a -j3 kernel build, updatedb, and a find on root with memory exhaustion taking place (eg. browser was loaded up). It was a matter of pride that these things could be done on Linux, because Windows would fall over under similar loads. Nowadays, Windows handles these situations better than Linux does due to Windows improvements and regressions in Linux.

Chalk it up to server-oriented performance tuning.

Re:No

[cheater512]

Use the deadline scheduler then if you don't want a complicated scheduler. It should be included in most prebuilt kernels.

Funnily enough deadline is recommended for server loads and CFQ is recommended for desktop use. The exact opposite of what you are suggesting.

And I have no doubt my computers can do exactly what you describe just fine without impacting desktop performance at all.
Just the other day I was doing network video streaming (1080p over SSHFS), I had 4 virtual machines running, plus Thunderbird, two dozen Chrome tabs, etc...
No impact on interactive performance at all.

If you have performance issues, there are steps to diagnose it (unlike on Windows).
You do not automatically just blame the scheduler because that is the stupidest argument ever.
I notice you don't have a shred of evidence (top output, iostat, etc...) to back up anything you've said.

Btw if a updatedb takes more than about 10 seconds you are doing something horribly wrong.
Most distributions have been using mlocate for several years now instead of slocate. updatedb took 5.8 seconds to run just now for me over 6TB of disks.

Re:No

[CAIMLAS]

Are you using an SSD? Was memory exhausted?

The scenarios I describe were/are disk contentious in scenarios at or near memory exhaustion, when the system dips into swap.

You can experience this as soon as the system starts dumping RAM pages to swap even today, assuming you've not got an SSD.

Compared to what?

[bmo]

"Are they moving too fast?""

Compared to what, Windows, IOS, OSX, What?

>known bug that got by review
>caught
>fixed rapidly instead of waiting for the next release

I don't see the problem.

If this was a regular occurrence, yeah, it'd be a problem. But it's infrequent enough to be "news."

Unlike Patch Tuesdays, which aren't.

--
BMO

Re:Compared to what?

[ericloewe]

Patch Tuesdays aren't news. Patch Tuesdays that break something are.

Re:Compared to what?

[jamesh]

"Are they moving too fast?""

Compared to what, Windows, IOS, OSX, What?

Compared to a speed where accidents like this are less likely to happen, if such a thing exists. It could be that OS release cycles are unsafe at any speed!

Re:Compared to what?

[rudy_wayne]

"Are they moving too fast?""

Compared to what, Windows, IOS, OSX, What?

A long time ago, I don't remember where it was, maybe LKM, Linux Torvalds said there would never, ever be a version 3 of the Linux Colonel. I thought that was a strange thing to say, even for him. I thought to myself "things are really gomna get weird when they get to 2.99.99.99.99.99".

So now, he's changed his mind and the version numbers are zipping along. Not as fast as the absurd version number inflation of Firefox and Chrome, but still a lot faster than it used to be.

In General, I don't have any Major problems with the Linux Colonel. But I don't think it's unreasonable to say "Whoa, what's your hurry? Maybe you should slow down a little bit."

Re:Compared to what?

[geekoid]

Now you're being redundant.

Re:Compared to what?

[Anonymous Coward]

Patch Tuesdays that break something aren't news either.

Re: Compared to what?

[O('_')O_Bush]

You mean, break everything. It is rare that a MS update doesn't break *something*.

Re:Compared to what?

[Anonymous Coward]

Compared to itself? Why is it the Linux crowd is always trying to battle other OSs instead of just concentrating on what theirs does? So fucking hung up on Apple and Microsoft that they're missing the real big picture.

Re:Compared to what?

[cheater512]

Although they are starting to become the norm.

Re:Compared to what?

[Anonymous Coward]

I rote a kool program that steals credit card numberz. and micro$ofts patch borken it.

Re:Compared to what?

[Anonymous Coward]

Would anything be different if they had stayed on 2.x.x?

Of course not!

A version number, when used correctly, is just an identifier. That the proprietary world has hijacked them for marketing purposes doesn't really change anything.

Re: Compared to what?

[Barlo_Mung_42]

Their massive install base works against them here. On Windows a one in a million bug will happen by next Tuesday.

Re:Compared to what?

[bmo]

>Last sentence.

http://i.qkme.me/3rpqih.jpg

--
BMO

Re:Compared to what?

[bmo]

>moderation "overrated"

Not like karma counts for anything, but I've always thought the "overrated" mod was chickenshit.

"I disagree with you, but I don't have the balls to reply"

--
BMO

Re:Compared to what?

[Anonymous Coward]

Moving too fast compared to the speed of available broadband in USA. I'm sure if Greg had faster connection he would have been able to pull it out in time.

Re:Compared to what?

[MikeBabcock]

Which would be why you should use an actual distribution kernel and not the compile-it-yourself variety if you need stability and testing.

Re:Compared to what?

[Anonymous Coward]

And the person who live in the toilet is a loo tenant

Re:Compared to what?

[ArsonSmith]

Perhaps the question should be "Am I adopting Linux kernels too fast?"

Well, if you were hit by this bug in a significant non-testing way then I'd say yes. Unless you are testing, don't install anything on release day.

Re:Compared to what?

[Osgeld]

problem is that most of the time it takes its sweet ass time

known bug that got by review
caught
wait half a decade for known bug to be fixed or more likley forgotten so it still remains in next version, goto 10

Re:Compared to what?

[deviated_prevert]

The word you meant to use is "kernel" as in "'kernel of truth." It's at the center, the core, the essential part. "Colonel" is a rank or designation, especially of the military. The more you know!

Some people use auto spell correction and don't even look at the results. This is why his use of the word colonel has a cap letter. On a Windows2000 forum years ago I called myself Colonel_Panic just for a laugh. It went over the head of the Windows only zealots on the forum, but it was good for a laugh when asking questions about debugging hardware bsods on Win2000 installs that I was administering at the time. Funny part is one nasty bsod was from a belkin card we were using. Ran it on a Slackware install an it was fine but it caused havoc with Win2000 installs even though the Win98 drivers were fine. Wound up costing a few grand to fix we just replaced the cards. LOL

Re:Compared to what?

[Anonymous Coward]

I used to work in the Windows division at Microsoft. The organization responsible for patches that appear on Windows Update (WinSE it is called) is craaazy conservative, to the point of it being a bad thing. As in, there are typically massive amounts of critical flaws that are fixed in the development branch, but will never be ported to the currently shipped version because no customer with deep enough pockets has complained about it.

I don't think it's a model worth emulating.

Re:Compared to what?

[semi-extrinsic]

Indeed. ArchLinux, which is the fastest bleeding-edge, rolling-release distro out there, never released 3.10.8 into [core]. [core] is now at 3.10.7 and an update is flagged to 3.10.9 within a couple of days, skipping .8. The only way you would install 3.10.8 is if you (a) roll your own kernel or (b) use the [testing] repo in Arch. Both of these indicate that you are not a regular user, you are a developer (in some sense).

Re: Compared to what?

[Carewolf]

Impossible bugs happen 12 times every single day on most computers (the amount of spontanious errors in RAM due to cosmic radiation and the lack of error correcting memory). Now add one extra error every week from an operating system bug. Can you spot it?

Re: Compared to what?

[robthebloke]

Clippy still works.

Re:Compared to what?

[Anonymous Coward]

Thank you. One of the few actually insightful posts, so far.

Re:Compared to what?

[Cramer]

Actually, it appears to be rolling along at the same pace it always has. The only thing that has changed (*AGAIN*) is Linus's idiotic idea of release numbering. He should stop sipping from the teacup of Microsoft and look instead at Cisco -- while it may have some oddity with special branching, it's had relatively the same release vs. development process for decades. I'm not saying we should return to the insanity that was 0.99pl5j1e, but 3.x isn't any better.

Re:Compared to what?

[bmo]

Fair point.

But the change to 3.0 was "just a number" I really don't give a shit about version numbering. Just tell me which one works.

--
BMO

Too Fast?

[Jeremiah Cornelius]

Let me try and catch up to it, and ask...

Seriously. Why is this even a question? Did a new stable release show up in your watch or your laptop - or your in flight entertainment system, over night?

Packagers and distribution maintainers aren't exactly up in arms about this...

Released kernels are the real testbed

[icebike]

As indicated in the debate on LKM, rc kernels get hardly any testing, although all of the tests it does get are mostly by highly motivated and astute testers

Most distros are releasing kernels at least one behind the developers tree, with not a great deal of incentive to update the kernel right away, (even if they make it available in a repository for those wanting it). So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.

So at most, this was an embarrassing incident, and not a bit deal. The amazing thing is that it was caught at all. Some of us remember kernels that got into production distros with serious things broken that should have been caught much earlier.

Re:Released kernels are the real testbed

[NotFamous]

So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.

One of these days I am going to meet Joe, and I am going to complement him on his Abs.

Re:Released kernels are the real testbed

[Anonymous Coward]

it doesn't hit Joe Sixpack's machine for several months.

I dunno. I feel someone who uses Linux on the desktop at all is probably not a Joe Sixpack. They're either a techie, or an elderly relative of a techie.

Re:Released kernels are the real testbed

[CAIMLAS]

From where I'm sitting, as someone who used to routinely build rc releases and use them, this is how things look.

Five, ten years ago you had people such as myself who would build RC (or AC, etc.) kernel trees to test things and see how they'd work. I know several people who regularly made LKML submissions, many of which turned out to contribute to fixes.

Today, using the rc releases isn't as practical because they're fairly divergent from distribution patchsets. A lot goes into a distribution kernel which isn't present in the vanilla kernel.org kernels, it seems.

More often than not, pulling everything together to build our own kernels isn't worth the extra effort: possibly due to the shortened cycle and possibly due to general code maturity, there's little benefit. Maybe our definitions of 'benefit' has changed, too - but arguably, the changes in the kernel today are nowhere near as drastic or significant as when (say) XFS was getting merged with additional kernel and disk schedulers.

Re:Released kernels are the real testbed

[Anonymous Coward]

One of these days I am going to meet Joe, and I am going to complement him on his Abs.

They mean beer, not abs...you queer.

Re:Released kernels are the real testbed

[DocHoncho]

One of these days I am going to meet Joe, and I am going to complement him on his Abs.

They mean beer, not abs...you queer.

Thanks for the clarification, genius.

Oh, and woosh.

What a stupid question.

[Slartibartfast]

Clearly, the author of the post has forgotten 3+ year intervals for kernel releases. Of the odd, quickly fixed pothole is the price of 6x release speed, "Hell, yes!" Is the answer.

Re:What a stupid question.

[Sponge Bath]

The result is mainline kernels (including "stable") are perpetual beta. Any important machine should only run a kernel vetted and tweaked by a trusted distribution.

Re:What a stupid question.

[Stumbles]

Ah yeah, like kernel.org isn't trusted. Yes I know you said "distribution" but really now.

Re:What a stupid question.

[Rich0]

Ah yeah, like kernel.org isn't trusted. Yes I know you said "distribution" but really now.

He wasn't talking about trusting that it didn't contain a trojan or something. By trust he meant vetted for quality.

It is a legitimate concern. The whole reason for having a release cycle is to have sufficient QA to prevent issues like this from happening. Distros provide that service - when Linus/Greg call a kernel done, they call it ready to start being tested. RHEL is still running 2.6 (albeit with backports).

Re:What a stupid question.

[TheGavster]

The problem with RHEL is that the true version of any package (for compatibility purposes) is the part after the dash. They'll call it "2.6.18" (I think that was the RHEL5 kernel), but really it's got most of the patches (but not all!) up to a much more recent version. If you have a piece of software from outside the RH repositories, and you want to know if your libraries are sufficiently up to date, you need to go dig up the SRPM for the version of each that you have installed and peruse the changelog feature checking for what has and has not been backported from the "real" latest version. I've never been able to find a magic decoder ring from RH proper to make this process of getting the true package version anything less than absolutely hellish.

Re:What a stupid question.

[Rich0]

Oh, I fully get that.

But that was what was meant by moving QA to the distro. In this case Red Hat basically circumvented the upstream release process entirely creating a new set of releases which met their own quality standards. Most distros can't afford to completely replicate the upstream release management for everything they distribute (or at least the important stuff).

The fact that many pay for RHEL basically (or just use it anyway via CentOS) speaks to the concerns people have about FOSS release management in general. If they trusted upstream maintainers to do the job right then Red Hat would go out of business.

Re:What a stupid question.

[coolsnowmen]

The fact that many pay for RHEL basically (or just use it anyway via CentOS) speaks to the concerns people have about FOSS release management in general. If they trusted upstream maintainers to do the job right then Red Hat would go out of business.

Even if every upstream maintainer was perfect, there is still a lot to do to maintain a linux distro. Didn't RH do a billion$ in business last year? I assume some of that is in selling SOMETHING related to RHEL.

I do not think you have an accurate understanding of Redhat platform's business model in conjunction with the challenges of maintaining and versioning software.
In my experience, people pay for RHEL for any number of the following reasons:
1) A guarantee not to break ABI for a given x.y version of RHEL, this means that your business software will still run after an update
2) An attempt to backport security fixes to software that the maintainer wouldn't. Think about this in conjunction with the previous point. For example, if an upstream product changes their interface, then rhel can just pull it in, because it would break all dependent software. But if the upstream changes, and then they issue a security fix, RHEL has determine if the security fix is needed, and then backport that change to the older software to maintain both security and compatibility- something that most projects wouldn't attempt to do. Because they are focussed on just making the single best piece of software, not a backwards compatible ecosystem (ain't nobody got time for that- well, I guess RHEL does)
3) They sell support. (this has been covered many times in /. discussions)
4) They sell many other business "middleware" services that all run on rhel. So the cost of rhel get wrapped up in the entire package. [They might want a JBOSS application server to replace one of it's more expensive competitors, and they pay RH to do some of the hard porting, well, their RH engineer, and RH consultant developed the demo on RHEL so that is what ended up getting installed.

Partial Disclosure: I have friends that work for redhat; and I respect/appreciate the job they've done with linux in the business world.

Re:What a stupid question.

[Rich0]

Agreed on all you posted, and much of my point was directed at #1-2, and to a lesser extent #3 (the reason they can offer support is that they make sure stuff doesn't get broken all the time). Most distros do not do any of these to the same extent, but the value of any linux distro is still often in the way they add a consistent quality layer to upstream release practices.

Now?

[Anonymous Coward]

OK, so the entire stable transition from 3.4 to 3.10 went like this:

1) kernel 3.n.1 compiles. Ship it!

2) Knowing that 3.n.1 has been out for three days, GKH leaves a message like, "System builders, go fsck yourself. Kernel 3.n.1 is out, stop using kernel 3.(n-1), it won't be updated, nothing to see here. Don't care if you're Fedora and you just released a new system based on a decent kernel. It's obsolete now. Just move on."

3) kernel 3.(n+1).0-rc2 is released, with six more weeks until it becomes the new kernel 3.n

4) release kernel 3.n. Wait three days

5) repeat step 1.

And now, because of one simple human mistake, there's a debate on the pace at which stable kernels are released?!? Sheesh!

P.S. -- GKH is a good guy to me, now that he's released a new LTS kernel. He's a hard worker, and I can't blame him for the way he goes about his work.

Re:Now?

[pr0fessor]

He is working on this one too so he was in two article in a row.

http://tech.slashdot.org/story/13/08/21/1959227/internet-infrastructure-for-everyone

That's what he said!

[fizzup]

I mistakenly didn't pull it out in time.

Re:That's what he said!

[Anonymous Coward]

I mistakenly didn't pull it out in time.

And he was also "going to fast".

Re:That's what he said!

[Anonymous Coward]

Countered by "Aren't we moving too fast?" - that's what she said.

(Erm. What were we talking about again?)

Re: That's what he said!

[Anonymous Coward]

Like that old adage. "Programming is like sex. Make one mistake, and support it for the rest of your life."

WHAT THE FUCK! WHAT THE FUCK!!!

[Anonymous Coward]

WHAT THE FUCK!

I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.

Test-driven development is not the solution to this problem. And my good gawd, behavior-driven development is even farther away from the solution than fucking test-driven development is.

Behavior-driven development is one of the biggest loads of shit to splash upon our profession in years. Customers and analysts will write tests? Riiiiiiiiiiiiight...

All that we get from BDD is half-arsed tests that don't work and clients or analysts who cry to high heaven about how they hate writing them. And we programmers can't just rewrite them in Java or C# or Python or C++ or whatever our project is using. Noooooooo! They all require stupid English-like syntaxes that need to be translated down into real code by some turdy tool named after a vegetable.

Fuck TDD. Fuck BDD twice over. And please, for the love of all that is good, NEVER suggest that either of them be used seriously, especially for a critical piece of software like the Linux kernel.

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[Anonymous Coward]

WHAT THE FUCK!

I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.

...

OK, one of those two must be blatantly false.

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[Motard]

Thanks Linus, but I think it's an established rule that you can't go releasing new versions of software until the ridicule surrounding your last release has died down. How else are we going to get stories for /.? Microsoft plays along. Why can't you?

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[Zeromous]

Only said fuck four times not including subject. Not Linus.

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[hermitdev]

If I had mod points, you'd be given one. I have Linus giving the finger to nVidia as my desktop wallpaper at work. That being said, I'm more of a Windows guy. I work on Linux, predominantly, and I use Windows as home exclusively (except for around 4 VMs) because I'm a gamer.

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[Billly Gates]

... Microsoft plays along. Why can't you?

Fucking Linus and Alen Cox are greedy mofus! No more supported updates for kernel 2.2 after April 2014 WTF!!

It still works fine and I am not willing to change just to make them happy

Software does not have rusticles form. It does not age and we had a good thing running back in 2001 in kernel 2.2 and it makes no sense at all to throw out a perfectly working and I do not want to run some bloated kernel with useless utilities and eye candy for teenagers.

We have software that is very expensive to change and you all hipsters on slashdot and the LKML fail to understand! All hardware better support it or we wont make your product and not one slashdotter or Linux user has given me a reason to change?

I can browse the web with Netscape just fine as Root as /dev/modem wont work with a wheel user account, so why should I change?

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[zidium]

Maybe he NEVER reads the comments?

Re:WHAT THE FUCK! WHAT THE FUCK!!!

[Xtifr]

Nope, even that doesn't excuse it. -- voice of experience

The problem with oversight from two persons

[Mister Liberty]

... is that it would move even faster.
Really, this is a non-problem. The 'system' worked.
Thank God they're no slick sleezeballs like Ballmer,
but acked and corrected.
Enjoy your new kernel. Relax. Be happy!

Re:The problem with oversight from two persons

[ArsonSmith]

But my heart monitor automatically downloads and installs the latest kernels within minutes of them being posted to kernel.org.

Re:The problem with oversight from two persons

[IwantToKeepAnon]

But my heart monitor automatically downloads and installs the latest kernels within minutes of them being posted to kernel.org.

Then you're a fool who deserves what he gets. You should be using a long-term support strategy and update ONLY when you get new HARDWARE and need the software to support the new features. Embedded code not only doesn't need to be bleeding edge, it should NOT be bleeding edge. I doubt your heart monitor needs new filesystem drivers or the latest GNOME or KDE desktops (just guessing here).

Nothing a few f-bombs for linus can't fix

[Anonymous Coward]

Because everyone's obviously learned their lessons

Re:Nothing a few f-bombs for linus can't fix

[Virtucon]

No that's only when you break User Space.

What's good.

[dutchwhizzman]

Why do you have to compare it to other operating systems? Just look at what should be the right way to do it, maybe learn from other operating systems, but don't just look at the speed of what others are doing and try and match that. If things go wrong because you're moving too fast, you should either slow down, or fix your methodology so you can deal with the speed. If things don't get adapted by distributions because it's a pain to keep supporting, slow down, or make it easier for them to support it. If things go too slow and you miss essential features that everybody needs, speed it up. It's not that hard to rely on your own merits and not be dependent on other operating systems to determine how fast you should be going.

Time for an LTS Option

[Anonymous Coward]

Time for an LTS option. RedHat, Canonical, Debian, should backport security fixes and maybe mature drivers to a LTS kernel for 5 years or so.

For that matter, go ahead and make a LTS gcc fork, backporting security fixes during that same time schedule.

Re:Time for an LTS Option

[Anonymous Coward]

Uhhh... isn't this how this is already done?

Re:Time for an LTS Option

[greg1104]

3.10 is the next LTS kernel by Linux standards. The existing long term kernels are 2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04), 2.6.34, 3.0, 3.2.50 (used in Ubuntu 12.04 LTS), and 3.4.59.

Re:Time for an LTS Option

[Anonymous Coward]

Amen !
  I thought 3.10 was supposed to be a LTS kernel.

I would have hoped for a little extra care and testing for a LTS where many distros will build LTS versions.
This will be around for at least 2 years.

Re:Time for an LTS Option

[kthreadd]

2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04)

Given the amount of backported features and hardware support that Red Hat continuously adds to their kernel I highly doubt that it looks anywhere close to 2.6.32 from kernel.org.

No, you want a frozen kernel

[robmv]

No, you want a frozen kernel. A stable kernel isn't one without bugs, is one where there aren't massive changes and you get dot releases with fixes

Absolutely not

[stox]

There are plenty of older kernels being actively maintained. Stable does not equal recommended for all production needs.

Maintainer is too nice of a guy

[Anonymous Coward]

"People want national championship banners. People want to talk about Indiana being competitive. How do we get there? We don't get there with milk and cookies."
- Bob Knight

No

[swillden]

Keep in mind that the stable kernel releases are not expected to be production-ready. Linus just produces the input to the actual testing and validation processes, which are performed by the distributions. That assumption is built into the kernel dev team's processes. Not that they intentionally release crap, but they don't perform the sort of in-depth testing that is required before you push a foundational piece of software onto millions of machines.

This is why we have Linux Distributions

[Khopesh]

I haven't needed to bypass my Linux distro and install a vanilla kernel in over ten years. I can wait. If it hasn't been packaged by the major distributions yet, it also hasn't been widely deployed. There are some highly competent kernel package maintainers out there, and they're likely to catch a lot of things themselves.

Then there's the package release and its early adopters (Debian, for example, releases the latest kernels in Experimental and/or Unstable, letting them cook for a while). I typically pick up new kernels when they're in Debian Testing or Unstable (if I really want a new feature). This minimizes the chance of running into problems.

(note, this works best for people using standard hardware. ymmv regarding embedded or obscure devices)

Cool Farts

[sexconker]

[Read as if you're Robert Preston in The Music Man addressing the town]

Now we're all familiar with hot farts here on Slashdot. That sharp exit of heated gas that warms your anus for a few seconds during its escape.
It's a unique sensation, and it's often uncomfortable! But my friends there is another way to fart. Yes, I said another way!

Why just last week I was sittin'. Sittin' in this very chair, browsin' this very site.
Yes I was sittin'. And while I was sittin' I felt that familiar pressure. The pressure we all know all too well. The pressure of a tight little bubble of gas winding it's way through my bowels.

But this time it was different. As I felt that fart knocking on my door I took a look around. I say, I looked around for anyone who would see or smell or hear.
Friends, family, coworkers, even gosh darn strangers. But my friends the coast was clear. Yes I was free and clear to let'r rip!

But I decided to try something a little bit different. I passed on my usual lean and "foof". I opted against the raucous blast. I say I did something just a little bit different that made all the difference in the world.

Oh I leaned to the left. I leaned to the left and raised my right cheek off the chair. I raised it up and I put it back down. Right on the right edge of that chair.
Then I leaned to the right. This time to the right, raising my left cheek up and settin' it down.

Now over there on the left edge of the seat was one ass cheek. And way over there on the right edge was the other.
But right in the middle, free and clear and stretched nice and taught was my anus. And my friends what a glorious, clean pink anus it is. I took that anus and I opened the valve nice and slow. Like openin' a shaken up bottle of pop.

And just like that bottle of pop my anus let out a slow "hisssssssss". Yes a hiss! And as I savored the extended release of that one little fart, I felt a sensation. A sensation like none I'd ever felt before on this green Earth.

There was a coolness. A coolness from that escaping gas that refreshed my anus and rectum better than one of ol' Doc Miller's suppositories. It was a coolness that lasted. Stayed with me all day long! It put a skip in my step and a twinkle in my eye and that's why, my friends, I'm here today. Tellin' you about this new great way to fart.

9X / 4.XX line will be better just don't use USB

[Joe_Dragon]

USB will suck in the first 4.XX one.

is_gate_vma bug

[Anonymous Coward]

Introduced in an RC, copied to stable, fixed in RC, still in stable a few releases later.

#define is_gate_vma(vma) ((vma) = &gate_vma)

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/arch/arm/kernel/process.c?id=refs/tags/v3.10.9#n469

If "Stable" isn't stable...

[Tarlus]

...then it's moving too fast.

The decline of quality of time

[rasper99]

As someone who tested drivers with it:
OK through about 3.2 then it started to decline.
Faster decline around 3.4
3.7 - Who needs NFS anyway? Took until 3.9 to fix that

Re:The decline of quality of time

[Billly Gates]

Which is why I do not like Linux. I favor FreeBSD. At least our terrible 5.x and 6.x releases were well understand.

It is kind of like the XP die hard loyalists who feel Windows 7 is too cutting edge and wont change. Better to stick with the devil when it comes to deficiencies.

I use Linux a lot more but only CentOS in a VM. I prefer something works and if I wanted cutting edge I never would have named my account this name and became a MS hater back in the old days when it BSOD all the time. Sadly Windows has surprised Linux when it comes to stability and now use my older nemisis OS as my main as I have given up on FreeBSD and Linux for that purpose.

Re:The decline of quality of time

[Osgeld]

sorry mods, I dont have my life to dedicate to hacking text files in a god damned console cause linux 3.2+ decided it doesnt know what my 1280x1024 mitusubishi monitor is any more

last damn straw

Is Torvalds abusing people again

[musth]

...in the current debate?

Re:Is Torvalds abusing people again

[kthreadd]

If you don't know who he is abusing, then it means it is probably you.

Rate of patch submission?

[gwstuff]

I follow kernel development only cursorily, looking at the kernel mailing list once in a while. But I get the distinct feeling that patch volumes have been higher over the past few months than they would be a few years ago. A version is simply something that group a set of tested patches. Generally, you don't want the sets to get too big, so it seems natural that the speed of version releases is keeping up.

It would be nice to see a plot of the number of commits and number of versions over time.

OHH EMMMM GEEEE!

[Provocateur]

Are we *gasp* agile?Or what!

still pissed there hasn't been a LTS since 3.4

[davydagger]

I am still pissed there hasn't been an LTS since 3.4, and they are NOT being released on a regular cadence

3.0, 3.2 and 3.4, released in short succession all are LTS, and since then none. I understand the devs can't maintain a zillion kernels, but could they at least space them out and/or release on a more stable cadence.

i.e. drop 3.0 or 3.2 for 3.10/11???

Re:still pissed there hasn't been a LTS since 3.4

[e5150]

3.10 is longterm, even tough kernel.org doesn't say so yet. http://www.kroah.com/log/blog/2013/08/04/longterm-kernel-3-dot-10/

Re: Too Fast

[Anonymous Coward]

Naw, I'm still using 2.6.18 and it's fine. :)

That's what he said...

[Anonymous Coward]

Apologizing for mistakenly not pulling out in time...hilarious.

Re: Too Fast

[kthreadd]

Are you on RHEL 5 maybe? Then download 2.6.18 from kernel.org and do a diff; it is quite far from 2.6.18. =)

Poor version control

[Anonymous Coward]

Releasing *two* version 3.10s is a horrible thing to do.
If they released 3.10 and it had a problem, then you release 3.10.1 or 3.11, you do *not* re-release 3.10, because then nobody can be sure which "3.10" they really have.

That's the point of versioning in the first place, isn't it???

Re:Poor version control

[andy.ruddock]

From the article :

"At 9 a.m. PT on Aug. 20, Linux kernel developer Greg Kroah-Hartman announced the 3.10.8 kernel update. At 3:44 p.m. PT, Kroah-Hartman announced the Linux 3.10.9 kernel."

So, it was a new release, not a re-release.

Yeah I have that problem all the time

[Anonymous Coward]

"went into the release, because I mistakenly didn't pull it out in time"

So who?

[Anonymous Coward]

Who is Torvalds going to act like a fucking cunt to over this?

YES

[TheRealDevTrash]

My God, slow it DOWN. I mean I have 1200 servers and they all got updated automtically and then again with this snafu. Who ever coded the automaticallly install the latest stable kernel patch needs to be taken to the toolshed and learned a lesson. And Linus? Why did you approve such an evil....wait...what? Admins can still update their kernels on their own time, based on their own testing platform? Huh.