Slashdot Mirror
Is the Stable Linux Kernel Moving Too Fast?
darthcamaro writes "Yesterday the stable Linux 3.10 kernel was updated twice — an error was made, forcing a quick re-issue. 'What happened was that a patch that was reported to be broken during the RC [release candidate] review process, went into the release, because I mistakenly didn't pull it out in time,' Greg Kroah-Hartman said. The whole incident however is now sparking debate on the Linux Kernel Mailing List about the speed of stable Linux kernel releases. Are they moving too fast?"
its moving along just fine. People make mistakes, get over it, its not the end of the world. Considering its current release speed, the amount of changes made over the long term the Linux kernel folks have as good or better track record than most other software houses.
My karma is not a Chameleon.
Neither will guarantee defectless & errorless code. They help, but you still need to do more. If you are relying only on either, then your process is broken.
"Are they moving too fast?""
Compared to what, Windows, IOS, OSX, What?
>known bug that got by review
>caught
>fixed rapidly instead of waiting for the next release
I don't see the problem.
If this was a regular occurrence, yeah, it'd be a problem. But it's infrequent enough to be "news."
Unlike Patch Tuesdays, which aren't.
--
BMO
"the fun thing about a kernel is that there is no good way to test it except to run it" --Greg Kroah Hartman
I work on PostgreSQL, and nothing goes out until it's been validated on the entire buildfarm. It's hard to have such a thing for the Linux kernel though, because it's so easy for a bug to break test machines. You need to catch when machines are responding, do a hardware reset, and then rollback to a known good kernel instead. It's much harder than most software testing to automate.
Let me try and catch up to it, and ask...
Seriously. Why is this even a question? Did a new stable release show up in your watch or your laptop - or your in flight entertainment system, over night?
Packagers and distribution maintainers aren't exactly up in arms about this...
"Flyin' in just a sweet place,
Never been known to fail..."
As indicated in the debate on LKM, rc kernels get hardly any testing, although all of the tests it does get are mostly by highly motivated and astute testers
Most distros are releasing kernels at least one behind the developers tree, with not a great deal of incentive to update the kernel right away, (even if they make it available in a repository for those wanting it). So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.
So at most, this was an embarrassing incident, and not a bit deal. The amazing thing is that it was caught at all. Some of us remember kernels that got into production distros with serious things broken that should have been caught much earlier.
Sig Battery depleted. Reverting to safe mode.
Clearly, the author of the post has forgotten 3+ year intervals for kernel releases. Of the odd, quickly fixed pothole is the price of 6x release speed, "Hell, yes!" Is the answer.
I mistakenly didn't pull it out in time.
WHAT THE FUCK!
I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.
Test-driven development is not the solution to this problem. And my good gawd, behavior-driven development is even farther away from the solution than fucking test-driven development is.
Behavior-driven development is one of the biggest loads of shit to splash upon our profession in years. Customers and analysts will write tests? Riiiiiiiiiiiiight...
All that we get from BDD is half-arsed tests that don't work and clients or analysts who cry to high heaven about how they hate writing them. And we programmers can't just rewrite them in Java or C# or Python or C++ or whatever our project is using. Noooooooo! They all require stupid English-like syntaxes that need to be translated down into real code by some turdy tool named after a vegetable.
Fuck TDD. Fuck BDD twice over. And please, for the love of all that is good, NEVER suggest that either of them be used seriously, especially for a critical piece of software like the Linux kernel.
... is that it would move even faster.
Really, this is a non-problem. The 'system' worked.
Thank God they're no slick sleezeballs like Ballmer,
but acked and corrected.
Enjoy your new kernel. Relax. Be happy!
Why do you have to compare it to other operating systems? Just look at what should be the right way to do it, maybe learn from other operating systems, but don't just look at the speed of what others are doing and try and match that. If things go wrong because you're moving too fast, you should either slow down, or fix your methodology so you can deal with the speed. If things don't get adapted by distributions because it's a pain to keep supporting, slow down, or make it easier for them to support it. If things go too slow and you miss essential features that everybody needs, speed it up. It's not that hard to rely on your own merits and not be dependent on other operating systems to determine how fast you should be going.
I was promised a flying car. Where is my flying car?
He is working on this one too so he was in two article in a row.
http://tech.slashdot.org/story/13/08/21/1959227/internet-infrastructure-for-everyone
Time for an LTS option. RedHat, Canonical, Debian, should backport security fixes and maybe mature drivers to a LTS kernel for 5 years or so.
For that matter, go ahead and make a LTS gcc fork, backporting security fixes during that same time schedule.
No, you want a frozen kernel. A stable kernel isn't one without bugs, is one where there aren't massive changes and you get dot releases with fixes
There are plenty of older kernels being actively maintained. Stable does not equal recommended for all production needs.
"To those who are overly cautious, everything is impossible. "
No that's only when you break User Space.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Keep in mind that the stable kernel releases are not expected to be production-ready. Linus just produces the input to the actual testing and validation processes, which are performed by the distributions. That assumption is built into the kernel dev team's processes. Not that they intentionally release crap, but they don't perform the sort of in-depth testing that is required before you push a foundational piece of software onto millions of machines.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Doesn't check it works perfectly on all hardware, but a good chunk of the testing could be done on VMs running monkey-scripts which are simply disposed of & a new image spun up if they break. That said, automating hardware resets is hardly an unsolved problem.
I haven't needed to bypass my Linux distro and install a vanilla kernel in over ten years. I can wait. If it hasn't been packaged by the major distributions yet, it also hasn't been widely deployed. There are some highly competent kernel package maintainers out there, and they're likely to catch a lot of things themselves.
Then there's the package release and its early adopters (Debian, for example, releases the latest kernels in Experimental and/or Unstable, letting them cook for a while). I typically pick up new kernels when they're in Debian Testing or Unstable (if I really want a new feature). This minimizes the chance of running into problems.
(note, this works best for people using standard hardware. ymmv regarding embedded or obscure devices)
Use my userscript to add story images to Slashdot. There's no going back.
The logs of the machines that break are the most important part of the test results here. You can't just throw them away when a VM dies without losing most of the testing value. If you're lucky, a busted kernel will stay alive long enough to create a crash dump when running on dedicated hardware. That's less likely to happen on VMs.
It's also possible to automate hardware resets with IPMI or similar lights-out management code. All of that takes special hardware though, and test code like this is painful to build.
VMs create a single 'hardware' platform. You aren't testing more than a fraction of the kernel.
And your automated resets will let you boot a new kernel easily (so does 'reboot' anyway), but if it does break you can sit there resetting over and over again in to the same broken kernel. Yay!
...then it's moving too fast.
/* No Comment */
As someone who tested drivers with it:
OK through about 3.2 then it started to decline.
Faster decline around 3.4
3.7 - Who needs NFS anyway? Took until 3.9 to fix that
You'd need VMs to emulate all sorts of hardware and architectures. We don't have all that yet. We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).
Which rises question of just why are they part of the kernel? Why does a mouse driver need to run at Ring 0?
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
For the purpose of release testing, though, the only thing you care about is whether or not there was a crash. If there was a crash, don't release. Back out the busted patch and release the working version. Then you can spend your time debugging the busted patch, which requires the logs and all.
SIGSEGV caught, terminating
wait... not that kind of sig.
I follow kernel development only cursorily, looking at the kernel mailing list once in a while. But I get the distinct feeling that patch volumes have been higher over the past few months than they would be a few years ago. A version is simply something that group a set of tested patches. Generally, you don't want the sets to get too big, so it seems natural that the speed of version releases is keeping up.
It would be nice to see a plot of the number of commits and number of versions over time.
Are we *gasp* agile?Or what!
WARNING: Smartphones have side effects--most of them undocumented.
You need to do some more reading on how Linux works.
- Michael T. Babcock (Yes, I blog)
I am still pissed there hasn't been an LTS since 3.4, and they are NOT being released on a regular cadence
3.0, 3.2 and 3.4, released in short succession all are LTS, and since then none. I understand the devs can't maintain a zillion kernels, but could they at least space them out and/or release on a more stable cadence.
i.e. drop 3.0 or 3.2 for 3.10/11???
Apologizing for mistakenly not pulling out in time...hilarious.
If you don't know who he is abusing, then it means it is probably you.
Are you on RHEL 5 maybe? Then download 2.6.18 from kernel.org and do a diff; it is quite far from 2.6.18. =)
"went into the release, because I mistakenly didn't pull it out in time"
From the article :
"At 9 a.m. PT on Aug. 20, Linux kernel developer Greg Kroah-Hartman announced the 3.10.8 kernel update. At 3:44 p.m. PT, Kroah-Hartman announced the Linux 3.10.9 kernel."
So, it was a new release, not a re-release.
God: An invisible friend for grown-ups.
You need to do some more reading on how Linux works.
So do you.
Linux is sort of a hybrid kernel now. Some hardware drivers are in ring 0. Quite a lot are no longer. libUSB for example allows userspace USB drivers. They work great. FUSE allows for user space filesystems which work great where absolute top performance is not necessary (eg sshfs, ftpfs etc).
A good fraction of the bluetooth stack, for example anything above L2CAP (the Bluetooth world equivalent of the IP stack's SCTP) , such as ATT and GATT is all userspace (and the non kernel side sucks donkey balls by the way). That means I could (if it didn't suck massive donkey balls) control all the various profiles with the majority of the code in userspace.
All the printer drivers are mostly in userspace (yay).
The graphics (X11) is largely in userspace for now...
Sound has a large userspace component and all the complex stuff like routing, mixing, and figuring out what to send where is in userspace (pulse or Jack).
The Linux kernel as-is is more than capable of running a mouse driver in userspace.
SJW n. One who posts facts.
I've always felt that a rolling release, as exemplified by FreeBSD, naturally stablizes things. They have two branches, STABLE and HEAD (a/k/a CURRENT). New code is commited to HEAD and only once it's stabilized is it MFC'd (Merged From Current). Once in the STABLE branch it's feature frozen, and receives continuous refinement and debugging for the life of the code. There has always been a saying that "FreeBSD works like running water", and I attribute this statement to the development process.
In one of Greg's presentations he stated that Kernel development is the largest and fastest moving software development project in the world, by an order of magnitude. I don't have the answers for what needs to be done, but it speaks volumes that Enterprise Linux distributions choose to use a kernel from 2009. Something needs to be done.
What's a decent alternative? FreeBSD?
Slashdot Valentines Beta Massacre: iT WORKED! The boycotts killed Beta!!
Hmmm :)
Nothing like a good troll to read in the afternoon. I love reading Slashdot at -1 :)
Not very well in this regard, according to the grandparent, which gets us back to my question: why do it this way?
But thanks for your +5 Informative comment anyway. I certainly know more having read it.
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
The All clears an ambiguity. This means the poster means that all the print drivers have most of their stuff in userspace. Rather than most of the print drivers are in userspace.
I don't follow printer drivers closely enough to know if the poster was right. I try and follow the nuance language, it is abused often.
Your parent's statement is not an oxymoron.
If every single print driver has components running in both ring 0 and userspace, but the preponderance of components (by number or 'size') of every single print driver is in userspace, then it is more precise to say that "all print drivers are mostly in userspace" as opposed to "printer drivers are mostly in userspace". The latter is semantically a superset of the former, as it could either mean the same thing, or also describe a situation where some printer drivers are completely implemented in ring 0, but the majority are completely or mostly implemented in userspace such that the preponderance of the set of all printer driver implementations resides in userspace.
In other words, your parent's statement is more precise about not only the aggregate population of printer drivers, but the distribution within the population. Whether that statement is actually correct or even the real intent of your parent poster's statement is another question :).
Good job on not paying attention to the last ten years of Linux development.
So the stuff that isn't a hardware driver attached to the Linux kernel isn't in the kernel ... good job on deducing that.
Printers aren't in the kernel because (get this) they aren't part of the hardware. The Bluetooth stack isn't hardware, the hardware is hardware. That stuff's in the kernel. The sound system? The hardware support is (again) in the kernel, the stuff that's at a higher level isn't.
There are some very specific edge cases, but for the most part, all actual hardware support in Linux is in the kernel because that's where it belongs in a monolithic design.
- Michael T. Babcock (Yes, I blog)
As though I gave myself +5 ... jeez.
The post I responded to was out of touch with how Linux is designed. If you want to know more, go do some reading. Its not hard.
- Michael T. Babcock (Yes, I blog)
My God, slow it DOWN. I mean I have 1200 servers and they all got updated automtically and then again with this snafu. Who ever coded the automaticallly install the latest stable kernel patch needs to be taken to the toolshed and learned a lesson. And Linus? Why did you approve such an evil....wait...what? Admins can still update their kernels on their own time, based on their own testing platform? Huh.
I used to be