Slashdot Mirror

← Back to Stories (view on slashdot.org)

Is the Stable Linux Kernel Moving Too Fast?

Posted by Soulskill on from the committed-to-committing dept.

darthcamaro writes "Yesterday the stable Linux 3.10 kernel was updated twice — an error was made, forcing a quick re-issue. 'What happened was that a patch that was reported to be broken during the RC [release candidate] review process, went into the release, because I mistakenly didn't pull it out in time,' Greg Kroah-Hartman said. The whole incident however is now sparking debate on the Linux Kernel Mailing List about the speed of stable Linux kernel releases. Are they moving too fast?"

47 of 156 comments (clear)

  1. No by Stumbles · · Score: 5, Insightful

    its moving along just fine. People make mistakes, get over it, its not the end of the world. Considering its current release speed, the amount of changes made over the long term the Linux kernel folks have as good or better track record than most other software houses.

    --
    My karma is not a Chameleon.
    1. Re:No by Stumbles · · Score: 2

      Well everyone has bellybuttons but that doesn't mean we all should gaze upon the great naval; at least for to long else your eyes will burn out.

      --
      My karma is not a Chameleon.
    2. Re:No by CAIMLAS · · Score: 3, Interesting

      Really?

      The current process resulted in us having CFQ + EXT3 as the default for a long time (some distros still have this). This basically means any sort of interactive performance is worse than horrible. The only reason we're beyond it now is because EXT3 is on its way out with EXT4 being preferred.

      IIRC, wasn't CFQ one of the first major infrastructural things put into 'stable' after this 'rapid release' approach was adopted?

      Also, udev.

      I'm sure there are other examples... and maybe I'm projecting a bit.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
    3. Re:No by vilanye · · Score: 2

      That is the way it should be.

      That's how Linux can thrive dispite the chaos of getting patches from 1000's of independant programmers.

    4. Re: No by coffbr01 · · Score: 4, Interesting

      I prefer to think that an immediate response from the community is a sign of an active and concerned user base. It might be worse if the community insisted on staying on LTS-type releases.

    5. Re:No by jhol13 · · Score: 3, Interesting

      Linux (the kernel) is accumulating new security holes at least at the same speed as they are fixed.
      Proof: Ubuntu kernel security hole fix rate has been constant for years.

      (actually I have not counted the actual number of holes, only the actual number of kernel security patches - these two should correlate though).

    6. Re: No by TCM · · Score: 2

      As if you can't have both. That's what branches are for! Duh.

      But of course you'd have to understand branches and their use first. Why call it stable if it's just another testbed where everything is shipped regardless? If "stable" is moving so fast that bugs that are already recognized cannot be pulled back in time, something is just wrong.

      --
      Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
    7. Re:No by semi-extrinsic · · Score: 2

      and yes, it also appears some webmasters code their websites assumiing their users have machines with quad cores and 16GB of RAM.

      This. I had to throw more RAM into my wife's laptop because a significant portion of the interior design blogs she reads are coded by seriously inept people.

      --
      for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
    8. Re:No by Error27 · · Score: 2

      I work in kernel security and I would say we have improved. You can't just tell people "don't make mistakes" and expect security to improve the only way you can improve is by improving the process.

      1) We've added a few exploit prevention techniques like hiding kernel pointers.
      2) Our fuzz testers have improved.
      3) Our static checkers have improved.

      But we're not perfect.

      For example, we earlier this year we merged user namespaces. Obviously this is tricky code which deals with security. People had been working on it since 2007, but even after five years we all knew there were going to be some security bugs which we had missed. Code has bugs. That's life. But user namespace is a valuable feature and we had done everything we knew how to do.

      Actually, in some ways, user namespaces will improve security overall because we can use it to remove a setuid binary from the Chrome browser.

      Btw, you can't just look at CVE count. If could be that the bug is old but it was only found recently because of the improved tools. Also two years ago we probably wouldn't have issued a CVE for info leaks like CVE-2013-2148.

  2. Compared to what? by bmo · · Score: 5, Insightful

    "Are they moving too fast?""

    Compared to what, Windows, IOS, OSX, What?

    >known bug that got by review
    >caught
    >fixed rapidly instead of waiting for the next release

    I don't see the problem.

    If this was a regular occurrence, yeah, it'd be a problem. But it's infrequent enough to be "news."

    Unlike Patch Tuesdays, which aren't.

    --
    BMO

    1. Re:Compared to what? by ericloewe · · Score: 2

      Patch Tuesdays aren't news. Patch Tuesdays that break something are.

    2. Re:Compared to what? by jamesh · · Score: 3

      "Are they moving too fast?""

      Compared to what, Windows, IOS, OSX, What?

      Compared to a speed where accidents like this are less likely to happen, if such a thing exists. It could be that OS release cycles are unsafe at any speed!

    3. Re:Compared to what? by geekoid · · Score: 2, Funny

      Now you're being redundant.

      --
      The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
    4. Re: Compared to what? by O('_')O_Bush · · Score: 2

      You mean, break everything. It is rare that a MS update doesn't break *something*.

      --
      while(1) attack(People.Sandy);
    5. Re: Compared to what? by Barlo_Mung_42 · · Score: 2

      Their massive install base works against them here. On Windows a one in a million bug will happen by next Tuesday.

    6. Re:Compared to what? by MikeBabcock · · Score: 3, Informative

      Which would be why you should use an actual distribution kernel and not the compile-it-yourself variety if you need stability and testing.

      --
      - Michael T. Babcock (Yes, I blog)
    7. Re:Compared to what? by semi-extrinsic · · Score: 2

      Indeed. ArchLinux, which is the fastest bleeding-edge, rolling-release distro out there, never released 3.10.8 into [core]. [core] is now at 3.10.7 and an update is flagged to 3.10.9 within a couple of days, skipping .8. The only way you would install 3.10.8 is if you (a) roll your own kernel or (b) use the [testing] repo in Arch. Both of these indicate that you are not a regular user, you are a developer (in some sense).

      --
      for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
  3. Re:TDD by greg1104 · · Score: 4, Insightful

    "the fun thing about a kernel is that there is no good way to test it except to run it" --Greg Kroah Hartman

    I work on PostgreSQL, and nothing goes out until it's been validated on the entire buildfarm. It's hard to have such a thing for the Linux kernel though, because it's so easy for a bug to break test machines. You need to catch when machines are responding, do a hardware reset, and then rollback to a known good kernel instead. It's much harder than most software testing to automate.

  4. Too Fast? by Jeremiah+Cornelius · · Score: 2, Insightful

    Let me try and catch up to it, and ask...

    Seriously. Why is this even a question? Did a new stable release show up in your watch or your laptop - or your in flight entertainment system, over night?

    Packagers and distribution maintainers aren't exactly up in arms about this...

    --
    "Flyin' in just a sweet place,
    Never been known to fail..."
  5. Released kernels are the real testbed by icebike · · Score: 4, Informative

    As indicated in the debate on LKM, rc kernels get hardly any testing, although all of the tests it does get are mostly by highly motivated and astute testers

    Most distros are releasing kernels at least one behind the developers tree, with not a great deal of incentive to update the kernel right away, (even if they make it available in a repository for those wanting it). So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.

    So at most, this was an embarrassing incident, and not a bit deal. The amazing thing is that it was caught at all. Some of us remember kernels that got into production distros with serious things broken that should have been caught much earlier.

    --
    Sig Battery depleted. Reverting to safe mode.
    1. Re:Released kernels are the real testbed by CAIMLAS · · Score: 4, Informative

      From where I'm sitting, as someone who used to routinely build rc releases and use them, this is how things look.

      Five, ten years ago you had people such as myself who would build RC (or AC, etc.) kernel trees to test things and see how they'd work. I know several people who regularly made LKML submissions, many of which turned out to contribute to fixes.

      Today, using the rc releases isn't as practical because they're fairly divergent from distribution patchsets. A lot goes into a distribution kernel which isn't present in the vanilla kernel.org kernels, it seems.

      More often than not, pulling everything together to build our own kernels isn't worth the extra effort: possibly due to the shortened cycle and possibly due to general code maturity, there's little benefit. Maybe our definitions of 'benefit' has changed, too - but arguably, the changes in the kernel today are nowhere near as drastic or significant as when (say) XFS was getting merged with additional kernel and disk schedulers.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
  6. That's what he said! by fizzup · · Score: 5, Funny

    I mistakenly didn't pull it out in time.

  7. WHAT THE FUCK! WHAT THE FUCK!!! by Anonymous Coward · · Score: 2, Interesting

    WHAT THE FUCK!

    I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.

    Test-driven development is not the solution to this problem. And my good gawd, behavior-driven development is even farther away from the solution than fucking test-driven development is.

    Behavior-driven development is one of the biggest loads of shit to splash upon our profession in years. Customers and analysts will write tests? Riiiiiiiiiiiiight...

    All that we get from BDD is half-arsed tests that don't work and clients or analysts who cry to high heaven about how they hate writing them. And we programmers can't just rewrite them in Java or C# or Python or C++ or whatever our project is using. Noooooooo! They all require stupid English-like syntaxes that need to be translated down into real code by some turdy tool named after a vegetable.

    Fuck TDD. Fuck BDD twice over. And please, for the love of all that is good, NEVER suggest that either of them be used seriously, especially for a critical piece of software like the Linux kernel.

    1. Re:WHAT THE FUCK! WHAT THE FUCK!!! by Anonymous Coward · · Score: 5, Funny

      WHAT THE FUCK!

      I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.

      ...

      OK, one of those two must be blatantly false.

    2. Re:WHAT THE FUCK! WHAT THE FUCK!!! by Motard · · Score: 4, Funny

      Thanks Linus, but I think it's an established rule that you can't go releasing new versions of software until the ridicule surrounding your last release has died down. How else are we going to get stories for /.? Microsoft plays along. Why can't you?

    3. Re:WHAT THE FUCK! WHAT THE FUCK!!! by Zeromous · · Score: 5, Funny

      Only said fuck four times not including subject. Not Linus.

      --
      ---Up Up Down Down Left Right Left Right B A START
  8. What's good. by dutchwhizzman · · Score: 3, Insightful

    Why do you have to compare it to other operating systems? Just look at what should be the right way to do it, maybe learn from other operating systems, but don't just look at the speed of what others are doing and try and match that. If things go wrong because you're moving too fast, you should either slow down, or fix your methodology so you can deal with the speed. If things don't get adapted by distributions because it's a pain to keep supporting, slow down, or make it easier for them to support it. If things go too slow and you miss essential features that everybody needs, speed it up. It's not that hard to rely on your own merits and not be dependent on other operating systems to determine how fast you should be going.

    --
    I was promised a flying car. Where is my flying car?
  9. No, you want a frozen kernel by robmv · · Score: 4, Informative

    No, you want a frozen kernel. A stable kernel isn't one without bugs, is one where there aren't massive changes and you get dot releases with fixes

  10. Absolutely not by stox · · Score: 2

    There are plenty of older kernels being actively maintained. Stable does not equal recommended for all production needs.

    --
    "To those who are overly cautious, everything is impossible. "
  11. Re:Nothing a few f-bombs for linus can't fix by Virtucon · · Score: 2

    No that's only when you break User Space.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
  12. Re:TDD by malacandrian · · Score: 2

    Doesn't check it works perfectly on all hardware, but a good chunk of the testing could be done on VMs running monkey-scripts which are simply disposed of & a new image spun up if they break. That said, automating hardware resets is hardly an unsolved problem.

  13. This is why we have Linux Distributions by Khopesh · · Score: 2

    I haven't needed to bypass my Linux distro and install a vanilla kernel in over ten years. I can wait. If it hasn't been packaged by the major distributions yet, it also hasn't been widely deployed. There are some highly competent kernel package maintainers out there, and they're likely to catch a lot of things themselves.

    Then there's the package release and its early adopters (Debian, for example, releases the latest kernels in Experimental and/or Unstable, letting them cook for a while). I typically pick up new kernels when they're in Debian Testing or Unstable (if I really want a new feature). This minimizes the chance of running into problems.

    (note, this works best for people using standard hardware. ymmv regarding embedded or obscure devices)

    --
    Use my userscript to add story images to Slashdot. There's no going back.
  14. Re:TDD by greg1104 · · Score: 3, Interesting

    The logs of the machines that break are the most important part of the test results here. You can't just throw them away when a VM dies without losing most of the testing value. If you're lucky, a busted kernel will stay alive long enough to create a crash dump when running on dedicated hardware. That's less likely to happen on VMs.

    It's also possible to automate hardware resets with IPMI or similar lights-out management code. All of that takes special hardware though, and test code like this is painful to build.

  15. If "Stable" isn't stable... by Tarlus · · Score: 2

    ...then it's moving too fast.

    --
    /* No Comment */
  16. Re:What a stupid question. by Rich0 · · Score: 4, Insightful

    Ah yeah, like kernel.org isn't trusted. Yes I know you said "distribution" but really now.

    He wasn't talking about trusting that it didn't contain a trojan or something. By trust he meant vetted for quality.

    It is a legitimate concern. The whole reason for having a release cycle is to have sufficient QA to prevent issues like this from happening. Distros provide that service - when Linus/Greg call a kernel done, they call it ready to start being tested. RHEL is still running 2.6 (albeit with backports).

  17. Re:TDD by hobarrera · · Score: 2

    You'd need VMs to emulate all sorts of hardware and architectures. We don't have all that yet. We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).

  18. Re:Time for an LTS Option by greg1104 · · Score: 5, Informative

    3.10 is the next LTS kernel by Linux standards. The existing long term kernels are 2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04), 2.6.34, 3.0, 3.2.50 (used in Ubuntu 12.04 LTS), and 3.4.59.

  19. Re:TDD by ultranova · · Score: 3, Insightful

    We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).

    Which rises question of just why are they part of the kernel? Why does a mouse driver need to run at Ring 0?

    --

    Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

  20. Re:TDD by The_Wilschon · · Score: 4, Insightful

    For the purpose of release testing, though, the only thing you care about is whether or not there was a crash. If there was a crash, don't release. Back out the busted patch and release the working version. Then you can spend your time debugging the busted patch, which requires the logs and all.

    --
    SIGSEGV caught, terminating

    wait... not that kind of sig.
  21. Rate of patch submission? by gwstuff · · Score: 2

    I follow kernel development only cursorily, looking at the kernel mailing list once in a while. But I get the distinct feeling that patch volumes have been higher over the past few months than they would be a few years ago. A version is simply something that group a set of tested patches. Generally, you don't want the sets to get too big, so it seems natural that the speed of version releases is keeping up.

    It would be nice to see a plot of the number of commits and number of versions over time.

  22. Re:TDD by MikeBabcock · · Score: 4, Informative

    You need to do some more reading on how Linux works.

    --
    - Michael T. Babcock (Yes, I blog)
  23. That's what he said... by Anonymous Coward · · Score: 2, Funny

    Apologizing for mistakenly not pulling out in time...hilarious.

  24. Re:Time for an LTS Option by kthreadd · · Score: 2

    2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04)

    Given the amount of backported features and hardware support that Red Hat continuously adds to their kernel I highly doubt that it looks anywhere close to 2.6.32 from kernel.org.

  25. Re:Poor version control by andy.ruddock · · Score: 3, Informative

    From the article :

    "At 9 a.m. PT on Aug. 20, Linux kernel developer Greg Kroah-Hartman announced the 3.10.8 kernel update. At 3:44 p.m. PT, Kroah-Hartman announced the Linux 3.10.9 kernel."

    So, it was a new release, not a re-release.

    --
    God: An invisible friend for grown-ups.
  26. Re:TDD by serviscope_minor · · Score: 5, Informative

    You need to do some more reading on how Linux works.

    So do you.

    Linux is sort of a hybrid kernel now. Some hardware drivers are in ring 0. Quite a lot are no longer. libUSB for example allows userspace USB drivers. They work great. FUSE allows for user space filesystems which work great where absolute top performance is not necessary (eg sshfs, ftpfs etc).

    A good fraction of the bluetooth stack, for example anything above L2CAP (the Bluetooth world equivalent of the IP stack's SCTP) , such as ATT and GATT is all userspace (and the non kernel side sucks donkey balls by the way). That means I could (if it didn't suck massive donkey balls) control all the various profiles with the majority of the code in userspace.

    All the printer drivers are mostly in userspace (yay).

    The graphics (X11) is largely in userspace for now...

    Sound has a large userspace component and all the complex stuff like routing, mixing, and figuring out what to send where is in userspace (pulse or Jack).

    The Linux kernel as-is is more than capable of running a mouse driver in userspace.

    --
    SJW n. One who posts facts.
  27. Re:still pissed there hasn't been a LTS since 3.4 by e5150 · · Score: 2

    3.10 is longterm, even tough kernel.org doesn't say so yet. http://www.kroah.com/log/blog/2013/08/04/longterm-kernel-3-dot-10/

  28. Re:TDD by ultranova · · Score: 2

    You need to do some more reading on how Linux works.

    Not very well in this regard, according to the grandparent, which gets us back to my question: why do it this way?

    But thanks for your +5 Informative comment anyway. I certainly know more having read it.

    --

    Forget magic. Any technology distinguishable from divine power is insufficiently advanced.