Slashdot Mirror


Live Patching Now Available For Linux

New submitter cyranix writes "You may never have to reboot your Linux machine ever again, even for kernel patching," and excerpts from the long (and nicely human-readable) description of newly merged kernel code that does what Ksplice has for quite a while (namely, offer live updating for Linux systems, no downtime required), but without Oracle's control. It provides a basic infrastructure for function "live patching" (i.e. code redirection), including API for kernel modules containing the actual patches, and API/ABI for userspace to be able to operate on the patches (look up what patches are applied, enable/disable them, etc). It's relatively simple and minimalistic, as it's making use of existing kernel infrastructure (namely ftrace) as much as possible. It's also self-contained, in a sense that it doesn't hook itself in any other kernel subsystem (it doesn't even touch any other code). It's now implemented for x86 only as a reference architecture, but support for powerpc, s390 and arm is already in the works (adding arch-specific support basically boils down to teaching ftrace about regs-saving).

117 comments

  1. Discussed in the Kernel Report by rgbe · · Score: 5, Informative

    This, among other things was discussed in the Kernel Report, at the recent Linux Conf in Auckland, New Zealand:
      https://www.youtube.com/watch?...

    1. Re:Discussed in the Kernel Report by buchner.johannes · · Score: 1

      ehm, could you point to the minute?

      --
      NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
    2. Re:Discussed in the Kernel Report by buchner.johannes · · Score: 4, Informative
      --
      NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
  2. No more downtime by ArcadeMan · · Score: 5, Funny

    Which means you can keep it up forever!

    (PHRASING!)

    1. Re:No more downtime by fulldecent · · Score: 2, Informative

      Holy shitsnacks. There are more Archer seasons? I need to step up my piracy.

      We are talking about Archer, right?

      --

      -- I was raised on the command line, bitch

    2. Re:No more downtime by aliquis · · Score: 2, Insightful

      Meanwhile Windows still seem to ask you to reboot about once a week (every second week?) to install the updates ..

    3. Re:No more downtime by johanwanderer · · Score: 1

      Or, you could just have rolling updates and the clients wouldn't know that the servers were rebooted.

    4. Re:No more downtime by Anonymous Coward · · Score: 0

      Which means you can keep it up forever!

      (PHRASING!)

      So it should be called Viagra4linux?

    5. Re:No more downtime by Anonymous Coward · · Score: 0

      We need to get phrasing back in the rotation

    6. Re:No more downtime by Anonymous Coward · · Score: 0

      Take it to maintain your hard drive

    7. Re:No more downtime by Creepy · · Score: 1

      Lack of patching never stopped me from striving for high uptimes, but now I can have high uptimes, safely.

    8. Re:No more downtime by Dutch+Gun · · Score: 1

      Actually, once a month - second Tuesday of the month is patch day.

      I read the commit notes, but am still a bit fuzzy at how the system can patch code that's currently in use by running processes... Does anyone understand the mechanism used to do this? The mention of stop_machine() suggests that processes affected by the patch are being shut down and restarted. If that's the case, then although the machine isn't necessarily rebooted, processes may go down and come back up during this procedure, right? Or am I misinterpreting this?

      --
      Irony: Agile development has too much intertia to be abandoned now.
    9. Re:No more downtime by BronsCon · · Score: 1, Offtopic

      If you have an uptime lasting longer than 4 decades, call your sysadmin.

      --
      APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
    10. Re:No more downtime by Atzanteol · · Score: 1

      Ubuntu pesters me nearly daily to reboot....

      --
      "Ignorance more frequently begets confidence than does knowledge"

      - Charles Darwin
    11. Re:No more downtime by gavron · · Score: 5, Informative

      Ok, so here's the simple answer. Note: I'm generalizing a lot to make this simple.

      All functions have a known entry point which you can think of a name that you can call like
      print("hello world"); -- calls "print" so it knows where "print" is.

      Somewhere in the memory was loaded the function print(). There's also a symbol which allows everyone who wants to call print() to know where it is.

      The livepatch loads a new function into memory. Let's call it print2(). It then goes over and makes the symbol that used to let everyone know where print() is point to print2(). Anyone that comes after this patch will still think they are calling print() but in fact will be calling print().

      The stop_machine() is part of how ksplice (the proprietary-vendor method does it). That is not part of kernel live patch (klp).

      What klp does is ensure that a process is in a "good point" to be messed with, and then changes its pointer to e.g. print().

      That allows no changes to affect the process until that pointer to print() is changed at which point any subsequent call to print() will run print() instead.

      Ehud
      P.S. I have some code from the early 1990s where we used to do this on VMS/OpenVMS. We literally patched the running kernel (much as is done here) and allowed a system to run for years with newer kernel code.

    12. Re:No more downtime by H0p313ss · · Score: 1

      Do you want to have puns? Because this is how you get puns!

      --
      XML is a known as a key material required to create SMD: Software of Mass Destruction
    13. Re:No more downtime by aliquis · · Score: 1

      Ubuntu pesters me nearly daily to reboot....

      I don't know whatever you run some rolling release / unstable version of it or not.

      But my experience would be that kernel upgrades is what requires reboots and even if running openSUSE with updates and pacman repositories enable of course you'll get new kernels but it's not like you have to reboot because of that or that anyone is forcing you.

      I don't know what Ubuntu says because I don't use it.

      In Windows when you get updates it gives a generic "fixes some problems in product" and nothing more really and then a link you can click which open your web-browser to reach yet another "fixes various problems in .." without really saying what or why you should bother.

      So you just have to install the crap and sometimes it ask you to reboot sometimes not.

      And I feel like I get updates more often than once a month for my Windows installation.

      And if it want to reboot it just say so, not why.

      And back when I had just installed it it asked me whatever I wanted to reboot. I didn't, and then it said it would reboot the machine for me regardless in a week. Fuck that.

      I got rid of that somehow (both that planned reboot and any such future activities) but it completely suck. Let me decide whatever I want to reboot please. And it's not like the solution for that problem was intuitive.

      With Ubuntu I wouldn't feel like I had to reboot. Even if I got kernel upgrades so what? If it's not broken already changes are I wouldn't have to reboot. Whatever things some time in the future would rely on running the latest stuff I don't know. Maybe I would figure out at that time. I don't feel like I'm forced to reboot such a system at least.

      And with "no-reboot" patches of the kernel even less so.

    14. Re:No more downtime by aliquis · · Score: 1

      I feel like it's more than that. Mind you in this case it likely had wanted me to reboot for some time because I had some upgrades and when it was back up it kinda asked me to reboot again but I guess that may have been the second Tuesday in the month. As in maybe I rebooted on say Sunday and then again on Wednesday or something such.

      As for updates only coming once a month I don't know.

    15. Re:No more downtime by Dutch+Gun · · Score: 1

      Ah, I see, so it's a real-time library function pointer fixup then... no need to shutdown and restart processes at all (which would make it much less useful, I guess). I guess by nature there wouldn't typically be any changes that should have extraneous side effects from C-style library functions, so this should work pretty well in practice.

      Thanks for the high-level explanation!

      --
      Irony: Agile development has too much intertia to be abandoned now.
    16. Re:No more downtime by Atzanteol · · Score: 1

      I *just* updated and it's pestering me to reboot again. Mostly libkrb stuff.

      Just because you ignore it doesn't mean it's not asking you to reboot.

      --
      "Ignorance more frequently begets confidence than does knowledge"

      - Charles Darwin
    17. Re:No more downtime by aliquis · · Score: 1

      I'm not sure openSUSE tell me to reboot / I don't know how it expresses it.

      Though I guess it would be interesting to get to know whatever any of the updates was security related or whatever it's just upgraded software in general.

      Also Windows didn't just told you you should reboot or asked you whatever you wanted it.

      It enforced the reboot regardless of what you picked.

      I understand I need to reboot for the kernel stuff to happen but in Linux I would worry less about being without those upgrades than I would in Windows.

      That I have no clue (though I guess there's some way to actually get one) what Windows updated doesn't help either.

      But mostly it REALLY wants me to reboot whereas I assume a Linux message about it would be like "You need to reboot to to complete these updates" (for lack of better word / sentence.)

    18. Re:No more downtime by Anonymous Coward · · Score: 0

      The best example of a PBKAC I've seen this year, but it's still early.

    19. Re:No more downtime by Anonymous Coward · · Score: 0

      Doesn't even make any sense. As for when I reboot I know that thank you. As for getting upgrades I just felt it happened more often than once a month. Maybe life just go past fast though.

      http://windows.microsoft.com/s...

      "Om du inte anvÃnder automatisk uppdatering bÃr du sÃka efter uppdateringar minst en gÃ¥ng i veckan. Microsoft ger oftast ut viktiga uppdateringar den andra eller fjÃrde tisdagen i mÃ¥naden. De kan Ãven ges ut vid andra tidpunkter."

      "MÃ¥nga uppdateringar installeras dock inte automatiskt."

      So even Microsofts text says if usually happen ever second OR fourth tuesday which to begin with would be about twice as often, but it also says that it can also happen at other times.

      And that many updates isn't installed automatically so I guess if Windows update has fetched some you can later run Windows update and get some others which wasn't fetched automatically beyond the 1 or 2 Tuesdays per month.

    20. Re:No more downtime by TheDarkMaster · · Score: 1

      Uptime is irrelevant for a desktop.

      --
      Religion: The greatest weapon of mass destruction of all time
    21. Re:No more downtime by loufoque · · Score: 1

      That should have been obvious from the summary, it said it was based on ftrace.
      ftrace allows to intercept function calls for the purpose of tracing or benchmarking.

    22. Re:No more downtime by Anonymous Coward · · Score: 0

      Congratulations VMS/ksplice/Linux, only 5/23/30 years after the Amiga did this ;)

      I have some code from the early 1990s where we used to do this on VMS/OpenVMS.

    23. Re:No more downtime by aliquis · · Score: 1

      Uptime is irrelevant for a desktop.

      I'm not talking "uptime" as a number. I'm talking having to reboot to get upgrades installed.

      Regardless clearly it's not irrelevant for me.

      I don't really give much of a shit how you feel it is. That doesn't apply to me. I've got my reasons.

    24. Re:No more downtime by aliquis · · Score: 1

      And there we go again, yet another important upgrade now, in this case an update for Windows Defender so that one won't require a reboot AFAIK.

      I may not install the upgrades as soon as they become available and hence they may add up before I finally install them so the occasions for them may be more often than on my machine.

      But:
      http://i58.tinypic.com/28bbqsm...

      So clearly there was 10 occasions in December, 6 in January and 5 so far in February for me.

      Eat my ass regarding once a month the second Tuesday.

      The ones regarding the base OS alone may be less frequent / once a month and maybe one don't have to reboot more often than that. It's not like I sit rebooting more often than I have to.

    25. Re:No more downtime by TheDarkMaster · · Score: 1

      Well, let me see... Having to restart a desktop computer is not a problem, after all you supposedly are not running a server. And if despite being a desktop you keeps it on 24/7 then you are wasting electricity, because you are not using your computer 24 hours a day non stop right? Then he be occasionally off is not a problem. And when Windows asks to restart it does not require you to do so immediately, you can finish what you was doing and then restart. What's the big problem with doing this?

      --
      Religion: The greatest weapon of mass destruction of all time
    26. Re:No more downtime by Dutch+Gun · · Score: 1

      I'm not very familiar with Linix programming yet, which is why I tried to ask nicely for an explanation. Sorry for not already being omniscient.

      --
      Irony: Agile development has too much intertia to be abandoned now.
    27. Re:No more downtime by Dutch+Gun · · Score: 1

      You'll see malware definitions (Windows Defender) pretty often, but these are tiny and never require a reboot. They should be practically invisible to you, and I don't think MS really considers them "patches" per se. I also see Flash updates more frequently - that also shouldn't require a reboot, but I don't use Flash so I'm not sure. Obviously, MS doesn't control the timing of those patches - that's Adobe, but MS apparently just pushes them along. There are occasionally out-of-band patches as well that don't fall under the regular schedule, but these seem to be fairly rare. On rare occasions, errors in patches are rolled back fairly soon after - I think that happened a couple of times in the last year - it almost always makes news.

      The "Patch Tuesday" was largely created to make things easier for enterprise customers by MS, so they'd have more predictable patching schedules. Generally, large enterprises will control the patching themselves, first testing the patches out on test machines, and then deploying them across the company workstations in a controlled fashion.

      For consumer customers, it just generally means we pull down the bulk of the patches directly from MS within a day or two of that time. Maybe your experience is different from mine, but I really do typically only have to reboot my Window machine once a month because of patches.

      --
      Irony: Agile development has too much intertia to be abandoned now.
    28. Re:No more downtime by aliquis · · Score: 1

      It is a problem for me.

      Try to get that out of your tiny (or gapping, what do I know) asshole (maybe it's just too much to try to get into your.. uhm.. ass to understand for you.)

      Ohnoz, I'm wasting electricity. Thanks for telling me!

      I've done that lots of years. Now I have functional sleep at least so sometimes but not always I use that which is much better than having it on. Hibernate doesn't work so I don't use that.

      It is a problem. It may not be for you but it is for me. Get over it.

      Windows did require me to restart, the restart after a week/a week + one day(?) was enforced.

      http://answers.microsoft.com/e...
      http://www.redmondpie.com/how-...

      Now shut up stupid. It doesn't matter that your opinion is different. Mine is what it is.

    29. Re:No more downtime by aliquis · · Score: 1

      May be patches to Windows Defender database but not for Windows. I never said I always had to reboot. And I likely exaggerated.

      The point was that Windows want me to reboot a lot and without additional hard to reach tweaking it even enforced reboot of its own will.

      I don't care about patch Tuesday. I think it's stupid to not send out the updates as soon as possible. If companies don't want them then fine let them handle it. Thought I guess they would suffer from such a solution. Having updates isn't a problem. Windows requiring reboots and worse forcing me to reboot is the problem.

    30. Re:No more downtime by TheDarkMaster · · Score: 1

      Well, it seems clear to me that you really should use a Linux server, if it's not what you already do. So why not stick with your server rather than offend others like a crybaby who was upset, crybaby?

      --
      Religion: The greatest weapon of mass destruction of all time
    31. Re:No more downtime by Anonymous Coward · · Score: 0

      I'm holding back on the offensive and cursing front..

  3. Reboot for Systemd by Anonymous Coward · · Score: 3, Funny

    Yup. Exactly.

    But then I guess the quest for epic uptime is bogus, right? Who the heck would want their system running 24/7 all the time?

    *waits for Systemd flamewar to break out*

    1. Re:Reboot for Systemd by Anonymous Coward · · Score: 1

      Yes, why would I want my laptop running 24/7?

    2. Re:Reboot for Systemd by Anonymous Coward · · Score: 1

      I certainly never want to test production systems to see if they'll boot after a power failure. This is great only for machines that actually have an uptime requirement (gathering live data from an experiment, waiting for an external trigger like alarm systems, etc.) If you have a high availability requirement, then you really ought to use failover instead of relying on the uptime of a single machine, and make sure your machines will recover from soft and hard reboots.

    3. Re:Reboot for Systemd by Anonymous Coward · · Score: 0

      Yes, that is what any person who actually manages servers does. Only the people running dumpster-dived servers in their parents' basements think that you keep one machine running forever.

    4. Re:Reboot for Systemd by geantvert · · Score: 1

      That's odd! I just upgraded systemd and no reboot was needed!

    5. Re:Reboot for Systemd by Anonymous Coward · · Score: 0

      No reboot needed or no reboot done?

    6. Re:Reboot for Systemd by Rich0 · · Score: 1

      No reboot needed or no reboot done?

      It isn't like the version in RAM magically stops working. If you want to use the features of the new version, then sure, you'll need to reboot.

      Then again, I don't see any reason why systemd could just re-exec itself, and have the new version inherit the state of the previous version. I mean, that is basically what kexec is doing.

      Not really specific to systemd either - if sysvinit has some bug and you want to fix it, you need to reboot. The main difference is that systemd does more and thus there may be more incentive to update it.

      If you're running a backport-oriented distro then most likely you'll have no more cause for rebooting for systemd than you'd have for rebooting for any of your applications.

    7. Re:Reboot for Systemd by geantvert · · Score: 1

      As the name indicates systemctl daemon-reexec is used to re-exec the daemon after an upgrade. No reboot needed.

    8. Re:Reboot for Systemd by caseih · · Score: 1

      Well when you post flame bait what do you expect, especially when it's FUD pure and simple. If you've used systemd, or done even a little bit of research you would know that 99% of systemd's suite of services does not run in pid 1. I think at this point pid 1 is rather stable and unchanging. It's the ancillary (and I might add mostly optional) components that are getting the updates lately it seems. I've run put in several updates for systemd since RHEL7 and Centos 7 came out, and *none* of them have required an reboot so far. Kernels update more frequently I might add, which did require a reboot until now. With systemd updates, individual subsystems can restart after an update. Only the little init 1 process would need a reboot to replace, but that's the case for *any* init system. And I'm sure the linux kernel could be modified to restart init 1 some day.

      Just heading off the FUD at the pass here.

    9. Re:Reboot for Systemd by Anonymous Coward · · Score: 0

      I don't see any reason why systemd could just re-exec itself

      You'd think that, but I just upgraded dbus due to CVE-2015-0245 and both the upgrade and attempting to manually restart it with "service dbus restart" gave me "System dbus already running, not starting."

      If that's how the rest of systemd is going to work when Debian switches to it, it's going to be a huge pain in the ass.

    10. Re:Reboot for Systemd by koinu · · Score: 1

      *waits for Systemd flamewar to break out*

      Ok, I'll try...

      Of course, as long as the kernel is patching itself, systemd will take over the kernel's role and keep your system running. Until next year, then the kernel won't get patches anymore, because systemd will fork it's own kernel service systemd-kerneld which will make Linux standalone kernel irrelevant.

    11. Re:Reboot for Systemd by Rich0 · · Score: 1

      Actually, it turns out that systemd can re-exec itself.

      However, you're now talking about dbus. The long-term solution to that is to just use kdbus, and as is the topic of the article, that can be upgraded in place as part of the kernel.

  4. Time to update the ol' t-shirt... by DigitAl56K · · Score: 5, Funny

    ... to a more extreme version:

    "I don't always test my code, but when I do it's via live patching the kernel on production"

  5. Doesn't hook itself in any other kernel subsystem? by Anonymous Coward · · Score: 3, Funny

    Is this the anti-systemd?

  6. Re:Doesn't hook itself in any other kernel subsyst by Anonymous Coward · · Score: 0

    Is this the anti-systemd?

    <GANDALF>It IS systemd.

    Or rather, systemd as it should have been.</GANDALF>

  7. Re:Sounds great by bill_mcgonigle · · Score: 4, Funny

    I can't possibly see this ever causing a problem with this because linux is very secure and there is absolutely no way for a malicious user to get elevated access on anything that runs linux. This includes android phones, which are totally invulnerable to hacking.

    You should suggest to Linus that he make kernel features configurable so people can do different builds for different targets. Put it in a dot-config file or something. Maybe in a future release...

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  8. Re:Sounds great by Anonymous Coward · · Score: 0, Flamebait

    I love recompiling kernels and updating config files whenever I buy a new mouse. It's the best system ever, and a great way to never open up security holes because all of those options are really well defined.

    Truly, this is the year of linux on the desk top.

  9. Unintended consequences? by marciot · · Score: 4, Interesting

    Maybe I’m old school, but this sort of bothers me. One of the nice things about rebooting is that it clears out old crud and gives you a reassurance that the system can bring itself up by its bootstraps. I can imagine live patching giving rise to a scenario where you have a machine that hasn’t been rebooted for years and when a power glitch finally brings it down, you find that what is on disk is different than what was in RAM and your kernel is corrupt or not bootable.

    I think live patching would make sense if we had non-volatile system RAM (i.e. universal memory), but until then, it seems like rebooting is a pretty good sanity check that things are alright.

    1. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Besides, when else would you fsck your disks?

    2. Re:Unintended consequences? by techno-vampire · · Score: 1

      In theory, at least, you patch or update the software image on disk and this allows the working copy in RAM to use those patches without being restarted. Thus, if and when you need to reboot, what you load is functionally identical to what you were running before. Of course, that's only in theory. In practice, there's always the possibility that what you get at reboot won't be quite the same as what you had before because of some sort of read/write glitch that slipped past the error checking and mucks things up. Yes, that can happen now, but if you need to reboot to get (let's say) a new kernel running, you don't have to wait for an emergency to find out and, you may be able to reboot into an older, working version until things get corrected. I didn't RTFA, so I don't know if this new process leaves you with a backup version or not.

      --
      Good, inexpensive web hosting
    3. Re:Unintended consequences? by bill_mcgonigle · · Score: 1

      So your use case allows for reboots - not every one does. And far more use cases allow for scheduled reboots, but not necessarily immediate reboots as soon as a security vulnerability is published.

      Fedora-derived (e.g. EL7) and Ubuntu disros can use Redhat's kpatch support (but no patches provided in EL7 -updates yet) whilst SuSE has kGraft which as of November has had a real update stream available. People don't run these things because it's easier than rebooting.

      Besides, fix your runtime problems, set up a test cluster, and monitor your systems better - relying on side-effects is never a good plan.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    4. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      gives you a reassurance that the system can bring itself up by its bootstraps

      I'm not sure I understand the logic in this.

      If you reboot and it doesn't, it is safe to say that it can't. And even after you reboot successfully, that is not a guarantee that will next time.

      Maybe you mean reboot at a convenient time where you can screw with it if you need to vs sudden death @ 5 o'clock. :D

    5. Re:Unintended consequences? by mmell · · Score: 1

      You're obviously not "old school" enough - we truly old ones remember working on mainframes and minicomputers - not x86 commodity-grade hardware. "Old ones" such as myself remember platforms which could withstand a disk, memory . . . even a processor failure without any service interruptions. I personally have worked on minis and mainframes with over ten year uptimes despite multiple hardware failures. The x86 stack can't even come close to that kind of reliability. This is a first step (possibly the last step - it may just be impossible to make microcomputers as hardware fault tolerant as real computers).

    6. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Your description doesn't fit reality in all cases.

      I apply patches to a test system using ksplice, and after a quick test-case check I then reboot the test system to make sure it can boot from scratch.

      So your "you must do this since you aren't" is actually telling me to do something I already do.

      However once the ksplice updated kernel is verified, and booting is verified, why on earth do you think it is a good idea to reboot the other 1500 VMs that are a perfect clone of the test system when clearly it is both not needed and proved to not be needed?

      What a waste of time, electricity, and resource IO on the virtual hosts...

    7. Re:Unintended consequences? by MachineShedFred · · Score: 1

      Yeah, the x86 stack doesn't need that kind of reliability, because of the inexpensiveness of the hardware. If you need that kind of uptime, you buy 3 and put them behind a load balancing scheme. You end up with more capacity, the same reliability, and still less expense. Especially in the world of virtualized server instances.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    8. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Yeah, the x86 stack doesn't need that kind of reliability, because of the inexpensiveness of the hardware. If you need that kind of uptime, you buy 3 and put them behind a load balancing scheme. You end up with more capacity, the same reliability, and still less expense. Especially in the world of virtualized server instances.

      Then you have to have redundant load balancers. Which means you need a load balancer for the load balancers.
      In the end, it's always a single point of failure. If you really want uptime, you use mainframes.

    9. Re:Unintended consequences? by bloodhawk · · Score: 1

      Maybe I’m old school, but this sort of bothers me. One of the nice things about rebooting is that it clears out old crud and gives you a reassurance that the system can bring itself up by its bootstraps. I can imagine live patching giving rise to a scenario where you have a machine that hasn’t been rebooted for years and when a power glitch finally brings it down, you find that what is on disk is different than what was in RAM and your kernel is corrupt or not bootable.

      I think live patching would make sense if we had non-volatile system RAM (i.e. universal memory), but until then, it seems like rebooting is a pretty good sanity check that things are alright.

      I agree, especially in this day and age where it is so easy to build scale out solutions that make restarting a server a non issue. I think worst of all I can imagine the nightmare of finding that one of the 30 or 40 patches you applied since last reboot caused a critical issue or incompatibility that is only discovered through a reload of drivers and now you have to try and track down which bitch did the damage.

    10. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Scrub your arrays from the bad memories, scrub your disks from the bad bits. The regular scrubbing is the answer, son.

    11. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Yeah, the x86 stack doesn't need that kind of reliability, because of the inexpensiveness of the hardware. If you need that kind of uptime, you buy 3 and put them behind a load balancing scheme. You end up with more capacity, the same reliability, and still less expense. Especially in the world of virtualized server instances.

      Then you have to have redundant load balancers. Which means you need a load balancer for the load balancers.
      In the end, it's always a single point of failure. If you really want uptime, you use mainframes.

      No it isn't. we run with failover load balancers for the load balancers. we run with multiple ISP connections, multiple datacenters, each server is patched into multiple switches. We really only have one single point of failure and that is the fibre that leaves the city we are in, it has been cut once in the last 10 years by accident, but then at that time we had mainframes and that was a single point of failure for them.

    12. Re:Unintended consequences? by bloodhawk · · Score: 1

      you have never heard of failover load balancers? every vendor supports various failover options.

    13. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Huh huh... Just don't run Windows servers...

      I have watched that "redundant server" crap fail. First server failed, failover, second failed, failover, third failed...

      All because MS hadn't implemented a proper way to reject a simple mail message that overflowed their disk.

    14. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      You have a higher user ID than the OP so what you say doesn't matter. No one cares about your mainframes. You can't game on them, they're trash.

    15. Re:Unintended consequences? by Blaskowicz · · Score: 1

      x86 did gain reliability features years ago, with the Nehalem-EX series and successors.

      http://www.anandtech.com/show/...

      Not sure if that's close enough for you. A year ago there were some additional RAS features (lower quality article : ) http://semiaccurate.com/2014/0...
      Perhaps it doesn't go as far as the most paranoid mainframes but I wonder if such systems can be called a minicomputer.

    16. Re:Unintended consequences? by mmell · · Score: 1

      Al, is that you?

    17. Re:Unintended consequences? by Blaskowicz · · Score: 1

      I find it funny, like a RAID array efficiently replicating the corrupted data or data loss over all disks.

    18. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      for users a reboot doesn't clear all the gnome or X crud out of tmp directory.....

      do yum updates (or similar) have a restart service flag (depending on what the update includes)?

    19. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Rebooting periodically can address memory leaks. It also forces memory to be re-allocated and flushes the caches. The hardware gets to experience a power off state. All good things from a systems housekeeping perspective.

      However you want to reboot on your own schedule and terms. Live patching can give you much better control over the timing of the system restarts and that's a Very Good Thing.

    20. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      "Getting rid of crud" is just symptom of broken system that does not cleanup properly (*cough*Windows*cough*) - that's not the Un*x way.

    21. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      that is hardly a windows server fail. That is a design fail of those that built the system. replication is intended to replicate data regardless of it being good or bad, and any retard that lets their disk overflow really shouldn't be running a critical system.

    22. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      All because MS hadn't implemented a proper way to reject a simple mail message that overflowed their disk.

      They have, it is called disk quotas/resource throttling and not hiring morons to do system configuration. you can also set the store sizes of exchange which will then reject the mail message so this sounds like you have a set of incompetent fools running your IT for something like that to happen.

    23. Re:Unintended consequences? by fnj · · Score: 1

      do yum updates (or similar) have a restart service flag (depending on what the update includes)?

      The hook is there; at least some packages do this when updated - but I wouldn't be too confident that they all do.

    24. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Nah, there were no empty threats of legal action against you. Doesn't match the basic pattern.

    25. Re:Unintended consequences? by Anonymous Coward · · Score: 0

      Eh... I have a SPARC server in production that has been running for eight years and counting. The only failures we've had were drives but they can be hot-swapped.

  10. Fresh boot... by michael_rendier · · Score: 1

    i'm kinda partial to seeing the errors so that i can fix them...*shrugs*...

    --
    There are three kinds of people in the world. Those that can count, and those that can't.
  11. Is Linux finally catching up to AIX? by Anonymous Coward · · Score: 1

    AIX had this in 1990s.

    1. Re:Is Linux finally catching up to AIX? by bill_mcgonigle · · Score: 2

      AIX had this in 1990s.

      and users too!

      LISP machines had this in the 70's, so there's that.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  12. Re:Sounds great by bill_mcgonigle · · Score: 1

    I love recompiling kernels and updating config files whenever I buy a new mouse.

    Stop building your own proprietary PCIe mouse cards, knucklehead. You can buy a decent USB mouse for $12 now.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  13. Re: Sounds great by Anonymous Coward · · Score: 2, Funny

    But I'm a professional Starcraft player. The lag of USB packet overhead is killing me!

  14. Older than this I think by davidwr · · Score: 2

    I think mainframes had this long before the '90s.

    From the 1970s, Tandem Computers (now part of HP) specialized in high-availability computing. I'm pretty sure they've had the ability to patch their equivalent of a kernel for ages.

    1980s-era electronic/digital telephone switches (the kind at telco switching offices, NOT your run-of-the-mill PBX) had uptimes measured in DECADES. I don't know if these switches had "live 'kernel' update" capability or not but they did have an "half and half" mode with "live fail-over" capability. One way of doing updates was to shut down one "half" gracefully, update it, bring it back up, leave it running for awhile, then repeat with the other "half". From a customer perspective, the switch was never "down".

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    1. Re:Older than this I think by Anonymous Coward · · Score: 0

      The Babbage Engine could accept live updates, but you had to be careful or you'd lose a hand.

    2. Re:Older than this I think by Anonymous Coward · · Score: 1

      The Babbage Engine could accept live updates, but you had to be careful or you'd lose a hand.

      The Antikythera mechanism [https://en.wikipedia.org/wiki/Antikythera_mechanism] could accept live updates but you had to slaughter a few slaves and spill their blood over the gears. Some employers still do this practice with lower level IT staff.

    3. Re:Older than this I think by boogahboogah · · Score: 1

      According to a friend, who was a switchman for AT&T/Bell Atlantic/Verizon for 36 years, the machines in a single office were a dual machine architecture, running off of the primary, and in case of a problem, would switch over to the secondary machine which was already up & running & doing stuff.

      For patching, you moved to the secondary machine, installed the patch on the primary, rebooted the primary & prayed it would work. If not, then you spent lots of time on the phone to geek central to find out what was going on...

       

  15. Obligatory Snark by _bug_ · · Score: 1, Funny

    This is awesome. Now that live patching is part of the kernel I expect it to be implemented in systemd very soon and all my GNU/systemd servers will never need a three finger salute again!

  16. Re: Sounds great by Anonymous Coward · · Score: 1

    You should switch to Zerg so that you can A-move your roaches.

  17. Ironicly, some older OSes are easy to "patch live" by davidwr · · Score: 3, Interesting

    The OSes that ran on 8086-era computers and on very early Macs, as well as most consumer 8-bit OSes could in principle be patched or even completely overwritten without a reboot.

    I vaguely remember an early Mac implementation of Lisp which basically "took over" the machine and gave you a command-line environment (look Ma! No menus!). You "ran" it by running a standard Mac application which basically took over the machine.

    I seem to remember some DOS (if you can call that an OS) programs that worked basically the same way: They loaded themselves into memory, kicked the OS out, then when they quit, they asked you to insert a DOS disk and re-loaded DOS from disk without doing a hardware/BIOS-level reboot (or they knew how to read the hard disk boot tracks and loaded it from there).

    With the advent of chips that provided real privilege levels and OSes that actually took advantage of them, such "takeovers" without the cooperation of the already-loaded OS became impossible by design (but still possible using exploits of course).

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
  18. kexec? by buckfeta2014 · · Score: 1

    I know TFS mentions ksplice, but I thought this was the purpose of kexec as well? So really, it's not anything new.

    --
    Buck Feta. You know what to do.
    1. Re:kexec? by Anonymous Coward · · Score: 0

      Attention, overseer(s) - defective human personality detected. Please repair or eradicate.

  19. You didn't upgrade systemd. by Anonymous Coward · · Score: 0

    You didn't upgrade systemd. You upgraded the systemd package. You won't actually start using the new version of systemd until you do a reboot. And you ought to pray while you do that, even if you aren't religious. Based on my experience with systemd, there is a very good chance that your system will not boot properly, likely due to systemd failing in one or more obscure ways.

    1. Re:You didn't upgrade systemd. by psmears · · Score: 2

      You didn't upgrade systemd. You upgraded the systemd package. You won't actually start using the new version of systemd until you do a reboot.

      No, that's completely wrong. The new version will run if the package upgrade script tells the daemon to re-exec itself. Which (at least in the case of RHEL7), it does.

  20. Reboot required to patch glibc by emil · · Score: 3, Insightful

    Ksplice and it's derivatives won't help you if you need to purge bad glibc code from memory, as we did for the recent "ghost" vulnerability.

    Still, it could potentially be nasty if exploited so we strongly recommend immediate patching and rebooting. Without a reboot, services using the old library will not be restarted,” Moore concluded.

  21. Re:Ironicly, some older OSes are easy to "patch li by hawaiian717 · · Score: 1

    That's how BootX (one of the Mac OS 8/9 Linux boot loaders for PowerPC Macs) worked. Mac OS would start loading, then a dialog would come up and you could select Mac OS or Linux. You could also run the application from Mac OS anytime after the OS was fully booted. In either case, when you selected Linux, it pushed Mac OS out of memory and Linux would start up.

    --
    End of Line.
  22. So the NSA will be live patching our kernels ... by Anonymous Coward · · Score: 1

    .... what could possibly go wrong?

  23. Not "never" by guruevi · · Score: 1

    Although KSplice is nice (and functionality like this has been in Solaris/AIX/... for a really long time), last time I looked at it, it didn't support live-patching everything. You couldn't just bump an entire kernel version (as is possible on Solaris), only patch modules with a very specific patch as long as there are no processes using any part of it.

    --
    Custom electronics and digital signage for your business: www.evcircuits.com
  24. Kernel patch == uptime, Firefox == downtime? by Anonymous Coward · · Score: 0

    I find it odd that the Linux kernel is able to support this, but major distributions (Fedora) are making userspace updates (like Firefox)[1] an "offline" process.

    This seems like quite a reversal from a few years ago.

    Maybe userspace updates will finally catch up to kernel updates and _not_ require downtime?

    [1] http://fedoraproject.org/wiki/Features/OfflineSystemUpdates

  25. Re:Ironicly, some older OSes are easy to "patch li by Anonymous Coward · · Score: 0

    It was DOS itself doing that, but not a complete kick out : just command.com. If an application needed the memory, the DOS would reclaim that used by the command processor and reload it when the application finished.

  26. Re:Sounds great by Anonymous Coward · · Score: 0

    You do know Windows supports live-patching of some parts of the code, right?
    It isn't that dangerous when the alternative is running code that's known to be broken.

  27. Re: Sounds great by Anonymous Coward · · Score: 0

    Not anymore it doesn't. Microsoft deprecated the hotpatch APIs in Win8. They were never used in production before that anyway. Maybe some special enterprise customers.

  28. Great by fisted · · Score: 2

    An ocean of new opportunities for rooted machines...

  29. Re:Sounds great by Anonymous Coward · · Score: 0

    And when it did most of it was in userspace - Heartbleed, ShellShock, GHOST ... to name a few recent ones. Restarting *everything* that depends on openssl or glibc is indeed possible without a reboot but really - reboot is just much easier.

    kSplice and LivePatch do nothing for live userspace patching - overhyped solutions to something that isn't really a big problem.

  30. Ksplice has been around since 2009 by skotfred · · Score: 0

    The only thing remotely "new" about this is that Oracle is now running the show, I've used it on some of my Ubuntu servers since late 2009.

    --
    http://www.giantgeek.com/blog/
  31. Re:Ironicly, some older OSes are easy to "patch li by Myen · · Score: 1

    That sounds more like kexec, where the running kernel is replaced (which also means existing processes are all killed). This newfangled thing is for live patching, where everything (including userland) stays up.

    The DOS part you are talking about works because it isn't doing multitasking; effectively, each app is the kernel as it runs. For later examples of this, any 386 or higher version of Windows (3.11 WFW, 95, ...) did basically the same thing.

  32. Re:Sounds great by Anonymous Coward · · Score: 0

    you aren't fucking serious are you? kernel level escalation of privileges occur at a pretty steady rate throughout the year. If you can leverage a remote exploit into user space you will almost certainly be able to elevate your privileges from that point in all but the most seriously locked down systems.

  33. Recent Linux Conf in Auckland, New Zealand? by lippydude · · Score: 1

    @rgbe: "This, among other things was discussed in the Kernel Report, at the recent Linux Conf in Auckland, New Zealand:

    How was this covered on slashdot, did I miss it?

  34. Re:LIVE GAY NIGGER ASS CUM FELCHING ON GNAA PPV by Anonymous Coward · · Score: 0

    Have to admit, it leaves a good anchor for someone to Hijack the thread and get a high position post to make their opinions on a matter known without having to go through the effort of getting an actual first post.

  35. You did upgrade systemd! by Gunstick · · Score: 1

    it actually updates systemd in ram by doing a reexec
    Something which was possible for other processes.
    To do this with the PID1 is something new.
    And if that is going to work all the time is a mystery.

    You could expect kernel panic on a systemd update.

    --
    Atari rules... ermm... ruled.