Live Patching Now Available For Linux

New submitter cyranix writes "You may never have to reboot your Linux machine ever again, even for kernel patching," and excerpts from the long (and nicely human-readable) description of newly merged kernel code that does what Ksplice has for quite a while (namely, offer live updating for Linux systems, no downtime required), but without Oracle's control. It provides a basic infrastructure for function "live patching" (i.e. code redirection), including API for kernel modules containing the actual patches, and API/ABI for userspace to be able to operate on the patches (look up what patches are applied, enable/disable them, etc). It's relatively simple and minimalistic, as it's making use of existing kernel infrastructure (namely ftrace) as much as possible. It's also self-contained, in a sense that it doesn't hook itself in any other kernel subsystem (it doesn't even touch any other code). It's now implemented for x86 only as a reference architecture, but support for powerpc, s390 and arm is already in the works (adding arch-specific support basically boils down to teaching ftrace about regs-saving).

Comments

Discussed in the Kernel Report

[rgbe]

This, among other things was discussed in the Kernel Report, at the recent Linux Conf in Auckland, New Zealand:
  https://www.youtube.com/watch?...

Re:Discussed in the Kernel Report

[buchner.johannes]

at 39:39

No more downtime

[ArcadeMan]

Which means you can keep it up forever!

(PHRASING!)

Re:No more downtime

[fulldecent]

Holy shitsnacks. There are more Archer seasons? I need to step up my piracy.

We are talking about Archer, right?

Re:No more downtime

[aliquis]

Meanwhile Windows still seem to ask you to reboot about once a week (every second week?) to install the updates ..

Re:No more downtime

[gavron]

Ok, so here's the simple answer. Note: I'm generalizing a lot to make this simple.

All functions have a known entry point which you can think of a name that you can call like
print("hello world"); -- calls "print" so it knows where "print" is.

Somewhere in the memory was loaded the function print(). There's also a symbol which allows everyone who wants to call print() to know where it is.

The livepatch loads a new function into memory. Let's call it print2(). It then goes over and makes the symbol that used to let everyone know where print() is point to print2(). Anyone that comes after this patch will still think they are calling print() but in fact will be calling print().

The stop_machine() is part of how ksplice (the proprietary-vendor method does it). That is not part of kernel live patch (klp).

What klp does is ensure that a process is in a "good point" to be messed with, and then changes its pointer to e.g. print().

That allows no changes to affect the process until that pointer to print() is changed at which point any subsequent call to print() will run print() instead.

Ehud
P.S. I have some code from the early 1990s where we used to do this on VMS/OpenVMS. We literally patched the running kernel (much as is done here) and allowed a system to run for years with newer kernel code.

Reboot for Systemd

[Anonymous Coward]

Yup. Exactly.

But then I guess the quest for epic uptime is bogus, right? Who the heck would want their system running 24/7 all the time?

*waits for Systemd flamewar to break out*

Time to update the ol' t-shirt...

[DigitAl56K]

... to a more extreme version:

"I don't always test my code, but when I do it's via live patching the kernel on production"

Doesn't hook itself in any other kernel subsystem?

[Anonymous Coward]

Is this the anti-systemd?

Re:Sounds great

[bill_mcgonigle]

I can't possibly see this ever causing a problem with this because linux is very secure and there is absolutely no way for a malicious user to get elevated access on anything that runs linux. This includes android phones, which are totally invulnerable to hacking.

You should suggest to Linus that he make kernel features configurable so people can do different builds for different targets. Put it in a dot-config file or something. Maybe in a future release...

Unintended consequences?

[marciot]

Maybe I’m old school, but this sort of bothers me. One of the nice things about rebooting is that it clears out old crud and gives you a reassurance that the system can bring itself up by its bootstraps. I can imagine live patching giving rise to a scenario where you have a machine that hasn’t been rebooted for years and when a power glitch finally brings it down, you find that what is on disk is different than what was in RAM and your kernel is corrupt or not bootable.

I think live patching would make sense if we had non-volatile system RAM (i.e. universal memory), but until then, it seems like rebooting is a pretty good sanity check that things are alright.

Re: Sounds great

[Anonymous Coward]

But I'm a professional Starcraft player. The lag of USB packet overhead is killing me!

Older than this I think

[davidwr]

I think mainframes had this long before the '90s.

From the 1970s, Tandem Computers (now part of HP) specialized in high-availability computing. I'm pretty sure they've had the ability to patch their equivalent of a kernel for ages.

1980s-era electronic/digital telephone switches (the kind at telco switching offices, NOT your run-of-the-mill PBX) had uptimes measured in DECADES. I don't know if these switches had "live 'kernel' update" capability or not but they did have an "half and half" mode with "live fail-over" capability. One way of doing updates was to shut down one "half" gracefully, update it, bring it back up, leave it running for awhile, then repeat with the other "half". From a customer perspective, the switch was never "down".

Re:Is Linux finally catching up to AIX?

[bill_mcgonigle]

AIX had this in 1990s.

and users too!

LISP machines had this in the 70's, so there's that.

Ironicly, some older OSes are easy to "patch live"

[davidwr]

The OSes that ran on 8086-era computers and on very early Macs, as well as most consumer 8-bit OSes could in principle be patched or even completely overwritten without a reboot.

I vaguely remember an early Mac implementation of Lisp which basically "took over" the machine and gave you a command-line environment (look Ma! No menus!). You "ran" it by running a standard Mac application which basically took over the machine.

I seem to remember some DOS (if you can call that an OS) programs that worked basically the same way: They loaded themselves into memory, kicked the OS out, then when they quit, they asked you to insert a DOS disk and re-loaded DOS from disk without doing a hardware/BIOS-level reboot (or they knew how to read the hard disk boot tracks and loaded it from there).

With the advent of chips that provided real privilege levels and OSes that actually took advantage of them, such "takeovers" without the cooperation of the already-loaded OS became impossible by design (but still possible using exploits of course).

Reboot required to patch glibc

[emil]

Ksplice and it's derivatives won't help you if you need to purge bad glibc code from memory, as we did for the recent "ghost" vulnerability.

Still, it could potentially be nasty if exploited so we strongly recommend immediate patching and rebooting. Without a reboot, services using the old library will not be restarted,” Moore concluded.

Re:You didn't upgrade systemd.

[psmears]

You didn't upgrade systemd. You upgraded the systemd package. You won't actually start using the new version of systemd until you do a reboot.

No, that's completely wrong. The new version will run if the package upgrade script tells the daemon to re-exec itself. Which (at least in the case of RHEL7), it does.

Great

[fisted]

An ocean of new opportunities for rooted machines...