Multiple Linux Distributions Affected By Crippling Bug In Systemd (agwa.name)

An anonymous reader writes: System administrator Andrew Ayer has discovered a potentially critical bug in systemd which can bring a vulnerable Linux server to its knees with one command. "After running this command, PID 1 is hung in the pause system call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system." According to the bug report, Debian, Ubuntu, and CentOS are among the distros susceptible to various levels of resource exhaustion. The bug, which has existed for more than two years, does not require root access to exploit.

Comments

I don't hate on systemd but this is really bad

[haruchai]

and been around for 2 years and doesn't require root access??
If this happened on Windows, I & many others would be scornful of it.

Re:I don't hate on systemd but this is really bad

[rossz]

How can you possibly overblow a bug that can bring down a system without root privileges?

Re:I don't hate on systemd but this is really bad

[somenickname]

Exactly this. You could probably paste a working and viable init.c into a Slashdot post and not cause it to emit the "Click to read more" link.

On the other hand, you can do this:

foo [ ~/src ]$ git clone https://github.com/systemd/sys...
foo [ ~/src ]$ cd systemd
foo [ ~/src/systemd ]$ wc -l `find . -name "*.c"` | tail -1
    374209 total

That's a bit more code than a traditional Unix init system...

Re:I don't hate on systemd but this is really bad

[somenickname]

I see where you are coming from and, yes, it's disingenuous for me to imply that all that code is running in PID 1. It's certainly not. But, my point is that systemd is gigantic because it has started to absorb other fundamental parts of the userland. And so those parts are now heavily reliant on PID 1 or a very near descendant. Instead of layers of software being built on more fundamental layers of software, you now have a nasty web of dependencies that will, in time, become unmaintainable.

We grey beards didn't do it how we did it for fun. We did it because once one layer of the system worked, we stopped caring about it and moved to the next layer. Systemd is compressing all the layers into a single, nasty web of interdependent processes that represent a single layer. The complexity of it *will* overwhelm the stability of it. It's just a matter of when.

Re:I don't hate on systemd but this is really bad

[lgw]

That's a bit more code than a traditional Unix init system...

That's a bit more code than an old-school Unix system.

EMACS and systemd are both credible complete operating systems, but EMACS is lighter weight, includes a web browser, and can emit textual log files. It's a clear victory for EMACS.

This is what we were talking about.

[Gravis Zero]

All the people that were telling you that this init system called Systemd was overly complex, unaudited and insecure had warned you that this was coming. All the "Troll -1" modding on people that posted such warning here did not prevent the inevitable.

Not convinced? Here's a graph of the number of issues opened/closed since systemd moved to github last year.

Re:First of many

[somenickname]

The kernel is a necessary evil that supports thousands (millions?) of different devices, dozens of architectures, dozens of file systems, etc, etc. It's also the quintessential open source project with a meritocracy based hierarchy that dictates how things get added to the kernel. Systemd is Lennart and his henchmen carving out a fiefdom. Big difference.

And of course the systemd devs throw a tantrum

[Anonymous Coward]

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d
Can't have anyone criticizing any aspect of the holy systemd.
Whole thing boils down to:
"Following security practices in an init system is hard, and you've never done it so leave us alone."
Completely ignoring the fact that the only reason they patched this thing is because he made a big deal out of it.
And on what planet is testing for corner cases like empty strings the domain of fuzz tools?
That seems like a pretty standard test case to me.
I can understand if you don't test for a 1MB string, but empty seems like a no brainer.

OpenRC

[Shane_Optima]

If you're dissatisfied with systemd and you don't need any of its fancier capabilities (which as an end user I'm assuming would be Docker stuff), please consider switching to a non-systemd distro as soon as possible and (if you can afford the time or money) contributing to their development. The more support systemd alternatives can garner, the more likely it is that projects to will resist unnecessary systemd dependencies and it might even be that systemd itself will eventually become more modular and moddable.

I'm not a hater. I cringe every time I see +5 comments claiming that systemd didn't fix anything. Declarative syntax is (at least in principle) a massive win, especially for distro builders. And LXC is amazing stuff, and I certainly cannot fault Red Hat for wanting containers to behave perfectly. Unless something like Genode scores a major coup, containers are definitely the future of secure and robust computing.

But the actual details of systemd's course have been hair-raising. It needs to be more UNIX-like and less draconian in its requirements and less toxic in its effects on the FOSS ecosystem and unfortunately (given Red Hat's behavior over the past decade) it appears that pushing alternatives hard is the only way they can conceivably be convinced to change their ways or reform anything moving forward.

I encourage all of the haters here to try and put your money where your mouth is. Install, use, support and help promote a distro like Devuan or even better: go and find one of the multiple OpenRC distros available. OpenRC can't be the all-in-one automagic solution systemd endeavors to be, but it doesn't hide tons of stuff in huge C binaries and it's addressed most of the common frustrations people have with SysV. Arch Linux has an OpenRC variant (the standard install uses systemd), Gentoo was the distro that started OpenRC years ago, and Alpine linux uses it (which isn't an ideal easy desktop distro, but it's amazing for those wanting a secure minimal distro to build on and last time I checked it does run XFCE and Firefox.) There are probably others.

Re:OpenRC

[Shane_Optima]

As far as I can tell, the problem is mostly one of personality and ideology. Poettering is a rather unpleasant person who is purposefully choosing to do things in an unnecessarily hostile and control-destroying manner, and Red Hat, while obviously having done tremendous things to help Linux over the years, has been in the business of promoting user-hostile asshatery and lock-in philosophies in recent years, something which hasn't been remarked on nearly as much as it should've (possibly because RHAT earned themselves too many OSS brownie points 10+ years ago and are still making useful contributions today.)

None of that excuses the legions of people around who have said that there was no problem with SysV, or that declarative syntax was worthless (tell that to someone who is trying to build a distro), or treated process control or containerization as inherently unworthy things to be concerned with. I do not say that systemd does these things especially well; I merely say that as concerns, they are intrinsically worthy of consideration.

And if you say that no one needs better containerization options or that stateless systems is a worthless concept or that Debian's old init scripts were the pinnacle of perfection, you clearly do not know what you are talking about. Red Hat does have some fiat sway, but if SysV init were easy to stick with we wouldn't have seen all of the major distros adopt systemd in lockstep. It's not completely worthless. Some of the problems it solves are indeed real. I'm not saying it's worth half a million lines of C code to solve those problems, but the problems are fact there.

But if you actually deny that any problems exist, you're basically just an unthinking hater and no one is paying you any attention except a certain subset of systemd haters who didn't need any more convincing.

This actually isn't so dissimilar to Trump and concerns over immigration or terrorism, but that's an analogy for another day.

Re:RTFA, please.

[grcumb]

That is far from a detailed description and more of a list of uninformed rants. Much better to read the informed reply to TFA here: https://medium.com/@davidtstra...

More clueless autonomic defensiveness without any reflection on what the impact of the bug actually is. I especially enjoyed this old chestnut as the author attempts to fisk the original bug report:

These accusations are true for every major production kernel (Windows, Linux, and BSD) and every alternative to systemd (in the sense that they’re almost all written in C and run many of their operations as root).

"SystemD, let me just stop you there. I know the Linux kernel. I've worked with the Linux kernel. You're no Linux kernel."

The incredible hubris of asserting parity with the core of the entire OS, the ignorance that underlies the statement that init was written in C and runs as root, so it's every bit as vulnerable... How the fuck do you even make code run? Do you even teh logic?

The SystemD team is the Microsoft of a new generation. Doubling down on their mistakes; shouting louder when they don't get their way; using every available ratiocination and intellectual contortion to excuse themselves; resorting to any means to make their strategy win, instead of stopping to ask themselves for once, 'Are we following a winning strategy here?'

Thank g*d I quit writing software last year. Dealing with Microsoft's mind-crushing blindness was enough for one lifetime. Now I can just grump about it and walk away.

Not surprising

[MrKaos]

I've made several requests for systemd proponents to supply a use case that SysV initd could not support and haven't received a satisfactory reply to this purely technical question. I was interested in what systemd could offer over initd. I find systemd proponents are overly veherment in their criticisms of initd proponents.

I sense this comes from an inability to address the issues raised and, perhaps a mindset that anyone who has an appreciation for initd's elegant power will simply be bulldozed into irrelevance. I think systemd's criticism of the rc scripts that starts a linux based system is valid criticism however we have to keep in mind that they were devised by Red Hat. It is dealing with rc shell scripts that are the brunt of the justification for systemd.

In that sense the unitd solution is tidy but also reveals the justification to replace initd is not based on a full understanding of its capabilities, or even an understanding of was it is, a process manager. rc scripts are only meant to prepare the system for entries in /etc/inittab, yet everyone tries to get everything done in rc, which serializes the Linux boot process. A parallell boot is completely achievable by using initd properly. I know there is more to it, like events and messaging, I'm just citing one example.

Yet I've never seen a Linux distro that's utilized initd's /etc/inittab file properly. Especially a Red Hat system. They don't use initd properly, the rc scripts are bloated with rewrites of what initd already does, and now we're replacing initd, keh? initd has yet to be utilized fully on modern linux systems.

Criticisms of sco the company aside: sco *as a distribution of unix* had an interesting adjunct feature to initd, the 'enable' and 'disable' command that managed entries in /etc/inittab, where you would configure the characteristics of the system you were running. Franky I think this is functionality is essentially

sed -i -e '/someProcess/ s/:on:/:off:/' /etc.inittab ; kill -1 1

I think initd would make a lot more sense to more people if this functionality had been available in Linux from the beginning. It is true that initd is beguiling in terms of it's simplicity wrt its power, but it is also very worthwhile. It is supposed to be small as that is where the skill is expressed.

initd is where you design the characteristics of the system, it is not an event manager and all the other things systemd is supposed to be. Something that does all the functionality systemd has, belongs as an inittab enty, not as the first process the kernel runs.

The point of a bug like this is not that it is a big deal itself, the big deal is the failure mode systemd has been revealed to have due to its complexity. This the type of concern I have about systemd, what else can trigger such a failure mode. I have seen initd in a variety of failure modes and not once has it ever consumed all system resources and disconnected running processes.

Now we've seen systemd do something that initd can't.