Multiple Linux Distributions Affected By Crippling Bug In Systemd (agwa.name)

An anonymous reader writes: System administrator Andrew Ayer has discovered a potentially critical bug in systemd which can bring a vulnerable Linux server to its knees with one command. "After running this command, PID 1 is hung in the pause system call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system." According to the bug report, Debian, Ubuntu, and CentOS are among the distros susceptible to various levels of resource exhaustion. The bug, which has existed for more than two years, does not require root access to exploit.

Comments

I don't hate on systemd but this is really bad

[haruchai]

and been around for 2 years and doesn't require root access??
If this happened on Windows, I & many others would be scornful of it.

Re:I don't hate on systemd but this is really bad

[rossz]

How can you possibly overblow a bug that can bring down a system without root privileges?

Re:I don't hate on systemd but this is really bad

[somenickname]

Exactly this. You could probably paste a working and viable init.c into a Slashdot post and not cause it to emit the "Click to read more" link.

On the other hand, you can do this:

foo [ ~/src ]$ git clone https://github.com/systemd/sys...
foo [ ~/src ]$ cd systemd
foo [ ~/src/systemd ]$ wc -l `find . -name "*.c"` | tail -1
    374209 total

That's a bit more code than a traditional Unix init system...

Re:I don't hate on systemd but this is really bad

[somenickname]

No, it's you who fails to see that what somenickname showed was not the number of lines of code in the systemd init but the number of lines of all the applications, deamons etc that is stored in the systemd source repository.

And that should be a gigantic red flag to anyone. Why does the init system need all that stuff?

Just like BSD stores all the code for their kernel and user space applications in a single repository.

That single repository represents hundreds or thousands of different projects. The "git clone" I did represents one single project.

It's just a guilt by git association.

No, it's guilt by assimilation. Big difference.

Re:I don't hate on systemd but this is really bad

[Sarten-X]

I'm not too terribly familiar with init's requirements, but isn't a "working and viable init.c" basically something like execl("/sbin/getty", "tty0");? It runs, it provides a login shell to the user... what more do you want?

Oh, you want preconfigured settings? Real Linux Users set that stuff by hand when they log in, but fine. We'll add that to the init daemon.

Multiple terminals, too? Fine, a bit of magic with getty, and you're good.

Oh, you want it to start vital services like networking? You could do that with ifconfig, but whatever... Sure, let's give it some network support.

Wait, and now you want to be able to configure all that without compiling? This is getting absurd, but if you insist, we can make a hundred little hundred-line shell scripts, and just run them.

...in different ways? You're really going to ask to run your shiny new server with completely different sets of services at different times, and you're just so spoiled that you can just reconfigure it as needed? Why the hell did we make the damned thing so configurable anyway, if you're not going to use it? Fine... Since you're asking so nicely, we'll throw in a bunch of folders... just link the scripts you want, and names the links so everything's in the right order.

Another request? What do you want from me now? You can't even keep a network operating reliably, and you want your init daemon to do the work for you? Alright, but this is the last straw. Now your configuration scripts can run in parallel, have dependencies, and they will run other scripts to see if they can run your services yet.

...

One of these steps apparently crosses a line, though, and causes enough discomfort that folks derail discussions.

Re:I don't hate on systemd but this is really bad

[somenickname]

I see where you are coming from and, yes, it's disingenuous for me to imply that all that code is running in PID 1. It's certainly not. But, my point is that systemd is gigantic because it has started to absorb other fundamental parts of the userland. And so those parts are now heavily reliant on PID 1 or a very near descendant. Instead of layers of software being built on more fundamental layers of software, you now have a nasty web of dependencies that will, in time, become unmaintainable.

We grey beards didn't do it how we did it for fun. We did it because once one layer of the system worked, we stopped caring about it and moved to the next layer. Systemd is compressing all the layers into a single, nasty web of interdependent processes that represent a single layer. The complexity of it *will* overwhelm the stability of it. It's just a matter of when.

Re:I don't hate on systemd but this is really bad

[dbIII]

Why does the init system need all that stuff?

Because it's not an init system anymore, it's Lennart trying to put his name on everything between the application the user runs and the kernel.

Re:I don't hate on systemd but this is really bad

[lgw]

That's a bit more code than a traditional Unix init system...

That's a bit more code than an old-school Unix system.

EMACS and systemd are both credible complete operating systems, but EMACS is lighter weight, includes a web browser, and can emit textual log files. It's a clear victory for EMACS.

First of many

[somenickname]

Putting this level of complexity at such a low level of the system is going to cause show stopping bugs. And, with every new release, more complexity is added.

Re:First of many

[somenickname]

The kernel is a necessary evil that supports thousands (millions?) of different devices, dozens of architectures, dozens of file systems, etc, etc. It's also the quintessential open source project with a meritocracy based hierarchy that dictates how things get added to the kernel. Systemd is Lennart and his henchmen carving out a fiefdom. Big difference.

Re:First of many

[Z00L00K]

The difference compared to systemd is that 'init' is small and have little overhead.

Systemd is trying to fix stuff that isn't broken.

WONTFIX

[Anonymous Coward]

This is not a bug, it's a feature. This is the systemd way and you've all being doing it wrong!

Regards,

The Systemd Developers

Diversity

[K. S. Kyosuke]

Fortunately, we have so many different Linux distributions that this is not a problem! (...right?)

This is what we were talking about.

[Gravis Zero]

All the people that were telling you that this init system called Systemd was overly complex, unaudited and insecure had warned you that this was coming. All the "Troll -1" modding on people that posted such warning here did not prevent the inevitable.

Not convinced? Here's a graph of the number of issues opened/closed since systemd moved to github last year.

And of course the systemd devs throw a tantrum

[Anonymous Coward]

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d
Can't have anyone criticizing any aspect of the holy systemd.
Whole thing boils down to:
"Following security practices in an init system is hard, and you've never done it so leave us alone."
Completely ignoring the fact that the only reason they patched this thing is because he made a big deal out of it.
And on what planet is testing for corner cases like empty strings the domain of fuzz tools?
That seems like a pretty standard test case to me.
I can understand if you don't test for a 1MB string, but empty seems like a no brainer.

OpenRC

[Shane_Optima]

If you're dissatisfied with systemd and you don't need any of its fancier capabilities (which as an end user I'm assuming would be Docker stuff), please consider switching to a non-systemd distro as soon as possible and (if you can afford the time or money) contributing to their development. The more support systemd alternatives can garner, the more likely it is that projects to will resist unnecessary systemd dependencies and it might even be that systemd itself will eventually become more modular and moddable.

I'm not a hater. I cringe every time I see +5 comments claiming that systemd didn't fix anything. Declarative syntax is (at least in principle) a massive win, especially for distro builders. And LXC is amazing stuff, and I certainly cannot fault Red Hat for wanting containers to behave perfectly. Unless something like Genode scores a major coup, containers are definitely the future of secure and robust computing.

But the actual details of systemd's course have been hair-raising. It needs to be more UNIX-like and less draconian in its requirements and less toxic in its effects on the FOSS ecosystem and unfortunately (given Red Hat's behavior over the past decade) it appears that pushing alternatives hard is the only way they can conceivably be convinced to change their ways or reform anything moving forward.

I encourage all of the haters here to try and put your money where your mouth is. Install, use, support and help promote a distro like Devuan or even better: go and find one of the multiple OpenRC distros available. OpenRC can't be the all-in-one automagic solution systemd endeavors to be, but it doesn't hide tons of stuff in huge C binaries and it's addressed most of the common frustrations people have with SysV. Arch Linux has an OpenRC variant (the standard install uses systemd), Gentoo was the distro that started OpenRC years ago, and Alpine linux uses it (which isn't an ideal easy desktop distro, but it's amazing for those wanting a secure minimal distro to build on and last time I checked it does run XFCE and Firefox.) There are probably others.

Re:OpenRC

[Shane_Optima]

As far as I can tell, the problem is mostly one of personality and ideology. Poettering is a rather unpleasant person who is purposefully choosing to do things in an unnecessarily hostile and control-destroying manner, and Red Hat, while obviously having done tremendous things to help Linux over the years, has been in the business of promoting user-hostile asshatery and lock-in philosophies in recent years, something which hasn't been remarked on nearly as much as it should've (possibly because RHAT earned themselves too many OSS brownie points 10+ years ago and are still making useful contributions today.)

None of that excuses the legions of people around who have said that there was no problem with SysV, or that declarative syntax was worthless (tell that to someone who is trying to build a distro), or treated process control or containerization as inherently unworthy things to be concerned with. I do not say that systemd does these things especially well; I merely say that as concerns, they are intrinsically worthy of consideration.

And if you say that no one needs better containerization options or that stateless systems is a worthless concept or that Debian's old init scripts were the pinnacle of perfection, you clearly do not know what you are talking about. Red Hat does have some fiat sway, but if SysV init were easy to stick with we wouldn't have seen all of the major distros adopt systemd in lockstep. It's not completely worthless. Some of the problems it solves are indeed real. I'm not saying it's worth half a million lines of C code to solve those problems, but the problems are fact there.

But if you actually deny that any problems exist, you're basically just an unthinking hater and no one is paying you any attention except a certain subset of systemd haters who didn't need any more convincing.

This actually isn't so dissimilar to Trump and concerns over immigration or terrorism, but that's an analogy for another day.

Re:Systemd was SUCH A GREAT IDEA

[a_n_d_e_r_s]

No its a technical struggle.

The UNIX philosofy is to make many smaller programs that does one thing and does it well. From a bug point of view that been godsend; smaller programs are easier to debug and test.

Large complex programs will always be a problem. Like webb browsers and systemd. The more complex a program becomes and the more it does the more complex is it to write secure code for all situations.

Re:RTFA, please.

[grcumb]

That is far from a detailed description and more of a list of uninformed rants. Much better to read the informed reply to TFA here: https://medium.com/@davidtstra...

More clueless autonomic defensiveness without any reflection on what the impact of the bug actually is. I especially enjoyed this old chestnut as the author attempts to fisk the original bug report:

These accusations are true for every major production kernel (Windows, Linux, and BSD) and every alternative to systemd (in the sense that they’re almost all written in C and run many of their operations as root).

"SystemD, let me just stop you there. I know the Linux kernel. I've worked with the Linux kernel. You're no Linux kernel."

The incredible hubris of asserting parity with the core of the entire OS, the ignorance that underlies the statement that init was written in C and runs as root, so it's every bit as vulnerable... How the fuck do you even make code run? Do you even teh logic?

The SystemD team is the Microsoft of a new generation. Doubling down on their mistakes; shouting louder when they don't get their way; using every available ratiocination and intellectual contortion to excuse themselves; resorting to any means to make their strategy win, instead of stopping to ask themselves for once, 'Are we following a winning strategy here?'

Thank g*d I quit writing software last year. Dealing with Microsoft's mind-crushing blindness was enough for one lifetime. Now I can just grump about it and walk away.

Devuan: a fork of Debian without systemd.

[Artemis3]

In the meantime you may avoid using systemd as init in Debian by installing sysvinit-core or in Ubuntu by installing upstart-sysv in your transition to a systemd-less distro such as Devuan.

If you are using Debian Jessie, you can switch to Devuan by simply changing repositories. Its still in beta so don't do it on production servers yet. But do plan your migration, before this gets out of hand.

Re:RTFA, please.

[dbIII]

It's another straw on a very overloaded camel's back - so this "small bug" just gets added to the list.
I've still got pretty well everything on linux still on RHEL6/CentOS6 because the vendors of an application used in the place don't trust systemd yet. The few workstations I have on something newer are as unstable as MS Windows machines used to be - one wouldn't even boot when systemd hung on a wireless mouse dongle - so much for parallel init! It should have failed then moved on instead of blocking every single time it hits that USB device, and the rumoured parallel init should have enabled it to be doing other stuff instead of completely blocking. That's just one, every few weeks there's some different fuckup of the sort we used to laugh at MS Windows.

Not surprising

[MrKaos]

I've made several requests for systemd proponents to supply a use case that SysV initd could not support and haven't received a satisfactory reply to this purely technical question. I was interested in what systemd could offer over initd. I find systemd proponents are overly veherment in their criticisms of initd proponents.

I sense this comes from an inability to address the issues raised and, perhaps a mindset that anyone who has an appreciation for initd's elegant power will simply be bulldozed into irrelevance. I think systemd's criticism of the rc scripts that starts a linux based system is valid criticism however we have to keep in mind that they were devised by Red Hat. It is dealing with rc shell scripts that are the brunt of the justification for systemd.

In that sense the unitd solution is tidy but also reveals the justification to replace initd is not based on a full understanding of its capabilities, or even an understanding of was it is, a process manager. rc scripts are only meant to prepare the system for entries in /etc/inittab, yet everyone tries to get everything done in rc, which serializes the Linux boot process. A parallell boot is completely achievable by using initd properly. I know there is more to it, like events and messaging, I'm just citing one example.

Yet I've never seen a Linux distro that's utilized initd's /etc/inittab file properly. Especially a Red Hat system. They don't use initd properly, the rc scripts are bloated with rewrites of what initd already does, and now we're replacing initd, keh? initd has yet to be utilized fully on modern linux systems.

Criticisms of sco the company aside: sco *as a distribution of unix* had an interesting adjunct feature to initd, the 'enable' and 'disable' command that managed entries in /etc/inittab, where you would configure the characteristics of the system you were running. Franky I think this is functionality is essentially

sed -i -e '/someProcess/ s/:on:/:off:/' /etc.inittab ; kill -1 1

I think initd would make a lot more sense to more people if this functionality had been available in Linux from the beginning. It is true that initd is beguiling in terms of it's simplicity wrt its power, but it is also very worthwhile. It is supposed to be small as that is where the skill is expressed.

initd is where you design the characteristics of the system, it is not an event manager and all the other things systemd is supposed to be. Something that does all the functionality systemd has, belongs as an inittab enty, not as the first process the kernel runs.

The point of a bug like this is not that it is a big deal itself, the big deal is the failure mode systemd has been revealed to have due to its complexity. This the type of concern I have about systemd, what else can trigger such a failure mode. I have seen initd in a variety of failure modes and not once has it ever consumed all system resources and disconnected running processes.

Now we've seen systemd do something that initd can't.

Re:Same old playbook

[Etcetera]

How long are systemd proponents going to evade accountability to crying about detractors, greybeards and positoning opponents as anti-change.

Any criticism of Systemd and out come a hoarde of Redhat supporters and astroturfers to change the focus swiftly from the technical to the political

It's fine to blame RedHat, Inc, and the *current* Fedora Community on top of it, but don't blame RedHat supporters.... Many of us using RHEL and its derivatives (CentOS/SL) think this thing is horrible and would have stepped in earlier if we'd been paying closer attention to Fedora-devel five or six years ago. We kind of just assumed that the adults who'd been in charge there were still making decent architectural decisions while we went on with our lives.

Unfortunately SystemD isn't the only one

[Casandro]

The whole "FreeDesktop" Movement seems to be about making Linux more and more incomprehensible.

My theory for why this is is like this:
There are lots of people now growing up when Windows kinda worked (since about 2000). At the same time, involvement in "Open Source" software is seen as a good career move. So they churn out some shitty badly designed code as potential recruiters cannot tell good from bad code. Also they take part in design processes without the experience necessary for this. The result are overcomplex buggy solutions which suck in manpower to maintain them.

Take a look at the *BSD people. The team maintaining OpenBSD is probably smaller than the SystemD team, yet they manage to maintain a whole operating system.