Multiple Linux Distributions Affected By Crippling Bug In Systemd (agwa.name)

An anonymous reader writes: System administrator Andrew Ayer has discovered a potentially critical bug in systemd which can bring a vulnerable Linux server to its knees with one command. "After running this command, PID 1 is hung in the pause system call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system." According to the bug report, Debian, Ubuntu, and CentOS are among the distros susceptible to various levels of resource exhaustion. The bug, which has existed for more than two years, does not require root access to exploit.

Comments

I don't hate on systemd but this is really bad

[haruchai]

and been around for 2 years and doesn't require root access??
If this happened on Windows, I & many others would be scornful of it.

Re:I don't hate on systemd but this is really bad

[bcarson]

There are plenty of people who don't need anymore reasons to hate on systemd. This won't get a pass just because it's Linux.

Re:I don't hate on systemd but this is really bad

[rossz]

How can you possibly overblow a bug that can bring down a system without root privileges?

Re:I don't hate on systemd but this is really bad

[Anonymous Coward]

You know you are in deep dodo when people compare the bloody init to the kernel...

Re: I don't hate on systemd but this is really bad

[Anonymous Coward]

It does not bring down the system, some services are delayed for 30 seconds (which is the timeout before daemons supporting systemd goes on without systemd).

Re:I don't hate on systemd but this is really bad

[somenickname]

Exactly this. You could probably paste a working and viable init.c into a Slashdot post and not cause it to emit the "Click to read more" link.

On the other hand, you can do this:

foo [ ~/src ]$ git clone https://github.com/systemd/sys...
foo [ ~/src ]$ cd systemd
foo [ ~/src/systemd ]$ wc -l `find . -name "*.c"` | tail -1
    374209 total

That's a bit more code than a traditional Unix init system...

Re:I don't hate on systemd but this is really bad

[F.Ultra]

And the reason they cannot is because apparently this bug exists in debug code (it's a ASSERT that is triggered) so only distributions that compiled with -DDEBUG are affected. Also the affected distributions was patched three days ago.

Re:I don't hate on systemd but this is really bad

[somenickname]

No, it's you who fails to see that what somenickname showed was not the number of lines of code in the systemd init but the number of lines of all the applications, deamons etc that is stored in the systemd source repository.

And that should be a gigantic red flag to anyone. Why does the init system need all that stuff?

Just like BSD stores all the code for their kernel and user space applications in a single repository.

That single repository represents hundreds or thousands of different projects. The "git clone" I did represents one single project.

It's just a guilt by git association.

No, it's guilt by assimilation. Big difference.

Re:I don't hate on systemd but this is really bad

[Sarten-X]

I'm not too terribly familiar with init's requirements, but isn't a "working and viable init.c" basically something like execl("/sbin/getty", "tty0");? It runs, it provides a login shell to the user... what more do you want?

Oh, you want preconfigured settings? Real Linux Users set that stuff by hand when they log in, but fine. We'll add that to the init daemon.

Multiple terminals, too? Fine, a bit of magic with getty, and you're good.

Oh, you want it to start vital services like networking? You could do that with ifconfig, but whatever... Sure, let's give it some network support.

Wait, and now you want to be able to configure all that without compiling? This is getting absurd, but if you insist, we can make a hundred little hundred-line shell scripts, and just run them.

...in different ways? You're really going to ask to run your shiny new server with completely different sets of services at different times, and you're just so spoiled that you can just reconfigure it as needed? Why the hell did we make the damned thing so configurable anyway, if you're not going to use it? Fine... Since you're asking so nicely, we'll throw in a bunch of folders... just link the scripts you want, and names the links so everything's in the right order.

Another request? What do you want from me now? You can't even keep a network operating reliably, and you want your init daemon to do the work for you? Alright, but this is the last straw. Now your configuration scripts can run in parallel, have dependencies, and they will run other scripts to see if they can run your services yet.

...

One of these steps apparently crosses a line, though, and causes enough discomfort that folks derail discussions.

Re:I don't hate on systemd but this is really bad

[Daemonik]

Please. If nobody reviewed the code then it's going to linger around until someone exploits it. Just like very friggen other piece of software ever written.

Re:I don't hate on systemd but this is really bad

[somenickname]

I see where you are coming from and, yes, it's disingenuous for me to imply that all that code is running in PID 1. It's certainly not. But, my point is that systemd is gigantic because it has started to absorb other fundamental parts of the userland. And so those parts are now heavily reliant on PID 1 or a very near descendant. Instead of layers of software being built on more fundamental layers of software, you now have a nasty web of dependencies that will, in time, become unmaintainable.

We grey beards didn't do it how we did it for fun. We did it because once one layer of the system worked, we stopped caring about it and moved to the next layer. Systemd is compressing all the layers into a single, nasty web of interdependent processes that represent a single layer. The complexity of it *will* overwhelm the stability of it. It's just a matter of when.

Re:I don't hate on systemd but this is really bad

[myowntrueself]

At least with Debian, can't you choose your init system at install time? Or is that no longer an option?

I choose the distribution to meet my needs. I wouldn't allow the init system to dictate which distro I use.

Its moderately hard to choose the init at install time and requires some changes to the installers boot command.

Its easy enough to strip out systemd after an install and trivially easy to upgrade from Wheezy to Jessie without systemd.

Re:I don't hate on systemd but this is really bad

[dbIII]

Why does the init system need all that stuff?

Because it's not an init system anymore, it's Lennart trying to put his name on everything between the application the user runs and the kernel.

Re:I don't hate on systemd but this is really bad

[lgw]

That's a bit more code than a traditional Unix init system...

That's a bit more code than an old-school Unix system.

EMACS and systemd are both credible complete operating systems, but EMACS is lighter weight, includes a web browser, and can emit textual log files. It's a clear victory for EMACS.

Re:I don't hate on systemd but this is really bad

[konohitowa]

You forgot to sign your comment with "MCSE".

Re:I don't hate on systemd but this is really bad

[Hognoxious]

The inid system does not need any of that stuff, the systemd project aims to present a unified set of administration tools

And that's the problem, right there.

Re:I don't hate on systemd but this is really bad

[Hognoxious]

It's better to be able to choose whichever tools you like. I'd take good over unified any day.

Re:I don't hate on systemd but this is really bad

[drinkypoo]

I'm not too terribly familiar with init's requirements, but isn't a "working and viable init.c" basically something like execl("/sbin/getty", "tty0");? It runs, it provides a login shell to the user... what more do you want?

init starts and stops processes to attain different run levels. That's it. Period, the end. One small tool, which does one thing correctly.

Multiple terminals, too? Fine, a bit of magic with getty, and you're good.

There is nothing whatsoever magical about getty. It is extremely straightforward. It is one small tool, which does one thing correctly.

Oh, you want it to start vital services like networking? You could do that with ifconfig, but whatever... Sure, let's give it some network support.

There is no reason to use anything more (or less!) than a script for configuring networking. Scripting is a central feature of Unix.

Wait, and now you want to be able to configure all that without compiling? This is getting absurd, but if you insist, we can make a hundred little hundred-line shell scripts, and just run them.

My Debian system has 57 init scripts. If that kind of complexity scares you, perhaps computers are not for you.

You're really going to ask to run your shiny new server with completely different sets of services at different times, and you're just so spoiled that you can just reconfigure it as needed? Why the hell did we make the damned thing so configurable anyway, if you're not going to use it? Fine... Since you're asking so nicely, we'll throw in a bunch of folders... just link the scripts you want, and names the links so everything's in the right order.

Yep. Using central Unix features like linking is the right way to do this. It utilizes the database-like aspects of the hierarchical filesystem, a central Unix feature that today you apparently take for granted.

Another request? What do you want from me now? You can't even keep a network operating reliably, and you want your init daemon to do the work for you?

In fact, I can keep a network operating reliably... and much easier without systemd making it more vulnerable because it is developed by morons. I would like to say amateurs, but these are professional incompetents.

One of these steps apparently crosses a line, though, and causes enough discomfort that folks derail discussions.

Everything systemd does crosses a line, because it does everything wrong.

Re:I don't hate on systemd but this is really bad

[jeremyp]

False

Re:I don't hate on systemd but this is really bad

[skids]

This doesn't tell me anything I didn't know, and has one unsubstantiated claim that normal IPC is "inefficient and quite unreliable".

Pointers to a case study where someone who actually knows how to use POSIX IPC runs the numbers would be appreciated.

Re:I don't hate on systemd but this is really bad

[Sarten-X]

init starts and stops processes to attain different run levels. That's it. Period, the end. One small tool, which does one thing correctly.

I'm a bit more familiar with the program than my sarcastic post let on. Originally, init's job was to be the first process, period. It'd launch a shell, which quickly became "launch a login prompt" and then changed to "launch a bunch of services" and finally "launch everything the user wants to run for this runlevel". In short, to answer the original request, running getty is just about all you need.

There is nothing whatsoever magical about getty. It is extremely straightforward. It is one small tool, which does one thing correctly.

Relax; it's literary exaggeration. Once upon a time, as I recall, getty would manage its own tty connections. Now it takes a parameter so you can tell it to connect to a given device. If you want init to support multiple ttys right from the start, you'll want that feature.

There is no reason to use anything more (or less!) than a script for configuring networking. Scripting is a central feature of Unix.

Anything you can do in a script can be done manually. Isn't that the whole point of having non-compiled scripts in the first place?

My Debian system has 57 init scripts. If that kind of complexity scares you, perhaps computers are not for you.

Mine has 96, averaging 120 lines per script. I never said it scares me. I'm merely making fun of the aversion to Systemd's complexity, by highlighting the fact that you already have such complexity, but it's pushed out and duplicated (usually poorly) in your init scripts.

Yep. Using central Unix features like linking is the right way to do this. It utilizes the database-like aspects of the hierarchical filesystem, a central Unix feature that today you apparently take for granted.

...Requiring arcane naming schemes for numbering is the "right way"? Co-opting a folder to be a database is the "right way"? Treating your unsorted filesystem as a sorted list is the "right way"? My intent was to only make light of runlevels' inelegant implementation (and touch on their near-obsolescence, as well), but you've done more than that...

In fact, I can keep a network operating reliably... and much easier without systemd making it more vulnerable because it is developed by morons. I would like to say amateurs, but these are professional incompetents.

Flames aside, I doubt very much that you can keep every network everywhere running reliably. That's partly the point of systemd: to put more adaptability into the service configuration, and let the daemon, rather than the user, determine the correct order for the system services to start.

As a concrete example, I've worked on an embedded system that had to determine its own place in a self-arranging network, set its IP address appropriately, then start system services to manage that network, including determining its own place. The project had a circular dependency that was vital to functionality. The original init scripts to handle that were a mess of conditions, delays, and retries, until the whole thing was scrapped for a single custom-written startup daemon that would undermine and override the whole sysvinit anyway. Systemd could very well have handled the thing, because it has more support for handling service dependencies than just "this one starts before that one".

Everything systemd does crosses a line, because it does everything wrong.

I'm going to need a bit more proof than your authoritative claim, though. Besides being different, what's so wrong with it?

Re:I don't hate on systemd but this is really bad

[somenickname]

My Debian system has 57 init scripts. If that kind of complexity scares you, perhaps computers are not for you.

Mine has 96, averaging 120 lines per script. I never said it scares me. I'm merely making fun of the aversion to Systemd's complexity, by highlighting the fact that you already have such complexity, but it's pushed out and duplicated (usually poorly) in your init scripts.

My grandparent post to this thread says that systemd has 374,000 lines of code in its git repo. You've described 11,500 lines of code, split into many different projects that is mostly boilerplate stuff. And, that's what systemd could actually do: Get rid of the boilerplate crap. As long as you did it in a sensible way, no one would argue with that. But, systemd reached Peak Sensibility like 7 or 8 years ago and has just devolved into... whatever it now is.

Yep. Using central Unix features like linking is the right way to do this. It utilizes the database-like aspects of the hierarchical filesystem, a central Unix feature that today you apparently take for granted.

...Requiring arcane naming schemes for numbering is the "right way"? Co-opting a folder to be a database is the "right way"? Treating your unsorted filesystem as a sorted list is the "right way"? My intent was to only make light of runlevels' inelegant implementation (and touch on their near-obsolescence, as well), but you've done more than that...

Hold on. Yes, that *is* the right way for most problems. You know who wrote my go-to database? Theodore Ts'o. You know who wrote my filesystem? Theodore Ts'o. You know who wrote my security software? The Linux Kernel Team.

Your problem is probably not a special snowflake problem. It has been solved before. Many, many times. And, on a modern computer, the incredible level of shit that you gain buy using the filesystem as your... well... everything, is amazing. Embrace the filesystem. It's your friend. Really.

I'm going to need a bit more proof than your authoritative claim, though. Besides being different, what's so wrong with it?

Please see my previous posts in this article.

Re:I don't hate on systemd but this is really bad

[drinkypoo]

...Requiring arcane naming schemes for numbering is the "right way"?

Arcane? It's called in alpha order. It's not arcane.

Now, arguably, it would have been nicer to use multiple directories instead of the letter prefix, but that would actually have increased complexity and it wasn't necessary. They chose the road of simplicity.

...Requiring arcane naming schemes for numbering is the "right way"?

There is nothing arcane about it. It is fully documented on the system itself. And yes. It is simple and it works and it doesn't necessitate yet another configuration file so yes, it is absolutely the right way. But it is still not arcane. Stop saying that, it's wrong.

Co-opting a folder to be a database is the "right way"?

Holy fucking shit. You actually said that? A filesystem is a database. Get that through your head right now, or you will never make any sense.

Treating your unsorted filesystem as a sorted list is the "right way"?

That is actually the dumbest thing I've seen on Slashdot today.

SELECT * FROM TABLE ORDER BY NAME ASC;

What's the difference between that and doing a readdir and sorting the results alphabetically? Answer, in one case the logic lives in a server and in the other case, the one we're talking about, the logic lives in a combination of the C library and your application and does exactly the same sort of thing. And they both work. Furthermore, the userland tools sort output by default, so it doesn't matter if the filesystem is in inode order or filename order.

My intent was to only make light of runlevels' inelegant implementation (and touch on their near-obsolescence, as well), but you've done more than that...

Yes, I've explained why they are simple and they work and they make use of basic UNIX facilities.

Flames aside, I doubt very much that you can keep every network everywhere running reliably. That's partly the point of systemd: to put more adaptability into the service configuration, and let the daemon, rather than the user, determine the correct order for the system services to start.

That does not belong in a daemon married to init. That belongs in an init-agnostic daemon.

As a concrete example, I've worked on an embedded system that had to determine its own place in a self-arranging network,

It's own "place"? That doesn't mean anything when you haven't explained what kind of place that might be. Speak English. Do you mean its physical location?

set its IP address appropriately, then start system services to manage that network,

Oh noes! Not starting services!

The project had a circular dependency that was vital to functionality.

Without knowing even what aspect this dependency occurred in, I cannot comment intelligently (and you have thus said nothing of value.)

The original init scripts to handle that were a mess of conditions, delays, and retries, until the whole thing was scrapped for a single custom-written startup daemon that would undermine and override the whole sysvinit anyway.

So you're shitty at writing init scripts. Or you tried to do something with init scripts that would have better been done supplementing your initrd. Or some other problem, you haven't given enough information to be useful.

Systemd could very well have handled the thing, because it has more support for handling service dependencies than just "this one starts before that one".

It's trivial for init scripts to check whether other services are running. In fact, you do it through init. You make initscript with $1="status" return a non-zero result if the process is n

First of many

[somenickname]

Putting this level of complexity at such a low level of the system is going to cause show stopping bugs. And, with every new release, more complexity is added.

Re:First of many

[somenickname]

The kernel is a necessary evil that supports thousands (millions?) of different devices, dozens of architectures, dozens of file systems, etc, etc. It's also the quintessential open source project with a meritocracy based hierarchy that dictates how things get added to the kernel. Systemd is Lennart and his henchmen carving out a fiefdom. Big difference.

Re:First of many

[ChunderDownunder]

It's what's happening above the kernel that troubles people. To paraphrase Greenspun,

Any sufficiently complicated Lennart init system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of GNU Hurd.

Re:First of many

[Z00L00K]

The difference compared to systemd is that 'init' is small and have little overhead.

Systemd is trying to fix stuff that isn't broken.

WONTFIX

[Anonymous Coward]

This is not a bug, it's a feature. This is the systemd way and you've all being doing it wrong!

Regards,

The Systemd Developers

Diversity

[K. S. Kyosuke]

Fortunately, we have so many different Linux distributions that this is not a problem! (...right?)

Re:Diversity

[Kazymyr]

My gentoo boxes shrugged for a half milisecond, then continued chugging along.

Comment removed

[account_deleted]

Comment removed based on user account deletion

This is what we were talking about.

[Gravis Zero]

All the people that were telling you that this init system called Systemd was overly complex, unaudited and insecure had warned you that this was coming. All the "Troll -1" modding on people that posted such warning here did not prevent the inevitable.

Not convinced? Here's a graph of the number of issues opened/closed since systemd moved to github last year.

Re:This is what we were talking about.

[nnull]

I don't even need to worry about this bug, systemd already hangs my system when rebooting because it can't properly unmount or mount my network drives.

"Activation of org.freedesktop.systemd1 timed out"

[adnonsense]

Not sure if it's directly related to this, but since my desktop OS (openSUSE) has adopted systemd I've had a number of amusing incidents like this:

Failed to start reboot.target: Activation of org.freedesktop.systemd1 timed out
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.

requiring a powercycle. Doesn't endear systemd to me in the least.

Bet Devuan/Slackware are not affected

[Anonymous Coward]

Everyone who mocks these distributions for not toeing the Debhat line can all enjoy my "told you so".

RTFA, please.

[Anonymous Coward]

I strongly urge people to RTFA for a detailed description of some of the technical problems with systemd.

It feels surreal that the most senior people in the Linux community, after decades of attempting to put out a credibly secure client and server platform, suddenly almost all decided to switch to this product.

Re:RTFA, please.

[F.Ultra]

That is far from a detailed description and more of a list of uninformed rants. Much better to read the informed reply to TFA here: https://medium.com/@davidtstra...

What does feel surreal is that people now all of a sudden pretend that SysV init where without exploits while going completely berserk when systemd have a non remote exploitable denial of service bug that cannot be used to take over the machine that also where patched three days ago...

Re:RTFA, please.

[grcumb]

That is far from a detailed description and more of a list of uninformed rants. Much better to read the informed reply to TFA here: https://medium.com/@davidtstra...

More clueless autonomic defensiveness without any reflection on what the impact of the bug actually is. I especially enjoyed this old chestnut as the author attempts to fisk the original bug report:

These accusations are true for every major production kernel (Windows, Linux, and BSD) and every alternative to systemd (in the sense that they’re almost all written in C and run many of their operations as root).

"SystemD, let me just stop you there. I know the Linux kernel. I've worked with the Linux kernel. You're no Linux kernel."

The incredible hubris of asserting parity with the core of the entire OS, the ignorance that underlies the statement that init was written in C and runs as root, so it's every bit as vulnerable... How the fuck do you even make code run? Do you even teh logic?

The SystemD team is the Microsoft of a new generation. Doubling down on their mistakes; shouting louder when they don't get their way; using every available ratiocination and intellectual contortion to excuse themselves; resorting to any means to make their strategy win, instead of stopping to ask themselves for once, 'Are we following a winning strategy here?'

Thank g*d I quit writing software last year. Dealing with Microsoft's mind-crushing blindness was enough for one lifetime. Now I can just grump about it and walk away.

Re:RTFA, please.

[dbIII]

It's another straw on a very overloaded camel's back - so this "small bug" just gets added to the list.
I've still got pretty well everything on linux still on RHEL6/CentOS6 because the vendors of an application used in the place don't trust systemd yet. The few workstations I have on something newer are as unstable as MS Windows machines used to be - one wouldn't even boot when systemd hung on a wireless mouse dongle - so much for parallel init! It should have failed then moved on instead of blocking every single time it hits that USB device, and the rumoured parallel init should have enabled it to be doing other stuff instead of completely blocking. That's just one, every few weeks there's some different fuckup of the sort we used to laugh at MS Windows.

Re:RTFA, please.

[rl117]

"Unchanging" does not mean "unmaintained".

The core C code of sysvinit was feature-complete, reviewed, debugged and tested years and years ago. Its original design goals were satisfied, and the project is "done". Software does not need continual churn to mark it as "maintained". The same applies to startpar/insserv and other ancillary bits. If you found a bug in sysvinit, I'd review and test it, and push a commit for it. I no longer maintain the Debian packaging, but I still have upstream commit rights should I need them.

Compare this with systemd. It doesn't have the same clearly-defined scope; it's not possible to say when it will be complete as a result. Software can be complete and finished.

Re:RTFA, please.

[rl117]

I'm well aware of the history and context. It was quite clear that a replacement would be welcome, providing that replacement was an improvement. I was initially quite hopeful when systemd came about; but it's proven to be unsuitable. There's more to software than features alone, and for this low level part of the system, it's critical to the system's functioning for it to be defect free. It has some interesting ideas, but the design and implementation leave a lot to be desired.

sysvinit (or any init) does not need to have a lot of features, so long as you can build more complex features and functionality on top. The very essence of modularity and substitutability. The priority is to be minimal, reliable and bug free. Just look at how many other systems run more advanced stuff on top of sysvinit. It doesn't have to be there in PID1 or provided by the same software package. sysvinit had a very clearly-defined role and within that constrained scope, it provided a working robust solution that worked for multiple decades. The design of sysvinit can be described in a couple of paragraphs of text; you'd need a book to describe systemd, and I doubt even the authors could fully describe it themselves. Overcomplexity and poor design has a cost, and systemd is already an unmaintainable mess.

The very fact that it had a small interface (signals, initctl, inittab) meant that it could very easily be replaced by other systems (and this was done multiple times). The fact that it was easily built upon meant that there were multiple systems built on top of it. It wasn't perfect, and it didn't have every feature everyone wanted. But the point is that it didn't have to while it could be used as a building block by others. systemd is at one extreme (large, complex, tightly-coupled) while s6 is at the other (tiny, simple, loosely-coupled); sysvinit is toward the s6 side of the spectrum; were I writing it from scratch, I'd lean more towards s6 and strip out the more complex bits into separate parts; PID1 doesn't need to deal with inittab for example, nor with runlevels or shutdown.

Same old playbook

[Anonymous Coward]

How long are systemd proponents going to evade accountability to crying about detractors, greybeards and positoning opponents as anti-change.

Any criticism of Systemd and out come a hoarde of Redhat supporters and astroturfers to change the focus swiftly from the technical to the political

Re:Same old playbook

[Etcetera]

How long are systemd proponents going to evade accountability to crying about detractors, greybeards and positoning opponents as anti-change.

Any criticism of Systemd and out come a hoarde of Redhat supporters and astroturfers to change the focus swiftly from the technical to the political

It's fine to blame RedHat, Inc, and the *current* Fedora Community on top of it, but don't blame RedHat supporters.... Many of us using RHEL and its derivatives (CentOS/SL) think this thing is horrible and would have stepped in earlier if we'd been paying closer attention to Fedora-devel five or six years ago. We kind of just assumed that the adults who'd been in charge there were still making decent architectural decisions while we went on with our lives.

And of course the systemd devs throw a tantrum

[Anonymous Coward]

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d
Can't have anyone criticizing any aspect of the holy systemd.
Whole thing boils down to:
"Following security practices in an init system is hard, and you've never done it so leave us alone."
Completely ignoring the fact that the only reason they patched this thing is because he made a big deal out of it.
And on what planet is testing for corner cases like empty strings the domain of fuzz tools?
That seems like a pretty standard test case to me.
I can understand if you don't test for a 1MB string, but empty seems like a no brainer.

Re:And of course the systemd devs throw a tantrum

[Anonymous Coward]

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d
Can't have anyone criticizing any aspect of the holy systemd.
Whole thing boils down to:
"Following security practices in an init system is hard, and you've never done it so leave us alone."
Completely ignoring the fact that the only reason they patched this thing is because he made a big deal out of it.
And on what planet is testing for corner cases like empty strings the domain of fuzz tools?
That seems like a pretty standard test case to me.
I can understand if you don't test for a 1MB string, but empty seems like a no brainer.

For those who don't want to follow OP's link:

The systemd project applies both unit testing and static/dynamic analysis to systemd. We’ve done this for years; I ran the first Coverity scans myself. Testing inputs of empty strings, excessively large data structures, and other invalid permutations is the realm of fuzz testing, which is a recent project even for the Linux kernel. Despite Linux being used for critical systems for decades, fuzz testing only began as side-projects “in beta” in 2007 and more earnestly in 2013. It’s clearly a valuable technique, but implying that comprehensive testing of invalid inputs is “obvious” is misleading about the state of major projects.

WHAT

THE

FUCK?!?!

It's too much to expect systemd to test for invalid inputs from non-privileged user-space?

Are you fucking kidding me?!?!?

Who the fuck is David Strauss? And when is he scheduled to matriculate from kindergarten?

Too much to expect him to test?!?!!?!

Pathetic. Thalidomide-brain pathetic.

Re:And of course the systemd devs throw a tantrum

[somenickname]

Well, at the very least, be glad that these guys are just writing your init system and not your website.

-- Little Bobby Tables

Re:And of course the systemd devs throw a tantrum

[Anonymous Coward]

Oh geez. He spends his entire diatribe picking apart each one of Ayers's points and saying each one thing alone isn't reason enough not to use systemd. No, maybe not. But all of those things PUT TOGETHER add up to plenty of reason not to use systemd.

I like the contradiction here as well:

"implying that comprehensive testing of invalid inputs is 'obvious' is misleading about the state of major projects"

then later,

"It's a stretch to use the label 'parsing' for what is mostly a string comparison against a fixed number of possibilities"

So, which is it?

1) testing for invalid inputs is waaay too hard because there are waaay too many of them and it ought to be done by some third party fuzzer instead of the systemd team so this isn't our fault at all

or

2) there are a small fixed number of possibilities to compare against and someone accidentally forgot to include "" among them

Then he degrades into the whole "hurr durr, this guy didn't even graduate until 2012, he isn't qualified to criticize my software" angle. Strauss himself didn't graduate until 2007 yet he fancies himself some grizzled dinosaur from the K&R era.

Come on, you guys fucked up, it happens to all of us, nobody has ever released totally bug free software. Admit your mistake and move on.

OpenRC

[Shane_Optima]

If you're dissatisfied with systemd and you don't need any of its fancier capabilities (which as an end user I'm assuming would be Docker stuff), please consider switching to a non-systemd distro as soon as possible and (if you can afford the time or money) contributing to their development. The more support systemd alternatives can garner, the more likely it is that projects to will resist unnecessary systemd dependencies and it might even be that systemd itself will eventually become more modular and moddable.

I'm not a hater. I cringe every time I see +5 comments claiming that systemd didn't fix anything. Declarative syntax is (at least in principle) a massive win, especially for distro builders. And LXC is amazing stuff, and I certainly cannot fault Red Hat for wanting containers to behave perfectly. Unless something like Genode scores a major coup, containers are definitely the future of secure and robust computing.

But the actual details of systemd's course have been hair-raising. It needs to be more UNIX-like and less draconian in its requirements and less toxic in its effects on the FOSS ecosystem and unfortunately (given Red Hat's behavior over the past decade) it appears that pushing alternatives hard is the only way they can conceivably be convinced to change their ways or reform anything moving forward.

I encourage all of the haters here to try and put your money where your mouth is. Install, use, support and help promote a distro like Devuan or even better: go and find one of the multiple OpenRC distros available. OpenRC can't be the all-in-one automagic solution systemd endeavors to be, but it doesn't hide tons of stuff in huge C binaries and it's addressed most of the common frustrations people have with SysV. Arch Linux has an OpenRC variant (the standard install uses systemd), Gentoo was the distro that started OpenRC years ago, and Alpine linux uses it (which isn't an ideal easy desktop distro, but it's amazing for those wanting a secure minimal distro to build on and last time I checked it does run XFCE and Firefox.) There are probably others.

Re:OpenRC

[Shane_Optima]

As far as I can tell, the problem is mostly one of personality and ideology. Poettering is a rather unpleasant person who is purposefully choosing to do things in an unnecessarily hostile and control-destroying manner, and Red Hat, while obviously having done tremendous things to help Linux over the years, has been in the business of promoting user-hostile asshatery and lock-in philosophies in recent years, something which hasn't been remarked on nearly as much as it should've (possibly because RHAT earned themselves too many OSS brownie points 10+ years ago and are still making useful contributions today.)

None of that excuses the legions of people around who have said that there was no problem with SysV, or that declarative syntax was worthless (tell that to someone who is trying to build a distro), or treated process control or containerization as inherently unworthy things to be concerned with. I do not say that systemd does these things especially well; I merely say that as concerns, they are intrinsically worthy of consideration.

And if you say that no one needs better containerization options or that stateless systems is a worthless concept or that Debian's old init scripts were the pinnacle of perfection, you clearly do not know what you are talking about. Red Hat does have some fiat sway, but if SysV init were easy to stick with we wouldn't have seen all of the major distros adopt systemd in lockstep. It's not completely worthless. Some of the problems it solves are indeed real. I'm not saying it's worth half a million lines of C code to solve those problems, but the problems are fact there.

But if you actually deny that any problems exist, you're basically just an unthinking hater and no one is paying you any attention except a certain subset of systemd haters who didn't need any more convincing.

This actually isn't so dissimilar to Trump and concerns over immigration or terrorism, but that's an analogy for another day.

Re:OpenRC

[dbIII]

we wouldn't have seen all of the major distros adopt systemd in lockstep

The distros have been in lockstep for a long time on a lot of issues due to RedHat doing a lot of the work. That gnome now depends on systemd on linux is another very major factor. Neither of those matters have anything to do with the quality of systemd. Like the sound situation of a few years ago there is only one option being worked on so we are told to like it or lump it.

Re:OpenRC

[Etcetera]

And if you say that no one needs better containerization options or that stateless systems is a worthless concept or that Debian's old init scripts were the pinnacle of perfection, you clearly do not know what you are talking about. Red Hat does have some fiat sway, but if SysV init were easy to stick with we wouldn't have seen all of the major distros adopt systemd in lockstep. It's not completely worthless. Some of the problems it solves are indeed real.

This is one thing that I always found weird. I was working almost solely in RH and RH-derived distros from 2000 onward. From my perspective, "initscripts" were fine. On RH/Fedora initscript standardization and /etc/init.d/functions calls were basically a solved problem. I had no idea how crappy Debian's were until I did some release work on it and Ubuntu last year. I could understand a lot more the rip-it-all-out motivation if systemd had come from Debianland over to RH, but it's just bizarre in retrospect that it was instigated on Fedora first when by and large RH was where init was the most non-broken.

I always want to travel back in time to the Fedora 14 release era and convince everyone to just follow Debian's lead and replace Bash with Dash instead if they want a boot speed boost.

Re:Systemd was SUCH A GREAT IDEA

[a_n_d_e_r_s]

No its a technical struggle.

The UNIX philosofy is to make many smaller programs that does one thing and does it well. From a bug point of view that been godsend; smaller programs are easier to debug and test.

Large complex programs will always be a problem. Like webb browsers and systemd. The more complex a program becomes and the more it does the more complex is it to write secure code for all situations.

Re:Doctor Doctor Give Me The News

[Applehu Akbar]

Try to explain to foreigners that cleave means to stick tight to or to split apart from, or that sanction is to permit or to forbid something, and they will run screaming.

Dodged a bullet

[93 Escort Wagon]

"Debian, Ubuntu, and CentOS are among the distros susceptible to various levels of resource exhaustion."

Whew! Thank goodness I run Red Hat!

Devuan: a fork of Debian without systemd.

[Artemis3]

In the meantime you may avoid using systemd as init in Debian by installing sysvinit-core or in Ubuntu by installing upstart-sysv in your transition to a systemd-less distro such as Devuan.

If you are using Debian Jessie, you can switch to Devuan by simply changing repositories. Its still in beta so don't do it on production servers yet. But do plan your migration, before this gets out of hand.

The failure of systemd is that ...

[fahrbot-bot]

The developers haven't stopped at what systemd needs to do and have gone on to what they want it to do, favoring the latter over the former.

Re:Doctor Doctor Give Me The News

[lgw]

inflammable = flammable. It's one of those unfortunate english words.

"In-" can mean both "not-" for latin root words, or "overly-" for other words like infamous or ingenious.

Here that's a coincidence, as the root verb is "inflame".

You simply don't know what an English word means until you know its etymology. Hey, at least you don't need to know its Kanji.

Re:Doctor Doctor Give Me The News

[wierd_w]

No, you can find it without that modifier in quite a few sultry harlequin romance novels.

Things like:

"He cleaved to her breasts with an insatiable hunger" and the like.

The phrase "Cleaved to" is ambiguous.

See above, but also see:

"while working in the butcher shop, Jimmy often cleaved to the sounds of classical music."

Does that mean he stuck with classical music nearly exclusively, or given then context, did he chop meat to the playing of classical music?

It was this ambiguity that the GP was discussing AC.

 

Re:Doctor Doctor Give Me The News

[RavenLrD20k]

I've got 3 separate servers that all run different OSes. 1 in-house with direct control running Gentoo with OpenRC. Then there's the two VPS's. One is running CentOs 6.7 with Upstart. Then there's the PoS VPS I have free on Microsoft Azure running Ubuntu something-or-other with SystemD. Nothing critical is on this server...it just serves as a lab environment and data passthrough. The only time I've ever run SystemD on a system I own with physical access was on my primary desktop...which is never permanently online to begin with.

There have been too many points with a systemd system that I don't trust. Nothing to date with the system has personally affected me to say it's as worthless as I think. I just never trusted it because it just felt too much like a Windows Registry clone in how it worked, which in itself screams that it cannot be trusted. This bug seems to prove my intuition correct.

Re:Doctor Doctor Give Me The News

[Z00L00K]

SystemD a black box that have a lot of features that's hard to understand unless you dig through the source code trying to trace down why it doesn't do what you want and why it doesn't tell you anything about what's wrong.

Sometimes

[Anonymous Coward]

when grey-haired conservative fuddy duddies warn of something, you should PAY ATTENTION even if you disagree.

In this case, the "conservative" is certainly not in the political sense, it's in the technical sense. The core philosophy of UNIX was: small dedicated programs doing discrete things (which can be easily developerd, debugged, tested, and yes... replaced/substituted. Many warned that systemd was the polar opposite and would inevitably invite this very sort of issue. The warnings were ignored because they were not consistent with what the cool kids wanted. It was much more cool to create a whole new gluttonous monster, than to do the hard work to fix a bunch of long-standing and not glamorous basic usability issues that might actually help Linux take over the desktop.

In the political sense a similar thing happened with Obamcare, where conservatives kept pointing out that the basic plan did not pass the economic "smell test", and that inevitably the rates would rise and the markets would fall apart because of the poor planning.

In both cases, the hard-charging progressives (in the technical sense for the former and the political sense for the latter) ranted and raved against the cautious conservatives flinging insults about being backwards, stuck in the mud, opposed to progress, etc rather than facing the actual criticisms, considering that thier opponents might have serious and valid concerns, and then addressing those concerns. In both cases, when the inevitable "I told you so" comments arise, the advocates of the changes get angry and complain and propose moving even further in their chose direction, without facing that the now proven problems are real and were real - they want to solve a real problem with politics and name calling.

Incidentally, before some partisan hack rates this "Troll", I'll point out that this is a trait of human nature and applies to the political right and technical conservatives as well. Some right-leaning "fiscal" conservatives love to propose reductions in social spending while ignoring left wingers who suggest some might be harmed, instead of facing the problems suggested. Some technical conservatives, particularly in places like the FAA, can actually suppress the increase in safety that modern systems could provide out of excessive fear of the risk of "new" (AOA indicators on small aircraft, and the typecerts required to put new avionics into older small aircraft come to mind)

Not surprising

[MrKaos]

I've made several requests for systemd proponents to supply a use case that SysV initd could not support and haven't received a satisfactory reply to this purely technical question. I was interested in what systemd could offer over initd. I find systemd proponents are overly veherment in their criticisms of initd proponents.

I sense this comes from an inability to address the issues raised and, perhaps a mindset that anyone who has an appreciation for initd's elegant power will simply be bulldozed into irrelevance. I think systemd's criticism of the rc scripts that starts a linux based system is valid criticism however we have to keep in mind that they were devised by Red Hat. It is dealing with rc shell scripts that are the brunt of the justification for systemd.

In that sense the unitd solution is tidy but also reveals the justification to replace initd is not based on a full understanding of its capabilities, or even an understanding of was it is, a process manager. rc scripts are only meant to prepare the system for entries in /etc/inittab, yet everyone tries to get everything done in rc, which serializes the Linux boot process. A parallell boot is completely achievable by using initd properly. I know there is more to it, like events and messaging, I'm just citing one example.

Yet I've never seen a Linux distro that's utilized initd's /etc/inittab file properly. Especially a Red Hat system. They don't use initd properly, the rc scripts are bloated with rewrites of what initd already does, and now we're replacing initd, keh? initd has yet to be utilized fully on modern linux systems.

Criticisms of sco the company aside: sco *as a distribution of unix* had an interesting adjunct feature to initd, the 'enable' and 'disable' command that managed entries in /etc/inittab, where you would configure the characteristics of the system you were running. Franky I think this is functionality is essentially

sed -i -e '/someProcess/ s/:on:/:off:/' /etc.inittab ; kill -1 1

I think initd would make a lot more sense to more people if this functionality had been available in Linux from the beginning. It is true that initd is beguiling in terms of it's simplicity wrt its power, but it is also very worthwhile. It is supposed to be small as that is where the skill is expressed.

initd is where you design the characteristics of the system, it is not an event manager and all the other things systemd is supposed to be. Something that does all the functionality systemd has, belongs as an inittab enty, not as the first process the kernel runs.

The point of a bug like this is not that it is a big deal itself, the big deal is the failure mode systemd has been revealed to have due to its complexity. This the type of concern I have about systemd, what else can trigger such a failure mode. I have seen initd in a variety of failure modes and not once has it ever consumed all system resources and disconnected running processes.

Now we've seen systemd do something that initd can't.

Re:Not surprising

[MrKaos]

the justification to replace initd is not based on a full understanding of its capabilities

I can't fathom what it is based on. Ego?

Me either, however they are pushing systemd awfully hard to destroy a core part of Linux stability.

The sane decision would have been to put it there as an option and let users choose.

Perhaps this is RH's "Imperial" phase, they've won their market share and now they can do what they want.

Re:Not surprising

[MrKaos]

Why not do your own research instead? Go and learn systemd and then compare and write up a nice unbiased report for us all to read.

I have been learning systemd and have more and more criticisms of it as I learn more. I have test systems set up with systemd and a bunch of middleware that we used to integrate with initd.

In my experiences systemd replaces a lot of general knowledge you can apply elsewhere (like shell scripting and regular expressions) with specific knowledge about systemd's and its properties, so I'm unsure what a report would achieve considering new capabilities are added to systemd all the time.

Frankly I think I would rather spend my energy on writing some adjuncts to initd (like enable and disable) that make it more accessible. I'm open to suggestions.

Re: Doctor Doctor Give Me The News

[corychristison]

Yet another Gentoo user here.

Wellz not 100% accurate because I've since moved on to Funtoo. Only because Gentoo stopped making OpenVZ templates at one point, and Funtoo was "close enough" for what I needed at the time.

Since then, I've moved all of my machines to Funtoo with the exception of two cPanel VM's I have running for clients that required cPanel and weren't open to an alternative.

Honestly I can't see myself using anything else. OpenRC does everything I need. Honestly the fact it depended on udev was worrying, but then they forked it into eudev so its completely uncoupled from systemd.

Unfortunately SystemD isn't the only one

[Casandro]

The whole "FreeDesktop" Movement seems to be about making Linux more and more incomprehensible.

My theory for why this is is like this:
There are lots of people now growing up when Windows kinda worked (since about 2000). At the same time, involvement in "Open Source" software is seen as a good career move. So they churn out some shitty badly designed code as potential recruiters cannot tell good from bad code. Also they take part in design processes without the experience necessary for this. The result are overcomplex buggy solutions which suck in manpower to maintain them.

Take a look at the *BSD people. The team maintaining OpenBSD is probably smaller than the SystemD team, yet they manage to maintain a whole operating system.

Re:Doctor Doctor Give Me The News

[mrvan]

My guess* is that they are from separate stems. In Dutch, kleven (clay-vun) is to stick together, and klieven (clee-vun) is to split apart.

No idea where the contradictory meaning in sanction comes from, in Dutch 'sanctioneren' (v) also has both meanings and people get confused.

*) And Internet confirms it :) http://www.etymonline.com/inde...

Re:Doctor Doctor Give Me The News

[JustAnotherOldGuy]

I've always loved the multiple meanings in "the boat is fast". Just like the word "secure", it can mean wildly different things:

If you give the command "SECURE THE BUILDING", here is what the different services would do:

The NAVY would turn out the lights and lock the doors.

The ARMY would surround the building with defensive fortifications, tanks and concertina wire.

The MARINE CORPS would assault the building, using overlapping fields of fire from all appropriate points on the perimeter.

The AIR FORCE would take out a three-year lease with an option to buy the property.