Slashdot Mirror
Does Systemd Make Linux Complex, Error-Prone, and Unstable? (ungleich.ch)
"Systemd developers split the community over a tiny detail that decreases stability significantly and increases complexity for not much real value." So argues Nico Schottelius, talking about his experiences as the CEO of a Swiss company providing VM hosting, datacenters, and high-speed fiber internet. Long-time Slashdot reader walterbyrd quotes Nico's essay:
While I am writing here in flowery words, the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis. This is even without calculating the security risks that come with systemd. Our objective is to create a great, easy-to-use platform for VM hosting, not to walk a tightrope...
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
1. You've moved having a basic understanding of the boot process, and the ability to fix things, from having a decent knowledge of bash to being a C wizard.
2. You've broken decades of understanding the boot process.
3. It breaks KISS, as it doesn't simply do startup. Hell, it does ntpd.
It breaks a lot of the *concept* of unix. Maybe to something preferred by a lot of people - but it also turns it into an alien mess to a lot of other people.
So the short answer is: Yes, systemd makes things unnecessarily complex with little benefit.
That matches my experience - losing a lot of time trying to figure out why things don't work. The improved boot time is lost several times over.
I completely agree. Troubleshooting is really a bitch with systemd, much more time-consuming. For instance, often systemctl reports a daemon as failed while it's not, or suddenly decides that it didn't start because of some mysterious arbitrary timeout while the daemon just needs some time to run a maintenance tasks at startup time. And getting anything of value out of the log is a pain in the ass.
Quite often I end up writing control shell scripts specifically to be called by systemd, because this junkware is too fragile and capricious to work with actual daemons. That says a lot about the overal usefulness of systemd.
Nothing has been gained with systemd, at least not on servers.
lucm, indeed.
Troubleshooting is really a bitch with systemd, much more time-consuming. For instance, often systemctl reports a daemon as failed while it's not, or suddenly decides that it didn't start because of some mysterious arbitrary timeout while the daemon just needs some time to run a maintenance tasks at startup time.
Not to mention that the damn logs are not plain text, which in itself complicates things before you even have the chance to start troubleshooting.
#DeleteChrome
moving to BSD is done like this:
Estimated total time 90 minutes (OpenBSD), unless you are still using a 32 bit machine on 811g*. Do not restore /etc and /var in place, but somewhere else (like /home/restore) and migrate the settings as you find you need them.
Once its working, delete the /home/restore subtree.
There should not be anything that is file system dependent anyway unless you are deeply into file system development.
+ Install from floppies is no longer recommended, and installing over the net will take a while if you have not done it before
* A 32 bit machine on 811g will probably still perform adequately for some purposes - such as a trial run to see if its as easy as I say.
Sent from my ASR33 using ASCII
People who complain about systemd the most seem to have been using Linux for a very long time and just "don't want to change".
no, that's not it. people who have been using linux for a long time usually *know the corner-cases better*. in other words, they know *exactly* why it doesn't work and won't work, they know *exactly* the hell that it can and will create, under what circumstances, and they know *precisely* how they've been betrayed by the rail-roaded decisions made by distros without consulting them as to the complexities of the scenario to which they have been (successfully up until that point) deploying a GNU/Linux system.
also they've done the research - looked up systemd vs other init systems on the CVE mitre databases and gone "holy fuck".
also they've seen - perhaps even reported bugs themselves over the years - how well bugs are handled, and how reasonable and welcoming (or in some sad cases not, but generally it's ok) the developers are... then they've looked up the systemd bug database and how pottering abruptly CLOSES LEGITIMATE BUGREPORTS and they've gone "WHAT the fuck??"
also, they've been through the hell that was the "proprietary world", if they're REALLY old they've witnessed first-hand the "Unix Wars" and if they're not that old they experienced the domination of Windows through the 1990s. they know what a monoculture looks like and how dangerous that is for a computing eco-system.
in short, i have to apologise for pointing this out: they can read the danger signs far better than you can. sorry! :)
Everyone* switched to systemd because everyone* was using something that was much, much worse. Traditional sysvinit is a joke for service startup, it can't even handle dependencies in a way that actually works reliably (sure, it works until a process fails to start or hangs, then all bets are off, and good luck keeping dependencies starting in the right order as the system changes). Upstart is a mess (with plenty of corner case bugs) and much harder to make sense of and use than systemd. I'm a much happier person writing systemd units than Upstart whatever-you-call-thems on the Ubuntu systems I have to maintain.
The problem with systemd is that although it does init systems *better* than everything else*, it's also trying to take over half a dozen more responsibilities that are none of its damn business. It's a monolithic repo, and it's trying as hard as it can to position itself as a hard dependency for every Linux system on the face of the planet. Distros needed* a new init system, and they got an attempt to take over the Linux ecosystem along with it.
* The exception is Gentoo, which for over 15 years has had an rc-script system (later rewritten as OpenRC) based on sysvinit as PID 1 but with real dependencies, easy to write initscripts, and all the features you might need in a server environment (works great for desktops too). It's the only distro that has had a truly server-worthy init system, with the right balance of features and understandability and ease of maintenance. Gentoo is the only major distro that hasn't switched to systemd, though it does offer systemd as an option for those who want it. OpenRC was proposed as a systemd alternative in the Debian talks, but Gentoo didn't advertise it, and nobody on the Debian side cared to give it a try. Interestingly Poettering seems to be *very* careful to *never, ever* mention OpenRC when he talks about how systemd is better than everything else. I wonder why. Gentoo developers have had to fork multiple things assimilated by systemd (like udev) in order to keep offering OpenRC as an option.
Technology doesn't hold still.
Doesn't mean that just because a technology is new it's automatically a good idea, great-grandpa.
Love, a NetBSD-on-everything millenial who'd pick Slackware if forced to use Linux
CLI paste? paste.pr0.tips!
Can I ask, why don't you and other admins/devs like you start to contribute to systemd?
Lennart Poettering has specifically said that he will not accept many important kinds of patches, for example he refuses to merge any patch that improves cross-platform compatibility.
And what's the reason, because people on forums are complaining? Because binary log files break the UNIX philosophy?
Here is my analysis of systemd, spread across multiple posts (links towards the bottom). It's poorly written software (the interfaces are bad, you can read through my links for more explanation), and that will only get worse over time if an effort isn't made to isolate it over time. This is basic system architecture.
"First they came for the slanderers and i said nothing."
Journald makes logging simple and convenient, right?
journald has been known to run out of memory and stop responding.
Due to the design of "oh just connect stdout of the process to journald" if you restart journald it closes all of those file descriptors and you silently lose all further logging from already running processes.
Journald, by design, will only log so much per process, meaning that if it's logged too much since startup/an error you're interested in, you've now lost it.
Why 'they' had to go for demonstrably broken binary logging using a new interface I don't know. They could have just extended the syslog format to make it mandatory to pass along program name and process ID in the message. Then they could have made it "easier" to find the logs by having a per program/facility directory under /var/log and then stuck to simple, plain text logging that the existing *nix tools can search with ease. But, nope, they had to go with this shitfest instead.
And that's only one component of the whole systemd shitfest. On a very simple Debian install I've had it exhibit issues with shutdown, hanging on something that is simple or ignorable.
Sadly I had to abandon using Devuan after a while. The only really supported version is the jessie (Debian old-stable) version, and I'm not sure even that gets timely security updates. Their equivalent of Debian stable (Stretch) is 'ascii' and got next to no updates during the few months I used it. Boot up was both nice and fast (a major systemd selling point) and reliable (unlike systemd). I guess they just need more in the way of human resources so that they can nail down which Debian packages have problematic systemd tentacles involved, then they could pass through other Debian updates as soon as they're available.
Though perhaps you struggle to type "journalctl" instead of "tail /var/log/messages" to see text on your console?
And if you're desperate to have a text log in addition to, or instead of a journal you can just set it up to forward messages to a traditional syslogger, or just dump text using a chron job.
So in summary, no it doesn't complicate anything. It provides functionality that a text log doesn't have but if you absolutely must have a text log you can do that too.
The problem is that systemd is full of bugs. When the boot process hangs, automounts fail, or shutdown gets stuck waiting on nfs (saying it will time out but the time out target keeps moving), troubleshooting requires knowing C. Those problems can't be fixed by config files and documentation. They are bugs in the C code which is far more complex than a boot system should be.
I'm curious if Redhat are regretting their decision on the early hard switch to systemd in RHEL7. I know for a fact their support system was flooded with issues from early adopters. I have friends in the industry still on RHEL6 as the upgrade to version 7 is a logistical nightmare and one who works at a reasonably large enterprise considering ditching Redhat entirely to go to a systemd-less alternative.
That faster boot time sure helps with servers that are only restarted once a year ...
The problem with systemd is that although it does init systems *better* than everything else*, it's also trying to take over half a dozen more responsibilities that are none of its damn business.
The only thing it "takes over" that some people seem to take exception to is the system log, and it does it for its own purposes. Yes, it can be argued that it would have been better to work with the generic syslog interface, but that just wasn't going to work. They would have had to rewrite large portions of it anyway that would have broken backwards-compatibility, so they just went with their own logging daemon instead, and they did their best to maintain backwards-compatibility by forwarding the log messages to the standard interface. Seems like a pretty reasonable decision to me. The other stuff--NFS, DNS, NTP, etc--is optional, and actually NOT recommended for general use. The standard system utilities are used unless you explicitly override them.
I really struggle to reconcile the Slashdot view that systemd is total crap and the fact that every major Linux distro has switched to it.
The Linux ecosystem is not sane. Redhat wanted more control of Linux so they pushed systemd. GNOME developers are easily distracted by shiny things (as proof I submit GNOME 3) so they went ahead and made GNOME dependent on it. And then Debian (which most Linux distributions are based upon) adopted systemd because GNOME depended on it. There were some other excuses, but that's the biggest reason. You can blame Redhat and Debian for this clusterfuck, and really, only a small handful of people in the Debian community are actually responsible for Debian's involvement. Debian's leaders were split almost down the middle on whether they should go to systemd. This is why major changes should require a 2/3 vote (or more!)
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Its not the Slashdot point of view, its a vocal minority. Personally I trust the developers at Red Hat more than random posters.
Your example is off base.
1. I seldom search my gzipped backlogs, because if something went wrong, it was recent. That is why they are gzipped.
2. If I am searching for a specific error message, like "$whatever_I_really_wanted_to_see", then generally no other program will spew that and the extra "grep dovecot" is unnecessary.
Generally this is all you need:
grep $whatever_I_really_wanted_to_see mail*
Which is even simpler than the journalctl bullshit.
Second, the example that you give above is not that difficult for an admin because zcat, cat, grep ARE ALL STANDARD TOOLS that I use for the output of virtually every program and service that I use. I do not want to have to use a different command (journalctl) to look at the output of every different program (systemd) that I run.