Slashdot Mirror
Does Systemd Make Linux Complex, Error-Prone, and Unstable? (ungleich.ch)
"Systemd developers split the community over a tiny detail that decreases stability significantly and increases complexity for not much real value." So argues Nico Schottelius, talking about his experiences as the CEO of a Swiss company providing VM hosting, datacenters, and high-speed fiber internet. Long-time Slashdot reader walterbyrd quotes Nico's essay:
While I am writing here in flowery words, the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis. This is even without calculating the security risks that come with systemd. Our objective is to create a great, easy-to-use platform for VM hosting, not to walk a tightrope...
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
The BSDâ(TM)s and Illumos. There is no reason to use the tire fire that is Linux. You have options!
Here's a list of actual problems that should have been solved instead of introducing the nightmare of systemd upon the Linux (Debian specifically) world:
- Forceful, unconditional kernel operations. When I say "unmount this filesystem," I'm not asking a question. When I say "terminate this process," I expect the process to be removed from memory and the runqueue, regardless of consequences.
- When I say "reboot" I mean "reboot." Hangs are not okay, ever.
- Actual, real soft NFS failures. Do not hang during boot for any reason unless that share is marked hard,nointr. Do not hang during shutdown/reboot, either.
- Enforce GPL-standard syntax on new incoming utilities. If you want into the package tree, use a GNU parsing library and use it correctly. Perhaps a standardized syntax wrapper available for package maintainers.
- Bolt simple parallelization, triggers and flow control onto init/rc.
- Drop this selinux shit. It's too complicated and causes more problems than it solves. Vulnerabilities come from bad code, not a lack of complex call ACLs. Security is a process, not a feature.
- Standardize and fix bluetooth support ffs.
My $0.02, as a 25-year Linux admin.
A government is a body of people notably ungoverned - AC
A big ol ball? My init.d was about 13 scripts big which were readable and editable. Ever tried to edit systemd files? Depending on systemd version you have to create overrides, modify symlinks or edit systemd files straight up which can be in about 5 different locations and on top of that, systemd can have overrides on any changes either with an update or just inherited.
Systemd makes every system into a dependency mess.
Remove/fail a hard drive and your system will boot into single user mode, not even remote access will be available so you better be near the machine just because it was in fstab and apparently everything in fstab is a hard dependency on systemd.
Custom electronics and digital signage for your business: www.evcircuits.com
So the short answer is: Yes, systemd makes things unnecessarily complex with little benefit.
That matches my experience - losing a lot of time trying to figure out why things don't work. The improved boot time is lost several times over.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Yep... With a flamebait headline, flamebait summary, and flamebait TFA, this should provide ample opportunity for well-reasoned and insightful discussion, I'm sure... Thanks, Slashdot!
Do one thing, and do it well. Systemd has eaten init, udev, inetd, syslog and soon dhcpd. Yes, that is getting ridiculous.
Systemd creates a dependency mess which means it cannot be replaced by simpler things, which wasn't the case before systemd.
I don't think there's a problem with the idea of systemd. Having a standard way to handle process start-up, dependencies, failures, recovery, "contracts", etc... isn't a bad, or unique, thing -- Solaris has Service Manager, for example. I think there's just too many things unnecessarily built into systemd rather than it utilizing external, usually, already existing utilities. Does systemd really need, for example, NFS, DNS, NTP services built-in? Why can't it run as PID 2 and leave PID1 for init to simply reap orphaned processes? Would make it easier to restart or handle a failed systemd w/o rebooting the entire system (or so I've read).
In short, systemd has too many things stuffed into its kitchen sink -- if you want that, use Emacs :-)
[ Note, I'm a fan and long-time user of Emacs, so the joke's in good fun. ]
It must have been something you assimilated. . . .
Betteridge's Law says no.
I'm still yet to have someone give me a legitimately non hysterical reason why "systemd bad" other than "its different"
This indicates a problem with understanding technology and technological explanations on your side, nothing else. "Safety in numbers" does not work for software.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I disagree on SELinux, not because its interface is well-designed (it is not), but because it is needed for some things.
On the rest, I fully agree. And instead, systemd solves things that were already solved and does it badly. The amount of stupidity in that decision is staggering.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Slackware or BSD. Everything else is not worth it.
Red Hat has managed to do what everybody compains about Windows. Embrace Extend & Extinguish. Open source means jack squat when 1 or 2 corporte entities control the whole infrastructure stack.
Perhaps you have had no problems with systemd because you aren't trying to use it to do much.
Lots of people, myself included, have had issues trying to get things which are trivial in pre-systemd or on other OSes to work properly and consistently on systemd. There are many, many, many examples of issues. If someone asked me for examples, I'd have a hard time deciding where to start because so many things have been gratuitously changed. If you really think there aren't examples, just read this thread.
On the other hand, I have yet to see real technical discussion about problems that systemd apparently is fixing. I honestly and openmindedly am curious about what makes systemd good, so I've tried on several occasions to find these discussions where good technical reasoning is used to explain the motivations behind systemd. If they exist, I haven't found any yet. I'm hoping some will appear as a result of this thread.
But you bring up the idea that the "market has spoken"? You do realize that a majority of users use Windows, right? And people in the United States are constantly electing politicians who directly hurt the people who vote for them more than anyone else. It's called marketing. Just because something has effective marketing doesn't mean it doesn't suck.
I have a PC with a Bios that tries to do everything, launching a bootloader that tries to do everything, to start a DesktopManager that treis to do everything, so I can run a browser that tries to do everything to see a website that tries to do everything.
And to think I started with Linux, because each thing had its own program and I could select what program did it.
Don't fight for your country, if your country does not fight for you.
systemd fails silently if something is wrong in /etc/fstab. It just doesn't finish booting. Which is moderately annoying if you have access to the system console and you can guess that an unchanged /etc/fstab from before systemd that worked for a while with systemd is now suddenly toxic
If you do not have easy access to the system console or you are not blessed with divine inspiration, that is quite a bit more than annoying. Thanks to the binary log files you cannot even boot something random and read the logs, but at least you aren't missing anything, because nothing pertinent to the error is logged anyway.
The problem is that one camp won't admit that old init is a pile of shit from the 80's whose only virtue is that the stench has faded over time, and the other camp won't admit that their new shiny toy needs to be understandable and debuggable.
A proper init system needs dependencies and service monitoring. init + monit does not cut it today. Systemd does that bit rather impressively well. It's just terrible at actually booting the system, all the early boot stuff that you could depend on old init to get right every time, or at least spit out aggressive messages about why it failed.
Finally! A year of moderation! Ready for 2019?
If systemd is so terrible, then why did a lot of the major distros switch over?
There's one use case that systemd is really good at, and that is making things easier for distro maintainers. The creators of systemd spent effort discussing it with distro maintainers, and figuring out what features they wanted. That is why they switched over.
Systemd isn't easier for any other demographic, though.
"First they came for the slanderers and i said nothing."
>To me, the fact that the major distros have adopted systemd is strong evidence that it is probably better
"Better" is a subjective term. Software (and any product really) does not have some absolute measurable utility. It's utility is specific to an audience. The fact that the major distros switch is probably strong evidence that systemd is "better" for distro developers. But the utility it brings them may not apply to all users, or even any particular user.
A big part of the reason people were upset was exactly that - the key reasons distros had for switching was benefits to people building distros which subsequent users would never experience. These should not have trumped the user experience.
All that would still have been fine - we could easily have ended up with a world that had systemd for those who wanted it, and didn't have it for those who didn't want it. Linux systems are supposed to be flexible enough that you can set them up to whatever purpose you desire.
So where the real anger came in was the systemd's obsessive feature-creep made it go into a lots and lots of areas that have nothing to do with it's supposed purpose (boot process management), in that area it's biggest advantages are only useful to people building distributions (who have to maintain thousands of packages and ensure they reliable handle their bootup requirements regardless of what combination of them is actually installed- systemd genuinely did make that easier on them - but no user or admin ever experiences that scenario). But that feature creep itself wasn't even the issue, the issue was that - as it entered into all these unrelated areas (login was the first of many) - it broke compatibility with the existing software to do those jobs. This meant that, if you built a system to support systemd, that same system could not use any alternatives. So now, you had to create hard dependencies on systemd to support it at all - for distros to gain those benefits, they had to remove the capacity for anybody to forgo them, or alternatively provide two versions of every package - even ones that never touch the boot process and get no benefit from systemd's changes there.
And the trouble is - in none of those other areas has it offered anything of significant value to anybody. Logind doesn't actually do anything that good old login didn't do anyway, but it's incompatible so a distro that compiles it's packages around logind can't work with anything else. Replacing the process handler... and not only did it not add any new functionality it broke some existing functionality (granted, in rarer edge cases -but there was no reason for any breakage at all because these were long-solved problems).
Many years ago, I worked as a unix admin for a company that developed for lots of different target unix systems. As such I had to maintain test environments running all the targets. I had several linux systems running about 5 different distros, I had solaris boxes with every version from 8 onwards (yep, actual Sparcs), I had IBM's running AIX, I even had two original (by then 30 year old) DEC Alphas running Tru64... and I had several HPUX boxes.
At the time, while adminning all these disparate unix environments on a day-to-day basis and learning all their various issues and problems - I came to announce routinely that Solaris pre-Version-10 had the worst init system in the word to admin, but the worst Unix in the world was definitely HPUX because HPUX was the only Unix where I could not, with absolute certainty, know that if I kill -9 a process - that process would definitely be gone. WIped out of memory and the process table with absolutely no regard for anything else - it's a nuclear option, and it's supposed to work that way - because sometimes that what you need to keep things running.
SystemD brought to Linux an init system that replicated everything I used to hate about the Solaris 8/9 init system - but what's worse than that, it brought the one breakage that got me to declare HPUX the absolute worst unix system
Unicode killed the ASCII-art *
if I have a problem with e.g. my dovecot instance on a server, with rsyslogd (as default installed on Debian) I get the fun of guessing which of mail.log, mail.info, or mail.err contains the messages I might like to see (with the mild suspicion that I ought to also glance at debug.log as well, just in case), then if I like to see things in chronological order I have the added amusment of running a command line like this:
zcat $(ls -tr /var/log/mail.log.*.gz) | cat /var/log/mail.log - | grep dovecot | grep $whatever_I_really_wanted_to_see
and I'll get most of what I'm looking for, along with anything else that contains the word dovecot.
[BTW hands up anyone that thinks a gzip file is a text file]
whereas with systemd it's just so bloody tedious:
journalctl -u dovecot | grep $whatever_I_really_wanted_to_see
Where's the fun in that?
Debian: GNU/Linux done the Linux way
So tell us, why didn't you simply install whatever log system you prefer? You are the classic example of the person with no clue what they are doing blaming systemd.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
As usual lack of knowledge is your real problem. ntp is an *optional* component of systemd. If you don't like theirs use your own. You don't have to know *any* C to use systemd. I don't even believe that you think you do. If you spent even a minimal time reading the documentation your problems would all magically disappear, but you don't want that. Complaining in ridiculous ways about non-issues is much more fun, and you can get lots of +1 mods from everyone else who learns everything they know about it from Slashdot and never invested any time learning about it.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Granted, I have never needed any kind of tampering or corruption mitigation in my log files over the last 20 years of Linux administration. So the value for at least my usage of journalctl has been sum negative because I don't see the value in a command that by default truncates log output.
So the answer for systemd is to workaround it by using a "legacy" service to restore decades of functionality.
SMF was the death knell for Solaris (along with the Oracle purchase), and it feels like systemd is going to be the anchor which drags Linux into the abyss.
--WooooHoooo--
Tough question. Depends what that functionality is. Compatibility is valuable but sometimes it must be sacrificed to deal with technical debt or make genuine progress. Even Microsoft had a huge compatibility break with Vista which was needed at the time (even if Vista itself was atrocious). /etc is a major deal - it utterly breaks with a standard around which disk space allocation is done professionally. /use ought to not even need backups because everything there is supposed to be installed and never hand edited. It means modifying backup strategy which is a big, very risky, cange. Logs aren't where I expect them. Boot errors flash on screen and disappear before you can read them so you have to remember to go look in the binary log to figure out if it was something serious.
It would depend what those features were, what benefits it gave me. It would be a trade off and should be evaluated as such. A major sacrifice requires an even more major advantage to be worthwhile. I've yet to see any such advantage from anything systemd has added. I'm not saying advantages don't exist, I'm saying whatever they may be they do not benefit me, personally, in any measurable way. The disadvantages however do, and compatibility is the least of them.
Config outside
I was never a fan of system V. It was a complicated, slow, mess if code duplication. It needed a replacement. I was championing Richard Gooch's make-init circa 2001 (and his devfs, the forerunner to udev, was in my kernels - I built a powerful hardware autoconfig system on it in 2005 when I built the first installable live CD distribution, the way they all work now: I invented it [I later discovered that pclinuxos had invented the same thing independently at the same time but Ubuntu for example still came on two disks, a live CD and separate text based installation disk and more than once I had machines where the live cd ran great but the installed system broke due to disparate hardware setup systems]). Later I praised upstart - it was a fantastic unit system that solved the issues with system V, retained compatibility but was easy to admin, standards and philosophy compliant and fast. It was even parallel.
That is the system that should have won the unit wars. I'm not a huge fan of Ubuntu's eclectic side, unity has always been a fugly unusable mess of a desktop to me - but upstart was great, that and PPAs are Ubuntu two most amazing accomplishments. Sadly one got lost instead of being the world changing tech it deserved to be and it lost to a wholly inferior technology for no sane reason.
It's the Amiga of the Linux world.
Unicode killed the ASCII-art *
Using text processing skills to process a generic text file isn't any harder than using journalctl. The difference is that the former is generically applicable to just about any other software on the planet, and the latter is for journald. It's not that complex to confer the journalctl benefits without ditching *native* text log capability, but they refuse to do so.
Using ForwardToSyslog just means there's an unnecessary middle-man, meaning both services must be functional to complete logging. The problem is the time when you actually want logs is the time when there's something going wrong. A few weeks ago was trying to support someone who did something pretty catastrophic to his system. One of the side effects was that it broke the syslog forwarding (syslog would still work, but nothing from journald would get to it). The other thing that happened would be for the system to lock out all access. I thought 'ok, I'll reboot and use jornalctl', but wait, on CentOS 7, journald defaults to not persisting journald across boot, because you have syslog to do that.
Of course the other problem (not entirely systemd project fault) was the quest to 'simplify' console output so he just saw 'fail' instead of the much more useful error messages that would formerly spam the console on experiencing the sort of problem he hit (because it would be terrible to have an 'ugly' console...). This hints about another source of the systemd controversy, that it's also symbolic of a lot of other design choices that have come out of the distros.
XML is like violence. If it doesn't solve the problem, use more.
Remind me again how useful journalctl and binary logs are when you can't remember the exact name of the unit? "tail -f /var/log/messages | grep dhcp" is a lot easier to remember than "journactl -f -u isc-dhcp-server" - and hopefully you ARE running isc-dhcp-server, because if it's a different server you're SOL.
Remind me again how useful journalctl and binary logs are when the only things that run on a system are "echo" and maybe "/bin/cat" if you're lucky?
And yes, I've had that happen. Problems with the init system usually result in systems that have minimal functionality available. SystemD has far too many dependencies to reliably reconstruct a system that has failed init.