Slashdot Mirror
Debate Over Systemd Exposes the Two Factions Tugging At Modern-day Linux
walterbyrd (182728) sends this article about systemd from Paul Venezia, who writes:
In discussions around the Web in the past few months, I've seen an overwhelming level of support of systemd from Linux users who run Linux on their laptops and maybe a VPS or home server. I've also seen a large backlash against systemd from Linux system administrators who are responsible for dozens, hundreds, or thousands of Linux servers, physical and virtual. ... The release of RHEL 7 has brought the reality of systemd to a significant number of admins whose mantra is stability over all else and who perhaps had not waded into the choppier waters of Fedora or Debian unstable to work with systemd before it arrived in RHEL.
Is there scope for a less intrusive fork of SystemD?
OpenRC + Init seems to be the commonly referenced replacement. UselessD seems to be a more reactionary replacement, which is also often mentioned. I still can't see what's wrong with init scripts :)
Can one expect a re-write of all these daemons by a small team with no history of working on these applications to be anywhere near free of bugs?
Not likely.
"First they came for the slanderers and i said nothing."
Another reason for the systemd hate is the fact that it's been brought in by the back door. I dislike MariaDB for much the same reason.
Il n'y a pas de Planet B.
Debian already stated that they would not go with systemd, and I am very happy about that decision. Even though the article seems to imply otherwise by lumping Debian into Redhat and Ubuntu. I do think it's cool that all of the arguments I have presented here are exactly what the author states as the problem. I also hope the decision by Redhat backfires and they finally figure out who pays them, since they seem to consistently pander to a very vocal subset of desktop users (dropped Redhat support 3 years ago and it has turned out to be a great decision, except I miss Kickstart because FAI is a huge pain in comparison).
There are numerous programs for desktops that you don't want on a server. Gnome/KDE take up precious disk space that can be used better otherwise, numerous programs and services have security concerns, so the best answer is only load what you need. Booting graphics requires extra hardware probing, all the gadgets polling networks, all of the desktop wiz bangs downloading content and goodies, etc.. Great for your desktop, but not on a server that hosts 400 web sites. I need real time logging to a central server, not local binary logs that have to be translated and transported with a new format (good luck if networking does not start, because we just lost some audit trail). When MySQL does not start on a production server, you need to figure out why as quickly as possible and fix it. Not have some application sitting in the background chain starting a broken application and causing extra work and precious time while monitoring spins trying to figure out if the DB is really up or really down. Yeah, I know.. rewrite all the monitoring applications fixes the problem.. but I should not have to do this.
I'll be watching this one close, wondering if Redhat will have to come out with a fancy option to support systemd with an option for init. The market and time will tell, or of course Redhat could try and fix everything in systemd that's lacking. They have done that before.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
Administrators dislike constraint based systems.
This should surprise no one. One of the problems with a constraint based system is that you don't control the precise ordering of things.
This doesn't bother the Debian folks, because their build system is a constraint based system. If they have a package to install which has dependencies, they don't control the actual build order of the dependencies, or of their dependencies, and so on. Turtles all the way down. You do an apt-get install foo, and it's going to try to build subcomponents when they become available to try to build. And if they fail to build, they don't care: they "try again later", in case something that happens later satisfies the dependency that wasn't satisfied the first time around.
This is very disturbing to system administrators, who like things to be both orderly and predictable. All dependencies should be mapped out, known, and explicit. If something gets tried now, and fails, the correct response isn't "We'll try again later!", it's "Stop! Someone fix this fucking thing, it's obviously broken".
The build system is not deterministic; if there are two components, and one has a subdependency on some X of "at least version N", and another has a subdependency on X of "at least version N+2", then depending on the vagaries of network overhead, it's possible that half your code gets built with version N and the other half gets built with N+2, and things break. Things breaking is in fact far more acceptable to a system administrator than "things act weird", and "things act weird" is at least deterministic for a given build instance, and far, far, more preferable than "things sometimes work and sometimes don't".
So system administrators dislike Debian for large system installations. And they dislike systemd for starting things up and shutting things down.
A desktop system is far, far more forgiving: "It's not working; I'll just reboot!". As long as things "mostly work", then things are great! "Look! It's as good as Windows!".
Note that launchd in Mac OS X has many of the same problems as systemd; it's also a constraint based system. It's somewhat worse, in that it insists on controlling file descriptors and sockets and Mach ports for the things it starts - which means you have to rewrite a lot of at least the startup code in most Open Source software to tolerate being run by something that opens the files and sockets that it expects to do itself. But that's just a lot of make-work, and people who are paid to do work are paid because it's not something they'd be willing to do voluntarily, for free, and that's what they're exchanging for the money they are getting in exchange for putting up with that part of the job.
Unlike the people making things work with launchd, though, the people expected to make things work with systemd aren't being paid. And so systemd represents make-work and change for chage's sake, which doesn't sit well with volunteers.
--
So yeah, a lot of people find systemd annoying. Kirk McKusick once accused "vnode" as being "the structure that's taken over the kernel"; in Linux, systemd is fast becoming "the program that's taken over user space".
How this will all play out, I don't know, but don't expect it to be resolved any time soon, given the dichotomy between the philosophies of the stakeholders involved.
Systemd articles have become the new Slashdot clickbait. It seems every freakin' day there's another "discussion" about that unholy abortion.
I do not fail; I succeed at finding out what does not work.
Debian already stated that they would not go with systemd,
I think you might want to check again. Ironically, for all the claims that SystemD is for a better desktop experience, Ubuntu was the last one to enable SystemD. Not sure what to think about that. I think the reality is that SystemD makes life easier for distro builders, not for users, and that is why it has won.
I'll be watching this one close, wondering if Redhat will have to come out with a fancy option to support systemd with an option for init.
I'm really interested in seeing that too.
"First they came for the slanderers and i said nothing."
At this time it hangs on:
systemd[1]: Failed to run event loop: Invalid argument
systemd[1]: Failed to run mainloop: Invalid argument
systemd-logind[123]: Failed to enable subscription: Message did not receive a reply (timeout by message bus)
systemd-logind[123]: Failed to fully start up daemon: Connection timed out
The last time, at least a few months ago, I solved it by downgrading systemd from version 208 or so to the previous version. In the last update (of the rest of the system) dhcpd, sshd and probably others stopped working so I decided to update systemd as well and got the error above. The distro is Archlinux ARM.
My x86_64 installs work flawlessly, except sometimes when a program segfaults and journald decides to take a core for itself for a few minutes.
Firstly using a pid files is an utterly stupid idea and quite frankly, anybody who can not see that when they first think of them or read about them should not be an admin on any critical systems. However, much as I like init, init doesnt do pids more elegantly, it doesnt do them at all. The kernel does that by kindly telling init when one of its children has died and arranging for it to be able tell what the pid was.
init doesnt do much at all and thats why it works so well. It simply takes whatever run level you want, reads through /etc/inittab to see what jobs to start for that run level and starts them. It then re-starts them if it gets a child death signal and /etc/inittab said respawn. Simplicity itself even though most Unixes now break it by having it start one job that handles everything else. Im guessing the one problem with init is that it cant handle a process that forks and then exits and maybe thats the reason /etc/inittab is dying. Shame.
I also think the kernel handing orphaned processes over to init is cheating a bit but I like it :)
- there is a link on words "an overwhelming level of support of systemd from Linux users" - and that prompted me to click on that link (in clear violation of /. codex) because I was hoping to see who are these people that overwhelmingly support systemd? (apart from Lennart himself, that is).
All I got was a blog by Paul Venezia claiming that there is "an overwhelming level of support of systemd from Linux users". The links proving that claim are suspiciously missing. The blog itself seem to be be more on the skeptical side too.
So unless I see an overwhelming level of support of systemd from someone that matters and someone who knows what he talks about, then I'm not inclined to take that statement at face value.
Systemd has it's downsides. The real downside is that you have so much complex code running as root. most other complaints can be dealt with.
Binary logfiles: You're not supposed to keep important log files on the local machine. Send them to your central logging facility where they are stored in a database. If the machine is still running, you can use the appropriate tools to look at the binary log files for debug. All your logging, stats and alerting should be centralized anyway.
Doesn't feel unixy: Get with the times. It's scriptable and tweakable more than ever. Just get used to the way it works.
Solution looking for a problem: Just not true, see the benefits.
Systemd is one of the options to solve some problems that have been pestering unix for a long time.
Dependency in services: Systemd can restart all dependencies on a service in the right sequence if you have to meddle with one part of a stack
long startup times: Systemd has the possibility to start up things in parallel. No long waits for earlier systems that your service doesn't depend on. Mostly useful for mobile users, but HA systems benefit too due to shorter maintenance downtime
Location/circumstances specific profiles: Depending on where you are and what kind of facilities you have available, your system can "adapt" by changing power profiles, network connectivity, firewalling and whatnot. Primary benefits are for mobile users, but servers changing load depending on things like overheating, having to run on UPS power and such are also quite useful.
Systemd isn't the only project that wants to work this way. Upstart is another one that at least wants to deal with the startup concurrency and dependency problems of classical init. Sun (Oracle) Solaris SMF is also a good example of this line of thinking.
While you can have doubts about the amount of complex code and forks to 3rd party code done by systemd while running as root, I don't think it's useful to complain that someone moved your cheese and took away the init scripts you used to use in the old days. Once you figure out how to work with the new tools, you'll find it's way more tweakable and controllable than classical init. If in the end you choose for init or a different alternative, that's up to you, but at least investigate and educate yourself, before you start complaining with arguments that just aren't true.
I was promised a flying car. Where is my flying car?
No, it's not 'simplicity itself'. Inittab has no notion of dependencies, that's why the whole RC-level stuff had to be invented. Inittab has no notion of restart policies so if a service dies immediately after restart then it's certainly possible to get gigabytes of logs filled with output of a failing daemon. Inittab does not redirect output, so it's easy to miss crucial logs. Inittab rules don't care about service's prerequisites like mounted network shares.
It's actually hard to find something that inittab can actually do well.
Probably best if you do some of your own investigation into Systemd as to what you can configure in and out of its usage. Don't rely on the misinformation peddled in these postings . e.g. one of the biggest bits of misinformation is that systemd only emits binary logs but if you check the config options you will see
"ForwardToSyslog=, ForwardToKMsg=, ForwardToConsole=, ForwardToWall= Control whether log messages received by the journal daemon shall be forwarded to a traditional syslog daemon, to the kernel log buffer (kmsg), to the system console, or sent as wall messages to all logged-in users. These options take boolean arguments. If forwarding to syslog is enabled but no syslog daemon is running, the respective option has no effect. By default, only forwarding wall is enabled. These settings may be overridden at boot time with the kernel command line options "systemd.journald.forward_to_syslog=", "systemd.journald.forward_to_kmsg=", "systemd.journald.forward_to_console=" and "systemd.journald.forward_to_wall=". When forwarding to the console, the TTY to log to can be changed with TTYPath=, described below."
"The hands that help are better far than lips that pray." - Robert Ingersoll (1833-1899)
1. systemd was FORCED down my throat by just issuing a "pacman -Syu". it took me THREE fucking days getting my desktop back to normal after sifting through a barely comprehensible arch wiki page. systemd had long since been adopted by other distros and i was amazed how BROKEN it still is.
2. systemd FORCED me to sit down and learn a new set of tools for NO reason. my system worked just fine before systemd. i've since migrated to openrc (thx gentoo devs and apg + artoo from the arch community for making it possible (relatively easy to adopt) on my system.
3. systemd VIOLATES EVERYTHING K.I.S.S. it tries to be much too clever and it makes me feel dumb
4. systemd GRABS everything. not just init. try to install gnome without systemd... no way you can do that without a major hassle.
5. systemd -FORCES me to jump through hoops and rely on the good will of some very nice folks to provide me with a viable alternative. i have to rely on experimental and often buggy packages to get back to before where i had NO PROBLEM with init.
i hate SYSTEMD and this pottering can go to hell for what i care. thx for a whole lot of lost time. if it where just nothing and didn't force me to waste all this time i wouldn't even have cared... but this means WAR!
fuck you pottering. fuck you arch linux guys for violating your core principles and fuck all those ZEALOTS not giving me a choice in the matter and think they know what's best for me. this is the worst time being a gnu/linux advocate in like EVER.
on a positive note: thank you windows 7 for staying predictably stupid and giving me a working system while my linux init was broken for three days.
And as a result, when you do /etc/init.d/apache stop, apache stops. When you do /etc/init.d/apache start, (drumroll please), apacghe starts. Just exactly what he said.
That is exactly what systemd does, without all the hacks of the script. And with systemd I can finally be sure if Apache started or not, and I can put a dependency to (for example) mysqld, because most web sites are using MySQL as database. All of that is not possible without major hacks in Sysvinit.
Like "/usr/bin/systemctl apache start"?
add a helper app that runs the daemon in a cgroup and sticks around to manage it?
That is exactly what systemd is, but it does per default for all services. You can still use script hackery if you want to.
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
365 days without a security patch. Does uptime make you more money than protecting your customer data?
FFS. What makes you think a server needs to reboot for patches? Your extensive Windows administration experience?
UNIX/Linux servers need to reboot for a kernel patch. Very rarely (maybe never?) should a system need to reboot for anything other than a kernel patch. The number of recent packages aside from the kernel that require this is growing and a stunningly distrubing trend (mostly related to systemd).
Now, when must a kernel patch be applied? When a security patch is applied that affects something exploitable from one of your publicly accessible services.
An example, bind running inside a chroot jail and an exploit that requires access to a file or handle outside the jail != kernel patch and reboot.
A kernel patch for a local privilege escalation exploitable as www user with apache listening publicly = patch the kernel and reboot.
See the difference? There have been probably hundreds of local privesc exploits since I started working with Linux that just did not apply, and we very safely ignored the patch. When one matters, it is applied and we reboot. I've had specialty boxes that went multiple years without the need to reboot. We are on two commercial grids with battery and generator power available. I reboot when necessary, but have 6 9's of uptime (discounting planned outages) and the reason it's only 6 is hardware failure. It's 5 9's *INCLUDING* the planned outages across about 150 servers.
Now, I actually support systemd. But a few things seriously turn me off about it.
1) It is almost viral in what it demands, incorporates, and forces to be installed.
2) It doesn't appear to be well planned, documented, or tested.
3) There's a lot of scary shit in the bugtracker that is still unresolved or even assigned (some more than a year old).
Now, I accept that systemd and the 1000 required subpackages (udev, dbus, avahi, the QRcode enabled http server, journactl, etc.) are under development and alpha software. I don't understand why my production servers are supposed to migrate there now. Fix the broken crap and we'll talk, but again - my fucking notebook stopped resolving without a reboot after a non-kernel patch. Fuck that in production. Message clear?
> From what I can see on the RHEL lists that have many professional admins, there's been no hue and cry, no sky falling, etc.
Dude,
I don't know about you, but I admin about 400 odd servers, we've got about 40,000 globally. I've still got RHEL 4 boxes (Soon to be decomm'ed) Only some (5 - 10) of the boxes I built last year are RHEL 6. Everything else is RHEL 5 still. It works, I've no need to go above that for our purposes.
Now, I've got some new re-purposed boxes that I've started building with RHEL 7, and I've just started dropping myself into systemd.
Changing the startup scripts of *every* vendors application out there? No commercial applications are setup for systemd, this is going to be a loooooooooooooooong drawn out process to make this work.
RHEL 7 is brand new, very few people have started using it, the customers haven't had a chance to comment on it yet.
Curiosity was framed; ignorance killed the cat. -- Author unknown
Init scripts work just fine in systemd, and will for as long as there are init scripts. So vendors and apps *can* provide systemd service definition files, but they don't have to. It's backwards compatible just like upstart was in RHEL6. So no there's not a loooooooooong drawn out process to make it work. I'm running a debian box right now with systemd and everything is still in init scripts.
You can't pin that one on Linus, not at all. Kay Sievers, one of the heavies of the L.P. cabal, as part of systemd's subsuming of udev came up with this rotten idea of Predictable Interface Names. I have never known in which universe such a scheme is predictable. His motivation was to avoid races in the enumeration of devices controlled by newly inserted network-driver modules. (If you build these drivers in the kernel, as I always do, this problem wouldn't arise.) There were three ways to handle this situation, but he chose the one that causes the most pain.
They went with #3 as a default. That's the problem you're having. You can add the kernel command-line parameter net.ifnames=0 to go back to #1. If you have multiple Ethernet interfaces (as in a router) and you don't build your own kernels, you might prefer #2.
At any rate, this change is not Linus' fault!