Slashdot Mirror
Ask Slashdot: Can You Say Something Nice About Systemd?
ewhac writes: "I'm probably going to deeply deeply regret this, but every time a story appears here mentioning systemd, a 700-comment thread of back-and-forth bickering breaks out which is about as informative as an old Bud Light commercial, and I don't really learn anything new about the subject. My gut reaction to systemd is (currently) a negative one, and it's very easy to find screeds decrying systemd on the net. However, said screeds haven't been enough to prevent its adoption by several distros, which leads me to suspect that maybe there's something worthwhile there that I haven't discovered yet. So I thought it might be instructive to turn the question around and ask the membership about what makes systemd good. However, before you stab at the "Post" button, there are some rules...
Bias Disclosure: I currently dislike systemd because — without diving very deeply into the documentation, mind — it looks and feels like a poorly-described, gigantic mess I know nothing about that seeks to replace other poorly-described, smaller messes which I know a little bit about. So you will be arguing in that environment."
Nice Things About systemd Rules:
Bias Disclosure: I currently dislike systemd because — without diving very deeply into the documentation, mind — it looks and feels like a poorly-described, gigantic mess I know nothing about that seeks to replace other poorly-described, smaller messes which I know a little bit about. So you will be arguing in that environment."
Nice Things About systemd Rules:
- Post each new Nice Thing as a new post, not as a reply to another post. This will let visitors skim the base level of comments for things that interest them, rather than have to dive through a fractally expanding tree of comments looking for things to support/oppose. It will also make it easier to follow the next rule:
- Avoid duplication; read the entire base-level of comments before adding a new Nice Thing. Someone may already have mentioned your Nice Thing. Add your support/opposition to that Nice Thing there, rather than as a new post.
- Only one concrete Nice Thing about systemd per base-level post. Keep the post focused on a single Nice Thing systemd does. If you know of multiple distinct things, write multiple distinct posts.
- Describe the Nice Thing in some detail. Don't assume, for example, that merely saying "Supports Linux cgroups" will be immediately persuasive.
- Describe how the Nice Thing is better than existing, less controversial solutions. systemd is allegedly better at some things than sysvinit or upstart or inetd. Why? Why is the Nice Thing possible in systemd, and impossible (or extremely difficult) with anything else? (In some cases, the Nice Thing will be a completely new thing that's never existed before; describe why it's good thing.)
We will assume out of the gate that systemd boots your system faster than ${SOMETHING_ELSE}, so no points for bringing that up. Bonus points are awarded for:
- Personal Experience. "I actually did this," counts for way more than, "The docs claim you can do this."
- Working Examples. Corollary to the above — if you did a Nice Thing with systemd, consider also posting the code/script/service file you wrote to accomplish it.
- Links to Supporting Documentation. If you leveraged a Nice Thing, furnish a link to the docs you used that describe the Nice Thing and its usage.
By
eldavojohn
Systemd has a nice ring to it. The way the syllables roll off my tongue pleases me greatly. It could be the title of a great anime series. It could even be the lost name of an ancient forgotten god-king. It might even be the name I give my first born. It sounds much more authoritative and genuine than sysvinit or upstart or inetd. For instance from my non-technical fourth grade perspective this is what I interpret the others to mean:
Contrary to your base assumptions, systemd does not actually boot faster on my Pentium II (Intel inside) system. I just like the way it sounds.
My work here is dung.
I've done migrational upgrades to System-d with Arch Linux with zero problems in addition to using it with new installations. It works fine, and I'm still really confused about the jihad-level hatred it seems to engender in some people.
AntiFA: An abbreviation for Anti First Amendment.
If you have a service called 'foo', then 'systemctl status foo' not only shows you whether the service is running, it also shows you the last 10 log entries created by that service. This is great when the service failed; usually the error message will be right there.
How does it do this? Well, because all processes created by the service are in the same cgroup, all of the log messages (and even anything they print to stdout, which would have been lost otherwise) can easily be tagged with the service name (and a bunch of other metadata). You can use journalctl to query the journal for the logs from a specific service, and systemctl status does this for you.
Using the information gleaned from this year old bug I was able to create a Tower of Hanoi solver using the dependency resolver's attempts to determine whether a NFS filesystem should be mounted after the network comes up or not. Every attempt to start Apache (which depended on the filesystem) represents moving a disc to the middle tower.
My problem with things like systemd is not that they have nice features - lots of things have nice features. Windows' Shadow Copies is a lovely feature that's a lot easier to configure that some alternative equivalents, etc.
It's that they put those nice features into some new paradigm of configuration when you could have just added them to the existing system.
'systemd' needs to stay optional, and I mean that explicitly. optional. Not "default", and not "optional, in the sense that you then have to do the maintainer's job for them because they are too lazy to consider people not using systemd, because systemd is the default, and the maintainer does not want to consider the people that dont want systemd, regardless of the reasons or circumstances." kind of way.
Systemd is potentially useful for only a subset of the linux ecosystem, and forcing this kind of change is a bad thing.
Please allow me to explain why:
Systemd seeks to be a "Be all, end all solution to system initialization", which ultimately means that it will itself try to cover every possible thing that its maintainers believe needs to or should happen during system init. That in and of itself means that it will be large and cumbersome; exactly the things that embedded linux should avoid, where ultra-minimalism is king. (We are talking systems that have just a few dozen megabytes of memory, and just a few hundred megahertz of processor power at the most. Having all that gobbled up by the init system as soon as power is applied is not going to win you any trophies, and boldly asserting that embedded devices need to obey a desktop paradigm is throwing the baby out with the bathwater.)
This is especially true with "reference distributions", like debian. Debian "console only" deployments with tools like debootstrap are reasonably common with embedded devices, as are deployments that make use of chroots for specific sandboxed services. A chroot does not need a full blown init like systemd. It is best served with a simple init script. Building a distro with the intention of killing simple init, and replacing it with a monolithic solution like systemd will make service daemons much more difficult to control in this way, and will actually rob core functionality away from the distro that goes that route--- exclusively in favor of desktop flavored deployments.
Linux is more than desktops.
Linux is routers.
Linux is home automation systems.
Linux is servers performing specialized functions.
Linux is so much more.
Please be more considerate about trying to force systemd into debian. Optional is OK. Optional like "gnome vs kde vs xfce vs $ManagerHere". Not "optional" like "unity on ubuntu". Debian is a reference distro, upon which many other distros are based. It has already found its niche in the linux ecosystem. Please dont try to reinvent it.
http://uselessd.darknedgy.net/ProSystemdAntiSystemd/
This comes from an "anti-systemd" source, but tries to cut through a lot of the controversy and hostility shown on both sides. Bear in mind, you only see the anti-systemd view on Slashdot, but you get just as much idiocy on the other side as well. For example, the Poettering "death threats" were actually a joke made by a bunch of people in an IRC channel. Here, read the log (ctrl-F "hitman"):
http://logs.nslu2-linux.org/livelogs/maemo/maemo.20130215.txt
I've been starting to migrate many of my services at home to containers to make them a bit easier to maintain (a bit of a tangent - having 5 containers instead of one host with 5 services means that you have to do 5x as many updates, but each update can at most break one thing at a time). This was trivial to do with systemd-nspawn.
With a command line that barely fills a terminal line I can launch a container, have it boot systemd inside the container, have a few bind mounts, and have it get its own IP like a lightweight VM. Within the container systemd just does whatever it is told to do, like launch ssh so that I can get in, configure the network, and launch whatever services the container was intended to provide. The container journal logs are symlinked back to the host log directory, so they're really easy to look at from the host.
Sure, you can do similar things with docker, but doing it with systemd involves less tooling in general.
Also, for simpler situations systemd-nspawn makes a VERY good substitute for chroot. In addition to doing everything chroot does, it starts a separate process namespace so you don't see outside processes from inside the container. It also automatically sets up /dev for you, sets up resolv.conf, etc - it can do all this while just spawning one program inside just like chroot does (so no need to run systemd inside). It can also set up bind mounts if you ask it to. When you exit it cleans up - no lingering bind mounts, or tmpfs, or /proc and such inside. Also, any mounts inside the container aren't visible outside, so you can run a backup on your chroot and not have it follow bind mounts, or try to save /proc/kcore or whatever. In fact, you could spawn 5 containers inside the same directory and they can have private /tmp and /dev and /proc while seeing changes each other make in the files in the actual chroot.
Maybe I'm unique in this regard, but as an admin, if something goes down on one of my servers, I want it to stay down until I intervene.
Firstly, if I'm properly monitoring the process, then I'll be alerted and can investigate.
Secondly, there may a *reason* the process goes down, and having it down may be a good thing. If someone's trying to fuzz our httpd process for exploitation, then it happily restarting will open up a wider attack window.
Autopilots on production servers seem like a bad idea to me.
Method of processing duck feet
One thing I really like about systemd is that when you stop a service, it actually stops.
I used to run monit with openrc and when you wanted to restart a service you had to play games to ensure that it was really killed, and that the service state was cleaned up, and so on. Just telling openrc to stop the service just wasn't reliable at all - it worked well when nothing was wrong, but if nothing was wrong chances are monit wouldn't be doing anything.
Systemd is very effective at containing processes and their children and when you stop them, they are all gone for good. If you want to restart a service, systemctl restart service will get the job done 100% of the time, assuming the configuration/etc lets it restart. It does support graceful shutdown of individual services, followed by process genocide.
This also applies to things like cron jobs you launch through it. When the parent process ends, anything left gets cleaned up.
Systemd was forced down my throat by Arch Linux. I didn't know anything about the controversy back then, so I just thought: "There's probably a good reason for this, let's get to work".
I read some docs and I liked the security features a lot! You can tighten services easily with a declarative syntax.
Here's a snippet from my ntpdate.service file. You don't need much systemd knowledge to guess at what each line does:
PrivateTmp=true
ReadOnlyDirectories=/
InaccessibleDirectories=/boot
InaccessibleDirectories=/root
InaccessibleDirectories=/etc/ssh
LimitNPROC=1
DeviceAllow=/dev/null rw
DeviceAllow=/dev/urandom r
User=nobody
Group=nobody
CapabilityBoundingSet=CAP_SYS_TIME
NoNewPrivileges=true
I ended up enjoying that work and tightened things so much that I hit a bug, which was resolved in just a few days: https://bugs.freedesktop.org/s...
But I still don't know how to configure the network properly T_T
Or even better : The nice thing is systemD is not portable. It will infect only linux.
aaaaaaa
A small thing I've come to appreciate with systemd is that it actually cares about exit codes. This applies to any unit, including timer units (the equivalent of cron jobs). I ported most of my cron scripts over to systemd and suddenly started noticing scripts which had been having non-zero exits for ages, but fcron just didn't care about exit codes.
You can tell systemd to ignore exit codes for a process, or specific exit codes. However, I've found that in general using systemd I have a lot more awareness of abnormalities in my services.
Sure, you can often get away with ignoring exit codes, just as you can often get away with ignoring compiler warnings. However, in getting rid of them I fixed a few problems ranging from trivial to important, and my system is more robust for it.
In which case a nanny process restart is useless. Thanks for making my point, idiot.
Many hardware failures are transient, and a process restart is a very effective fix, at least in the short term. In the longer term, you'd better have monitoring in place so you know that the restarts are happening and can decide when to fix the hardware.
In addition, many process crashes are caused not by hardware failures, but by obscure software defects, and a process restart is not only effective at getting the production system back online, but arguably is a complete solution to the problem if the defect is sufficiently obscure that it's very rarely triggered, and hence not worth the large amount of effort required to identify and fix it.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Maybe I'm unique in this regard, but as an admin, if something goes down on one of my servers, I want it to stay down until I intervene.
I have the opposite view. If something goes down:
1) I want to know about it. I want some monitoring system to send me an alert of some description.
2) I want it set up in a way that it won't clobber it's own log files. Preservation of the reason something failed is key.
3) I want it to automatically restart if possible. I say if possible because I want to avoid a re-start loop but if something goes down and comes back up then the end user is happy as uptime is maximised and providing criteria 1 and 2 are met I'm no worse off as an admin. Believe it or not random crashes may sometimes happen and may have nothing to do with the config or the system itself. I may be caused by some edge case that was hit by a user, and in those cases I am unlikely to find the problem quickly and in the interest of not limiting user access would be inclined to get the system up and running as quickly as possible and then sort through the log history into what the actual problem was afterwards anyway.
The reality is that before systemd showed up, there wasn't really any project with an active upstream that tried to solve the plumbing problem (I'm not talking about init in isolation here). Each distro had to invent their own hacks, some of them good, some of them not.
The fact is that the community that is beginning to form around systemd is much more healthy than the scattered bits and not-quite-fitting pieces we had before. Maybe that's sad, I don't know. I think that in the end, the unification around systemd will allow competitors to form (just implement the interesting subset of the systemd interface and you can integrate with all distros!). So long term we'll end up with a much more vibrant plumbing for Linux.
In which case a nanny process restart is useless. Thanks for making my point, idiot.
Your logic is flawed. Random bit flips sometimes happen. They don't necessarily take down the whole machine.
Background: I've professionally administered Unix and Linux machines for >25 years, including various BSDs, Linuxes, Irix, HP-UX, Solaris/SunOS, AIX, etc. I've been certified by several vendors or distributions, including, since 1999, Red Hat (which gives me quite a bit of background on their specific implementations over the years). I don't work for a company doing development of any OS or platform. heck, other than random 401K type aggregate ownership, I don't own stock in any company that cares about this issue at a deep level, to the best of my knowledge :)
Personal Bias about this thread: You pretty much lose all the credibility possible with me when you start of with "I ... dislike systemd because ... it looks ... like a poorly-described, gigantic mess I know nothing about ... ." (It's the "disliking something you know nothing about" that bugs me). Otherwise, I don't care much about this debate on a personal level. I currently admin boxes using systemd as well as everything else, and nothing about systemd has caused me anywhere near the heartache that it seems people who haven't used it much seem to feel about it.
Seriously, there're thousands of pages of documentation about what it is, how it works, and what most if not all of the design basis decisions are/were. I'll link you to a few of them because hey, you can get to slashdot and post, but you can't seem to use Google ;) (tongue in cheeck, of course). There're plenty of folks who DO have great detailed reasons on why they don't like bits and pieces of it, and you should be able to compare them to the various info I'm linking below.
Systemd has tons of upside and tons of downside. Most are pretty well detailed, although many of the gut reactions people seem to have to it are based on a lack of understanding about how it works and what it's compatible with, to wit "I can't use shell scripts for anything at startup anymore!" , "All of my old chkconfig or SysV scripts can't be included at all!", "It kills off syslog!", "The only reason it exists is to make laptops boot faster and in the server world we don't care", etc. Those are easily researched and the actual basis (or lack thereof) pretty easily found.
So, for why the systemd setup looks like it does, you can go back 4.5 years to where the announcement and rationale is described. Speed is part of it, as is device changability, as is double-forking, resource limits, and service state checking and recovery. Yes, it's a load of stuff. Definitely a system-wide approach VS a semi-random collection of various ways to do things all tacked together (which is, frankly, what most Unix and Unixlike systems are, through survival of the fittest).
http://0pointer.de/blog/projec...
http://0pointer.de/blog/projec...
http://0pointer.de/blog/projec...
http://0pointer.de/blog/projec...
http://0pointer.de/blog/projec...
Since RedHat's obviously the largest major proponent and arguably the source of the most production users, here's their documentation:
https://access.redhat.com/docu...
Here's the project page with loads of links about the software and uses cases:
http://www.freedesktop.org/wik...
And of course so many questions have been raised the developers have posted their rebuttals to myths or misunderstandings.
http:/
Caveat: I am a server admin.
With systemd, one can't even remotely log a journal natively, which is par for the course in many server environments. You can't make this stuff up, please see bug #1098132 (https://bugzilla.redhat.com/show_bug.cgi?id=1098132) At the time I'm writing this, that functionality still just *doesn't exist* in systemd/journalctl. It was almost pushed into the last revision, but the bugs were show-stoppers so was pulled. Even if it does go into the next systemd revision, how long until it's really kosher for solid/production environments? 1 year? Two? Three? Yet it's being pushed as the default *now*.
To explain why this is important: if someone cracks in, the log files are going to be one of the first things they muck with. You have locked down syslog/et al, so you can tell if they muck with the binaries, but the log files themselves are considered compromised. Logging remotely at the same time before the data even hits the disk creates another layer of safety, which you simply can't do with systemd/journalctl. One day you will, assuming you want the binary journal and journalctl, but even if you wanted those now you couldn't. Yes, you could rsync over a copy of the files, but there's a reason why people aren't just doing that for regular logs and this is going long.
Over the last decades, people have worked out tried-and-true systems for documenting and verifying logging. So from an audit perspective, with journalctl you can't lock down syslog-ng and that process, because you've introduced an untrustworthy one above it. They know the current strengths and limitations -- to the point that there are legal/liability issues involved for many if their logging goes wonky. Journalctl adds a layer between the systems they have that work (and even have protocols in place for in terms of auditing), and as of right now adds a *wonky* layer that's very buggy. And by buggy I mean prone to corruption and simply not doing as its told.
Here's the real kicker: most of this issue would go away if systemd was willing to allow the user to not use journalctl and let things like syslog-ng have that data in the raw. It is choosing not to allow that as a tactic, as opposed to what the users and tech itself wants, and so here we are.
Also, this entire proposition by samzenpus is inane. When one thinks backwards from what the motivations might be, none of them are good and make me lose that much more respect for the site.