Slashdot Mirror
Ask Slashdot: Can You Say Something Nice About Systemd?
ewhac writes: "I'm probably going to deeply deeply regret this, but every time a story appears here mentioning systemd, a 700-comment thread of back-and-forth bickering breaks out which is about as informative as an old Bud Light commercial, and I don't really learn anything new about the subject. My gut reaction to systemd is (currently) a negative one, and it's very easy to find screeds decrying systemd on the net. However, said screeds haven't been enough to prevent its adoption by several distros, which leads me to suspect that maybe there's something worthwhile there that I haven't discovered yet. So I thought it might be instructive to turn the question around and ask the membership about what makes systemd good. However, before you stab at the "Post" button, there are some rules...
Bias Disclosure: I currently dislike systemd because — without diving very deeply into the documentation, mind — it looks and feels like a poorly-described, gigantic mess I know nothing about that seeks to replace other poorly-described, smaller messes which I know a little bit about. So you will be arguing in that environment."
Nice Things About systemd Rules:
Bias Disclosure: I currently dislike systemd because — without diving very deeply into the documentation, mind — it looks and feels like a poorly-described, gigantic mess I know nothing about that seeks to replace other poorly-described, smaller messes which I know a little bit about. So you will be arguing in that environment."
Nice Things About systemd Rules:
- Post each new Nice Thing as a new post, not as a reply to another post. This will let visitors skim the base level of comments for things that interest them, rather than have to dive through a fractally expanding tree of comments looking for things to support/oppose. It will also make it easier to follow the next rule:
- Avoid duplication; read the entire base-level of comments before adding a new Nice Thing. Someone may already have mentioned your Nice Thing. Add your support/opposition to that Nice Thing there, rather than as a new post.
- Only one concrete Nice Thing about systemd per base-level post. Keep the post focused on a single Nice Thing systemd does. If you know of multiple distinct things, write multiple distinct posts.
- Describe the Nice Thing in some detail. Don't assume, for example, that merely saying "Supports Linux cgroups" will be immediately persuasive.
- Describe how the Nice Thing is better than existing, less controversial solutions. systemd is allegedly better at some things than sysvinit or upstart or inetd. Why? Why is the Nice Thing possible in systemd, and impossible (or extremely difficult) with anything else? (In some cases, the Nice Thing will be a completely new thing that's never existed before; describe why it's good thing.)
We will assume out of the gate that systemd boots your system faster than ${SOMETHING_ELSE}, so no points for bringing that up. Bonus points are awarded for:
- Personal Experience. "I actually did this," counts for way more than, "The docs claim you can do this."
- Working Examples. Corollary to the above — if you did a Nice Thing with systemd, consider also posting the code/script/service file you wrote to accomplish it.
- Links to Supporting Documentation. If you leveraged a Nice Thing, furnish a link to the docs you used that describe the Nice Thing and its usage.
Okay, I'll play along: Systemd was the final straw that made me switch my Linux servers to FreeBSD.
See, was that so difficult?
Using the information gleaned from this year old bug I was able to create a Tower of Hanoi solver using the dependency resolver's attempts to determine whether a NFS filesystem should be mounted after the network comes up or not. Every attempt to start Apache (which depended on the filesystem) represents moving a disc to the middle tower.
Are you trying to imply that all the hatred I've heard about pulseaudio was also unjustified?
Only if you have a setup like Potterings, since he rewrote the pulseaudio configuration to work on his computer and just deleted features he wasn't using to "keep it easy".
Sort of like how if you're using systemd in an NFS world you're going to have a bad day, since he still hasn't made their dependency resolution work correctly with networked filesystems. Funny that rc dealt with that ages ago by having S01mount S05network S10networkmount but I'm sure there's a very good reason systemd is incapable of using mount -a -t nfs,cifs,smb,afs,etc, I suppose we'll need the kernel rewritten to support a systemd-mount program to solve this problem, which will of course be incompatible with standard mount.
...for better or worse, quite special conditions have to be met in the engineering department of a user's brain in order to like systemd.
Just installing systemd over sysv speed booting up from a minute to 10 seconds, and shutting down from half a minute to under one second. I always hated Linux couldn't handle shutting the fuck down efficiently, now it does.
A long time ago I used to do that with inittab. And then suddenly that was wrong, we weren't supposed to use inittab anymore, everything was supposed to bin in an init script.
Whatever
Remember how awesome pulseaudio is? Well what if we made your ENTIRE SYSTEM that awesome?
Oh I could only dream. Imagine that, a system which can cope with plug and play and access to multiple pieces of hardware at the same time. A system which can seemlessly switch outputs as hardware becomes available. My god that would be an amazing system.
Actually you've demonstrated the parent's point very well. Pulseaudio like systemd solves very real problems. It had teething issues and now seems to work rather well. If my experience with systemd will be anything like it is with Pulseaudio then I say bring it on.
(stay with me here...) Once upon a time there was a community. In the community were lots of different opinions - Slackware, Redhat, Debian, the weird *BSD folk - we all worked together, despite being of different religions. We'd yell at each other, and to an outsider we'd look as though we hated each other, but we were yelling at each other at the same bar while buying each other drinks. We yelled at each other because that's just what we liked to do. We had a certain set of rules that we all followed - and those rules were our real religion. We contributed code upstream. We filed bug reports. We did code review. We contributed. We Kept It Simple, Stupid. RMS was one of our major prophets - maybe even a god (though, we often started rolling our eyes and heading home for the night if he showed up at the bar to drink with us). We laughed at people who would declare, year after year, that this would be the Year of The Linux Desktop.
Then, along came two things - Ubuntu, and modern capitalism/culture/media/whatever - a mindset where there should be no plan, just go go go new feature new feature new feature go go go (I'm looking at you Agile, facebook, google...). Suddenly, the highest and best praise your project can get became whether it was "disruptive."
The *NIX/FOSS community would not have been a place for this to take hold, were it not for Ubuntu. Ubuntu decided they would break all our paradigms - they'd refuse to contribute patches upstream, they'd take simple processes that worked well and left tremendous power in the user's hands, and replace them with very broken messes of stuff. (In contrast to what we had...) they'd make an experience that mostly worked for complete novices - to be distinguished from most other distros that rarely worried much if even their initial installer failed because meh, you should know enough to know how to fix it yourself. They'd ignore religious ideals like only using OSS. And last but most certainly not least, they replaced init.d.
Problem is, when a lot of new people started in on the scene via Ubuntu (and the like), the established distros decided that they had always wanted their distro to be the desktop featured in The Year of the Linux Desktop, and realized they were losing overall "market" share (@#$%@ for those nitwits thinking of people as a "market," when we had been a "community" for ages), even though the number of users of each of the major distros was still increasing. So they looked around at what Ubuntu was doing to become popular, and tried to decide what to adopt from it. Unfortunately, this new crop of people included the likes of Lennart Poettering, who would have ideas such as this one, regarding systemd. Instead of seeing diversity and differences as good things, those of his ilk decided to destroy (yes, a harsh word...but it's pretty much accurate) the FOSS community. An entire set of ideals just...disappeared. No longer are simple things kept simple, no longer is "Do one thing and do it well" followed, no longer do we try to let open inter-connectivity organically solve problems of integration (instead, we just birth a giant Rock Biter to mow our laws).
Systemd came from a new set of ideals where solving problems that don't exist is great, so long as the big bad Establishment is taken out. I actually saw it as a bit of agism - where youth expected to be peers to those who had been around for ages, and when they weren't immediately accepted as experts they just co-opted the entire environment and left us old farts without any toys anymore. Oh wait...you wanted something good about systemd. Um, well, my laptop now boots 0.5 seconds faster than it otherwise would have, even if I no longer know why and can no longer really do anything about it. That's good, right?
Systemd during a normal boot is doing some detailed logging of the boot process that let's you do a simper version of bootchart. This allows you to find out how long each service took to boot and also do a graph of them.
It's concerned complimentary to the actual bootchart...
[1] http://0pointer.de/blog/projec...
Wait, what? It's the other way around. If the dependencies are systemd managed, systemd will wait until they report back to be up and running before starting dependants. This is opposed to the SysV rc way, which just blithely ignores all output and continues booting unless the rc script actually hangs.
In fact, one of the most common complaints is that systemd keeps hanging on filesystems in fstab that are not mandatory to mount on boot, instead of ignoring the mount error and leaving them to be mounted manually later.
"I know I will be modded down for this": where's the option '-1, Asking for it'?
Back in 1993 I worked on a network process controller for Inmarsat-A. It had it's own custom init process group leader which erected unix domain IPCs between certain children, some shared memory segments, and performed a waitpid to relaunch any child process that died. I was new to the project and new to unix so when I suggested we use inittab to take over this task, the team lead just laughed. Init was not up to the task. The arrival of systemd finally addresses this use case: a process group leader that can care for of a hierarchy of tightly coupled processes. Systemd may be complicated, but so is the intended use case.
I wonder if systemd was named with any irony to match the "System D" mentioned in Orwell's "Down and Out in Paris and London" and referenced by Anthony Bourdain's "Kitchen Confidential?"
System D refers to the whatever-means-necessary, MacGyver-ing, theiving, gerry-rigging, etc. that chefs and other restaurant workers may do to ensure that the service works without management or patrons noticing a problem. The term comes from "dbroillard" (Down and Out p78):
"Dbroillard is what every plongeur [dish washer] wants to be called. A dbroillard is a man who, even when he is told to do the impossible, will se dbroiller--get it done somehow."
Design for Use, not Construction!
In which case a nanny process restart is useless. Thanks for making my point, idiot.
Your logic is flawed. Random bit flips sometimes happen. They don't necessarily take down the whole machine.
Poettering get to pretend that he's Linus. That, and it manages to start all of the services correctly on my system without hanging about 9 times out of 10. Aside from that? as previous posters have said, if a service stops/crashes for some reason? I want it to stay stopped until i can figure out what is wrong. MySQLD? out of disk space? I don't want that to keep on yo-yoing up and down until i have it fixed. I agree, SystemD should be optional, you want it on your phone/IOT device? fine. My server that I reboot about once every 2-3 months? I want init.
OK I'll try this. Traditionally Init services were about starting processes. They didn't have capacities for keeping processes running. Which meant that for processes that need to be kept running and for which there was a real possibility of failure (most of them) the init had to start a process management system which then started the functional process. This has been the status for years. With systemd there is a process maintenance component standard in the init system. Which means that processes not only start but are capable of being kept in a stable state easily and automatically. Process management stops being something system admins work hard at and instead becomes something that happens out of the box.
Funny that rc dealt with that ages ago by having S01mount S05network S10networkmount but I'm sure there's a very good reason systemd is incapable of using mount -a -t nfs,cifs,smb,afs,etc,
Really? I confess I haven't looked at how to configure systemd much, but from what I have seen, that sort of setup could be supported easily enough by writing the right unit files (or whatever they're called) and specifying the dependencies.
Yes, exactly. So, by developing a clear understanding of systemd syntax and dependency specification, you should be able to solve the dependency problem that rc/sysvinit solved fifteen years ago by specifying that dependent processes start after their requirements.
This is actually a silly example. If all of the start-up processes are loaded sequentially, then order is really not a hard thing to sort out. You can do it manually, in your head, and your brain can process what's going to happen. Systemd introduces a massive complication by allowing init processes to happen in parallel. So, now you've got this horrible optimization to do, where you separate all of the separate init processes into mutually independent sequences that start in parallel. So, your brain can tell that you should start network file systems only after you've started the network, but how about the audio driver? Can you start named in one process while starting apache in another?
My point is that the dependency issue is really something that systemd introduces, and the benefit of its automatic dependency resolution is only relevant in the context of systemd. If it's really important to you that your computer be able to start multiple daemons in parallel, rather than wait the 5 seconds it takes to start bind before starting apache or samba or whatever, then systemd does that and solves the problems it introduces. If you'd rather understand what happens as your system starts up and not trust someone else's optimization, then systemd is not for you.
We've had problems a few times with systemd (usually the next boot after an upgrade to a package). Without exception, the failed boot occured with next-to-no meaningful error on the console and was more difficult to debug than if it had been using sysvinit. I personally, as a sysadmin for ~16 years, don't see a problem with sysvinit that justifies tearing it out of Linux for a more complicated, more opaque replacement.
Not to mention crashes caused by rare, hard-to-reproduce race conditions.
Indeed. That is one sub-category of the obscure software defects I mentioned. It's probably the best example, actually.
It's interesting to describe the approach used by many systems at Google (where I work; I'm now in Android but used to work on Google servers): a common pattern at Google is to crash immediately upon detection of any error. This is actually just a logical extrapolation from Google's long-used approach of building reliable systems on top of cheap, unreliable, commodity hardware, applying the same notion to individual software components. System designs assume that anything can fail at any time, and are built to recover gracefully, possibly with some degradation. So there is extensive infrastructure to fail a request over to to another process instance, and to automatically restart any failed processes. And of course there is extensive and detailed monitoring, with lots of statistical analysis of failure modes plus charting of everything to enable patterns to be recognized and various forms of alerting, ranging from automated bug filings, to e-mails, to pages delivered to on-call engineers.
Given that approach to fault tolerance, it's often very reasonable, at least for non-Java processes, to simply abort/crash whenever anything goes wrong. Restarts are automated and fast (for non-Java processes; JVMs are a bit slower to start) and monitoring and alerting take care of letting people know what happened and how often. This includes both hardware and software-related failures. Monitoring also pays special attention to processes that fail repeatedly (called "flapping") upon restart and generates high-priority alerts. The restart infrastructure will also slow and even stop the restarts of flapping processes.
Anyone who's used the googletest unit test framework for C++ may have wondered about the extensive support for and documentation of so-called "death tests", which allow you to verify that your code crashes when it's supposed to, and in the right way. This is a consequence of this particular approach to fault tolerance; if crashing is part of your reliability plan, you need to test that your code crashes when and how it's supposed to.
None of this has anything to do with systemd, of course. The fact that a strategy is effective in Google's environment is utterly meaningless in single-server contexts. In this case, though, auto-restart plus monitoring and flapping control seems like something that could usefully work in many contexts, perhaps even as the default.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.