Slashdot Mirror


Debate Over Systemd Exposes the Two Factions Tugging At Modern-day Linux

walterbyrd (182728) sends this article about systemd from Paul Venezia, who writes: In discussions around the Web in the past few months, I've seen an overwhelming level of support of systemd from Linux users who run Linux on their laptops and maybe a VPS or home server. I've also seen a large backlash against systemd from Linux system administrators who are responsible for dozens, hundreds, or thousands of Linux servers, physical and virtual. ... The release of RHEL 7 has brought the reality of systemd to a significant number of admins whose mantra is stability over all else and who perhaps had not waded into the choppier waters of Fedora or Debian unstable to work with systemd before it arrived in RHEL.

19 of 863 comments (clear)

  1. How about we hackers? by Anonymous Coward · · Score: 5, Insightful

    I know quite a few of us in hacker culture who hate the fact tha systemd does not feel UNIXy at all. It breaks practically every principle of the UNIX philosophy. Reminds me of working with windows, and that is never fun.

    1. Re: How about we hackers? by armanox · · Score: 5, Interesting

      Or we have learned that you can't argue with Red Hat. As a company we have decided against upgrading to RHEL 7 because of systemd, and likely will be migrating to FreeBSD when it is no longer supported.

      I'm waiting for our research team to get bored and start finding holes in systemd

      --
      I'm starting to think GNU is the problem with "GNU/Linux" these days.
    2. Re: How about we hackers? by Anonymous Coward · · Score: 5, Funny

      That's exactly the point.

    3. Re:How about we hackers? by 0123456 · · Score: 5, Insightful

      365 days without a security patch. Does uptime make you more money than protecting your customer data?

      Most of my servers are behind firewalls with no incoming connections through the Internet. And, yes, uptime matters when we're doing something more critical than serving funny cat videos.

    4. Re:How about we hackers? by s.petry · · Score: 5, Interesting

      As for the unix philosophy, init systems pre-systemd hardly did just one thing and hardly did it well.

      Every time I see stuff like this I simply laugh. Having worked with *nix for over 30 years I have never had init "not work well" or "not work" as people try and claim. This is 30 years, with at least 8 brands of *nix, and more servers than I can count any longer (ranging from 1CPU to 128CPU E10K/F15K, so my opinion is not based on limited experience).

      Systemd is not going to be any better, than Sun's SMF. SMF added nothing over init, except for some sales guy got to tell everyone how great it was. Maybe systemd is going to be worse though... at least SMF was script hackable as long as you could parse and edit XML, and that is not really possible with systemd (last I checked). And in that net zero gain, what did all of the Sun customers get? Lots and lots of costs to develop new scripts and new monitoring tools because SMF was different (not because monitoring was broken). Meanwhile anything that was important stayed out of SMF and went to legacy mode "init" scripts anyway since we could be extremely granular and detailed in a startup

      --

      -The wise argue that there are few absolutes, the fool argues that there are no probabilities.

    5. Re:How about we hackers? by Wheely · · Score: 5, Interesting

      I have similar length and breadth of experience of Unix systems and to be fair, I have seen init break but only once and it was when I broke it myself. I forgot to put an & and the end of a "sleep 20000 /dev/tty10" while trying to keep a serial line to a printer working properly. Next re-boot hung the machine but I was able to guess what the problem was.

      When I first saw SMF break I had absolutely no clue why I couldnt ssh into the machine nor where to start looking. It was when I discovered that sshd startup was dependent on utmp being available which depended on filesystem mounting being successful that I knew for sure that systemd style init was nothing I wanted.

      For me, scanning through /etc/inittab and being able to see exactly what is going on in the initialisation stage is the essence of Unix. Even this is sadly being slowly broken even before the utterly pointless systemd was born.

    6. Re:How about we hackers? by Anonymous Coward · · Score: 5, Informative

      At this time it hangs on:

      systemd[1]: Failed to run event loop: Invalid argument
      systemd[1]: Failed to run mainloop: Invalid argument
      systemd-logind[123]: Failed to enable subscription: Message did not receive a reply (timeout by message bus)
      systemd-logind[123]: Failed to fully start up daemon: Connection timed out

      The last time, at least a few months ago, I solved it by downgrading systemd from version 208 or so to the previous version. In the last update (of the rest of the system) dhcpd, sshd and probably others stopped working so I decided to update systemd as well and got the error above. The distro is Archlinux ARM.
      My x86_64 installs work flawlessly, except sometimes when a program segfaults and journald decides to take a core for itself for a few minutes.

    7. Re:How about we hackers? by MrKaos · · Score: 5, Insightful

      As for the unix philosophy, init systems pre-systemd hardly did just one thing and hardly did it well.

      Are you sure you are using it correctly. Whilst fussy, init is hardly complicated - perhaps you are thinking of rc?

      How does systemd remind you of windows?

      I think the binary log files is a good start.

      Have you actually *used* either in a system administration capacity?

      Yes, we've been testing systemd in-house extensively. It has compelling feature that I like (unit files are a big improvement) however the monolithic approach is a turn off. If it was a replacement for rc, I'd back it, however as a replacement for initd it is not.

      Whilst there is much pontificating about systemd, I think it is great for desktop systems however I can't see many enterprise deployments using it, it's just not ready for prime time. risk==downtime==2am working==no way

      I don't care if you call me a holdout. I know how to make systemd do what I want because I use it. Init is still simpler and more robust because while you see the blocking/slow startup as a problem, most experienced admins see it as init advertising what is wrong.

      systemd solves a problem that isn't really there.

      --
      My ism, it's full of beliefs.
    8. Re:How about we hackers? by devent · · Score: 5, Informative

      And as a result, when you do /etc/init.d/apache stop, apache stops. When you do /etc/init.d/apache start, (drumroll please), apacghe starts. Just exactly what he said.

      That is exactly what systemd does, without all the hacks of the script. And with systemd I can finally be sure if Apache started or not, and I can put a dependency to (for example) mysqld, because most web sites are using MySQL as database. All of that is not possible without major hacks in Sysvinit.

      /sbin/manage start apache

      Like "/usr/bin/systemctl apache start"?

      add a helper app that runs the daemon in a cgroup and sticks around to manage it?

      That is exactly what systemd is, but it does per default for all services. You can still use script hackery if you want to.

      --
      http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
    9. Re:How about we hackers? by Anonymous Coward · · Score: 5, Informative

      365 days without a security patch. Does uptime make you more money than protecting your customer data?

      FFS. What makes you think a server needs to reboot for patches? Your extensive Windows administration experience?

      UNIX/Linux servers need to reboot for a kernel patch. Very rarely (maybe never?) should a system need to reboot for anything other than a kernel patch. The number of recent packages aside from the kernel that require this is growing and a stunningly distrubing trend (mostly related to systemd).

      Now, when must a kernel patch be applied? When a security patch is applied that affects something exploitable from one of your publicly accessible services.

      An example, bind running inside a chroot jail and an exploit that requires access to a file or handle outside the jail != kernel patch and reboot.

      A kernel patch for a local privilege escalation exploitable as www user with apache listening publicly = patch the kernel and reboot.

      See the difference? There have been probably hundreds of local privesc exploits since I started working with Linux that just did not apply, and we very safely ignored the patch. When one matters, it is applied and we reboot. I've had specialty boxes that went multiple years without the need to reboot. We are on two commercial grids with battery and generator power available. I reboot when necessary, but have 6 9's of uptime (discounting planned outages) and the reason it's only 6 is hardware failure. It's 5 9's *INCLUDING* the planned outages across about 150 servers.

      Now, I actually support systemd. But a few things seriously turn me off about it.

      1) It is almost viral in what it demands, incorporates, and forces to be installed.
      2) It doesn't appear to be well planned, documented, or tested.
      3) There's a lot of scary shit in the bugtracker that is still unresolved or even assigned (some more than a year old).

      Now, I accept that systemd and the 1000 required subpackages (udev, dbus, avahi, the QRcode enabled http server, journactl, etc.) are under development and alpha software. I don't understand why my production servers are supposed to migrate there now. Fix the broken crap and we'll talk, but again - my fucking notebook stopped resolving without a reboot after a non-kernel patch. Fuck that in production. Message clear?

    10. Re:How about we hackers? by TheRaven64 · · Score: 5, Insightful

      I don't know why you've been modded troll. The problem isn't binary files, it's complex files. All of your log files are binary, the difference is that you have a load of small tools that can work with the ASCII / UTF-8 text ones easily. As long as there's a small program that can be statically linked and run from a recovery medium to turn the log files into something that other tools can handle (or, ideally, can search them faster) then there's no issue. The problem is systems where you need the entire GUI and a big chunk of the userland applications stack working to be able to read logs.

      --
      I am TheRaven on Soylent News
    11. Re: How about we hackers? by Anonymous Coward · · Score: 5, Insightful

      That's right, Linux is monolithic, but on the other hand--and this is a crucial difference--Linus is hugely concerned about preventing breakage. Of all the large packages I use, the kernel is the one that gives me the least worry when it comes to upgrade time.

      L. Poettering, on the other hand, seems to relish in breaking things. He sure isn't big on commiserating with people whose systems his code has broken.

  2. Are you sure? by phantomfive · · Score: 5, Interesting

    I've seen an overwhelming level of support of systemd from Linux users who run Linux on their laptops and maybe a VPS or home server

    I haven't seen an overwhelming support anywhere. Most people who run Linux on their laptops say, "meh, as long as it boots."

    This article is a LOT better than the previous one by the same author. He makes his point clearly, that he doesn't like SystemD, as a sysadmin he doesn't want binary log files etc; but if someone else feels like they need systemD, maybe there should be a fork to make place for those people. It's a fairly kind, open attitude. Maybe we should have more of that around here.

    --
    "First they came for the slanderers and i said nothing."
    1. Re:Are you sure? by phantomfive · · Score: 5, Informative

      Is there scope for a less intrusive fork of SystemD?

      OpenRC + Init seems to be the commonly referenced replacement. UselessD seems to be a more reactionary replacement, which is also often mentioned. I still can't see what's wrong with init scripts :)

      Can one expect a re-write of all these daemons by a small team with no history of working on these applications to be anywhere near free of bugs?

      Not likely.

      --
      "First they came for the slanderers and i said nothing."
    2. Re:Are you sure? by serviscope_minor · · Score: 5, Insightful

      I think the reality is that SystemD makes life easier for distro builders, not for users, and that is why it has won.

      I think this is the underlying cause as to why the old guard are upset, and what a lot of the lawn-invaders don't really understand. It's not really about systemd.

      Linux used to be our system. It was unashamedly by hackers for hackers. The user was king because the user was a hacker and Linux built by like minded users. If there was something that sucked to set up or sucked to use it wouldn't win out because why would anyone want to make a system worse for themselves. Furthermore the builders were derived from all walks of hackerdom. Some were distro builders, some web developers, some kernel hackers and so on and so forth.

      For systemd, I don't even know if there's much wrong with it. But it is indicative of a deeper rift. Linux systems are now primarily build by professional distro builders and they don't do much on Linux except build distros day to day. And the vast influx of corporate money means that it's getting harder and harder (though not impossible yet) to avoid its effects.

      The end result is that Linux is no longer the ultimate hacker system, made by techies for techies. It used to be uncompromisingly awesome by the standards of the time for such people.

      Now compromises have had to be made, and the old guard are feeling the effects of the change. This amazing system which once you could bend to your will in any way imaginable is beginning to approach the type of opaque black box that they fought so hard to escape.

      That's the problem. Systemd is just yet another instance where it bubbles to the surface.

      --
      SJW n. One who posts facts.
  3. Re:It's about control by caseih · · Score: 5, Insightful

    Today, the competent administrator can control startup, dependencies, etc on a granular basis. With systemd, that control has gone - somewhere else.

    How so? Systemd has removed my ability to start and stop services?

    How would a package mess with systemd's configuration? It's readily apparently no clue about systemd. Hint, it's no different than it was before. A package drops its own service definition file in a directory (sound familiar?). That's it. It's no different in this area than any other init system. If the file is bad, the service just won't start. Just as it was before. Runlevels or targets are defined the same way: with simple symlinks. Really in this aspect, systemd is no different than upstart or plain old system v init.

    This post is one example why the debate gets so heated. People like you post stuff that's only nearly half true, without knowing anything about systemd, except the name of one of the authors. FUD plain and simple. A technical debate is fine, but you've got to actually know what you're talking about before you start debating. So far I've seen zero technical debate on this site regarding systemd. Certainly no one is willing to own up to the flaws in traditional init that have led to systemd's development. It's extremely disheartening to see this kind of irrational fear instead of technical discussion.

  4. Administrators dislike constraint based systems by tlambert · · Score: 5, Informative

    Administrators dislike constraint based systems.

    This should surprise no one. One of the problems with a constraint based system is that you don't control the precise ordering of things.

    This doesn't bother the Debian folks, because their build system is a constraint based system. If they have a package to install which has dependencies, they don't control the actual build order of the dependencies, or of their dependencies, and so on. Turtles all the way down. You do an apt-get install foo, and it's going to try to build subcomponents when they become available to try to build. And if they fail to build, they don't care: they "try again later", in case something that happens later satisfies the dependency that wasn't satisfied the first time around.

    This is very disturbing to system administrators, who like things to be both orderly and predictable. All dependencies should be mapped out, known, and explicit. If something gets tried now, and fails, the correct response isn't "We'll try again later!", it's "Stop! Someone fix this fucking thing, it's obviously broken".

    The build system is not deterministic; if there are two components, and one has a subdependency on some X of "at least version N", and another has a subdependency on X of "at least version N+2", then depending on the vagaries of network overhead, it's possible that half your code gets built with version N and the other half gets built with N+2, and things break. Things breaking is in fact far more acceptable to a system administrator than "things act weird", and "things act weird" is at least deterministic for a given build instance, and far, far, more preferable than "things sometimes work and sometimes don't".

    So system administrators dislike Debian for large system installations. And they dislike systemd for starting things up and shutting things down.

    A desktop system is far, far more forgiving: "It's not working; I'll just reboot!". As long as things "mostly work", then things are great! "Look! It's as good as Windows!".

    Note that launchd in Mac OS X has many of the same problems as systemd; it's also a constraint based system. It's somewhat worse, in that it insists on controlling file descriptors and sockets and Mach ports for the things it starts - which means you have to rewrite a lot of at least the startup code in most Open Source software to tolerate being run by something that opens the files and sockets that it expects to do itself. But that's just a lot of make-work, and people who are paid to do work are paid because it's not something they'd be willing to do voluntarily, for free, and that's what they're exchanging for the money they are getting in exchange for putting up with that part of the job.

    Unlike the people making things work with launchd, though, the people expected to make things work with systemd aren't being paid. And so systemd represents make-work and change for chage's sake, which doesn't sit well with volunteers.

    --

    So yeah, a lot of people find systemd annoying. Kirk McKusick once accused "vnode" as being "the structure that's taken over the kernel"; in Linux, systemd is fast becoming "the program that's taken over user space".

    How this will all play out, I don't know, but don't expect it to be resolved any time soon, given the dichotomy between the philosophies of the stakeholders involved.

  5. And apps while we're at it by knorthern+knight · · Score: 5, Insightful

    It's not just the init, it's also the applications that are being infected with Lennart-ware, e.g. gnumeric. It's a great spreadsheet, but recently it's been picking up various egregious hard-coded dependancies that simply don't make sense. This occurs mostly via GTK, which seems to pull in a significant chunk of GNOME.

    I run a minimalist Gentoo desktop, and I notice when additional dependancies are dragged in. The past year or 2 has seen goffice, ghostscript, harfbuzz, dbus, and various other crap become hard-coded dependancies for gnumeric. It was not necessary a couple of years ago. If I had several million dollars, I'd hire a bunch of progragrammers to port gnumeric from being dependant on GTK to being dependant on FLTK (Fast Light ToolKit) http://www.fltk.org/ Some of the money would go to ongoing maintenance.

    Another few million dollars, and I'd like to hire a team to hack and slash away at Firefox. I was around when "Phoenix" was forked as a lightweight alternative to the Mozilla web-browser. I savoured that promise. That promise has been dashed into the ground, with a Firefox that's bigger, heavier, and slower than the original Mozilla ever was. Time for a new fork.

    I want GNU-Linu-x, not GNOME-Lenna-x

    --

    I'm not repeating myself
    I'm an X window user; I'm an ex-Windows user
  6. Re:Not true. There's a different division by Endymion · · Score: 5, Insightful

    Your'e close - the split is indeed between the older Unix types and people that just want to be "users", but you need to recalibrate where their relative positions. Those of us that are against being forced to use[1] systemd see this in a different light. As computers became inexpensive over the last decade, a new generation of younger people joined the Linux community. They were young an inexperience, and often made well-known mistakes in their software. Thats was ok - we were all n00bs at first, and many of us tried to gently nudge the inexperience developeers in useful directions. Very few listened, and have now decided that anything "old" is bad.

    Listening to those that came before you is important, if you want to avoid making the same mistakes. A lot of those lessons are collected under what many refer to as the "Unix Philosophy". Mostly, that "philosophy" is jsut a handful of tricks that make maintainance saner. A lot of the stuff that people claim is "overcomplicated", "messy" or an "archaic design" is such an "ugly" state for a reason, and those messy bits are bugfixes. The nice ideal design we all starty with rarely fits exactly when we introduce it to the problems and unforseen circumstances in the real world. That ugly spaghetti-code-style hack that seems to ignore and bypass the "correct" way? That is probably a bug fix, and by removing it you probably reintroduce the bug.

    You call us luddites, but heed our warning at your own peril. Some bugs and bad designs have happened before, and a key reason why we don't like systemd is that it makes one of the worst mistakes you can ever make when designing software: it throws out the supposedly "old" or "ugly" parts. I suggest readoing Joel Spolsky's famous essay on this topic:

    you can ask almost any programmer today about the code they are working on. "It's a big hairy mess," they will tell you. "I'd like nothing better than to throw it out and start over."

    Why is it a mess?

    "Well," they say, "look at this function. It is two pages long! None of this stuff belongs in there! I don't know what half of these API calls are for."
    [...]
    The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it. It doesn't acquire bugs just by sitting around on your hard drive. Au contraire, baby! Is software supposed to be like an old Dodge Dart, that rusts just sitting in the garage? Is software like a teddy bear that's kind of gross if it's not made out of all new material?

    Back to that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes.

    Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
    [...]
    When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.

    Systemd is still at the early stage, where it can get away with this kind of bad design, but as more and more people start to use it and the never-ending list of Real World Problems starts to creep in, the systemd developers - and the distros that joined them - are goign to have one nasty mess on their ihands. It is going to be a nightmare to all of the bugfixes and real-world mess that was thrown away because it was "old".

    We tried to warn them, and were labeled luddites.

    Well, as B5's Londo Mollari put it:

    "Ah, arrogance and stupidity a

    --
    Ce n'est pas une signature automatique.