Slashdot Mirror
Does Systemd Make Linux Complex, Error-Prone, and Unstable? (ungleich.ch)
"Systemd developers split the community over a tiny detail that decreases stability significantly and increases complexity for not much real value." So argues Nico Schottelius, talking about his experiences as the CEO of a Swiss company providing VM hosting, datacenters, and high-speed fiber internet. Long-time Slashdot reader walterbyrd quotes Nico's essay:
While I am writing here in flowery words, the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis. This is even without calculating the security risks that come with systemd. Our objective is to create a great, easy-to-use platform for VM hosting, not to walk a tightrope...
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
[W]hat the Devuan developers are doing is creating stability. Think about it not in a few repeating systemd bugs or about the insecurity caused by a huge, monolithic piece of software running with root privileges. Why do people favor Linux on servers over Windows? It is very easy: people don't use Windows, because it is too complex, too error prone and not suitable as a stable basis. Read it again. This is exactly what systemd introduces into Linux: error prone complexity and instability. With systemd the main advantage to using Linux is obsolete.
The essay argues that while Devuan foisted another choice into the community, "it is not their fault. Creating Devuan is simply a counteraction to ensure Linux stays stable. which is of high importance for a lot of people."
Oh my! I'm going to hang back, make some popcorn, step into some Tyvek coveralls, grab a hard-hat and safety goggles and enjoy the show!
If you want news from today, you have to come back tomorrow.
The BSDâ(TM)s and Illumos. There is no reason to use the tire fire that is Linux. You have options!
Here's a list of actual problems that should have been solved instead of introducing the nightmare of systemd upon the Linux (Debian specifically) world:
- Forceful, unconditional kernel operations. When I say "unmount this filesystem," I'm not asking a question. When I say "terminate this process," I expect the process to be removed from memory and the runqueue, regardless of consequences.
- When I say "reboot" I mean "reboot." Hangs are not okay, ever.
- Actual, real soft NFS failures. Do not hang during boot for any reason unless that share is marked hard,nointr. Do not hang during shutdown/reboot, either.
- Enforce GPL-standard syntax on new incoming utilities. If you want into the package tree, use a GNU parsing library and use it correctly. Perhaps a standardized syntax wrapper available for package maintainers.
- Bolt simple parallelization, triggers and flow control onto init/rc.
- Drop this selinux shit. It's too complicated and causes more problems than it solves. Vulnerabilities come from bad code, not a lack of complex call ACLs. Security is a process, not a feature.
- Standardize and fix bluetooth support ffs.
My $0.02, as a 25-year Linux admin.
A government is a body of people notably ungoverned - AC
A big ol ball? My init.d was about 13 scripts big which were readable and editable. Ever tried to edit systemd files? Depending on systemd version you have to create overrides, modify symlinks or edit systemd files straight up which can be in about 5 different locations and on top of that, systemd can have overrides on any changes either with an update or just inherited.
Systemd makes every system into a dependency mess.
Remove/fail a hard drive and your system will boot into single user mode, not even remote access will be available so you better be near the machine just because it was in fstab and apparently everything in fstab is a hard dependency on systemd.
Custom electronics and digital signage for your business: www.evcircuits.com
How are they going to get in and stay in if admins have real logs and can detect persistence?
Domestic spying is now "Benign Information Gathering"
So the short answer is: Yes, systemd makes things unnecessarily complex with little benefit.
That matches my experience - losing a lot of time trying to figure out why things don't work. The improved boot time is lost several times over.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
1. You've moved having a basic understanding of the boot process, and the ability to fix things, from having a decent knowledge of bash to being a C wizard.
2. You've broken decades of understanding the boot process.
3. It breaks KISS, as it doesn't simply do startup. Hell, it does ntpd.
It breaks a lot of the *concept* of unix. Maybe to something preferred by a lot of people - but it also turns it into an alien mess to a lot of other people.
yet another rant with zero details.... "all the problems caused by systemd" and yet not a single one listed.
Use "journalctl" to see the details log.
It must have been something you assimilated. . . .
it turns out that, on arm embedded systems at the very least, where context-switching is a little slower and booting off of microsd cards results in amplification of any performance-related issues associated with drive reads/writes when compared to an SSD or HDD, sysvinit easily outperforms systemd for boot times.
Do one thing, and do it well. Systemd has eaten init, udev, inetd, syslog and soon dhcpd. Yes, that is getting ridiculous.
Systemd creates a dependency mess which means it cannot be replaced by simpler things, which wasn't the case before systemd.
I don't think there's a problem with the idea of systemd. Having a standard way to handle process start-up, dependencies, failures, recovery, "contracts", etc... isn't a bad, or unique, thing -- Solaris has Service Manager, for example. I think there's just too many things unnecessarily built into systemd rather than it utilizing external, usually, already existing utilities. Does systemd really need, for example, NFS, DNS, NTP services built-in? Why can't it run as PID 2 and leave PID1 for init to simply reap orphaned processes? Would make it easier to restart or handle a failed systemd w/o rebooting the entire system (or so I've read).
In short, systemd has too many things stuffed into its kitchen sink -- if you want that, use Emacs :-)
[ Note, I'm a fan and long-time user of Emacs, so the joke's in good fun. ]
It must have been something you assimilated. . . .
This is Slashdot, any ill you can imagine can be traced back to systemd.
“Common sense is not so common.” — Voltaire
Betteridge's Law says no.
- Drop this selinux shit. It's too complicated and causes more problems than it solves. Vulnerabilities come from bad code, not a lack of complex call ACLs. Security is a process, not a feature.
If you want to disable SELinux then disable SELinux, but not writing "bad code" isn't an option when even OpenSSL get major holes.
As long as people want new features there will either be new security vulnerabilities or software you can't afford and never gets completed. If SELinux adds enough security to be worth your bother then go for it, if not then disable it.
I stole this Sig
code contribution: /code/systemd
rm -rf
So the short answer is: Yes, systemd makes things unnecessarily complex with little benefit.
That matches my experience - losing a lot of time trying to figure out why things don't work. The improved boot time is lost several times over.
I completely agree. Troubleshooting is really a bitch with systemd, much more time-consuming. For instance, often systemctl reports a daemon as failed while it's not, or suddenly decides that it didn't start because of some mysterious arbitrary timeout while the daemon just needs some time to run a maintenance tasks at startup time. And getting anything of value out of the log is a pain in the ass.
Quite often I end up writing control shell scripts specifically to be called by systemd, because this junkware is too fragile and capricious to work with actual daemons. That says a lot about the overal usefulness of systemd.
Nothing has been gained with systemd, at least not on servers.
lucm, indeed.
Now that you converted all your start scripts etc... to systemd, to you really want to go back to init?
Slashdot, fix the reply notifications... You won't get away with it...
Troubleshooting is really a bitch with systemd, much more time-consuming. For instance, often systemctl reports a daemon as failed while it's not, or suddenly decides that it didn't start because of some mysterious arbitrary timeout while the daemon just needs some time to run a maintenance tasks at startup time.
Not to mention that the damn logs are not plain text, which in itself complicates things before you even have the chance to start troubleshooting.
#DeleteChrome
Drop this selinux shit. It's too complicated and causes more problems than it solves.
I think the utility called audit2allow summarizes well the immense "value" of selinux.
generate SELinux policy allow/dontaudit rules from logs of denied operations
https://manpages.debian.org/un...
The first time I heard about it I thought it was a prank.
lucm, indeed.
I tried your patch and it's amazing. Please submit a pull request upstream as soon a possible.
it's a fine OS, the only thing it's missing is a really good init system.
I disagree on SELinux, not because its interface is well-designed (it is not), but because it is needed for some things.
On the rest, I fully agree. And instead, systemd solves things that were already solved and does it badly. The amount of stupidity in that decision is staggering.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
In my experience managing systemd unit files is GREAT!
What are you using them for? Are you a sysadmin? A debian init script writer? An embedded systems builder?
I wish half the effort that went into b!tching and moaning would go into a decent alternative but compatible alternative/fork to systemd (ie. works with same unit files etc).
The reason there isn't a compatible alternative is because the code is too complex.
"First they came for the slanderers and i said nothing."
I've been Unix admin in various environments for 20+ years. My first Linux install was in the early 90's with a pile of 3 1/2" floppy disks that I downloaded from Usenet. I think the first kernel I ever got compiled and working from scratch was 0.87. I've learned, and forgotten, more than I care to remember about Solaris, AIX, HPUX, IRUX & Linux.
I no longer care to admin anything but my own few systems as I've developed other interests and career paths.
I just got done, this very evening, installing Devuan on the laptop I'm using to make this post. The installation process was trivial. I've had Devuan running on other laptops and virtual machines for about a year now. I couldn't be happier, and I'll never go back to a systemd corrupted distro. I just want my stuff to work, and keep working, and not require hours to fix when something does go wrong.
systemd: may you rot in hell.
"Every time I see an adult on a bicycle, I no longer despair for the future of the human race." - H. G. Wells
Yeah, yeah I know the history of its development and how log files are binary and the whole debug kernel flag fiasco. And I don't care. By the time I used systemd, that had already long passed.
I switched from Squeeze to Jessie a couple years ago, had some growing pains as I learned how to use systemd... but that was it. No stability issues, no bugs. Can't say whether things run better, but they definitely don't run worse.
I had only really been using Linux for a few years before the onset of systemd, and honestly I think that's part of the problem. People who complain about systemd the most seem to have been using Linux for a very long time and just "don't want to change". Whether its nostalgia or sunk-cost fallacy, I can't say, but beyond that it seems much more like a philosophical difference than a practical one. It just reminds me of people's refusal to use the metric system, for no better reason than they are unfamiliar with it.
If systemd is so terrible, then why did a lot of the major distros switch over? If they didn't, it would just be a footnote in the history of open source: "Hey remember when they tried to replace sysV and init with that stupid thing with the binary log files? What was it called? SystemP?" The fact that Devaun has not overtaken Debian in any real way (at least from what I've seen, feel free to correct me if I'm wrong) indicates that my experience with systemd is the norm, not the exception. The market has spoken.
I read TFA, there is not one single specific bug or instability mentioned about systemd. What is the "tiny detail" that split the community? I have no idea, because TFA doesn't say what it is. I know that part of the philosophy behind Linux is "figuring it out yourself", but if you don't explain to me these low level kernel details (if that's even what they are; again, I have no idea), then don't expect people like me to be on your side. Linux is just a tool to me, I don't have any emotional attachment to it, so if things are working OK I am not going to start poking around under the hood just because someone posts an article claiming there are problems, but never specifying what those problems are and how they affect me as a user.
Honestly TFA reads like "We are having development problems, therefore systemd sucks." I get that when major changes to the platform happens there are going to be issues and annoyances, but that's the way software development has always been and will always be. Even if systemd was perfect there would still be all kinds of compatibility issues and new conventions that developers would have to adapt to. That's what I would expect to happen whenever any major change is made to a widely used and versatile platform like Linux.
Even Linus doesn't really care:
"I don't actually have any particularly strong opinions on systemd itself. I've had issues with some of the core developers that I think are much too cavalier about bugs and compatibility, and I think some of the design details are insane (I dislike the binary logs, for example), but those are details, not big issues."
I'm not saying systemd is "better" or "the right answer". If you want to stick to distros that don't use it, that's up to you. But what I am saying is, get over it.
Only crack the nuts that crack. You don't put the ones that don't crack in the sack.
- Fix the logging.
Seriously. It's nearly impossible to troubleshoot issues if messages are just swallowed and not either output to the screen (which systemd has broken completely) or to the journal.
I started out using Slackware over twenty years ago. Biggest reason I left it was the libc5/glibc2 debacle. It became nonviable for me to remain on Slackware at that time. Bounced around through several package-managed distros and eventually ended up on Debian. I've now watched Debian become increasingly complex, sometimes needlessly, and sometimes because different major components have features that other major components lack. This means even if one is running something like xfce for the windowmanager one has to have a lot of Gnome or KDE stuff installed for basic stuff to work.
It's gotten worse since Systemd entered the picture. Honestly pulseaudio is still not mature, not sure how the person who didn't get that working right was entrusted to replace init.
Do not look into laser with remaining eye.
"but servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration made it too expensive to run Debian or Ubuntu."
PRO TIP: Run Slackware. Slackware is cleaner and does everything Debian does and has never been tainted with systemd.
Your thin skin doesn't make me a troll
Linux (and before that: UNIX) has always had a "look how clever I am, writing all this obscure code" mentality. Since not too long after its inception, complexity - as a way of displaying the developers' prowess - has always been favoured over simplicity and elegance.
Whether you look at systemd, or the print subsystem or emacs or sendmail. They are all over-complex and if not intended to freeze-out users without the time, inclination or ability to grok them, then to achieve this through bad design which leads to complicated implementations.
Good design is difficult. Too hard for most coders. And it does seem that with kernel development and the systems that surround it, most of the design decisions are simply left up to the people writing the code to make, themselves. While this is standard procedure for a teenager sitting in a darkened bedroom, knocking out .... code, it is strictly amateur-hour stuff. You would have hoped that the linux community would have moved past that in these last 30 years.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
I have a PC with a Bios that tries to do everything, launching a bootloader that tries to do everything, to start a DesktopManager that treis to do everything, so I can run a browser that tries to do everything to see a website that tries to do everything.
And to think I started with Linux, because each thing had its own program and I could select what program did it.
Don't fight for your country, if your country does not fight for you.
Betteridge's law of headlines meets slashdot's anti-systemd bias. Could be interesting
It's always amusing reading comments like this. I remember hearing this kind of thing when it came to software distribution.
"Installing Microsoft Office is so easy. I install it all the time with no issues. People are just so whiny. Why don't they just grow up and get used to it?"
Now try installing it across a few thousand, or maybe 50,000, PCs on a staggered schedule across a business where the OS installation is supposedly standardized. There's a world out there that you don't understand, and it has issues that you can't even comprehend. Their concerns are not your concerns.
If anything, it shows that the linux base is so big that major distros are now unable or unwilling to cater to the needs of professional system administrators.
Most consumers couldn't care less how stuff works under the hood. It's bizarre that that may be a reality in the Linux world, but here we are.
apt purge systemd
add http://angband.pl/debian/ to /etc/apt/sources.list before doing that and it will actually succeed. okok it's a bit more complex than that, but you can read the instructions online which are neeearly as simple :)
Windows is a very complex system. Not necessarily because it needs to be complex, but rather because of "wouldn't it be great if we could also..." thinking. Take the registry. Good idea in its core, a centralized repository for all configuration files. Great. But wouldn't it be nice if we could also store some states in there? And we could put the device database in there, too. And how about the security settings? And ...
And eventually you had the mess you have now, where we're again putting configuration files into the %appdata% directory. But when we have configuration in there already anyway, couldn't we... and we could sync this for roaming, ya know...
Which is the second MS disease. How many users actually need roaming? 2, maybe 3 out of 10? The rest is working on a stationary desktop, never moving, never roaming. But they have to have this feature, needed or not. And if you take a look through the services, you'll notice that a lot of services that you simply know you don't need MUST run because the OS needs them for some freakish reason. Because of "wouldn't it be great if this service did also...".
systemd now brought this to the Linux world. Yes, it can do a lot. But unfortunately it does so, whether you need it or not. And it requires you to take these "features" into account when configuring it, even if you have exactly zero use for them and wouldn't potentially not even know just wtf they're supposed to do.
systemd is as overengineered as many Windows components. And thus of course as error prone. And while it can make things more manageable for huge systems, everything becomes more convoluted and complicated for anyone that has no use for these "wouldn't it be great if it also..." features.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
fwiw Slackware has improved their package management quite a bit, might be worth checking it out again.
"First they came for the slanderers and i said nothing."
Can I ask, why don't you and other admins/devs like you start to contribute to systemd?
Lennart Poettering has specifically said that he will not accept many important kinds of patches, for example he refuses to merge any patch that improves cross-platform compatibility.
And what's the reason, because people on forums are complaining? Because binary log files break the UNIX philosophy?
Here is my analysis of systemd, spread across multiple posts (links towards the bottom). It's poorly written software (the interfaces are bad, you can read through my links for more explanation), and that will only get worse over time if an effort isn't made to isolate it over time. This is basic system architecture.
"First they came for the slanderers and i said nothing."
Journald makes logging simple and convenient, right?
journald has been known to run out of memory and stop responding.
Due to the design of "oh just connect stdout of the process to journald" if you restart journald it closes all of those file descriptors and you silently lose all further logging from already running processes.
Journald, by design, will only log so much per process, meaning that if it's logged too much since startup/an error you're interested in, you've now lost it.
Why 'they' had to go for demonstrably broken binary logging using a new interface I don't know. They could have just extended the syslog format to make it mandatory to pass along program name and process ID in the message. Then they could have made it "easier" to find the logs by having a per program/facility directory under /var/log and then stuck to simple, plain text logging that the existing *nix tools can search with ease. But, nope, they had to go with this shitfest instead.
And that's only one component of the whole systemd shitfest. On a very simple Debian install I've had it exhibit issues with shutdown, hanging on something that is simple or ignorable.
Sadly I had to abandon using Devuan after a while. The only really supported version is the jessie (Debian old-stable) version, and I'm not sure even that gets timely security updates. Their equivalent of Debian stable (Stretch) is 'ascii' and got next to no updates during the few months I used it. Boot up was both nice and fast (a major systemd selling point) and reliable (unlike systemd). I guess they just need more in the way of human resources so that they can nail down which Debian packages have problematic systemd tentacles involved, then they could pass through other Debian updates as soon as they're available.
>do not tinker under the hood, don't use Linux professionally, and don't run servers.
I do all of these things, I still don't whine hysterically about systemd.
>To me, the fact that the major distros have adopted systemd is strong evidence that it is probably better
"Better" is a subjective term. Software (and any product really) does not have some absolute measurable utility. It's utility is specific to an audience. The fact that the major distros switch is probably strong evidence that systemd is "better" for distro developers. But the utility it brings them may not apply to all users, or even any particular user.
A big part of the reason people were upset was exactly that - the key reasons distros had for switching was benefits to people building distros which subsequent users would never experience. These should not have trumped the user experience.
All that would still have been fine - we could easily have ended up with a world that had systemd for those who wanted it, and didn't have it for those who didn't want it. Linux systems are supposed to be flexible enough that you can set them up to whatever purpose you desire.
So where the real anger came in was the systemd's obsessive feature-creep made it go into a lots and lots of areas that have nothing to do with it's supposed purpose (boot process management), in that area it's biggest advantages are only useful to people building distributions (who have to maintain thousands of packages and ensure they reliable handle their bootup requirements regardless of what combination of them is actually installed- systemd genuinely did make that easier on them - but no user or admin ever experiences that scenario). But that feature creep itself wasn't even the issue, the issue was that - as it entered into all these unrelated areas (login was the first of many) - it broke compatibility with the existing software to do those jobs. This meant that, if you built a system to support systemd, that same system could not use any alternatives. So now, you had to create hard dependencies on systemd to support it at all - for distros to gain those benefits, they had to remove the capacity for anybody to forgo them, or alternatively provide two versions of every package - even ones that never touch the boot process and get no benefit from systemd's changes there.
And the trouble is - in none of those other areas has it offered anything of significant value to anybody. Logind doesn't actually do anything that good old login didn't do anyway, but it's incompatible so a distro that compiles it's packages around logind can't work with anything else. Replacing the process handler... and not only did it not add any new functionality it broke some existing functionality (granted, in rarer edge cases -but there was no reason for any breakage at all because these were long-solved problems).
Many years ago, I worked as a unix admin for a company that developed for lots of different target unix systems. As such I had to maintain test environments running all the targets. I had several linux systems running about 5 different distros, I had solaris boxes with every version from 8 onwards (yep, actual Sparcs), I had IBM's running AIX, I even had two original (by then 30 year old) DEC Alphas running Tru64... and I had several HPUX boxes.
At the time, while adminning all these disparate unix environments on a day-to-day basis and learning all their various issues and problems - I came to announce routinely that Solaris pre-Version-10 had the worst init system in the word to admin, but the worst Unix in the world was definitely HPUX because HPUX was the only Unix where I could not, with absolute certainty, know that if I kill -9 a process - that process would definitely be gone. WIped out of memory and the process table with absolutely no regard for anything else - it's a nuclear option, and it's supposed to work that way - because sometimes that what you need to keep things running.
SystemD brought to Linux an init system that replicated everything I used to hate about the Solaris 8/9 init system - but what's worse than that, it brought the one breakage that got me to declare HPUX the absolute worst unix system
Unicode killed the ASCII-art *
To me, the fact that the major distros have adopted systemd is strong evidence that it is probably better.
Raises the question, better for whom? Systemd seems to make some things easier for distro maintainers, at the cost of fucking shit up for users and admins.
That said, Debian's vote on the matter was essentially 50:50, and they're going to keep supporting SysV init. Most distros are descendants of Debian, so there's that. Redhat switched for obvious reasons (having the main systemd developer on their payroll and massively profiting from increased support demands).
With Debian and Redhat removed, what remains on the list of major distros?
Yeah.. strong evidence...
CLI paste? paste.pr0.tips!
In the UNIX server world, hardware usually doesn't change during runtime (init 3 or init 5). Thus a boot process that starts new processes in a pre-determined, finely tuned sequence until all services are running, makes sense. All dependencies are already solved before the system boots up (and if not, you change your boot sequence until they are). And in this case, shell scripts as glue between the processes make thoroughly sense, as they provide a deterministic, linear approach. For the few cases when hardware changes during runtime (USB drives etc.pp.), you have a wrapper that handles it, but it's not something init is concerned with.
But if you have a system where the hardware changes all the time during runtime, because mice get plugged in and graphic tablets removed, monitors are sometimes on, sometimes not, you have to support projectors for some time, you have to support several different "power down" states (S1, S2, S3...), computers get put into docking stations and removed etc.pp., init is just clumsy. There are so many drivers to be loaded and unloaded, services to be started, to be stopped, to be removed from memory, you have so many dependencies to be solved on the fly, that your shell scripts take ages to tune, and the boot sequence is depending on the hardware currently plugged in. You would need dozens of init states to make sense of it all, each one with the correct set of started and stopped services, and the changes between states happen often.
This is not the usual server environment (yet), but it is the daily life of Linux running on other devices.
The reason there isn't a compatible alternative is because the code is too complex.
That makes no sense at all. How could the complexity of systemd's code have any effect on the difficulty of writing a compatible system?
Watch this Heartland Institute video
if I have a problem with e.g. my dovecot instance on a server, with rsyslogd (as default installed on Debian) I get the fun of guessing which of mail.log, mail.info, or mail.err contains the messages I might like to see (with the mild suspicion that I ought to also glance at debug.log as well, just in case), then if I like to see things in chronological order I have the added amusment of running a command line like this:
zcat $(ls -tr /var/log/mail.log.*.gz) | cat /var/log/mail.log - | grep dovecot | grep $whatever_I_really_wanted_to_see
and I'll get most of what I'm looking for, along with anything else that contains the word dovecot.
[BTW hands up anyone that thinks a gzip file is a text file]
whereas with systemd it's just so bloody tedious:
journalctl -u dovecot | grep $whatever_I_really_wanted_to_see
Where's the fun in that?
Debian: GNU/Linux done the Linux way
... is he's an arrogant fool with an overrated sense of his own abilities who will not listen to advice from people who know a lot more and have a lot more experience of using linux/unix in a critical enviroment than he does.
Why red hat and other distros are in thrall to his buffoon and his 2nd rate software is a mystery that I suspect even Mulder and Scully would find a challenge to solve.
As usual lack of knowledge is your real problem. ntp is an *optional* component of systemd. If you don't like theirs use your own. You don't have to know *any* C to use systemd. I don't even believe that you think you do. If you spent even a minimal time reading the documentation your problems would all magically disappear, but you don't want that. Complaining in ridiculous ways about non-issues is much more fun, and you can get lots of +1 mods from everyone else who learns everything they know about it from Slashdot and never invested any time learning about it.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Granted, I have never needed any kind of tampering or corruption mitigation in my log files over the last 20 years of Linux administration. So the value for at least my usage of journalctl has been sum negative because I don't see the value in a command that by default truncates log output.
So the answer for systemd is to workaround it by using a "legacy" service to restore decades of functionality.
SMF was the death knell for Solaris (along with the Oracle purchase), and it feels like systemd is going to be the anchor which drags Linux into the abyss.
--WooooHoooo--
The problem is that systemd is full of bugs. When the boot process hangs, automounts fail, or shutdown gets stuck waiting on nfs (saying it will time out but the time out target keeps moving), troubleshooting requires knowing C. Those problems can't be fixed by config files and documentation. They are bugs in the C code which is far more complex than a boot system should be.
... their only arguments are hyperbolic personal attacks. (Look, I can do that too!)
Hint: Your side is just as stupid as your opposing site. There is no sane or reasonable, let alone sensible side. Because that is how Americans are. At least it is beyond their *tiny* mental box.
Regarding systemd, I state *both* A and B:
A) Monolithic "frameworks" have always been a stupid idea. Because they disable you from plugging them into *your* system, and force you to plug into *theirs*. Because they want to dominate you! And they are mutually exclusive as a result of that.
B) Traditional init systems are very limited and badly limiting nowadays. Like still using DOS as the underpinnings of your actual system. A more generic event/trigger system is much more sensible.
THE PROBLEM IS: That systemd throws away what's good about traditional init systems (like "everything is a file"; modularity; being able to do things with a simple file manager, text editor and maybe a script.).
It could have done the event/trigger thing *without* sacrificing modularity (tools that do *one* thing, and do it right!).
It could have acted less like a dominatrix on a power trip, swallowing everything.
The base ideas were good. The personality of the way it was implemented, was that of a complete egocentric psychopathic asshole with a god complex.
Give me a sane eventd, and I will ditch the old init system before you can blink.
I am an old fart. My first Unix machine was a VME 68xx running Unix Version 7 around 1986. I am mostly a developper, but I've been doing sysadmin work as an aside (unavoidable in small companies) more or less continuously for the last 30 years.
Recently, on upgrading my Debian home server (can't remember if it was Wheezy->Jessie or Jessie->Stretch), the server did not come back on the network after the upgrade. Go down to the garage where it lives: single user mode. No explanation nothing. After wasting 2 hours trying to guess what was happening, the explanation was that there was a stray entry in fstab. Nothing related to the real important stuff (/ or /usr), something like /proc/bus/usb or such. Systemd just decided that single user was the right thing to do. No ssh, no nothing. If the server had been remote, this would have been a major issue, instead of a couple of uncomfortable hours (restarting from backups would have been possible but would have changed a quasi-routine thing into one or more days of work).
I can't remember a machine being so nasty to me since the 90s (Unixware maybe :) )
I'm curious if Redhat are regretting their decision on the early hard switch to systemd in RHEL7. I know for a fact their support system was flooded with issues from early adopters. I have friends in the industry still on RHEL6 as the upgrade to version 7 is a logistical nightmare and one who works at a reasonably large enterprise considering ditching Redhat entirely to go to a systemd-less alternative.
That faster boot time sure helps with servers that are only restarted once a year ...
Tough question. Depends what that functionality is. Compatibility is valuable but sometimes it must be sacrificed to deal with technical debt or make genuine progress. Even Microsoft had a huge compatibility break with Vista which was needed at the time (even if Vista itself was atrocious). /etc is a major deal - it utterly breaks with a standard around which disk space allocation is done professionally. /use ought to not even need backups because everything there is supposed to be installed and never hand edited. It means modifying backup strategy which is a big, very risky, cange. Logs aren't where I expect them. Boot errors flash on screen and disappear before you can read them so you have to remember to go look in the binary log to figure out if it was something serious.
It would depend what those features were, what benefits it gave me. It would be a trade off and should be evaluated as such. A major sacrifice requires an even more major advantage to be worthwhile. I've yet to see any such advantage from anything systemd has added. I'm not saying advantages don't exist, I'm saying whatever they may be they do not benefit me, personally, in any measurable way. The disadvantages however do, and compatibility is the least of them.
Config outside
I was never a fan of system V. It was a complicated, slow, mess if code duplication. It needed a replacement. I was championing Richard Gooch's make-init circa 2001 (and his devfs, the forerunner to udev, was in my kernels - I built a powerful hardware autoconfig system on it in 2005 when I built the first installable live CD distribution, the way they all work now: I invented it [I later discovered that pclinuxos had invented the same thing independently at the same time but Ubuntu for example still came on two disks, a live CD and separate text based installation disk and more than once I had machines where the live cd ran great but the installed system broke due to disparate hardware setup systems]). Later I praised upstart - it was a fantastic unit system that solved the issues with system V, retained compatibility but was easy to admin, standards and philosophy compliant and fast. It was even parallel.
That is the system that should have won the unit wars. I'm not a huge fan of Ubuntu's eclectic side, unity has always been a fugly unusable mess of a desktop to me - but upstart was great, that and PPAs are Ubuntu two most amazing accomplishments. Sadly one got lost instead of being the world changing tech it deserved to be and it lost to a wholly inferior technology for no sane reason.
It's the Amiga of the Linux world.
Unicode killed the ASCII-art *
I'm inclined to agree, BUT... I actually quite like system unit (files). It's a great way to daemonise a very simple program (perhaps one you got handed by your devs who know nothing about sysadmin, or perhaps some crap downloaded from the Internet because someone thought it would be useful). Getting simple stuff to work with systemd is actually super-easy. All those symlinks are really just enable/disable, although I'd love to see the actual files in some obvious directory - I'm not sure how much of that is systemd or distro vendor, but what the hell is multi-user-wants or whatever it's called? Either way, stopping every last daemon having to have its own watchdog process and letting "the system" take care of all that sort of thing seems like a really sensible move to me.
I have had horrendous trouble trying to get some esoteric daemons to work with systemd though. Trying to make an old init.d script work in systemd is a world of pain. Trying to have half a dozen 'linked' systemd units to fire up a half dozen daemons in the right order is really really horrible. It becomes easier to rewrite the launcher script to work on STDOUT and then run that with systemd. That's a problem because now you're not really using systemd, and so all of systemd is really just an overhead to what you're doing. Resolving this is partly the job of daemon writers, but partly the job of systemd itself (and could be solved there by allowing one system unit start multiple processes, conditionally run programs before startup, etc). Daemon writers aren't going to get onboard with systemd until it's easier for them to use it than to ignore it, so something's got to happen here (IMHO).
IMHO also, binary logs are crap - that's probably the single worst design item of it. We've all got along with plain text for years, we've got our log-shipping infrastructure and whatever else, and we've got a handful of aliases or scripts to do the sorts of cut-and-splice jobs we want. Having to use a different toolset for one or two log files is just a pain in the neck, and I have to go read the manpage or google every time I want to look at the bloomin' log because I've forgotten the command line options because I don't use them often enough.
If I had my way, I'd separate the binary logging stuff from systemd, and make it an optional system 'enhancement' for things that want to use it. In 5+ years of making cool tools for it (perhaps even implementing a log-shipping/aggregating solution), then start pushing that as the "next big thing" - until then, make it optional. Making binary logs a mandatory requirement for systemd just makes the "dislike surface" of systemd that much bigger and isn't a necessary dependency.
Whatever form of logging there is, systemd makes debugging really hard. It doesn't seem to suck up STDOUT/STDERR when you really need it to, it doesn't seem to tell you the command it ran that it thought had failed when you want it to, it doesn't give you the response code, it doesn't tell you why it considered it failed, etc etc. I can't even begin to think of incremental ways to change what it currently does because if I'm honest, I don't really understand it well enough. Bottom line: this is the worse experience of it, and probably the reason daemon writers will continue to resist systemd as much as they can. If systemd doesn't get this sort of thing fixed, then it's a huge barrier to entry, and means the likes of slashdot comments will continue to be negative.
In short: I think it came from a place of good sense, but then got too embroiled in things it shouldn't have. If it's really the "perfect awesome" that we all need, then we should have been given incremental steps to get there, rather than getting thrown in a the deep end with some software which has a few kinks in the pipes still.
I'd say it is worse having to type something different for one log on the system, when the other 100+ are plain text and so accessible with the old tools we've all learned backwards. It means you don't have the necessary switches or key presses to hand because you don't do it often enough.
"journalctl" might be the best thing since sliced bread, but making it a hard requirement of systemd makes adoption of systemd that much harder. IMHO, systemd should "pick it's battles" and concentrate on managing system processes and worry about log formats another time.
Or in other words, a simple, reliable and clear solution was replaced with a gigantic KISS violation. No engineer worth the name will ever do that. And if it needs doing, any good engineer will make damned sure there is an easy and clean way back. The systemd people seem to be hell-bent on making it as hard as possible to not use their monster. That alone is a good reason to stay away from it.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Indeed. And on top of that they try to make it as hard as possible to do without systemd. This really seems to be a case of "if you cannot compete on merit, force users into it", which is completely unacceptable.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
There is a long history of Poettering and Sivers marking even clear security problems as "will not fix". Look for them yourself, they are not hard to find.
However your stance implies that you are just trying to sabotage the discussion. Your blue-eyed innocence is an obvious lie. Despicable.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
The point being that text format is more universally readable, and also should it get corrupted, it has a better shot of still being readable.
On the other hand, pure binary logging was not necessary to achieve what they wanted. In fact, strictly speaking a split format of fixed-size, well aligned binary metadata alongside a text record of the variable length data would have been even *better* performance and still be readable.
XML is like violence. If it doesn't solve the problem, use more.
I don't agree. Systemd is the most visible part of a clear trend within Red Hat, consisting in an attempt to make their particular version of Linux THE canonical Linux, to the point that, if you are not using Red Hat, or some derived distribution, things will not work. In essence, Red Hat is attempting to out-MS MS by polluting and warping Linux needlessly but surely. The latest: they have come up with the 'timedatectl' command, which does exactly the same as 'date'. The latter is to be deprecated. Red Hat, the MS wannabee. They will not pull it off, but they are inflicting a lot of damage on Linux in the process.
Using text processing skills to process a generic text file isn't any harder than using journalctl. The difference is that the former is generically applicable to just about any other software on the planet, and the latter is for journald. It's not that complex to confer the journalctl benefits without ditching *native* text log capability, but they refuse to do so.
Using ForwardToSyslog just means there's an unnecessary middle-man, meaning both services must be functional to complete logging. The problem is the time when you actually want logs is the time when there's something going wrong. A few weeks ago was trying to support someone who did something pretty catastrophic to his system. One of the side effects was that it broke the syslog forwarding (syslog would still work, but nothing from journald would get to it). The other thing that happened would be for the system to lock out all access. I thought 'ok, I'll reboot and use jornalctl', but wait, on CentOS 7, journald defaults to not persisting journald across boot, because you have syslog to do that.
Of course the other problem (not entirely systemd project fault) was the quest to 'simplify' console output so he just saw 'fail' instead of the much more useful error messages that would formerly spam the console on experiencing the sort of problem he hit (because it would be terrible to have an 'ugly' console...). This hints about another source of the systemd controversy, that it's also symbolic of a lot of other design choices that have come out of the distros.
XML is like violence. If it doesn't solve the problem, use more.
I really struggle to reconcile the Slashdot view that systemd is total crap and the fact that every major Linux distro has switched to it.
The Linux ecosystem is not sane. Redhat wanted more control of Linux so they pushed systemd. GNOME developers are easily distracted by shiny things (as proof I submit GNOME 3) so they went ahead and made GNOME dependent on it. And then Debian (which most Linux distributions are based upon) adopted systemd because GNOME depended on it. There were some other excuses, but that's the biggest reason. You can blame Redhat and Debian for this clusterfuck, and really, only a small handful of people in the Debian community are actually responsible for Debian's involvement. Debian's leaders were split almost down the middle on whether they should go to systemd. This is why major changes should require a 2/3 vote (or more!)
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
.
I am now actively looking for a distribution that does not use systemd.
Sometimes I have to wonder if the wrong feature goal (faster start-up times) was over-emphasized in the whiplash switch to systemd by the distributions. For me, the faster start up is just not that big of a deal anymore, now that I've moved to SSD.
Remind me again how useful journalctl and binary logs are when you can't remember the exact name of the unit? "tail -f /var/log/messages | grep dhcp" is a lot easier to remember than "journactl -f -u isc-dhcp-server" - and hopefully you ARE running isc-dhcp-server, because if it's a different server you're SOL.
Remind me again how useful journalctl and binary logs are when the only things that run on a system are "echo" and maybe "/bin/cat" if you're lucky?
And yes, I've had that happen. Problems with the init system usually result in systems that have minimal functionality available. SystemD has far too many dependencies to reliably reconstruct a system that has failed init.
I mean a sizeable company that has it's own Linux distro, and makes money supporting that distro.
Red Hat seems to be the enterprise Linux company.
I suppose IBM, and Oracle, somewhat support Linux. But for those companies, Linux is just a sideline.
I bet if there were a company like Red Hat, that had a non-systemd alternative, the Red Hat competitor would win.
Your example is off base.
1. I seldom search my gzipped backlogs, because if something went wrong, it was recent. That is why they are gzipped.
2. If I am searching for a specific error message, like "$whatever_I_really_wanted_to_see", then generally no other program will spew that and the extra "grep dovecot" is unnecessary.
Generally this is all you need:
grep $whatever_I_really_wanted_to_see mail*
Which is even simpler than the journalctl bullshit.
Second, the example that you give above is not that difficult for an admin because zcat, cat, grep ARE ALL STANDARD TOOLS that I use for the output of virtually every program and service that I use. I do not want to have to use a different command (journalctl) to look at the output of every different program (systemd) that I run.
Yes, when you focus on what you intend to happen, issues seem so clear, don't they? But all this lack of responsiveness to your demands arises from attempts to contain potential unintended consequences.
I have a pet peeve, which is people who rank "crash" or "hang" bugs at the apex of the failure scale, above data corruption and faulty outputs. Think of an automated trading system that enters an abnormal state and starts generating random buy and sell orders for example. Or a system which is supposed to authorize access sensitive data. This is why operating systems are designed to crash when unexpected states are encountered. It's also why they're designed to restrict or alter powerful operations like unmounting a drive.
Now I'm with you -- a system should be responsive to what the operator demands, even if it is a potentially bad idea. I think computers (smartphones especially) should have genuine power buttons that cut off the power supply from the system. But just because getting the computer to do what I *intend* is simpler doesn't mean that the operation itself is simple; the operator has to be prepared for the complex consequences of his action.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
The problem is that systemd is full of bugs. When the boot process hangs, automounts fail, or shutdown gets stuck waiting on nfs (saying it will time out but the time out target keeps moving), troubleshooting requires knowing C. Those problems can't be fixed by config files and documentation. They are bugs in the C code which is far more complex than a boot system should be.
Exactly! This is the MASSIVE point that pro-systemd-ers completely fail to address.
If any portion of my sysvinit system fails to process... at the console I can ctrl-c, carry on, and figure the issue out normally.
Perfect example, case in point: I recently upgraded a box to Debian Jessie and forgot to remove systemd before rebooting. I lost remote access to the box because for whatever reason, systemd was waiting for dhcp on an interface that was supposed to be set up as static, and was frozen in the process.... Inconveniently despite having console access, I couldn't ctrl-c, ctrl-d, or anything. Completely unresponsive while waiting for dhcp. This is unforgivable of an init system. Yes, I am a C coder, so I can very well find the bit that is waiting for dhcp and add a SIGINT handler, but why would I? SystemD is a steaming pile that no one wanted, and that solves problems that no one had, and is the solution that no one wants. It's been pushed as the 'next-best-thing' and clueless people have went with it.
The arbitrary file locations, random symlink requirements, and overall complexity is NOT what is needed in an init system. This throws away DECADES of acquired knowledge and startup knowhow that WORKS (99% of the time).. and when it doesn't... it's SIMPLE to debug. in sysv (or just about any other init system under the sun) I don't have to read a tome of documentation or fire up a C development studio to figure out why my dependencies aren't coming up.
The goal of computer science is to build something that will last at least until we've finished building it.
It breaks a lot of the *concept* of unix.
And therein lies the problem. After all, Linux is based on Minix and Minix is "a POSIX-compliant (since version 2.0), Unix-like computer operating system".
POSIX, of course, is a standard derived from the various versions of Unix that existed when the standard was first thought of; and Unix systems are characterized by a modular design that is sometimes called the "Unix philosophy".
The original draft of the Unix philosophy stated:
Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.
It was later revised to:
Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.
And, of course, systemd isn't compatible with either specification, which means it would (and should) be rejected by the Unix world and, thus, by Minix, which follows the Unix philosophy; as such, it should be rejected by Linux, which is based on Minix and also strives to follow the Unix philosophy.
Why is that so hard for systemd proponents to understand? Just make a fork of Linux that doesn't purport to be Unix-like and limit systemd's reach to that fork and everyone will be happy. We chose Linux because we wanted a Unix-like environment and your systemd violates that choice.
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
Absolutely. There are some things that are definitely GOOD about systemd. The extensibility/overloading of the service/unit files is a good example of something that works well and is implemented in a way that makes a lot of sense. For example, you can have a service file at /usr/lib/systemd/system/somesystem.service /etc/systemd/system/somesystem.service.d/*.conf
And then modify functionality with units under
It's easy to do, and works nicely with packaging systems so that you can create an addon package to modify or add behavior without editing the file(s) supplied by the original package. The way you can build dependency chains is also quite useful.
There's also some stuff that is lame, like the binary logs and the needed to run journalctl or systemctl to figure out WTF your daemon is doing when it fails, or how the binary log can be corrupted so that you can *never* figure out what happened in some situations.
The biggest problem is the lack of compromise. A lot of people in SystemD-land have a "my way or the highway" attitude, whereas a lot of people in init-land have a "change is bad" mentality.