Linux Mint Will Continue To Provide Both Systemd and Upstart

jones_supa writes: After Debian adopted systemd, many other Linux distributions based on that operating system made the switch as well. Ubuntu has already rolled out systemd in 15.04, but Linux Mint is providing dual options for users. The Ubuntu transition was surprisingly painless, and no one really put up a fight, but the Linux Mint team chose the middle ground. The Mint developers consider that the project needs to still wait for systemd to become more stable and mature, before it will be the default and only option.

Comments

The pain isn't in the switch

[Anonymous Coward]

it's in trying to resolve problems later on, when you'll find that systemd helps you obscure the source of the trouble instead of resolve it.

Re:The pain isn't in the switch

[Antique Geekmeister]

It's also in maintaining distinct management tools, configuration analysis, and configuration tools for DHCP, NTP, the new binary logging, space for logging of kernel booting, and whatever other components of the one man band have been added lately. systemd, and especially its core developer Lennart Pottering, are attempting to create a "stateless Linux", and the ramifications are spilling over into other parts of the system.

Quoting from the b log at http://0pointer.net/blog/proje...

> A Stateless System goes one step further: a system like this never stores /etc or /var on persistent storage, but always comes up with pristine vendor state.

That is a _huge_ shift in system management, and directly violates the file system hierarchy where host specific configurations are stored in /etc or persistent data in /var/lib. They're basically taking all the daemon specific parameters from /etc and putting them in /run, and they're going to run into most of the same problems but in unfamiliar layouts. Every component that stores history in /var/lib or configuration in /etc will have to be rewritten, including long-standing conventions such as /etc/hosts, /etc/resolv.conf, and /etc/nsswitch.conf.

Re:The pain isn't in the switch

[Anonymous Coward]

As someone who has spent the last week plus upgrading his server to Jessie, I completely disagree that the pain isn't in the switch. It may very well be that more pain is coming but the switch itself has been quite painful and isn't over yet. How the fuck do you do a meaningful Google search on "systemd complains that starting Apache fails and shows the service as stopped but the process actually starts and Apache is working just fine, systemd apparently just doesn't know it?"

Re:The pain isn't in the switch

[gbjbaanb]

a system like this never stores /etc or /var on persistent storage

including /var/log?? Well, I suppose that's one way to hide the fact that systemd's logs get corrupted.

Re:The pain isn't in the switch

[kthreadd]

Sounds like Apache is forking and running as a daemon, and you haven't told systemd to expect that behaviour.

Re:The pain isn't in the switch

[Kazymyr]

Well for one thing by obfuscating system logs in a way that is likely (and expected) to fail.

Re:The pain isn't in the switch

[hitmark]

The only behavior systemd "expects" is for daemons to talk to it either via dbus or libsystemd.

http://ewontfix.com/15/

Everything else have some kind of problem attach that is just waiting for the admin to tun his back on the rack.

Re:The pain isn't in the switch

[Bengie]

Bug report: Logs keep getting corrupted and cannot read them at all
Rejected with reason: Delete the corrupted logs and move on

SystemD - works well when it works, fails spectacularly when it fails.

Re:The pain isn't in the switch

[Junta]

SystemD - works well when it works, fails spectacularly when it fails.

Incidentally that is precisely the way Windows is. Windows is exceedingly structured and engineered (contrary to some beliefs), but as a a result when it fails... boy does it go down so hard that no one is going to bring it back. Not surprising many of the principles in Windows design match the design principles of many modern linux distros: Binary configuration files, binary log files, increasingly complex IPC schemes, and less and less friendly to simple scripts (though increasingly better for complex scripting).

Re:The pain isn't in the switch

[Anonymous Coward]

bug report: /var/log/* corrupted, all text files replaced with zeroes from hard disk failure
Rejected with reason: Delete the corrupted logs and move on

For fucks sake, grow up.... ALL files types are liable to corruption and depending on the extent of corruption, you ain't gonna be able to read them, text or otherwise.

The operative words here are "depending on the extent of corruption".

You can read text logs with a sector editor if there's any possibility at all of pulling raw data off the hardware. For lesser damage - well, Linux is awash in text utility programs and if all else fails, put it on Windows and use the Wordpad program to read them.

Binary logs are a different matter. To read them at all, you need a matching decoding program. And the damage to the logfiles must be within the limits of the decoding program's ability to skip around damage. You're probably not going to get anything usable from a raw sector dump.

Re:The pain isn't in the switch

[Junta]

dconf uses binary configuration files. As I've said many times, while systemd catches a lot of flak for messing with long standing conventions, it is far from alone in modifying the experience (dbus, dconf, systemd, udev all do interesting things, each component bringing interesting capability with varying degrees of drawbacks).

I think udev generally does a great job of delivering useful capability with minimal downside. Then dconf (most people don't even realize the binary dconf files exist, even when they poke dconf extensively). dbus starts going off the rails (many things that once were simple enough to explore/grep around for are now only possible through internet search of obscure dbus-send commands). systemd I actually consider less bad than having to do more and more dbus stuff.

Re:The pain isn't in the switch

[Junta]

I have contended with corrputed files plenty. If they are plain text, it's highly unlikely that a glitch leaves it such that a human can't piece together what is left. If it relies heavily upon common features of binary formats (compression, alignment to very particular addresses, section headers), it's quite easily unrecoverable. Basically the exact things that make them attractive (performance, efficiency, analysis) make them lose all meaning pretty quickly if part is missing or something.

Re:The pain isn't in the switch

[kthreadd]

Ubuntu 14.04 uses Upstart.

Re:The pain isn't in the switch

[ookaze]

I have contended with corrputed files plenty. If they are plain text, it's highly unlikely that a glitch leaves it such that a human can't piece together what is left. If it relies heavily upon common features of binary formats (compression, alignment to very particular addresses, section headers), it's quite easily unrecoverable. Basically the exact things that make them attractive (performance, efficiency, analysis) make them lose all meaning pretty quickly if part is missing or something.

But that's the problem: people who constantly rage against systemd and its binary logfiles are not AT ALL interested in knowing if they can read it, they never tried, they never looked at explanations of how it works or anything.
The fact is that you can actually use "strings" (the shell command) against your journal file and get A LOT of text logs with a lot more information than in a standard log file. Of course you lose all the semantic like the integrity of the text you're reading.
Which is because journald won't even compress small logs because it would take too much space. Only big log lines or blobs are compressed usually.
All this talk about systemd binary logs being unreadable is just plain lies and BS from detractors.
Of course, if you read your log with journalctl, it will get you all the extra benefits.

Re:The pain isn't in the switch

[squiggleslash]

IIRC these "binary" logs you speak of are simply text files with indexing information attached. They're not encrypted or compressed, you can, at worst, use the "strings" command to pull the information, and if you're doing raw sector dumps from a hard drive then you'll be able to read them there too.

Of course, if the log files are zero'd out, like the bug report you quote, then it doesn't matter whether they're "binary" or text, you're screwed. That has nothing to do with systemd.

Re:Year of the Linux

[jeremyp]

Linux is merely the kernel used by systemd. The correct name of the operating system is systemd/GNU/Linux.

Yeah mint!

[BlackPignouf]

One more reason to use Mint.
That's the best they could do out of this situation.

Re:Yeah mint!

[Anonymous Coward]

Precisely. I can't get that everyone and his dog is rolling over and playing dead with this. They *ALL* remember what came of all of Lennart's OTHER projects. Stuff looking for problems to realistically solve. Stuff that's still not QUITE "right" and the man, rather than finishing the job, moves on to the next disaster. This is no different. This has no place in a "mission critical" space- and yet they're letting them do it. I question the sanity of Red Hat in bankrolling this travesty and then jamming it down our throats in the FIRST damn place. Just like with PulseAudio. No compelling anything, really, and the proponents keep thinking that network service on this little userland mixer feature (which should've been a service of the DRIVER layer for one piece, the mixer) is the greatest thing- when it's part of the latency problems with sound. First hint it's stupid...Android doesn't use it.

Re:Yeah mint!

[Feral Nerd]

One more reason to use Mint. That's the best they could do out of this situation.

The thing about staying neutral in a war of religion is that you run the risk of being branded a heretic by both sides.

Re:Yeah mint!

[phantomfive]

The reason so many distros have adopted systemd is because it fills a need: it does something useful for them. This topic has been exposited. In the second link, note how much shorter the unit file is than the init script. That is why distro maintainers have adopted systemd, because they have to write less code.

So Lennart found something people needed, and built it. That's why people adopted it. You don't have to like systemd, but you should at least understand why it's been adopted.

Are the "GNU" and "Linux" parts still needed?

[Anonymous Coward]

I've been learning more about systemd lately, and I'm liking what I'm seeing. It's cohesive, yet it's completely modular. It's modern. Its binary log files make debugging easier. It's getting more and more critical functionality every day, and this makes my Linux system easier to use each time I update it.

The deeper I dig into systemd, and the more I use it, the more I question why we even need the GNU utilities and the Linux kernel.

I hope to see the most useful parts of the GNU toolchain and the most useful parts of the Linux kernel folded into systemd eventually. This way I don't have to install a Linux distro. I can just install systemd itself, and it will give me everything I need.

If my computer boots directly to systemd, and systemd provides the command line tools and the UI, then my boot time will be shaved down to almost nothing, and I'll get to use some really nice and modular command line tools. I won't have to try to remember awkward shell, sed, grep, and awk commands and their flags, because if systemd provided alternatives, I know they would be easier to use.

I've found that systemd is all about empowering me, as a user. It's about letting me get more done with less effort. GNU and Linux aren't always about that. They're about doing things for philosophical reasons, or for scratching an itch of their developers. Systemd is all about getting work done, which is why I've found it to be so useful. If it can do the things that the GNU toolchain and the Linux kernel are doing, but it can do them better, then I'd prefer just to use systemd directly.

Re:Are the "GNU" and "Linux" parts still needed?

[Anonymous Coward]

Take note modern shitposters. This is what a good troll looks like.

If everyone who wants to derail a discussion would put in as much effort as this the Internet would be saved.

Re:But...

[gbjbaanb]

To be fair, Linux has always been multiple components that you can chose which one suits you best - whether its vi or emacs, gnome or kde, sendmail or postfix, apache or nginx, etc

This is a good thing, where you can swap out component A for B for any reason, and keeps the project competing with each other to get better and better.

If only you could swap out Systemd so easily, things would be great.

Re:Systemd detractors are like climate change deni

[sjames]

Actually, it seems quite the opposite. We have the systemd crowd claiming that it's simpler even when there is a whole new level of complexity in systemd they don't even know about (hint, look for the systemd craziness in /lib). Like the climate change deniers, they believe that since they haven't personally seen a problem in their simple and vanilla system that there isn't a problem at all.

Lennart Poettering is cancer on the face of Linux

[Anonymous Coward]

systemd is an abomination.

Linux's benefit for a long time now has been the ability to swap functionality out you don't like for other functionality that you do. I don't know who in their right mind thought it was okay to move critical system infrastructure systems (init, time, logging, etc) into the hands of an untested piece of software which clearly has very deep issues, written mostly by a developer who has a track record of making extraordinarily poor choices in software development and writing poor code.

What's stunning to me is that smart people in leadership positions in a variety of distributions have decided to support such an asinine system. What's even more stunning is that Lennart didn't stop them, realizing his code was not appropriate in production environments and doing the responsible thing, which is to say "We're far from ready". systemd's position as PID 1 on Linux systems creates an enormous SPOF given the complexity of the code. The only sane position systemd developers can take is "we're not ready, please don't use this even as a test in your released distributions".

For all practical purposes, the rapid and unseemly adoption of systemd means that many enterprises running distributions that now rely upon systemd have to make the decision to not trust their distribution any more if they consider their systems mission critical. This is going to make people move to FreeBSD, Oracle, Windows, non-systemd distributions, microservice/microkernels, etc in rapid fashion. It is going to literally kill Linux for the people who have not yet figured out how to deal with the loss of machines (the majority of the enterprise world). And that may be a good thing, in the sense that Linux has in many ways become indistinguishable and directionless in the sea of operating system options. It tries to be far too many things to far too many people.

Lennart likes to whine about how much the community hates him and how much people take it out on him personally. Reality is that outside of a few people who take it much too far, the reason people don't like Lennart is because he makes poor decisions with poor forethought and his advocacy and arrogance for his own approach have a seriously negative effect on a great many people, enterprises, and efforts. He likes to think he is a genius and we simply don't understand his vision, but for a great many people, his vision is the antithesis of what we like about Linux and Unix. Systemd developers don't understand the arguments of simplicity, composability, and small programs that do one thing well.

The truth is the fault doesn't rely totally upon Lennart and his team: Some of the blame can also be assigned to Linus for poor stewardship, but Linus has a set of complex motives and organizations that influence him. Linus should have killed this stuff much earlier.

I think in a few years, we'll realize what a mistake we made in giving Mr. Poettering any chance of credibility in operating system software development.
I hope it comes sooner rather than later.

Re:But...

[gl4ss]

because some software expects it and doesn't run without it.
shit software, but software anyways.

that's whats bad about it, really. it's just not a startup replacement but one that aims to turn developers into developing software that can't work without it,, without trouble.

why would startup utility provide user authentication? to conquer everything of course.

I like how this got marked troll

[drinkypoo]

It's a fact that the fix for corrupt logs, which systemd will often corrupt if you power-cycle your system, is to delete them and throw them away. https://lwn.net/Articles/621453/ https://lists.fedoraproject.org/pipermail/devel/2013-July/185702.html It's a fact that systemd will only sometimes recover any part of its bullshit binary logs, and only any part after any error. So if journald truncates a file because it shits itself, which it has been known to do, then you lose the whole log.

Moderators, you might want to familiarize yourself with the festering pile of shit which is Lennart Poettering, as well as the pile of crap which is systemd, before you moderate any systemd discussions. Because you clearly have no idea what you're doing.

Re:I like how this got marked troll

[phantomfive]

Moderators, you might want to familiarize yourself with the festering pile of shit which is Lennart Poettering, as well as the pile of crap which is systemd, before you moderate any systemd discussions.

Please, tell us how you really feel! It's not clear to me from this sentence.

Re:I like how this got marked troll

[drinkypoo]

Sincerely, all this hate towards systemd looks like pure zealotry.

Yeah, that's what they said about our love for Linux in the first place. Now that it's successful, and specific people are helping specific corporations gain undue control of it and at the expense of quality, they say it again about our desire to protect what made us care about it in the first place. We love Unixlikes which are actually "like Unix" ("The Unix Way", blah blah blah) not solely for aesthetic reasons, but for specific technical reasons which systemd compromises.

True, like any piece of software, systemd surely must have issues (binary logs seem like one) that should be fixed or parts that may be improved, but all this constant bashing from some members against it, is just purely irrational.

It's constantly being bashed by "some members" of the community because it's still a problem. It's not like we pointed out the problems and they were fixed, or the community rejected it. We pointed out real problems which were summarily ignored, and which are now becoming real problems for more people. This is not production-quality software. I am not in theory against everything they are trying to do, but I am absolutely against the way they are trying to do it, and who is trying to do it as he has proven both his incompetence and unwillingness to consider user feedback in the past.

In any case, normal users and readers of /. are sick and tired of your constant rantings about systemd.

Riiiiight, that's why we keep having these massive arguments about systemd, because you are all so tired of it. Just like you don't need the FCC to take things off the air because you can change the channel, you don't need us to stop discussing systemd and why it's literally both bad and wrong. You can just skip the discussion.

Re:I like how this got marked troll

[Damouze]

The issue is not with systemd corrupting the binary logs, or with the filesystem corrupting the binary logs, but with the fact that they are -binary- logs. A log file should be an ordinary text file. Nothing more, nothing less.

To say nothing about the very concept of systemd being a BAD IDEA. One daemon to rule them all and in the darkness bind them? Give me a break. It used to be that Linux was a fairly faithful UNIX-like operating system. And you know what makes a good UNIX or UNIX-like operating system? The tools, specifically those that are well-written and do one thing and do it well (with reasonable exceptions). Systemd is neither well-written nor does it do one single thing very well, well maybe corrupt your log files and crash very, very spectacularly.

By the way, if you're so worried about your filesystem getting corrupted, why use ext4 at all? It contains serious design and implementation flaws, specifically concerning the journal. XFS is by far a better journalled filesystem, with none of those issues. It is simply the best choice, especially if your concerned with the integrity of your data.

Re:I like how this got marked troll

[Peter H.S.]

The issue is not with systemd corrupting the binary logs, or with the filesystem corrupting the binary logs, but with the fact that they are -binary- logs. A log file should be an ordinary text file. Nothing more, nothing less.

The glaring problems with flat file, unstructured text logs have been discussed for decades now. Plain text log files, like the original syslog(3) interface was simply bad ideas that Linux inherited. The main commercial selling point with any syslog implementation today is exactly that you can avoid several of these problems by using a DB.

An overhaul of the creaking legacy Linux logging system have been needed for a decade at least. Remember, the Rsyslog project was started exactly to fix such problems. They failed partly because distros and developers don't care about working together for a common goal and because there was no other central developer hub who would take the initiative.

systemd have finally fixed so many of the classic syslog problems. I don't care if you think syslog is perfect and never can be improved, and Rsyslog should be avoided for providing binary DB storing option, but please don't think that other people agree to this postulate.

And systemd's journald is designed to be 100% backwards compatible with syslog, so if your setup have legacy need, it is trivial to get a systemd distro to use legacy syslog text logs.

Thank you, Mint

[The Infamous Grimace]

Seriously. Thank you for giving me a choice. If I want to try systemd and see if I run into issues with my particular use-cases, I can. If I want to avoid the possibility of conflicts and continue with the (admittedly crufty) sysvinit scripts I can.

The mint team is doing a right thing.

[cryptogranny]

The mint team is doing a right thing. I'm writing this from a CentOS 7 machine that I'm using to test systemd. One day, when the XFS partition was corrupted I realized that the system can't do usual self check and self repair. Why? Because systemd.fsck module runs fsck.xfs which according to the manual "simply exits with a zero exit status". You can boot with init=/bin/bash, but can't correct system scripts to change the behavior, everything is in binaries now.

Re:Lennart Poettering is cancer on the face of Lin

[jbolden]

Let me just point out, Oracle Linux 7 is systemd only. Oracle switched Jul 23, 2014. So yet another enterprise vendor that doesn't buy into the whole "systemd isn't mission critical ready" meme you all are pushing.

Logs via network

[bouldin]

SysD's binary logs have another, serious flaw: they are not designed to be sent over a network. This has been an intrinsic part of syslog for a looong time.

There are a few tools to send journald logs remotely, but they rely on tailing the binary log, then reformatting it and transmitting it over HTTPS to another tool on the destination system.

I found a feature request for journald/sysd/whatever to enable network logging, and the solution they are adding is to... tail the binary logs with basically the same tools.

Putting the disk in the middle and depending on file tailing is such a bad solution. Many things can cause this to fail; it's a total kludge.

I discovered this when investigating how to send journald logs to Splunk. The solution there is to... tail the binary logs with journald to a text file and have Splunk monitor the text file.

Now, this is not a fatal flaw. Journald could (I assume) be enhanced to natively send logs over a network. But, this shows that systemd is not enterprise-grade or production-grade, and was not designed with that kind of reliability in mind.

Re:Logs via network

[drinkypoo]

The real, fundamental problem isn't even having binary logging, they are free to do that in whatever miserably thought out bugfest they like. The problem is that they thought binary logging should be the primary means, and that is just stupid. I am all for a binary log... right next to the good old text log like we've always had. And if this was what they had done, then they would have got no argument on this issue. Instead, they threw away what we had, what was working, what all our tools were expecting, and replaced it with something that works sometimes.