So now you are going from saying that people do not use DB's for logging unless they are clueless; (Let me quote you on this:)
And yes, I have extensive experience with log analysis. You do not turn the logs into a database unless you are fully clueless.
..to that DB's can be a good thing? And please remember that ACID doesn't give any protection against file corruption, nor does it prevent user space programs corrupting the logs by generating obviously impossible field values. But then again, neither does the journal nor plain text logs.
Again, for those who need to run a full DB for their logs, systemd's binary file format is great, because it allows the journal to be exported in industry standard formats like JSON. That way the remote log-sink database can receive and store rich meta data in a totally stable and structured way; changing hostnames, IP's, or even different wordings or even different and changing languages used by the daemon log output isn't a problem with the journal, since it is based on field values, not complex regex-ing of unstructured, undocumented, unstable, language specific words.
Oh, and tamper proof, cryptographically "sealed" logs too (FFS) if you want that.
If your remote log-sink solution isn't a full DB, you still gets all the benefits of receiving structured log entries with eg. full microsecond precision timestamps _and_ monotonic timestamps. It is trivial to convert eg. JSON output with defined field names to any other structured format, while converting and aggregating unstructured text files is a pain.
It is also great for those who only needs local logging; the journal has many of the advantages of a full DB, but without the complexity and overhead. Its append based file format is also much more robust against file level corruption than databases. Since the log files are structured and indexed with field values, they are easy to perform powerful, yet simple queries on.
How do you find all syslog entries with the priority level "error" generated by the previous boot only? With the journal it is : "journalctl -b -1 -p err"
And how do you generate a full list of every executable, including their path?
Since you can "tab" trough the values in the journal this can easily be done: "journalctl -F _EXE "
And with the -x switch, the help database is activated, giving further explanation on what the log entry means, and gives direct links to upstream support, perhaps linking directly to a page that explains the error code etc:
mar 07 16:46:16 localhost systemd-logind[546]: New session 1 of user Peter H.S. (log entry above, help database output below:) -- Subject: A new session 1 has been created for user Peter H.S -- Defined-By: systemd -- Support: http://lists.freedesktop.org/m... -- Documentation: http://www.freedesktop.org/wik...
There are seriously many people who just trawls through long log files (even using vi apparently), because it is hard to grep for anything unless you already know what to grep for, and because regex'es are well, regex'es; difficult to use, understand and to remember all the many variations. Basically, newbies can't read or filter Linux syslog log-files by other means than trawling through them.
They can't have a useful GUI either because it is impossible to make a distro-agnostic syslog gui. Again, a problem that systemd's journal solves.
The bottom line is, that the only virtue syslog files have, namely that they are human readable, is a serious hindrance for their use too: they can't add monotonic timestamps and micro-precision timestamps and other meta-data without being excessively difficult to read for humans.
With the journal you can give machine parsers exactly the structured log info they need and can benefit
I am not twisting your words. Let me quote you verbatim:
That does not make systemd better. It makes it worse as it is so complex is not had dependencies on tools. Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.
Isn't it straight up obvious that you claim that external contributors can be fended off because they can't afford licenses? Isn't this exactly what you are claiming?
Since this claim is totally untrue, isn't it then obvious that you are utterly confused about how Coverty and Jenkins work, and have no clue whatsoever on how systemd development is done?
No, I'm not misunderstanding. You will need a repair option to make the logs readable.
Oh yes you were, let me quote you verbatim on this:
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy.
You thought that apparently journald had a fsck option, it doesn't, which just goes to show how little you know about systemd.
As a sys admin I'm not interested in what it is doing.
As a Sysadmin you should know how your logging system works and the limitations it has. This is just basic SA stuff. In this case you should be able to discern about malformed log entries and actual file corruption of the logs. This is apparently something that is really hard for you to understand.
Neither you or the tin pot systemd crew have any idea what logging actually means.
Yeah right, like you were the smart one; I am sure you are a hot shot Linux developer with a CS degree that has to fend off job offers every week from leading firms, because that pretty much describe the leading systemd developers, that includes among many, Greg KH, the kernel developer who maintains the stable Linux kernels etc. probably number two after Linus T. in the kernel group, or Poettering who has over a decades Linux developing experience etc.
I could go on, but the CV's of the systemd developers are really impressive, and many of them work for leading Linux distros like Red Hat, Canonical, Suse, Debian etc. ; if these Linux distros know nothing about logging, please write a peer reviewed paper that sets them all straight about this.
Good luck with your BSD adventure, be sure to tell them that BSD is all about choice, so that they should support several init-systems simultaneously and by the way, break up their source repos in many smaller independent groups or else they are "bloated" and "monolithic".
And don't cry when your BSD fork makes a systemd clone and throws away their old obsolete legacy script based init-system, because this is exactly what is going to happen, and yes, BSD's will get binary logging too, it is only a matter of time. All these changes will be "forced down your throat" no matter how much you whine about.
BSD is for people who hate Linux, so you will feel right welcome there. It is just a matter of time before you will kowtow to the party-line about how GPL is bad because it doesn't allow BSD sponsors to close source the code etc.
Yes you are confused about several things, including licensing payment. Your claim that people are discouraged from contributed because Coverty/Jenkins may discover that their patch have security issues or breaks the build is laughable; of course such patches should be rewritten, and the submitter would be glad they such problems was discovered. Asking the submitter to rewrite the patch, or that the developer with commit access fix it, is everyday work in FOSS land. It happens all the time and it discourages nobody; this is something the submitter learns from. Really, should the alternative be to accept every broken patch just so not hurt the submitters feeling? I don't think anybody wants that.
Regarding your claim that the systemd developers don't play nice to others and suffer from "God complex" (talk about hyperboles), then this is plain wrong:
You can find lots of statements from people actually contributing, even small patches, that the systemd developers where really friendly and helpful. The proof is also in the numbers; there hundreds of such minor contributors to systemd, something that strongly indicate a good and including developer community. Compare that to the almost non-existing non-systemd developer eco systems.
Sure, the systemd developers sometimes says no to certain patches or features, but again, this isn't having a "God complex" but about project leadership and the technical know-how to reject things that are bad for the project. This happens in all FOSS projects.
Ah, so you where trying to use zfs on Fedora, despite that this isn't supported. Problems may occur in such cases but the problems have nothing to do with systemd since systemd works fine with zfs.
Seriously? Try again. I am not confused at all. But you are blind to what is happening.
But you really seems to be very confused since you erroneously talk about systemd contributors needing to pay licensing fees: "Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.
That is just totally misunderstanding how things works, so it really seems like you are very confused about even basic facts how systemd is developed.
Disagree or not; logging directly to a full DB is very common, which is why it is a selling point for every commercial syslog implementation I know of.
Regarding the indexing and structuring of log files, this is exactly what journald does. This dramatically reduces work needed to analyse them, even if you re-index them and convert them another structured file format like JSON. Working with indexed and structured files with defined field values, is always better than working with unstructured, un-indexed text files.
systemd's journal log files will always have an advantage over the poorly defined syslogd text logs, no matter how you spin it.
What a load of claptrap. People log to log files for reasons of the lowest common denominator. We have things called 'databases' for this kind of stuff and there's perfectly good reasons we don't use them for logging which should be obvious to anyone with an ounce of sense.
Really, are you unaware how common it is to aggregate log files in databases? This is a major selling point for Rsyslog that it is able to do so. In fact, Rainer started Rsyslog exactly in order to overcome the many deficiencies with syslog(3), including the severe limitations of unstructured text files.
systemd's journal is pretty much a stroke of genius in that it overcomes all the limitations of unstructured, unindexed text logs, while not being a full blown DB either (the journal files are basically appended text files with a different line delimiter and an index in front).
Log files keep on growing every year, because people log more and more, and systems are running more and more daemons. Analysing such logs means machine parsing, and structured and indexed log formats like the journal have a huge advantage for this kind of work.
Since the journald binary logging file format is stable and fully documented....
I sincerely hope Red Hat and Lennart is paying for this piece of contrived PR.
The arguments against it are based on contrived scenarios
Corrupted logs on a perfectly running system isn't a contrived scenario.
It is when the corruption means squat for the ability to actually read the logs.
For some reason you seem to think that "corruption" discovered by "journalctl" means the logs are unreadable, but as explained, they are often marked "corrupted" just because a single field value in a single log entry was discovered as impossible. This doesn't mean that the log file can't be read.
....through the Linux/Unix concept of piping.
Piping eh? Wow. You give the impression you haven't heard of it before;-).
I don't care what you think, as long as you agree that the whole notion of not being able to use standard Linux text tools like grep together with systemd's journal is just plain wrong and a total non-issue.
There just aren't any good arguments against structured, indexed log files that can be programmatically accessed and has rich meta-data.
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy. I don't know where editing files was mentioned.
You misunderstand; there _isn't_ a "fsck" option for systemd journals and there won't be because that would be equivalent of "editing the log files".
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
You simply don't understand how journald works. Let me explain again: systemd marks certain log entries as "corrupted" if eg. they are plain wrong, that is the client is sending obviously wrong or malformed log entries. That way you avoid impossible field values that may otherwise have screwed the log watch or log analysing programs; think of it as discarding bad data. This is the most common "journal" corruption that people discover when using the journalctl "verify" option. It is usually totally harmless and usually limited to a single log entry. It happens all the time with syslogd text logs too, people just don't discover them because syslogd doesn't have log file integrity checks.
When journald discover such malformed entries, it log rotates as a simple preventive security measure.
Sure, the journald log files are sensitive to real file corruption too (but so is syslogd log files), so clean unmounts of file-systems is a good idea (it always is). But the whole issue is massively overblown by people who doesn't use systemd anyway.
It would just be an optional KDE module; KDE already have a rather excellent GUI for configuring systemd (Kcmsystemd) that works as an optional control panel module.
Coverty is static code analysis that find potential stability and security problems. https://scan.coverity.com/ Think "Unix Lint".
Many FOSS projects like the Linux kernel and LibreOffice uses Coverty. It isn't a magic wand, but it plain works nevertheless. No systemd contributor needs a licence in order to contribute to systemd, even if they get commit access. Coverty can trivially be replaced or complemented by other static code analysers, or even be omitted; it is something helpful, not something systemd depend upon.
The Jenkins builder provides: "Continuous Integration with automated test execution", something that really improves code stability since it ensures that eg. a patch isn't committed if it breaks the build.
Again, no contributor needs any kinds of licence for this, and again, this is a nice to have thing, not something systemd depends upon.
You seem to be arguing against code discipline and the use of automated tools, just because systemd have them (while SysVinit doesn't). That is just bizarre.
Anyway, systemd's developer community is extremely vibrant and growing all the time; there are hundreds of people who have contributed already with new ones coming every month. So there doesn't seem to be anything that scares contributors away, unlike, sorry for putting the boot in, the tiny and deteriorating non-systemd developer eco-system.
The major point of systemd's binary logs are that they are both structured, indexed, and contain rich meta-data, something plain syslog logs isn't.
That again means that when you export your logs to a log-sink, you can do so in a format that fits directly into the log-sinks database like JSON format and preserve the rich meta-data like the "_MACHINE_ID" field, that uniquely are tying every log-entry to a certain machine. You can also be sure that that the program claiming to generate the entry actually is correct since journald provides kernel guarantee for this.
If you choose to keep the logs in journald's binary format on the log sink server, you gain the benefit of indexed fields when analysing data; a huge win when it comes to random access. So much faster than trawling through text logs (O(n) complexity and all that).
Every log entry can be traced back to a certain machine (not just hostname), many log fields have kernel guarantee for what they say, besides having integrity checking and even strong cryptographically "forward secure sealing" (FSS) security against tampering. The journald logs are simply designed from the ground up to be read an analysed on other computers than the one they were generated on, either in their native format or as exported logs.
Since the journald can be accessed programmatically, it can aggregate and analyse logs across different languages and has strong immunity against changing wording of the daemons log output (it operates on fields, not words).
And since the journal logs are structured identically by a documented standard, they are trivial to aggregate, unlike the output of many different syslog implementations.
Since the journald binary logging file format is stable and fully documented, and can be accessed programmatically with language bindings, you don't need a specific binary like "journalctl" to read them. In fact, there are there is a Rsyslog module that allows it to directly read (and export with metadata) systemd journal's. There are also Python modules etc. that acts as journal readers etc. In fact, it is quite possible to make a systemd journal reader that works on non-systemd platforms like MS-Windows or OSX.
The journald collate all logging on the Linux machine, that means everything can be trivially exported to a remote log-sink, including the kernel ring buffer and the binary utmp and wtmp log files that syslog doesn't know about, and it can include early boot and late shutdown log info because it can work in initramfs, something syslog can't.
How about journald being designed as signal-safe from the ground up (unlike syslog) and that it doesn't silently drop messages under load etc. etc.....
systemd's binary log format is simply a massive win for both the enterprise user and the Linux newbie and everything in between.
The arguments against it are based on contrived scenarios, like professional admins that doesn't have (systemd) boot medias and lack access to a pc, a usb stick and a internet connection so they can make one in 5 minutes.
All the standard Linux text tools like grep, tee, sed, sort etc. work with systemd's journal through the Linux/Unix concept of piping.
Sure, systemd's journal still doesn't have as many tool sets and projects around it as syslog. But that it is simply a matter of time.
There are several very good projects on the Wiki page. My favourites are probably:
Project: Port Amarok to Qt5/Kf5/Plasma5: Something I use every day.
Project: Port KSystemLog to use journald as a backend: With systemd it is actually possible to make a distro agnostic GUI log viewer that isn't just a "less" with windows decorations. I like using the CLI "journalctl", but a GUI, perhaps with some log watch support and real time panel notifications about "syslog level: Error" events and above would be nice.
Project: Implement PDF Poppler features: I like Okular very much, so more features like linearized pdf support would be nice.
I'm using it and learning with each thing that breaks. Most recent was zfs and there were a few before that. If it hasn't ever caused you any trouble I doubt you've been using it much.
zfs has been working for years now with systemd, despite it not being part of the Linux kernel and therefore not really supported by many major distros.
So if you are experience troubles, it is most likely distro specific troubles like using a distro that haven't converted to systemd yet (Debian) or using zfs on a distro that didn't officially support it in the first place (CentOS/Fedora) and perhaps because you haven't read the zfs support docs carefully enough.
The systemd bugs I have encountered have been really trivial and not pertaining to core functions; it has never let me down by not being able to stop or disable daemons, or having run away processes, or suddenly consume lots of memory or cpu time. It is far better than SysVinit and Upstart in that regard too.
No, systemd detractors really is a tiny minority; some ways this really shows is how there are almost no developers working to maintain even critical needed infra structure for non-systemd distros; ConsoleKit has been abandoned for years now, eudev is just a shadow fork of udev with no independent development going on, and several key components like systemd-shim and cgmanager are only kept alive by paid Canonical developers and a few Debian devs; once Ubuntu and Debian shifts to systemd, those projects will languish too. SysVinit will properly also deteriorate completely; Red Hat/Suse was the defacto upstream before, and now it is only Debian as long as it last. So the non-systemd infrastructure will probably deteriorate further as the commercial distros stops to maintain it.
In short, almost no developers are working on maintaining non-systemd infrastructure, this reflects how few the systemd-detractors really are.
The recent Debian debate also show how few the systemd detractors really are when the numbers are shown: The system-detractors made a lot of noise on the Debian mailing list, but after the technical committee had decided that systemd should be the new Debian Linux init system, the detractors were unable to even gather 5 (like in five) Debian developers out of around 1000 to sponsor a vote on this subject.
Even the GR bill trying to keep other init-systems equally supported was clobbered at the GR vote.
So going by the noise on the mailing lists, the systemd-detractors seemed like a force, but when voting they where nowhere to be seen.
Same with Linux distros; you would think that the non-systemd distro ranks would be swelling with the numbers of systemd-refugees. This certainly doesn't seem to be the case. A couple of rather obscure distros like Funtoo and Void are among the few distros that don't want to support systemd. Slackware is undecided on the issue, and Gentoo etc. support systemd, with a growing number of its users that prefer it to OpenRC.
Also no medium/major commercial non-systemd Linux distro have emerged this last couple of years, this is a strong indication that the paying costumers wants systemd, and doesn't care at all for the alleged superiority of SysVinit. Several companies made it clear during the Debian debate that they favoured systemd, none spoke for SysVinit or Upstart.
No wonder; systemd is great and it solves real world problems like daemon management and security much better than any other alternative.
I have been using systemd since its inception, and it has been rock solid. It is better tested and documented than most other FOSS projects; a Jenkins back-end and Coverty scans, integrated self tests, and rather strict coding practise really helps too in that regard. Take fx. SysVinit; their developers doesn't even have build test framework, so the only way to test whether a new patch doesn't break everything, is to make a live boot. No wonder they haven't made a release for many years despite the patches are accumulating.
Claiming that it isn't ready yet is just plain wrong and just seems to be some sorry excuse for not bothering to learn something new.
Thanks, that's interesting. I always thought one of the strengths of Linux was that the OS was highly separated from the GUI.
Don't believe him, what he says is demonstrably untrue. Both KDE and Gnome runs just fine on non-systemd distros and on BSD. This couldn't happen if they somehow where totally dependent on systemd.
The real story is that systemd provides some very needed OS infrastructure that Desktop Environments (DE's) like KDE and XFCE needs. The old non-systemd infrastructure that DE's could use instead, like ConsoleKit (CK), hasn't been maintained for years. The DE developers have warned the non-systemd distros for years that they needed to maintain CK, or that things would stop working, but they were ignored.
So Gnome, KDE, XFCE etc. still have support for non-systemd distros that are using eg., CK's login API, but things are beginning to fall apart since nobody have bothered to maintain that for years.
The point is that that Gnome and KDE played no role in why all major Linux distros have shifted to systemd. The simple fact is, that systemd simply is light-years ahead of any competing init-system; integrated groups support so each and every process can be controlled regarding resources, no more hard to write and debug shell scripts to start daemons (executable config files; who thought that was a good idea?), integrated kernel Capabilities(7) to provide defence in depth for all running daemons etc.
The TL;DR resolution: journalctl --fsck after rotation or pump it into a traditional syslog
Lennart Poettering's comments about it:
"our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse"
Totally sensible solution, why would you edit log files? They are not meant to be changed.
Besides that, the log entries that are mangled are in them self important info; How can you detect a pattern of a misbehaving program that sends wrong field values if you delete the entry every time it happens? Or what if the log corruption is caused by a failing disk where the bad blocks are spreading like wildfire? If you silently delete such entries and files, you delete the evidence for that something is wrong.
If everyone hates systemd so much, why is it being incorporating into all these Linux distributions? Have all the major ones incorporated it? Does this "evil" Poettering guy really have that much clout in all the disparate distros?
The systemd haters are actually a tiny but vocal minority. They tend to flash-mob systemd threads, so you can often see here on slashdot, how a little handful of systemd-haters post 10-20 anti-systemd posts in anything remotely related to systemd. They seem like they are many, but when counting they are quite few.
No distro have lost users because of switching to systemd, in fact, systemd is part of the whole OS container wave that are fuelling the Linux engine at the moment. Not a single non-systemd commercial Linux distro have emerged since all the major Linux distro announced their shift to systemd, so the server market seems firmly behind systemd.
One reason why Canonical is changing to systemd as fast as it can, is because their OS container costumers are impatiently tapping their feet, waiting for systemd integration.
People have started to ignore this small, sometimes very toxic minority for quite some time, since the anti-systemd people are basically uninformed about any technical aspects of systemd, because they rely on hearsay and random hate blogs for their information about systemd instead of actually reading the systemd documentation.
Don't get your panties in bunch just because you discover that the journal is "corrupted"; usually it is just trivial stuff like a malformed timestamp or a field value that isn't valid in a single log entry. journald actually test and validate incoming log messages and report errors as it finds them. Even then, you can often read the log entry, but of course, the invalid field value will be missing.
The reason why people discover "corrupted" journal log files is because systemd's journal have integrity checks, unlike syslog who doesn't and where the admin therefore never will know if there are spurious or invalid log entries unless he happens to stumble upon them by chance.
Seriously, if you really care about each and every log entry, you should never have been using syslog anyway; it simply silently drops messages under load, and both local and remote logging are inherently unreliable. Yeah, I know Rainer have made a signal-safe syslog library, but does anybody actually use it?
There are simply too many Linux newbies who seems to be unaware of decade long struggle to improve the many, many problems with syslog. The "rsyslog" team have fought heroically, but often in vain, to try to fix many of the problems.
Syslog, like cron and SysVinit are among the several pieces of Linux infrastructure that can't be changed or radically improved. Because if you change your syslog/cron/SysVinit implementation, it would be incompatible with syslog/cron/SysVinit, so no one would use it, because user space programs doesn't support it etc. A circular dependency that prevent all progress. The systemd project have finally broken Linux free from that quagmire, so now we can actually have Linux infrastructure developed in the same pace as the kernel and user space programs and daemons.
systemd's journal is, warts and all, a massive improvement over the what existed before.
Take a look at FreeBSD Gnome, for example. I am sure that their patches to make Gnome portable again flow back upstream.
Perhaps, but this is exactly what I said; only the minimal effort to make the DE's run on BSD is made by BSD developers; when it comes to making the actual DE there seems to be no BSD developers helping out. In short, BSD developers aren't pulling their share of the load when it comes to DE development.
OpenSSH is portable. This is the difference between Linux and BSD developers. [snip: the usual anti-Linux ranting]. most Unices/BSDs already have the solutions for the problems and why should they accept something that has to be ported with a lot of effort? It is only being done with things that are worth to port.
Exactly, this is why DE's like KDE and Gnome shouldn't accept BSD patches anymore, but just make the BSD developers maintain the DE's in their own source trees, just like OpenSSH. If it is worth for BSD to maintain them, then they can do it, if not, then why bother Linux developers to make it BSD compatible.
As it is now, BSD is simply dragging Linux DE development down without the BSD developers contributing anything instead. At some point this has to stop; either the BSD community starts contributing, or the Linux community will stop working for free.
It would require a radical shift among BSD developers and the companies that sponsor them to make any serious inroad into the desktop. AFAIK, there are almost no BSD developers contributing to DE's like KDE or Gnome.
This is probably because the focus for BSD's are servers; their sponsors pay for making server software that may be close sourced. All the major DE's are using GPL toolkits, so BSD developers are unlikely to make any contributions besides the minimal required work to make the DE's work on BSD.
In case that eg. Wayland support don't materialize on BSD, then I find it much more likely that DE's like KDE and Gnome will split their code, leaving it up to BSD developers to maintain whatever they need to make those DE's run, in the same way OpenBSD does with OpenSSH.
None of that speaks to why systemd needs to suck in everything under the sun that has a server mode (like the gimp and open office examples above).
It doesn't. That a particular distro/package maintainer utilizes their distro's init-system and service manager for launching a daemon is hardly a problem or even controversial.
And just because something's launched often doesn't mean it has to be sucked into systemd. Angry Birds is launched on Linux more often than most stuff the systemd guys play with -- but that doesn't mean all games need insane dependancies on an init system.
It seems to me that have some problems with understanding what an init-system does. SysVinit/systemd/SMF deals with starting daemons and similar processes, not end user programs like games. Of course, if a game has a server mode what uses sockets and what not, then it probably is convenient to use a proper init-system.
Your container example seems to be taking the wrong approach too.
Lightweight containers like Docker seem to suggest it's best to run a single service within a container --- so the last thing such a system needs an init system -- let alone the most bloated init system in the world. A it turns out, it's quite a pain in the neck to run systemd in a docker container.
There are many different OS container modes, from running single services, to full blown servers. Each mode has it advantages and disadvantages. That flexibility is exactly what makes OS containers interesting.
Unlike other OS container implementations, systemd also support running unmodified Linux distros as containers. That means that I can boot a standard Debian distro on top of my Fedora distro, or running a newer unstable version of my distro on top. It takes seconds or minutes (depending on net speed) to launch such a container. A great way to test and debug new stuff without bothering with a full VM. The "machine concept" that allows maintaining OS containers from the "outside" is also way cool.
All in all, systemd is on its way to become the best OS container manager the world have ever seen, and that is great news for Linux.
Slashdot Mirror
So now you are going from saying that people do not use DB's for logging unless they are clueless; (Let me quote you on this:)
And yes, I have extensive experience with log analysis. You do not turn the logs into a database unless you are fully clueless.
And please remember that ACID doesn't give any protection against file corruption, nor does it prevent user space programs corrupting the logs by generating obviously impossible field values. But then again, neither does the journal nor plain text logs.
Again, for those who need to run a full DB for their logs, systemd's binary file format is great, because it allows the journal to be exported in industry standard formats like JSON. That way the remote log-sink database can receive and store rich meta data in a totally stable and structured way; changing hostnames, IP's, or even different wordings or even different and changing languages used by the daemon log output isn't a problem with the journal, since it is based on field values, not complex regex-ing of unstructured, undocumented, unstable, language specific words.
Oh, and tamper proof, cryptographically "sealed" logs too (FFS) if you want that.
If your remote log-sink solution isn't a full DB, you still gets all the benefits of receiving structured log entries with eg. full microsecond precision timestamps _and_ monotonic timestamps. It is trivial to convert eg. JSON output with defined field names to any other structured format, while converting and aggregating unstructured text files is a pain.
It is also great for those who only needs local logging; the journal has many of the advantages of a full DB, but without the complexity and overhead. Its append based file format is also much more robust against file level corruption than databases. Since the log files are structured and indexed with field values, they are easy to perform powerful, yet simple queries on.
How do you find all syslog entries with the priority level "error" generated by the previous boot only?
With the journal it is : "journalctl -b -1 -p err"
And how do you generate a full list of every executable, including their path?
Since you can "tab" trough the values in the journal this can easily be done: "journalctl -F _EXE "
And with the -x switch, the help database is activated, giving further explanation on what the log entry means, and gives direct links to upstream support, perhaps linking directly to a page that explains the error code etc:
Example:
# journalctl -b -x -u systemd-logind.service
mar 07 16:46:16 localhost systemd-logind[546]: New session 1 of user Peter H.S.
(log entry above, help database output below:)
-- Subject: A new session 1 has been created for user Peter H.S
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/m...
-- Documentation: http://www.freedesktop.org/wik...
There are seriously many people who just trawls through long log files (even using vi apparently), because it is hard to grep for anything unless you already know what to grep for, and because regex'es are well, regex'es; difficult to use, understand and to remember all the many variations. Basically, newbies can't read or filter Linux syslog log-files by other means than trawling through them.
They can't have a useful GUI either because it is impossible to make a distro-agnostic syslog gui. Again, a problem that systemd's journal solves.
The bottom line is, that the only virtue syslog files have, namely that they are human readable, is a serious hindrance for their use too: they can't add monotonic timestamps and micro-precision timestamps and other meta-data without being excessively difficult to read for humans.
With the journal you can give machine parsers exactly the structured log info they need and can benefit
I am not twisting your words. Let me quote you verbatim:
That does not make systemd better. It makes it worse as it is so complex is not had dependencies on tools. Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.
Isn't it straight up obvious that you claim that external contributors can be fended off because they can't afford licenses? Isn't this exactly what you are claiming?
Since this claim is totally untrue, isn't it then obvious that you are utterly confused about how Coverty and Jenkins work, and have no clue whatsoever on how systemd development is done?
No, I'm not misunderstanding. You will need a repair option to make the logs readable.
Oh yes you were, let me quote you verbatim on this:
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy.
You thought that apparently journald had a fsck option, it doesn't, which just goes to show how little you know about systemd.
As a sys admin I'm not interested in what it is doing.
As a Sysadmin you should know how your logging system works and the limitations it has. This is just basic SA stuff.
In this case you should be able to discern about malformed log entries and actual file corruption of the logs. This is apparently something that is really hard for you to understand.
Neither you or the tin pot systemd crew have any idea what logging actually means.
Yeah right, like you were the smart one; I am sure you are a hot shot Linux developer with a CS degree that has to fend off job offers every week from leading firms, because that pretty much describe the leading systemd developers, that includes among many, Greg KH, the kernel developer who maintains the stable Linux kernels etc. probably number two after Linus T. in the kernel group, or Poettering who has over a decades Linux developing experience etc.
I could go on, but the CV's of the systemd developers are really impressive, and many of them work for leading Linux distros like Red Hat, Canonical, Suse, Debian etc. ; if these Linux distros know nothing about logging, please write a peer reviewed paper that sets them all straight about this.
Good luck with your BSD adventure, be sure to tell them that BSD is all about choice, so that they should support several init-systems simultaneously and by the way, break up their source repos in many smaller independent groups or else they are "bloated" and "monolithic".
And don't cry when your BSD fork makes a systemd clone and throws away their old obsolete legacy script based init-system, because this is exactly what is going to happen, and yes, BSD's will get binary logging too, it is only a matter of time.
All these changes will be "forced down your throat" no matter how much you whine about.
BSD is for people who hate Linux, so you will feel right welcome there. It is just a matter of time before you will kowtow to the party-line about how GPL is bad because it doesn't allow BSD sponsors to close source the code etc.
Yes you are confused about several things, including licensing payment.
Your claim that people are discouraged from contributed because Coverty/Jenkins may discover that their patch have security issues or breaks the build is laughable; of course such patches should be rewritten, and the submitter would be glad they such problems was discovered. Asking the submitter to rewrite the patch, or that the developer with commit access fix it, is everyday work in FOSS land. It happens all the time and it discourages nobody; this is something the submitter learns from.
Really, should the alternative be to accept every broken patch just so not hurt the submitters feeling? I don't think anybody wants that.
Regarding your claim that the systemd developers don't play nice to others and suffer from "God complex" (talk about hyperboles), then this is plain wrong:
You can find lots of statements from people actually contributing, even small patches, that the systemd developers where really friendly and helpful. The proof is also in the numbers; there hundreds of such minor contributors to systemd, something that strongly indicate a good and including developer community. Compare that to the almost non-existing non-systemd developer eco systems.
Sure, the systemd developers sometimes says no to certain patches or features, but again, this isn't having a "God complex" but about project leadership and the technical know-how to reject things that are bad for the project. This happens in all FOSS projects.
Fedora - was working then broke again.
Ah, so you where trying to use zfs on Fedora, despite that this isn't supported. Problems may occur in such cases but the problems have nothing to do with systemd since systemd works fine with zfs.
Seriously? Try again. I am not confused at all. But you are blind to what is happening.
But you really seems to be very confused since you erroneously talk about systemd contributors needing to pay licensing fees: "Of course this also means external contributors can easily be fended off, as nobody can afford the licenses on their private budget.
That is just totally misunderstanding how things works, so it really seems like you are very confused about even basic facts how systemd is developed.
Disagree or not; logging directly to a full DB is very common, which is why it is a selling point for every commercial syslog implementation I know of.
Regarding the indexing and structuring of log files, this is exactly what journald does. This dramatically reduces work needed to analyse them, even if you re-index them and convert them another structured file format like JSON.
Working with indexed and structured files with defined field values, is always better than working with unstructured, un-indexed text files.
systemd's journal log files will always have an advantage over the poorly defined syslogd text logs, no matter how you spin it.
contain rich meta-data....
What a load of claptrap. People log to log files for reasons of the lowest common denominator. We have things called 'databases' for this kind of stuff and there's perfectly good reasons we don't use them for logging which should be obvious to anyone with an ounce of sense.
Really, are you unaware how common it is to aggregate log files in databases? This is a major selling point for Rsyslog that it is able to do so.
In fact, Rainer started Rsyslog exactly in order to overcome the many deficiencies with syslog(3), including the severe limitations of unstructured text files.
systemd's journal is pretty much a stroke of genius in that it overcomes all the limitations of unstructured, unindexed text logs, while not being a full blown DB either (the journal files are basically appended text files with a different line delimiter and an index in front).
Log files keep on growing every year, because people log more and more, and systems are running more and more daemons. Analysing such logs means machine parsing, and structured and indexed log formats like the journal have a huge advantage for this kind of work.
Since the journald binary logging file format is stable and fully documented....
I sincerely hope Red Hat and Lennart is paying for this piece of contrived PR.
Ah, so now facts and reality is propaganda. Take a look here
http://www.freedesktop.org/wik...
and here (about the file journal file format)
http://www.freedesktop.org/wik...
The arguments against it are based on contrived scenarios
Corrupted logs on a perfectly running system isn't a contrived scenario.
It is when the corruption means squat for the ability to actually read the logs.
For some reason you seem to think that "corruption" discovered by "journalctl" means the logs are unreadable, but as explained, they are often marked "corrupted" just because a single field value in a single log entry was discovered as impossible. This doesn't mean that the log file can't be read.
....through the Linux/Unix concept of piping.
Piping eh? Wow. You give the impression you haven't heard of it before ;-).
I don't care what you think, as long as you agree that the whole notion of not being able to use standard Linux text tools like grep together with systemd's journal is just plain wrong and a total non-issue.
There just aren't any good arguments against structured, indexed log files that can be programmatically accessed and has rich meta-data.
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy. I don't know where editing files was mentioned.
You misunderstand; there _isn't_ a "fsck" option for systemd journals and there won't be because that would be equivalent of "editing the log files".
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
You simply don't understand how journald works. Let me explain again: systemd marks certain log entries as "corrupted" if eg. they are plain wrong, that is the client is sending obviously wrong or malformed log entries. That way you avoid impossible field values that may otherwise have screwed the log watch or log analysing programs; think of it as discarding bad data. This is the most common "journal" corruption that people discover when using the journalctl "verify" option. It is usually totally harmless and usually limited to a single log entry. It happens all the time with syslogd text logs too, people just don't discover them because syslogd doesn't have log file integrity checks.
When journald discover such malformed entries, it log rotates as a simple preventive security measure.
Sure, the journald log files are sensitive to real file corruption too (but so is syslogd log files), so clean unmounts of file-systems is a good idea (it always is). But the whole issue is massively overblown by people who doesn't use systemd anyway.
A logging system corrupts logs, in a format specified by it, because it checks their integrity. I can only suggest checking into a clinic somewhere.
I can only suggest that you start to read up on systemd and the journal. You seem utterly confused about even basic facts.
It would just be an optional KDE module; KDE already have a rather excellent GUI for configuring systemd (Kcmsystemd) that works as an optional control panel module.
You seem to be somewhat confused here:
Coverty is static code analysis that find potential stability and security problems. https://scan.coverity.com/
Think "Unix Lint".
Many FOSS projects like the Linux kernel and LibreOffice uses Coverty. It isn't a magic wand, but it plain works nevertheless. No systemd contributor needs a licence in order to contribute to systemd, even if they get commit access. Coverty can trivially be replaced or complemented by other static code analysers, or even be omitted; it is something helpful, not something systemd depend upon.
The Jenkins builder provides: "Continuous Integration with automated test execution", something that really improves code stability since it ensures that eg. a patch isn't committed if it breaks the build.
Again, no contributor needs any kinds of licence for this, and again, this is a nice to have thing, not something systemd depends upon.
You seem to be arguing against code discipline and the use of automated tools, just because systemd have them (while SysVinit doesn't). That is just bizarre.
Anyway, systemd's developer community is extremely vibrant and growing all the time; there are hundreds of people who have contributed already with new ones coming every month. So there doesn't seem to be anything that scares contributors away, unlike, sorry for putting the boot in, the tiny and deteriorating non-systemd developer eco-system.
The major point of systemd's binary logs are that they are both structured, indexed, and contain rich meta-data, something plain syslog logs isn't.
That again means that when you export your logs to a log-sink, you can do so in a format that fits directly into the log-sinks database like JSON format and preserve the rich meta-data like the "_MACHINE_ID" field, that uniquely are tying every log-entry to a certain machine. You can also be sure that that the program claiming to generate the entry actually is correct since journald provides kernel guarantee for this.
If you choose to keep the logs in journald's binary format on the log sink server, you gain the benefit of indexed fields when analysing data; a huge win when it comes to random access. So much faster than trawling through text logs (O(n) complexity and all that).
Every log entry can be traced back to a certain machine (not just hostname), many log fields have kernel guarantee for what they say, besides having integrity checking and even strong cryptographically "forward secure sealing" (FSS) security against tampering.
The journald logs are simply designed from the ground up to be read an analysed on other computers than the one they were generated on, either in their native format or as exported logs.
Since the journald can be accessed programmatically, it can aggregate and analyse logs across different languages and has strong immunity against changing wording of the daemons log output (it operates on fields, not words).
And since the journal logs are structured identically by a documented standard, they are trivial to aggregate, unlike the output of many different syslog implementations.
Since the journald binary logging file format is stable and fully documented, and can be accessed programmatically with language bindings, you don't need a specific binary like "journalctl" to read them. In fact, there are there is a Rsyslog module that allows it to directly read (and export with metadata) systemd journal's. There are also Python modules etc. that acts as journal readers etc.
In fact, it is quite possible to make a systemd journal reader that works on non-systemd platforms like MS-Windows or OSX.
The journald collate all logging on the Linux machine, that means everything can be trivially exported to a remote log-sink, including the kernel ring buffer and the binary utmp and wtmp log files that syslog doesn't know about, and it can include early boot and late shutdown log info because it can work in initramfs, something syslog can't.
How about journald being designed as signal-safe from the ground up (unlike syslog) and that it doesn't silently drop messages under load etc. etc.....
systemd's binary log format is simply a massive win for both the enterprise user and the Linux newbie and everything in between.
The arguments against it are based on contrived scenarios, like professional admins that doesn't have (systemd) boot medias and lack access to a pc, a usb stick and a internet connection so they can make one in 5 minutes.
All the standard Linux text tools like grep, tee, sed, sort etc. work with systemd's journal through the Linux/Unix concept of piping.
Sure, systemd's journal still doesn't have as many tool sets and projects around it as syslog. But that it is simply a matter of time.
There are several very good projects on the Wiki page. My favourites are probably:
Project: Port Amarok to Qt5/Kf5/Plasma5: Something I use every day.
Project: Port KSystemLog to use journald as a backend: With systemd it is actually possible to make a distro agnostic GUI log viewer that isn't just a "less" with windows decorations. I like using the CLI "journalctl", but a GUI, perhaps with some log watch support and real time panel notifications about "syslog level: Error" events and above would be nice.
Project: Implement PDF Poppler features: I like Okular very much, so more features like linearized pdf support would be nice.
I'm using it and learning with each thing that breaks. Most recent was zfs and there were a few before that. If it hasn't ever caused you any trouble I doubt you've been using it much.
zfs has been working for years now with systemd, despite it not being part of the Linux kernel and therefore not really supported by many major distros.
So if you are experience troubles, it is most likely distro specific troubles like using a distro that haven't converted to systemd yet (Debian) or using zfs on a distro that didn't officially support it in the first place (CentOS/Fedora) and perhaps because you haven't read the zfs support docs carefully enough.
The systemd bugs I have encountered have been really trivial and not pertaining to core functions; it has never let me down by not being able to stop or disable daemons, or having run away processes, or suddenly consume lots of memory or cpu time. It is far better than SysVinit and Upstart in that regard too.
No, systemd detractors really is a tiny minority; some ways this really shows is how there are almost no developers working to maintain even critical needed infra structure for non-systemd distros; ConsoleKit has been abandoned for years now, eudev is just a shadow fork of udev with no independent development going on, and several key components like systemd-shim and cgmanager are only kept alive by paid Canonical developers and a few Debian devs; once Ubuntu and Debian shifts to systemd, those projects will languish too. SysVinit will properly also deteriorate completely; Red Hat/Suse was the defacto upstream before, and now it is only Debian as long as it last. So the non-systemd infrastructure will probably deteriorate further as the commercial distros stops to maintain it.
In short, almost no developers are working on maintaining non-systemd infrastructure, this reflects how few the systemd-detractors really are.
The recent Debian debate also show how few the systemd detractors really are when the numbers are shown: The system-detractors made a lot of noise on the Debian mailing list, but after the technical committee had decided that systemd should be the new Debian Linux init system, the detractors were unable to even gather 5 (like in five) Debian developers out of around 1000 to sponsor a vote on this subject.
Even the GR bill trying to keep other init-systems equally supported was clobbered at the GR vote.
So going by the noise on the mailing lists, the systemd-detractors seemed like a force, but when voting they where nowhere to be seen.
Same with Linux distros; you would think that the non-systemd distro ranks would be swelling with the numbers of systemd-refugees. This certainly doesn't seem to be the case. A couple of rather obscure distros like Funtoo and Void are among the few distros that don't want to support systemd. Slackware is undecided on the issue, and Gentoo etc. support systemd, with a growing number of its users that prefer it to OpenRC.
Also no medium/major commercial non-systemd Linux distro have emerged this last couple of years, this is a strong indication that the paying costumers wants systemd, and doesn't care at all for the alleged superiority of SysVinit. Several companies made it clear during the Debian debate that they favoured systemd, none spoke for SysVinit or Upstart.
No wonder; systemd is great and it solves real world problems like daemon management and security much better than any other alternative.
I have been using systemd since its inception, and it has been rock solid. It is better tested and documented than most other FOSS projects; a Jenkins back-end and Coverty scans, integrated self tests, and rather strict coding practise really helps too in that regard. Take fx. SysVinit; their developers doesn't even have build test framework, so the only way to test whether a new patch doesn't break everything, is to make a live boot. No wonder they haven't made a release for many years despite the patches are accumulating.
Claiming that it isn't ready yet is just plain wrong and just seems to be some sorry excuse for not bothering to learn something new.
Thanks, that's interesting. I always thought one of the strengths of Linux was that the OS was highly separated from the GUI.
Don't believe him, what he says is demonstrably untrue. Both KDE and Gnome runs just fine on non-systemd distros and on BSD. This couldn't happen if they somehow where totally dependent on systemd.
The real story is that systemd provides some very needed OS infrastructure that Desktop Environments (DE's) like KDE and XFCE needs. The old non-systemd infrastructure that DE's could use instead, like ConsoleKit (CK), hasn't been maintained for years. The DE developers have warned the non-systemd distros for years that they needed to maintain CK, or that things would stop working, but they were ignored.
So Gnome, KDE, XFCE etc. still have support for non-systemd distros that are using eg., CK's login API, but things are beginning to fall apart since nobody have bothered to maintain that for years.
The point is that that Gnome and KDE played no role in why all major Linux distros have shifted to systemd. The simple fact is, that systemd simply is light-years ahead of any competing init-system; integrated groups support so each and every process can be controlled regarding resources, no more hard to write and debug shell scripts to start daemons (executable config files; who thought that was a good idea?), integrated kernel Capabilities(7) to provide defence in depth for all running daemons etc.
Read here for more reason to what systemd can and why it was developed:
http://0pointer.de/blog/projec...
Here is one of many:
https://bugs.freedesktop.org/s...
The TL;DR resolution: journalctl --fsck after rotation or pump it into a traditional syslog
Lennart Poettering's comments about it:
"our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse"
Totally sensible solution, why would you edit log files? They are not meant to be changed.
Besides that, the log entries that are mangled are in them self important info; How can you detect a pattern of a misbehaving program that sends wrong field values if you delete the entry every time it happens?
Or what if the log corruption is caused by a failing disk where the bad blocks are spreading like wildfire? If you silently delete such entries and files, you delete the evidence for that something is wrong.
If everyone hates systemd so much, why is it being incorporating into all these Linux distributions? Have all the major ones incorporated it? Does this "evil" Poettering guy really have that much clout in all the disparate distros?
The systemd haters are actually a tiny but vocal minority. They tend to flash-mob systemd threads, so you can often see here on slashdot, how a little handful of systemd-haters post 10-20 anti-systemd posts in anything remotely related to systemd. They seem like they are many, but when counting they are quite few.
No distro have lost users because of switching to systemd, in fact, systemd is part of the whole OS container wave that are fuelling the Linux engine at the moment. Not a single non-systemd commercial Linux distro have emerged since all the major Linux distro announced their shift to systemd, so the server market seems firmly behind systemd.
One reason why Canonical is changing to systemd as fast as it can, is because their OS container costumers are impatiently tapping their feet, waiting for systemd integration.
People have started to ignore this small, sometimes very toxic minority for quite some time, since the anti-systemd people are basically uninformed about any technical aspects of systemd, because they rely on hearsay and random hate blogs for their information about systemd instead of actually reading the systemd documentation.
Don't get your panties in bunch just because you discover that the journal is "corrupted"; usually it is just trivial stuff like a malformed timestamp or a field value that isn't valid in a single log entry. journald actually test and validate incoming log messages and report errors as it finds them. Even then, you can often read the log entry, but of course, the invalid field value will be missing.
The reason why people discover "corrupted" journal log files is because systemd's journal have integrity checks, unlike syslog who doesn't and where the admin therefore never will know if there are spurious or invalid log entries unless he happens to stumble upon them by chance.
Seriously, if you really care about each and every log entry, you should never have been using syslog anyway; it simply silently drops messages under load, and both local and remote logging are inherently unreliable. Yeah, I know Rainer have made a signal-safe syslog library, but does anybody actually use it?
There are simply too many Linux newbies who seems to be unaware of decade long struggle to improve the many, many problems with syslog. The "rsyslog" team have fought heroically, but often in vain, to try to fix many of the problems.
Syslog, like cron and SysVinit are among the several pieces of Linux infrastructure that can't be changed or radically improved. Because if you change your syslog/cron/SysVinit implementation, it would be incompatible with syslog/cron/SysVinit, so no one would use it, because user space programs doesn't support it etc. A circular dependency that prevent all progress. The systemd project have finally broken Linux free from that quagmire, so now we can actually have Linux infrastructure developed in the same pace as the kernel and user space programs and daemons.
systemd's journal is, warts and all, a massive improvement over the what existed before.
Take a look at FreeBSD Gnome, for example. I am sure that their patches to make Gnome portable again flow back upstream.
Perhaps, but this is exactly what I said; only the minimal effort to make the DE's run on BSD is made by BSD developers; when it comes to making the actual DE there seems to be no BSD developers helping out. In short, BSD developers aren't pulling their share of the load when it comes to DE development.
OpenSSH is portable. This is the difference between Linux and BSD developers. [snip: the usual anti-Linux ranting]. most Unices/BSDs already have the solutions for the problems and why should they accept something that has to be ported with a lot of effort? It is only being done with things that are worth to port.
Exactly, this is why DE's like KDE and Gnome shouldn't accept BSD patches anymore, but just make the BSD developers maintain the DE's in their own source trees, just like OpenSSH. If it is worth for BSD to maintain them, then they can do it, if not, then why bother Linux developers to make it BSD compatible.
As it is now, BSD is simply dragging Linux DE development down without the BSD developers contributing anything instead. At some point this has to stop; either the BSD community starts contributing, or the Linux community will stop working for free.
It would require a radical shift among BSD developers and the companies that sponsor them to make any serious inroad into the desktop. AFAIK, there are almost no BSD developers contributing to DE's like KDE or Gnome.
This is probably because the focus for BSD's are servers; their sponsors pay for making server software that may be close sourced. All the major DE's are using GPL toolkits, so BSD developers are unlikely to make any contributions besides the minimal required work to make the DE's work on BSD.
In case that eg. Wayland support don't materialize on BSD, then I find it much more likely that DE's like KDE and Gnome will split their code, leaving it up to BSD developers to maintain whatever they need to make those DE's run, in the same way OpenBSD does with OpenSSH.
None of that speaks to why systemd needs to suck in everything under the sun that has a server mode (like the gimp and open office examples above).
It doesn't.
That a particular distro/package maintainer utilizes their distro's init-system and service manager for launching a daemon is hardly a problem or even controversial.
And just because something's launched often doesn't mean it has to be sucked into systemd. Angry Birds is launched on Linux more often than most stuff the systemd guys play with -- but that doesn't mean all games need insane dependancies on an init system.
It seems to me that have some problems with understanding what an init-system does. SysVinit/systemd/SMF deals with starting daemons and similar processes, not end user programs like games. Of course, if a game has a server mode what uses sockets and what not, then it probably is convenient to use a proper init-system.
Your container example seems to be taking the wrong approach too.
Lightweight containers like Docker seem to suggest it's best to run a single service within a container --- so the last thing such a system needs an init system -- let alone the most bloated init system in the world. A it turns out, it's quite a pain in the neck to run systemd in a docker container.
There are many different OS container modes, from running single services, to full blown servers. Each mode has it advantages and disadvantages. That flexibility is exactly what makes OS containers interesting.
Unlike other OS container implementations, systemd also support running unmodified Linux distros as containers. That means that I can boot a standard Debian distro on top of my Fedora distro, or running a newer unstable version of my distro on top. It takes seconds or minutes (depending on net speed) to launch such a container. A great way to test and debug new stuff without bothering with a full VM.
The "machine concept" that allows maintaining OS containers from the "outside" is also way cool.
All in all, systemd is on its way to become the best OS container manager the world have ever seen, and that is great news for Linux.