I suppose this bias toward old, good documentation is contributing to my dislike of Systemd, I don't see the same docs for Systemd that I saw for System V and BSD init structures.
Every man-page I have used is done by the book: good examples, relevant "See also" references, and sometimes even links to further information. I really like the fact that all config files have extensive man pages too (man journald.conf fx), and that they also provide "big picture" documentation like "man bootup".
I have often seen how a question on the systemd devel list resulted in clarification of the documentation, so they take their documentation seriously.
On top of the man pages, there is also a huge amount of information on the projects homepage, spanning from overview videos, to low level developer information. http://www.freedesktop.org/wik...
Those are "the systemd requirements for the filesystem", what systemd needs. It provides no clarity about what goes _in_ those directories, nor why,
Yes, those are the basics for systemd to work. Again, rather simple requirements. It doesn't state what goes into what, because this is something the distros dictate, not systemd. The systemd developers can merely suggest where to put things, but distros won't necessarily follow those suggestions.
systemd is designed with different distro policies in mind, so they provide "Presets" to help facilitate the different policies and needs of different distros: http://freedesktop.org/wiki/So...
Compare this checklist with the actual FHS documents, and you'll see the distinction. The result is confusion, such as the systemd migration of "/media" attachable storage to the individual user's owned subdiroectories in "/run".
The FHS is primarily concerned about directory layout, not so much the content of these. And there is nothing in FHS 3.0 that contradict the slightest that user attached storage is mounted in the/run hierarchy. All such disc layout reorgs of the distro I have tried, have been discussed on mailing lists etc. before they are implemented. Again, this is something the distros dictate, not systemd.
Allow me to restate your comment:
> Systemd will continue to work for Linux distros that don't care for stateless boot etc
Rather, systemd will continue to re-arrange, and break, stable subsystems without warning to the user communities.
Come on. It is the distros that dictates exactly what features systemd have turned on, and how the FH layout is. There are no systemd developers sneaking into the distro repos at night and secretly turning on features without documenting them.
You may not follow the eg. development mailing list of your distro where such things are discussed, nor care about reading release notes. But Linux distros have always evolved this way, with new ways of doing things, often from release to release.
Note that/run/systemd/resolve/resolv.conf should not be used directly but only through a symlink from/etc/resolv.conf.
I fail to see how this in any way breaks any stable subsystem. 1. systemd-resolved is an entirely optional feature. If the distros don't want it, they can just turn it off. 2. Whatever its/etc/run requirements is, this is a new feature, not a change of something old and stable. So it doesn't break anything at all.
I really don't think you can give just a single non-contrieved example of how systemd are breaking stable subsystems for users without warning.
While more documentation is always better, I think the systemd project is way better documented than most other software projects, and certainly much better documented than many people think it is.
I'm not even going to mention what happens if you accidentally follow generations of sytem standard behavior and install NTP alongside an active systemd NTP daemon. It's not a pretty site, and the accumulated clock management confusion if they're not pointed to the same NTP servers can break Kerberos authentication in startlingly short order.
Running two NTP-clients at the same time was an amazingly stupid thing to do even pre-systemd. AFAIK, systemd per default tries to avoid this by using "conflict" statements in the service files, so if one NTP client or daemon is run, the others are suppressed. I don't think eg. SysV
The systemd requirements for the file system is documented here: http://www.freedesktop.org/wik... These are rather simple requirements and probably very close to the old FHS 3.0 proposal, I think/run is the only new thing here compared to FHS 2.3
AFAIK, all other changes of the FHS are merely suggestions like getting extra OS container features when using btrfs, and in case of stateless boot, something for the future. Systemd will continue to work for Linux distros that don't care for stateless boot etc.
You mean boot logging? There are numerous ways to solve this which would have made more sense, including a small change to the linux kernel to make it stop throwing away boot logs when they are large.
The kernel ring-buffer size can be user specified, so no need to ever lose such messages. The problem is that during boot, only messages directed at/dev/kmsg is recorded until rootfs is mounted and syslogd is running. This is hard to do something about since it is a kernel limitation that only one device at the time can own/dev/log. journald catches all messages, including those to/dev/kmsg and/dev/log, and it does this from early boot. So a journal based system will always contain more info than a pure syslog system.
So use syslog-ng, which can be used in place of syslog without changes to other daemons, unlike journald. Which again, is the reason we're even having this conversation.
Regarding silently dropping messages: AFAIK all syslog daemons does this per design (or lack thereof). The main problem is that calls to syslog() would be blocking if implemented in a safe manner.
Wait, you don't care about human-readability, but you don't want the logs to become unreadably long? Make up your fucking mind. You're just making shit up, I can tell because that shit is contradictory.
You misunderstand; a reason why syslog messages doesn't contain monotonic timestamps etc. is that it makes the log entries extremely long and hard to read for humans. It is a kind of limitation of the "design". journald doesn't have such limitations since it is field based, and the output format is easily adapted to what view you want. If you only want wall time stamps, you can get that, if you want monotonic time stamps, you get that, and only that.
So with the journal, you can add more fields in the future without making the log entries unwieldy, something impossible with syslog.
Regarding human readable: I only care about that for the subset of information I intend to read, and this is exactly what journalctl does. That way you can have logs that are easy to parse for both humans and machines; each consumer gets exactly the view of the data that they want in a form they can use.
that text logs simply don't scale
Still not an argument against text logs, only an argument for also implementing binary logs. And I wouldn't put them into their own shitty files with their own shitty format, I would put them into an RDBMS. Putting them into crappy, easily-corrupted binary log files is just asking for trouble, and guess what? It's causing trouble.
Yes, the scaling thing is a good argument against using flat file text logs. You really need indexes when analysing such huge amount of data, which is exactly what people are doing at the moment; they invest in often proprietary commercial solutions in order to massage their flat text file log data into something usable.
RDBMS have been tried for a long time and they are not a good general purpose solution for local logging. They are also much more prone to corruption than the journal's simpler binary storage format.
What? So now you don't like flat files or regex? Why don't you just fuck off to Windows and let us have our Linux? You clearly don't like Unix.
While regex is a powerful tool, the old joke about trying to solve a problem using regex, and then discover you now have _two_ problems, is true too. Discounting the many variations of regex that each program like grep/egrep uses, it simply becomes unwieldy when you are using regex epressions that are +120 characters long. Making ad hoc queries with regex really is painful when the complexity rises beyond the basics. The systemd journal is so much better regarding this since it is field based.
I assume you are talking about FHS 3.0 here, but remember, that is only a proposal, not a standard. Also note how the FHS 3.0 process has been stalled for years. The bottom line today is that there is FHS 2.3, and then everybody doing their own thing , perhaps with some cross-distro cooperating. The end result is no common formal standard for the FHS among the distros.
Personally I don't think there will be a FHS 3.0 until the dust has settled regarding the current layout reorg,
The reorg of the FHS that the systemd developer are doing, are caused by good technical reasons. Unix and later Linux have had several of these reorgs over the decades, and it is a process that probably never stops.
So until the FHS reorg to make about OS containers, and stateless boot work in a sensible manner, the Linux FHS will be in a slight flux. Not a big problem really. I guess at some point somebody will make a FHS 3.X out of what the systemd distros does at some point in the future. Personally I prefer this approach of practise getting formalized into a standard. That way you are sure to have both a good working standard, and a standard that people actually use.
Judging by this thread, it does give at least one advantage, that being the ability to at least partially recover and analyze corrupted logs.
The same is very much true for systemd's journald files. They are just appended text files with another delimiter +index, so in worst case a hex editor or "strings" may read them.
"journalctl" is designed to cope with corrupted journals and read them very well. It should be noted here that corruption of journal files as discovered by the --verify option, mostly are very trivial errors like a monotonic timestamps of the wrong length etc. Such errors only affects a single log entry, and often even that log entry can be read without problems. Despite using journald since it became standard on Fedora (F15) I have never experienced any data loss nor having journal files that I was unable to read.
Plain text log files, like the original syslog(3) interface was simply bad ideas that Linux inherited.
No, that was a brilliant idea that Linux inherited.
Come on, syslog(3) wasn't even signal-safe and designed as dog slow too. It was crap. People have been working ever since to make syslog thread safe (still interim), etc.
What classic syslog problems? I haven't had a problem with log rotation, really the last remaining issue, in approximately forever.
That syslogd can't log until rootfs is started and most of the machine is up, that it is dog slow and silently drops messages under load, that there is no common API for accessing the content of the data, that the data is basically free form, deviating from machine to machine (an industry exist dedicated to this problem alone) also making it impossible to make a distro agnostic GUI log viewer, that meta-data like monotonic timestamps are basically impossible to add without the log entries becoming unreadable long, that text logs simply don't scale (again, an industry exist to solve this problem, see Splunk etc.), that using regex becomes intolerable and unwieldy above a certain point, that in order to grep for something in the log file, you basically already have to know the name of the function and what the problem is, this makes syslog log files basically impossible to use for newbies, so they have to resort to trawling through the logs like reading a book.
Really, flat file, un-indexed log files meant to be read by a human SA is something that belongs to another era. What the present and the future needs are binary, structured and indexed logs with a stable API that can be parsed by machines and that works even at mega-scales.
And systemd's journald is designed to be 100% backwards compatible with syslog, so if your setup have legacy need, it is trivial to get a systemd distro to use legacy syslog text logs.
And syslog gets the messages late, and may not get them at all.
Well, these are know problems with syslogd implementations; they silently drop messages under even moderate load. It is their ancient design, so using pure syslog you will never know whether something was logged or dropped. With journald you will at least be informed if log messages are dropped. The latter should be a rare occurrence since journald is so much faster than syslogd.
journald goes to great lengths by using per app log rate limiting in order not saturate syslogd, but the inherent limitations in syslog means that this will always be an best effort. Still, it probably isn't worse than using syslogd alone.
I'm curious: why not just make an indexed database logging system on top of, rather than instead of, text logs? i.e. write out log entries in some structured but still text-only format in one file, and then maintain an efficient binary index of entries in a separate file. That way you get all the nice search etc, while still being able to grep the raw text if you really need to.
There are several problems with solution: it isn't backwards compatible with syslog (journald strives for 100% backwards compatibility).
It doesn't solve the problems that combining meta-data with log entries in flat text files, makes the text logs hard to read for humans since the log lines become far too long and complex.
It doesn't really gives any advantage of pure binary data. Remember grep, sed, tee and every other standard Linux text tool works with systemd's journal through the standard Linux/Unix concept of piping, so there is nothing lost by going pure binary.
The reorg of the FHS is quite needed in these days of OS containers, sandboxing and virtualization, so while placing stuff under "/run" is something new (part of the proposed FHS 3.0 standard btw), there are good technical reasons for it, including that/run is guaranteed to available from early boot (initramfs).
Thanks for the links. Last time I looked into this, the remote features were still a feature request.
Maybe I'm not understanding you, but I thought journal was meant to be superior to syslog and eventually supplant it.
Well, yes and no. journald was meant as a total overhaul of _local_ Linux logging. It provides "metal-to-metal" logging by being able to operate in initramsfs before rootfs is mounted, and pivot back to initramfs (Dracut) after rootfs has been unmounted. It collates all logs into a single view, instead of the myriad of scattered legacy log files (including binary logs like utmp etc). It also vastly improve the way programmers can use logging, since syslog(3) etc. have so many limitations regarding speed, messages per seconds etc. It also provide structured logs with plenty of metadata like monotonic timestamps that are troublesome in syslog text logs.
Rsyslog and similar syslog implementation works both as a _local_ logging system _and_ as a remote log sink.
journald doesn't work as a general log sink for remote logging, nor does it provide DB interfaces and various other similar plug-ins. So for general log-sinks, Rsyslog was always meant to be choice.
journald was also designed to be 100% compatible with syslog since it was obvious that legacy requirements would make that a necessity. So again, for legacy setups, syslog was always the designed choice to run on top of journald.
The issue is not with systemd corrupting the binary logs, or with the filesystem corrupting the binary logs, but with the fact that they are -binary- logs. A log file should be an ordinary text file. Nothing more, nothing less.
The glaring problems with flat file, unstructured text logs have been discussed for decades now. Plain text log files, like the original syslog(3) interface was simply bad ideas that Linux inherited. The main commercial selling point with any syslog implementation today is exactly that you can avoid several of these problems by using a DB.
An overhaul of the creaking legacy Linux logging system have been needed for a decade at least. Remember, the Rsyslog project was started exactly to fix such problems. They failed partly because distros and developers don't care about working together for a common goal and because there was no other central developer hub who would take the initiative.
systemd have finally fixed so many of the classic syslog problems. I don't care if you think syslog is perfect and never can be improved, and Rsyslog should be avoided for providing binary DB storing option, but please don't think that other people agree to this postulate.
And systemd's journald is designed to be 100% backwards compatible with syslog, so if your setup have legacy need, it is trivial to get a systemd distro to use legacy syslog text logs.
Bug report: Logs keep getting corrupted and cannot read them at all
Rejected with reason: Delete the corrupted logs and move on
I doubt that you can provide a link for that. Perhaps you are hinting to the RFE about "pruning" journal logs containing "corrupted" field values?
SystemD - works well when it works, fails spectacularly when it fails.
That is part of the Unix philosophy: (Rule of Repair : http://en.wikipedia.org/wiki/U...), or "if you have to fail, fail early and fail noisily". systemd is build around that; so if disks specified in fstab doesn't show up, it won't boot but give a rescue shell instead. This prevents silent data loss; a classic example of "Rule of Repair".
Been using systemd on Fedora for several years now; it has been rock solid. systemd is just so much better than any alternative out there, and for every release it just keeps on getting better. I love its logging system and the easy and natural way it handles OS containers and system service management.
Stateless boot is nothing new in Linux. What the systemd developers want is to make Linux generic enough to make stateless boot work for those who needs it, instead of each and every developer group making their own proprietary tools. Stateless boot are mostly for embedded but also allow for some interesting network booted options.
There is a lot of work to be done before it is possible, but with systemd people are at least working on it.
Regarding the FHS and where to put things like using/run. systemd is only doing the sensible thing here. The FHS 2.0 is ancient, so the systemd way of doing things is likely to become FHS 3.0, and then it will be a Linux standard.
SysD's binary logs have another, serious flaw: they are not designed to be sent over a network. This has been an intrinsic part of syslog for a looong time.
Of course you can push journald logs over the network in several different ways, and you can also receive them using the journald. But the systemd journald isn't substitute for syslog, but meant to complement it. Just use rsyslog when it is relevant, or just systemd's journald when that suffices.
Here is how to receive or pull systemd journals across the network: It can either pull log requests or passively wait for a connection: http://www.freedesktop.org/sof...
PA can provide both things, by suspending out if your need shift from consumer audio (PA) to Pro audio using jackd. No need to dedicate your box to either; they can both work for whenever people need different solutions.
Are you really saying that a feature of pulseaudio is the ability to disable pulseaudio and use something else?
Yes I do. It is great that you can use exactly the program you need, even if their usage normally are orthogonal. Please note that PA doesn't disable itself, it just automatically suspend until needed again. PA and jackd where made for different reasons and solves different problems.
I see a lot of posts complaining about anti-systemd zealotry here, but clearly there is a lot of zealotry all around. I can't recall any other topic in Linux history that has been this divisive, and that is part of the problem.
Well, some people thought they could stop the systemd progress by waging a negative campaign against it and its developers. That of course was a predictable failure.
Personally I don't care whatever init system people run. Even if I find e.g. SysVinit inferior to systemd I have no problem with people using it, and they owe me no explanation for doing so.
Systemd-opponents don't agree upon what they do like, they can only agree on what they don't like. This make their tactics so rabid and aggressive when attacking systemd, its developers, the distros that uses it and even its users. They simply don't have a positive alternative they can agree upon, so they are left with negative attacks.
Yes, and most of that existing industry will have to rewrite everything from scratch without file format compatibility.
journald has excellent export capacity, it is trivial to convert everything into JSON or similar industry standards without tedious data massaging. Unlike the many slightly varying syslog implementation with deviating output, journald output is always predictable since has a stable API and is structured around fields. This is a massive win for everybody, especially those who needs to put their logs into full DB's (for legal or whatever reasons).
"Another problem with text log files are their limited ability to incorporate much meta-data, since that makes the log lines unreadably long. So having full microsecond precision and monotonic timestamps is a problem."
Really? Here's the microsecond monotonic timestamp on my current box: 242841.31760194. That's 15 bytes. A 64-bit binary timestamp is 16 bytes. The text version is shorter! And given that the vast majority of usages do not require microsecond resolution (which isn't even really accurate, anyhow), the text approach would lead to substantially _smaller_ files.
Adding full microsecond precision and monotonic timestamps and wall time timestamps + all the other interesting meta-data and the log message itself, are making the text log entries extremely long and unfriendly to read for humans. With journald you only see such meta-data info if you ask for it. Much better solution. And if you are concerned about log sizes, journald has transparent compression. With text logs you need to unpack or use specialised tools to manipulate them if compressed.
Plain text logs sucks for so many reasons, their only redeeming feature seems to be you don't need to pipe them through a tool in order to read them. I have much respect for the Rsyslog developers, and I remember how I hoped they would fix the most of the glaring syslog problems when they started out a decade ago and in their own way tried to solve the problems that journald does. They partially failed, not because they weren't good, but because changing certain stuff in Linux userland is extremely hard.
sysvinit is crude and simple because it pushes all complexity out of init into userspace and because it offers no worthwhile init functionality. That is why Linux endured decades of remote exploits since SysVinit wouldn't take responsibility for dropping daemon privileges or do socket activation like systemd does, so daemons didn't have to use complex and error prone privilege dropping code just to get a port below 1024.
With systemd and new style daemons you can just avoid all that crap in the daemons. Much better to have one central place for security code made by experts than each and every daemon needs to compensate for the abysmal lack of functionality and security features of SysVinit.
I don't care if some people think that SysVinit is the best init system ever made and that it has no errors and can't be improved and Linux should use it forever and ever. We happen to be a lot of people who don't see it that way, which is why we like systemd and chooses distros that support that.
Or, you know, pretty much noone really cares about having indexed and structured log files. Separating into log files based on hostname and application name has worked really well for me for ages. And, because you're still going to have a human write the log messages, you're still going to have to eyeball the output at some point.
Great that syslog works for your usercase. I doesn't for me and so many others. There is an entire industry build around the need to beat syslog files into some sort of structure and give them an index so you can do number crunching on them.
These days log files grows so fast and is so important for business decisions that only machine parsing can do the job properly. The journal is excellent in that regard too.
Another problem with text log files are their limited ability to incorporate much meta-data, since that makes the log lines unreadably long. So having full microsecond precision and monotonic timestamps is a problem.
Yes, at some point a Mark I eyeball is going to look at a tiny subset of the data, but these isn't a problem either with systemd's journal; all the usual standard Linux text tools work through piping. And there are language bindings up the yazoo so you can manipulate it the journal files directly with Python scripts etc. It is so easy to extract the data that you need.
Personally I find syslog text files pretty hostile to newbies; unless they grep well and know a variation or two of regex, they are bound to just plow through the logs trying to identify some error. Really, many of them even resort to editors for reading log files. And the DE developers can't make a distro agnostic GUI log viewer better than "less" because there is no stable text log API. With systemd's journal they can because it has a stable API and language bindings etc.
With journalctl it is so easy to make even powerful extractions like; just show me logfiles generated the last 5 minutes (journalctl --since -5m) or all errors with syslog level "warning" and above created since this boot (journalctl -b -p warning). There is tab completion of every command option and every journal field and ususally even of the values of those fields. No need to memorise obscure arguments and subsystem names.
Sure it is possible and has been done before. My point is that no distros seem to use such a init-system and that no other init-system have anywhere near all the nice features that systemd has.
Rather typical LKML discussion in my opinion, but blown out in all proportions by the systemd-haters because they wrongly think that Linus don't like systemd nor its programmers because of this.
No it wasn't. Linus made his views on when this stuff will get merged, if ever, abundantly clear. It's not blown out of all proportion. That's what was written.
You don't seem to have experienced many LKML flame fests before, so I think you will be sadly disappointed when Linus merges kdbus.
Linus Torvalds have long signalled that he wants kdbus in the kernel.
No he hasn't you liar. I'm afraid this propaganda ain't going to have any effect whatsoever on his views.
Have you even read the lkml? Linus is deep into discussion of the code and even defended kdbus design decisions. He would never do that if he didn't intend to merge it as some point when enough people thought it was good enough.
Also notice how Linus appeared in the first kdbus patch threads. He didn't comment on the code, so the only reason he mailed was to signal to everybody that he followed this, and this was just business as usual.
Sure, the kdbus maintainers will have to do a good enough job in fixing problems before it is merged, but as it looks now, this doesn't seem to be a problem.
kdbus is good stuff that can benefit everybody, and while it isn't a shining-new general purpose IPC that some kernel developers want, it does solve real world problems, and that is exactly what Linus think his kernel is all about.
You know what's different about consumer and pro audio? Nothing. As it turns out, consumers also want high-fidelity audio, and they want to be able to mix multiple streams. The only difference lies in what the industry is willing to sell us.
This not about spending money or HiFi, but on what you use the audio system for. On consumer audio you want low cpu and battery drain and accept the tiny latency associated with this; that a music stream start 20ms second later doesn't matter. It is also important that features such as streaming, bluetooth head sets integration etc. works out of the box etc.
With Pro audio latency is king. cpu and battery drain means nothing.
PA can provide both things, by suspending out if your need shift from consumer audio (PA) to Pro audio using jackd. No need to dedicate your box to either; they can both work for whenever people need different solutions.
A bug from 2007 on Ubuntu that famously fouled up their PA implementation, hardly a serious complaint. This 2012 shows how PA even beat Androids audio stack (Audioflinger) regarding CPU and poweruse. http://arunraghavan.net/2012/0...
There are several such benchmarks that shows how good PA is in this regard.
Unlike you I have several years of experience with systemd. You just seem to parrot the party line propaganda from the tin-foil hat systemd-haters.
Don't run with the pack. Try to actual examine systemd by reading the documentation and get some experience yourselves. It takes some time to really understand how systemd works.
Did you actually read my post which was about a very specific point about binary versus text versus INDEXED text log files
systemd's journal is a basically a textfile that uses another line delimiter and have an index. Try "strings" on it, or a hex editor.
Well if so, that's new. It never used to be the case that the binary format was stable. So, I'm going to ask you to provide a link to the description of the binary structure.
Here is is the systemd interface and portability chart. Notice sd_journal and the journal file format: http://www.freedesktop.org/wik...
There are loads of other developer information about accessing the journal with Python language bindings etc.
Jesus fucking H. Christ on a stick what is wrong with you? I specifically went into detail about how noisy the pro-systemd people like you insist on a dichotomy between unstructured text files and binary file formats. There is no fucking dichotomy you moron. If you actually read my sodding post instead of going off on a rant then you would actually become marginally more informed and stop spouting the same old shit repeatedly.
Well, you are not the first person to think about how to extend text logs with some kind of index, and it would be nice if it could have microprecision timestamps, and monotonic timestamps like the journal does. The problem is that is unworkable.
As I said, several projects have tried to fix this without any luck for the last couple of decades. Another problem with such a solution is that standard Linux text tools like grep, tee, sort etc. won't work with such a solution where the index isn't part of the text file, and metadata like microsecond time stamps tend to make the log lines unreadable long.
With systemd's journal all standard text tools work with the index and metadata, so you can grep monotonic timestamps by a simple pipe.
If it was possible to have indexed, and structured plain text log files that didn't require a special "translator" in order for the standard Linux text tools to work, it would have been implemented a decade ago.
The whole point I was trying to make that looking back on sound on Linux in the days before ALSA and PA shouldn't be done with rose tinted glasses. Claiming that, like the OP does, that everything went backwards with the invention of ALSA and later PA is just absurd. Those where the bad days regarding sound on Linux. There is nothing to miss from those days, including ISA sound cards.
With ALSA and later PA there were actually people working on the Linux soundstack and developers that could coordinate bug fixing. And PA was a huge help for DE developers that where so tired of trying to maintain their own sound servers and dealing with really low level kernel stuff like drivers, when all they wanted to do was to develop for the desktop.
Slashdot Mirror
I suppose this bias toward old, good documentation is contributing to my dislike of Systemd, I don't see the same docs for Systemd that I saw for System V and BSD init structures.
systemd is a seriously well documented project. At the moment there are around 202 man-pages.
http://www.freedesktop.org/sof...
Every man-page I have used is done by the book: good examples, relevant "See also" references, and sometimes even links to further information. I really like the fact that all config files have extensive man pages too (man journald.conf fx), and that they also provide "big picture" documentation like "man bootup".
I have often seen how a question on the systemd devel list resulted in clarification of the documentation, so they take their documentation seriously.
On top of the man pages, there is also a huge amount of information on the projects homepage, spanning from overview videos, to low level developer information.
http://www.freedesktop.org/wik...
Those are "the systemd requirements for the filesystem", what systemd needs. It provides no clarity about what goes _in_ those directories, nor why,
Yes, those are the basics for systemd to work. Again, rather simple requirements. It doesn't state what goes into what, because this is something the distros dictate, not systemd. The systemd developers can merely suggest where to put things, but distros won't necessarily follow those suggestions.
systemd is designed with different distro policies in mind, so they provide "Presets" to help facilitate the different policies and needs of different distros:
http://freedesktop.org/wiki/So...
Compare this checklist with the actual FHS documents, and you'll see the distinction. The result is confusion, such as the systemd migration of "/media" attachable storage to the individual user's owned subdiroectories in "/run".
The FHS is primarily concerned about directory layout, not so much the content of these. And there is nothing in FHS 3.0 that contradict the slightest that user attached storage is mounted in the /run hierarchy.
All such disc layout reorgs of the distro I have tried, have been discussed on mailing lists etc. before they are implemented. Again, this is something the distros dictate, not systemd.
Allow me to restate your comment:
> Systemd will continue to work for Linux distros that don't care for stateless boot etc
Rather, systemd will continue to re-arrange, and break, stable subsystems without warning to the user communities.
Come on. It is the distros that dictates exactly what features systemd have turned on, and how the FH layout is. There are no systemd developers sneaking into the distro repos at night and secretly turning on features without documenting them.
You may not follow the eg. development mailing list of your distro where such things are discussed, nor care about reading release notes. But Linux distros have always evolved this way, with new ways of doing things, often from release to release.
OTher examples abound. For example, the new replacement for /etc/resolv.conf says at http://www.tin.org/bin/man.cgi...
Note that /run/systemd/resolve/resolv.conf should not be used directly but only through a symlink from /etc/resolv.conf.
I fail to see how this in any way breaks any stable subsystem. /etc /run requirements is, this is a new feature, not a change of something old and stable. So it doesn't break anything at all.
1. systemd-resolved is an entirely optional feature. If the distros don't want it, they can just turn it off.
2. Whatever its
I really don't think you can give just a single non-contrieved example of how systemd are breaking stable subsystems for users without warning.
While more documentation is always better, I think the systemd project is way better documented than most other software projects, and certainly much better documented than many people think it is.
I'm not even going to mention what happens if you accidentally follow generations of sytem standard behavior and install NTP alongside an active systemd NTP daemon. It's not a pretty site, and the accumulated clock management confusion if they're not pointed to the same NTP servers can break Kerberos authentication in startlingly short order.
Running two NTP-clients at the same time was an amazingly stupid thing to do even pre-systemd. AFAIK, systemd per default tries to avoid this by using "conflict" statements in the service files, so if one NTP client or daemon is run, the others are suppressed.
I don't think eg. SysV
The systemd requirements for the file system is documented here: /run is the only new thing here compared to FHS 2.3
http://www.freedesktop.org/wik...
These are rather simple requirements and probably very close to the old FHS 3.0 proposal, I think
AFAIK, all other changes of the FHS are merely suggestions like getting extra OS container features when using btrfs, and in case of stateless boot, something for the future. Systemd will continue to work for Linux distros that don't care for stateless boot etc.
You mean boot logging? There are numerous ways to solve this which would have made more sense, including a small change to the linux kernel to make it stop throwing away boot logs when they are large.
The kernel ring-buffer size can be user specified, so no need to ever lose such messages. The problem is that during boot, only messages directed at /dev/kmsg is recorded until rootfs is mounted and syslogd is running. This is hard to do something about since it is a kernel limitation that only one device at the time can own /dev/log. journald catches all messages, including those to /dev/kmsg and /dev/log, and it does this from early boot. So a journal based system will always contain more info than a pure syslog system.
So use syslog-ng, which can be used in place of syslog without changes to other daemons, unlike journald. Which again, is the reason we're even having this conversation.
Regarding silently dropping messages: AFAIK all syslog daemons does this per design (or lack thereof). The main problem is that calls to syslog() would be blocking if implemented in a safe manner.
Wait, you don't care about human-readability, but you don't want the logs to become unreadably long? Make up your fucking mind. You're just making shit up, I can tell because that shit is contradictory.
You misunderstand; a reason why syslog messages doesn't contain monotonic timestamps etc. is that it makes the log entries extremely long and hard to read for humans. It is a kind of limitation of the "design".
journald doesn't have such limitations since it is field based, and the output format is easily adapted to what view you want. If you only want wall time stamps, you can get that, if you want monotonic time stamps, you get that, and only that.
So with the journal, you can add more fields in the future without making the log entries unwieldy, something impossible with syslog.
Regarding human readable: I only care about that for the subset of information I intend to read, and this is exactly what journalctl does. That way you can have logs that are easy to parse for both humans and machines; each consumer gets exactly the view of the data that they want in a form they can use.
that text logs simply don't scale
Still not an argument against text logs, only an argument for also implementing binary logs. And I wouldn't put them into their own shitty files with their own shitty format, I would put them into an RDBMS. Putting them into crappy, easily-corrupted binary log files is just asking for trouble, and guess what? It's causing trouble.
Yes, the scaling thing is a good argument against using flat file text logs. You really need indexes when analysing such huge amount of data, which is exactly what people are doing at the moment; they invest in often proprietary commercial solutions in order to massage their flat text file log data into something usable.
RDBMS have been tried for a long time and they are not a good general purpose solution for local logging. They are also much more prone to corruption than the journal's simpler binary storage format.
What? So now you don't like flat files or regex? Why don't you just fuck off to Windows and let us have our Linux? You clearly don't like Unix.
While regex is a powerful tool, the old joke about trying to solve a problem using regex, and then discover you now have _two_ problems, is true too. Discounting the many variations of regex that each program like grep /egrep uses, it simply becomes unwieldy when you are using regex epressions that are +120 characters long. Making ad hoc queries with regex really is painful when the complexity rises beyond the basics. The systemd journal is so much better regarding this since it is field based.
I assume you are talking about FHS 3.0 here, but remember, that is only a proposal, not a standard. Also note how the FHS 3.0 process has been stalled for years.
The bottom line today is that there is FHS 2.3, and then everybody doing their own thing , perhaps with some cross-distro cooperating. The end result is no common formal standard for the FHS among the distros.
Personally I don't think there will be a FHS 3.0 until the dust has settled regarding the current layout reorg,
The reorg of the FHS that the systemd developer are doing, are caused by good technical reasons. Unix and later Linux have had several of these reorgs over the decades, and it is a process that probably never stops.
So until the FHS reorg to make about OS containers, and stateless boot work in a sensible manner, the Linux FHS will be in a slight flux. Not a big problem really.
I guess at some point somebody will make a FHS 3.X out of what the systemd distros does at some point in the future. Personally I prefer this approach of practise getting formalized into a standard. That way you are sure to have both a good working standard, and a standard that people actually use.
Judging by this thread, it does give at least one advantage, that being the ability to at least partially recover and analyze corrupted logs.
The same is very much true for systemd's journald files. They are just appended text files with another delimiter +index, so in worst case a hex editor or "strings" may read them.
"journalctl" is designed to cope with corrupted journals and read them very well. It should be noted here that corruption of journal files as discovered by the --verify option, mostly are very trivial errors like a monotonic timestamps of the wrong length etc. Such errors only affects a single log entry, and often even that log entry can be read without problems. Despite using journald since it became standard on Fedora (F15) I have never experienced any data loss nor having journal files that I was unable to read.
Plain text log files, like the original syslog(3) interface was simply bad ideas that Linux inherited.
No, that was a brilliant idea that Linux inherited.
Come on, syslog(3) wasn't even signal-safe and designed as dog slow too. It was crap. People have been working ever since to make syslog thread safe (still interim), etc.
What classic syslog problems? I haven't had a problem with log rotation, really the last remaining issue, in approximately forever.
That syslogd can't log until rootfs is started and most of the machine is up, that it is dog slow and silently drops messages under load, that there is no common API for accessing the content of the data, that the data is basically free form, deviating from machine to machine (an industry exist dedicated to this problem alone) also making it impossible to make a distro agnostic GUI log viewer, that meta-data like monotonic timestamps are basically impossible to add without the log entries becoming unreadable long, that text logs simply don't scale (again, an industry exist to solve this problem, see Splunk etc.), that using regex becomes intolerable and unwieldy above a certain point, that in order to grep for something in the log file, you basically already have to know the name of the function and what the problem is, this makes syslog log files basically impossible to use for newbies, so they have to resort to trawling through the logs like reading a book.
Really, flat file, un-indexed log files meant to be read by a human SA is something that belongs to another era. What the present and the future needs are binary, structured and indexed logs with a stable API that can be parsed by machines and that works even at mega-scales.
And systemd's journald is designed to be 100% backwards compatible with syslog, so if your setup have legacy need, it is trivial to get a systemd distro to use legacy syslog text logs.
And syslog gets the messages late, and may not get them at all.
Well, these are know problems with syslogd implementations; they silently drop messages under even moderate load. It is their ancient design, so using pure syslog you will never know whether something was logged or dropped. With journald you will at least be informed if log messages are dropped. The latter should be a rare occurrence since journald is so much faster than syslogd.
journald goes to great lengths by using per app log rate limiting in order not saturate syslogd, but the inherent limitations in syslog means that this will always be an best effort. Still, it probably isn't worse than using syslogd alone.
I'm curious: why not just make an indexed database logging system on top of, rather than instead of, text logs? i.e. write out log entries in some structured but still text-only format in one file, and then maintain an efficient binary index of entries in a separate file. That way you get all the nice search etc, while still being able to grep the raw text if you really need to.
There are several problems with solution: it isn't backwards compatible with syslog (journald strives for 100% backwards compatibility).
It doesn't solve the problems that combining meta-data with log entries in flat text files, makes the text logs hard to read for humans since the log lines become far too long and complex.
It doesn't really gives any advantage of pure binary data. Remember grep, sed, tee and every other standard Linux text tool works with systemd's journal through the standard Linux/Unix concept of piping, so there is nothing lost by going pure binary.
The reorg of the FHS is quite needed in these days of OS containers, sandboxing and virtualization, so while placing stuff under "/run" is something new (part of the proposed FHS 3.0 standard btw), there are good technical reasons for it, including that /run is guaranteed to available from early boot (initramfs).
I believe this re-org even predates systemd.
Thanks for the links. Last time I looked into this, the remote features were still a feature request.
Maybe I'm not understanding you, but I thought journal was meant to be superior to syslog and eventually supplant it.
Well, yes and no. journald was meant as a total overhaul of _local_ Linux logging. It provides "metal-to-metal" logging by being able to operate in initramsfs before rootfs is mounted, and pivot back to initramfs (Dracut) after rootfs has been unmounted. It collates all logs into a single view, instead of the myriad of scattered legacy log files (including binary logs like utmp etc).
It also vastly improve the way programmers can use logging, since syslog(3) etc. have so many limitations regarding speed, messages per seconds etc.
It also provide structured logs with plenty of metadata like monotonic timestamps that are troublesome in syslog text logs.
Rsyslog and similar syslog implementation works both as a _local_ logging system _and_ as a remote log sink.
journald doesn't work as a general log sink for remote logging, nor does it provide DB interfaces and various other similar plug-ins. So for general log-sinks, Rsyslog was always meant to be choice.
journald was also designed to be 100% compatible with syslog since it was obvious that legacy requirements would make that a necessity. So again, for legacy setups, syslog was always the designed choice to run on top of journald.
The issue is not with systemd corrupting the binary logs, or with the filesystem corrupting the binary logs, but with the fact that they are -binary- logs. A log file should be an ordinary text file. Nothing more, nothing less.
The glaring problems with flat file, unstructured text logs have been discussed for decades now. Plain text log files, like the original syslog(3) interface was simply bad ideas that Linux inherited. The main commercial selling point with any syslog implementation today is exactly that you can avoid several of these problems by using a DB.
An overhaul of the creaking legacy Linux logging system have been needed for a decade at least. Remember, the Rsyslog project was started exactly to fix such problems. They failed partly because distros and developers don't care about working together for a common goal and because there was no other central developer hub who would take the initiative.
systemd have finally fixed so many of the classic syslog problems. I don't care if you think syslog is perfect and never can be improved, and Rsyslog should be avoided for providing binary DB storing option, but please don't think that other people agree to this postulate.
And systemd's journald is designed to be 100% backwards compatible with syslog, so if your setup have legacy need, it is trivial to get a systemd distro to use legacy syslog text logs.
Bug report: Logs keep getting corrupted and cannot read them at all
Rejected with reason: Delete the corrupted logs and move on
I doubt that you can provide a link for that. Perhaps you are hinting to the RFE about "pruning" journal logs containing "corrupted" field values?
SystemD - works well when it works, fails spectacularly when it fails.
That is part of the Unix philosophy: (Rule of Repair : http://en.wikipedia.org/wiki/U...), or "if you have to fail, fail early and fail noisily". systemd is build around that; so if disks specified in fstab doesn't show up, it won't boot but give a rescue shell instead. This prevents silent data loss; a classic example of "Rule of Repair".
Been using systemd on Fedora for several years now; it has been rock solid. systemd is just so much better than any alternative out there, and for every release it just keeps on getting better. I love its logging system and the easy and natural way it handles OS containers and system service management.
Stateless boot is nothing new in Linux. What the systemd developers want is to make Linux generic enough to make stateless boot work for those who needs it, instead of each and every developer group making their own proprietary tools. Stateless boot are mostly for embedded but also allow for some interesting network booted options.
There is a lot of work to be done before it is possible, but with systemd people are at least working on it.
Regarding the FHS and where to put things like using /run.
systemd is only doing the sensible thing here. The FHS 2.0 is ancient, so the systemd way of doing things is likely to become FHS 3.0, and then it will be a Linux standard.
SysD's binary logs have another, serious flaw: they are not designed to be sent over a network. This has been an intrinsic part of syslog for a looong time.
Of course you can push journald logs over the network in several different ways, and you can also receive them using the journald. But the systemd journald isn't substitute for syslog, but meant to complement it. Just use rsyslog when it is relevant, or just systemd's journald when that suffices.
Here is the document info for "journal export format"
http://www.freedesktop.org/wik...
Here is how to receive or pull systemd journals across the network: It can either pull log requests or passively wait for a connection:
http://www.freedesktop.org/sof...
Here is how to serve journald log entries across the network:
http://www.freedesktop.org/sof...
Are you really saying that a feature of pulseaudio is the ability to disable pulseaudio and use something else?
Yes I do. It is great that you can use exactly the program you need, even if their usage normally are orthogonal. Please note that PA doesn't disable itself, it just automatically suspend until needed again. PA and jackd where made for different reasons and solves different problems.
I see a lot of posts complaining about anti-systemd zealotry here, but clearly there is a lot of zealotry all around. I can't recall any other topic in Linux history that has been this divisive, and that is part of the problem.
Well, some people thought they could stop the systemd progress by waging a negative campaign against it and its developers. That of course was a predictable failure.
Personally I don't care whatever init system people run. Even if I find e.g. SysVinit inferior to systemd I have no problem with people using it, and they owe me no explanation for doing so.
Systemd-opponents don't agree upon what they do like, they can only agree on what they don't like. This make their tactics so rabid and aggressive when attacking systemd, its developers, the distros that uses it and even its users. They simply don't have a positive alternative they can agree upon, so they are left with negative attacks.
Yes, and most of that existing industry will have to rewrite everything from scratch without file format compatibility.
journald has excellent export capacity, it is trivial to convert everything into JSON or similar industry standards without tedious data massaging. Unlike the many slightly varying syslog implementation with deviating output, journald output is always predictable since has a stable API and is structured around fields. This is a massive win for everybody, especially those who needs to put their logs into full DB's (for legal or whatever reasons).
"Another problem with text log files are their limited ability to incorporate much meta-data, since that makes the log lines unreadably long. So having full microsecond precision and monotonic timestamps is a problem."
Really? Here's the microsecond monotonic timestamp on my current box: 242841.31760194. That's 15 bytes. A 64-bit binary timestamp is 16 bytes. The text version is shorter! And given that the vast majority of usages do not require microsecond resolution (which isn't even really accurate, anyhow), the text approach would lead to substantially _smaller_ files.
Adding full microsecond precision and monotonic timestamps and wall time timestamps + all the other interesting meta-data and the log message itself, are making the text log entries extremely long and unfriendly to read for humans. With journald you only see such meta-data info if you ask for it. Much better solution. And if you are concerned about log sizes, journald has transparent compression. With text logs you need to unpack or use specialised tools to manipulate them if compressed.
Plain text logs sucks for so many reasons, their only redeeming feature seems to be you don't need to pipe them through a tool in order to read them.
I have much respect for the Rsyslog developers, and I remember how I hoped they would fix the most of the glaring syslog problems when they started out a decade ago and in their own way tried to solve the problems that journald does. They partially failed, not because they weren't good, but because changing certain stuff in Linux userland is extremely hard.
sysvinit is crude and simple because it pushes all complexity out of init into userspace and because it offers no worthwhile init functionality. That is why Linux endured decades of remote exploits since SysVinit wouldn't take responsibility for dropping daemon privileges or do socket activation like systemd does, so daemons didn't have to use complex and error prone privilege dropping code just to get a port below 1024.
With systemd and new style daemons you can just avoid all that crap in the daemons. Much better to have one central place for security code made by experts than each and every daemon needs to compensate for the abysmal lack of functionality and security features of SysVinit.
I don't care if some people think that SysVinit is the best init system ever made and that it has no errors and can't be improved and Linux should use it forever and ever. We happen to be a lot of people who don't see it that way, which is why we like systemd and chooses distros that support that.
Or, you know, pretty much noone really cares about having indexed and structured log files. Separating into log files based on hostname and application name has worked really well for me for ages. And, because you're still going to have a human write the log messages, you're still going to have to eyeball the output at some point.
Great that syslog works for your usercase. I doesn't for me and so many others. There is an entire industry build around the need to beat syslog files into some sort of structure and give them an index so you can do number crunching on them.
These days log files grows so fast and is so important for business decisions that only machine parsing can do the job properly. The journal is excellent in that regard too.
Another problem with text log files are their limited ability to incorporate much meta-data, since that makes the log lines unreadably long. So having full microsecond precision and monotonic timestamps is a problem.
Yes, at some point a Mark I eyeball is going to look at a tiny subset of the data, but these isn't a problem either with systemd's journal; all the usual standard Linux text tools work through piping. And there are language bindings up the yazoo so you can manipulate it the journal files directly with Python scripts etc. It is so easy to extract the data that you need.
Personally I find syslog text files pretty hostile to newbies; unless they grep well and know a variation or two of regex, they are bound to just plow through the logs trying to identify some error. Really, many of them even resort to editors for reading log files. And the DE developers can't make a distro agnostic GUI log viewer better than "less" because there is no stable text log API. With systemd's journal they can because it has a stable API and language bindings etc.
With journalctl it is so easy to make even powerful extractions like; just show me logfiles generated the last 5 minutes (journalctl --since -5m) or all errors with syslog level "warning" and above created since this boot (journalctl -b -p warning). There is tab completion of every command option and every journal field and ususally even of the values of those fields. No need to memorise obscure arguments and subsystem names.
Sure it is possible and has been done before. My point is that no distros seem to use such a init-system and that no other init-system have anywhere near all the nice features that systemd has.
Rather typical LKML discussion in my opinion, but blown out in all proportions by the systemd-haters because they wrongly think that Linus don't like systemd nor its programmers because of this.
No it wasn't. Linus made his views on when this stuff will get merged, if ever, abundantly clear. It's not blown out of all proportion. That's what was written.
You don't seem to have experienced many LKML flame fests before, so I think you will be sadly disappointed when Linus merges kdbus.
Linus Torvalds have long signalled that he wants kdbus in the kernel.
No he hasn't you liar. I'm afraid this propaganda ain't going to have any effect whatsoever on his views.
Have you even read the lkml? Linus is deep into discussion of the code and even defended kdbus design decisions. He would never do that if he didn't intend to merge it as some point when enough people thought it was good enough.
Also notice how Linus appeared in the first kdbus patch threads. He didn't comment on the code, so the only reason he mailed was to signal to everybody that he followed this, and this was just business as usual.
Sure, the kdbus maintainers will have to do a good enough job in fixing problems before it is merged, but as it looks now, this doesn't seem to be a problem.
kdbus is good stuff that can benefit everybody, and while it isn't a shining-new general purpose IPC that some kernel developers want, it does solve real world problems, and that is exactly what Linus think his kernel is all about.
You know what's different about consumer and pro audio? Nothing. As it turns out, consumers also want high-fidelity audio, and they want to be able to mix multiple streams. The only difference lies in what the industry is willing to sell us.
This not about spending money or HiFi, but on what you use the audio system for. On consumer audio you want low cpu and battery drain and accept the tiny latency associated with this; that a music stream start 20ms second later doesn't matter. It is also important that features such as streaming, bluetooth head sets integration etc. works out of the box etc.
With Pro audio latency is king. cpu and battery drain means nothing.
PA can provide both things, by suspending out if your need shift from consumer audio (PA) to Pro audio using jackd. No need to dedicate your box to either; they can both work for whenever people need different solutions.
Actually, pulseaudio has notably drained batteries in the past, because shitcode.
A bug from 2007 on Ubuntu that famously fouled up their PA implementation, hardly a serious complaint. This 2012 shows how PA even beat Androids audio stack (Audioflinger) regarding CPU and poweruse.
http://arunraghavan.net/2012/0...
There are several such benchmarks that shows how good PA is in this regard.
Unlike you I have several years of experience with systemd. You just seem to parrot the party line propaganda from the tin-foil hat systemd-haters.
Don't run with the pack. Try to actual examine systemd by reading the documentation and get some experience yourselves. It takes some time to really understand how systemd works.
Did you actually read my post which was about a very specific point about binary versus text versus INDEXED text log files
systemd's journal is a basically a textfile that uses another line delimiter and have an index. Try "strings" on it, or a hex editor.
Well if so, that's new. It never used to be the case that the binary format was stable. So, I'm going to ask you to provide a link to the description of the binary structure.
Here is is the systemd interface and portability chart. Notice sd_journal and the journal file format:
http://www.freedesktop.org/wik...
Here is the file format;
http://www.freedesktop.org/wik...
There are loads of other developer information about accessing the journal with Python language bindings etc.
Jesus fucking H. Christ on a stick what is wrong with you? I specifically went into detail about how noisy the pro-systemd people like you insist on a dichotomy between unstructured text files and binary file formats. There is no fucking dichotomy you moron. If you actually read my sodding post instead of going off on a rant then you would actually become marginally more informed and stop spouting the same old shit repeatedly.
Well, you are not the first person to think about how to extend text logs with some kind of index, and it would be nice if it could have microprecision timestamps, and monotonic timestamps like the journal does. The problem is that is unworkable.
As I said, several projects have tried to fix this without any luck for the last couple of decades.
Another problem with such a solution is that standard Linux text tools like grep, tee, sort etc. won't work with such a solution where the index isn't part of the text file, and metadata like microsecond time stamps tend to make the log lines unreadable long.
With systemd's journal all standard text tools work with the index and metadata, so you can grep monotonic timestamps by a simple pipe.
If it was possible to have indexed, and structured plain text log files that didn't require a special "translator" in order for the standard Linux text tools to work, it would have been implemented a decade ago.
The whole point I was trying to make that looking back on sound on Linux in the days before ALSA and PA shouldn't be done with rose tinted glasses. Claiming that, like the OP does, that everything went backwards with the invention of ALSA and later PA is just absurd. Those where the bad days regarding sound on Linux. There is nothing to miss from those days, including ISA sound cards.
With ALSA and later PA there were actually people working on the Linux soundstack and developers that could coordinate bug fixing. And PA was a huge help for DE developers that where so tired of trying to maintain their own sound servers and dealing with really low level kernel stuff like drivers, when all they wanted to do was to develop for the desktop.