Although the language itself isn't truly, truly bad, the only thing that made it tolerable was a library like Qt. STL and Boost are so shocking it isn't even funny, as well as the attitude amongst many that they are some kind of 'standard'. That's where C++ really did go off the rails.
You misunderstand; there _isn't_ a "fsck" option for systemd journals and there won't be because that would be equivalent of "editing the log files".
No, I'm not misunderstanding. You will need a repair option to make the logs readable. If there is not, they're left in a corrupted state which is just plain daft and is no use to anyone or you're leaving it to people to trawl through whatever format documentation there is to piece it back together. I have *never* had to do this with a text log file.
I don't know where this 'editing' notion comes from. You can't edit them......they're binary. I'm relying on journald to keep these in a consistent format. Either give me a consistent and readable log file or get the fuck out of my way. Dead simple.
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
You simply don't understand how journald works. Let me explain again
Ahhhh, yes, the 'You don't understand' angle........
The claptrap you've written there has nothing do do with the original comment and reply that I wrote - that somehow this corruption occurs because of other issues, namely disk corruption. This is happening on an otherwise perfectly running system.
systemd marks certain log entries as "corrupted" if eg. they are plain wrong, that is the client is sending obviously wrong or malformed log entries.
As a sys admin I'm not interested in what it is doing. It is producing corrupted logs regardless I'm afraid. I don't care how or why, but it does greatly concern me that journald is somehow deciding what is or isn't corrupted log data. Neither you or the tin pot systemd crew have any idea what logging actually means. That's not something for a logging system to decide and it's amusing that this is actually sanitising and 'editing' logs.
Once again, either give me a consistent and readable log file that I don't have to wash through *any* further tools or get the fuck out of my way.
Indeed. And if you pipe your logs directly into a database, you get this little thing called ACID, which apparently the systemd folks have never heard about. Their current solution is about the worst one possible. Closed, proprietary, unreliable. A "fair weather only" system that falls over dead as soon as the conditions are not ideal anymore.
It's bloody laughable isn't it? I can have the low-level text logs available and I can pipe into a database and aggregate and search as I want with a mature piece of software. But no, what we get is a completely new and totally immature piece of software and a new format that rolls over when there are problems.
Really, are you unaware how common it is to aggregate log files in databases? This is a major selling point for Rsyslog that it is able to do so.
No, I am aware but you're putting up a strawman there for obvious reasons. The point is that the lowest common denominator is always the plain text log file for reasons that this bug report makes clear.
However, given that we can have the option to drop logs into databases or pipe and arrange them how we want kind of begs the question why we need the binary journald format. You're going to have difficulty squirming your way out of that one.
Ah, so now facts and reality is propaganda.
The bug reports on corruption ain't a piece of propaganda I'm afraid. You can shout from the hills that it is nicely documented as much as you want, but this remains a problem I'm afraid. I'm not the slightest bit interested in trawling through documentation so I can piece back together a log file that the logging system itself has corrupted. It's wheel spinning.
It is when the corruption means squat for the ability to actually read the logs.
1. journald is causing this corruption.
2. What makes you think that plain text logs can't be read in the event of any corruption (which I have seen once in a blue moon by the way)? When you have a binary format the clue is in the name - you can read it or you can't. If you can't read it then it needs repaired, and one has to hope that the repair works. If it doesn't good luck.....
For some reason you seem to think that "corruption" discovered by "journalctl" means the logs are unreadable, but as explained, they are often marked "corrupted" just because a single field value in a single log entry was discovered as impossible. This doesn't mean that the log file can't be read.
In other words you haven't the slightest clue as to what this corruption is or where it's coming from. I can count on the fingers of one hand the number of times I've seen corruption in plain text logs and all of them have been down to hardware failure. I have always had a fighting chance of piecing together the low level plain text logs to find out what happened.
As a sys admin I really don't give a shit and I'm not the slightest bit interested in washing it through a fsck system once I actually discover it or having to trawl through its format documentation so I can piece it back together. It's retarded.
Quite apart from Freedesktop and systemd documentation generally being a constantly moving target.......
I don't care what you think, as long as you agree that the whole notion of not being able to use standard Linux text tools like grep together with systemd's journal is just plain wrong and a total non-issue.
It is when I have to pipe it through yet another application that generates corrupted logs.
There just aren't any good arguments against structured, indexed log files that can be programmatically accessed and has rich meta-data.
Apart from the fact that we can already do that (and we can use far, far more mature software than journald), and we can also have the low-level text logs available as a last resort to piece together what happened, I don't have to run a fsck on them and I don't need to trawl through its format documentation so I can piece them back together in the event of inevitable issues. No, nothing apart from that;-).
Yeah, especially odd that the systemd folks didn't just adopt it. There'd be a LOT less to bitch about in the Linux "community" if they'd simply ported an already-stable daemon.
It would have been if they were simply developing an init system. However, systemd isn't one.
What a load of claptrap. People log to log files for reasons of the lowest common denominator. We have things called 'databases' for this kind of stuff and there's perfectly good reasons we don't use them for logging which should be obvious to anyone with an ounce of sense.
Since the journald binary logging file format is stable and fully documented....
I sincerely hope Red Hat and Lennart is paying for this piece of contrived PR.
The arguments against it are based on contrived scenarios
Corrupted logs on a perfectly running system isn't a contrived scenario.
....through the Linux/Unix concept of piping.
Piping eh? Wow. You give the impression you haven't heard of it before;-).
When you pretend to be a foolish version of someone else......
It's not terribly hard I'm afraid.
If you really have a valid point to make, argue against your opponents' best points. Don't make an ad hominem attack against a caricature of the opponent.
If only you could hold everyone in the 'discussion' to such high standards.........
This is the merest tip of the iceberg of the problems we're going to have.
There are a lot of 'difficult' developers in the community that we've all heard of or had to deal with but no one I knows compares to the obvious issues that Lennart seems to have.
So, your issue with systemd is that it works just like syslog, only that systemd makes the issue more transparent by showing corrupted logs instead of just silently drop them like syslog?
No:
1. This is happening *a lot* more frequently than on syslog based systems, and on seemingly perfectly running systems at that.
2. With a binary log file you've lost the whole thing. Unless the fsck happens (I still can't get over that - fsck on log files) and works then you've lost it. Putting together a partial text log is certainly possible. I would expect we will see a whole lot of third-party tools cropping up in the future to attempt to 'solve' these corruptions.
In the case of a failing disk or file system, the likes of rsyslog would be worse off because the daemons don't even know the logs are corrupt.
No, you're actually worse off with journald. Piecing together a partial text file is actually possible. However, if you lose a corrupt binary file you lose the whole thing. It's all or nothing.
" Obviously nobody with the least bit of clue would expect logs to be cleanly written on failing disks, hence this is not even a subject for discussion" exactly so if rsyslog logs are corrupted its expected but if journald logs are corrupted, its the fault of systemd, great logic
For the umpteenth time, these corruptions are not being caused by failing disks, but that is the only thing that you have here so things are understandably getting a little desperate.
These reports are happening on perfectly running systems and they are a lot more frequent than they should be from a logging system.
Utterly false. The idea that syslog doesn't have corruption is false.I have seen syslog corruption many times. Whether it's truncated lines, merged lines because half of an old truncated line has a new message appended, blocks of 4k zero bytes, or single bit or single character errors.
Ahhhh, the 'syslog is just as bad' defence. We're going to see more of this kind of defensive butt covering as the inevitable problems continue to crawl out of the woodwork with the smorgasbord of systemd components.
In particular if a syslog file is truncated mid-line by either disk full, system failure, filesystem bug or drive bug, the best thing syslog could do when it resumes after boot would be to rotate log files at that point, instead of appending to the truncated file.
I can literally count on the fingers of a lot less than one hand the number of times I've seen this, and they're all related to a rotation procedure not being done properly, a problem in the software you're logging for or a catastrophic failure of the system. syslog hasn't found the fountain of eternal youth, so one can only apologise for that, but I have never heard of the kinds of corruption caused with it on a perfectly running system as journald has chalked up.
These are quite rare, but not so rare that I haven't seen them maybe 50 times in 20 years in Linux syslog files.
Yer. Go figure.
I have no opinion on systemd particularly, but with regard to this single thing of rotating logs on detecting corruption, instead of attempting to patch the corruption or continue appending to the file, I think the right decision was made, from the perspective of an admin wanting the best available information after a problem.
Personally, I would prefer an investigation into why logs are being corrupted like this and a willingness to take it seriously rather than a 'corruption happens, rotate' attitude, but I'm just funny like that.
Totally sensible solution, why would you edit log files? They are not meant to be changed.
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy. I don't know where editing files was mentioned.
Or what if the log corruption is caused by a failing disk where the bad blocks are spreading like wildfire? If you silently delete such entries and files, you delete the evidence for that something is wrong.
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
The reason why people discover "corrupted" journal log files is because systemd's journal have integrity checks, unlike syslog who doesn't and where the admin therefore never will know if there are spurious or invalid log entries unless he happens to stumble upon them by chance.
Seriously, if you really care about each and every log entry, you should never have been using syslog anyway; it simply silently drops messages under load, and both local and remote logging are inherently unreliable. Yeah, I know Rainer have made a signal-safe syslog library, but does anybody actually use it?
What a load of utter claptrap. Just when you think the brain damage and denial can't get worse........it does, in spades.
A logging system corrupts logs, in a format specified by it, because it checks their integrity. I can only suggest checking into a clinic somewhere.
Journald is is not forced. Like everything in systemd the services are modular, and you can use them or not. Debian for instance does not use journald.
Repeating this tripe like a broken record won't make it true I'm afraid. Yes, people are using rsyslog....so they can avoid having their logs corrupted.
90% of systemd's suite of utilities are not part of init, and not even required or used by most people and their distros.
Strange that you've just described a very bog standard duck and pretty much repeated verbatim the OP's problem. This is kind of strange for something that purports to be an init system, no? I think you need to sit down for a little while, lay off the keyboard and have a think about the ramifications of what you wrote there.
It does, though, make containers and cloud a lot easier for those who want to do that. Certainly makes administration better on servers.
No it doesn't - and especially in the latter case. I've been managing containers for quite a long time and frankly, I'm not the slightest bit interested in companies who want to fuck up the Linux userspace so they can cram as many hapless customers on to a piece of cloud hardware as they can. That is simply all that it is for.
Well, of course he would as ConsoleKit was the current component at the time, so he had to. That doesn't alter the fact that ConsoleKit is being deprecated and Lennert has said as much. Anyone hoping to depend on it to get around systemd is going to be disappointed.
Slashdot Mirror
Looks like the memory card on the the black box has been "lost". Is this true? How is it possible if the black box is designed to withstand 3500 g ?
It's not. One being damaged, once in a blue moon. Two, as Oscar Wilde would have put it......
Also, why isn't data streamed to ground stations nowadays? And why black boxes do not float ?
Good question.
We know absolutely nothing, despite the pathetic attempts to convince us otherwise.
The data recorder would have corroborated everything but of course, that's damaged with its data card missing.
Best of all worlds. And guess what, in the grand scheme of things, the price is a drop in the bucket compared to salaries.
Alas, everyone has to pay salaries and then it's a question of how much you have left to get anything done. Nice try though.
Although the language itself isn't truly, truly bad, the only thing that made it tolerable was a library like Qt. STL and Boost are so shocking it isn't even funny, as well as the attitude amongst many that they are some kind of 'standard'. That's where C++ really did go off the rails.
You misunderstand; there _isn't_ a "fsck" option for systemd journals and there won't be because that would be equivalent of "editing the log files".
No, I'm not misunderstanding. You will need a repair option to make the logs readable. If there is not, they're left in a corrupted state which is just plain daft and is no use to anyone or you're leaving it to people to trawl through whatever format documentation there is to piece it back together. I have *never* had to do this with a text log file.
I don't know where this 'editing' notion comes from. You can't edit them......they're binary. I'm relying on journald to keep these in a consistent format. Either give me a consistent and readable log file or get the fuck out of my way. Dead simple.
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
You simply don't understand how journald works. Let me explain again
Ahhhh, yes, the 'You don't understand' angle........
The claptrap you've written there has nothing do do with the original comment and reply that I wrote - that somehow this corruption occurs because of other issues, namely disk corruption. This is happening on an otherwise perfectly running system.
systemd marks certain log entries as "corrupted" if eg. they are plain wrong, that is the client is sending obviously wrong or malformed log entries.
As a sys admin I'm not interested in what it is doing. It is producing corrupted logs regardless I'm afraid. I don't care how or why, but it does greatly concern me that journald is somehow deciding what is or isn't corrupted log data. Neither you or the tin pot systemd crew have any idea what logging actually means. That's not something for a logging system to decide and it's amusing that this is actually sanitising and 'editing' logs.
Once again, either give me a consistent and readable log file that I don't have to wash through *any* further tools or get the fuck out of my way.
Indeed. And if you pipe your logs directly into a database, you get this little thing called ACID, which apparently the systemd folks have never heard about. Their current solution is about the worst one possible. Closed, proprietary, unreliable. A "fair weather only" system that falls over dead as soon as the conditions are not ideal anymore.
It's bloody laughable isn't it? I can have the low-level text logs available and I can pipe into a database and aggregate and search as I want with a mature piece of software. But no, what we get is a completely new and totally immature piece of software and a new format that rolls over when there are problems.
Really, are you unaware how common it is to aggregate log files in databases? This is a major selling point for Rsyslog that it is able to do so.
No, I am aware but you're putting up a strawman there for obvious reasons. The point is that the lowest common denominator is always the plain text log file for reasons that this bug report makes clear.
However, given that we can have the option to drop logs into databases or pipe and arrange them how we want kind of begs the question why we need the binary journald format. You're going to have difficulty squirming your way out of that one.
Ah, so now facts and reality is propaganda.
The bug reports on corruption ain't a piece of propaganda I'm afraid. You can shout from the hills that it is nicely documented as much as you want, but this remains a problem I'm afraid. I'm not the slightest bit interested in trawling through documentation so I can piece back together a log file that the logging system itself has corrupted. It's wheel spinning.
It is when the corruption means squat for the ability to actually read the logs.
1. journald is causing this corruption.
2. What makes you think that plain text logs can't be read in the event of any corruption (which I have seen once in a blue moon by the way)? When you have a binary format the clue is in the name - you can read it or you can't. If you can't read it then it needs repaired, and one has to hope that the repair works. If it doesn't good luck.....
For some reason you seem to think that "corruption" discovered by "journalctl" means the logs are unreadable, but as explained, they are often marked "corrupted" just because a single field value in a single log entry was discovered as impossible. This doesn't mean that the log file can't be read.
In other words you haven't the slightest clue as to what this corruption is or where it's coming from. I can count on the fingers of one hand the number of times I've seen corruption in plain text logs and all of them have been down to hardware failure. I have always had a fighting chance of piecing together the low level plain text logs to find out what happened.
As a sys admin I really don't give a shit and I'm not the slightest bit interested in washing it through a fsck system once I actually discover it or having to trawl through its format documentation so I can piece it back together. It's retarded.
Quite apart from Freedesktop and systemd documentation generally being a constantly moving target.......
I don't care what you think, as long as you agree that the whole notion of not being able to use standard Linux text tools like grep together with systemd's journal is just plain wrong and a total non-issue.
It is when I have to pipe it through yet another application that generates corrupted logs.
There just aren't any good arguments against structured, indexed log files that can be programmatically accessed and has rich meta-data.
Apart from the fact that we can already do that (and we can use far, far more mature software than journald), and we can also have the low-level text logs available as a last resort to piece together what happened, I don't have to run a fsck on them and I don't need to trawl through its format documentation so I can piece them back together in the event of inevitable issues. No, nothing apart from that ;-).
With exceptionally good reason...........
Yeah, especially odd that the systemd folks didn't just adopt it. There'd be a LOT less to bitch about in the Linux "community" if they'd simply ported an already-stable daemon.
It would have been if they were simply developing an init system. However, systemd isn't one.
contain rich meta-data....
What a load of claptrap. People log to log files for reasons of the lowest common denominator. We have things called 'databases' for this kind of stuff and there's perfectly good reasons we don't use them for logging which should be obvious to anyone with an ounce of sense.
Since the journald binary logging file format is stable and fully documented....
I sincerely hope Red Hat and Lennart is paying for this piece of contrived PR.
The arguments against it are based on contrived scenarios
Corrupted logs on a perfectly running system isn't a contrived scenario.
....through the Linux/Unix concept of piping.
Piping eh? Wow. You give the impression you haven't heard of it before ;-).
You asked for it you idiot.......
When you pretend to be a foolish version of someone else......
It's not terribly hard I'm afraid.
If you really have a valid point to make, argue against your opponents' best points. Don't make an ad hominem attack against a caricature of the opponent.
If only you could hold everyone in the 'discussion' to such high standards.........
Because it's true.
This is the merest tip of the iceberg of the problems we're going to have.
There are a lot of 'difficult' developers in the community that we've all heard of or had to deal with but no one I knows compares to the obvious issues that Lennart seems to have.
So, your issue with systemd is that it works just like syslog, only that systemd makes the issue more transparent by showing corrupted logs instead of just silently drop them like syslog?
No:
1. This is happening *a lot* more frequently than on syslog based systems, and on seemingly perfectly running systems at that.
2. With a binary log file you've lost the whole thing. Unless the fsck happens (I still can't get over that - fsck on log files) and works then you've lost it. Putting together a partial text log is certainly possible. I would expect we will see a whole lot of third-party tools cropping up in the future to attempt to 'solve' these corruptions.
This is just.......basic.
Lots of things cause log corruption.
In the case of a failing disk or file system, the likes of rsyslog would be worse off because the daemons don't even know the logs are corrupt.
No, you're actually worse off with journald. Piecing together a partial text file is actually possible. However, if you lose a corrupt binary file you lose the whole thing. It's all or nothing.
" Obviously nobody with the least bit of clue would expect logs to be cleanly written on failing disks, hence this is not even a subject for discussion" exactly so if rsyslog logs are corrupted its expected but if journald logs are corrupted, its the fault of systemd, great logic
For the umpteenth time, these corruptions are not being caused by failing disks, but that is the only thing that you have here so things are understandably getting a little desperate.
These reports are happening on perfectly running systems and they are a lot more frequent than they should be from a logging system.
Utterly false. The idea that syslog doesn't have corruption is false.I have seen syslog corruption many times. Whether it's truncated lines, merged lines because half of an old truncated line has a new message appended, blocks of 4k zero bytes, or single bit or single character errors.
Ahhhh, the 'syslog is just as bad' defence. We're going to see more of this kind of defensive butt covering as the inevitable problems continue to crawl out of the woodwork with the smorgasbord of systemd components.
In particular if a syslog file is truncated mid-line by either disk full, system failure, filesystem bug or drive bug, the best thing syslog could do when it resumes after boot would be to rotate log files at that point, instead of appending to the truncated file.
I can literally count on the fingers of a lot less than one hand the number of times I've seen this, and they're all related to a rotation procedure not being done properly, a problem in the software you're logging for or a catastrophic failure of the system. syslog hasn't found the fountain of eternal youth, so one can only apologise for that, but I have never heard of the kinds of corruption caused with it on a perfectly running system as journald has chalked up.
These are quite rare, but not so rare that I haven't seen them maybe 50 times in 20 years in Linux syslog files.
Yer. Go figure.
I have no opinion on systemd particularly, but with regard to this single thing of rotating logs on detecting corruption, instead of attempting to patch the corruption or continue appending to the file, I think the right decision was made, from the perspective of an admin wanting the best available information after a problem.
Personally, I would prefer an investigation into why logs are being corrupted like this and a willingness to take it seriously rather than a 'corruption happens, rotate' attitude, but I'm just funny like that.
Totally sensible solution, why would you edit log files? They are not meant to be changed.
They aren't meant to be corrupted in this manner either, but apparently journald has a fsck option in an attempt to fix it. Crazy. I don't know where editing files was mentioned.
Or what if the log corruption is caused by a failing disk where the bad blocks are spreading like wildfire? If you silently delete such entries and files, you delete the evidence for that something is wrong.
Ahhhh, the whole 'disk corruption' misdirection. Yes, corruption could also be caused by solar flares, but this isn't, nor is it caused by a failing disk. It is caused by the journalling system itself. No investigation of this has happened, according to the maintainer 'it happens', we rotate and move on. Lunacy of the top order.
The reason why people discover "corrupted" journal log files is because systemd's journal have integrity checks, unlike syslog who doesn't and where the admin therefore never will know if there are spurious or invalid log entries unless he happens to stumble upon them by chance.
Seriously, if you really care about each and every log entry, you should never have been using syslog anyway; it simply silently drops messages under load, and both local and remote logging are inherently unreliable. Yeah, I know Rainer have made a signal-safe syslog library, but does anybody actually use it?
What a load of utter claptrap. Just when you think the brain damage and denial can't get worse........it does, in spades.
A logging system corrupts logs, in a format specified by it, because it checks their integrity. I can only suggest checking into a clinic somewhere.
Journald is is not forced. Like everything in systemd the services are modular, and you can use them or not. Debian for instance does not use journald.
Repeating this tripe like a broken record won't make it true I'm afraid. Yes, people are using rsyslog....so they can avoid having their logs corrupted.
Have a look at the title of this article and read it back a few times if you're not getting it.
Have look at the title of this article and read it back to yourself a couple of times.
Funny-looking duck you have there.
90% of systemd's suite of utilities are not part of init, and not even required or used by most people and their distros.
Strange that you've just described a very bog standard duck and pretty much repeated verbatim the OP's problem. This is kind of strange for something that purports to be an init system, no? I think you need to sit down for a little while, lay off the keyboard and have a think about the ramifications of what you wrote there.
It does, though, make containers and cloud a lot easier for those who want to do that. Certainly makes administration better on servers.
No it doesn't - and especially in the latter case. I've been managing containers for quite a long time and frankly, I'm not the slightest bit interested in companies who want to fuck up the Linux userspace so they can cram as many hapless customers on to a piece of cloud hardware as they can. That is simply all that it is for.
Well, of course he would as ConsoleKit was the current component at the time, so he had to. That doesn't alter the fact that ConsoleKit is being deprecated and Lennert has said as much. Anyone hoping to depend on it to get around systemd is going to be disappointed.