So, how is systemd supposed to add mongod's stderr to the journal, the mongo startup script is sending it to/dev/null.
It gets worse, even if we fix/etc/init.d/mongod to not redirect stdout/stderr it's calling the function "daemon" from/etc/rc.d/init.d/functions (sic) to launch the program, and that does:
# A function to start a program. daemon() {
# Test syntax. [...]
# And start it up.
if [ -z "$user" ]; then
$cgroup $nice/bin/bash -c "$corelimit >/dev/null 2>&1 ; $*"
else
$cgroup $nice runuser -s/bin/bash $user -c "$corelimit >/dev/null 2>&1 ; $*"
fi
So if/usr/bin/mongod writes to stderr then systemd never gets to see it because of the fucked up centos sysvinit scripts!
Yes, it's the sysvinit scripts that are preventing you seeing the message, not systemd!
As far as why people think systemd doesn't log, it's because it doesn't at least on any of the servers where I've looked. I don't know why people question that.
Well, I question it because it's not true on any machine I'm running systemd on. For example on the laptop I'm typing this message on:
# journalctl -p 0..3 | wc -l 149
What distros are you running that don't log error messages? Did you report it as a bug?
[root@localhost ~]# cat/etc/redhat-release CentOS Linux release 7.0.1406 (Core) [root@localhost ~]# systemctl start broken_systemd [root@localhost ~]# systemctl status broken_systemd broken_systemd.service - Broken systemd example
Loaded: loaded (/etc/systemd/system/broken_systemd.service; disabled)
Active: failed (Result: exit-code) since Fri 2015-02-13 13:59:11 CET; 23s ago
Process: 16880 ExecStart=/root/broken_systemd.sh (code=exited, status=1/FAILURE)
Main PID: 16880 (code=exited, status=1/FAILURE)
Feb 13 13:59:11 localhost.localdomain systemd[1]: Starting Broken systemd example... Feb 13 13:59:11 localhost.localdomain systemd[1]: Started Broken systemd example. Feb 13 13:59:11 localhost.localdomain broken_systemd.sh[16880]: Example systemd service Feb 13 13:59:11 localhost.localdomain broken_systemd.sh[16880]: Error that should not be thrown away Feb 13 13:59:11 localhost.localdomain systemd[1]: broken_systemd.service: main process exited, code=exited, status=1/FAILURE) Feb 13 13:59:11 localhost.localdomain systemd[1]: Unit broken_systemd.service entered failed state. [root@localhost ~]# exit
You are, of course, totaly correct. My suggestion was rubbish.
Think about it, the original "problem" was that launching the "broken_systemd" service with systemctl gives no error, and returns no error status.
But how could it? Imagine that "broken_systemd" runs for three hours and fails. Do you want systemctl to hang around until it fails. Or for three minutes? Or three seconds? Where do you put the arbitrary limit?
systemctl start starts the service. It will tell you if the service can't be started (if the program can't be found, or if there is a startup script (Type=forking) which fails) but it can't tell you that the process started will fail at some time in the future.
# aptitude install mongodb The following NEW packages will be installed:
libboost-dev{a} libboost-filesystem1.55.0{a} libboost1.55-dev{a}
libgoogle-perftools4{a} libpcrecpp0{a} libsnappy1{a}
libtcmalloc-minimal4{a} libunwind8{a} libv8-3.14.5{a} mongodb
mongodb-clients{a} mongodb-dev{a} mongodb-server{a} [...]
Let's simulate your problem:
# echo 11289 >/var/lib/mongodb/mongod.lock # systemctl start mongodb # systemctl status mongodb * mongodb.service - An object/document-oriented database
Loaded: loaded (/lib/systemd/system/mongodb.service; enabled)
Active: failed (Result: exit-code) since Thu 2015-02-12 23:31:44 CET; 5s ago
Docs: man:mongod(1)
Process: 11321 ExecStart=/usr/bin/mongod --config/etc/mongodb.conf (code=exited, status=100)
Main PID: 11321 (code=exited, status=100)
Feb 12 23:31:44 russia mongod[11321]: all output going to:/var/log/mongodb/mongodb.log Feb 12 23:31:44 russia systemd[1]: mongodb.service: main process exited, code=exited, status=100/n/a Feb 12 23:31:44 russia systemd[1]: Unit mongodb.service entered failed state.
Huh? "all output going to:/var/log/mongodb/mongodb.log"? Maybe that's why we see nothing in the journal?
In the mongodb.conf file I find:
#where to log logpath=/var/log/mongodb/mongodb.log
And the doc says:
--logpath <path>
Specify a path for the log file that will hold all diagnostic logging information.
Unless specified, mongod will output all log information to the standard output.
So, let's take out the logpath and try again, and now we get the error in the journal:
# journalctl -u mongodb -- Logs begin at Sun 2015-02-08 13:27:50 CET, end at Thu 2015-02-12 23:38:53 CET. -- Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] MongoDB starting : pid=11372 port=27017 dbp Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] db version v2.4.10 Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] git version: nogitversion Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] build info: Linux julia 3.16-2-amd64 #1 SMP Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] allocator: tcmalloc Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/ Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] ** WARNING: Readahead for/var/lib/mongodb Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] ** We suggest setting it to 256KB Feb 12 23:38:53 russia mongod[11372]: Thu Feb 12 23:38:53.858 [initandlisten] ** http://dochub.mongodb.org/core/ Feb 12 23:38:53 russia mongod[11372]: ************** Feb 12 23:38:53 russia mongod[11372]: old lock file:/var/lib/mongodb/mongod.lock. probably means unclean shutdown, Feb 12 23:38:53 russia mongod[11372]: but there are no journal files to recover. Feb 12 23:38:53 russia mongod[11372]: this is likely human error or filesystem corruption. Feb 12 23:38:53 russia mongod[11372]: please make sure that your journal directory is mounted. Feb 12 23:38:53 russia mongod[11372]: found 1 dbs. Feb 12 23:38:53 russia mongod[11372]: see: http://dochub.mongodb.org/core/repair for more information
So, configuration error. systemd "threw away" nothing.
> Where did you get the idea that this is "policy"?
Because complaints to the mailing list are ignored and the posters insulted. Here's one example I saved that I just ran on an updated CentOS to confirm that it is still a problem. This shows the systemd guys prevent the troubleshooting sysadmins need to manage our systems.
So, while we're busy accusing people of being liars, care to give a link to the "complaint to the mailing list" or bug report you made for this problem.
The problem with the exit code is that you're missing "Type=forking" in the service file. If I add that then I get:
# systemctl start broken_systemd Job for broken_systemd.service failed. See 'systemctl status broken_systemd.service' and 'journalctl -xn' for details. # echo $? 1
So it looks like you've got:
1. CentOS bug losing stdout/stderr 2. User error losing exit status
systemd shills may be liars, but systemd haters display symptoms of idiocy.
> the default mysql config writes error messages to a file
Are all of the systemd shills childish liars? That is not the default configuration for MySQL community edition from Oracle. The word error doesn't even appear in/etc/my.cnf.
# apt-get install mysql-server Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed:
libdbd-mysql-perl libdbi-perl libhtml-template-perl libterm-readkey-perl
mysql-client-5.5 mysql-server-5.5 mysql-server-core-5.5 [...] Setting up libhtml-template-perl (2.95-1)... Setting up mysql-server (5.5.40-1)... # grep error/etc/mysql/my.cnf log_error =/var/log/mysql/error.log
Why are anti systemd people so quick to accuse others of things like being "childish liars"?
Just found a very nice script that proves the problem:
# systemctl status -l broken_systemd * broken_systemd.service - Broken systemd example
Loaded: loaded (/etc/systemd/system/broken_systemd.service; disabled)
Active: failed (Result: exit-code) since Thu 2015-02-12 21:49:30 CET; 2min 0s ago
Process: 8831 ExecStart=/root/broken_systemd.sh (code=exited, status=1/FAILURE)
Main PID: 8831 (code=exited, status=1/FAILURE)
Feb 12 21:49:30 russia broken_systemd.sh[8831]: Example systemd service Feb 12 21:49:30 russia broken_systemd.sh[8831]: Error that should not be thrown away Feb 12 21:49:30 russia systemd[1]: broken_systemd.service: main process exited, code=exited, status=1/FAILURE Feb 12 21:49:30 russia systemd[1]: Unit broken_systemd.service entered failed state.
Looks like your version of systemd is broken.
I have
# dpkg -l systemd [...] ii systemd 215-8 amd64 system and service manager
Yup, it works correctly on Debian too -- everything that goes to syslog gets into the journal.
I've been making an effort to find some kind of confirmation or source for these "systemd doesn't log by policy" claims, which, AFAIK are the opposite of the truth (systemd/journald catch output to stdout/stderr that used to get lost).
So far the only one I've been able to get a handle on is the "mysql didn't start and there was no info in the journal" one, which appears to be because the default mysql config writes error messages to a file in/var/log instead of to syslog.
How is systemd/journald supposed to know that it should put the contents of random disk files in the journal?
The basic system of Unix permissions while being suitable in many cases, can be more difficult to manage in others. What if i want to write some rules to say that a particular user should not have access to certain directories, or to say they should only have access to certain directories.
If you want to get into windows ACL hell it's easy:
Tha's bizzare. Exactlty what log messages were missing?
Huh? That's systemd's policy. I've setup hundreds of CentOS 7 servers for customers since last Sept, and I can't remember ever seeing a start-up error in the systemd journal.
Not my experience.
AFAIK systemd logs everything it gets given.
Systemd logs everything that gets sent to syslog, everything that gets sent to the kernel log and the stdout/stderr of almost all things it starts (you can put "StandardError=null" in a service file if you want, for example ModemManager.service).
I suppose some lunatic could put DefaultStandardOutput=null, DefaultStandardError=null in system.conf, but that's user (or distro) error, not systemd.
If you don't understand why https is a very poor replacement for syslog over a network that may well be unreliable. I will assume your not really familiar with low level hardware and the failure modes. You do not seem to get the overhead and delay of TCP connections in general and thus why it's a poor choice for logging.
Well I just tried that on Debian, and the problem wasn't that systemd threw away any log messages, the problem was that the default mysql configuration writes its error messages to/var/log/mysql/error.log rather than the journal.
Removing "log_error =/var/log/mysql/error.log" from the mysql configuration gets exactlt what you want:
# systemctl start mysql Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details. # systemctl -l status mysql.service * mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql)
Active: failed (Result: exit-code) since Thu 2015-02-12 15:38:55 CET; 42s ago
Process: 2633 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 5482 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Feb 12 15:38:25 celtic mysqld[5839]: 150212 15:38:25 InnoDB: Operating system error number 13 in a file operation. Feb 12 15:38:25 celtic mysqld[5839]: InnoDB: The error means mysqld does not have the access rights to Feb 12 15:38:25 celtic mysqld[5839]: InnoDB: the directory. Feb 12 15:38:25 celtic mysqld[5839]: InnoDB: File name./ibdata1 Feb 12 15:38:25 celtic mysqld[5839]: InnoDB: File operation call: 'open'. Feb 12 15:38:25 celtic mysqld[5839]: InnoDB: Cannot continue operation. Feb 12 15:38:55 celtic mysql[5482]: Starting MySQL database server: mysqld. failed! Feb 12 15:38:55 celtic systemd[1]: mysql.service: control process exited, code=exited status=1 Feb 12 15:38:55 celtic systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon. Feb 12 15:38:55 celtic systemd[1]: Unit mysql.service entered failed state.
I can't see how a crappy mysql configuration is systemd's fault.
Slashdot Mirror
So I install Centos7 and mongodb and, in fact you are right, the error message is not written to the systemd journal.
Why?
Let's look at /etc/init.d/mongod, here's how it starts /usr/bin/mongodb:
So, how is systemd supposed to add mongod's stderr to the journal, the mongo startup script is sending it to /dev/null.
It gets worse, even if we fix /etc/init.d/mongod to not redirect stdout/stderr it's calling the function "daemon" from /etc/rc.d/init.d/functions (sic) to launch the program, and that does:
So if /usr/bin/mongod writes to stderr then systemd never gets to see it because of the fucked up centos sysvinit scripts!
Yes, it's the sysvinit scripts that are preventing you seeing the message, not systemd!
The lack of logging of login attempts is a pretty serious security problem.
It sure is.
Where's the bug report?
As far as why people think systemd doesn't log, it's because it doesn't at least on any of the servers where I've looked. I don't know why people question that.
Well, I question it because it's not true on any machine I'm running systemd on. For example on the laptop I'm typing this message on:
What distros are you running that don't log error messages? Did you report it as a bug?
Fuck off you tedious troll. systemd is hiding nothing.
systemd is correct to move on from those out of date and hard to understand concepts. I like the fact that errors are hidden from my admins
Moron. systemd is hiding nothing. You are a brainless troll.
Moron. Complaints should not be ignored. It is possible that the original complaint or bug report didn't include enough information.
You need to get a life, preferably in some sphere that has no impact on real human beings.
Moron. systemd throws nothing away.
Ok, I tried it on Debian and it works.
Now what happens on CentOS.
Works for me.
You are, of course, totaly correct. My suggestion was rubbish.
Think about it, the original "problem" was that launching the "broken_systemd" service with systemctl gives no error, and returns no error status.
But how could it? Imagine that "broken_systemd" runs for three hours and fails. Do you want systemctl to hang around until it fails. Or for three minutes? Or three seconds? Where do you put the arbitrary limit?
systemctl start starts the service. It will tell you if the service can't be started (if the program can't be found, or if there is a startup script (Type=forking) which fails) but it can't tell you that the process started will fail at some time in the future.
Interesting. Like I say it demonstrably works for me on Debian. Mongodb don't seem to have a version for Debian Sid/Jessie yet.
What version of RedHat are you using?
(Did you ever make a bugreport about this? What number?)
Okeydoke, let's try it.
Let's simulate your problem:
Huh? "all output going to: /var/log/mongodb/mongodb.log"? Maybe that's why we see nothing in the journal?
In the mongodb.conf file I find:
And the doc says:
So, let's take out the logpath and try again, and now we get the error in the journal:
So, configuration error. systemd "threw away" nothing.
> Where did you get the idea that this is "policy"?
Because complaints to the mailing list are ignored and the posters insulted. Here's one example I saved that I just ran on an updated CentOS to confirm that it is still a problem. This shows the systemd guys prevent the troubleshooting sysadmins need to manage our systems.
So, while we're busy accusing people of being liars, care to give a link to the "complaint to the mailing list" or bug report you made for this problem.
Because so far I can't find it.
Ok, just tried it on recent Debian sid:
Interesting
The problem with the exit code is that you're missing "Type=forking" in the service file. If I add that then I get:
So it looks like you've got:
1. CentOS bug losing stdout/stderr
2. User error losing exit status
systemd shills may be liars, but systemd haters display symptoms of idiocy.
> the default mysql config writes error messages to a file
Are all of the systemd shills childish liars? That is not the default configuration for MySQL community edition from Oracle. The word error doesn't even appear in /etc/my.cnf.
Why are anti systemd people so quick to accuse others of things like being "childish liars"?
Just found a very nice script that proves the problem:
http://linux.slashdot.org/comments.pl?sid=6953777&cid=49039657
Well, I just tried that example on Debian sid and it worked for me.
http://linux.slashdot.org/comments.pl?sid=6953777&cid=49041613
So there is no evidence that "systemd shills are liars", just that Centos is a buggy piece of shit.
Ok, just tried it on recent Debian sid:
Interesting, but:
Looks like your version of systemd is broken.
I have
No, it's absolutely the best way to explain what an orbit is.
I don't understand.
On commercial platforms with a GUI there is a front end.
On Linux with a GUI there is a front end.
On Linux without a GUI you use command line tools, nmcli for example.
Hell we used to build unidirectional ethernet cables for log servers, it's rather hard to leak data to a hacker over an air gap after all.
Twisted pair, not coax I presume :-)
Yup, it works correctly on Debian too -- everything that goes to syslog gets into the journal.
I've been making an effort to find some kind of confirmation or source for these "systemd doesn't log by policy" claims, which, AFAIK are the opposite of the truth (systemd/journald catch output to stdout/stderr that used to get lost).
So far the only one I've been able to get a handle on is the "mysql didn't start and there was no info in the journal" one, which appears to be because the default mysql config writes error messages to a file in /var/log instead of to syslog.
How is systemd/journald supposed to know that it should put the contents of random disk files in the journal?
The basic system of Unix permissions while being suitable in many cases, can be more difficult to manage in others. What if i want to write some rules to say that a particular user should not have access to certain directories, or to say they should only have access to certain directories.
If you want to get into windows ACL hell it's easy:
Wifi support in Windows is a mess.
Wifi support in MacOS is a mess.
Wifi support in Android is a mess.
Wifi support is a mess. Does it work anywhere?
Tha's bizzare. Exactlty what log messages were missing?
Huh? That's systemd's policy. I've setup hundreds of CentOS 7 servers for customers since last Sept, and I can't remember ever seeing a start-up error in the systemd journal.
Not my experience.
AFAIK systemd logs everything it gets given.
Systemd logs everything that gets sent to syslog, everything that gets sent to the kernel log and the stdout/stderr of almost all things it starts (you can put "StandardError=null" in a service file if you want, for example ModemManager.service).
I suppose some lunatic could put DefaultStandardOutput=null, DefaultStandardError=null in system.conf, but that's user (or distro) error, not systemd.
Where did you get the idea that this is "policy"?
If you don't understand why https is a very poor replacement for syslog over a network that may well be unreliable. I will assume your not really familiar with low level hardware and the failure modes. You do not seem to get the overhead and delay of TCP connections in general and thus why it's a poor choice for logging.
UDP over an unreliable network is better?
Well I just tried that on Debian, and the problem wasn't that systemd threw away any log messages, the problem was that the default mysql configuration writes its error messages to /var/log/mysql/error.log rather than the journal.
Removing "log_error = /var/log/mysql/error.log" from the mysql configuration gets exactlt what you want:
I can't see how a crappy mysql configuration is systemd's fault.
He's claiming that you don't get any 3, 2, 1, or 0 (error, critical, alert or emergency) messages.
At least on Debian that's just not true.