However, I have never ever needed a pid file. If I have written the deamon myself, it ensures it "ps" only produces the correct "hit" or can be readily found in/proc (if the OS supports it) or can otherwise be communicated with via an IPC. For other daemons I will trust to careful ps and grep and have never stopped the wrong thing.
Ok that might be a bit annoying for most its true. However, as all daemons are the child of init, init will be informed by the kernel if a child dies and wait() will get the pid so all that difficulty could be fixed in a few extra lines to init.
The grouping of everything together is achieved by cgroups, not systemd so theres no reason why you cant arrange that using standard sysvinit. To be honest as cgroup like technology has been around in other systems before, youd wonder why nobody ever bothered to implement such a solution before, maybe nobody saw the need.
I can see your point but in the cases where I have had to parse binary logs that come to mind i.e utmp files and BSM audit logs, it was significantly more annoying than parsing something like syslog with grep/awk/sed/cut/expr etc etc.
It occurs to me that the problem you are trying to address is only a problem because maybe you havent found the right tools and maybe havent split your logs up into logical files rather than just using syslog.
The tool you want to parse your logs is so good it seems like magic. It is an unbelievable tool. It indexes log files, extracts reports, draws graphs, alerts and keeps your coffee warm. It is http://www.splunk.com/ you can use it for free if you dont index too much information.
Like so many enterprise tools, including all monitoring software, it cant read binary logs.
I just need to get this clear what you are saying.
Are you saying that a "daemonized" process needs to keep track of what pid it had itself and that it has to keep it in a pidfile? I just wondered because nothing ever, ever, ever needs a pidfile especially as any service can get its own pid by going "getpid()"
And then you are saying that one advantage of systemd is that you get the pid in syslog and perhaps some other data about the process itself. I had never thought of that. I mean I never thought when I read something like "can not write to/oracle; device full" that having the pid of the process and that it was owned by oracle would be particularly useful.
Maybe it is sometimes for some people. Seems a terrible upheaval for a very small gain.
What Sys V does isnt dependency checking, it is simply an order of execution. True, you could put something in one that hangs the thing, I did myself by accident once but its a very easy fix and it takes two minutes to come up with a timeout strategy if you want one. Here Ill try ok. This one took me two minutes to come up with and test. Im sure there are many other ways of doing it that are simpler. The shell is such a wonderful tool isnt it and that makes it great for initialising your system.
export PARENT=$? (sleep 10 && kill $PARENT) & commands that could hang kill $!
You can put a "trap" in there if you want to do more than just exit of course.
Doing your own dependency checking IS trivial and I am informed BSD even provide a tool to do it. I admit it might be more difficult for red hat to do it and my advise is that they shouldnt.
You do not understand a high security system even though you think you do.
Think steel plated walled rooms, double locked steel doors requiring two security certified people with two keys. Tell me Im complaining when I have to arrange for people to go down there because a useless tool like utmp needs to be safeguarded.
What you fail to understand and I dont blame you is that vendor driven dependency checking as a concept is actually broken. The Solaris dependency is correct if you look at it from one perspective and wrong if you look at it from another. smf should not decide if utmp is more important than sshd especially as that decision may be different under different circumstances. In this particular case it matters not if the utmp filesytem is mounted, sshd would still have worked even though the utmp records may get a bit screwed and in this instance uptime was more important than audit logs.
By the way, please do not extoll the virtues of utmp for user logging in a highly secure environment. We log and monitor every system call, every exec, every connect, every bind and everything typed at a keyboard.
I can guarantee it because sys V init doesnt enforce dependencies so sshd would have been started and it would have created a new utmp file in the root filesystem and I could have logged in and fixed everything.
It wasnt that the vendor (this is Solaris if you remember) got the dependencies wrong. The dependencies are actually right if you look at it from one perspective (user audit logs should be preserved) and wrong if you look at it from another perspective (I am not interested in standard unix auditing logs and need sshd up). With systemd type systems the vendor has to choose one and can not know which is right for might feel differently on different occasions.
For sys v init, no option is chosen and everything attempts to start and if it fails tells me why in a standard log file that anybody can read, even my monitoring software.
If you need dependency checking it is trivial for an administrator to set that up using the sys V model.
Point 1. No. I guarantee that on a sys V init system I would have been able to log in and easily see that a filesystem that should have been mounted wasnt mounted and then diagnose and fix it. ssh will cope just fine if there is no utmp file but you might end up with another utmp file than your usual one which is presumably why the dependency exists (those records might be important to you). However, I dont depend on utmp for my auditing records but smf (or systemd) can not know that and there you have the essence of the problem. It didnt let me log in because it thought I needed something which I dont.
Point 2. Indeed but/etc/inittab tells you all the places you need to look and you can go and look at them and read them. Sorry but that seems rather tidy to me.
Please stop. Ive been at this game a very long time. It took very little time to determine what was wrong and to fix it.
My point was that I couldnt ssh in because a filesystem was corrupt and had to use the console. That is stupid as well as very time consuming and expensive in the high security environment in which these systems live.
I see the logic of course. utmp is updated when you log in with ssh so sshd depends on utmp and having utmp requires having a file system to put it on so there is a dependency on the mounts. What concerns me is that the init system was trying to be clever rather than realising that if the filesystem with utmp didnt mount, a utmp file would still be created by ssh and be useable albeit probably on the root filesystem and fundamentally, I could log in.
Im not even concerned that the mounts are marked as failed if one completely unrelated filesytem fails to mount as that is merely a problem of implementation and can be fixed.
The fundamental problem here is that systemd (or smf in this case) are not clever enough to understand the intricacies of all the things along the dependency chain, can leave you in a very bad state and are far more difficult to debug than a single file whose last few lines contain the words "could not mount/tmp/whatever"
You are correct that the/etc/inittab has no notion of dependencies but you do and thats the key.
I can certainly see that whole S021startsomething.sh is a bit of an ugly hack but its a workable hack. If the dependency thing is such a problem for you why not have a simple dependency file that something reads and describes dependencies and perhaps what to do when dependencies are not met. You can easily implement something like that with the current model. Cant see why you need to over engineer a behemoth just to solve that? Personally Ive never broken a system because a service started when a dependency had failed but I have found myself unable to log in to a Solaris box because one filesystem didnt mount.
I have never seen gigabytes of logs filled with output of failing daemons and have never missed crucial logs in a start up session. If you have then perhaps you need to review your own policies.
Actually, most of the new sysadmins who join my team do not get taught by crusty old sysadmins but learn by themselves. Every single one of them chooses to use a init script to start something up when given the choice though.
And, why would I need a portable init script for forking a daemon with or without a limited capability set and why would I want to do it in five lines when I can do it with one that everybody can clearly understand in/etc/inittab?
I have no problem with control groups. They provide features that have been more or less provided for in several different ways before and are a handy feature but killing daemons reliably has not been a problem for the forty or so years before cgroups came along.
You seem to believe these crusty old unix admins with years of experience have nothing to teach you so Im gad you dont work on critical systems but I urge you to re-consider your outlook. It will save you a lot of trouble in all aspects of your life.
I dont think input validation is any less of an issue with a binary. You could argue that at least with a script everybody can see if its doing something stupid.
I have used it. I have also used several systems like it on various Unix systems as the concepts behind systemd are not new. I have adapted to more changes in Unix/Linux and learned more new technologies that you can shake a stick at over the years and continue to do so.
My experience with systemd like initialisation systems is that they are not so easy for the new, fledgling administrators to understand and therefore are a source of application down time. I dont like them as a result. Is that OK with you?
Maybe you could learn a few things from people that have seen the same mistakes you make a hundred times before.
Firstly using a pid files is an utterly stupid idea and quite frankly, anybody who can not see that when they first think of them or read about them should not be an admin on any critical systems. However, much as I like init, init doesnt do pids more elegantly, it doesnt do them at all. The kernel does that by kindly telling init when one of its children has died and arranging for it to be able tell what the pid was.
init doesnt do much at all and thats why it works so well. It simply takes whatever run level you want, reads through/etc/inittab to see what jobs to start for that run level and starts them. It then re-starts them if it gets a child death signal and/etc/inittab said respawn. Simplicity itself even though most Unixes now break it by having it start one job that handles everything else. Im guessing the one problem with init is that it cant handle a process that forks and then exits and maybe thats the reason/etc/inittab is dying. Shame.
I also think the kernel handing orphaned processes over to init is cheating a bit but I like it:)
I have similar length and breadth of experience of Unix systems and to be fair, I have seen init break but only once and it was when I broke it myself. I forgot to put an & and the end of a "sleep 20000/dev/tty10" while trying to keep a serial line to a printer working properly. Next re-boot hung the machine but I was able to guess what the problem was.
When I first saw SMF break I had absolutely no clue why I couldnt ssh into the machine nor where to start looking. It was when I discovered that sshd startup was dependent on utmp being available which depended on filesystem mounting being successful that I knew for sure that systemd style init was nothing I wanted.
For me, scanning through/etc/inittab and being able to see exactly what is going on in the initialisation stage is the essence of Unix. Even this is sadly being slowly broken even before the utterly pointless systemd was born.
Remember when we moved from a.out to elf executable format? That was a lot of work but we all could see the benefit and although software would break, everyone would know why.
I think I'm with you on this. Compared to Solaris, HPUX and even Aix, Linux has no place in an enterprise looking to keep costs down.
One thing that made me thump my head against a wall with SMF was when a system booted but I couldnt ssh into it. sshd was not started because utmp was not "enabled" because mounting the filesystems failed because a single file system didnt mount. Ok, its fixable but sshd ran perfectly if started manually and it delayed getting production systems up by a few minutes. We get journalists knocking at the door if our systems are down.
I hate this stuff, none of my admins remember where the damn log files are because they play with it so rarely. A load of scripts run in sequence can easily be followed, however rusty you are.
Slashdot Mirror
OK thanks for the clarification.
However, I have never ever needed a pid file. If I have written the deamon myself, it ensures it "ps" only produces the correct "hit" or can be readily found in /proc (if the OS supports it) or can otherwise be communicated with via an IPC. For other daemons I will trust to careful ps and grep and have never stopped the wrong thing.
Ok that might be a bit annoying for most its true. However, as all daemons are the child of init, init will be informed by the kernel if a child dies and wait() will get the pid so all that difficulty could be fixed in a few extra lines to init.
The grouping of everything together is achieved by cgroups, not systemd so theres no reason why you cant arrange that using standard sysvinit. To be honest as cgroup like technology has been around in other systems before, youd wonder why nobody ever bothered to implement such a solution before, maybe nobody saw the need.
Actually, the binary logs bother me the most.
I can see your point but in the cases where I have had to parse binary logs that come to mind i.e utmp files and BSM audit logs, it was significantly more annoying than parsing something like syslog with grep/awk/sed/cut/expr etc etc.
It occurs to me that the problem you are trying to address is only a problem because maybe you havent found the right tools and maybe havent split your logs up into logical files rather than just using syslog.
The tool you want to parse your logs is so good it seems like magic. It is an unbelievable tool. It indexes log files, extracts reports, draws graphs, alerts and keeps your coffee warm. It is http://www.splunk.com/ you can use it for free if you dont index too much information.
Like so many enterprise tools, including all monitoring software, it cant read binary logs.
Hes right.
I was there too.
cron does care about exit codes. Any cron job returning a non zero exit code will have "rc=X" in the cron log and it can even mail you the stderr.
Admittedly it will also have "rc=1" if it couldnt run the job at all e.g. if the user account is locked but mostly the cron log doesnt lie.
I just need to get this clear what you are saying.
Are you saying that a "daemonized" process needs to keep track of what pid it had itself and that it has to keep it in a pidfile? I just wondered because nothing ever, ever, ever needs a pidfile especially as any service can get its own pid by going "getpid()"
And then you are saying that one advantage of systemd is that you get the pid in syslog and perhaps some other data about the process itself. I had never thought of that. I mean I never thought when I read something like "can not write to /oracle; device full" that having the pid of the process and that it was owned by oracle would be particularly useful.
Maybe it is sometimes for some people. Seems a terrible upheaval for a very small gain.
I wish you hadnt posted this as Anonymous Coward.
It is extremely good and currently has a score of 0.
What Sys V does isnt dependency checking, it is simply an order of execution. True, you could put something in one that hangs the thing, I did myself by accident once but its a very easy fix and it takes two minutes to come up with a timeout strategy if you want one. Here Ill try ok. This one took me two minutes to come up with and test. Im sure there are many other ways of doing it that are simpler. The shell is such a wonderful tool isnt it and that makes it great for initialising your system.
export PARENT=$?
(sleep 10 && kill $PARENT) &
commands that could hang
kill $!
You can put a "trap" in there if you want to do more than just exit of course.
Doing your own dependency checking IS trivial and I am informed BSD even provide a tool to do it. I admit it might be more difficult for red hat to do it and my advise is that they shouldnt.
You do not understand a high security system even though you think you do.
Think steel plated walled rooms, double locked steel doors requiring two security certified people with two keys. Tell me Im complaining when I have to arrange for people to go down there because a useless tool like utmp needs to be safeguarded.
What you fail to understand and I dont blame you is that vendor driven dependency checking as a concept is actually broken. The Solaris dependency is correct if you look at it from one perspective and wrong if you look at it from another. smf should not decide if utmp is more important than sshd especially as that decision may be different under different circumstances. In this particular case it matters not if the utmp filesytem is mounted, sshd would still have worked even though the utmp records may get a bit screwed and in this instance uptime was more important than audit logs.
By the way, please do not extoll the virtues of utmp for user logging in a highly secure environment. We log and monitor every system call, every exec, every connect, every bind and everything typed at a keyboard.
I can guarantee it because sys V init doesnt enforce dependencies so sshd would have been started and it would have created a new utmp file in the root filesystem and I could have logged in and fixed everything.
It wasnt that the vendor (this is Solaris if you remember) got the dependencies wrong. The dependencies are actually right if you look at it from one perspective (user audit logs should be preserved) and wrong if you look at it from another perspective (I am not interested in standard unix auditing logs and need sshd up). With systemd type systems the vendor has to choose one and can not know which is right for might feel differently on different occasions.
For sys v init, no option is chosen and everything attempts to start and if it fails tells me why in a standard log file that anybody can read, even my monitoring software.
If you need dependency checking it is trivial for an administrator to set that up using the sys V model.
Point 1. No. I guarantee that on a sys V init system I would have been able to log in and easily see that a filesystem that should have been mounted wasnt mounted and then diagnose and fix it. ssh will cope just fine if there is no utmp file but you might end up with another utmp file than your usual one which is presumably why the dependency exists (those records might be important to you). However, I dont depend on utmp for my auditing records but smf (or systemd) can not know that and there you have the essence of the problem. It didnt let me log in because it thought I needed something which I dont.
Point 2. Indeed but /etc/inittab tells you all the places you need to look and you can go and look at them and read them. Sorry but that seems rather tidy to me.
Please stop. Ive been at this game a very long time. It took very little time to determine what was wrong and to fix it.
My point was that I couldnt ssh in because a filesystem was corrupt and had to use the console. That is stupid as well as very time consuming and expensive in the high security environment in which these systems live.
I see the logic of course. utmp is updated when you log in with ssh so sshd depends on utmp and having utmp requires having a file system to put it on so there is a dependency on the mounts. What concerns me is that the init system was trying to be clever rather than realising that if the filesystem with utmp didnt mount, a utmp file would still be created by ssh and be useable albeit probably on the root filesystem and fundamentally, I could log in.
Im not even concerned that the mounts are marked as failed if one completely unrelated filesytem fails to mount as that is merely a problem of implementation and can be fixed.
The fundamental problem here is that systemd (or smf in this case) are not clever enough to understand the intricacies of all the things along the dependency chain, can leave you in a very bad state and are far more difficult to debug than a single file whose last few lines contain the words "could not mount /tmp/whatever"
Persistent arent you.
You are correct that the /etc/inittab has no notion of dependencies but you do and thats the key.
I can certainly see that whole S021startsomething.sh is a bit of an ugly hack but its a workable hack. If the dependency thing is such a problem for you why not have a simple dependency file that something reads and describes dependencies and perhaps what to do when dependencies are not met. You can easily implement something like that with the current model. Cant see why you need to over engineer a behemoth just to solve that? Personally Ive never broken a system because a service started when a dependency had failed but I have found myself unable to log in to a Solaris box because one filesystem didnt mount.
I have never seen gigabytes of logs filled with output of failing daemons and have never missed crucial logs in a start up session. If you have then perhaps you need to review your own policies.
Actually, most of the new sysadmins who join my team do not get taught by crusty old sysadmins but learn by themselves. Every single one of them chooses to use a init script to start something up when given the choice though.
And, why would I need a portable init script for forking a daemon with or without a limited capability set and why would I want to do it in five lines when I can do it with one that everybody can clearly understand in /etc/inittab?
I have no problem with control groups. They provide features that have been more or less provided for in several different ways before and are a handy feature but killing daemons reliably has not been a problem for the forty or so years before cgroups came along.
You seem to believe these crusty old unix admins with years of experience have nothing to teach you so Im gad you dont work on critical systems but I urge you to re-consider your outlook. It will save you a lot of trouble in all aspects of your life.
Regards
I dont think input validation is any less of an issue with a binary. You could argue that at least with a script everybody can see if its doing something stupid.
I have used it. I have also used several systems like it on various Unix systems as the concepts behind systemd are not new. I have adapted to more changes in Unix/Linux and learned more new technologies that you can shake a stick at over the years and continue to do so.
My experience with systemd like initialisation systems is that they are not so easy for the new, fledgling administrators to understand and therefore are a source of application down time. I dont like them as a result. Is that OK with you?
Maybe you could learn a few things from people that have seen the same mistakes you make a hundred times before.
Firstly using a pid files is an utterly stupid idea and quite frankly, anybody who can not see that when they first think of them or read about them should not be an admin on any critical systems. However, much as I like init, init doesnt do pids more elegantly, it doesnt do them at all. The kernel does that by kindly telling init when one of its children has died and arranging for it to be able tell what the pid was.
init doesnt do much at all and thats why it works so well. It simply takes whatever run level you want, reads through /etc/inittab to see what jobs to start for that run level and starts them. It then re-starts them if it gets a child death signal and /etc/inittab said respawn. Simplicity itself even though most Unixes now break it by having it start one job that handles everything else. Im guessing the one problem with init is that it cant handle a process that forks and then exits and maybe thats the reason /etc/inittab is dying. Shame.
I also think the kernel handing orphaned processes over to init is cheating a bit but I like it :)
I have similar length and breadth of experience of Unix systems and to be fair, I have seen init break but only once and it was when I broke it myself. I forgot to put an & and the end of a "sleep 20000 /dev/tty10" while trying to keep a serial line to a printer working properly. Next re-boot hung the machine but I was able to guess what the problem was.
When I first saw SMF break I had absolutely no clue why I couldnt ssh into the machine nor where to start looking. It was when I discovered that sshd startup was dependent on utmp being available which depended on filesystem mounting being successful that I knew for sure that systemd style init was nothing I wanted.
For me, scanning through /etc/inittab and being able to see exactly what is going on in the initialisation stage is the essence of Unix. Even this is sadly being slowly broken even before the utterly pointless systemd was born.
Me too.
Remember when we moved from a.out to elf executable format? That was a lot of work but we all could see the benefit and although software would break, everyone would know why.
I think I'm with you on this. Compared to Solaris, HPUX and even Aix, Linux has no place in an enterprise looking to keep costs down.
Good job nobody called or those seconds might have been forever lost.
Are using a UI for "date" as your argument?
Admittedly, setting the timezone is pain these days but thats because of the same thinking i.e make a simple thing as complex as possible.
Me too, but then I never left :)
At the same time, not critical devices can't block startup.
But all these systems do exactly that too and its harder to find out why when they do.
I dont have three, I have four but Im very much with you on this.
Linux with systemd is finished for me at home and at work, well thats tough to change but I have started the debate.
One thing that made me thump my head against a wall with SMF was when a system booted but I couldnt ssh into it. sshd was not started because utmp was not "enabled" because mounting the filesystems failed because a single file system didnt mount. Ok, its fixable but sshd ran perfectly if started manually and it delayed getting production systems up by a few minutes. We get journalists knocking at the door if our systems are down.
I hate this stuff, none of my admins remember where the damn log files are because they play with it so rarely. A load of scripts run in sequence can easily be followed, however rusty you are.
Geography is not altered by your political affiliations.