Reporter Logging Improvements

[Jason Stephenson]

The reporter, clark-kent.pl (aka Clark), currently spews text to stdout/stderr. This is wrong on 2 fronts:

1. When started with the --daemon option those messages get dropped on the floor after the user who started the process logs out, and

2. daemon processes are expected to close stdin, stdout, and stderr and then use syslog to log any messages.

I propose that we follow UNIX/SUSP conventions and do #2 when clark-kent.pl is started as a daemon.

Branch to follow.

[Jason Stephenson (jstephenson)]

This bug also affects OpenSRF because the daemonize funciton in OpenSRF::Utils does not do everything that it should.

Section 13.3 of Advanced Programming in the Unix Environment, 3rd Edition, lists the requirements for a process to daemonize itself:

1. Call umask to set the file mode creation mask to a known value, usually 0. The inherited file mode creation mask could be set to deny certain permissions. If the daemon process creates files, it may want to set specific permissions. For example, if it creates files with group-read and group-write enabled, a file mode creation mask that turns off either of these permissions would undo its efforts. On the other hand, if the daemon calls library functions that result in files being created, then it might make sense to set the file mode create mask to a more restrictive value (such as 007), since the library functions might not allow the caller to specify the permissions through an explicit argument.

2. Call fork and have the parent exit. This does several things. First, if the daemon was started as a simple shell command, having the parent terminate makes the shell think that the command is done. Second, the child inherits the process group ID of the parent but gets a new process ID, so we’re guaranteed that the child is not a process group leader. This is a prerequisite for the call to setsid that is done next.

3. Call setsid to create a new session. The three steps listed in Section 9.5 occur. The process (a) becomes the leader of a new session, (b) becomes the leader of a new process group, and (c) is disassociated from its controlling terminal.

Under System V–based systems, some people recommend calling fork again at this point, terminating the parent, and continuing the daemon in the child. This guarantees that the daemon is not a session leader, which prevents it from acquiring a controlling terminal under the System V rules (Section 9.6). Alternatively, to avoid acquiring a controlling terminal, be sure to specify O_NOCTTY whenever opening a terminal device.

4. Change the current working directory to the root directory. The current working directory inherited from the parent could be on a mounted file system. Since daemons normally exist until the system is rebooted, if the daemon stays on a mounted file system, that file system cannot be unmounted.

Alternatively, some daemons might change the current working directory to a specific location where they will do all their work. For example, a line printer spooling daemon might change its working directory to its spool directory.

5. Unneeded file descriptors should be closed. This prevents the daemon from holding open any descriptors that it may have inherited from its parent (which could be a shell or some other process). We can use our open_max function (Figure 2.17) or the getrlimit function (Section 7.11) to determine the highest descriptor and close all descriptors up to that value.

6. Some daemons open file descriptors 0, 1, and 2 to /dev/null so that any library routines that try to read from standard input or write to standard output or standard error will have no effect. Since the daemon is not associated with a terminal dev...

[Jason Stephenson (jstephenson)]

I have branches ready that I have tested with production data over the past few days. The branches can be installed independently of each other, but to get the full benefit, both OpenSRF and Evergreen should be updated at the same time.

The OpenSRF branch is https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/dyrcona/lp1933984-perl-utils-daemonize-improvements

The OpenSRF branch improves the behavior of the daemonize function by

1. Setting the umask to a known, useful, value: 022
2. Closing all open file descriptors.
3. Opening STDIN, STDOUT, and STDERR to /dev/null.

It also adds a hard dependency on the BSD::Resource Perl module. This module is presently required for mod_perl, but I added it as an explicit dependency in case the mod_perl dependencies ever change. The dependency change is in a separate commit from the code changes in case it is deemed unnecessary.

The Evergreen branch is https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dyrcona/lp1933984-teach-clark-about-syslog

The Evergreen changes affect only the clark-kent.pl program. Most messages will be logged to syslog rather than to STDERR/STDOUT. It was neccessary to log all warnings during reporting to syslog because the reporting child processes daemonize themselves regardless of whether the main process is started as a daemon or not. (I consider changing that behavior to be a separate bug.)

Only those issues which occur before the main procces would daemonize itself are sent to the STDERR stream of the console. These generally affect process startup. Any fatal DBI errors and a failure to create the lock file in the main process are logged to both syslog and to STDERR. The latter do not go to the console when the main process is daemonized.

[Jason Stephenson (jstephenson)]

I was interrupted while writing the previous comment.

I wanted to add that I have targeted this as a new feature for Evergreen, though I consider it to be a bug. If anyone wants to add backports targets for 3.5, 3.6, and/or 3.7, feel free to do it yourself to ask me to do it.

I have been testing this with 3.5.3 and production data. I intend to install the changes in production, soon. As I'd like to get log messages to find out what is happening when our reporter "randomly" crashes.