Perl services can crash with a "Can't kill a non-numeric process ID" error

[Galen Charlton]

Perl app listeners can occasionally throw the following exception:

server: died with error Can't kill a non-numeric process ID at /opt/sequoia/apps/sequoia-perl5lib/lib/perl5/OpenSRF/Server.pm line 335.

As with bug 1953044, when this happens, the listener will kill its drones and attempt to reset itself (though the reset doesn't work for other reasons that I'll document in a separate bug).

We have seen this on Perl 5.20.2 and Perl 5.28.1 systems, though I'm not sure that the Perl version matters.

OpenSRF 3.1+

[Galen Charlton (gmc)]

A patch is available at the tip of

user/gmcharlt/lp1953047_only_kill_numeric_pids / https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/gmcharlt/lp1953047_only_kill_numeric_pids

As the commit message indicates, this is just a band-aid, but one that may help us track down the root cause.

[Galen Charlton (gmc)]

Noting that the non-handling of failed forks by Perl apps mentioned in the commit message is the subject of bug 1546683

[Galen Charlton (gmc)]

I just saw one scenario that triggered the error: doing a service restart where the listener didn't go away after a TERM and needed an INT:

* timed out waiting on open-ils.actor pid=23389 to die
* sending INT signal to pid=23389 open-ils.actor