Monit fills logs when reading 0 byte pid file

[agent 8131]

When the pid file monit reads for a service is a 0 byte file monit enters a tight loop and spews this error to the syslog:

monit: Error reading pid from file

It will write this hundreds of times a second until all disk space is consumed.

I've verified this on monit 5.3.2-1 on Ubuntu 12.04 but not on other versions as of yet.

[Sergey B Kirpichev (skirpichev)]

> spews this error to the syslog:
> monit: Error reading pid from file

Is this is the only error you have seen in the log file? Or there are others?

I think, there is - e.g. monit tries to restart this service. If this is the case - this is expected. You should limit the number of restarts for this service, like this:
-->8--
if 2 restarts within 3 cycles then unmonitor
-->8--
And you should fix this service - pid file shouldn't be empty!

[agent 8131 (agent-8131)]

> Is this is the only error you have seen in the log file? Or there are others?

This is the only message

> I think, there is - e.g. monit tries to restart this service. If this is the case - this is expected.

This is not the case, monit is not trying to restart the service, it is entering a tight loop trying to read the PID file, as I stated. All of my monit configs have the number of restarts limited.

> And you should fix this service - pid file shouldn't be empty!

Obviously, but that does not change the fact that this bug exists in monit. Monit should fail gracefully, handling an empty PID file the same way it would handle a non-existent PID file.

[Sergey B Kirpichev (skirpichev)]

> This is the only message

Sorry. Then I'm unable to reproduce this on the current Debian package. Can you check this version?

> This is not the case, monit is not trying to restart the service, it is entering a tight loop trying to read the PID file, as I stated.

Here is my steps to check this:
1) install Sid's version of the package
2) install openntpd + edit /etc/init.d/openntpd to exclude -p option
3) manually create (touch on start) and remove (on stop) empty pid file in /etc/init.d/openntpd
4) include /etc/monit/monitrc.d/openntpd in /etc/monit/monitrc
5) restart openntpd
6) restart monit

After the last step:
$ cat /var/log/monit.log

[MSK Feb 21 23:13:49] info : monit HTTP server started
[MSK Feb 21 23:13:49] info : 'sid.test' Monit started
[MSK Feb 21 23:13:49] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[MSK Feb 21 23:13:49] error : 'ntpd' process is not running
[MSK Feb 21 23:13:49] info : 'ntpd' trying to restart
[MSK Feb 21 23:13:49] info : 'ntpd' stop: /etc/init.d/openntpd
[MSK Feb 21 23:13:49] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[...]
[MSK Feb 21 23:13:49] info : 'ntpd' start: /etc/init.d/openntpd
[MSK Feb 21 23:13:49] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[...]

[MSK Feb 21 23:14:19] error : 'ntpd' failed to start
[MSK Feb 21 23:16:19] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[MSK Feb 21 23:16:19] error : 'ntpd' process is not running
[MSK Feb 21 23:16:19] info : 'ntpd' trying to restart
[MSK Feb 21 23:16:19] info : 'ntpd' stop: /etc/init.d/openntpd
[MSK Feb 21 23:16:19] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[MSK Feb 21 23:16:19] info : 'ntpd' stopped
[MSK Feb 21 23:16:19] info : 'ntpd' start: /etc/init.d/openntpd
[MSK Feb 21 23:16:20] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[...]
[MSK Feb 21 23:19:21] error : monit: Error reading pid from file '/var/run/openntpd.pid'
[MSK Feb 21 23:19:21] error : 'ntpd' failed to start
[MSK Feb 21 23:21:21] error : 'ntpd' service restarted 4 times within 4 cycles(s) - unmonitor

Yes, I see a lot of spam, but it works well for me.

> Monit should fail gracefully, handling an empty PID file the same way it would handle a non-existent PID file.

That's very different situations, imho.

[Sergey B Kirpichev (skirpichev)]

Same for 1:5.4-2.

Can you reproduce this:
-->8--
This is not the case, monit is not trying to restart the service, it is entering a tight loop trying to read the PID file
-->8--
for this monit's version?

[agent 8131 (agent-8131)]

Easy to reproduce. Here's syslog-ng on Ubuntu 12.04:

monit/conf.d/syslog-ng:
check process syslog-ng with pidfile /var/run/syslog-ng.pid
  start program = "/etc/init.d/syslog-ng start"
  stop program = "/etc/init.d/syslog-ng stop"

make sure monit status is running

echo > /var/run/syslog-ng.pid

Wait for monit to notice, then logs fill with:
monit: Error reading pid from file '/var/run/syslog-ng.pid'

Not the best example because syslog-ng will notice the empty pid file and start a new instance so this will not fill the logs entirely but any application that does not actively check the state of its PID file will.

[Sergey B Kirpichev (skirpichev)]

On Sat, Mar 01, 2014 at 07:10:15AM -0000, agent 8131 wrote:
> Easy to reproduce. Here's syslog-ng on Ubuntu 12.04:
>
> monit/conf.d/syslog-ng:
> check process syslog-ng with pidfile /var/run/syslog-ng.pid
> start program = "/etc/init.d/syslog-ng start"
> stop program = "/etc/init.d/syslog-ng stop"

I don't any restriction for number of restarts here. Is
this is a full config?

[dinexi (im-dinexi)]

the config:
===
check process nginx with pidfile /var/run/nginx.pid
  group web
  start program = "/etc/init.d/nginx start"
  stop program = "/etc/init.d/nginx stop"
  if failed host 127.0.0.1 port 80 protocol http request '/robots.txt' then restart
  if 3 restarts within 5 cycles then alert
===
and the excerpt from the log:
===
[CDT May 8 02:43:36] error : monit: Error reading pid from file '/var/run/nginx.pid'
[CDT May 8 02:43:36] error : monit: Error reading pid from file '/var/run/nginx.pid'
[CDT May 8 02:43:36] info : 'php-fpm' start: /etc/init.d/php5-fpm
[CDT May 8 02:45:36] error : 'nginx' service restarted 5 times within 5 cycles(s) - alert
[CDT May 8 02:45:36] error : 'nginx' failed protocol test [HTTP] at INET[127.0.0.1:80/robots.txt] via TCP -- HTTP error: Server returned status 404
[CDT May 8 02:45:36] info : 'nginx' trying to restart
[CDT May 8 02:45:36] info : 'php-fpm' stop: /etc/init.d/php5-fpm
[CDT May 8 02:45:36] info : 'nginx' stop: /etc/init.d/nginx
[CDT May 8 02:45:36] info : 'nginx' start: /etc/init.d/nginx
> [CDT May 8 02:45:36] error : monit: Error reading pid from file '/var/run/nginx.pid'
> [CDT May 8 02:45:36] error : monit: Error reading pid from file '/var/run/nginx.pid'
===

[Arkadiy Kulev (eth-ethaniel)]

This bug is still here (monit 5.20.0).

http://stackoverflow.com/questions/41773333/monit-is-continuously-throwing-error-reading-pid-from-file

In my case, the whole startup is slowed down because monit is "initializing" the failing service which has an empty .pid file. It catches itself into a loop.

[MSK Feb 19 21:21:41] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:41] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:42] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:44] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:45] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:46] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:47] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:48] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:49] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:50] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:51] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:52] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:53] error : Error reading pid from file '/var/run/sphinx/searchd.pid'
[MSK Feb 19 21:21:54] error : Error reading pid from file '/var/run/sphinx/searchd.pid'