monasca-agent restart fails

[Bradley Klein]

We've seen this several times in our environment -- where the monasca agent fails to restart with:

17:12:32 2015-08-12T17:12:07+00:00 Error: Could not start Service[monasca-agent]: Execution of '/etc/init.d/monasca-agent start' returned 1: * Starting Monasca Monitoring Agent (using supervisord) monasca-agent
17:12:32 2015-08-12T17:12:07+00:00 Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
17:12:32 2015-08-12T17:12:07+00:00 For help, use /var/lib/monasca-agent/bin/supervisord -h
17:12:32 2015-08-12T17:12:07+00:00 ...fail!
17:12:32 2015-08-12T17:12:07+00:00 ...fail!
17:12:32 2015-08-12T17:12:07+00:00 2
17:12:32 2015-08-12T17:12:07+00:00 Monasca Monitoring Agent (supervisor) is NOT running all child processes
17:12:32 2015-08-12T17:12:07+00:00 * Stopping Monasca Monitoring Agent (stopping supervisord) monasca-agent
17:12:32 2015-08-12T17:12:07+00:00 ...done.
17:12:32 2015-08-12T17:12:07+00:00 Error: /Stage[main]/Monasca::Agent/Service[monasca-agent]/ensure: change from stopped to running failed: Could not start Service[monasca-agent]: Execution of '/etc/init.d/monasca-agent start' returned 1: * Starting Monasca Monitoring Agent (using supervisord) monasca-agent
17:12:32 2015-08-12T17:12:07+00:00 Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
17:12:32 2015-08-12T17:12:07+00:00 For help, use /var/lib/monasca-agent/bin/supervisord -h
17:12:32 2015-08-12T17:12:07+00:00 ...fail!
17:12:32 2015-08-12T17:12:07+00:00 ...fail!
17:12:32 2015-08-12T17:12:07+00:00 2
17:12:32 2015-08-12T17:12:07+00:00 Monasca Monitoring Agent (supervisor) is NOT running all child processes
17:12:32 2015-08-12T17:12:07+00:00 * Stopping Monasca Monitoring Agent (stopping supervisord) monasca-agent
17:12:32 2015-08-12T17:12:07+00:00 ...done.

[Ryan Bak (ryanmbak)]

This is occuring because the processes somehow become orphaned. 'service monasca-agent status' says monasca-agent is stopped, so our agent tries to start the agent, but the processes are still running, so the start fails with the above error. We have not been able to determine how the agent gets in this state.

[Ryan Bak (ryanmbak)]

We also just discovered another behavior, likely related, where doing a restart or stop fails to actually stop the agent, even though the status shows it is running.

[Tim Kuhlman (timkuhlman)]

Is it possible agent was started without the init script? Also what init system are using upstart, systemd or plain old sys V?

[Allan G (greental)]

need more details if reproduceable