unit stuck executing update-status
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
juju-core |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Hi,
I found a long-running environment with three units running update-status and never getting out of it.
juju version: 1.25.10
cloud provider: openstack
Here are relevant log excerpts. Apparently the failure happens on 2017-03-13 06:49, but I added a few lines of context.
juju unit logs logs (for one of the three units):
2017-03-08 12:26:45 INFO config-changed + service nagios-nrpe-server reload
2017-03-08 12:26:45 INFO config-changed * Reloading nagios-nrpe configuration files nagios-nrpe
2017-03-08 12:26:45 INFO config-changed ...done.
2017-03-08 17:45:50 WARNING juju.worker.
hook here, but we can't yet
2017-03-13 06:49:43 ERROR juju.worker.
2017-03-13 06:49:43 WARNING juju.worker.
ker: "leadership-
2017-03-13 06:49:47 WARNING juju.worker.
ker: "leadership-
2017-03-13 06:49:49 WARNING juju.worker.
ker: "leadership-
(and then the same message every few seconds for 1h+)
Machine 0 logs:
2017-03-11 13:01:19 WARNING juju.worker.
caused by: request (http://
{"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
2017-03-11 13:01:19 WARNING juju.worker.
caused by: request (http://
{"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
2017-03-13 06:49:42 ERROR juju.state.
state changing too quickly; try again soon
2017-03-13 08:51:50 ERROR juju.rpc server.go:573 error writing response: write tcp 10.25.8.
2017-03-13 08:51:50 ERROR juju.rpc server.go:573 error writing response: write tcp 10.25.8.
2017-03-13 08:52:40 INFO juju.cmd supercommand.go:37 running jujud [1.25.10-
2017-03-13 08:52:40 DEBUG juju.agent agent.go:491 read agent config, format "1.18"
2017-03-13 08:52:40 INFO juju.cmd.jujud machine.go:419 machine agent machine-0 start (1.25.10-
2017-03-13 08:52:40 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/
2017-03-13 08:52:40 INFO juju.cmd.jujud upgrade.go:88 no upgrade steps required or upgrade steps for 1.25.10 have already been run.
This is resolved by restarting jujud-machine-0 (as can be seen in the log above).
Thanks,
Laurent
tags: | added: canonical-is |
I forgot to mention that IP 10.25.8.154 (seen in machine-0's logs) is the IP of the machine where the mentioned nrpe unit runs.