The juju 2.4.7 agent (and likely newer) can get itself into a situation where it fails to reconnect to the API server. Usually this happens when the HA controllers are unavailable for a period (e.g. taken offline for repair/recovery).
The agent is currently finishes with:
2018-12-30 05:55:09 ERROR juju.worker.dependency engine.go:632 "migration-inactive-flag" manifold worker returned unexpected error: migration lookup failed: no reachable servers
2018-12-30 05:55:12 ERROR juju.worker.dependency engine.go:632 "migration-minion" manifold worker returned unexpected error: migration lookup: migration lookup failed: no reachable servers
2018-12-30 05:55:45 ERROR juju.worker.dependency engine.go:632 "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot get machine 74: no reachable servers
2018-12-30 05:55:45 ERROR juju.worker.dependency engine.go:632 "migration-inactive-flag" manifold worker returned unexpected error: migration lookup failed: no reachable servers
2018-12-30 05:55:45 ERROR juju.worker.dependency engine.go:632 "migration-minion" manifold worker returned unexpected error: migration lookup: migration lookup failed: no reachable servers
2018-12-30 05:56:16 ERROR juju.worker.dependency engine.go:632 "migration-inactive-flag" manifold worker returned unexpected error: migration lookup failed: no reachable servers
2018-12-30 05:56:25 ERROR juju.worker.dependency engine.go:632 "migration-minion" manifold worker returned unexpected error: migration lookup: migration lookup failed: no reachable servers
2018-12-30 05:56:25 ERROR juju.worker.dependency engine.go:632 "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot get machine 74: no reachable servers
2018-12-30 05:56:50 ERROR juju.worker.dependency engine.go:632 "migration-inactive-flag" manifold worker returned unexpected error: migration lookup failed: no reachable servers
2018-12-30 05:57:27 ERROR juju.worker.dependency engine.go:632 "migration-inactive-flag" manifold worker returned unexpected error: migration lookup failed: no reachable servers
2018-12-30 05:57:29 ERROR juju.worker.dependency engine.go:632 "migration-minion" manifold worker returned unexpected error: migration lookup: migration lookup failed: no reachable servers
2018-12-30 05:57:35 ERROR juju.worker.dependency engine.go:632 "upgrader" manifold worker returned unexpected error: cannot set agent version: cannot get machine 74: no reachable servers
Even though controllers are reachable:
$ nc -vz 10.25.2.111 17070
Connection to 10.25.2.111 17070 port [tcp/*] succeeded!
Restarting the juju-machine-X service via systemctl results in the agent immediately re-establishing a connection.
The juju_goroutines output is attached.
Full logs from juju-machine-74 machine.log are available at https:/ /private- fileshare. canonical. com/~jsing/ lp1810712/ juju-machine- 74.log. gz