kill-controller is stuck, lots of "lease manager stopped" errors

[Casey Marshall]

juju-2.0-beta5, lxd controller, lxd 2.0.0-0ubuntu4

`juju kill-controller` is stuck and not making progress in tearing down my models & controller.

Messages writing to machine-0.log on the controller every 3 seconds like this:

2016-04-21 16:47:47 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:47:50 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:47:53 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:47:56 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:47:59 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:02 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:05 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:08 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:11 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:14 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:17 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:20 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:23 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped
2016-04-21 16:48:26 ERROR juju.worker.dependency engine.go:526 "is-responsible-flag" manifold worker returned unexpected error: lease manager stopped

I'm going to nuke the lxc containers and try again with master to see if the issue has been fixed...

[Cheryl Jennings (cherylj)]

I saw the same problem when trying to clean up the environment in bug #1572237, and I think this is the same underlying issue. I think for that bug, the fix William is working on is specific to the pinger, but there is a more general issue of sub-state workers not being managed workers yet. We may use this bug for that issue.

And, there's also bug #1566426 which asks for kill-controller to time out when destroying through the API when we have cases like this where the environment is broken.

[Cheryl Jennings (cherylj)]

Looks like William has PRs for this in master:
https://github.com/juju/juju/pull/5355
https://github.com/juju/juju/pull/5367

He's evaluating the risk of backporting to 1.25.

[Anastasia (anastasia-macmood)]

@William - have you considered whether the fix can be backported to 1.25 as per comment #2?

[Anastasia (anastasia-macmood)]

From William:

It should be possible [to backport], yes.
Worst case it wants to run some worker that 1.25 doesn't have, but that should be trivial to change.

[Adam Stokes (adam-stokes)]

I've seen this on juju 2 beta17 with the same lease manager errors. The agents become in a blocked state with the agent status as failed.

Bouncing jujud on the controller unwedges this and my deployments will continue.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.