Can't destroy service -- juju status "failed" & unknown relation 0 error in log

[Casey Marshall]

Juju 2.0, built from master at 0bd48e6.

Steps to reproduce:

I deployed an etcd charm with 3 units, related it to prometheus, everything worked, no problem.

Did `juju destroy-service etcd` and two of the etcd units died and went away. However, one of them is stuck and repeated `juju destroy-service etcd` commands do not remove it.

The unit agent log on the stuck unit shows:

2016-05-13 13:32:41 INFO juju.worker.uniter resolver.go:120 committing "relation-broken" hook
2016-05-13 13:32:42 INFO juju.worker.uniter uniter.go:355 unit "etcd/2" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:42 ERROR juju.worker.dependency engine.go:526 "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:42 INFO juju.worker.leadership tracker.go:182 etcd/2 will renew etcd leadership at 2016-05-13 13:33:11.869693571 +0000 UTC
2016-05-13 13:32:45 INFO worker.uniter.jujuc tools.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-etcd-2
2016-05-13 13:32:45 INFO worker.uniter.jujuc tools.go:40 was a symlink, now looking at /var/lib/juju/tools/2.0-beta7.1-xenial-amd64
2016-05-13 13:32:45 INFO juju.worker.uniter.relation relations.go:431 joining relation "prometheus:scrape etcd:target"
2016-05-13 13:32:45 INFO juju.worker.uniter.relation relations.go:467 joined relation "prometheus:scrape etcd:target"
2016-05-13 13:32:45 INFO juju.worker.uniter uniter.go:164 unit "etcd/2" started
2016-05-13 13:32:45 INFO juju.worker.uniter uniter.go:203 hooks are retried true
2016-05-13 13:32:45 INFO juju.worker.uniter resolver.go:120 committing "relation-broken" hook
2016-05-13 13:32:45 INFO juju.worker.uniter uniter.go:355 unit "etcd/2" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:45 ERROR juju.worker.dependency engine.go:526 "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:48 INFO worker.uniter.jujuc tools.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-etcd-2
2016-05-13 13:32:48 INFO worker.uniter.jujuc tools.go:40 was a symlink, now looking at /var/lib/juju/tools/2.0-beta7.1-xenial-amd64
2016-05-13 13:32:48 INFO juju.worker.uniter.relation relations.go:431 joining relation "prometheus:scrape etcd:target"
2016-05-13 13:32:48 INFO juju.worker.uniter.relation relations.go:467 joined relation "prometheus:scrape etcd:target"
2016-05-13 13:32:48 INFO juju.worker.uniter uniter.go:164 unit "etcd/2" started
2016-05-13 13:32:48 INFO juju.worker.uniter uniter.go:203 hooks are retried true
2016-05-13 13:32:48 INFO juju.worker.uniter resolver.go:120 committing "relation-broken" hook
2016-05-13 13:32:48 INFO juju.worker.uniter uniter.go:355 unit "etcd/2" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:48 ERROR juju.worker.dependency engine.go:526 "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:51 INFO worker.uniter.jujuc tools.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-etcd-2
2016-05-13 13:32:51 INFO worker.uniter.jujuc tools.go:40 was a symlink, now looking at /var/lib/juju/tools/2.0-beta7.1-xenial-amd64
2016-05-13 13:32:51 INFO juju.worker.uniter.relation relations.go:431 joining relation "prometheus:scrape etcd:target"
2016-05-13 13:32:51 INFO juju.worker.uniter.relation relations.go:467 joined relation "prometheus:scrape etcd:target"
2016-05-13 13:32:51 INFO juju.worker.uniter uniter.go:164 unit "etcd/2" started
2016-05-13 13:32:51 INFO juju.worker.uniter uniter.go:203 hooks are retried true
2016-05-13 13:32:51 INFO juju.worker.uniter resolver.go:120 committing "relation-broken" hook
2016-05-13 13:32:51 INFO juju.worker.uniter uniter.go:355 unit "etcd/2" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
2016-05-13 13:32:51 ERROR juju.worker.dependency engine.go:526 "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0

[Casey Marshall (cmars)]

I'm able to recover from this condition with `juju remove-machine --force`, but that `unknown relation: 0` error is probably indicative of some problems.

[Cheryl Jennings (cherylj)]

Casey - if you have logs from the controller, can you attach them?

[Casey Marshall (cmars)]

Here's the machine log from the controller: https://pastebin.canonical.com/156442/

[Anastasia (anastasia-macmood)]

This has been fixed in Juju 2.1x by the fix of bug # 1643430