inappropriate "relation-joined" for unit bounces untiter repeatedly

[Tim Penhey]

From a solutions QA test run of k8s on vsphere. Seems that we have a race condition somewhere.

Three units showed up as "failed", and on inspection the uniters are repeatedly restarting.

 https://solutions.qa.canonical.com/#/qa/testRun/a6491ea8-88bf-40e7-b49e-5b1c2b468a38

See graylog/0, easyrsa/0 and vsphere-integrator

There is potentially more than one bug here.

From greylog/0 we see the following in the unit logs:

2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:153 joining relation "filebeat:logstash graylog:beats"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:189 joined relation "filebeat:logstash graylog:beats"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:153 joining relation "apache2:reverseproxy graylog:website"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:189 joined relation "apache2:reverseproxy graylog:website"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:153 joining relation "graylog:elasticsearch elasticsearch:client"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:189 joined relation "graylog:elasticsearch elasticsearch:client"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:153 joining relation "graylog:mongodb mongodb:database"
2020-05-27 16:34:36 INFO juju.worker.uniter.relation statetracker.go:189 joined relation "graylog:mongodb mongodb:database"

From the machine-lock.log file, we only see relation-created for two of those relations:

2020-05-27 15:54:25 === agent unit-graylog-0 started ===
2020-05-27 15:54:40 unit-graylog-0: meterstatus (meter-status-changed), waited 0s, held 4s
2020-05-27 16:10:30 unit-graylog-0: uniter (run install hook), waited 0s, held 14m16s
2020-05-27 16:12:05 unit-graylog-0: uniter (run relation-created (19; app: mongodb) hook), waited 0s, held 2s
2020-05-27 16:12:06 unit-graylog-0: uniter (run relation-created (22; app: filebeat) hook), waited 0s, held 2s
2020-05-27 16:12:25 unit-graylog-0: uniter (run leader-elected hook), waited 0s, held 19s
2020-05-27 16:12:29 unit-graylog-0: uniter (run config-changed hook), waited 0s, held 4s
2020-05-27 16:13:14 unit-graylog-0: uniter (run start hook), waited 0s, held 45s
2020-05-27 16:13:14 unit-graylog-0: uniter (run relation-joined (17; unit: apache2/0) hook), waited 0s, held 0s
2020-05-27 16:13:18 unit-graylog-0: uniter (run relation-joined (17; unit: apache2/0) hook), waited 0s, held 0s
2020-05-27 16:13:23 unit-graylog-0: uniter (run relation-joined (17; unit: apache2/0) hook), waited 0s, held 0s
2020-05-27 16:13:28 unit-graylog-0: uniter (run relation-joined (17; unit: apache2/0) hook), waited 0s, held 0s
2020-05-27 16:13:36 unit-graylog-0: uniter (run relation-joined (17; unit: apache2/0) hook), waited 0s, held 0s

The uniter bounces with the following error:

2020-05-27 16:13:19 ERROR juju.worker.dependency engine.go:671 "uniter" manifold worker returned unexpected error: preparing operation "run relation-joined (17; unit: apache2/0) hook": inappropriate "relation-joined" for "apache2/0": relation is broken and cannot be changed further
2020-05-27 16:13:19 DEBUG juju.worker.dependency engine.go:673 stack trace:
relation is broken and cannot be changed further
/workspace/_build/src/github.com/juju/juju/worker/uniter/relation/state.go:58: inappropriate "relation-joined" for "apache2/0"
/workspace/_build/src/github.com/juju/juju/worker/uniter/relation/relationer.go:123:
/workspace/_build/src/github.com/juju/juju/worker/uniter/operation/executor.go:140: preparing operation "run relation-joined (17; unit: apache2/0) hook"
/workspace/_build/src/github.com/juju/juju/worker/uniter/resolver/loop.go:97:

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/11634