<name>_relation_departed is sometimes called randomly

[Simon Déziel]

Note: This is probably the same as LP: #1945514 but I'm opening a new bug to provide fresh data.

Sometimes, when deploying an app, a relation-departed event is triggered at random:

$ juju deploy --resource snap=./lxd-cloud-cell_0+git.6c40e0c_amd64.snap ./lxd-cloud-cell_ubuntu-22.04-amd64.charm --config name=foo -n 3
Located local charm "lxd-cloud-cell", revision 37
Deploying "lxd-cloud-cell" from local charm "lxd-cloud-cell", revision 37 on ubuntu@22.04/stable

$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
test overlord juju-lxd/default 3.1.0 unsupported 18:57:52-05:00

App Version Status Scale Charm Channel Rev Exposed Message
lxd-cloud-cell active 3 lxd-cloud-cell 37 no

Unit Workload Agent Machine Public address Ports Message
lxd-cloud-cell/99 active idle 99 10.184.166.81
lxd-cloud-cell/100* active idle 112 10.184.166.56
lxd-cloud-cell/101 active idle 98 10.184.166.29

Machine State Address Inst id Base AZ Message
98 started 10.184.166.29 juju-a608ca-98 ubuntu@22.04 Running
99 started 10.184.166.81 juju-a608ca-99 ubuntu@22.04 Running
112 started 10.184.166.56 juju-a608ca-112 ubuntu@22.04 Running
113 started 10.184.166.31 juju-a608ca-113 ubuntu@22.04 Running
114 started 10.184.166.13 juju-a608ca-114 ubuntu@22.04 Running
115 started 10.184.166.136 juju-a608ca-115 ubuntu@22.04 Running

Relation provider Requirer Interface Type Message
lxd-cloud-cell:cluster lxd-cloud-cell:cluster cell-cluster peer

Something, somehow went wrong:

$ juju debug-log --level DEBUG --replay --exclude-module juju | grep -F ' 18:52:' | grep -F 'cluster_relation_departed.'
unit-lxd-cloud-cell-100: 18:52:31 DEBUG unit.lxd-cloud-cell/100.juju-log cluster:37: Emitting Juju event cluster_relation_departed.
unit-lxd-cloud-cell-101: 18:52:31 DEBUG unit.lxd-cloud-cell/101.juju-log cluster:37: Emitting Juju event cluster_relation_departed.
unit-lxd-cloud-cell-99: 18:52:33 DEBUG unit.lxd-cloud-cell/99.juju-log cluster:37: Emitting Juju event cluster_relation_departed.
unit-lxd-cloud-cell-99: 18:52:35 DEBUG unit.lxd-cloud-cell/99.juju-log cluster:37: Emitting Juju event cluster_relation_departed.

That wasn't something I asked for ;)

Please find attached all the needed pieces to reproduce as well as juju logs I hope are useful to track this down.

[Simon Déziel (sdeziel)]

I ran into this again but this time with juju 3.1.2:

$ juju deploy ./lxd-cloud-cell_ubuntu-22.04-amd64.charm cell-a --resource snap=./lxd-cloud-cell_0+git.6c40e0c_amd64.snap --config name=foo -n 3
Located local charm "lxd-cloud-cell", revision 34
Deploying "cell-a" from local charm "lxd-cloud-cell", revision 34 on ubuntu@22.04/stable

$ juju debug-log --color --level DEBUG --replay | grep -E 'cell-a/(2[89]|30)\.' | grep event
unit-cell-a-28: 20:31:59 DEBUG unit.cell-a/28.juju-log Emitting Juju event install.
unit-cell-a-29: 20:31:59 DEBUG unit.cell-a/29.juju-log Emitting Juju event install.
unit-cell-a-30: 20:31:59 DEBUG unit.cell-a/30.juju-log Emitting Juju event install.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-30: 20:32:02 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-29: 20:32:03 DEBUG unit.cell-a/29.juju-log Emitting Juju event config_changed.
unit-cell-a-28: 20:32:03 DEBUG unit.cell-a/28.juju-log Emitting Juju event config_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event config_changed.
unit-cell-a-29: 20:32:06 DEBUG unit.cell-a/29.juju-log Emitting Juju event start.
unit-cell-a-28: 20:32:06 DEBUG unit.cell-a/28.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:08 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:09 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell...