leader-settings-changed firing before unit departure

Bug #1964582 reported by Mia Altieri
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Ian Booth

Bug Description

Hi team. I had some weird behavior for this set up / situation

For Example. If you have these units:
```
A *
B
C
```
and `A` departs then you get these events hooks for each unit:
`A`: `leader-settings-changed` , `mongodb-storage-detaching`, `mongodb-relation-departed`(for A-B), `mongodb-relation-departed`(for A-C), `stop`, `remove`
`B`: `mongodb-relation-departed` (for A-B)
`C`: `mongodb-relation-departed` (for A-C),

According to @wallyworld and @rwcarlsen this is unexpected behaviour and should be looked into. As requested, [this](https://github.com/canonical/mongodb-operator/tree/DPE-65-relation-departed) is my repo/branch where I observed this behaviour. It can be observed by first creating the necessary environment according to [CONTRIBUTING.md](https://github.com/canonical/mongodb-operator/blob/main/CONTRIBUTING.md). Once the environment has been created you can reproduce this with:
```
# pack and deploy
charmcraft pack
juju deploy -n3 ./*charm

# in a separate session
juju debug-hooks <leader_unit>

# remove unit
juju remove-unit <leader_unit
```

Revision history for this message
Ian Booth (wallyworld) wrote :

Looking at the intent of the code, the unit agent has this logic:

// We want to run the leader settings hook if we're
// not the leader and the settings have changed.
if !localState.Leader && localState.LeaderSettingsVersion != remoteState.LeaderSettingsVersion {
    return opFactory.NewRunHook(hook.Info{Kind: hooks.LeaderSettingsChanged})
}

So we should only run leader-settings-changed if the unit is not the leader. Unit A was the leader at the time it was removed - and we are supposed to only take away leadership when unit A goes to Dead, ie after all hooks have been run. So it does seem something is amiss here.

Changed in juju:
milestone: none → 2.9.27
importance: Undecided → High
status: New → Triaged
Revision history for this message
Ian Booth (wallyworld) wrote :

The logic which determined what to run for leadership changes did not take account of a unit which was dying.

https://github.com/juju/juju/pull/13834

Changed in juju:
assignee: nobody → Ian Booth (wallyworld)
status: Triaged → In Progress
Revision history for this message
Ian Booth (wallyworld) wrote :

Looking at the code, this behaviour has been in place for a long time, probably ever since leadership was first implemented. So we'll need to consider if this is a change we can reasonably make in 2.9 since there may be charms which rely on this hook being fired during unit teardown.

Changed in juju:
milestone: 2.9.27 → 2.9.28
Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.