Leadership must be dropped before removing lead unit

[Robert Ayres]

Juju version 1.23.3-trusty-amd64 through 2.3.8

If the lead unit is removed, it can remain leader right up until the point it is destroyed.

If the lead unit is unaware that it is being removed, which is the case prior to Juju 2.4, then it will make bad decisions and destroy data. For example, decommissioning nodes in a database cluster it thinks is going away, when in fact it will shortly be destroyed taking the data it thoughtfully drained with it.

With Juju 2.4, if the charm examines the charm state then it can tell it is being removed. The situation is better, with the remaining problem that leadership decisions will need to be deferred until the doomed leader is destroyed, and a new hook kicked off somewhere such as update-status, at which point a new leader is appointed and it can make the necessary decisions.

[William Reade (fwereade)]

This sounds to me like another reason to implement target/active state on top of "current" state (what we currently show via the relation-* tools).

Active and target states are not *quite* the same, but I think they're both necessary to get a complete picture of what's going on inside your service. In either case, the state is a list of relations and units in those relations (although *not* their settings); the "active" list is composed of the relations that exists and the units that have entered their scope, and the "target" list is the projected relation state in the distant future (so dying relations and units are not shown; but all others are, even if no units have come up yet).

As a first cut, to be fleshed out more in future, it could look something like a `--view` param for `relation-ids` and `relation-list`, allowing values:

  * "current" -- the default value, works as today; usefully consistent lies about current state
  * "target" -- all accessible Alive relations, and all Alive units therein, including self
  * "active" -- all accessible relations, Alive and Dying; and all units, Alive and Dying, currently participating therein

In your particular case, I think you'd just check for your own presence with `--view=target`; if the target state has a happy community of units including yourself, you can expect to be around for a while; if you're not present, you should take the opportunity to hand over any critical state to your siblings; and if nobody's present then everything's shutting down.

I believe the distinction between "target" and "active" is necessary to allow charms to make the right decisions as they scale up and down; I'm very much open to the idea that there should be an analogous `minion-list` command accessible to service leaders, but aware that it's technically redundant with any existing peer relation (i.e. you *could* `relation-list --target $(relation-ids --target my-peer-relation`, but that's squicky).

I also think that we should *not* implement hooks to react to changes in target or active state. Changes to active state would be very close to 1:1 with the existing joined/departed hooks; changes to target state are likewise going to cluster unhelpfully and obscure logic flow; and in any case the active/target state change is merely a preview of an event that is *likely* to happen in the future.

I'm less keen on implementing --view for relation-get, though. The existing view is a weird mixture of current and target anyway, and trying to use it with target-state units or relations will make relation-get much more confusing and no more useful that I can see.

Thoughts?

[Stuart Bishop (stub)]

This is causing failover to fail in my updated PostgreSQL charm, where the leader & master can be completely destroyed before leadership is lost or discovering that it is the doomed unit, and it never signals failover. In my case, I think I can rescue the situation in the leader-elected hook.

[Robert Ayres (robert-ayres)]

Has this actually been fixed in Juju 2.0?

Just closing for the sake of lowering the bug count won't make bugs go away.

[Anastasia (anastasia-macmood)]

@Robert
The bug is closed because "juju-core" project tracks bugs for Juju 1. This project only accepts Critical bugs. Consequently, this bug will not be fixed for Juju 1, current series 1.25.

The project that tracks Juju 2 bugs is "juju". We have moved all the bugs that have been explicitly opened against Juju 2. As Juju 2 is very different to Juju 1, majority of bugs against Juju 1 are not applicable.

This one, as you rightly pointed out, needs to be addressed. I'll re-target it to "juju".

We have sent an email out to juju-dev 2 months ago advertising our move. We have also waited for 2 months to make sure that no bugs have been forgotten or left behind before we closed "juju-core". Apologies if decisive closure of this bug took you by surprise :D Thank you for reaching out!

[Ian Booth (wallyworld)]

Once a lead unit transitions to a life state of Dead upon destruction, it is appropriate for its leadership to be relinquished at that point. This would happen as part of the cleanup process, and a consequence is that a new lead unit would be elected immediately. I don't think there's a need to expose to the charm the ability to abdicate explicitly.

[Stuart Bishop (stub)]

When does transition to 'a life state of Dead upon destruction' happen exactly?

My main concern is that leadership needs to be relinquished before relation-departed hooks are run on the unit. Once relation-departed hooks start running, the unit has an incomplete picture of the application state and can no longer make informed decisions.

[Trent Lloyd (lathiat)]

These two bugs appear to be effectively duplicates #1469731 and #1532085

[Ian Booth (wallyworld)]

The answer #6, there are leader only calls that can be mad even relations are departed, eg updating application data bag. Until a unit is dead, it can still invoke hook commands.

The safest option is to wait for the unit to be dead, meaning it has left all relation scopes and is just about to be removed.