Leadership must be dropped before removing lead unit

Bug #1469731 reported by Robert Ayres on 2015-06-29
This bug affects 3 people
Affects Status Importance Assigned to Milestone
postgresql (Juju Charms Collection)

Bug Description

Juju version 1.23.3-trusty-amd64 through 2.3.8

If the lead unit is removed, it can remain leader right up until the point it is destroyed.

If the lead unit is unaware that it is being removed, which is the case prior to Juju 2.4, then it will make bad decisions and destroy data. For example, decommissioning nodes in a database cluster it thinks is going away, when in fact it will shortly be destroyed taking the data it thoughtfully drained with it.

With Juju 2.4, if the charm examines the charm state then it can tell it is being removed. The situation is better, with the remaining problem that leadership decisions will need to be deferred until the doomed leader is destroyed, and a new hook kicked off somewhere such as update-status, at which point a new leader is appointed and it can make the necessary decisions.

Curtis Hovey (sinzui) on 2015-06-29
tags: added: leadership
Changed in juju-core:
importance: Undecided → Medium
status: New → Triaged
William Reade (fwereade) wrote :

This sounds to me like another reason to implement target/active state on top of "current" state (what we currently show via the relation-* tools).

Active and target states are not *quite* the same, but I think they're both necessary to get a complete picture of what's going on inside your service. In either case, the state is a list of relations and units in those relations (although *not* their settings); the "active" list is composed of the relations that exists and the units that have entered their scope, and the "target" list is the projected relation state in the distant future (so dying relations and units are not shown; but all others are, even if no units have come up yet).

As a first cut, to be fleshed out more in future, it could look something like a `--view` param for `relation-ids` and `relation-list`, allowing values:

  * "current" -- the default value, works as today; usefully consistent lies about current state
  * "target" -- all accessible Alive relations, and all Alive units therein, including self
  * "active" -- all accessible relations, Alive and Dying; and all units, Alive and Dying, currently participating therein

In your particular case, I think you'd just check for your own presence with `--view=target`; if the target state has a happy community of units including yourself, you can expect to be around for a while; if you're not present, you should take the opportunity to hand over any critical state to your siblings; and if nobody's present then everything's shutting down.

I believe the distinction between "target" and "active" is necessary to allow charms to make the right decisions as they scale up and down; I'm very much open to the idea that there should be an analogous `minion-list` command accessible to service leaders, but aware that it's technically redundant with any existing peer relation (i.e. you *could* `relation-list --target $(relation-ids --target my-peer-relation`, but that's squicky).

I also think that we should *not* implement hooks to react to changes in target or active state. Changes to active state would be very close to 1:1 with the existing joined/departed hooks; changes to target state are likewise going to cluster unhelpfully and obscure logic flow; and in any case the active/target state change is merely a preview of an event that is *likely* to happen in the future.

I'm less keen on implementing --view for relation-get, though. The existing view is a weird mixture of current and target anyway, and trying to use it with target-state units or relations will make relation-get much more confusing and no more useful that I can see.


Stuart Bishop (stub) wrote :

This is causing failover to fail in my updated PostgreSQL charm, where the leader & master can be completely destroyed before leadership is lost or discovering that it is the doomed unit, and it never signals failover. In my case, I think I can rescue the situation in the leader-elected hook.

Changed in juju-core:
status: Triaged → Won't Fix
Robert Ayres (robert-ayres) wrote :

Has this actually been fixed in Juju 2.0?

Just closing for the sake of lowering the bug count won't make bugs go away.

Anastasia (anastasia-macmood) wrote :

The bug is closed because "juju-core" project tracks bugs for Juju 1. This project only accepts Critical bugs. Consequently, this bug will not be fixed for Juju 1, current series 1.25.

The project that tracks Juju 2 bugs is "juju". We have moved all the bugs that have been explicitly opened against Juju 2. As Juju 2 is very different to Juju 1, majority of bugs against Juju 1 are not applicable.

This one, as you rightly pointed out, needs to be addressed. I'll re-target it to "juju".

We have sent an email out to juju-dev 2 months ago advertising our move. We have also waited for 2 months to make sure that no bugs have been forgotten or left behind before we closed "juju-core". Apologies if decisive closure of this bug took you by surprise :D Thank you for reaching out!

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
milestone: none → 2.1.0
Curtis Hovey (sinzui) on 2017-02-17
Changed in juju:
milestone: 2.1-rc2 → none
Changed in juju:
importance: Medium → High
tags: added: charm charmers
Stuart Bishop (stub) on 2018-05-25
summary: - Leader should probably change when removing unit
+ Leadership must be dropped before removing lead unit
Stuart Bishop (stub) on 2018-05-25
description: updated
tags: added: canonical-is
Ian Booth (wallyworld) on 2019-02-19
tags: added: teardown
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers