Multiple units told they are leaders

[Jacek Nykis]

In my environment I had 5 out of 8 units end up with hook errors.

On closer inspection I noticed that all failed units wrongly thought they were leaders and were trying to "leader-set" which was failing.

Once I run "juju resolved" on all affected units they worked fine.

I could not find anything interesting in the logs.

This is the 2nd bug I noticed recently that affects leadership election so they may be related. The other is bug 1721159.

juju 1.25.13
Ubuntu 14.04.5 LTS

[Junien F (axino)]

I have seen kind of the same thing just today, on a fresh 2.2.4 (<24h) model :

2017-10-12 19:28:26 INFO juju-log leader-elected fired. This unit is the new leader: foo/3
2017-10-12 19:28:26 DEBUG leader-elected ERROR cannot write leadership settings: cannot write settings: not the leader
[...]
2017-10-12 19:28:26 DEBUG leader-elected subprocess.CalledProcessError: Command '['leader-set', 'leader_id=foo/3']' returned non-zero exit status 1
2017-10-12 19:28:26 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1

[John A Meinel (jameinel)]

can you describe the charm where you're seeing this? and if there are steps to reproduce?

[Junien F (axino)]

It's our good friend ubuntu-repository-cache, same as in the linked bug. It's one of the rare (only ?) charm that actively uses leadership / has leader-* hooks, which could explain why we're seeing leadership problems only with it.

No step to reproduce as far as I'm aware, unfortunately.

I guess I can resolve the hook error and enable TRACE debug on that env, and wait for a repro ?

[John A Meinel (jameinel)]

We are unlikely to fix this in 1.25 unless there is a clear reproduction case.

[John A Meinel (jameinel)]

If we can reproduce this, then it is very likely worth fixing. Especially the 2.2.4 case seems to be we are actively electing a leader and it is getting demoted within the same second.
I wonder if the problem is actually that it has not successfully been promoted entirely? (could we have thought we requested the txn but it hasn't been fully committed before the next step?
)

[Jacek Nykis (jacekn)]

We've just hit this in a production environment running juju 2.2.6

We ended up with a non-leader unit stuck with failed "leader-elected" hook. The hook was trying to "leader-set" which returned error. We recovered by running "exit 0" inside debug-hooks.

So maybe the solution is to cancel "leader-elected" hooks on non-leaders?

[Haw Loeung (hloeung)]

Seeing this as well with a 2.2.6 environment.

| ubuntu@ip-172-30-18-69:~$ sudo juju-run ubuntu-repository-cache/5 'is-leader'
| False

| ubuntu@ip-172-30-18-69:~$ sudo juju-run ubuntu-repository-cache/5 'leader-get'
| leader_id: ubuntu-repository-cache/4

At some point, this unit must have been elected as the leader and now it keeps trying to run the leader-elected hook.

[John A Meinel (jameinel)]

This sounds like we got into a bad state where a hook failed, and our logic
around automatically retrying hooks that are in error state, is not a good
fit around leader-elected.

The one option from the charm is to just return if is_leader returns false
instead of failing. That would avoid having the hook go into error state
and triggering juju to retry the hook.

John
=:->

On Tue, Dec 5, 2017 at 3:27 AM, Haw Loeung <email address hidden>
wrote:

> Seeing this as well with a 2.2.6 environment.
>
> | ubuntu@ip-172-30-18-69:~$ sudo juju-run ubuntu-repository-cache/5
> 'is-leader'
> | False
>
> | ubuntu@ip-172-30-18-69:~$ sudo juju-run ubuntu-repository-cache/5
> 'leader-get'
> | leader_id: ubuntu-repository-cache/4
>
> At some point, this unit must have been elected as the leader and now it
> keeps trying to run the leader-elected hook.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1723184
>
> Title:
> Multiple units told they are leaders
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1723184/+subscriptions
>

[Junien F (axino)]

This is still happening with 2.4.2 by the way :

2018-08-29 09:13:59 INFO juju-log leader-elected fired. This unit is the new leader: ubuntu-repository-cache/2
2018-08-29 09:13:59 DEBUG leader-elected ERROR cannot write leadership settings: cannot write settings: not the leader

[Tim Penhey (thumper)]

Possibly a timing issue where the leadership changed before the unit was able to run the leader-elected hook.

Juju should ensure that the unit is still the leader before running the leader-elected hook to handle this case.

[Joseph Phillips (manadart)]

2.4: https://github.com/juju/juju/pull/9353
develop: https://github.com/juju/juju/pull/9438

[Joseph Phillips (manadart)]

The patch to address this included some logic that should have been removed.

It is addressed by https://github.com/juju/juju/pull/10301, and is the the edge versions for 2.5+.