"ERROR could not determine leader" but `juju status` says there is a leader

[Casey Marshall]

(mojo-stg-ua-event-bus)stg-ua-event-bus@wendigo:~$ juju version
2.6.10-trusty-amd64

`juju status` says:
Model Controller Cloud/Region Version SLA Timestamp
stg-ua-event-bus prodstack-is prodstack-45/bootstack-ps45 2.6.10 unsupported 20:52:36Z

I'm not able to run actions on the leader of an application that is scaled out:

(mojo-stg-ua-event-bus)stg-ua-event-bus@wendigo:~$ juju run-action kafka/leader list-topics
ERROR could not determine leader for "kafka"

In the past, I used to be able to run actions on the leader. My scripts take advantage of this, so I don't have to do `juju status` jq magic to find a unit to run actions on.

What's weird is there seems to be a "leader" in `juju status` output (resorting to jq magic...):

(mojo-stg-ua-event-bus)stg-ua-event-bus@wendigo:~$ juju status kafka --format json| jq -r '.applications["kafka"].units|to_entries[]|select(.value["leader"])|.key'
kafka/53

(mojo-stg-ua-event-bus)stg-ua-event-bus@wendigo:~$ juju status kafka --format json| jq -r '.applications["kafka"].units|to_entries[]|select(.value["leader"])|.value'
...
  "public-address": "10.15.118.104",
  "open-ports": [
    "9093/tcp"
  ],
  "machine": "101",
  "leader": true,
  "juju-status": {
    "version": "2.6.10",
    "since": "15 Nov 2019 21:59:29Z",
    "current": "idle"
  },
  "workload-status": {
    "since": "25 Jul 2019 10:48:11Z",
    "message": "ready",
    "current": "active"
  }
}

No obvious errors in the unit logs for this application.

[Ian Booth (wallyworld)]

What happens at the back end is a call to ApplicationLeaders() on the *State struct.
It seems kafka is not in the known list of leadership lease holders.

Can you confirm if you are running with legacy leases or raft leases?

If raft leases, can you look at the leaseholders collection to see if there's a "application-leadership" lease for kafka in there?

[Christian Muirhead (2-xtian)]

Could you post the contents of /var/log/juju/lease.log from the controller machine(s)? If there's an error writing the lease claim to the db it should be logged there.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Alexander Balderson (asbalderson)]

We had this come up in a solutions qa deploy.

2020-07-12-09:19:10 root ERROR [localhost] Command failed: juju run -m kubernetes -u vault/leader 'export VAULT_ADDR=http://127.0.0.1:8200 && /snap/bin/vault operator init -key-shares=5 -key-threshold=3'
2020-07-12-09:19:10 root ERROR [localhost] STDOUT follows:
ERROR could not determine leader for "vault"

there was a small window where the lease expired for vault but not near where the command was called:

2020/07/12 08:16:56.529201 claimed "cde71e82-923c-4d64-8c68-0a80ffbbb8c4:singular-controller#887bf8ef-631c-4471-8d70-7c1fda32c1bd#" for "machine-0"
2020/07/12 08:16:56.661828 claimed "cde71e82-923c-4d64-8c68-0a80ffbbb8c4:singular-controller#cde71e82-923c-4d64-8c68-0a80ffbbb8c4#" for "machine-0"
2020/07/12 08:16:56.691419 claimed "f78772e4-cf6e-4803-8315-6c43d26e7538:singular-controller#f78772e4-cf6e-4803-8315-6c43d26e7538#" for "machine-0"
2020/07/12 08:26:41.343650 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:singular-controller#fe56cd3b-dfbd-4c17-83bc-78d42e9688f0#" for "machine-0"
2020/07/12 08:33:00.393369 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#nagios#" for "nagios/0"
2020/07/12 08:33:02.912030 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#prometheus#" for "prometheus/0"
2020/07/12 08:33:18.370298 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#easyrsa#" for "easyrsa/0"
2020/07/12 08:33:21.564650 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#vault#" for "vault/2"
2020/07/12 08:33:49.436550 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#elasticsearch#" for "elasticsearch/0"
2020/07/12 08:34:21.771141 expired "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#vault#"
2020/07/12 08:34:21.774884 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#vault#" for "vault/2"
2020/07/12 08:34:28.219153 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#filebeat#" for "filebeat/0"
2020/07/12 08:34:54.569267 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#canonical-livepatch#" for "canonical-livepatch/0"
2020/07/12 08:34:57.683200 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#ntp#" for "ntp/0"
2020/07/12 08:35:00.619431 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#nrpe-host#" for "nrpe-host/0"
2020/07/12 08:35:19.526314 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#hacluster-vault#" for "hacluster-vault/0"
2020/07/12 08:35:31.667826 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#telegraf#" for "telegraf/0"
2020/07/12 08:37:27.725550 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#graylog#" for "graylog/0"
2020/07/12 08:38:53.440411 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#apache2#" for "apache2/0"
2020/07/12 08:38:55.374714 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#grafana#" for "grafana/0"
2020/07/12 08:38:56.086081 claimed "fe56cd3b-dfbd-4c17-83bc-78d42e9688f0:application-leadership#ceph-osd#" for "ceph-osd/5"
2020/07/12 08:38:57.904492 claimed "fe56cd3...

[Pen Gale (pengale)]

This might be a duplicate of https://bugs.launchpad.net/juju/+bug/1532085

@asbalderson does vault/2 exist at this point?

[John A Meinel (jameinel)]

If you have units that are cycling (more common on K8s than on other clouds), then there is a higher likelyhood that there will be times that there is no leader. The current design is that we guarantee to never have more than 1 leader, but not that there will always be a leader. (we give a unit a lease for 1 minute, if it dies, there will be a time where there is no leader before the next one takes over.)

I know there have been some requests that if we know a unit is going away completely, we could expire its lease faster, so something else could take over leadership. I'd personally actually rather see a K8s agent that is the "application agent" that is always the leader, rather than unit agents that feel like they might at any time become leader.
(Especially in pod configuration, all the unit agents run in the same pod anyway, so there isn't an HA reason to have different units.)

[Ponnuvel Palaniyappan (pponnuvel)]

Facing a similar issue but the leader is permanently - not just a momentary problem - missing/inaccessible.

`juju status` does show there's a leader. But any command issued to `<unit>/leader` fails with:
```
# juju run --unit <unit>/leader 'relation-ids db-router'
ERROR could not determine leader for "<unit>"
```

I looked through the raft logs and it all looks OK.

In the mongoDB, there's no leadership entry in the leaseholders collection at all for the unit that `juju status` says is the leader.

[John A Meinel (jameinel)]

One thing you could try, is to force the leader to not renew its lease (by stopping the associated unit agent) and then start it again. That should trigger a new election, which should then also update the database.

At the very least, you could then grab logs from around that time to see if something else is failing.

```
$ juju ssh unit/X
$$ juju_stop_unit unit/X
$$ sleep 65
$$ juju_start_unit unit/X
```

[Ponnuvel Palaniyappan (pponnuvel)]

```
$ juju ssh unit/X
$$ juju_stop_unit unit/X
$$ sleep 65
$$ juju_start_unit unit/X
```

This approach did work in my test environment (where artificially recreated by manually removed the leader entry from leaseholders collection). However, this didn't work in the user environment where this issue originated from.

[Ponnuvel Palaniyappan (pponnuvel)]

I also asked the user to insert the leadership entry into the
leaseholders collection (which doesn't have any entry for the
leadership of this unit). However, even inserting a leader -- whatever
`juju status` reports as the leader for the application -- still doesn't work.

[Ponnuvel Palaniyappan (pponnuvel)]

The above was reported on Juju 2.9.11 (originally reported against 2.6.10).

From the controller logs and Raft logs, there's not anything to suggest problems.

That DB surgery didn't fix is still a puzzle to me as I could artificially remove
a leader entry to reproduce it and re-inserting it appear to correct it. So why
the leadership entry doesn't get added into the leaseholder collection is the problematic
bit as the Raft lease logs suggest the leasing -> expiring -> reclaiming works normally.

[Juan M. Tirado (tiradojm)]

If you're not running in HA could you check if the link below helps?

https://discourse.charmhub.io/t/cannot-run-command-juju-add-machine-it-stays-in-pending-state/5218/3

[Ponnuvel Palaniyappan (pponnuvel)]

@Juan

I've checked that before but didn't think there was any "storage issues" here.

Tried manually revoking the lease for the cinder-mysql-router application (the problematic one)
with `juju_revoke_leases -m <model-uuid> -l cinder-mysql-router` and no success.

But didn't go through
"
Stopping the controller.
Deleting the contents of /var/lib/juju/raft.
Starting the controller.
"

as there's no indication of any corruption or storage issues. But if you can confirm it is completely safe to get rid of /var/lib/juju/raft, I suppose that's worth a try given where we are.

[Juan M. Tirado (tiradojm)]

This is safe for controllers not running on HA. It may help to fix your current issue.

[Ponnuvel Palaniyappan (pponnuvel)]

Hi Juan,

The user environment actually has HA with 3 controllers. So deleting '/var/lib/juju/raft' isn't possible unfortunately.

[Joseph Phillips (manadart)]

This what to do.

- Shut down the controllers.
- Delete the contents of /var/lib/juju/raft on all controllers.
- Get this file: https://private-fileshare.canonical.com/~manadart/raft-logs.
- Drop it into the directory above on one controller and start it.
- Start the other controllers.

The file is a Raft log from a bootstrapped controller that simply has a log entry indicating that this node is the leader. It will ensure that you can come back up after deleting the directory on HA controllers.

[Ponnuvel Palaniyappan (pponnuvel)]

@Joseph, that workaround did work, Thanks.

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.