New unit fails to see the whole cluster

[Loïc Gomez]

Adding a new unit today, I've been hit by a bug where the new unit failed to join the cluster and see the rest of it.

2023-06-27 05:18:08 INFO unit.ubuntu-repository-cache/6.juju-log server.go:316 cluster:1: Syncing known_hosts @ /home/www-sync/.ssh/known_hosts.
2023-06-27 05:18:08 INFO unit.ubuntu-repository-cache/6.juju-log server.go:316 cluster:1: Updating metadata on a peer
2023-06-27 05:18:08 ERROR unit.ubuntu-repository-cache/6.juju-log server.go:316 cluster:1: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 379, in dispatch
    _invoke(hook_handlers)
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/charm/reactive/ubuntu_repository_cache.py", line 225, in cluster_relation_joined
    mirror.peer_update_metadata()
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/charm/lib/ubuntu_repository_cache/mirror.py", line 371, in peer_update_metadata
    _nonleader_update_metadata()
  File "/var/lib/juju/agents/unit-ubuntu-repository-cache-6/charm/lib/ubuntu_repository_cache/mirror.py", line 276, in _nonleader_update_metadata
    leader_rel = rel[config['leader_unit']]
KeyError: 'ubuntu-repository-cache/4'

Unit is in status hook failed: "cluster-relation-changed"

This is on a unit that properly joined the cluster:
ubuntu@machine-6:~$ sudo juju-run ubuntu-repository-cache/5 "relation-list -r cluster:1"
ubuntu-repository-cache/0
ubuntu-repository-cache/1
ubuntu-repository-cache/2
ubuntu-repository-cache/4
ubuntu-repository-cache/6

This is on the unit failing to join the cluster:
ubuntu@machine-7:~$ sudo juju-run ubuntu-repository-cache/6 "relation-list -r cluster:1"
ubuntu@machine-7:~$

[Loïc Gomez (kotodama)]

Also had to run `juju resolved ubuntu-repository-cache/6` about 8 times or so to get it to resolve and start syncing data.