Charm can fail due to missing private key on scale out

[Liam Young]

In a recent deployment a single unit failed because it had received a certificates relation joined event but had no private key available to generate a csr.

Looking at the code the generation of the private key happens when an install event is received BUT if the peer relation is not ready the request is deferred. In the case of this unit the event was deferred, then the pod was killed (I assume by k8s), a new pod was created and the install hook did not fire so no key was generated.

2023-09-15T09:50:14.783Z [container-agent] 2023-09-15 09:50:14 ERROR juju-log certificates:86: Uncaught exception while in charm code:
2023-09-15T09:50:14.783Z [container-agent] Traceback (most recent call last):
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/./src/charm.py", line 498, in <module>
2023-09-15T09:50:14.783Z [container-agent] main(OVNCentralOperatorCharm)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops/main.py", line 441, in main
2023-09-15T09:50:14.783Z [container-agent] _emit_charm_event(charm, dispatcher.event_name)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops/main.py", line 149, in _emit_charm_event
2023-09-15T09:50:14.783Z [container-agent] event_to_emit.emit(*args, **kwargs)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops/framework.py", line 344, in emit
2023-09-15T09:50:14.783Z [container-agent] framework._emit(event)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops/framework.py", line 841, in _emit
2023-09-15T09:50:14.783Z [container-agent] self._reemit(event_path)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops/framework.py", line 930, in _reemit
2023-09-15T09:50:14.783Z [container-agent] custom_handler(event)
2023-09-15T09:50:14.783Z [container-agent] File "/var/lib/juju/agents/unit-ovn-central-1/charm/venv/ops_sunbeam/relation_handlers.py", line 909, in _on_certificates_relation_joined
2023-09-15T09:50:14.783Z [container-agent] private_key=private_key.encode(),
2023-09-15T09:50:14.783Z [container-agent] AttributeError: 'NoneType' object has no attribute 'encode'

[Guillaume Boutry (gboutry)]

Fix commited in https://review.opendev.org/c/openstack/charm-ops-sunbeam/+/899713