race condition when deploying multiple etcd applications against one easyrsa

[Nobuto Murata]

How to reproduce:

juju deploy ./<attached-bundle>.yaml
juju-wait -w (or wait until all etcd applications are up and running)

for app in {c..f}; do
    juju deploy --series=bionic etcd my-etcd-$app
    juju add-relation easyrsa my-etcd-$app
done

Expected result:

All etcd applications are up and running.

Actual:
Applications added after the initial bundle deployment will be stuck with "Missing relation to certificate authority" even if those actually have relations to easyrsa.

Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.0.9.97 Certificate Authority connected.
my-etcd-a/0* active idle 1 10.0.9.193 2379/tcp Healthy with 1 known peer
my-etcd-b/0* active idle 2 10.0.9.64 2379/tcp Healthy with 1 known peer
my-etcd-c/0* blocked idle 3 10.0.9.170 Missing relation to certificate authority.
my-etcd-d/0* blocked idle 4 10.0.9.140 Missing relation to certificate authority.
my-etcd-e/0* blocked idle 5 10.0.9.145 Missing relation to certificate authority.
my-etcd-f/0* blocked idle 6 10.0.9.56 Missing relation to certificate authority.

[George Kraft (cynerva)]

This looks like it has the same root cause as https://bugs.launchpad.net/charm-etcd/+bug/1832883. The easyrsa charm only publishes client certs once. In your case, juju most likely had not established all of the relations when easyrsa decided to publish its certs. The relations that were established after that moment were never provided the client certs.

Here is a workaround. After the deployment has settled, you can force easyrsa to re-publish the client cert to all of its relations:

juju run --unit easyrsa/0 -- charms.reactive clear_flag easyrsa.global-client-cert.created

Can you try that and see if it helps?

[Nobuto Murata (nobuto)]

@Geroge,

Indeed, the command to clear the flag worked. It didn't take effect immediately, but probably it was picked up by update-status hook or something later.

[Cory Johns (johnsca)]

Fixed in https://github.com/charmed-kubernetes/layer-easyrsa/pull/22

[George Kraft (cynerva)]

Cherry-picked to stable branch:

https://github.com/charmed-kubernetes/layer-easyrsa/commit/32c5e99b3e984962fbc177cad5628bc77a5da997