race condition when deploying multiple etcd applications against one easyrsa

Bug #1835056 reported by Nobuto Murata on 2019-07-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
EasyRSA Charm
Undecided
Cory Johns

Bug Description

How to reproduce:

juju deploy ./<attached-bundle>.yaml
juju-wait -w (or wait until all etcd applications are up and running)

for app in {c..f}; do
    juju deploy --series=bionic etcd my-etcd-$app
    juju add-relation easyrsa my-etcd-$app
done

Expected result:

All etcd applications are up and running.

Actual:
Applications added after the initial bundle deployment will be stuck with "Missing relation to certificate authority" even if those actually have relations to easyrsa.

Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.0.9.97 Certificate Authority connected.
my-etcd-a/0* active idle 1 10.0.9.193 2379/tcp Healthy with 1 known peer
my-etcd-b/0* active idle 2 10.0.9.64 2379/tcp Healthy with 1 known peer
my-etcd-c/0* blocked idle 3 10.0.9.170 Missing relation to certificate authority.
my-etcd-d/0* blocked idle 4 10.0.9.140 Missing relation to certificate authority.
my-etcd-e/0* blocked idle 5 10.0.9.145 Missing relation to certificate authority.
my-etcd-f/0* blocked idle 6 10.0.9.56 Missing relation to certificate authority.

Nobuto Murata (nobuto) wrote :
George Kraft (cynerva) wrote :

This looks like it has the same root cause as https://bugs.launchpad.net/charm-etcd/+bug/1832883. The easyrsa charm only publishes client certs once. In your case, juju most likely had not established all of the relations when easyrsa decided to publish its certs. The relations that were established after that moment were never provided the client certs.

Here is a workaround. After the deployment has settled, you can force easyrsa to re-publish the client cert to all of its relations:

juju run --unit easyrsa/0 -- charms.reactive clear_flag easyrsa.global-client-cert.created

Can you try that and see if it helps?

Nobuto Murata (nobuto) wrote :

@Geroge,

Indeed, the command to clear the flag worked. It didn't take effect immediately, but probably it was picked up by update-status hook or something later.

Cory Johns (johnsca) on 2019-07-09
Changed in charm-easyrsa:
status: New → Confirmed
Cory Johns (johnsca) on 2019-07-09
Changed in charm-etcd:
status: New → In Progress
Changed in charm-easyrsa:
status: Confirmed → In Progress
assignee: nobody → Cory Johns (johnsca)
Cory Johns (johnsca) wrote :
Changed in charm-easyrsa:
status: In Progress → Fix Committed
Changed in charm-etcd:
status: In Progress → Fix Committed
Changed in charm-easyrsa:
milestone: none → 1.15+ck1
no longer affects: charm-etcd
Changed in charm-easyrsa:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers