missing CA cert

[Marian Gasparovic]

During deployment leader unit stays in Blocked state reporting Missing CA cert although auto-generate-root-ca-cert is set to True.

I did not find anything strange in log files.

juju srashdump - https://oil-jenkins.canonical.com/artifacts/345242f2-6680-4d39-9d36-e6d59c9eafff/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-10-04-03.38.32.tar.gz

Other files

https://oil-jenkins.canonical.com/artifacts/345242f2-6680-4d39-9d36-e6d59c9eafff/index.html

It happened only in once test run so far

[Moises Emilio Benzan Mora (moisesbenzan)]

Also found on this run:
https://solutions.qa.canonical.com/testruns/testRun/699d3da7-5208-4bae-bfe5-661218adc4c5

Link to artifacts: https://oil-jenkins.canonical.com/artifacts/699d3da7-5208-4bae-bfe5-661218adc4c5/index.html

However this is a kubernetes deployment on baremetal (fkb-master-kubernetes-focal-baremetal)

[Marian Gasparovic (marosg)]

I just hit it in manual run with k8s on baremetal.

[Felipe Reyes (freyes)]

Hello,

Looking at this bundle[0], the option auto-generate-root-ca-cert is not set for vault:

```
  vault:
    bindings:
      ? ''
      : internal-space
    charm: cs:vault
    constraints: spaces=oam-space
    num_units: 3
    to:
    - '6'
    - '7'
    - '8'
```

the logs confirm that this option wasn't set to True.

 ~/Downloads/e4863acb-a33d-42e4-b677-ebb9dc5714b7 $ grep -P '^tracer: set flag config.*.auto-generate-root-ca-cert' vault_*/var/log/juju/unit-vault-*.log | cut -d' ' -f4 | sort | uniq -c
      3 config.changed.auto-generate-root-ca-cert
    344 config.default.auto-generate-root-ca-cert

Is there an overlay that I could be missing where the config option is set to True?

[0] https://oil-jenkins.canonical.com/artifacts/345242f2-6680-4d39-9d36-e6d59c9eafff/generated/generated/kubernetes/bundle.yaml

[Paul Goins (vultaire)]

I've hit this on a cloud which was upgraded from bionic to focal. The vault cluster itself appears totally fine on all 3 units, but one of the charm units is reporting this message.

[Nikolay Vinogradov (nikolay.vinogradov)]

Just seen that behavior deploying ussuri/edge bundle with 1-unit vault charm from 1.7/stable with auto-generate-root-ca-cert: true.

After unsealing vault the charm shows "Missing root CA":

```
    vault/0* blocked idle 2/lxd/4 172.27.105.143 8200/tcp Missing CA cert
      vault-mysql-router/0* active idle 172.27.105.143 Unit is ready
```

Then few moments after it goes back to normal "Unit is ready (active: true, mlock: disabled)".