Need to support multiple certificate in "trusted_ssl_ca" option

[Chi Wai CHAN]

Currently, "trusted_ssl_ca" is only effective when there's only one certificate in it. If multiple certificates are set in "trusted_ssl_ca", the certificates', for example by doing

```
cat root_ca_1.crt root_ca_2.crt > multiple_root_cas.crt
```

, the symbolic links will not be properly created in "/etc/ssl/certs" because `update-ca-certificates` [0] does not support more than one certificate (see man page of `update-ca-certificates` and `openssl rehash`; the latter is used internally in `update-ca-certificates`).

However, the symbolic links are important to `check_ssl_cert` in verifying the certificate chain. If the symbolic links are missing, it will cause `check_ssl_cert` to report errors like "unable to get local issuer certificate".

We need supports for setting multiple certificates in "trusted_ssl_ca". For example, the charm code should be able to split the "trusted_ssl_ca" content containing multiple certificates into multiple files containing single certificate. This will allow symbolic links to be created properly.

[0] https://git.launchpad.net/charm-openstack-service-checks/tree/src/reactive/openstack_service_checks.py#n207

[Nobuto Murata (nobuto)]

What's the actual use case of this? Like each OpenStack endpoints are signed by different CA trusted chains?

Usually importing one CA is fine and any intermediate CAs should be sent by the endpoints themselves.

[Chi Wai CHAN (raychan96)]

For typical OpenStack deployment with vault, the support for multiple certificates is not necessary, only the root_ca from vault is enough for `check_ssl_cert` to work properly. However, I can imagine in some corner cases that someone needs to use 3rd party certificate (e.g. provided by a customer) to configure ssl per-application basis (e.g. cinder). The certificate is not necessary the signed by the root CA from vault. In that case, we might need to trust two root CAs, one from the vault, one from the root CA of the 3rd party.

However, I do want to say this may not happen very often, and it may not be urgent to implement it right away. Therefore, I am marking it as a wishlist to track it. Nevertheless, having the ability to trust more than one CA should be a welcome enhancement, and should be backward compatible.

[John Lettman (jplettman)]

A customer hits these problems due to their intermediates and mixed certificate environment.

$ sudo -u nagios /usr/local/lib/nagios/plugins/check_ssl_cert -H barbican.api.customer-xyz.com -p 9312 -u /v1 -c 14 -w 30 --ignore-sct --ignore-ocsp
SSL_CERT CRITICAL barbican.api.customer-xyz.com Cannot verify certificate: unable to get local issuer certificate|days_chain_elem1=79;30;14;; days_chain_elem2=2749;30;14;;

[John Lettman (jplettman)]

I will add that we do have a non-destructive workaround here for the case I described:

1. Inspect the contents of /etc/ssl/certs/openstack-service-checks.pem
2. Copy each certificate into /usr/local/share/ca-certificates/[customer]-[cert-number].crt
3. Ensure each certificate file is o+r mode
4. Execute update-ca-certificate --fresh

One solution for the charm could be to conduct this operation automatically when encountering a multi-certificate chain in the `trusted_ssl_ca` configuration option.

[Eric Chen (eric-chen)]

Migrate to https://github.com/canonical/charm-openstack-service-checks/issues