Re-creating root CA on vault causes Horizon services to not function correctly

[Diko Parvanov]

After a vault generate-root-ca action is performed and a new root CA is issued, all related openstack services certificates are renewed, however testing from some units to, for example, curl keystone API fails with 'SSL certificate problem: unable to get local issuer certificate'.

Either this information is not properly propagated by the relations or not properly handled by the related charms.

Manually fixing this by adding the new ca certificate, run-action get-root-ca on vault and then upload to /usr/loca/share/ca-certificates and running update-ca-certificates as root.

[Chris Sanders (chris.sanders)]

Subscribing field high due to the impact on operations as well as handovers and deployments which sometimes need to re-generate certificates.

[Liam Young (gnuoy)]

I'm not sure what I'm missing but I can't reproduce this. When I run generate-root-ca and reissue certs I see /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt change on the dashboard units and running curl from the openstack-dashboard against the keystone endpoint continues to work.

Is this problem intermittent or is it consistently reproducible ? Is it only the ca certificate on the dashboard units that need updating or are other applications affected too?

[Diko Parvanov (dparv)]

How about when you have a external ssl certifiated installed in the openstack-dashboard units?

I know that putting openstack-dashboard ssl_cert/ssl_key/ssl_ca takes precedence over what vault issues (by the internal root CA) which probably only updates the ca-certificates with the one configured on openstack-dashboard ssl_ca option and just drops the information via the vault certificates relation

[Liam Young (gnuoy)]

Am I right in saying that scenario is:

1) Deploy with vaults relations
2) Issue certificates via vault
3) Add certs to openstack-dashboard application via ssl_* config options
4) vault reissues certs
5) openstack-dashboard application now has CA cert from vault installed. The CA from ssl_ca is no longer installed.

If this is correct then what behaviour are you after ? Both CAs installed simultaneously ?

[Diko Parvanov (dparv)]

Not quite. The scenario is:

1) Deploy with vault relations
2) Issue certificates via vault
[some time later customer decides to use different ssl certs on Horizon]
3) Add certificates, provided by customer via ssl_* config option to openstack-dashboard
4) Re-issue vault root certificate
5) Openstack-dashboard now doesn't trust new root CA from vault, only has the CA from ssl_ca config option and can't access keystone endpoints, as keystone doesn't have custom ssl_* installed, only vault issued certificates.

When vault re-issues certificates and openstack-dashboard has a custom ssl_ca assigned the charm should combine the new vault certificates via the relation with the configured ssl_ca, otherwise the communication to keystone will stop.

[Diko Parvanov (dparv)]

I guess it's something openstack-dashboard charm should handle, vault has nothing wrong, as it correctly issues the certificates. Thus marking vault-charm as invalid.

[Liam Young (gnuoy)]

I haven't been able to reproduce this but from the description of the issue I think this change will help:

https://github.com/juju/charm-helpers/pull/522

[Liam Young (gnuoy)]

I am hopeful that https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/764551 will fix the issue. Please can you see if the bug still exists with cs:~openstack-charmers-next/openstack-dashboard ?