update of ssl_cert does not restart apache so old cert is still used

[Xav Paice]

Due to certificate expiry, we replaced the ssl_cert and ssl_key for openstack-dashboard (cs:openstack-dashboard-304, cloud:bionic-stein).

When testing, I saw that the certificate provided was still the old one:

curl -vvI https://dash-url.somesite
...snip
* start date: May 15 15:46:30 2020 GMT
* expire date: May 15 16:16:29 2021 GMT
...snip

After restarting apache2 on the units, the correct cert was used.

This should be triggered by the config-changed hook.

[Alex Kavanagh (ajkavanagh)]

Triaged: High due to SSL cert 'breaking'.

The issue is that, I think, although the cert is changed, the actual config files don't change as the names of the certs are consistent. A couple of interesting ways to change this are (assuming this is the correct analysis):

1. Always just restart apache on the config-changed hook; this is the brute force and easiest method.
2. Add a context for the SSL key, cert, ca files such that they get tested for changes and thus the restart handler that is built into the config-change hook would automatically restart apache if they changed. This has the added complication of needing to change how the current contexts work due to identity-service relations being involved.

I'm guessing that 1. is the simplest.

[Chris MacNaughton (chris.macnaughton)]

in addition to restarting Apache on config-changed, the easy change would also need to restart Apache in the certificate-relation-changed hook

[Cory Johns (johnsca)]

Another, perhaps more simple and reliable approach would be to wrap the `write_file()` calls in configure_cert [1] with `with restart_on_change({filename: ...})` [2]. This should only require a few lines of change (plus tests) and handle all potential cases where the certs are updated. The CA file should also be covered, as well.

[1]: https://github.com/juju/charm-helpers/blob/2ec6847/charmhelpers/contrib/openstack/context.py#L1037-L1042

[2]: https://github.com/juju/charm-helpers/blob/2ec6847/charmhelpers/core/host.py#L697

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/793387

[Hernan Garcia (hernandanielg)]

I have tried to reproduce this issue but couldn't

This is what I did

➜ ~ juju deploy cs:openstack-dashboard-304
➜ ~ juju config openstack-dashboard openstack-origin
cloud:bionic-stein

Loaded a certificate

➜ juju config openstack-dashboard ssl_key="$(base64 my.key)" ssl_cert="$(base64 my.cert)"

Checked that the certificate was correctly loaded

➜ ~ echo | openssl s_client -showcerts -connect 10.17.245.230:443 2>/dev/null | openssl x509 -inform pem -noout -subject -startdate -enddate
subject=C = US, ST = Denial, L = Springfield, O = Dis, CN = mycert
notBefore=Jun 3 16:13:32 2021 GMT
notAfter=Jun 3 16:13:32 2022 GMT

Changed the certificate

➜ juju config openstack-dashboard ssl_key="$(base64 my2.key)" ssl_cert="$(base64 my2.cert)"

Checked certificate again and saw that it was correctly updated

➜ ~ echo | openssl s_client -showcerts -connect 10.17.245.230:443 2>/dev/null | openssl x509 -inform pem -noout -subject -startdate -enddate
subject=C = US, ST = Denial, L = Springfield, O = Dis, CN = mycert2
notBefore=Jun 3 16:13:11 2021 GMT
notAfter=Jun 3 16:13:11 2022 GMT

Am I missing something?

[Alex Kavanagh (ajkavanagh)]

Caching is a tricky thing. It may be that my original analysis is wrong. Try hitting the apache servers a few more times before changing the cert files?

[Hernan Garcia (hernandanielg)]

Sadly did lot of requests and still cert got updated without any issues

Loaded first cert
➜ juju config openstack-dashboard ssl_key="$(base64 my.key)" ssl_cert="$(base64 my.cert)"

Checked the certificate
➜ echo | openssl s_client -showcerts -connect 10.17.245.181:443 2>/dev/null | openssl x509 -inform pem -noout -subject -startdate -enddate
subject=C = US, ST = Denial, L = Springfield, O = Dis, CN = mycert
notBefore=Jun 3 16:13:32 2021 GMT
notAfter=Jun 3 16:13:32 2022 GMT

Made hundred requests
➜ for i in {0..100} ; do curl -sk -vvI https://10.17.245.181 >&/dev/null ; done

Changed the certificate
➜ juju config openstack-dashboard ssl_key="$(base64 my2.key)" ssl_cert="$(base64 my2.cert)"

Then checked the cert again and it was already using the new one
➜ echo | openssl s_client -showcerts -connect 10.17.245.181:443 2>/dev/null | openssl x509 -inform pem -noout -subject -startdate -enddate
subject=C = US, ST = Denial, L = Springfield, O = Dis, CN = mycert2
notBefore=Jun 3 16:13:11 2021 GMT
notAfter=Jun 3 16:13:11 2022 GMT

Can't reproduce this issue and might need to abandon the review as discussed

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Hernan Garcia <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/793387
Reason: Can't reproduce bug

[Alex Kavanagh (ajkavanagh)]

Triaging this back to incomplete as it doesn't seem to be reproducible. This might mean it was a transient error. If the OP could add any additional information (e.g. logs) then we might be able to take another look at it. Thanks!

[Garrett Neugent (thogarre)]

Hello,

I'm marking this as new, as we've encountered this on another customer cloud today.

The version info is:

charm version: cs:openstack-dashboard-302
release: openstack-origin: cloud:bionic-stein

We saw that after updating the ssl config (ssl_cert, ssl_key, ssl_ca), the files were placed on the unit, but apache2 itself was not restarted during the config-changed hook:

ubuntu@juju-a00f9c-22-lxd-4:/etc/apache2/ssl/horizon$ ll
total 40
dr-xr-xr-x 2 root root 4096 Apr 22 2020 ./
dr-xr-xr-x 3 root root 4096 Apr 22 2020 ../
-rw-r----- 1 root root 6054 Nov 29 18:25 cert_10.225.0.22
-rw-r----- 1 root root 2687 Apr 22 2020 cert_10.225.0.81
-rw-r----- 1 root root 6054 Nov 29 18:25 cert_horizon.redacted
-rw-r----- 1 root root 1869 Nov 29 18:25 key_10.225.0.22
-rw-r----- 1 root root 1869 Apr 22 2020 key_10.225.0.81
-rw-r----- 1 root root 1869 Nov 29 18:25 key_horizon.redacted
ubuntu@juju-a00f9c-22-lxd-4:/etc/apache2/ssl/horizon$ date
Mon Nov 29 19:15:31 UTC 2021

ubuntu@juju-a00f9c-22-lxd-4:/etc/apache2/ssl/horizon$ systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Wed 2021-08-18 15:47:30 UTC; 3 months 11 days ago
 Main PID: 445 (apache2)
    Tasks: 0 (limit: 314572)
   CGroup: /system.slice/apache2.service
           ├─ 445 /usr/sbin/apache2 -k start ...

[Xav Paice (xavpaice)]

Reproduced today on two environments, though with charm version 304.

Bionic cloud, Stein.

I've attached a unit log (log.log) from the openstack-dashboard log, showing the config-changed hook and subsequent cluster-relation-changed.