Keystone does not update its certificate request when adding HA to an existing single-unit deployment

[Peter Matulis]

When trying to add HA to a single unit (single IP to VIP) with the hacluster charm Keystone is not updating its certificate request.

It looks like the Keystone server certificate is not being updated to account for the new address being used (the VIP). When Keystone is solicited by a cloud client a connection error results.

Please see attachment which shows that Glance fails to connect to Keystone.

[Peter Matulis (petermatulis)]

I just hit this again. Looking deeper, only the certificate of the keystone leader is not being updated:

The leader is keystone/0 (10.246.114.58) and the VIP is 10.246.116.11.

$ echo | openssl s_client -showcerts -servername 10.246.116.11 -connect 10.246.116.11:5000 2>/dev/null | openssl x509 -inform pem -noout -text | grep DNS
                DNS:juju-b63bf5-0-lxd-2.maas, IP Address:10.246.114.58

$ echo | openssl s_client -showcerts -servername 10.246.116.11 -connect 10.246.116.11:5000 2>/dev/null | openssl x509 -inform pem -noout -text | grep DNS
                DNS:juju-b63bf5-2-lxd-6.maas, IP Address:10.246.114.37, IP Address:10.246.116.11

$ echo | openssl s_client -showcerts -servername 10.246.116.11 -connect 10.246.116.11:5000 2>/dev/null | openssl x509 -inform pem -noout -text | grep DNS
                DNS:juju-b63bf5-1-lxd-6.maas, IP Address:10.246.114.38, IP Address:10.246.116.11

The keystone application looks like this:

 Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/2 10.246.114.58 5000/tcp Unit is ready
  keystone-hacluster/0* active idle 10.246.114.58 Unit is ready and clustered
  keystone-mysql-router/0* active idle 10.246.114.58 Unit is ready
keystone/1 active idle 1/lxd/6 10.246.114.38 5000/tcp Unit is ready
  keystone-hacluster/1 active idle 10.246.114.38 Unit is ready and clustered
  keystone-mysql-router/2 active idle 10.246.114.38 Unit is ready
keystone/2 active idle 2/lxd/6 10.246.114.37 5000/tcp Unit is ready
  keystone-hacluster/2 active idle 10.246.114.37 Unit is ready and clustered
  keystone-mysql-router/1 active idle 10.246.114.37 Unit is ready

[Peter Matulis (petermatulis)]

Workaround
----------

Redo the relation between the keystone and vault applications:

$ juju remove-relation vault:certificates keystone:certificates
$ juju add-relation vault:certificates keystone:certificates

[Felipe Reyes (freyes)]

On Fri, 2022-02-04 at 21:34 +0000, Peter Matulis wrote:
> Workaround
> ----------
>
> Redo the relation between the keystone and vault applications:
>
> $ juju remove-relation vault:certificates keystone:certificates
> $ juju add-relation vault:certificates keystone:certificates
>
Another workaround could be:

juju run-action vault/leader --wait reissue-certificates

[Peter Matulis (petermatulis)]

> Another workaround could be:
>
> juju run-action vault/leader --wait reissue-certificates

I did try that but it did not work.