Race in openstack ssl certificates generation can result in missing os-*-hostname SANs

[Drew Freiberger]

The vault code appears to be specifically set up to avoid generating an application's certificate from multiple application units, this means the first unit relate to vault on the certificates interface will determine which SANS will be added to the relation. This should be adapted to be additive.

For an example, if I have a 3 unit HA cluster for keystone, and it uses three separate network bindings for admin, public, and internal spaces, and a VIP is defined for each in such a manner:

public-space: 10.0.0.0/24
admin-space: 10.0.1.0/24
internal-space: 10.0.2.0/24

10.0.0.250 keystone.mydomain.tld
10.0.1.250 keystone-admin.mydomain.tld
10.0.2.250 keystone-internal.mydomain.tld

bindings:
  public: public-space
  admin: admin-space
  internal: internal-space
options:
  vip: 10.0.0.250,10.0.1.250,10.0.2.250
  os-public-hostname: keystone.mydomain.tld
  os-admin-hostname: keystone-admin.mydomain.tld
  os-internal-hostname: keystone-internal.mydomain.tld

The get_certificate_request is going to happen regardless of whether this is the leader unit, whether it has any of the VIPs defined on the local host, etc by the first related keystone (or other openstack application) unit. For example:
https://github.com/openstack/charm-keystone/blob/master/hooks/keystone_hooks.py#L854

In charmhelpers.contrib.openstack.cert_utils, get_certificate_request only requests certificate to be generated against IP addresses and hostnames that have bindings on the current unit.

If the certificates relation is joined prior to HA settling, or if the certificates relation is joined by a unit not hosting all three of the VIPs, at least one of the os-*-hostname entries will be left out of the request.sans data sent to vault.

So, if keystone/0 is juju-model-0-lxd-0 and keystone/1 is juju-model-1-lxd-0 and keystone/2 is juju-model-2-lxd-0, and 0/lxd/0 is the first to relate to vault on the certificates interface, and keystone/2 owns all of the VIPs, the keystone certificate will be devoid of the keystone*.mydomain.tld SANs.

I would suggest either the openstack layer needs to be multi-unit IP and FQDN aware when making the request to the certificates interface, or the application certificate needs to be reissued with an extended list of sans each time a new unit of the application relates and requests additional sans to be added.

This is a theoretical bug discovered while performing research to answer the following juju discussion topic. https://discourse.juju.is/t/juju-host-names-and-vault-ssl-issuance-best-practice-questions/3018

If there are other items in place in the code which I've missed, please feel free to mark this as Invalid.

[Drew Freiberger (afreiberger)]

Added charm-helpers and openstack charms because it may be possible to solve at the openstack charms layers based upon changing the os-$net_type-hostname tests within charmhelpers:

https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/cert_utils.py#L124-L144

[Drew Freiberger (afreiberger)]

I've found that the vip and it's dns name (which would end up returning os-*-hostname if ptr records for the VIPs are correctly configured) will be registered from each node per https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/cert_utils.py#L130

I've validated in a live environment that this works as intended. Marking invalid.