vip ssl cert and keys are not always configured

[Liam Young]

If aodh is deployed in a cluster and the cluster forms after other relations have completed then ssl certs and keys for the vip are not configured.

To reproduce deploy aodh in a cluster but with a missing units (1 less than cluster_count). Wait for relations to form then add the final unit.

After the cluster has formed /etc/apache2/ssl/aodh/ will have the cert and keys for the local ip but not the vip

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/536776

[Liam Young (gnuoy)]

fwiw cs:~gnuoy/aodh-8 contains the proposed fix

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536776
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=6cfcb8b79877499f177e2bbf076f7f50cdc228af
Submitter: Zuul
Branch: master

commit 6cfcb8b79877499f177e2bbf076f7f50cdc228af
Author: Liam Young <email address hidden>
Date: Tue Jan 23 10:47:44 2018 +0000

    Render SSL config when resolve_address changes

    When supplying certs and keys directly to the charm then
    get_certs_and_keys is hardcoded to return a cn of None. However,
    when the certs are written to disk configure_cert checks if the cn
    is None and if it is sets the cn to
    os_ip.resolve_address(endpoint_type=os_ip.INTERNAL). This causes an
    issue because generating the certs and keys is gated on a change to
    the return of get_certs_and_keys which always sets cn to None,
    however when the charm is clustered os_ip.resolve_address will
    return the vip which should change the cn. This means that if the
    charm transitions from un-clustered to clustered ssl certs will not
    be reconfigured if they have been configured already despite the cn
    having changed.

    This patch changes the default of cn in get_certs_and_keys to be
    os_ip.resolve_address(endpoint_type=os_ip.INTERNAL). Although this
    is a change in behaviour I believe that no charms are calling
    get_certs_and_keys directly and the risk is very low.

    Change-Id: I598bc822afa535fc865a333033069cfe05d7259d
    Closes-Bug: #1744886

[Dmitrii Shcherbakov (dmitriis)]

Should charm-gnocchi be added as well?

[Xav Paice (xavpaice)]

Tried upgrading an aodh from 18.02 to cs:~gnuoy/aodh-8 but that didn't actually add/change any certs.

Rebuilt the aodh, gnocchi and designate charms and uploaded to cs:~xavpaice/charm, deployed a fresh bundle with the new charms, and there was no change to the result.

[Xav Paice (xavpaice)]

FWIW, 'openssl s_connect -connect ${default_binding_IP}:8032 -servername ssl-host.fqdn' works just fine, providing me certificates etc. If I change that to port 8042, i.e. connecting via haproxy, the default binding works OK but I have two endpoints (admin and internal) on different namespaces to the default. In this case, each haproxy listener has only one backend, whereas the default namespace as 3, and openssl s_connect gives me no certificates when I connect to those IPs.

Reproducer:

- build an openstack cluster with 3 units each of aodh, gnocchi and designate
- use the haproxy subordinate to cluster those apps
- ensure endpoints are all ssl with fqdns
- ensure each endpoint is on a separate space
- try to connect using the openstack client for that project (e.g. aodh list... etc)

[Xav Paice (xavpaice)]

note: apache is configured with vhosts only on the default namespace binding, but haproxy backends are pointing at IP addresses in the correct space - i.e. there's no apache vhost listening on that IP address.

If I edit the apache config to replace the IP:8032 to *:8032, openssl returns a cert, and I can use the API successfully - however the charm re-writes the change pretty fast and returns us back to broken.

[Xav Paice (xavpaice)]

cs:~xavpaice/aodh-1, cs:~xavpaice/gnocchi-1 and cs:~xavpaice/designate-1 contain a workaround whereby the template for the apache ssl vhost specifies * rather than an address, until such time as we have a better fix.

[Nobuto Murata (nobuto)]

@Xav, your issue may be this one instead: https://bugs.launchpad.net/charms.openstack/+bug/1769973

[James Page (james-page)]

Marking the charm bug tasks as Fix Releases; as Nobuto points out it sounds like the other issue described here has also been fixed as of the charm release at the end of May.