Masakari endpoint not using HTTPS even with vault:certificates relation enabled

[Hybrid512]

Hi,

It seems SSL is not enabled on Masakari endpoint even though vault:certificates relation is added.

Here is my Masakari configuration :

=======================================================
series: focal
applications:

  masakari:
    charm: cs:masakari-11
    channel: stable
    num_units: 3
    to:
    - lxd:3
    - lxd:4
    - lxd:5
    options:
      debug: false
      evacuation-delay: 10
      openstack-origin: distro
      use-internal-endpoints: false
      use-syslog: false
      verbose: false
      vip: 192.168.111.225
      worker-multiplier: 0.25

  masakari-hacluster:
    charm: cs:hacluster-76
    channel: stable
    options:
      cluster_count: 3
      maas_credentials: <MAAS ADMIN CREDENTIALS>
      maas_url: http://<MAAS-IP>:5240/MAAS

  masakari-monitors:
    charm: cs:masakari-monitors-9
    channel: stable

  masakari-mysql-router:
    charm: cs:mysql-router-10
    channel: stable

  masakari-pacemaker-remote:
    charm: cs:pacemaker-remote-9
    channel: stable
    options:
      enable-resources: false
      enable-stonith: true

relations:
- - masakari:ha
  - masakari-hacluster:ha
- - nova-compute:juju-info
  - masakari-pacemaker-remote:juju-info
- - masakari-hacluster:pacemaker-remote
  - masakari-pacemaker-remote:pacemaker-remote
- - nova-compute:juju-info
  - masakari-monitors:container
- - keystone:identity-credentials
  - masakari-monitors:identity-credentials
- - masakari-monitors:certificates
  - vault:certificates
- - masakari:identity-service
  - keystone:identity-service
- - masakari:shared-db
  - masakari-mysql-router:shared-db
- - masakari-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - masakari:amqp
  - rabbitmq-server:amqp
- - masakari:certificates
  - vault:certificates
=======================================================

(as you can see, the relation for certificates is there)

and here is my endpoints list :

=======================================================
+----------------------------------+-----------+--------------+-----------------+---------+-----------+-----------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+-----------------------------------------------+
| 5f0a0c070eaa4b5e8c43540f9ed30e46 | RegionOne | barbican | key-manager | True | admin | https://192.168.111.230:9312 |
| 532f04ea7cb540b28c7f27c3442d67c9 | RegionOne | barbican | key-manager | True | internal | https://192.168.111.230:9311 |
| 74a17a8cd9f8424380c88ad292785452 | RegionOne | barbican | key-manager | True | public | https://192.168.111.230:9311 |
| 318f0e627e154c2fb6243b6729a1d8b6 | RegionOne | cinderv2 | volumev2 | True | admin | https://192.168.111.226:8776/v2/$(tenant_id)s |
| bd6dc8e1fa874850a9cba450c25e08a6 | RegionOne | cinderv2 | volumev2 | True | internal | https://192.168.111.226:8776/v2/$(tenant_id)s |
| cfe544ea53f14ec096b2dbb913b9ad7c | RegionOne | cinderv2 | volumev2 | True | public | https://192.168.111.226:8776/v2/$(tenant_id)s |
| 59cc15d7b63a41288eb05670919a7976 | RegionOne | cinderv3 | volumev3 | True | admin | https://192.168.111.226:8776/v3/$(tenant_id)s |
| b2304999c35e4cc0959658765feb3c02 | RegionOne | cinderv3 | volumev3 | True | internal | https://192.168.111.226:8776/v3/$(tenant_id)s |
| f8fde8d1e5304707b08e2242d4cbe256 | RegionOne | cinderv3 | volumev3 | True | public | https://192.168.111.226:8776/v3/$(tenant_id)s |
| 19f323f3a092484bb79a2db7176c8cc6 | RegionOne | designate | dns | True | admin | https://192.168.111.237:9001 |
| 31df7d60656c495a80ca2ad7c68e6062 | RegionOne | designate | dns | True | internal | https://192.168.111.237:9001 |
| 60aa8a256927420299f934f97795ece2 | RegionOne | designate | dns | True | public | https://192.168.111.237:9001 |
| b98dfc05c9d74002960afb090f87f21e | RegionOne | glance | image | True | admin | https://192.168.111.223:9292 |
| 13c50c405e674464becb31462922d905 | RegionOne | glance | image | True | internal | https://192.168.111.223:9292 |
| dc81de546b764af7993eb54f14cbd851 | RegionOne | glance | image | True | public | https://192.168.111.223:9292 |
| 0d75c6e98c064dce91811286093e3504 | RegionOne | gnocchi | metric | True | admin | https://192.168.111.235:8041 |
| 4a68bbaced3245a39653e836cad52e84 | RegionOne | gnocchi | metric | True | internal | https://192.168.111.235:8041 |
| c23a9727337d491aa9bef7afb6a93a2b | RegionOne | gnocchi | metric | True | public | https://192.168.111.235:8041 |
| 09b56aad2e8b42f19f6123a167fd4c20 | RegionOne | heat | orchestration | True | admin | https://192.168.111.224:8004/v1/$(tenant_id)s |
| a9aa6ce5cb0242d29fd1e3bd308285b2 | RegionOne | heat | orchestration | True | internal | https://192.168.111.224:8004/v1/$(tenant_id)s |
| 0deaa913655140e4979cf0dd1c1e22c0 | RegionOne | heat | orchestration | True | public | https://192.168.111.224:8004/v1/$(tenant_id)s |
| b42f92583d2e4910936fe605a5ade327 | RegionOne | heat-cfn | cloudformation | True | admin | https://192.168.111.224:8000/v1 |
| 1032db22fee64c519e18b54b20e84fef | RegionOne | heat-cfn | cloudformation | True | internal | https://192.168.111.224:8000/v1 |
| aef67d7d7cb94d488fd74c7c4c8946d5 | RegionOne | heat-cfn | cloudformation | True | public | https://192.168.111.224:8000/v1 |
| 5a2be383cae84d41ac83b86477242d98 | RegionOne | image-stream | product-streams | True | admin | http://192.168.112.27 |
| 04d02021fd634a57826475b58688ea04 | RegionOne | image-stream | product-streams | True | internal | http://192.168.112.27 |
| 8acbf1048e0241bb95ce3c5e36719257 | RegionOne | image-stream | product-streams | True | public | http://192.168.112.27 |
| 94748b1062a8453081ca6102a7d78452 | RegionOne | keystone | identity | True | admin | https://192.168.111.222:35357/v3 |
| 1c38a6e60673443a90284a6d22f1d747 | RegionOne | keystone | identity | True | internal | https://192.168.111.222:5000/v3 |
| 443d6e0d85ec44829390a9a634148716 | RegionOne | keystone | identity | True | public | https://192.168.111.222:5000/v3 |
| 2dfbe1d561484fd881b36de5c5777c6a | RegionOne | masakari | instance-ha | True | admin | http://192.168.111.225:15868/v1/%(tenant_id)s |
| 9e7f24ca01404f978bdddbd2f1963ea6 | RegionOne | masakari | instance-ha | True | internal | http://192.168.111.225:15868/v1/%(tenant_id)s |
| 6024e02726c24ec380815556c36a665b | RegionOne | masakari | instance-ha | True | public | http://192.168.111.225:15868/v1/%(tenant_id)s |
| da164ca9d7f744e493708f6617488634 | RegionOne | neutron | network | True | admin | https://192.168.111.227:9696 |
| e7fadfcfc64b44ba9d7dfb4d89e1f1be | RegionOne | neutron | network | True | internal | https://192.168.111.227:9696 |
| d4d90840c2684a728caff0199ce0a8be | RegionOne | neutron | network | True | public | https://192.168.111.227:9696 |
| a54c9bcb5305450eb2e7ca92b891b618 | RegionOne | nova | compute | True | admin | https://192.168.111.228:8774/v2.1 |
| c595da7e5e8643fdbc7dca7890d4fb10 | RegionOne | nova | compute | True | internal | https://192.168.111.228:8774/v2.1 |
| bc9aedc052ce47b19261294b7c928f0d | RegionOne | nova | compute | True | public | https://192.168.111.228:8774/v2.1 |
| 4aa4e6b2ad4547c4b2eda4c9fddb4c2d | RegionOne | octavia | load-balancer | True | admin | https://192.168.111.231:9876 |
| 5a1d4beae5774c0f947cd2a962f59a9f | RegionOne | octavia | load-balancer | True | internal | https://192.168.111.231:9876 |
| 36e74732b297477988ce0c8235c2b738 | RegionOne | octavia | load-balancer | True | public | https://192.168.111.231:9876 |
| 9d14cac7d6de4a06b7564ae80ab4514d | RegionOne | placement | placement | True | admin | https://192.168.111.229:8778 |
| 7616831fc9ad4bd4a59e507f3734d22e | RegionOne | placement | placement | True | internal | https://192.168.111.229:8778 |
| 446bcca965cc4d7f9cf88315573cf024 | RegionOne | placement | placement | True | public | https://192.168.111.229:8778 |
| 1fc5452dbf7f404ea89083c08e4decc3 | RegionOne | s3 | s3 | True | admin | https://192.168.111.236:443/ |
| 168c2478b8c8492aa74365ef7cf45163 | RegionOne | s3 | s3 | True | internal | https://192.168.111.236:443/ |
| ddfc6d03103c47408151845651bb4355 | RegionOne | s3 | s3 | True | public | https://192.168.111.236:443/ |
| 06b6454cd25d46c29d57ace14a4b928f | RegionOne | swift | object-store | True | admin | https://192.168.111.236:443/swift |
| fad9c70c19714dad859bcfde87d501e1 | RegionOne | swift | object-store | True | internal | https://192.168.111.236:443/swift/v1 |
| b7a37fd7640344c58031f3eeeda4fd06 | RegionOne | swift | object-store | True | public | https://192.168.111.236:443/swift/v1 |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+-----------------------------------------------+
=======================================================

As you can see, Masakari endpoint is configured for HTTP and not HTTPS.

[Vladimir Grevtsev (vlgrevtsev)]

This bug is still actual, can someone take a look please?

+ field-high as the expected functionality doesn't work (TLS endpoint isn't updated in Keystone database).

Just have checked on freshly Masakari deployment: the endpoint in Keystone is http:// indeed, and:

# the certs are present:
root@juju-2d3b82-0-lxd-10:~# ls /etc/apache2/ssl/masakari/
cert_10.35.174.113 cert_10.35.84.244

# but https-related apache conf is empty
~# cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
root@juju-2d3b82-0-lxd-10:~#

[Vladimir Grevtsev (vlgrevtsev)]

masakari unit flags:

juju run --unit masakari/0 'charms.reactive get_flags' | pastebinit
https://paste.ubuntu.com/p/zbH2HSCT7p/

[Vladimir Grevtsev (vlgrevtsev)]

overlay used for deploying masakari: https://paste.ubuntu.com/p/9NmmVS6M4Z/

# VIPs are also in-place
$ juju config masakari vip
10.35.84.213 10.35.174.113
$ juju config neutron-api vip
10.35.174.104 10.35.84.204
$ juju config keystone vip
10.35.174.100 10.35.84.200

$ juju spaces
Name Space ID Subnets
alpha 0
infra-vms 4 10.34.94.0/23
                          10.35.85.192/26
internal-space 3 10.35.174.0/25
oam-space 1 10.35.81.0/24
overlay-space 5 10.35.83.128/25
public-space 2 10.35.84.128/25

Comparing the network bindings between Masakari and any other app:

[endpoint is http in keystone]
$ juju show-application masakari
masakari:
  charm: masakari
  series: focal
  channel: stable
  principal: true
  exposed: false
  remote: false
  endpoint-bindings:
    "": oam-space
    admin: internal-space
    amqp: internal-space
    certificates: internal-space
    cluster: oam-space
    ha: oam-space
    identity-service: internal-space
    internal: internal-space
    public: public-space
    shared-db: internal-space

[endpoint is https in keystone]
$ juju show-application neutron-api
neutron-api:
  charm: neutron-api
  series: focal
  channel: stable
  principal: true
  exposed: false
  remote: false
  endpoint-bindings:
    "": oam-space
    admin: internal-space
    amqp: oam-space
    certificates: internal-space
    cluster: oam-space
    etcd-proxy: oam-space
    external-dns: oam-space
    ha: oam-space
    identity-service: oam-space
    infoblox-neutron: oam-space
    internal: internal-space
    midonet: oam-space
    neutron-api: oam-space
    neutron-load-balancer: oam-space
    neutron-plugin-api: oam-space
    neutron-plugin-api-subordinate: oam-space
    nrpe-external-master: oam-space
    public: public-space
    shared-db: internal-space
    vsd-rest-api: oam-space

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-masakari/+/805079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-masakari/+/805079
Committed: https://opendev.org/openstack/charm-masakari/commit/3760b7211f6428dfd7321d4adf8d7ddaed7535f2
Submitter: "Zuul (22348)"
Branch: master

commit 3760b7211f6428dfd7321d4adf8d7ddaed7535f2
Author: Chris MacNaughton <email address hidden>
Date: Wed Aug 18 11:14:26 2021 -0500

    Specify certificates relation to update endpoints.

    When the certificates relation is unspecified with the
    configure_tls method invocation, the masakari endpoints
    seem to ignore the update to TLS. This change specifies
    the relation which causes the endpoints to update correctly.

    Closes-Bug: #1935986
    Change-Id: Ib1a6ca1ddf64950ff13cf3a8904d9848710d96a5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-masakari/+/805142

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-masakari/+/805142
Committed: https://opendev.org/openstack/charm-masakari/commit/532ddc91f80084e731bf231fa591aee50fe8af38
Submitter: "Zuul (22348)"
Branch: stable/21.04

commit 532ddc91f80084e731bf231fa591aee50fe8af38
Author: Chris MacNaughton <email address hidden>
Date: Wed Aug 18 11:14:26 2021 -0500

    Specify certificates relation to update endpoints.

    When the certificates relation is unspecified with the
    configure_tls method invocation, the masakari endpoints
    seem to ignore the update to TLS. This change specifies
    the relation which causes the endpoints to update correctly.

    Closes-Bug: #1935986
    Change-Id: Ib1a6ca1ddf64950ff13cf3a8904d9848710d96a5
    (cherry picked from commit 3760b7211f6428dfd7321d4adf8d7ddaed7535f2)