Custom api-listening-port breaks certificates relation

[Natalia Litvinova]

Hi team,

After the deployment of cinder with config api-listening-port=443 and with using Vault as a self-signed CA I see the following:

SSL exception connecting to https://cinder.<mydomain>:443/v3/e86e28971b34446fb4ca2c9b043b8050/os-quota-sets/e86e28971b34446fb4ca2c9b043b8050: HTTPSConnectionPool(host='cinder.<mydomain>', port=443): Max retries exceeded with url: /v3/e86e28971b34446fb4ca2c9b043b8050/os-quota-sets/e86e28971b34446fb4ca2c9b043b8050 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

Checking the certs assigned to this endpoint I see:

$ openssl s_client -connect cinder.<mydomain>:443 2>/dev/null
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 337 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

I ran the action for Vault to reissue the certificates, it didn't help. Changing the port back to the default one - 8776 fixes this issue, but unfortunately this deployment requires port 443 for cinder.

Environment:
Ubuntu Jammy
Openstack Yoga
Juju 3.4.2
MAAS 3.4.1
Cinder charm yoga/stable rev 677
Vault charm 1.8/stable rev 209

I will attach the juju status.
Step to reproduce:
Deploy Cinder with custom port 443 and use Vault for certificates

[Alex Kavanagh (ajkavanagh)]

I had a quick poke around in the cinder context code, and in hooks/cinder_context.py around line 130 we have:

class ApacheSSLContext(SSLContext):
    interfaces = ['https']
    external_ports = [8776]
    service_namespace = 'cinder'

    def __call__(self):
        # late import to work around circular dependency
        from cinder_utils import service_enabled
        if not service_enabled('api'):
            return {}
        return super(ApacheSSLContext, self).__call__()

The ApacheSSLContext is the class that drives creation of the apache http vhost file. Apache handles the SSL termination, even in an haproxy-ed cluster.

I'd hazard a guess, but it would be good to confirm, that the apache conf file, when api-listening-port is set to 443, still lists 8776 as the listening port. This would be useful to confirm.

If not, then we'd need to do a deeper dive and try to recreate the issue.

[Natalia Litvinova (natalytvinova)]

So indeed this is /etc/apache2/conf-available/cinder-wsgi.conf:

Listen 8776
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D(us)" cinder_combined

<VirtualHost *:8776>
    WSGIDaemonProcess cinder-wsgi processes=5 threads=1 user=cinder group=cinder display-name=%{GROUP}
    WSGIProcessGroup cinder-wsgi
    WSGIScriptAlias / /usr/bin/cinder-wsgi
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>

    ErrorLog /var/log/apache2/cinder_error.log
    CustomLog /var/log/apache2/cinder.log cinder_combined

    <Directory /usr/bin>
        <IfVersion >= 2.4>
            Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
</VirtualHost>
~

[Alex Kavanagh (ajkavanagh)]

Ah, that's interesting. There is no SSL configuration in that file.

I may be on the wrong path. Thinking more deeply:

1. the :443 port needs to terminate on haproxy.
2. This is then proxied to apache on 8776
3. The cert information needs to be on the <VirtualHost..> definition.

What do the proxy configs look like for the relevant units. I'm hoping they listen on 443 but proxy to 8776. Then the issue is why aren't the certs being written.

[Natalia Litvinova (natalytvinova)]

Can you suggest the location of the proxy config you're talking about?

[Alex Kavanagh (ajkavanagh)]

Hi Natalia

> Can you suggest the location of the proxy config you're talking about?

Sure; I mean the ones on the cinder units, which will be at: /etc/haproxy/haproxy.cfg

Thanks!

[Natalia Litvinova (natalytvinova)]

Hi Alex,

Sure, here is the file, I don't see 8776 mentioned there though

[Alex Kavanagh (ajkavanagh)]

So the listen port is going to be 433 in Apache on the cinder units, in a file called /etc/apache2/sites-enabled/wsgi-openstack-api.conf

The haproxy.cfg is pointing to that port (443) on the backend units. Please could you grab that file and add it to the bug.

[Natalia Litvinova (natalytvinova)]

Hi Alex,
here you go

[Myles Penner (mylesjp)]

@freyes and I had a look at this bug and believe the issue stems from ApacheSSLContext defined in cinder_contexts.py[0] never fetching the provided port value in the api-listening-port config option:

The template that renders cinder-wsgi.conf uses 'ext_ports' as the external port[1]
ApacheSSLContext in charm-helpers provides the template that 'ext_ports' value[2]
ApacheSSLContext in cinder_contexts.py defines hard codes external_port as 8776[3]

It doesn't appear that the user-provided port value in 'api-listening-port' gets fetched at all.

A possible solution could be as follows:

# charm-cinder/.../hooks/cinder_context.py

class ApacheSSLContext(SSLContext):
    interfaces = ['https']
    external_ports = [8776]
    service_namespace = 'cinder'

    def __call__(self):
        # late import to work around circular dependency
        from cinder_utils import service_enabled
        if not service_enabled('api'):
            return {}
        # fetch the user-provided port if provided and use it as 'external_ports'
        if config('api-listening-port') > 0 and config('api-listening-port') < 65536:
            self.external_ports = [config('api-listening-port')]
        return super(ApacheSSLContext, self).__call__()

[0] https://opendev.org/openstack/charm-cinder/src/branch/master/hooks/cinder_contexts.py#L128
[1] https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/templates/openstack_https_frontend#L6
[2] https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/context.py#L1258
[3] https://opendev.org/openstack/charm-cinder/src/branch/master/hooks/cinder_contexts.py#L130