Octavia renders the same SSL certificate for different OpenStack endpoints when using Vault as an intermediate CA

Bug #1865447 reported by Przemyslaw Hausman
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Octavia Charm
Triaged
Medium
Unassigned

Bug Description

The certificates for Octavia endpoints are the same for all endpoints (internal, public, admin), which is incorrect. It results in a failed connections to a certain endpoint, because the hostname in CN does not match the hostname in request.

STEPS TO REPRODUCE

1. Deploy OpenStack with Octavia and Vault as an intermediate CA. Create relation "octavia:certificates - vault:certificates"

2. Set up different hostnames for octavia endpoints, e.g.:

  octavia:
    options:
      os-public-hostname: octavia.example.com
      os-internal-hostname: octavia-internal.example.com
      os-admin-hostname: octavia.example.com

3. Verify, that the certificates for different octavia endpoints point to the same file.

sudo cat /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 9866
<VirtualHost 10.127.1.108:9866>
    ServerName octavia-internal.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/octavia/cert_octavia-internal.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/octavia/key_octavia-internal.example.com
    ProxyPass / http://localhost:9856/
    ProxyPassReverse / http://localhost:9856/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 10.127.12.92:9866>
    ServerName octavia.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/octavia/cert_octavia.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/octavia/key_octavia.example.com
    ProxyPass / http://localhost:9856/
    ProxyPassReverse / http://localhost:9856/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

sudo ls -l /etc/apache2/ssl/octavia/
total 24
-rw-r----- 1 root octavia 1532 Mar 2 08:01 cert_eth2.juju-439b02-25-lxd-7.example.com
lrwxrwxrwx 1 root root 74 Mar 2 06:57 cert_octavia-internal.example.com -> /etc/apache2/ssl/octavia/cert_eth2.juju-439b02-25-lxd-7.example.com
lrwxrwxrwx 1 root root 74 Mar 2 06:57 cert_octavia.example.com -> /etc/apache2/ssl/octavia/cert_eth2.juju-439b02-25-lxd-7.example.com
-rw-r----- 1 root octavia 1678 Mar 2 08:01 key_eth2.juju-439b02-25-lxd-7.example.com
lrwxrwxrwx 1 root root 73 Mar 2 06:57 key_octavia-internal.example.com -> /etc/apache2/ssl/octavia/key_eth2.juju-439b02-25-lxd-7.example.com
lrwxrwxrwx 1 root root 73 Mar 2 06:57 key_octavia.example.com -> /etc/apache2/ssl/octavia/key_eth2.juju-439b02-25-lxd-7.example.com

4. Verify, that CN in the certificate is missing octavia-internal.example.com

sudo openssl x509 -in /etc/apache2/ssl/octavia/cert_eth2.juju-439b02-25-lxd-7.example.com -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            35:ff:ef:b3:0d:ec:06:af:d9:23:9a:ae:93:aa:1d:7c:e3:7b:9e:27
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Vault Root Certificate Authority (charm-pki-local)
        Validity
            Not Before: Mar 2 06:56:35 2020 GMT
            Not After : Mar 2 05:57:04 2021 GMT
        Subject: CN = octavia.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:cb:1d:4e:ba:52:03:ac:34:a3:05:39:ce:bd:b6:
                    de:c0:9d:1a:fd:8b:0c:fd:3e:24:69:83:6f:ce:7d:
                    bf:21:e8:52:a0:31:dd:df:76:63:c5:a4:95:db:23:
                    7a:a7:bc:d0:7c:49:14:2c:2a:a4:45:53:5c:16:6c:
                    dc:26:02:ef:a8:0a:d8:48:da:70:f3:3d:36:00:39:
                    c4:9c:7f:33:37:ca:b6:5b:83:3d:4d:21:76:68:8f:
                    c8:7c:12:85:d5:de:60:3e:02:94:e0:7a:59:11:85:
                    39:88:06:05:81:4b:c5:5a:eb:b3:09:ed:b4:2a:d4:
                    94:22:5e:98:6c:90:73:aa:ca:6d:69:ca:8e:98:16:
                    08:12:0c:86:fa:8d:8f:dd:2c:07:3c:84:7f:6d:35:
                    ea:33:fd:6d:02:a1:8e:36:72:22:1c:3e:84:be:3a:
                    45:f1:2a:18:2f:73:a0:95:43:d2:86:dc:57:db:a0:
                    56:e4:0c:16:4b:ca:4e:78:46:57:be:6d:6a:3a:b0:
                    d5:96:cd:46:a1:c2:93:90:60:50:c9:e6:94:2f:3c:
                    dd:c4:16:cf:f5:8a:62:ce:12:8d:a2:f4:97:bb:8e:
                    ae:c2:e6:a4:d7:ae:04:b3:43:78:74:c5:33:11:65:
                    43:9d:12:9f:3d:57:6d:0f:ad:b5:b0:96:d8:54:e4:
                    77:4d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                21:26:04:F3:82:DE:A3:08:D8:1C:A3:B3:E0:C6:9E:BB:CC:A5:A7:B4
            X509v3 Authority Key Identifier:
                keyid:78:C7:77:34:9D:2B:70:29:14:11:A5:CB:86:3E:B0:BA:5B:95:0F:86

            Authority Information Access:
                CA Issuers - URI:http://10.127.1.43:8200/v1/charm-pki-local/ca

            X509v3 Subject Alternative Name:
                DNS:octavia.example.com, DNS:octavia.example.com, IP Address:10.127.12.42, IP Address:10.127.12.92
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://10.127.1.43:8200/v1/charm-pki-local/crl

    Signature Algorithm: sha256WithRSAEncryption
         44:f4:a8:da:54:4b:53:9d:7b:93:bf:8f:52:d9:47:31:05:e9:
         8b:9d:ae:56:bf:74:c1:5a:f2:18:66:20:d8:5b:d5:4a:56:76:
         6e:c2:ce:85:cd:4a:94:b7:57:d5:6f:61:0a:73:64:25:43:51:
         86:d0:e1:9e:89:8a:49:71:61:8a:61:f0:fe:c9:6d:fe:8b:42:
         f4:7f:5c:90:72:14:75:d4:35:cc:93:91:62:3c:ec:df:3a:cf:
         f7:f4:c5:4d:f1:c9:3d:9e:d8:16:7d:a5:82:1b:13:68:84:03:
         28:2e:f0:4a:59:0e:c5:00:58:e3:31:2f:69:48:00:57:8b:77:
         7c:1e:86:b1:2a:01:bf:df:7f:2f:0e:03:fc:34:2c:40:3a:8e:
         e2:d8:dd:13:1e:7b:ab:d3:bd:bd:26:31:e7:b4:da:3c:83:ae:
         c5:22:1a:d2:6d:dc:62:e1:b0:81:ae:4a:26:a0:26:e8:52:ca:
         c0:90:87:02:a9:36:18:c0:e2:f1:a4:b0:9c:64:3c:ae:78:ee:
         19:e3:26:99:79:9c:7c:62:26:7f:90:7d:39:c1:42:33:ab:b8:
         4e:1f:23:58:29:20:82:c7:fd:d6:f2:b8:4f:7e:22:b0:63:6f:
         77:dd:11:68:5b:79:24:06:1e:c6:1e:c0:e5:a7:c0:76:13:07:
         0c:55:db:e6

5. Trigger a failure due to incorrect certificate. Go to the "Network Topology" in OpenStack Dashboard.

"Network Topology" dashboard fails to load. And in neutron log you can see the following error message:

requests.exceptions.SSLError: HTTPSConnectionPool(host='octavia-internal.example.com', port=9876): Max retries exceeded with url: /v2.0/lbaas/l7policies?project_id=69d2ec2c94b34921bb54eb81c6905688 (Caused by SSLError(CertificateError("hostname 'octavia-internal.example.com' doesn't match either of 'octavia.example.com', 'octavia.example.com', '10.127.12.42', '10.127.12.92'",),))

WORKAROUND

Configure all Octavia endpoints to point to the same domain, e.g. octavia.example.com.

Revision history for this message
Przemyslaw Hausman (phausman) wrote :
Revision history for this message
Przemyslaw Hausman (phausman) wrote :
Revision history for this message
Przemyslaw Hausman (phausman) wrote :
Revision history for this message
Przemyslaw Hausman (phausman) wrote :

juju run --unit octavia/0 'relation-ids certificates'
certificates:304

juju run --unit octavia/0 'relation-list -r certificates:304'
vault/0
vault/1
vault/2

juju run --unit octavia/0 'relation-get -r certificates:304 - octavia/0'
cert_requests: '{"eth2.juju-439b02-25-lxd-7.example.com": {"sans": ["10.127.2.213"]},
  "octavia.example.com": {"sans": ["10.127.12.42", "10.127.12.92", "octavia.example.com"]}}'
certificate_name: fe75e775-7754-49b8-ad6d-6e71823c5193
common_name: octavia-internal.example.com
egress-subnets: 10.127.2.213/32
ingress-address: 10.127.2.213
private-address: 10.127.2.213
sans: '["10.127.1.108", "10.127.1.42", "octavia-internal.example.com"]'
unit_name: octavia_0

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Interesting, I am seeing the same as Przemyslaw, but when I ran the opposite (from Vault side), there is no "sans" information:

$ juju run --unit vault/0 'relation-ids certificates'
certificates:315
certificates:316
certificates:317
certificates:318
certificates:319
certificates:320
certificates:321
certificates:322
certificates:323
certificates:324
certificates:325
certificates:326
certificates:327
$ juju run --unit vault/0 'relation-list -r certificates:325'
octavia/0
octavia/1
octavia/2
$ juju run --unit vault/0 'relation-get -r certificates:325 - vault/0'
egress-subnets: 10.32.17.32/32
ingress-address: 10.32.17.32
private-address: 10.32.17.32

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Nevermind my last comment, I was grabbing info sent by vault/0

Revision history for this message
Yoshi Kadokawa (yoshikadokawa) wrote :

I am having a similar issue with placement charm.
https://bugs.launchpad.net/charm-placement/+bug/1866741

Changed in charm-octavia:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
David Ames (thedac) wrote :

Work on this bug is being done on https://bugs.launchpad.net/charm-placement/+bug/1866741. The bug effects all reactive charms.

Marking this one as duplicate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.