Installing openstack certificate with misconfigured "X509v3 Subject Alternative Name" renders system inoperable

[Chris Winnicki]

Brief Description
-----------------
Installing openstack certificate with misconfigured "X509v3 Subject Alternative Name" renders system inoperable
* Where the subjectAltName is not a Fully Qualified Domain Name
* Both controllers end up in stand-by mode

Severity
--------
Major: System/Feature is usable - renders the system inoperable

Steps to Reproduce
------------------
Install 1+1 system (aka: AIO-DX)
Enable https on the system
Add FQDN system paramer: penstack helm endpoint_domain
Install openstack certificate with mismatched FQDN in the "X509v3 Subject Alternative Name" field

Expected Behavior
------------------
After installing https Openstack certificate, even if the "X509v3 Subject Alternative Name"
is misconfigured system should still be operational

Actual Behavior
----------------
Both controllers end up in stand-by mode

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
1+1 system (aka: AIO-DX)
( Internal Lab name: yow-cgcs-wildcat-3_6 )

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20190415T233001Z"
JOB="STX_build_master_master"
<email address hidden>"

Last Pass
---------
n/a (First time feature testing)

Timestamp/Logs
--------------
See attached logs:
ALL_NODES_20190422.191652.tar.partaa
ALL_NODES_20190422.191652.tar.partab
ALL_NODES_20190422.191652.tar.partac
ALL_NODES_20190422.191652.tar.partad
ALL_NODES_20190422.191652.tar.partae
ALL_NODES_20190422.191652.tar.partaf
ALL_NODES_20190422.191652.tar.partag
ALL_NODES_20190422.191652.tar.partah
ALL_NODES_20190422.191652.tar.partai
ALL_NODES_20190422.191652.tar.partaj
ALL_NODES_20190422.191652.tar.partak
ALL_NODES_20190422.191652.tar.partal
ALL_NODES_20190422.191652.tar.partam
ALL_NODES_20190422.191652.tar.partan

To reconstruct original log file:
1) Download all the logs
2) execute: cat ALL_NODES_20190422.191652.tar.part* > ALL_NODES_20190422.191652.tar

Test Activity
-------------
Feature Testing

Other
-----
Example of certificate with misconfigured subjectAltName

cwinnick@yow-cwinnick-lx:/folk/cgts/users/cwinnick/https-cert-mgmt$ openssl x509 -in server-with-key.pem -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14478747910509723221 (0xc8eed8834e5c7e55)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Alameda, O=Wind River Inc, <email address hidden>
        Validity
            Not Before: Apr 12 14:11:30 2019 GMT
            Not After : Apr 11 14:11:30 2020 GMT
        Subject: C=CA, ST=Ont, L=ott, O=wrs, OU=tis, CN=*.<email address hidden>
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b6:5c:45:8d:bf:02:17:4a:8d:41:0f:ee:60:28:
                    8b:36:0f:c9:cd:f1:9f:f1:e0:22:59:d6:5a:f9:0a:
                    33:fa:b5:67:c7:04:f5:1b:92:3d:d5:5e:db:57:e8:
                    75:8f:a5:d9:b3:5b:29:c0:a0:5d:ec:05:b6:59:38:
                    35:f5:ef:15:8c:7e:d0:fb:5c:51:1f:d5:4d:f5:93:
                    76:7f:cb:5a:6f:4a:ad:ab:a6:2c:5f:d9:04:bb:ca:
                    dc:e1:b6:3e:b7:77:a1:e6:3a:88:28:d1:61:10:2b:
                    33:ef:61:c4:b0:2d:3a:0e:c7:af:69:fd:96:d9:6d:
                    33:ee:90:e7:a2:2d:f8:89:35:98:6a:16:cb:7f:14:
                    c5:67:a8:26:1f:0a:90:93:43:19:b1:2e:54:37:7e:
                    50:b3:7c:f3:0a:74:2f:66:df:5c:fd:3e:68:be:66:
                    24:6e:a4:68:e0:61:f0:21:c6:6e:8c:27:5c:02:25:
                    f8:a8:0c:35:cc:0e:08:b5:96:0c:55:72:6e:81:b6:
                    5a:42:aa:cc:48:00:68:5c:61:fc:c7:47:9d:6a:19:
                    58:ec:66:46:cb:28:f6:f2:be:5d:19:55:64:4d:b7:
                    f3:ea:f3:3b:b6:34:cb:21:0d:48:de:c0:27:50:d9:
                    6a:96:86:dd:0e:d2:85:0c:18:eb:99:a1:50:d0:c8:
                    7d:73
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:cgcs-wildcat-3-6
    Signature Algorithm: sha256WithRSAEncryption
         0a:b9:e7:41:58:64:c7:ef:73:ad:89:94:7d:6b:9b:3a:dd:b0:
         a6:8e:d8:46:49:b3:1a:99:54:c0:37:ea:84:74:7e:82:72:cf:
         cf:2e:89:8f:3b:ca:ba:b9:14:f0:9a:23:77:d9:32:e9:57:62:
         ff:7b:dd:69:00:67:30:54:c8:60:7a:93:c3:59:68:14:e3:c5:
         3e:90:d4:87:fc:66:c5:90:f0:6d:01:47:51:4d:46:7d:ec:75:
         fe:e5:72:e4:16:27:74:af:c6:98:fc:06:84:50:2a:6d:e6:fe:
         91:41:ba:ef:b3:84:7b:78:ed:a3:92:22:21:55:a6:a9:c2:45:
         ac:25:01:9a:5e:c4:4e:5f:5b:05:bb:f8:9d:ad:ef:e6:00:af:
         94:d1:ab:60:d4:cf:a0:db:60:26:ee:46:7a:37:5b:42:cd:06:
         1c:c9:21:95:47:df:ae:09:0c:6a:29:d1:ea:f3:91:1a:bb:0e:
         47:bb:1a:16:6a:b5:13:11:c1:5d:1a:12:c1:a9:c4:91:55:fb:
         56:cc:1c:d6:48:16:87:59:53:7d:3c:51:4e:72:bf:9c:66:96:
         35:df:d8:c9:15:f5:7c:46:ec:78:4a:14:ee:de:61:7e:5e:49:
         ca:7a:29:96:f8:3f:b6:8a:b2:89:43:1d:5c:c1:02:be:c2:86:
         4b:a8:ec:c3

[Frank Miller (sensfan22)]

Marking as release gating; medium priority as the system should not be in dual standby mode when using an incorrect certificate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671479
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=6ac774799e1db3d2bf9cc5c053593005e8d9d763
Submitter: Zuul
Branch: master

commit 6ac774799e1db3d2bf9cc5c053593005e8d9d763
Author: Teresa Ho <email address hidden>
Date: Wed Jul 17 23:43:25 2019 -0400

    Add checks for dns name in certificate

    This commit adds validation of https certificate for openstack
    to ensure the provisioned endpoint domain matches the dns names
    or the common name in the certificate.

    If the domain is example.com, then the CN or SAN value must be
    *.example.com.
    SAN values, such as *.*.example.com, bad.*.example.com would not be
    accepted.

    Closes-Bug: 1826227

    Change-Id: Ib50491282e8bf15b42f008d4eb55a7db9d64999b
    Signed-off-by: Teresa Ho <email address hidden>

[Boris Shteinbock (bshteinb)]

# Testing Status
PASSED

# Configuration
standard system:
2 controllers
2 computes

# Load Tested
2019-09-11_00-10-00

1. System was installed
2. Controller nodes configured and unlocked
3. Https was enabled
[sysadmin@controller-0 ~(keystone_admin)]$ system modify --https_enabled true
+----------------------+--------------------------------------+
| Property | Value |
+----------------------+--------------------------------------+
| contact | None |
| created_at | 2019-09-25T18:07:37.356030+00:00 |
| description | None |
| https_enabled | True |
| location | None |
| name | d4677edf-6718-4950-9bcd-0398edec3d14 |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.09 |
| system_mode | duplex |
| system_type | Standard |
| timezone | UTC |
| updated_at | 2019-09-25T18:09:34.401338+00:00 |
| uuid | b13a6324-c7b1-42db-abef-73517c538463 |
| vswitch_type | none |
+----------------------+--------------------------------------+

4. A misconfigured certificate was installed.
[sysadmin@controller-0 ~(keystone_admin)]$ system certificate-install ./cert-with-key-openstack.pem
WARNING: For security reasons, the original certificate,
containing the private key, will be removed,
once the private key is processed.
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | c72eb73a-affc-4d40-b7a4-ff223001af64 |
| certtype | ssl |
| signature | ssl_14478747910509723273 |
| start_date | 2019-09-25 18:25:53+00:00 |
| expiry_date | 2020-09-24 18:25:53+00:00 |
+-------------+--------------------------------------+

5. System is still fully operational and https accessible:
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
| 4 | compute-0 | worker | unlocked | enabled | available |
| 5 | compute-1 | worker | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+

5. System was...