Fails to verify certificates for internal and admin endpoints when using Vault

[Yoshi Kadokawa]

When Vault is used as a Root CA, the expectation is that all certifications and keys for Placements endpoints are created as the other OpenStack components.
However, with the example bundle[0], certifications and keys were not created for internal and admin endpoint.

Here is the output of /etc/apache2/ssl/placement/

$ ll /etc/apache2/ssl/placement
total 10
dr-xr-xr-x 2 root root 10 Mar 10 03:52 ./
dr-xr-xr-x 3 root root 3 Mar 10 03:52 ../
-rw-r----- 1 root placement 1483 Mar 10 03:53 cert_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 49 Mar 10 03:52 cert_placement-admin.test -> /etc/apache2/ssl/placement/cert_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 49 Mar 10 03:52 cert_placement-internal.test -> /etc/apache2/ssl/placement/cert_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 49 Mar 10 03:52 cert_placement.test -> /etc/apache2/ssl/placement/cert_juju-6f67e2-2.lxd
-rw-r----- 1 root placement 1678 Mar 10 03:53 key_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 48 Mar 10 03:52 key_placement-admin.test -> /etc/apache2/ssl/placement/key_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 48 Mar 10 03:52 key_placement-internal.test -> /etc/apache2/ssl/placement/key_juju-6f67e2-2.lxd
lrwxrwxrwx 1 root root 48 Mar 10 03:52 key_placement.test -> /etc/apache2/ssl/placement/key_juju-6f67e2-2.lxd

As you can see, only one cert and key are created and all others are just a link to that.
Because of this, internal endpoint is not reachable, and fails to create instances.

[0] https://pastebin.ubuntu.com/p/gdjRK4Td98/

[Yoshi Kadokawa (yoshikadokawa)]

I have attached the output of

$ sudo openssl x509 -in /etc/apache2/ssl/placement/cert_juju-6f67e2-2.lxd -noout -text

As you can see, there is no subject name for admin and internal hostnames.

-- EXCERPT --
Subject: CN = placement.test
-- SKIP --
X509v3 Subject Alternative Name:
    DNS:placement.test, DNS:placement.test, IP Address:172.31.16.205
-- SKIP --

[Yoshi Kadokawa (yoshikadokawa)]

Here is the crashdump when reproduced with the bundle mentioned in the description.

[Yoshi Kadokawa (yoshikadokawa)]

Since this is blocking a deployment. I have subscribed this to field-critical.

[Yoshi Kadokawa (yoshikadokawa)]

I believe the symptom of this issue is quite close as this one.
https://bugs.launchpad.net/charm-octavia/+bug/1865447

[Chris MacNaughton (chris.macnaughton)]

The vault<-> placement relation does seem to have cert data for all of the endpoints (enjoy this temporary key data) so I suspect that there's an issue with the rendering code:

placement_0.processed_requests: '{"juju-c1823a-tls-test-2.project.serverstack": {"cert":
  "-----BEGIN CERTIFICATE-----\nMIIEOjCCAyKgAwIBAgIUJLjvliMwY7xE7YlqjxrejvSZ0uIwDQYJKoZIhvcNAQEL\nBQAwPTE7MDkGA1UEAxMyVmF1bHQgUm9vdCBDZXJ0aWZpY2F0ZSBBdXRob3Jp
dHkg\nKGNoYXJtLXBraS1sb2NhbCkwHhcNMjAwMzEwMDY1NTU3WhcNMjEwMzEwMDU1NjI3\nWjA1MTMwMQYDVQQDEypqdWp1LWMxODIzYS10bHMtdGVzdC0yLnByb2plY3Quc2Vy\ndmVyc3RhY2swggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCh+lAJ/yK9\njKjNm/eXZzVD3e6ovIsKNULDF1XN3o6uB9bGJ8jGBW7ZbVgXf/pklB2bNHKrQ7qj\nJfywJwBj7WPmLX5NuM6/X3e+sygwVfJM7zVhSyxQ1o6pbU
nP1Dp1U8fQ0oDjLMl4\nguqpdM1UeYZv0MiyUggRj2T+xW+PqNEFrcehipgex84AMIFdKsihYsUZgjHFRv4/\nzlKPQkUYeSaVTxrRyR5yEWhKqKHr/AuKBTwzvUgzRqPrH0VFt2UD/972AKn55+oQ\nLyGges
gFqAuxl9NUOnzn7Vzn0Z0KrjrQSdtEuDgRRqnM3x/SMRYWz2APd4LJi8pE\nTXXE/ozibCdxAgMBAAGjggE4MIIBNDAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0G
A1UdDgQWBBTlFQK/Rn1IhwlXAGRreqHS\nOmwwlTAfBgNVHSMEGDAWgBTOTMcOY/Yy4YU0LDtbvQVsrBZoEDBHBggrBgEFBQcB\nAQQ7MDkwNwYIKwYBBQUHMAKGK2h0dHA6Ly8xMC41LjAuMzE6ODIwMC92MS
9jaGFy\nbS1wa2ktbG9jYWwvY2EwOwYDVR0RBDQwMoIqanVqdS1jMTgyM2EtdGxzLXRlc3Qt\nMi5wcm9qZWN0LnNlcnZlcnN0YWNrhwQKBQBNMD0GA1UdHwQ2MDQwMqAwoC6GLGh0\ndHA6Ly8xMC41LjAuMz
E6ODIwMC92MS9jaGFybS1wa2ktbG9jYWwvY3JsMA0GCSqG\nSIb3DQEBCwUAA4IBAQCozkrwvww3rdtjUT6iflx+moqNWWkgyukYdMhLaSd0NrC6\nlRpm7F+ho2z++xKNjEQsAOE1FuFiRWOfWRKRErjXMOB9
nL3nuY9wB4wO65tR1faw\nPp0hjOxrYxw435+RRAU2o9O/vLJdvWHsTF7uL5T6cvJ4t/JtPNNA18f6SzFw+vqz\nBeG5tlm1d4VB4AVE1Ld0ErtNaa9kSk1XzKFkJ9TuXII/qLDRSrO8nSCMFQZ8fxwO\nCr86
ROHTgOHjIKRFZCDBfElzHWWR7pYEz/f9TzSYOTYD1aoWA8nOOyX2m/o5ngUg\nCp1amJs+V9lE1aHtxUrFNKXFq5MBiNiCEjKh7vsU\n-----END
  CERTIFICATE-----", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAofpQCf8ivYyozZv3l2c1Q93uqLyLCjVCwxdVzd6OrgfWxifI\nxgVu2W1YF3/6ZJQdmzRyq0O6oyX8sC
cAY+1j5i1+TbjOv193vrMoMFXyTO81YUss\nUNaOqW1Jz9Q6dVPH0NKA4yzJeILqqXTNVHmGb9DIslIIEY9k/sVvj6jRBa3HoYqY\nHsfOADCBXSrIoWLFGYIxxUb+P85Sj0JFGHkmlU8a0ckechFoSqih6/wL
igU8M71I\nM0aj6x9FRbdlA//e9gCp+efqEC8hoHrIBagLsZfTVDp85+1c59GdCq460EnbRLg4\nEUapzN8f0jEWFs9gD3eCyYvKRE11xP6M4mwncQIDAQABAoIBAQCTnt1zPuMafSca\nvBpaEeWphIoNnkfZ
ddDynEHG7h563QoQbhG85xavccfnIuvA3nxdBt+61m8yYVx5\n7hGdAK0bCjsh+lvybb9kPUNTSgEZvKiZkzlAM0qxfrjWgEUGyORCnJZ6dEbpXecf\nqSO46Uhsf4tpePmBh34z8xxJgUF+Of+CzmPdNQ5Mab
J6I77uhA19rk13dbvo9YKm\nCWeBEqFbH76qNmIw+QfMEvtMmBHnJGv4Xa1fZ+mKSqWmeoUfbJ8r+QmydnqzyaAl\nATRbGCYf52zPLEezBwulNNtziPms8jirpqcsrM2GBjL7xLJr4vcltb1Yr00kS8v1\nr+
9OgfMBAoGBANA1St6iED3X3hZD2rtgQT5oLVOF8IDf/Tvh8pATtZa6NAHv8hbI\nTTNp79JP9QK9hgt1qbDfqOB8gSlFGCId+VWcmujCjZIatmbiBYoeNjEQ5+uOKze3\ng+BoQw0F1M9pAAQ1O/FieP30ZLqF
SJUX+3iLMmZFjWu7BEU72fyayUdtAoGBAMco\nb0nuLLZ8bq1UDpcEW8P+XLTwZDDY6yq4gaRgwGWfKhLfjiW8tt1raHmddAKy4+rg\nL9IKoIln8pjgmH2yCUkfsEY1Fs6N1RYp6ciVcOOCHK72ktdUsejveS
ne+btFXDHd\nxQZXwu3xN/lZ1iIDVK4UgHyH6rsrS9zu7fRXVcmVAoGBAKMIJ9D+i8MxJfi/NyYj\nr9LjFiLhrTnsqkjamunQVQ9qTFD8Bs9qFnFc0WoYK9zydGTCxJX3/C+TrjuXm9cT\nK2qrDu2Vce9mtU
8ISpglIf/3ofJjx1mAfGYane4zk7i5GkcxO/e+SMlqNfmLZhNA\nzStNq7BDZAJoVWa0//L5bQRdAoGAfJsD5lGqkBKSfMfWyQ3lDF+dSWgOh7Fwka4X\...

[Chris MacNaughton (chris.macnaughton)]

As keystone doesn't exhibit this behaviour in the in the given bundle, I believe this to be an issue in charms.openstack around rendering provided TLS certificates

[Andrew McLeod (admcleod)]

Confirmed that all keys (since it is one key) point only do the public endpoint

            X509v3 Subject Alternative Name:
                DNS:placement.test, DNS:placement.test, IP Address:10.5.0.14

[Andrew McLeod (admcleod)]

Also confirmed #6 - keystone has a cert for each endpoint, each containing correct X509v3 Subject Alternative Name

[Andrew McLeod (admcleod)]

This does appear to be an issue in charms.openstack - but I am as yet unable to pinpoint the exact location - also this therefore affects all reactive charms.

The appropriate data is in the placement end of the relation:

cert_requests: '{"juju-82bdef-admcleod-2.project.serverstack": {"sans": ["10.5.0.14"]},
  "placement-admin.test": {"sans": ["10.5.0.14", "placement-admin.test"]}, "placement.test":
  {"sans": ["10.5.0.14", "placement.test"]}}'
certificate_name: 4936a579-0301-4035-9c5a-8f73636f8ef3
common_name: placement-internal.test
egress-subnets: 10.5.0.14/32
ingress-address: 10.5.0.14
private-address: 10.5.0.14
sans: '["10.5.0.14", "placement-internal.test"]'
unit_name: placement_0

[Nobuto Murata (nobuto)]

@Andrew, thanks for the analysis. I need to confirm it with Yoshi, but I believe two of three endpoints have the same network space and the IP address, then the remaining one is on a separate space and has a different IP address.

[Nobuto Murata (nobuto)]

Nah, I've checked the deployment artifacts used for the customer deployment, then each endpoint is in separate network spaces so IP addresses are different each other.

[Andrew McLeod (admcleod)]

My finding may be completely erroneous - I tried to revalidate my findings and it looks like I was wrong, so #10 can be ignored.

[Andrew McLeod (admcleod)]

https://github.com/juju-solutions/interface-tls-certificates/blob/master/requires.py#L250

This is called multiple times for each net_type (internal, admin, public) and I can see that the content of the primary certificate is changing, i.e. being overwritten, rather than being updated/ammended

[David Ames (thedac)]

Initial testing of Chris' suggestion is promising.

The following patch to charms.openstack can be tested:

diff --git a/charms_openstack/charm/classes.py b/charms_openstack/charm/classes.py
index 1c186ac..8568f42 100644
--- a/charms_openstack/charm/classes.py
+++ b/charms_openstack/charm/classes.py
@@ -953,14 +953,14 @@ class HAOpenStackCharm(OpenStackAPICharm):
                         tls_object['cert'],
                         tls_object['key'],
                         cn=tls_object['cn'])
- cert_utils.create_ip_cert_links(
- os.path.join('/etc/apache2/ssl/', self.name))
- if not os_utils.snap_install_requested() and changed:
- self.configure_apache()
- ch_host.service_reload('apache2')
-
- self.remove_state('ssl.requested')
- self.set_state('ssl.enabled', True)
+ cert_utils.create_ip_cert_links(
+ os.path.join('/etc/apache2/ssl/', self.name))
+ if not os_utils.snap_install_requested() and changed:
+ self.configure_apache()
+ ch_host.service_reload('apache2')
+
+ self.remove_state('ssl.requested')
+ self.set_state('ssl.enabled', True)
             else:
                 self.set_state('ssl.enabled', False)

Further testing is in progress.

[David Ames (thedac)]

Pastebin version of patch to avoid squashing of spaces:
https://pastebin.ubuntu.com/p/6CvHVgpNK5/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/712176

[David Ames (thedac)]

Required next steps:

* Validate, review and land [0]
* Rebuild reactive charms with this change

Optional next steps:
* Consider erring out in configure_cert when the filename is a sym link
* Backport IFF safe
* Consider deprecating [1] and refactoring apache2 site config (see bellow)

As we do not appear to send SANs of all hostnames, the sym linking probably has never worked in any charms. It probably exists, to avoid apache2 failing to start when it expects a certificate that is not there. We should consider refactoring the apache2 site config to only render sites we expect to exist and therefore remove the need to sym link certificates.

[0] https://review.opendev.org/712176
[1] https://github.com/juju/charm-helpers/blame/650d8a506f2f3d8c2cf6834d3c975bd6dbfe8014/charmhelpers/contrib/openstack/cert_utils.py#L148

[David Ames (thedac)]

Validated against the original post's bundle [0] with an updated placement charm.

I would still like a second pair of eyes to confirm.

[0] https://pastebin.ubuntu.com/p/gdjRK4Td98/

[Yoshi Kadokawa (yoshikadokawa)]

Thanks for the patch.
I have deployed with the patch, and I could also confirm that the patch is working.
If I'm understanding it correctly, this issue affects to all openstack charms that are in reactive.
However, I've also checked with designate, which is also a reactive charm, this one does not have this issue.
Here is the bundle that I have tested with.
https://pastebin.ubuntu.com/p/rPHs8CBJXx/

[Yoshi Kadokawa (yoshikadokawa)]

Sorry, my last comment was sent halfway.
What I meant was that despite Designate is in reactive charm, Designate worked just out of the box without any issues for the SSL certs.
I was a little concerned if this really always happens on reactive OpenStack charms.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/712176
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=72cfde8308684905363e80e4ca5db57f18573927
Submitter: Zuul
Branch: master

commit 72cfde8308684905363e80e4ca5db57f18573927
Author: David Ames <email address hidden>
Date: Tue Mar 10 11:24:46 2020 -0700

    Avoid multiple calls to create_ip_cert_links

    Prior to this change, the call to create_ip_cert_links was inside the
    for loop of tls objects. This caused LP Bug#1866741 where a sym link
    with the same certificate name would get overwritten with a new cert.
    This meant the last certificate of [internal, admin, public] would be
    written to all three.

    This change calls create_ip_cert_links and state sets only once. It also
    adds a warning log entry if we accidentally overwrite sym linked files in
    the future.

    Change-Id: I4b36d7fd5e74ab3a894eb4a2451b5876fd777552
    Partial-Bug: #1866741

[David Ames (thedac)]

@Yoshi,

Acknowledged, and thank you for pointing this out. I confirmed that designate does not seem to have the problem, however it seems to be by accident. Designate happens to get its admin, internal, public certs first before its host cert, where placement (and others) do not tripping over the bug.

2020-03-11 17:45:22 WARNING juju-log certificates:19: XXX configure cert cert_designate-admin.test
2020-03-11 17:45:23 WARNING juju-log certificates:19: XXX call to create_ip_cert
2020-03-11 17:46:14 WARNING juju-log certificates:19: XXX configure cert cert_designate-internal.test
2020-03-11 17:46:15 WARNING juju-log certificates:19: XXX call to create_ip_cert
2020-03-11 17:46:47 WARNING juju-log certificates:19: XXX configure cert cert_designate.test
2020-03-11 17:46:47 WARNING juju-log certificates:19: XXX call to create_ip_cert
2020-03-11 17:47:19 WARNING juju-log certificates:19: XXX configure cert cert_juju-acdb48-zaza-316fe218e683-0.project.serverstack
2020-03-11 17:47:19 WARNING juju-log certificates:19: XXX call to create_ip_cert

2020-03-11 17:43:42 WARNING juju-log certificates:18: XXX configure cert cert_juju-acdb48-zaza-316fe218e683-5.project.serverstack
2020-03-11 17:43:42 WARNING juju-log certificates:18: XXX call to create_ip_cert
2020-03-11 17:43:49 WARNING juju-log certificates:18: XXX configure cert cert_placement-admin.test
2020-03-11 17:43:50 WARNING juju-log certificates:18: XXX call to create_ip_cert
2020-03-11 17:43:56 WARNING juju-log certificates:18: XXX configure cert cert_placement-internal.test
2020-03-11 17:43:56 WARNING juju-log certificates:18: XXX call to create_ip_cert
2020-03-11 17:44:03 WARNING juju-log certificates:18: XXX configure cert cert_placement.test
2020-03-11 17:44:03 WARNING juju-log certificates:18: XXX call to create_ip_cert

I suspect this is alphanumeric ordering by the vault charm, but regardless the bug stands, the fix should correct this for all cases. Rebuilds of all reactive charms are in progress:

https://review.opendev.org/#/q/topic:rebuild+(status:open+OR+status:merged)

[Nobuto Murata (nobuto)]

Hi,

According to the status:
https://review.opendev.org/#/q/topic:rebuild+(status:open+OR+status:merged)
While charm-aodh has some CI failures, most of the charm rebuild requests has been merged.

However, charm-placement, for example, in the charm store still doesn't have the change (yet).
https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/placement-6/archive/repo-info

When can we expect to have it in ~openstack-charmers-next and/or ideally in the stable channel? That would be helpful to plan the action plan in ongoing projects.

Thanks,

[David Ames (thedac)]

@Nobuto,

Thanks for pointing this out. I expected the openstack-charmers-next version to already have had the fix. I confirmed that it does not.

I am investigating now.

[David Ames (thedac)]

@Nobuto,

I have pushed and confirmed that cs:~openstack-charmers-next/placement-8 contains fix.

Other reactive charms are in the process of being rebuilt.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/713205

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713205
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=8702d27254c6e73ffd383bada69ddcafd0c7f970
Submitter: Zuul
Branch: stable/20.02

commit 8702d27254c6e73ffd383bada69ddcafd0c7f970
Author: David Ames <email address hidden>
Date: Tue Mar 10 11:24:46 2020 -0700

    Avoid multiple calls to create_ip_cert_links

    Prior to this change, the call to create_ip_cert_links was inside the
    for loop of tls objects. This caused LP Bug#1866741 where a sym link
    with the same certificate name would get overwritten with a new cert.
    This meant the last certificate of [internal, admin, public] would be
    written to all three.

    This change calls create_ip_cert_links and state sets only once. It also
    adds a warning log entry if we accidentally overwrite sym linked files in
    the future.

    Change-Id: I4b36d7fd5e74ab3a894eb4a2451b5876fd777552
    Partial-Bug: #1866741
    (cherry picked from commit 72cfde8308684905363e80e4ca5db57f18573927)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/713291

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713291
Committed: https://git.openstack.org/cgit/openstack/charm-placement/commit/?id=72a060167fbf782bb0842c65bbed8948f165ca53
Submitter: Zuul
Branch: stable/20.02

commit 72a060167fbf782bb0842c65bbed8948f165ca53
Author: Alex Kavanagh <email address hidden>
Date: Mon Mar 16 17:02:50 2020 +0000

    Avoid multiple calls to create_ip_cert_links

    (This is a rebuild of the charm to pull in the fix that is in stable
    charms.openstack.)

    Prior to this change, the call to create_ip_cert_links was inside the
    for loop of tls objects. This caused LP Bug#1866741 where a sym link
    with the same certificate name would get overwritten with a new cert.
    This meant the last certificate of [internal, admin, public] would be
    written to all three.

    This change calls create_ip_cert_links and state sets only once. It also
    adds a warning log entry if we accidentally overwrite sym linked files in
    the future.

    Change-Id: Idd99fac8b5c48755b2ef2797deaa688cd2544281
    Partial-Bug: #1866741

[Nobuto Murata (nobuto)]

Those two have landed onto stable/20.02.

[Nobuto Murata (nobuto)]

https://review.opendev.org/#/c/712265/
https://review.opendev.org/#/c/712263/

[David Ames (thedac)]

Confirmed AODH did not yet have the fix. Using the rebuild review to do son now:

https://review.opendev.org/#/c/712254/

[David Ames (thedac)]

https://review.opendev.org/#/c/712254/ has landed.

I confirmed cs:~openstack-charmers-next/aodh-105 has the fix in the charm store.

[James Page (james-page)]

I have a number of rebuilds and cherry-picks todo for reactive charms for stable which will also resolve these outstanding tasks for the 20.02 stable release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/716640

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716640
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=06d8be372cbdd8afde9a94d55519eee5fd845360
Submitter: Zuul
Branch: stable/20.02

commit 06d8be372cbdd8afde9a94d55519eee5fd845360
Author: David Ames <email address hidden>
Date: Tue Mar 24 23:10:42 2020 +0000

    Fix HAProxy back ends

    Requires changes to charms.openstack and layer-openstack-api.

    Closes-Bug: #1858132
    Depends-On: I5716c28dd45b2d33325cf09487107543b8a42626
    (cherry picked from commit d6622db11d400d4321739bcddc7870ee4f2879d0)

    Rebuild also resolves bugs relating to certificate verification
    for internal and admin endpoints.

    Closes-Bug: #1866741

    Change-Id: Iaf20a0cd7bd2bc9accb20bd8987ed87289059e1c