Certificate provided with ssl_ca configuration option not installed in system certificate store

[Nicolas Pochet]

When deploying a bundle with TLS and adding the charm octavia-diskimage-retrofit, it fails when trying to run the retrofit-image action with the following error:

juju run-action --wait octavia-diskimage-retrofit/leader retrofit-image
unit-octavia-diskimage-retrofit-1:
  id: dee15cf0-ec18-4ac5-8ceb-046b5ae2d4aa
  message: 'Could not find versioned identity endpoints when attempting to authenticate.
    Please check that your auth_url is correct. SSL exception connecting to https://keystone-internal.customer:35357:
    HTTPSConnectionPool(host=''keystone-internal.customer'',
    port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad
    handshake: Error([(''SSL routines'', ''tls_process_server_certificate'', ''certificate
    verify failed'')],)",),))'
  status: failed
  timing:
    completed: 2019-11-06 08:31:16 +0000 UTC
    enqueued: 2019-11-06 08:31:12 +0000 UTC
    started: 2019-11-06 08:31:12 +0000 UTC
  unit: octavia-diskimage-retrofit/1

The ssl_ca parameter is configured properly but it does not seem to be anywhere on the deployed unit.
Looking at the charm code https://github.com/openstack/charm-octavia-diskimage-retrofit/blob/master/src/lib/charm/openstack/glance_retrofitter.py#L46 shows that it uses the system CA bundle.
Grepping on that file for the certificate gives no result. That means that the CA certificate that is passed through the config is not in /etc/ssl/certs/ca-certificates.crt

Looking, in the charm itself how this parameters is used gives nothing:
charm pull cs:octavia-diskimage-retrofit-2
cs:octavia-diskimage-retrofit-2
➜ /tmp cd octavia-diskimage-retrofit
➜ octavia-diskimage-retrofit grep -R ssl_ca .
./templates/parts/section-rabbitmq-oslo:{% if amqp.ssl_ca -%}
./templates/parts/section-rabbitmq-oslo:kombu_ssl_ca_certs = {{ amqp.ssl_ca_file }}
./templates/parts/section-oslo-messaging-rabbit:{% if amqp.ssl_ca -%}
./templates/parts/section-oslo-messaging-rabbit:ssl_ca_file = {{ amqp.ssl_ca_file }}
./config.yaml: "ssl_ca":

It seems that the ssl_ca config is there but not used at all.

[Nicolas Pochet (npochet)]

Subscribed field-medium as it impacts all deployments with Octavia where one wants to generate amphorae automatically and not rely on the test-images.

[Alex Kavanagh (ajkavanagh)]

It's an "accident" that the ssl_ca option got included into the charm. It's the "fault" of charms.openstack and the default handler called "certificates.available" which isn't activated by this charm. i.e. the configuration items are in the charm, but they are not used.

octavia and SSL is really used with vault.

To fix this the default "certificates.available" could be used and they SSL CA stuff should get to the right place.

[Przemyslaw Hausman (phausman)]

I'm hitting the same issue on customer deployment.
Vault is not used here, so there's no 'certificates' relation.

The workaround is to manually provision octavia-diskimage-retrofit charm with CA certificate:

juju scp tls/root-ca.pem octavia-diskimage-retrofit/0:/home/ubuntu

juju run --unit octavia-diskimage-retrofit/0 '\
  sudo mkdir -p /usr/local/ca-certificates/custom; \
  sudo mv /home/ubuntu/root-ca.pem /usr/local/ca-certificates/custom/root-ca.crt; \
  sudo chown root:root /usr/local/ca-certificates/custom/root-ca.crt; \
  sudo update-ca-certificates'

[Marco Filipe Moutinho da Silva (mfmsilva)]

Issue seems to have been fixed.
I've launched a test with the configuration found here (https://paste.ubuntu.com/p/RCjMRRZfp3/) and then tried to curl the https endpoint from within the retrofit container:

* common name: 10.5.0.15 (matched)
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
* SSL certificate verify ok.
} [5 bytes data]
> GET / HTTP/1.1
> Host: 10.5.0.15:5000
> User-Agent: curl/7.58.0
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 300 Multiple Choices
< Date: Tue, 12 May 2020 11:04:44 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Content-Length: 595
< Vary: X-Auth-Token
< X-Distribution: Ubuntu
< Content-Type: application/json
<

Certificate is recognized as valid and the request goes through fine. Anyone still facing the issue?

[Marco Filipe Moutinho da Silva (mfmsilva)]

Please disregard my comment above. Turns out it was because the retrofit was on the same machine as Keystone (which is correctly installing the CA).
Will test further.

[Frode Nordahl (fnordahl)]

Note that octavia-diskimage-retrofit is a subordinate charm and it is meant to be deployed together with a principle charm that will take care of configuring the system certificate store.

I would consider the exposure of the ssl_\* configuration options and the certificates relation on the charm as a bug, and I would also consider the lack of documentation around the above statement a bug.

The charm should use the system certificate store when calling out to API clients.

@OP: What is the principle charm used in the deployment in question, and is it configured to install CA in the system certificate store?

[Przemyslaw Hausman (phausman)]

In my case octavia-diskimage-retrofit was related with glance-simplestreams-sync. And g-s-s had ssl_ca config option set.

[Frode Nordahl (fnordahl)]

glance-simplestreams-sync does indeed not install the certificate provided with the ``ssl_ca`` configuration option in the system certificate store.

Adding gss to the bug and keeping the octavia-diskimage-retrofit task open so we track removal of the ssl_* configuration options and expanding the documentation so it points users to principle charm for TLS configuration.

Would also like to note that we have gss+retrofit with TLS certificates issued from Vault in the Octavia charm gate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/727784

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/727784
Committed: https://git.openstack.org/cgit/openstack/charm-octavia-diskimage-retrofit/commit/?id=e769190e59a8ab3ba5cecde43a757761f1ff7009
Submitter: Zuul
Branch: master

commit e769190e59a8ab3ba5cecde43a757761f1ff7009
Author: Frode Nordahl <email address hidden>
Date: Wed May 13 16:31:20 2020 +0200

    Update documentation and remove TLS configuration options

    Change-Id: I59b8627f700ca964c85c043529ccdb33973cfb08
    Closes-Bug: #1851474

[Vern Hart (vern)]

Okay, so this is still a bug in glance-simplestreams-sync.

I'm adding ssl_ca and gss is putting the cacert in /etc/glance-simplestreams-sync/cacert.pem and is also adding it to identity.yaml.

Watching the logs I see that it is indeed making a valid ssl connection to keystone but when it turns around and tries to talk to radosgw (the swift endpoint) it's failing with "certificate verify failed".

Looking in glance-simplestreams-sync.py (the cron job) I see that there is a section where it creates the /usr/local/share/ca-certificates/ directory, writes a cert, and runs update-ca-certificates but that code only seems to get executed if rabbit_hosts is in identity.yaml and if rabbit_use_ssl is true.

To work around this, I took advantage of the cert already being in the unit:

  juju run --unit glance-simplestreams-sync/leader "
    sudo cp /etc/glance-simplestreams-sync/cacert.pem \
      /usr/local/share/ca-certificates/glance-simplestreams-sync.crt
    sudo update-ca-certificates --fresh"

[James Page (james-page)]

https://review.opendev.org/736736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736736
Committed: https://git.openstack.org/cgit/openstack/charm-glance-simplestreams-sync/commit/?id=2dfcecf8d2fa76654a6f9773dadef2648c4db110
Submitter: Zuul
Branch: master

commit 2dfcecf8d2fa76654a6f9773dadef2648c4db110
Author: James Page <email address hidden>
Date: Thu Jun 18 15:43:45 2020 +0100

    Ensure ssl_ca is always installed to system certs

    If an encoded SSL CA cert is provided via configuration ensure
    it is installed to the system certificate store.

    Change-Id: I58f4b89996dc734c4253c2bc2237616434c8143c
    Closes-Bug: 1851474

[Alexander Litvinov (alitvinov)]

Bug exists in glance-simplestreams-sync-32, can we backport the fix to the current charm ?
We are using ssl_ca in almost every customer deployment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.05
Review: https://review.opendev.org/738610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738610
Committed: https://git.openstack.org/cgit/openstack/charm-glance-simplestreams-sync/commit/?id=961bd4b4e7dfeee64616c6a55340ffdaad5d2688
Submitter: Zuul
Branch: stable/20.05

commit 961bd4b4e7dfeee64616c6a55340ffdaad5d2688
Author: James Page <email address hidden>
Date: Thu Jun 18 15:43:45 2020 +0100

    Ensure ssl_ca is always installed to system certs

    If an encoded SSL CA cert is provided via configuration ensure
    it is installed to the system certificate store.

    Add focal to list of default images to sync into glance. This
    resolved zaza testing failures and ensures that all LTS releases
    of Ubuntu are sync'ed by default.

    Change-Id: I58f4b89996dc734c4253c2bc2237616434c8143c
    Closes-Bug: 1851474
    (cherry picked from commit 2dfcecf8d2fa76654a6f9773dadef2648c4db110)

[Vern Hart (vern)]

This has been fixed in the latest release of the charm, cs:glance-simplestreams-sync-33.