tripleo

User provided certificate OctaviaClientCert is missing after deployment

Bug #1874712 reported by Gregory Thiemonge on 2020-04-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Gregory Thiemonge	tripleo victoria-1 "tripleo victoria"

Bug Description

Originally reported in https://bugzilla.redhat.com/show_bug.cgi?id=1827578

Description of problem:

I tried to deploy Octavia with my own certificates and keys (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/networking_guide/sec-octavia#config-octavia-certs-keys).
I used the following parameter file (octavia_parameters.yaml):

parameter_defaults:
    OctaviaCaCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----

    OctaviaCaKey: |
        -----BEGIN RSA PRIVATE KEY-----
        <EDITED>
        -----END RSA PRIVATE KEY-----

    OctaviaClientCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----
        -----BEGIN PRIVATE KEY-----
        <EDITED>
        -----END PRIVATE KEY-----

OctaviaCaKeyPassphrase: <EDITED>

OctaviaGenerateCerts: false

Included in my overcloud_deploy.sh script (penultimate line):

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock1.rdu2.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /home/stack/octavia_parameters.yaml \
--log-file overcloud_deployment_90.log

After deployment, OctaviaClientCert (/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/client.pem) is missing on the controllers.

[root@controller-0 ~]# find /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private/cakey.pem
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem

And Octavia services throws exceptions when trying to communicate with an amphora:

2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 329, in execute
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server amp_info = self.amphora_driver.get_info(amphora)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 368, in get_info
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server self._populate_amphora_api_version(amphora)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 105, in _populate_amphora_api_version
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server amphora)['api_version']
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 702, in get_api_version
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server r = self.get(amp, retry_404=False)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 662, in request
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server r = _request(**reqargs)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server return self.request('GET', url, **kwargs)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server resp = self.send(prep, **send_kwargs)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server r = adapter.send(request, **kwargs)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 416, in send
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server self.cert_verify(conn, request.url, verify, cert)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 586, in cert_verify
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server self).cert_verify(conn, url, verify, cert)
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 250, in cert_verify
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server "invalid path: {}".format(conn.cert_file))
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server OSError: Could not find the TLS certificate file, invalid path: /etc/octavia/certs/client.pem
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server

Version-Release number of selected component (if applicable):
train

How reproducible:
100%

Steps to Reproduce:
1. Deploy Octavia with user-provided certificates (using the parameter file in the description)

Actual results:
client.pem is missing on controllers, Octavia cannot configure amphorae.

Expected results:
client.pem should be present on controllers, and Octavia services should be able to communicate with amphorae

Additional info:

Tags:

Gregory Thiemonge (gthiemonge) on 2020-04-24

Changed in tripleo:
assignee:	nobody → Gregory Thiemonge (gthiemonge)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-24: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/722683

Bogdan Dobrelya (bogdando) on 2020-04-27

Changed in tripleo:
milestone:	none → victoria-1
importance:	Undecided → High
tags:	added: train-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-06: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/722683
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=022c8f84254cd402e7cb3d128ef7c5601baafb55
Submitter: Zuul
Branch: master

commit 022c8f84254cd402e7cb3d128ef7c5601baafb55
Author: Gregory Thiemonge <email address hidden>
Date: Fri Apr 24 14:43:51 2020 +0200

Fix missing OctaviaClientCert* parameters

    OctaviaClientCert and OctaviaClientCertFile parameters
    were incorrectly removed in change
    Ia64668f9ef6efc91a05594ca34c35614d338fdb6.

    That breaks the user-provided certificate & key feature in Octavia
    deployments by not copying the client certificate to the controllers,
    Octavia services are then unable to communicate with running amphorae.

This commit restores those parameters.

Change-Id: I42b48a10512ef817203705a201c0b30d8d1bd50b
Closes-Bug: #1874712

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-07: Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726067

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-24: Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/726067
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3214bf08c467f74073a4046ee0bd1a1738468f5e
Submitter: Zuul
Branch: stable/train

commit 3214bf08c467f74073a4046ee0bd1a1738468f5e
Author: Gregory Thiemonge <email address hidden>
Date: Fri Apr 24 14:43:51 2020 +0200

Fix missing OctaviaClientCert* parameters

    OctaviaClientCert and OctaviaClientCertFile parameters
    were incorrectly removed in change
    Ia64668f9ef6efc91a05594ca34c35614d338fdb6.

This commit restores those parameters.

    Change-Id: I42b48a10512ef817203705a201c0b30d8d1bd50b
    Closes-Bug: #1874712
    (cherry picked from commit 022c8f84254cd402e7cb3d128ef7c5601baafb55)