**Bug Report**
What happened:
Deploying a new 2023.2 environment with Octavia, I am unable to deploy a minimal load balancer. When deploying via the UI, an Unexpected Error is seen and in the log, the following is found:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base [None req-83ac7a07-8914-4a97-bc88-3474d355519e - 79729fc3a96846689ffd35f38db66a03 - - default default] Error retrieving subnet (subnet id: 5a11f8ef-00eb-49a6-b8ca-1bdb0c154440.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.32.24.220:9696/v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base httplib_response = self._make_request(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self._validate_conn(conn)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base conn.connect()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connection.py", line 419, in connect
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self.sock = ssl_wrap_socket(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return ssl_context.wrap_socket(sock)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 446, in wrap_socket
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return GreenSSLSocket(sock, *a, _context=self, **kw)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 140, in __init__
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self.do_handshake()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self._call_trampolining(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return func(*a, **kw)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self._sslobj.do_handshake()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/adapters.py", line 489, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = conn.urlopen(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base retries = retries.increment(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 1014, in _send_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = self.session.request(method, url, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/sessions.py", line 587, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = self.send(prep, **send_kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/sessions.py", line 701, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base r = adapter.send(request, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/adapters.py", line 563, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise SSLError(e, request=request)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base requests.exceptions.SSLError: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/octavia/network/drivers/neutron/base.py", line 189, in _get_resource
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resource = getattr(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/network/v2/_proxy.py", line 5111, in get_subnet
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self._get(_subnet.Subnet, subnet)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 61, in check
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return method(self, expected, actual, *args, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 665, in _get
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return res.fetch(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1698, in fetch
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base response = session.get(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self.request(url, 'GET', **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 190, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base response = super().request(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self.session.request(url, method, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 923, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = send(**kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 1018, in _send_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise exceptions.SSLError(msg)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.32.24.220:9696/v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
I have automation that is deploying these environments and the 2023.1 version I deploy does not have this issue. In 2023.2, I disabled jobboard with enable_octavia_jobboard: "no" but I've also tried keeping it enabled and enabling Redit (enable_redis: "yes") but the same was still seen. Job board is the latest change I encountered, so I bring it up in case that is related.
This is a self-signed certificate being used, since these are non-production environments. The SSL verification is not an issue in 2023.1 but just 2023.2 so far.
What you expected to happen:
Deploy a load balancer without an error. The self-sign certificate should validate properly, as it should be added to all the containers.
How to reproduce it (minimal and precise):
- Deploy the base Kolla-Ansible and add in Octavia (globals.yml is below)
- Select the admin project if needed
- Navigate to Project > Network > Load Balancers
- Create a new load balancer. I cannot create balancer with pool members and monitoring or without them. I've only been filling in the requirements but no doing pool member or monitoring to save time.
**Environment**:
* OS (e.g. from /etc/os-release):
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
* Kernel (e.g. `uname -a`): 5.14.0-284.25.1.el9_2.x86_64
* Docker version if applicable (e.g. `docker version`): 24.0.7
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): stable/2023.2
* Are you using official images from Docker Hub or self built? Official
* Share your inventory file, globals.yml and other configuration files if relevant
---
kolla_base_distro: "rocky"
network_interface: "eth0"
neutron_external_interface: "eth1"
enable_haproxy: "yes"
kolla_internal_vip_address: "10.32.24.220"
nova_compute_virt_type: "kvm"
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"
kolla_enable_tls_backend: "yes"
kolla_verify_tls_backend: "no"
# For Rocky/EL
openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt"
workaround_ansible_issue_8743: yes
enable_octavia: "yes"
enable_neutron_provider_networks: "yes"
# Octavia requires Redis for the jobboard being enabled, which is default as of Kolla-Ansible 2023.2
# Alternatively, the octavia jobboard can be disabled in the globals.yml
# https://docs.openstack.org/octavia/latest/install/install-amphorav2.html
enable_octavia_jobboard: "no"
# enable_redis: "yes"
octavia_amp_network:
name: lb-mgmt-net
provider_network_type: vlan
provider_segmentation_id: 3
provider_physical_network: physnet1
external: true
shared: false
subnet:
name: lb-mgmt-subnet
cidr: "10.32.32.0/22"
allocation_pool_start: "10.32.34.30"
allocation_pool_end: "10.32.34.34"
gateway_ip: "10.32.32.1"
enable_dhcp: yes
I have the same issue. Works fine without self-signed certificates.
I think this is because of the commit: https:/ /opendev. org/openstack/ octavia/ commit/ 3e6fd13bfb00078 5d5eb9efe2139ee 739e074460# diff-cb0e7091c8 b7bce29aa22c0dc 66ea3548bcfa5f6
I have made a patch for a quick workaround. See the attachment.