kolla-ansible

Octavia LB Create in 2023.2 Throws CERTIFICATE_VERIFY_FAILED

Bug #2046382 reported by Korey on 2023-12-13

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Invalid	Undecided	Unassigned
	octavia	New	Undecided	Unassigned

Bug Description

**Bug Report**

What happened:
Deploying a new 2023.2 environment with Octavia, I am unable to deploy a minimal load balancer. When deploying via the UI, an Unexpected Error is seen and in the log, the following is found:

2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base [None req-83ac7a07-8914-4a97-bc88-3474d355519e - 79729fc3a96846689ffd35f38db66a03 - - default default] Error retrieving subnet (subnet id: 5a11f8ef-00eb-49a6-b8ca-1bdb0c154440.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.32.24.220:9696/v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base httplib_response = self._make_request(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self._validate_conn(conn)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base conn.connect()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connection.py", line 419, in connect
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self.sock = ssl_wrap_socket(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return ssl_context.wrap_socket(sock)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 446, in wrap_socket
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return GreenSSLSocket(sock, *a, _context=self, **kw)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 140, in __init__
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self.do_handshake()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self._call_trampolining(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return func(*a, **kw)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base self._sslobj.do_handshake()
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/adapters.py", line 489, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = conn.urlopen(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base retries = retries.increment(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 1014, in _send_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = self.session.request(method, url, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/sessions.py", line 587, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = self.send(prep, **send_kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/sessions.py", line 701, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base r = adapter.send(request, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/requests/adapters.py", line 563, in send
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise SSLError(e, request=request)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base requests.exceptions.SSLError: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred:
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last):
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/octavia/network/drivers/neutron/base.py", line 189, in _get_resource
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resource = getattr(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/network/v2/_proxy.py", line 5111, in get_subnet
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self._get(_subnet.Subnet, subnet)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 61, in check
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return method(self, expected, actual, *args, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 665, in _get
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return res.fetch(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1698, in fetch
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base response = session.get(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self.request(url, 'GET', **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py", line 190, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base response = super().request(
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/adapter.py", line 257, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base return self.session.request(url, method, **kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 923, in request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base resp = send(**kwargs)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base File "/var/lib/kolla/venv/lib64/python3.9/site-packages/keystoneauth1/session.py", line 1018, in _send_request
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base raise exceptions.SSLError(msg)
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.32.24.220:9696/v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440: HTTPSConnectionPool(host='10.32.24.220', port=9696): Max retries exceeded with url: /v2.0/subnets/5a11f8ef-00eb-49a6-b8ca-1bdb0c154440 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
2023-12-13 19:50:00.485 41 ERROR octavia.network.drivers.neutron.base

I have automation that is deploying these environments and the 2023.1 version I deploy does not have this issue. In 2023.2, I disabled jobboard with enable_octavia_jobboard: "no" but I've also tried keeping it enabled and enabling Redit (enable_redis: "yes") but the same was still seen. Job board is the latest change I encountered, so I bring it up in case that is related.

This is a self-signed certificate being used, since these are non-production environments. The SSL verification is not an issue in 2023.1 but just 2023.2 so far.

What you expected to happen:

Deploy a load balancer without an error. The self-sign certificate should validate properly, as it should be added to all the containers.

How to reproduce it (minimal and precise):

- Deploy the base Kolla-Ansible and add in Octavia (globals.yml is below)
- Select the admin project if needed
- Navigate to Project > Network > Load Balancers
- Create a new load balancer. I cannot create balancer with pool members and monitoring or without them. I've only been filling in the requirements but no doing pool member or monitoring to save time.

**Environment**:
* OS (e.g. from /etc/os-release):
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
* Kernel (e.g. `uname -a`): 5.14.0-284.25.1.el9_2.x86_64
* Docker version if applicable (e.g. `docker version`): 24.0.7
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): stable/2023.2
* Are you using official images from Docker Hub or self built? Official
* Share your inventory file, globals.yml and other configuration files if relevant

---
kolla_base_distro: "rocky"
network_interface: "eth0"
neutron_external_interface: "eth1"
enable_haproxy: "yes"
kolla_internal_vip_address: "10.32.24.220"
nova_compute_virt_type: "kvm"
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"
kolla_enable_tls_backend: "yes"
kolla_verify_tls_backend: "no"
# For Rocky/EL
openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt"
workaround_ansible_issue_8743: yes

enable_octavia: "yes"
enable_neutron_provider_networks: "yes"

# Octavia requires Redis for the jobboard being enabled, which is default as of Kolla-Ansible 2023.2
# Alternatively, the octavia jobboard can be disabled in the globals.yml
# https://docs.openstack.org/octavia/latest/install/install-amphorav2.html
enable_octavia_jobboard: "no"
# enable_redis: "yes"

octavia_amp_network:
    name: lb-mgmt-net
    provider_network_type: vlan
    provider_segmentation_id: 3
    provider_physical_network: physnet1
    external: true
    shared: false
    subnet:
        name: lb-mgmt-subnet
        cidr: "10.32.32.0/22"
        allocation_pool_start: "10.32.34.30"
        allocation_pool_end: "10.32.34.34"
        gateway_ip: "10.32.32.1"
        enable_dhcp: yes

Revision history for this message

Ilia Kerbs (ikerbs) wrote on 2023-12-31:

octavia.patch Edit (1.0 KiB, text/plain)

I have the same issue. Works fine without self-signed certificates.

I think this is because of the commit: https://opendev.org/openstack/octavia/commit/3e6fd13bfb000785d5eb9efe2139ee739e074460#diff-cb0e7091c8b7bce29aa22c0dc66ea3548bcfa5f6

I have made a patch for a quick workaround. See the attachment.

Revision history for this message

Korey (entizer) wrote on 2024-01-10:

Thanks @ikerbs.

What is the best way to apply this patch either before or after the deployment with Kolla-Ansible?

Revision history for this message

Ilia Kerbs (ikerbs) wrote on 2024-01-19:

Hello, @entizer

As of now, I had:
1. Forked openstack/octavia repository
2. Applied the patch
3. Built Octavia containers from the fork (edited kolla-build.conf, section [octavia-base])
4. Deployed custom containers

Revision history for this message

Korey (entizer) wrote on 2024-01-19:

Thanks @ikerbs! I thought that might be the method but wanted to make sure there was not a simpler way with kolla-ansible applying it. Much appreciated!

Revision history for this message

Wodel Youchi (wodel) wrote on 2024-03-11:

Download full text (5.8 KiB)

Hi,

I am experiencing the same problem, I tried to rebuild the octavia image using the patch, but the build didn't work, could you help me with more detailed steps?

This is what I did :
1. I created a local registry
2. cloned octavia source code from git version 2023.2
3. applied the patch on this file : octavia/kolla-src/octavia/common/clients.py
4. I created a virtual venv with podman as engine
5. I created kolla-build.conf like this at first
[DEFAULT]
base = rocky
namespace = kolla
push = true
registry = 192.168.2.35:4000

[octavia] <--------------------- I used this first
type = local
location = /root/work/octavia

6. I started the build like this :
kolla-build.py --config-file ./kolla_build.conf --engine podman -d --docker-healthchecks --logs-dir /root/work/logs --openstack-release 2023.2 octavia

Result : The octavia images were built, but not from the local source code but from the Internet. And further, verifying the octavia-api image for example did show that the client.py file does contain the patch.

I modified the kolla-build.conf like this :
[DEFAULT]
base = rocky
namespace = kolla
push = true
registry = 192.168.2.35:4000

[octavia-base] <--------------------- Then I used this
type = local
location = /root/work/octavia

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [19 lines of output] ...

Hi,

I am experiencing the same problem, I tried to rebuild the octavia image using the patch, but the build didn't work, could you help me with more detailed steps?

[octavia]  <--------------------- I used this first
type = local
location = /root/work/octavia

6. I started the build like this :
kolla-build.py --config-file ./kolla_build.conf --engine podman -d --docker-healthchecks --logs-dir /root/work/logs --openstack-release 2023.2  octavia

I modified the kolla-build.conf like this : 
[DEFAULT]
base = rocky
namespace = kolla
push = true
registry = 192.168.2.35:4000

[octavia-base]  <--------------------- Then I used this
type = local
location = /root/work/octavia

But this time the images did not get built correctly and I got many errors and I am not a developer
ia     && mkdir -p /etc/octavia     && cp -r /octavia/etc/* /etc/octavia/     && touch /usr/local/bin/kolla_octavia_extend_start     && chmod 644 /usr/local/bin/kolla_extend_start /usr/loca
l/bin/kolla_octavia_extend_start                                                                                                                                                              
ERROR:kolla.common.utils.octavia-base:Processing /octavia                                                                                                                                     
ERROR:kolla.common.utils.octavia-base:Preparing metadata (setup.py): started                                                                                                                  
ERROR:kolla.common.utils.octavia-base:Preparing metadata (setup.py): finished with status 'error'                                                                                             
ERROR:kolla.common.utils.octavia-base:error: subprocess-exited-with-error                                                                                                                     
                                                                                                                                                                                             
 × python setup.py egg_info did not run successfully.                                                                                                                                       
 │ exit code: 1                                                                                                                                                                              
 ╰─> [19 lines of output]                                                                                                                                                                    
     [pbr] Generating ChangeLog                                                                                                                                                              
     /var/lib/kolla/venv/lib64/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and othe
r standards-based tools.                                                                                                                                                                      
       warnings.warn(                                                                                                                                                                        
     /var/lib/kolla/venv/lib64/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards
-based tools.                                                                                                                                                                                 
       warnings.warn(                                                                                                                                                                        
     error: Multiple top-level packages discovered in a flat-layout: ['etc', 'httpd', 'specs', 'devstack', 'elements', 'playbooks', 'releasenotes'].                                         
                                                                                                                                                                                             
     To avoid accidental inclusion of unwanted files or directories,                                                                                                                         
     setuptools will not proceed with this build.
       
     If you are trying to create a single distribution with multiple packages
     on purpose, you should not rely on automatic discovery.
     Instead, consider the following options:
       
     1. set up custom discovery (`find` directive with `include` or `exclude`)
     2. use a `src-layout`
     3. explicitly set `py_modules` or `packages` with a list of names
       
     To find more information, look for "package discovery" on setuptools docs.
     [end of output]
   
 note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR:kolla.common.utils.octavia-base:error: metadata-generation-failed

I want to learn how to patch and rebuild an image in case of a problem. Any help will be appreciated.

Regards.

Revision history for this message

Dave (pagerlemon) wrote on 2024-04-06:

I have the same issue and I can't figure out how to apply the patch.

Can someone please help with some more detailed steps?

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2024-04-08:

route fix to the Octavia project.

Changed in kolla-ansible:
status:	New → Invalid

Revision history for this message

Gregory Thiemonge (gthiemonge) wrote on 2024-04-08:

Thanks for reporting it,

Can you share the config files of the octavia services?

Revision history for this message

mohamed mira (momo91) wrote on 2024-04-11:

octavia.conf Edit (3.9 KiB, text/plain)

I have the same issue, you find attached my octavia.conf file.

Revision history for this message

Korey (entizer) wrote on 2024-04-11:

#10

octavia.conf Edit (2.9 KiB, text/plain)

Here is my octavia.conf as well, with removed secrets. This is from Bobcat (2023.2).

Revision history for this message

Korey (entizer) wrote on 2024-04-11 (last edit on 2024-04-11):

#11

To add to the thread, others were asking how to implement the patch that @ikerbs mentioned. They were kind enough to send me a message with details, which I'll place below if it helps while we work through the issue. That said, I've been on other projects and not tested it myself yet. Thanks @ikerbs!

=====

As I have described previously:
1. Forked openstack/octavia repository
2. Applied the patch
3. Built Octavia containers from the fork (edited kolla-build.conf, section [octavia-base])
4. Deployed custom containers

I found that this is the most convenient way to apply custom patches until upstream is fixed by Octavia devs.
Octavia devs may implement this in another way - my patch is just a straightforward solution.
You can do this with an existing or new environment/installation.

As far as I understand the problem is that the current version of the code is missing self-signed certs support.
So you need to modify the code in the Octavia container and fix it. That leads to custom container builds.

Another simple solution - buy a valid certificate and pass it to Kolla-Ansible on deployment. Or acquire a free valid certificate from Lets Encrypt - Kolla-Ansible has support for this. But I never tried this feature.

If you have no other option but self-signed certificates, then as an alternative way, you can modify the kolla octavia-base Dockerfile and apply my patch directly on the container build.

To do this you need to clone Kolla and make a custom virtualenv:

git clone https://github.com/openstack/kolla
cd kolla && git checkout 17.1.0
mkvirtualenv kolla-17.1.0
pip install -e .
pip install tox==3.28.0 docker==6.1.3
tox -e genconfig

Then modify etc/kolla/kolla-build.conf and add:
[DEFAULT]
registry = YOUR_CONTAINER_REGISTRY

Then copy the patch to the docker/octavia/octavia-base directory and add lines to the docker/octavia/octavia-base/Dockerfile.j2:

COPY octavia.patch /octavia.patch
RUN cd /var/lib/kolla/venv/lib/python${KOLLA_DISTRO_PYTHON_VERSION}/site-packages && git apply /octavia.patch

Build custom containers and push them to your private registry with the command:
kolla-build --config-file etc/kolla/kolla-build.conf --push

Then adjust kolla-ansible global.yml config:
docker_registry: "YOUR_CONTAINER_REGISTRY"
docker_registry_insecure: "yes"
openstack_tag: "YOUR_CONTAINERS_TAG"

And deploy custom containers.

With this workflow, you can skip forking openstack/octavia repository.
But anyway you need to build custom containers and deploy them.

Anyway, I think the best practice is to have a container registry for custom containers and not rely on a public container registry.

Hope, this will help.

To add to the thread, others were asking how to implement the patch that @ikerbs mentioned.  They were kind enough to send me a message with details, which I'll place below if it helps while we work through the issue.  That said, I've been on other projects and not tested it myself yet.  Thanks @ikerbs!