octavia

[neutron]/endpoint_type is not respected

Bug #2049551 reported by Mohammed Naser on 2024-01-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	octavia	Fix Released	Undecided	Unassigned

Bug Description

For an existing environment that was functional, after upgrading to 2023.2, it can no longer create load balancers.

2024-01-16 21:47:18.081 10 ERROR wsme.api [None req-1768fc21-085d-48e4-95e1-18b19248e6a8 - 4cb4feb4eed947b8a686fb21be17eea0 - - default default] Server-side error: "SSL exception connecting to https://network.199-204-45-49.nip.io/v2.0/extensions/security-group: HTTPSConnectionPool(host='network.199-204-45-49.nip.io', port=443): Max retries exceeded with url: /v2.0/extensions/security-group (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))". Detail:
Traceback (most recent call last):

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)

File "/var/lib/openstack/lib/python3.10/site-packages/eventlet/green/ssl.py", line 446, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)

File "/var/lib/openstack/lib/python3.10/site-packages/eventlet/green/ssl.py", line 140, in __init__
self.do_handshake()

File "/var/lib/openstack/lib/python3.10/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
return self._call_trampolining(

File "/var/lib/openstack/lib/python3.10/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
return func(*a, **kw)

File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
self._sslobj.do_handshake()

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/var/lib/openstack/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(

File "/var/lib/openstack/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='network.199-204-45-49.nip.io', port=443): Max retries exceeded with url: /v2.0/extensions/security-group (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/session.py", line 1014, in _send_request
resp = self.session.request(method, url, **kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)

requests.exceptions.SSLError: HTTPSConnectionPool(host='network.199-204-45-49.nip.io', port=443): Max retries exceeded with url: /v2.0/extensions/security-group (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/var/lib/openstack/lib/python3.10/site-packages/wsmeext/pecan.py", line 82, in callfunction
result = f(self, *args, **kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 453, in post
self._validate_vip_request_object(load_balancer, context=context)

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 293, in _validate_vip_request_object
subnet = validate.subnet_exists(

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/validate.py", line 344, in subnet_exists
network_driver = utils.get_network_driver()

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/common/utils.py", line 66, in get_network_driver
network_driver = stevedore_driver.DriverManager(

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/driver.py", line 54, in __init__
super(DriverManager, self).__init__(

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/named.py", line 78, in __init__
extensions = self._load_plugins(invoke_on_load,

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/extension.py", line 218, in _load_plugins
self._on_load_failure_callback(self, ep, err)

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/extension.py", line 206, in _load_plugins
ext = self._load_one_plugin(ep,

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/named.py", line 156, in _load_one_plugin
return super(NamedExtensionManager, self)._load_one_plugin(

File "/var/lib/openstack/lib/python3.10/site-packages/stevedore/extension.py", line 242, in _load_one_plugin
obj = plugin(*invoke_args, **invoke_kwds)

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 45, in __init__
super().__init__()

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 41, in __init__
self.sec_grp_enabled = self._check_extension_enabled(SEC_GRP_EXT_ALIAS)

File "/var/lib/openstack/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 60, in _check_extension_enabled
if self.network_proxy.find_extension(extension_alias):

File "/var/lib/openstack/lib/python3.10/site-packages/openstack/network/v2/_proxy.py", line 1170, in find_extension
return self._find(

File "/var/lib/openstack/lib/python3.10/site-packages/openstack/proxy.py", line 500, in _find
return resource_type.find(

File "/var/lib/openstack/lib/python3.10/site-packages/openstack/resource.py", line 2297, in find
return match.fetch(session, microversion=microversion, **params)

File "/var/lib/openstack/lib/python3.10/site-packages/openstack/resource.py", line 1698, in fetch
response = session.get(

File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 395, in get
return self.request(url, 'GET', **kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/openstack/proxy.py", line 190, in request
response = super().request(

File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 257, in request
return self.session.request(url, method, **kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/session.py", line 923, in request
resp = send(**kwargs)

File "/var/lib/openstack/lib/python3.10/site-packages/keystoneauth1/session.py", line 1018, in _send_request
raise exceptions.SSLError(msg)

keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://network.199-204-45-49.nip.io/v2.0/extensions/security-group: HTTPSConnectionPool(host='network.199-204-45-49.nip.io', port=443): Max retries exceeded with url: /v2.0/extensions/security-group (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://network.199-204-45-49.nip.io/v2.0/extensions/security-group: HTTPSConnectionPool(host='network.199-204-45-49.nip.io', port=443): Max retries exceeded with url: /v2.0/extensions/security-group (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

You'll see that it's using the public address even when `[neutron]/endpoint_type` is set to `internalURL`

Revision history for this message

Mohammed Naser (mnaser) wrote on 2024-01-16:

I tried this but it didn't work:

https://review.opendev.org/c/openstack/octavia/+/905794

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-17: Fix proposed to octavia (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/octavia/+/905805

Changed in octavia:
status:	New → In Progress

Revision history for this message

Takashi Kajinami (kajinamit) wrote on 2024-01-17:

I don't have a handly deployment to test octavia now but am wondering if https://review.opendev.org/c/openstack/octavia/+/905805 can solve the problem.

Revision history for this message

Mohammed Naser (mnaser) wrote on 2024-01-17:

Takashi: What do you think about the change that Rico did for this?

https://review.opendev.org/c/openstack/octavia/+/905794

Revision history for this message

Gregory Thiemonge (gthiemonge) wrote on 2024-01-17:

This code should have handled the deprecation of the endpoint_type setting:

https://opendev.org/openstack/octavia/src/commit/5750e4512d622450f6ecab1c7001ebd5637e0d53/octavia/common/config.py#L948-L952

basically, it sets the [neutron]/valid_interfaces setting with the value of [neutron]/endpoint_type (as a list), the update should have been straighforward, I'm going to check what could happen there

Revision history for this message

Noel Ashford (nashford77) wrote on 2024-01-31:

Download full text (5.1 KiB)

I tried this as well - does not work....

openstack.exceptions.ResourceNotFound: No Subnet found for 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf: Client Error for url: https://int.dave.openstack.tunninet.com:9696/v2.0/subnets/1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf, Subnet 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line 82, in callfunction
result = f(self, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 453, in post
self._validate_vip_request_object(load_balancer, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 308, in _validate_vip_request_object
self._validate_subnets_share_network_but_no_duplicates(load_balancer)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 243, in _validate_subnets_share_network_but_no_duplicates
used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 250, in get_subnet
return self._get_resource('subnet', subnet_id, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 197, in _get_resource
raise getattr(base, '%sNotFound' % ''.join(

octavia.network.base.SubnetNotFound: subnet not found (subnet id: 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).
: octavia.network.base.SubnetNotFound: subnet not found (subnet id: 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).

==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [31/Jan/2024:00:15:50 -0500] "POST /v2.0/lbaas/loadbalancers HTTP/1.1" 500 128 1485190 "-" "heat-engine keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"

==> /var/log/kolla/octavia/octavia-api.log <==
2024-01-31 00:15:51.686 734 ERROR wsme.api [None req-580cb5f0-3c34-4ec6-bf6c-5596590d51b1 - e5b9296fbd9e4d9ea5e925780c64690f - - default default] Server-side error: "subnet not found (subnet id: 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).". Detail:
Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 189, in _get_resource
resource = getattr(

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/network/v2/_proxy.py", line 5111, in get_subnet
return self._get(_subnet.Subnet, subnet)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 61, in check
return method(self, expected, actual, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 665, in _get
return res.fetch(

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1711, in fetch
self._translate_response(response, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1287, in _transla...

I tried this as well - does not work....

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line 82, in callfunction
    result = f(self, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 453, in post
    self._validate_vip_request_object(load_balancer, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 308, in _validate_vip_request_object
    self._validate_subnets_share_network_but_no_duplicates(load_balancer)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 243, in _validate_subnets_share_network_but_no_duplicates
    used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 250, in get_subnet
    return self._get_resource('subnet', subnet_id, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 197, in _get_resource
    raise getattr(base, '%sNotFound' % ''.join(

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 189, in _get_resource
    resource = getattr(

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/network/v2/_proxy.py", line 5111, in get_subnet
    return self._get(_subnet.Subnet, subnet)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 61, in check
    return method(self, expected, actual, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 665, in _get
    return res.fetch(

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1711, in fetch
    self._translate_response(response, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1287, in _translate_response
    exceptions.raise_from_response(response, error_message=error_message)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/exceptions.py", line 250, in raise_from_response
    raise cls(

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line 82, in callfunction
    result = f(self, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 453, in post
    self._validate_vip_request_object(load_balancer, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 308, in _validate_vip_request_object
    self._validate_subnets_share_network_but_no_duplicates(load_balancer)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 243, in _validate_subnets_share_network_but_no_duplicates
    used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 250, in get_subnet
    return self._get_resource('subnet', subnet_id, context=context)

File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 197, in _get_resource
    raise getattr(base, '%sNotFound' % ''.join(

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-08: Fix proposed to octavia (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/octavia/+/912062

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-08: Fix merged to octavia (master)

Reviewed: https://review.opendev.org/c/openstack/octavia/+/905794
Committed: https://opendev.org/openstack/octavia/commit/7352dc8f1eca62870f5e5a351e3cfcbccc3f3260
Submitter: "Zuul (22348)"
Branch: master

commit 7352dc8f1eca62870f5e5a351e3cfcbccc3f3260
Author: Mohammed Naser <email address hidden>
Date: Tue Jan 16 17:13:19 2024 -0500

fix: specify endpoint info. for neutron client

Closes bug: #2049551

Change-Id: I80a266e500958415a70d462ddfe57e9e03e6ef13

Changed in octavia:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-14: Fix included in openstack/octavia 14.0.0.0rc1

This issue was fixed in the openstack/octavia 14.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-03: Fix merged to octavia (stable/2023.2)

#10

Reviewed: https://review.opendev.org/c/openstack/octavia/+/912062
Committed: https://opendev.org/openstack/octavia/commit/c664c865b8dfbb5065bbb34c8adb6ed1df5ce28d
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit c664c865b8dfbb5065bbb34c8adb6ed1df5ce28d
Author: Mohammed Naser <email address hidden>
Date: Tue Jan 16 17:13:19 2024 -0500

fix: specify endpoint info. for neutron client

Closes bug: #2049551

Change-Id: I80a266e500958415a70d462ddfe57e9e03e6ef13

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.