Did any other sections like NOVA etc change as well? Should all instances
of endpoint_type be removed from octavia or just Neutron?

I updated all sections to use the new monikers in one attempt and only
[neutron] updated in the second attempt.

As it complained about the below, I tried to update [glance], [nova]
[cinder] and still it gives me the same ?

It got further but seemingly broke on the other services it needs
regardless of the above approaches.

[image: image.png]



==> /var/log/kolla/octavia/octavia-worker.log <==
2024-02-01 12:12:56.958 731 WARNING octavia.common.base_taskflow [-] Task
'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait'
(1747f037-6b94-448b-98f5-e2fad599f03d) transitioned into state 'REVERTED'
from state 'REVERTING'
2024-02-01 12:12:57.134 731 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
instance. Retrying.: requests.exceptions.ConnectionError:
HTTPSConnectionPool(host='172.16.5.129', port=9443): Max retries exceeded
with url: // (Caused by
NewConnectionError('<urllib3.connection.HTTPSConnection object at
0x7f1b9057fa60>: Failed to establish a new connection: [Errno 111]
Connection refused'))

==> /var/log/kolla/octavia/octavia-api-access.log <==
2024-02-01 12:12:32.207 733 WARNING openstack [None
req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'block-storage': Encountered an
exception attempting to process config for project 'cinder' (service type
'block-storage'): no such option valid_interfaces in group [cinder]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[cinder]
2024-02-01 12:12:32.208 733 WARNING openstack [None
req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'compute': Encountered an exception
attempting to process config for project 'nova' (service type 'compute'):
no such option valid_interfaces in group [nova]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[nova]
2024-02-01 12:12:32.208 733 WARNING openstack [None
req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'image': Encountered an exception
attempting to process config for project 'glance' (service type 'image'):
no such option valid_interfaces in group [glance]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[glance]
2024-02-01 12:12:32.242 736 WARNING openstack [None
req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'block-storage': Encountered an
exception attempting to process config for project 'cinder' (service type
'block-storage'): no such option valid_interfaces in group [cinder]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[cinder]
2024-02-01 12:12:32.242 736 WARNING openstack [None
req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'compute': Encountered an exception
attempting to process config for project 'nova' (service type 'compute'):
no such option valid_interfaces in group [nova]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[nova]
2024-02-01 12:12:32.242 736 WARNING openstack [None
req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'image': Encountered an exception
attempting to process config for project 'glance' (service type 'image'):
no such option valid_interfaces in group [glance]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[glance]
2024-02-01 12:12:32.917 733 INFO
octavia.api.drivers.amphora_driver.v2.driver [None
req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Amphora provider created VIP port
94aa1bdc-a3f5-422b-a4af-7a72a2696175 for load balancer
127ec943-3ac6-46c0-b9ec-eca28aafaed4.
2024-02-01 12:12:32.921 733 INFO octavia.api.v2.controllers.load_balancer
[None req-1ee9dfa6-a33d-4550-951a-d05260daa944 -
e5b9296fbd9e4d9ea5e925780c64690f - - default default] Sending create Load
Balancer 127ec943-3ac6-46c0-b9ec-eca28aafaed4 to provider amphora


Additional Info


==> /var/log/kolla/octavia/octavia-api.log <==
2024-02-01 12:14:11.449 736 WARNING openstack [None
req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'block-storage': Encountered an
exception attempting to process config for project 'cinder' (service type
'block-storage'): no such option valid_interfaces in group [cinder]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[cinder]
2024-02-01 12:14:11.450 736 WARNING openstack [None
req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'compute': Encountered an exception
attempting to process config for project 'nova' (service type 'compute'):
no such option valid_interfaces in group [nova]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[nova]
2024-02-01 12:14:11.451 736 WARNING openstack [None
req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'image': Encountered an exception
attempting to process config for project 'glance' (service type 'image'):
no such option valid_interfaces in group [glance]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[glance]
2024-02-01 12:14:11.625 736 INFO octavia.api.v2.controllers.member [None
req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Sending create Member
b385a83d-8f36-4a0d-b801-199351b3172b to provider amphora

==> /var/log/kolla/octavia/octavia-worker.log <==
2024-02-01 12:14:11.684 731 INFO octavia.controller.queue.v2.endpoints [-]
Creating member 'b385a83d-8f36-4a0d-b801-199351b3172b'...

==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [01/Feb/2024:12:14:11 -0500] "POST
/v2.0/lbaas/pools/b1d75929-cfdc-4f7e-92b2-f5b38ef5be4c/members HTTP/1.1"
201 513 619189 "-" "heat-engine keystoneauth1/5.3.0 python-requests/2.28.2
CPython/3.10.12"
192.168.5.99 - - [01/Feb/2024:12:14:11 -0500] "GET
/v2.0/lbaas/pools/b1d75929-cfdc-4f7e-92b2-f5b38ef5be4c/members/b385a83d-8f36-4a0d-b801-199351b3172b
HTTP/1.1" 200 513 95692 "-" "heat-engine keystoneauth1/5.3.0
python-requests/2.28.2 CPython/3.10.12"

==> /var/log/kolla/octavia/octavia-api.log <==
2024-02-01 12:14:12.309 735 WARNING openstack [None
req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'block-storage': Encountered an
exception attempting to process config for project 'cinder' (service type
'block-storage'): no such option valid_interfaces in group [cinder]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[cinder]
2024-02-01 12:14:12.310 735 WARNING openstack [None
req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'compute': Encountered an exception
attempting to process config for project 'nova' (service type 'compute'):
no such option valid_interfaces in group [nova]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[nova]
2024-02-01 12:14:12.311 735 WARNING openstack [None
req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Disabling service 'image': Encountered an exception
attempting to process config for project 'glance' (service type 'image'):
no such option valid_interfaces in group [glance]:
oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
[glance]
2024-02-01 12:14:12.495 735 INFO octavia.api.v2.controllers.member [None
req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
- - default default] Sending create Member
7cd72bca-c140-4ab0-a081-7c8db5b94033 to provider amphora


ᐧ

On Thu, Feb 1, 2024 at 4:08 AM Gregory Thiemonge <email address hidden>
wrote:

> --
>
> [neutron]
> region_name = TN_DEV_NY_5_NET
> endpoint_type = internal
> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
> endpoint = https://int.noel.openstack.tunninet.com:9696
>
> --
>
> In 2023.2, we have deprecated the `endpoint_type` and `endpoint`
> parameters in favor of `valid_interfaces` and `endpoint_override`.
> we added some code to convert automatically the "old" settings to the new
> ones, but I think there's a bug (I'm reproducing a similar issue in my env).
>
> I found a possible workaround, if you rename the keys of the settings
> with the new names (only in the neutron section), it should work
> correctly:
>
> [neutron]
> region_name = TN_DEV_NY_5_NET
> valid_interfaces = internal
> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
> endpoint_override = https://int.noel.openstack.tunninet.com:9696
>
> can you try that?
>
> On my side, I'm working on a bugfix.
>
> ** Changed in: octavia
>    Importance: Undecided => High
>
> ** Changed in: octavia
>        Status: New => Confirmed
>
> ** Changed in: octavia
>      Assignee: (unassigned) => Gregory Thiemonge (gthiemonge)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2051604
>
> Title:
>   Magnum -> Octavia, Neutron driver - Not using region or availability
>   zone to search, rather picking a random neutron endpoint causing
>   lookup failures on subnets
>
> Status in octavia:
>   Confirmed
>
> Bug description:
>   t is lookup up on the wrong endpoint... I check endpoints, no errors
>   there - seems there is no filter to tell it what to look at.... 2023.2
>   is my version. I never saw this issue prior to 2023.x & amphorav2
>
>   Trigger = Call Magnum COE create, it creates a LB, looks up the subnet
>   and chooses the wrong endpoint when the LB is being made in the
>   neutron driver for octavia....
>
>   2024-01-29 17:15:55.984 733 ERROR wsme.api [None
> req-026ef7fd-67aa-412a-89e1-a7529c45e72c - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Server-side error: "subnet not found (subnet id:
> 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).". Detail:
>   Traceback (most recent call last):
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
> line 189, in _get_resource
>       resource = getattr(
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/network/v2/_proxy.py",
> line 5111, in get_subnet
>       return self._get(_subnet.Subnet, subnet)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line
> 61, in check
>       return method(self, expected, actual, *args, **kwargs)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line
> 665, in _get
>       return res.fetch(
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py",
> line 1711, in fetch
>       self._translate_response(response, **kwargs)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py",
> line 1287, in _translate_response
>       exceptions.raise_from_response(response, error_message=error_message)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/exceptions.py",
> line 250, in raise_from_response
>       raise cls(
>
>   openstack.exceptions.ResourceNotFound: No Subnet found for
>   1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf: Client Error for url:
>
> https://int.dave.openstack.tunninet.com:9696/v2.0/subnets/1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf
> ,
>   Subnet 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf could not be found.
>
>   The above exception was the direct cause of the following exception:
>
>   Traceback (most recent call last):
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line
> 82, in callfunction
>       result = f(self, *args, **kwargs)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
> line 453, in post
>       self._validate_vip_request_object(load_balancer, context=context)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
> line 308, in _validate_vip_request_object
>       self._validate_subnets_share_network_but_no_duplicates(load_balancer)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
> line 243, in _validate_subnets_share_network_but_no_duplicates
>       used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
> line 250, in get_subnet
>       return self._get_resource('subnet', subnet_id, context=context)
>
>     File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
> line 197, in _get_resource
>       raise getattr(base, '%sNotFound' % ''.join(
>
>
>
> https://docs.openstack.org/octavia/latest/_modules/octavia/network/drivers/neutron/base.html#BaseNeutronDriver.get_subnet
>
>   def _get_resources_by_filters(self, resource_type, unique_item=False,
>                                     **filters):
>           """Retrieves item(s) from filters. By default, a list is
> returned.
>
>           If unique_item set to True, only the first resource is returned.
>           """
>           try:
>               resources = getattr(
>                   self.network_proxy, f"{resource_type}s")(**filters)
>               conversion_function = getattr(
>                   utils,
>                   'convert_%s_to_model' % resource_type)
>               try:
>                   # get first item to see if there is at least one resource
>                   res_list = [conversion_function(next(resources))]
>               except StopIteration:
>                   # pylint: disable=raise-missing-from
>                   raise os_exceptions.NotFoundException(
>                       f'No resource of type {resource_type} found that
> matches '
>                       f'given filter criteria: {filters}.')
>
>               if unique_item:
>                   return res_list[0]
>               return res_list + [conversion_function(r) for r in resources]
>
>           except os_exceptions.NotFoundException as e:
>               message = _('{resource_type} not found '
>                           '({resource_type} Filters: {filters}.').format(
>                   resource_type=resource_type, filters=filters)
>               raise getattr(base, '%sNotFound' % ''.join(
>                   [w.capitalize() for w in resource_type.split('_')]
>               ))(message) from e
>           except Exception as e:
>               message = _('Error retrieving {resource_type} '
>                           '({resource_type} Filters: {filters}.').format(
>                   resource_type=resource_type, filters=filters)
>               LOG.exception(message)
>               raise base.NetworkException(message) from e
>
>   [docs]
>       def get_network(self, network_id, context=None):
>           return self._get_resource('network', network_id, context=context)
>
>   [docs]
>       def get_subnet(self, subnet_id, context=None):
>           return self._get_resource('subnet', subnet_id, context=context)
>
>   Somewhere, it is not using context to create the correct filter.... I
>   am dead in the water on octavia usage without this, its also impacting
>   magnum as i using octavia for this ;0 Thoughts ?
>
>   cross posting here: https://bugs.launchpad.net/kolla-
>   ansible/+bug/2051602
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/octavia/+bug/2051604/+subscriptions
>
>