It also worked to standup magnum (which calls octavia) fully by using the
old moniker for all but Neutron (but with the above errors that dont make
sense as i didnt pass them)
ᐧ

On Thu, Feb 1, 2024 at 12:38 PM Noel Ashford <email address hidden> wrote:

> OK, an update ... The above config changing everything worked with the
> above errors tho?!
>
> Here is what happens when i do not update the new moniker for glance,
> cinder, nova etc.... (which makes no sense to me?)
>
> ==> /var/log/kolla/octavia/octavia-api.log <==
> 2024-02-01 12:36:03.842 733 WARNING openstack [None
> req-366da871-ab3b-4e98-9780-0f29a93cfe14 - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'block-storage': Encountered an
> exception attempting to process config for project 'cinder' (service type
> 'block-storage'): no such option valid_interfaces in group [cinder]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [cinder]
> 2024-02-01 12:36:03.843 733 WARNING openstack [None
> req-366da871-ab3b-4e98-9780-0f29a93cfe14 - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'compute': Encountered an exception
> attempting to process config for project 'nova' (service type 'compute'):
> no such option valid_interfaces in group [nova]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [nova]
> 2024-02-01 12:36:03.843 733 WARNING openstack [None
> req-366da871-ab3b-4e98-9780-0f29a93cfe14 - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'image': Encountered an exception
> attempting to process config for project 'glance' (service type 'image'):
> no such option valid_interfaces in group [glance]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [glance]
> 2024-02-01 12:36:03.884 737 WARNING openstack [None
> req-4b65bd1b-3e6d-437d-a879-c08a1eaf225a - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'block-storage': Encountered an
> exception attempting to process config for project 'cinder' (service type
> 'block-storage'): no such option valid_interfaces in group [cinder]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [cinder]
> 2024-02-01 12:36:03.884 737 WARNING openstack [None
> req-4b65bd1b-3e6d-437d-a879-c08a1eaf225a - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'compute': Encountered an exception
> attempting to process config for project 'nova' (service type 'compute'):
> no such option valid_interfaces in group [nova]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [nova]
> 2024-02-01 12:36:03.885 737 WARNING openstack [None
> req-4b65bd1b-3e6d-437d-a879-c08a1eaf225a - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Disabling service 'image': Encountered an exception
> attempting to process config for project 'glance' (service type 'image'):
> no such option valid_interfaces in group [glance]:
> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
> [glance]
> 2024-02-01 12:36:03.994 733 INFO octavia.api.v2.controllers.member [None
> req-366da871-ab3b-4e98-9780-0f29a93cfe14 - e5b9296fbd9e4d9ea5e925780c64690f
> - - default default] Sending create Member
> 4a551e43-42b8-49ea-a989-42a28c1466ef to provider amphora
>
>
> Why when i dont provide this is it saying this?
>
> Unrelated, I also see this:
>
> I also saw this Error in the logs:
>
>
> ==> /var/log/kolla/octavia/octavia-api-error.log <==
> 2024-02-01 12:14:01.370602
> /var/lib/kolla/venv/lib/python3.10/site-packages/octavia/db/repositories.py:1017:
> SAWarning: Coercing Subquery object into a select() for use in IN(); please
> pass a select() construct explicitly
> 2024-02-01 12:14:01.370654
> filter(models.LoadBalancer.id.in_(lb_subquery)).first())
> 2024-02-01 12:14:01.373214
> /var/lib/kolla/venv/lib/python3.10/site-packages/octavia/db/repositories.py:1017:
> SAWarning: SELECT statement has a cartesian product between FROM element(s)
> "pool" and FROM element "load_balancer".  Apply join condition(s) between
> each element to resolve.
> 2024-02-01 12:14:01.373248
> filter(models.LoadBalancer.id.in_(lb_subquery)).first())
>
>
> Also, what about [certificates]?
>
> endpoint_type = internal
> or
> valid_interfaces = internal
> ?
>
>
> ᐧ
>
> On Thu, Feb 1, 2024 at 12:19 PM Noel Ashford <email address hidden> wrote:
>
>> Did any other sections like NOVA etc change as well? Should all instances
>> of endpoint_type be removed from octavia or just Neutron?
>>
>> I updated all sections to use the new monikers in one attempt and only
>> [neutron] updated in the second attempt.
>>
>> As it complained about the below, I tried to update [glance], [nova]
>> [cinder] and still it gives me the same ?
>>
>> It got further but seemingly broke on the other services it needs
>> regardless of the above approaches.
>>
>> [image: image.png]
>>
>>
>>
>> ==> /var/log/kolla/octavia/octavia-worker.log <==
>> 2024-02-01 12:12:56.958 731 WARNING octavia.common.base_taskflow [-] Task
>> 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait'
>> (1747f037-6b94-448b-98f5-e2fad599f03d) transitioned into state 'REVERTED'
>> from state 'REVERTING'
>> 2024-02-01 12:12:57.134 731 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to
>> instance. Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.5.129', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.HTTPSConnection object at
>> 0x7f1b9057fa60>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>>
>> ==> /var/log/kolla/octavia/octavia-api-access.log <==
>> 2024-02-01 12:12:32.207 733 WARNING openstack [None
>> req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'block-storage': Encountered an
>> exception attempting to process config for project 'cinder' (service type
>> 'block-storage'): no such option valid_interfaces in group [cinder]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [cinder]
>> 2024-02-01 12:12:32.208 733 WARNING openstack [None
>> req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'compute': Encountered an exception
>> attempting to process config for project 'nova' (service type 'compute'):
>> no such option valid_interfaces in group [nova]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [nova]
>> 2024-02-01 12:12:32.208 733 WARNING openstack [None
>> req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'image': Encountered an exception
>> attempting to process config for project 'glance' (service type 'image'):
>> no such option valid_interfaces in group [glance]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [glance]
>> 2024-02-01 12:12:32.242 736 WARNING openstack [None
>> req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'block-storage': Encountered an
>> exception attempting to process config for project 'cinder' (service type
>> 'block-storage'): no such option valid_interfaces in group [cinder]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [cinder]
>> 2024-02-01 12:12:32.242 736 WARNING openstack [None
>> req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'compute': Encountered an exception
>> attempting to process config for project 'nova' (service type 'compute'):
>> no such option valid_interfaces in group [nova]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [nova]
>> 2024-02-01 12:12:32.242 736 WARNING openstack [None
>> req-5bb468c8-0f06-4351-ab5f-4b5754d62775 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'image': Encountered an exception
>> attempting to process config for project 'glance' (service type 'image'):
>> no such option valid_interfaces in group [glance]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [glance]
>> 2024-02-01 12:12:32.917 733 INFO
>> octavia.api.drivers.amphora_driver.v2.driver [None
>> req-1ee9dfa6-a33d-4550-951a-d05260daa944 - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Amphora provider created VIP port
>> 94aa1bdc-a3f5-422b-a4af-7a72a2696175 for load balancer
>> 127ec943-3ac6-46c0-b9ec-eca28aafaed4.
>> 2024-02-01 12:12:32.921 733 INFO octavia.api.v2.controllers.load_balancer
>> [None req-1ee9dfa6-a33d-4550-951a-d05260daa944 -
>> e5b9296fbd9e4d9ea5e925780c64690f - - default default] Sending create Load
>> Balancer 127ec943-3ac6-46c0-b9ec-eca28aafaed4 to provider amphora
>>
>>
>> Additional Info
>>
>>
>> ==> /var/log/kolla/octavia/octavia-api.log <==
>> 2024-02-01 12:14:11.449 736 WARNING openstack [None
>> req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'block-storage': Encountered an
>> exception attempting to process config for project 'cinder' (service type
>> 'block-storage'): no such option valid_interfaces in group [cinder]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [cinder]
>> 2024-02-01 12:14:11.450 736 WARNING openstack [None
>> req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'compute': Encountered an exception
>> attempting to process config for project 'nova' (service type 'compute'):
>> no such option valid_interfaces in group [nova]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [nova]
>> 2024-02-01 12:14:11.451 736 WARNING openstack [None
>> req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'image': Encountered an exception
>> attempting to process config for project 'glance' (service type 'image'):
>> no such option valid_interfaces in group [glance]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [glance]
>> 2024-02-01 12:14:11.625 736 INFO octavia.api.v2.controllers.member [None
>> req-3db6d72d-420c-47e0-9420-7ef25282058e - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Sending create Member
>> b385a83d-8f36-4a0d-b801-199351b3172b to provider amphora
>>
>> ==> /var/log/kolla/octavia/octavia-worker.log <==
>> 2024-02-01 12:14:11.684 731 INFO octavia.controller.queue.v2.endpoints
>> [-] Creating member 'b385a83d-8f36-4a0d-b801-199351b3172b'...
>>
>> ==> /var/log/kolla/octavia/octavia-api-access.log <==
>> 192.168.5.99 - - [01/Feb/2024:12:14:11 -0500] "POST
>> /v2.0/lbaas/pools/b1d75929-cfdc-4f7e-92b2-f5b38ef5be4c/members HTTP/1.1"
>> 201 513 619189 "-" "heat-engine keystoneauth1/5.3.0 python-requests/2.28.2
>> CPython/3.10.12"
>> 192.168.5.99 - - [01/Feb/2024:12:14:11 -0500] "GET
>> /v2.0/lbaas/pools/b1d75929-cfdc-4f7e-92b2-f5b38ef5be4c/members/b385a83d-8f36-4a0d-b801-199351b3172b
>> HTTP/1.1" 200 513 95692 "-" "heat-engine keystoneauth1/5.3.0
>> python-requests/2.28.2 CPython/3.10.12"
>>
>> ==> /var/log/kolla/octavia/octavia-api.log <==
>> 2024-02-01 12:14:12.309 735 WARNING openstack [None
>> req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'block-storage': Encountered an
>> exception attempting to process config for project 'cinder' (service type
>> 'block-storage'): no such option valid_interfaces in group [cinder]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [cinder]
>> 2024-02-01 12:14:12.310 735 WARNING openstack [None
>> req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'compute': Encountered an exception
>> attempting to process config for project 'nova' (service type 'compute'):
>> no such option valid_interfaces in group [nova]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [nova]
>> 2024-02-01 12:14:12.311 735 WARNING openstack [None
>> req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Disabling service 'image': Encountered an exception
>> attempting to process config for project 'glance' (service type 'image'):
>> no such option valid_interfaces in group [glance]:
>> oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group
>> [glance]
>> 2024-02-01 12:14:12.495 735 INFO octavia.api.v2.controllers.member [None
>> req-8dfeb88b-3ce9-46e7-bebf-01079cc1819b - e5b9296fbd9e4d9ea5e925780c64690f
>> - - default default] Sending create Member
>> 7cd72bca-c140-4ab0-a081-7c8db5b94033 to provider amphora
>>
>>
>> ᐧ
>>
>> On Thu, Feb 1, 2024 at 4:08 AM Gregory Thiemonge <
>> <email address hidden>> wrote:
>>
>>> --
>>>
>>> [neutron]
>>> region_name = TN_DEV_NY_5_NET
>>> endpoint_type = internal
>>> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
>>> endpoint = https://int.noel.openstack.tunninet.com:9696
>>>
>>> --
>>>
>>> In 2023.2, we have deprecated the `endpoint_type` and `endpoint`
>>> parameters in favor of `valid_interfaces` and `endpoint_override`.
>>> we added some code to convert automatically the "old" settings to the
>>> new ones, but I think there's a bug (I'm reproducing a similar issue in my
>>> env).
>>>
>>> I found a possible workaround, if you rename the keys of the settings
>>> with the new names (only in the neutron section), it should work
>>> correctly:
>>>
>>> [neutron]
>>> region_name = TN_DEV_NY_5_NET
>>> valid_interfaces = internal
>>> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
>>> endpoint_override = https://int.noel.openstack.tunninet.com:9696
>>>
>>> can you try that?
>>>
>>> On my side, I'm working on a bugfix.
>>>
>>> ** Changed in: octavia
>>>    Importance: Undecided => High
>>>
>>> ** Changed in: octavia
>>>        Status: New => Confirmed
>>>
>>> ** Changed in: octavia
>>>      Assignee: (unassigned) => Gregory Thiemonge (gthiemonge)
>>>
>>> --
>>> You received this bug notification because you are subscribed to the bug
>>> report.
>>> https://bugs.launchpad.net/bugs/2051604
>>>
>>> Title:
>>>   Magnum -> Octavia, Neutron driver - Not using region or availability
>>>   zone to search, rather picking a random neutron endpoint causing
>>>   lookup failures on subnets
>>>
>>> Status in octavia:
>>>   Confirmed
>>>
>>> Bug description:
>>>   t is lookup up on the wrong endpoint... I check endpoints, no errors
>>>   there - seems there is no filter to tell it what to look at.... 2023.2
>>>   is my version. I never saw this issue prior to 2023.x & amphorav2
>>>
>>>   Trigger = Call Magnum COE create, it creates a LB, looks up the subnet
>>>   and chooses the wrong endpoint when the LB is being made in the
>>>   neutron driver for octavia....
>>>
>>>   2024-01-29 17:15:55.984 733 ERROR wsme.api [None
>>> req-026ef7fd-67aa-412a-89e1-a7529c45e72c - e5b9296fbd9e4d9ea5e925780c64690f
>>> - - default default] Server-side error: "subnet not found (subnet id:
>>> 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).". Detail:
>>>   Traceback (most recent call last):
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
>>> line 189, in _get_resource
>>>       resource = getattr(
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/network/v2/_proxy.py",
>>> line 5111, in get_subnet
>>>       return self._get(_subnet.Subnet, subnet)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line
>>> 61, in check
>>>       return method(self, expected, actual, *args, **kwargs)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line
>>> 665, in _get
>>>       return res.fetch(
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py",
>>> line 1711, in fetch
>>>       self._translate_response(response, **kwargs)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py",
>>> line 1287, in _translate_response
>>>       exceptions.raise_from_response(response,
>>> error_message=error_message)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/exceptions.py",
>>> line 250, in raise_from_response
>>>       raise cls(
>>>
>>>   openstack.exceptions.ResourceNotFound: No Subnet found for
>>>   1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf: Client Error for url:
>>>
>>> https://int.dave.openstack.tunninet.com:9696/v2.0/subnets/1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf
>>> ,
>>>   Subnet 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf could not be found.
>>>
>>>   The above exception was the direct cause of the following exception:
>>>
>>>   Traceback (most recent call last):
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line
>>> 82, in callfunction
>>>       result = f(self, *args, **kwargs)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
>>> line 453, in post
>>>       self._validate_vip_request_object(load_balancer, context=context)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
>>> line 308, in _validate_vip_request_object
>>>
>>> self._validate_subnets_share_network_but_no_duplicates(load_balancer)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py",
>>> line 243, in _validate_subnets_share_network_but_no_duplicates
>>>       used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
>>> line 250, in get_subnet
>>>       return self._get_resource('subnet', subnet_id, context=context)
>>>
>>>     File
>>> "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py",
>>> line 197, in _get_resource
>>>       raise getattr(base, '%sNotFound' % ''.join(
>>>
>>>
>>>
>>> https://docs.openstack.org/octavia/latest/_modules/octavia/network/drivers/neutron/base.html#BaseNeutronDriver.get_subnet
>>>
>>>   def _get_resources_by_filters(self, resource_type, unique_item=False,
>>>                                     **filters):
>>>           """Retrieves item(s) from filters. By default, a list is
>>> returned.
>>>
>>>           If unique_item set to True, only the first resource is
>>> returned.
>>>           """
>>>           try:
>>>               resources = getattr(
>>>                   self.network_proxy, f"{resource_type}s")(**filters)
>>>               conversion_function = getattr(
>>>                   utils,
>>>                   'convert_%s_to_model' % resource_type)
>>>               try:
>>>                   # get first item to see if there is at least one
>>> resource
>>>                   res_list = [conversion_function(next(resources))]
>>>               except StopIteration:
>>>                   # pylint: disable=raise-missing-from
>>>                   raise os_exceptions.NotFoundException(
>>>                       f'No resource of type {resource_type} found that
>>> matches '
>>>                       f'given filter criteria: {filters}.')
>>>
>>>               if unique_item:
>>>                   return res_list[0]
>>>               return res_list + [conversion_function(r) for r in
>>> resources]
>>>
>>>           except os_exceptions.NotFoundException as e:
>>>               message = _('{resource_type} not found '
>>>                           '({resource_type} Filters: {filters}.').format(
>>>                   resource_type=resource_type, filters=filters)
>>>               raise getattr(base, '%sNotFound' % ''.join(
>>>                   [w.capitalize() for w in resource_type.split('_')]
>>>               ))(message) from e
>>>           except Exception as e:
>>>               message = _('Error retrieving {resource_type} '
>>>                           '({resource_type} Filters: {filters}.').format(
>>>                   resource_type=resource_type, filters=filters)
>>>               LOG.exception(message)
>>>               raise base.NetworkException(message) from e
>>>
>>>   [docs]
>>>       def get_network(self, network_id, context=None):
>>>           return self._get_resource('network', network_id,
>>> context=context)
>>>
>>>   [docs]
>>>       def get_subnet(self, subnet_id, context=None):
>>>           return self._get_resource('subnet', subnet_id, context=context)
>>>
>>>   Somewhere, it is not using context to create the correct filter.... I
>>>   am dead in the water on octavia usage without this, its also impacting
>>>   magnum as i using octavia for this ;0 Thoughts ?
>>>
>>>   cross posting here: https://bugs.launchpad.net/kolla-
>>>   ansible/+bug/2051602
>>>
>>> To manage notifications about this bug go to:
>>> https://bugs.launchpad.net/octavia/+bug/2051604/+subscriptions
>>>
>>>