Magnum -> Octavia, Neutron driver - Not using region or availability zone to search, rather picking a random neutron endpoint causing lookup failures on subnets

[Noel Ashford]

It is lookup up on the wrong endpoint... I check endpoints, no errors there - seems there is no filter to tell it what to look at.... 2023.2 is my version. I never saw this issue prior to 2023.x

Trigger = Call Magnum COE create, it creates a LB, looks up the subnet and chooses the wrong endpoint when the LB is being made in the neutron driver for octavia....

2024-01-29 17:15:55.984 733 ERROR wsme.api [None req-026ef7fd-67aa-412a-89e1-a7529c45e72c - e5b9296fbd9e4d9ea5e925780c64690f - - default default] Server-side error: "subnet not found (subnet id: 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf).". Detail:
Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 189, in _get_resource
    resource = getattr(

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/network/v2/_proxy.py", line 5111, in get_subnet
    return self._get(_subnet.Subnet, subnet)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 61, in check
    return method(self, expected, actual, *args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/proxy.py", line 665, in _get
    return res.fetch(

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1711, in fetch
    self._translate_response(response, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/resource.py", line 1287, in _translate_response
    exceptions.raise_from_response(response, error_message=error_message)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/openstack/exceptions.py", line 250, in raise_from_response
    raise cls(

openstack.exceptions.ResourceNotFound: No Subnet found for 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf: Client Error for url: https://int.dave.openstack.tunninet.com:9696/v2.0/subnets/1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf, Subnet 1b98d0d6-3eaa-490d-a9a4-e351fb0ebedf could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/wsmeext/pecan.py", line 82, in callfunction
    result = f(self, *args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 453, in post
    self._validate_vip_request_object(load_balancer, context=context)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 308, in _validate_vip_request_object
    self._validate_subnets_share_network_but_no_duplicates(load_balancer)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/api/v2/controllers/load_balancer.py", line 243, in _validate_subnets_share_network_but_no_duplicates
    used_subnets[subnet_id] = network_driver.get_subnet(subnet_id)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 250, in get_subnet
    return self._get_resource('subnet', subnet_id, context=context)

  File "/var/lib/kolla/venv/lib/python3.10/site-packages/octavia/network/drivers/neutron/base.py", line 197, in _get_resource
    raise getattr(base, '%sNotFound' % ''.join(

[Noel Ashford (nashford77)]

https://docs.openstack.org/octavia/latest/_modules/octavia/network/drivers/neutron/base.html#BaseNeutronDriver.get_subnet

def _get_resources_by_filters(self, resource_type, unique_item=False,
                                  **filters):
        """Retrieves item(s) from filters. By default, a list is returned.

        If unique_item set to True, only the first resource is returned.
        """
        try:
            resources = getattr(
                self.network_proxy, f"{resource_type}s")(**filters)
            conversion_function = getattr(
                utils,
                'convert_%s_to_model' % resource_type)
            try:
                # get first item to see if there is at least one resource
                res_list = [conversion_function(next(resources))]
            except StopIteration:
                # pylint: disable=raise-missing-from
                raise os_exceptions.NotFoundException(
                    f'No resource of type {resource_type} found that matches '
                    f'given filter criteria: {filters}.')

            if unique_item:
                return res_list[0]
            return res_list + [conversion_function(r) for r in resources]

        except os_exceptions.NotFoundException as e:
            message = _('{resource_type} not found '
                        '({resource_type} Filters: {filters}.').format(
                resource_type=resource_type, filters=filters)
            raise getattr(base, '%sNotFound' % ''.join(
                [w.capitalize() for w in resource_type.split('_')]
            ))(message) from e
        except Exception as e:
            message = _('Error retrieving {resource_type} '
                        '({resource_type} Filters: {filters}.').format(
                resource_type=resource_type, filters=filters)
            LOG.exception(message)
            raise base.NetworkException(message) from e

[docs]
    def get_network(self, network_id, context=None):
        return self._get_resource('network', network_id, context=context)

[docs]
    def get_subnet(self, subnet_id, context=None):
        return self._get_resource('subnet', subnet_id, context=context)

Somewhere, it is not using context to create the correct filter.... I am dead in the water on octavia usage without this, its also impacting magnum as i using octavia for this ;0 Thoughts ?

[Noel Ashford (nashford77)]

cross posting here: https://bugs.launchpad.net/kolla-ansible/+bug/2051602

[Noel Ashford (nashford77)]

Sorry, wrong URL for cross post: https://bugs.launchpad.net/octavia/+bug/2051604

[Noel Ashford (nashford77)]

Seems to not be honoring region etc. at a high level...