[ovn] neutron-ovn-tempest-slow job fail tests relying on FIP

[yatin]

Example failure:- https://2f4a32f753edcd6fd518-38c49964a79149719549049b602122d6.ssl.cf5.rackcdn.com/906628/1/experimental/neutron-ovn-tempest-slow/1b35fb8/testr_results.html

Fails as:-
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 191, in setUp
    self._deploy_tenant(self.primary_tenant)
  File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 354, in _deploy_tenant
    self._set_access_point(tenant)
  File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 321, in _set_access_point
    self._assign_floating_ips(tenant, server)
  File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 325, in _assign_floating_ips
    floating_ip = self.create_floating_ip(
  File "/opt/stack/tempest/tempest/scenario/manager.py", line 1132, in create_floating_ip
    result = client.create_floatingip(**floatingip_kwargs)
  File "/opt/stack/tempest/tempest/lib/services/network/floating_ips_client.py", line 30, in create_floatingip
    return self.create_resource(uri, post_data)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 62, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 300, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 852, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {'type': 'ExternalGatewayForFloatingIPNotFound', 'message': 'External network 43cd92cd-4957-4770-9945-584e8d4da9e3 is not reachable from subnet e354d5ec-5be0-4536-9348-fd819e3f7464. Therefore, cannot associate Port 63869e58-05b8-4a18-be25-77500966df61 with a Floating IP.', 'detail': ''}

neutron-server trace:-
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn [None req-1674312b-7443-4d41-bd9b-5f69f5173083 tempest-TestNetworkAdvancedServerOps-1277687553 tempest-TestNetworkAdvancedServerOps-1277687553-project-member] Unable to add router interface to lrouter 31bdd6ba-45cf-45bd-aa3d-907a217ce2a3. Interface info: {'id': '31bdd6ba-45cf-45bd-aa3d-907a217ce2a3', 'tenant_id': 'f7ebd951642c4987ad034d6180f81784', 'port_id': 'f8606824-f658-4634-b9ce-2f8e2ce0d1c3', 'network_id': '2f74ef7b-8374-4f36-a4bf-fbf225b87c48', 'subnet_id': 'd61fb992-2d87-4198-82f1-a687609c3e7c', 'subnet_ids': ['d61fb992-2d87-4198-82f1-a687609c3e7c']}: neutron_lib.exceptions.SubnetNotFound: Subnet a283b869-2e54-44da-b109-06da46933d06 could not be found.
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn Traceback (most recent call last):
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/services/ovn_l3/service_providers/ovn.py", line 122, in _process_add_router_interface
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self.l3plugin._ovn_client.create_router_port(context, router.id,
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1754, in create_router_port
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self._update_lrouter_port(context, router_port,
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1762, in _update_lrouter_port
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self._get_nets_and_ipv6_ra_confs_for_router_port(context, port))
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1236, in _get_nets_and_ipv6_ra_confs_for_router_port
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn subnet = self._plugin.get_subnet(context, subnet_id)
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn return f_with_retry(*args, **kwargs,
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn with excutils.save_and_reraise_exception():
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self.force_reraise()
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn raise self.value
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn return f(*args, **kwargs)
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn with excutils.save_and_reraise_exception() as ectxt:
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self.force_reraise()
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn raise self.value
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn return f(*args, **kwargs)
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn with excutils.save_and_reraise_exception():
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self.force_reraise()
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn raise self.value
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn return f(*dup_args, **dup_kwargs)
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1184, in get_subnet
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn subnet_obj = self._get_subnet_object(context, id)
Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 282, in _get_subnet_object
Jan 26 15:10:35.550111 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn raise exceptions.SubnetNotFound(subnet_id=id)
Jan 26 15:10:35.550111 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn neutron_lib.exceptions.SubnetNotFound: Subnet a283b869-2e54-44da-b109-06da46933d06 could not be found.
Jan 26 15:10:35.550111 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn

Builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-slow&project=openstack%2Fneutron&branch=master&skip=0

Recently merged patches[1] seems to have triggered this, other neutron ovn scenario job likely not impacted as those explicitly enable extensions.

[1] https://review.opendev.org/q/topic:%222023-aa-l3-gw-multihoming%22+status:merged+file:neutron/common/ovn/extensions.py

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

The issue was triggered by [1], one of the patches for OVN GW multihoming, but this patch is not the cause of it.

Since [1], if the router has a GW port and FIP is distributed, we update the local router port [2]. This method is retrieving the subnet of the external network. If this operation is done by a non-admin user, it will fail as reported. How to reproduce it:
* As admin, create an external network.
* As non-admin, in other project, create a router.
* Assign the external GW to the router (this step should be done first)
* Create a private network and subnet, and assign the subnet to the router --> that will trigger the error.

This error is due to the way we handle the subnet RBACs. The subnet object inherits the network RBACs. In the network query, we add an extra query hook filter [3] that add the following condition:
  rbac_model.action == rbac_db_models.ACCESS_EXTERNAL

That means external networks are also retrieved as "shared" networks. But this is not happening with the subnet query, leading to the reported error.

Because it is needed an urgent fix, I'll propose raising the context in the ``_get_nets_and_ipv6_ra_confs_for_router_port`` method. A proper fix handling the subnet RBACs should be proposed later.

[1]https://review.opendev.org/c/openstack/neutron/+/874199
[2]https://github.com/openstack/neutron/blob/ca8e5b62e8f093a1d1ceb37a988e76d0a7ecd42f/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1755
[3]https://github.com/openstack/neutron/blob/ca8e5b62e8f093a1d1ceb37a988e76d0a7ecd42f/neutron/db/external_net_db.py#L41-L57

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/907312

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/907313

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/907312
Committed: https://opendev.org/openstack/neutron/commit/70e51eb2baac3c891d735aa5ffbeb4fca425f29b
Submitter: "Zuul (22348)"
Branch: master

commit 70e51eb2baac3c891d735aa5ffbeb4fca425f29b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 29 23:25:44 2024 +0000

    [OVN] Use elevated context to retrieve subnet in router port configuration

    The method ``_get_nets_and_ipv6_ra_confs_for_router_port`` can be called
    from a non-admin user request, when updating or creating a local router
    port. If the router external gateway network is "external" (as it should
    be) but is not explicitly shared (a network RBAC with action
    "access_as_shared"), the user won't retrieve the corresponding subnet.

    NOTE: is is *not* needed to apply both "access_as_shared" and
    "access_as_external" RBACs to a network. Please read c#1 in the LP bug
    for more context.

    Related-Bug: #2051831
    Change-Id: I161f1a6021c0da2d0063f8cb249b3bb9d7b6d5ae

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/907949

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/908256

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/908257

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/908256
Committed: https://opendev.org/openstack/neutron/commit/b1d7df2c17dd656fdb07c415a53da78a81d22c0f
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit b1d7df2c17dd656fdb07c415a53da78a81d22c0f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 29 23:25:44 2024 +0000

    [OVN] Use elevated context to retrieve subnet in router port configuration

    The method ``_get_nets_and_ipv6_ra_confs_for_router_port`` can be called
    from a non-admin user request, when updating or creating a local router
    port. If the router external gateway network is "external" (as it should
    be) but is not explicitly shared (a network RBAC with action
    "access_as_shared"), the user won't retrieve the corresponding subnet.

    NOTE: is is *not* needed to apply both "access_as_shared" and
    "access_as_external" RBACs to a network. Please read c#1 in the LP bug
    for more context.

    Related-Bug: #2051831
    Change-Id: I161f1a6021c0da2d0063f8cb249b3bb9d7b6d5ae
    (cherry picked from commit 70e51eb2baac3c891d735aa5ffbeb4fca425f29b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/908257
Committed: https://opendev.org/openstack/neutron/commit/68542c04a11a5585986571dac15fd72e11fb8f73
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 68542c04a11a5585986571dac15fd72e11fb8f73
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jan 29 23:25:44 2024 +0000

    [OVN] Use elevated context to retrieve subnet in router port configuration

    The method ``_get_nets_and_ipv6_ra_confs_for_router_port`` can be called
    from a non-admin user request, when updating or creating a local router
    port. If the router external gateway network is "external" (as it should
    be) but is not explicitly shared (a network RBAC with action
    "access_as_shared"), the user won't retrieve the corresponding subnet.

    NOTE: is is *not* needed to apply both "access_as_shared" and
    "access_as_external" RBACs to a network. Please read c#1 in the LP bug
    for more context.

    Related-Bug: #2051831
    Change-Id: I161f1a6021c0da2d0063f8cb249b3bb9d7b6d5ae
    (cherry picked from commit 70e51eb2baac3c891d735aa5ffbeb4fca425f29b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/907949
Committed: https://opendev.org/openstack/neutron-lib/commit/fc2f383dd698321f01dc8ec56dff44ba4ec66157
Submitter: "Zuul (22348)"
Branch: master

commit fc2f383dd698321f01dc8ec56dff44ba4ec66157
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Feb 2 06:26:13 2024 +0000

    Add extension "subnet-external-network"

    This extension adds a new field to the "subnet" resource:
    "router:external". This boolean field, that is False by default,
    represents if the subnet belongs to an external network.

    Related-Bug: #2051831
    Change-Id: I75a9c30f1e8031d40a548df345b02fbe0bc47706

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/911105

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/912273

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/912273
Committed: https://opendev.org/openstack/neutron-lib/commit/9db26809f9352d8a9536383c81e6db9f5afe5ffe
Submitter: "Zuul (22348)"
Branch: master

commit 9db26809f9352d8a9536383c81e6db9f5afe5ffe
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Mar 8 01:18:58 2024 +0000

    Enable filtering flag in subnet 'router:external' field

    This flag was incorrectly set to False in the API extension
    implementation. This extension is still not implemented in Neutron.

    Change-Id: I87ea4e8bf4b4bc6a6934a349ac28765107c1536a
    Related-Bug: #2051831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/911105
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/0da38af84cfb82d43adc768f985268471a2f1bfd
Submitter: "Zuul (22348)"
Branch: master

commit 0da38af84cfb82d43adc768f985268471a2f1bfd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Mar 5 00:03:27 2024 +0000

    Add extension "subnet-external-network"

    Added extension "subnet-external-network" to:
    * neutron-tempest-plugin-base
    * neutron-tempest-plugin-openvswitch
    * neutron-tempest-plugin-openvswitch-iptables_hybrid

    The ``SubnetsSearchCriteriaTest.test_list_validation_filters`` test
    case now filters the subnets by 'shared' and 'router:external' fields,
    same as the network test case. It should behave the same with this
    new extension.

    Related-Bug: #2051831
    Change-Id: I13144e3d722c06c918cd47860f378bf4bdaa0bf7