[unit test][xena+] test_port_deletion_prevention fails when runs in isolation

[yatin]

Can be reproduced by Just running:-
tox -epy3 -- test_port_deletion_prevention
or run any of the below tests individually:-
neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase.test_port_deletion_prevention_handles_missing_port
neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase.test_port_deletion_prevention_handles_missing_port

Fails as below:-
neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase.test_port_deletion_prevention_handles_missing_port
------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/ykarel/work/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)

      File "/home/ykarel/work/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)

      File "/home/ykarel/work/openstack/neutron/neutron/tests/unit/extensions/test_l3.py", line 4491, in test_port_deletion_prevention_handles_missing_port
    pl.prevent_l3_port_deletion(context.get_admin_context(), 'fakeid')

      File "/home/ykarel/work/openstack/neutron/neutron/db/l3_db.py", line 1742, in prevent_l3_port_deletion
    port = port or self._core_plugin.get_port(context, port_id)

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
    return f_with_retry(*args, **kwargs,

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
    with excutils.save_and_reraise_exception() as ectxt:

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
    return f(*dup_args, **dup_kwargs)

      File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper
    return fn(*args, **kwargs)

      File "/home/ykarel/work/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1628, in get_port
    lazy_fields = [models_v2.Port.port_forwardings,

    AttributeError: type object 'Port' has no attribute 'port_forwardings'

It's reproducible Since Xena+ since the inclusion of patch https://review.opendev.org/c/openstack/neutron/+/790691

It do not reproduce if there are other test runs(from the test class) before this test which involve other requests(like network get/create etc) apart from the ones modified in above patch.

Considering above point if this test is modified to run other requests like below then it succeeds:-
self.plugin.get_ports_count(context.get_admin_context())
or
self.plugin.get_networks_count(context.get_admin_context())
or
with self.port/network():
    self.assertIsNone(
        pl.prevent_l3_port_deletion(context.get_admin_context(), 'fakeid')
    )

The issue was originally noticed in a downstream job where unit tests were executed as part of package build. As the tests suite was executed with 100+ concurrency, by chance this test got executed in a worker first of any other test of the class.

Me not sure if there are other test cases which will fail like this when run in isolation.

[Slawek Kaplonski (slaweq)]

Patch proposed https://review.opendev.org/c/openstack/neutron/+/890342

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/890232

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/890233

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/890234

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/890235

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/890236

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/890235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/890232
Committed: https://opendev.org/openstack/neutron/commit/f0dec9fa8d5d3023435be5dad6c1a5601debb662
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit f0dec9fa8d5d3023435be5dad6c1a5601debb662
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 2 16:27:50 2023 +0200

    [UT] Create network to make lazy loading in the models_v2 possible

    In the test_port_deletion_prevention_handles_missing_port test which is
    part of the classes:
    * neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase
    * neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase

    it is needed to call pecan application to e.g. create network to make
    all models to be ready. Otherwise things like port_forwarding attribute
    in the Port class, which is loaded in lazy mode isn't available and test
    was failing when was run in the isolated environment.
    It wasn't failing in the gate as there were other tests run by the same
    worker before this one and then it was all initialized properly.

    Closes-Bug: #2028285
    Change-Id: Ie2382540d7c0a8813f093ddf51d82fe530026d71
    (cherry picked from commit 14b2f4f60f1f1c00f41545394bd9761f57b6e37f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/890233
Committed: https://opendev.org/openstack/neutron/commit/2e50aef67aa7dfcdd7e35432e0d34b065d3e3266
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 2e50aef67aa7dfcdd7e35432e0d34b065d3e3266
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 2 16:27:50 2023 +0200

    [UT] Create network to make lazy loading in the models_v2 possible

    In the test_port_deletion_prevention_handles_missing_port test which is
    part of the classes:
    * neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase
    * neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase

    it is needed to call pecan application to e.g. create network to make
    all models to be ready. Otherwise things like port_forwarding attribute
    in the Port class, which is loaded in lazy mode isn't available and test
    was failing when was run in the isolated environment.
    It wasn't failing in the gate as there were other tests run by the same
    worker before this one and then it was all initialized properly.

    Closes-Bug: #2028285
    Change-Id: Ie2382540d7c0a8813f093ddf51d82fe530026d71
    (cherry picked from commit 14b2f4f60f1f1c00f41545394bd9761f57b6e37f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/890234
Committed: https://opendev.org/openstack/neutron/commit/8ebd7dbdf007b440d0aa79270d7d55f742652f19
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 8ebd7dbdf007b440d0aa79270d7d55f742652f19
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 2 16:27:50 2023 +0200

    [UT] Create network to make lazy loading in the models_v2 possible

    In the test_port_deletion_prevention_handles_missing_port test which is
    part of the classes:
    * neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase
    * neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase

    it is needed to call pecan application to e.g. create network to make
    all models to be ready. Otherwise things like port_forwarding attribute
    in the Port class, which is loaded in lazy mode isn't available and test
    was failing when was run in the isolated environment.
    It wasn't failing in the gate as there were other tests run by the same
    worker before this one and then it was all initialized properly.

    Closes-Bug: #2028285
    Change-Id: Ie2382540d7c0a8813f093ddf51d82fe530026d71
    (cherry picked from commit 14b2f4f60f1f1c00f41545394bd9761f57b6e37f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/890236
Committed: https://opendev.org/openstack/neutron/commit/c7d375c3788bece3b792b4e1638f760adfcc04b9
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c7d375c3788bece3b792b4e1638f760adfcc04b9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 2 16:27:50 2023 +0200

    [UT] Create network to make lazy loading in the models_v2 possible

    In the test_port_deletion_prevention_handles_missing_port test which is
    part of the classes:
    * neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase
    * neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase

    it is needed to call pecan application to e.g. create network to make
    all models to be ready. Otherwise things like port_forwarding attribute
    in the Port class, which is loaded in lazy mode isn't available and test
    was failing when was run in the isolated environment.
    It wasn't failing in the gate as there were other tests run by the same
    worker before this one and then it was all initialized properly.

    Closes-Bug: #2028285
    Change-Id: Ie2382540d7c0a8813f093ddf51d82fe530026d71
    (cherry picked from commit 14b2f4f60f1f1c00f41545394bd9761f57b6e37f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

[Andrey Semenov (wilful)]

In the recent installation of the service, I encountered a similar error. If you create a server with an error, restart the service, then the service can no longer process API requests.

ZED: openstack-neutron-21.1.2-1.el9s.noarch

# openstack server list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2a94913d-2438-444f-b862-dcfaf3b4c393)

sgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
sgi response = self._view_builder.detail(
sgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail
sgi self._add_security_grps(request, list(servers_dict["servers"]),
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 184, in wrapped
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 135, in index
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs)
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 144, in get
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation return {self.collection: self.plugin_lister(*lister_args,
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 226, in wrapped
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation return f_with_retry(*args, **kwargs,
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 142, in wrapped
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-11-11 10:30:35.615 423268 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-11-11 10:30:35.615 423268 ERR...

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.

[Brian Haley (brian-haley)]

The comment on a failure in Zed looked to not have the fix - version 21.1.2, version 21.2.0 or greater is required. Will close this as the fixes have been released.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.