Intermittent failures of test_agent_metadata_port_ip_update_event

[Miro Tomaska]

test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_metadata_port_ip_update_event with

ft1.4: neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_metadata_port_ip_update_eventtesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/ovn/metadata/test_metadata_agent.py", line 345, in test_agent_metadata_port_ip_update_event
    self._test_agent_events(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/ovn/metadata/test_metadata_agent.py", line 228, in _test_agent_events
    self.sb_api.lsp_bind(mdt_port_name, self.chassis_name).execute(
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 50, in execute
    t.add(self)
  File "/usr/lib64/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/api.py", line 120, in transaction
    del self._nested_txns_map[cur_thread_id]
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/api.py", line 71, in __exit__
    self.result = self.commit()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
    raise result.ex
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
    command.run_idl(txn)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/schema/ovn_southbound/commands.py", line 93, in run_idl
    binding = idlutils.row_by_value(self.api.idl, 'Port_Binding',
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
    raise RowNotFound(table=table, col=column, match=match)
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Binding with logical_port=ovn-mdt-feb321c1-e4b9-46f7-89ee-3fa202838a8f

This is due to a race condition between when the metadata port is committed to NB DB, northd reflecting in SB DB and test calling logical switch port bind in SB DB.

proosed solution: Poll check SB DB before calling sb_api.lsp_bind

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a68/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/a680936/testr_results.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/878549

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/878549
Committed: https://opendev.org/openstack/neutron/commit/9d5163301310a8c3cb72e18cbaed89afbc07f674
Submitter: "Zuul (22348)"
Branch: master

commit 9d5163301310a8c3cb72e18cbaed89afbc07f674
Author: Miro Tomaska <email address hidden>
Date: Fri Mar 24 11:51:06 2023 -0500

    Fix intermittent failures in finding metada port in SB DB

    The test_agent_metadata_port_ip_update_event can sometimes fail
    with finding the port in SB DB Port_Binding table. This is
    due to a race condition between when the port is commited to NB DB
    ,reflected in SB DB and calling lsp_bind on SB DB.
    This patch simply adds metadata port create event check before
    calling lsp_bind on the metadata port.

    Closes-Bug: #2012754
    Change-Id: I07fa45401788da6b963830e72a7b3a3cd54662e1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/897846

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/897846
Committed: https://opendev.org/openstack/neutron/commit/ec588b0bc00b12867f718b860b67bdeaeb6224a3
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ec588b0bc00b12867f718b860b67bdeaeb6224a3
Author: Miro Tomaska <email address hidden>
Date: Fri Mar 24 11:51:06 2023 -0500

    Fix intermittent failures in finding metada port in SB DB

    The test_agent_metadata_port_ip_update_event can sometimes fail
    with finding the port in SB DB Port_Binding table. This is
    due to a race condition between when the port is commited to NB DB
    ,reflected in SB DB and calling lsp_bind on SB DB.
    This patch simply adds metadata port create event check before
    calling lsp_bind on the metadata port.

    Closes-Bug: #2012754
    Change-Id: I07fa45401788da6b963830e72a7b3a3cd54662e1
    (cherry picked from commit 9d5163301310a8c3cb72e18cbaed89afbc07f674)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.