master: neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle is failing - "RuntimeError: Timed out waiting for port '<> to transition to get status 'DOWN'."

[Ronelle Landy]

periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle is failing on master integration jobs since 2021-12-30:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master

The error reported is:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 80, in wait_until_true
    eventlet.sleep(sleep)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 263, in test_trunk_subport_lifecycle
    self._wait_for_port(port, status=constants.DOWN)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 168, in _wait_for_port
    "status {!r}.".format(port['id'], status)))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 84, in wait_until_true
    raise exception
RuntimeError: Timed out waiting for port 'f73c37e7-bac0-40ab-82dd-0b331cd062b0' to transition to get status 'DOWN'.

The full log is linked below:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/04beffa/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Ronelle Landy (rlandy)]

Last passing job has:

http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos8/component/network/df/94/df9435a9a6fab9492c4f23d9ab0f1507841430c7_940098e0/

openstack-neutron-19.1.0-0.20211221080843.df9435a.el8.noarch.rpm 2021-12-21 08:21 26K

First failing job has:

http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos8/component/network/0e/bf/0ebf5006bb416435af88b03272247e1f8a3c7921_b71b543e/

openstack-neutron-19.1.0-0.20211223142350.ca90d8c.el8.src.rpm 2021-12-23 14:27 13M

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/e700845/logs/undercloud/var/log/extra/errors.txt.gz

2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers [req-b8550d7d-13ae-4f0c-b092-d914a2cb4f5c - - - - -] Mechanism driver 'ovn' failed in update_port_postcommit: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch_Port with name=c3d8f0f1-6fdf-441d-a625-f341a78a37f0
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 482, in _call_on_drivers
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 847, in update_port_postcommit
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers port_object=original_port)
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 597, in update_port
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers ovn_port = self._nb_idl.lookup('Logical_Switch_Port', port['id'])
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 208, in lookup
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers notify_handler=notify_handler)
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers return self._lookup(table, record)
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers row = idlutils.row_by_value(self, rl.table, rl.column, record)
2021-12-30 18:15:42.736 ERROR /var/log/containers/neutron/server.log.1: 28 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/site-packages/ovsdbapp/bac...

[Douglas Viroel (dviroel)]

Same test is failing for periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master

Tempest failure:
https://logserver.rdoproject.org/89/36489/9/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/968d844/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Ronelle Landy (rlandy)]

Also impacting c9 master:

https://logserver.rdoproject.org/openstack-periodic-integration-main-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/81c96bd/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Slawek Kaplonski (slaweq)]

I think that this can be broken by patch https://review.opendev.org/c/openstack/neutron/+/821544 but I will need to confirm that.

[Slawek Kaplonski (slaweq)]

Lets test revert of that patch in https://review.rdoproject.org/r/c/testproject/+/37697 to see if it will pass.

[Luis Tomas Bolivar (ltomasbo)]

Fix for the problem is here https://review.opendev.org/c/openstack/neutron/+/823412

[Luis Tomas Bolivar (ltomasbo)]

OVN related bug causing the problem that requires the workaround in neutron: https://bugzilla.redhat.com/show_bug.cgi?id=2037433

[Soniya Murlidhar Vyas (svyas)]

Fix has been merged-https://review.opendev.org/c/openstack/neutron/+/823412
Can we close this bug now?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.