[victoria]RuntimeError: Timed out waiting for port '41d91748-bcdf-42fa-853d-1861167c9c9e' to transition to get status 'ACTIVE'. failing on fs020-victoria

[Bhagyashri Shewale]

neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test is failing standalone-full-tempest-scenario-master

Error log:
 Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 80, in wait_until_true
    eventlet.sleep(sleep)
  File "/usr/lib/python3.6/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 221, in test_trunk_subport_lifecycle
    self._wait_for_port(port)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 141, in _wait_for_port
    "status {!r}.".format(port['id'], status)))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 84, in wait_until_true
    raise exception
RuntimeError: Timed out waiting for port '41d91748-bcdf-42fa-853d-1861167c9c9e' to transition to get status 'ACTIVE'.

Affected job: periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria

links:

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria/e9e47c0/logs/undercloud/var/log/tempest/stestr_results.html.gz
[2]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria

[Bhagyashri Shewale (bhagyashri-shewale)]

Added test in the tempest skiplist: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/811701

[Ananya Banerjee (frenzyfriday)]

We are seeing this on periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria as well

[Ananya Banerjee (frenzyfriday)]

Recent logs

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/d3a5e35/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/f0f8da9/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

[Ananya Banerjee (frenzyfriday)]

Recent fs 20 logs:

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria/fba0cb1/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[Ananya Banerjee (frenzyfriday)]

Skiplist: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/830119

[Slawek Kaplonski (slaweq)]

If this happens only in last few days, maybe the root cause is patch https://review.opendev.org/c/openstack/neutron/+/823271 ? Lets test it in https://review.rdoproject.org/r/c/testproject/+/39640

[Ananya Banerjee (frenzyfriday)]

periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri is also hitting this

Logs: https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri/8e7e77b/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Slawek Kaplonski (slaweq)]

Luis Tomas found that there is missing backport https://review.opendev.org/c/openstack/neutron/+/830123 which causes that problem. It's now approved and in the gate so should be fixed soon.