[OVN] Tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails randomly

[Maciej Jozefczyk]

We can see occasional test failures of tempest test with OVN:

neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle

Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 78, in wait_until_true
    eventlet.sleep(sleep)
  File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/usr/local/lib/python3.6/dist-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 240, in test_trunk_subport_lifecycle
    self._wait_for_port(port)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 141, in _wait_for_port
    "status {!r}.".format(port['id'], status)))
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 82, in wait_until_true
    raise exception
RuntimeError: Timed out waiting for port 'cffcacde-a34e-4e1a-90ca-8d48776b9851' to transition to get status 'ACTIVE'.

Example failure:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d8c/717851/2/check/neutron-ovn-tempest-ovs-release/d8c0282/testr_results.html

[Maciej Jozefczyk (maciejjozefczyk)]

This problem seems to be serious issue:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:*test_ovn_db_sync* AND build_name: \"neutron-functional\" AND message:failed

In last week we noticed 51 failures on gates.

[Maciej Jozefczyk (maciejjozefczyk)]

Not really, my bad.

For OVN-related tests:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:*test_ovn_db_sync* AND build_name: \"neutron-functional\" AND message:failed

We have 2 refs that are causing the problem:
https://review.opendev.org/#/c/717350
https://review.opendev.org/#/c/722415

Those two patches are in WIP.

I haven't found any other failure related to that issue recently.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/830990

[yatin (yatinkarel)]

Last two comments looks for other issue and not for trunk_subport_lifecycle scenario test, pushed patch to re enable the test, the test looks stable now as seen in TripleO jobs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/830990
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/c7965d9d8ab48437bfb0de585af00620c0ddfc43
Submitter: "Zuul (22348)"
Branch: master

commit c7965d9d8ab48437bfb0de585af00620c0ddfc43
Author: yatinkarel <email address hidden>
Date: Fri Feb 25 19:21:42 2022 +0530

    Re enable trunk_subport_lifecycle scenario test

    The test is working fine now, so let's re enable
    it.

    Related-Bug: #1874447
    Change-Id: Ic7e31d4c309491c89a846bb790c0554d4cc85bb3