networking-ovn-tempest-dsvm-ovs-release-python2 is very flaky

Bug #1927985 reported by Flavio Fernandes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
New
Undecided
Unassigned

Bug Description

While backporting fixes to stable/train, I see that the
networking-ovn-tempest-dsvm-ovs-release-python2 job is failing on the following test a lot of times:

https://zuul.opendev.org/t/openstack/builds?job_name=networking-ovn-tempest-dsvm-ovs-release-python2&project=openstack/networking-ovn

```
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity_soft_reboot[id-a8a02c9b-b453-49b5-89a2-cce7da6680fb]
-----------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 317, in test_subport_connectivity_soft_reboot
    vlan_subnet=vlan_subnet)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_trunk.py", line 210, in _configure_vlan_subport
    subnets=[vlan_subnet])

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/ip.py", line 75, in configure_vlan_subport
    addresses = self.list_addresses()

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/ip.py", line 120, in list_addresses
    output = self.execute('address', *command)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/ip.py", line 58, in execute
    timeout=self.timeout).stdout

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py", line 69, in execute
    ssh_client=ssh_client)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py", line 103, in execute_remote_command
    stdout = ssh_client.exec_command(command, timeout=timeout)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in wrapped_f
    return self(f, *args, **kw)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/tenacity/__init__.py", line 360, in iter
    return fut.result()

      File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()

      File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/neutron_tempest_plugin/common/ssh.py", line 171, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)

      File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 160, in exec_command
    with transport.open_session() as channel:

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/paramiko/transport.py", line 879, in open_session
    timeout=timeout,

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/paramiko/transport.py", line 1001, in open_channel
    event.wait(0.1)

      File "/usr/lib/python3.6/threading.py", line 551, in wait
    signaled = self._cond.wait(timeout)

      File "/usr/lib/python3.6/threading.py", line 299, in wait
    gotit = waiter.acquire(True, timeout)

      File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()

    fixtures._fixtures.timeout.TimeoutException
```

We need to either disable this test for python 2 in stable/train for networking-ovn, or fix it,
or both. ;)

Revision history for this message
Flavio Fernandes (ffernand) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/networking-ovn/+/791008

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/networking-ovn/+/791479

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-ovn (master)

Change abandoned by "Flavio Fernandes <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/networking-ovn/+/791479
Reason: .

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/train)

Reviewed: https://review.opendev.org/c/openstack/networking-ovn/+/791008
Committed: https://opendev.org/openstack/networking-ovn/commit/4c5cb1198ea221c0c711c027d7f461f1f0e87240
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4c5cb1198ea221c0c711c027d7f461f1f0e87240
Author: Flavio Fernandes <email address hidden>
Date: Wed May 12 10:25:03 2021 -0400

    Mark scenario trunk tests as unstable for now for python 2 job

    It's failing from time to time and it's reported in the related bug.
    So lets mark this test as unstable for now, until we will fix the issue
    to make our gate more stable.

    As the parent job (networking-ovn-base) already had a tempest_black_regex,
    set, this child job explicitly adds the parent's regex to the list. Which
    means the IPv6Test is also part of
    networking-ovn-tempest-dsvm-ovs-release-python2's tempest_black_regex.

    Related-Bug: #1927985

    Change-Id: I49df20960fbb142ca5dd1affe715d2a7bd580bcf

tags: added: in-stable-train
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.