[stable/rocky] neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-rocky job fails

[Bernard Cafarelli]

With some other failures (pip mirrors, grenade, https://bugs.launchpad.net/bugs/1923413 , etc) now fixed, rocky backports are almost back in green except this job.

Sample backports failing:
https://review.opendev.org/c/openstack/neutron/+/779780
https://review.opendev.org/c/openstack/neutron/+/777123

We have 2 separate issues, first the testr_results.html generation fails on subunit2html
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 124173: ordinal not in range(128)

This is https://review.opendev.org/c/openstack/os-testr/+/700778 but it is not included in version used in rocky (and newer do not support py2)

This should probably be not needed if tests are all passing (we just need a bit more digging to find failing tests)

Second problem, some tests fail 100% or most of the time:
neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_to_2_servers
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity
(both seen in all rechecks I checked)
neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_router_east_west_traffic
neutron_tempest_plugin.scenario.test_portsecurity.PortSecurityTest.test_port_security_removed_added
(I saw these a few times)

At least one of the tests fails on missing ncat:
2021-04-15 04:18:09.915121 | controller | 2021-04-15 03:39:35,818 25332 INFO [tempest.lib.common.ssh] ssh connection to cirros@172.24.5.201 successfully created
2021-04-15 04:18:09.915129 | controller | 2021-04-15 03:39:36,071 25332 DEBUG [neutron_tempest_plugin.common.shell] Executing command 'ncat --version 2>&1' on local host (timeout=None)...
2021-04-15 04:18:09.915141 | controller | 2021-04-15 03:39:36,080 25332 DEBUG [neutron_tempest_plugin.common.shell] Command 'ncat --version 2>&1' failed (exit_status=127):
2021-04-15 04:18:09.915167 | controller | stderr:
2021-04-15 04:18:09.915180 | controller |
2021-04-15 04:18:09.915188 | controller | stdout:
2021-04-15 04:18:09.915197 | controller | /bin/sh: 1: ncat: not found

Maybe something changed in base image we use?

[Bernard Cafarelli (bcafarel)]

Strange, at least some of these tests should *not* run on rocky, at least they are excluded in job definition:
https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/zuul.d/rocky_jobs.yaml#L203

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/786657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/786657
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/a4cb8077a8efa311b01ded3d2c27fc484016da13
Submitter: "Zuul (22348)"
Branch: master

commit a4cb8077a8efa311b01ded3d2c27fc484016da13
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 16 15:33:29 2021 +0200

    Unset new rolevar for run-tempest role for rocky jobs

    It was done switched to use new rolevar for newer branches in [1]
    some time ago but it seems that as rocky and queens jobs are inheriting
    from the jobs defined for master branch, they have both regex variables
    set and that is causing problems when running CI jobs for older
    branches.
    It is like that because it seems that roles defined in Tempest repo
    (run-tempest) is always used from master branch, even if tempest is
    pinned in devstack to be e.g. 23.0.0 (for Rocky).
    Because of that, as for example our job
    neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-rocky
    inherits from the neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid
    it happened that it had defined both "tempest_black_regex" and
    "tempest_exclude_regex" so both tasks:

    run-tempest : Build exclude regex (old param)
    run-tempest : Build exclude regex (new param)

    were skipped and there was no list with tests to skip passed to tempest
    really.

    To avoid that, we need to define jobs for rocky branch "from
    scratch" and not inherit from the jobs defined in zuul.d/master.yaml
    as such jobs for master branch is using tempest_exclude_regex variable
    and there is no way to unset it in the rocky job.

    [1] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/775257

    Change-Id: Iafa52d60050c12ea10ab1bfb4d6147f886303ea5
    Closes-Bug: #1924315

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin train-last release.