neutron-tempest-plugin-scenario jobs on stable/rocky and stable/queens are failing with POST_FAILURE every time

[Slawek Kaplonski]

It seems that there is error during generation of the test results file:

2021-11-10 21:38:28.832885 | TASK [fetch-subunit-output : Generate testr_results.html file]
2021-11-10 21:38:30.122424 | controller | neutron_tempest_plugin.scenario.test_mtu.NetworkWritableMtuTest.test_connectivity_min_max_mtu[id-bc470200-d8f4-4f07-b294-1b4cbaaa35b9]
2021-11-10 21:38:30.122548 | controller | neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity[id-a8a02c9b-b453-49b5-89a2-cce7da66aafb]
2021-11-10 21:38:30.122613 | controller | Traceback (most recent call last):
2021-11-10 21:38:30.122666 | controller | File "/usr/local/bin/subunit2html", line 11, in <module>
2021-11-10 21:38:30.122746 | controller | sys.exit(main())
2021-11-10 21:38:30.122808 | controller | File "/usr/local/lib/python2.7/dist-packages/os_testr/subunit2html.py", line 761, in main
2021-11-10 21:38:30.123606 | controller | result.stopTestRun()
2021-11-10 21:38:30.123673 | controller | File "/usr/local/lib/python2.7/dist-packages/testtools/testresult/real.py", line 549, in stopTestRun
2021-11-10 21:38:30.125201 | controller | sink.stopTestRun()
2021-11-10 21:38:30.125351 | controller | File "/usr/local/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1775, in stopTestRun
2021-11-10 21:38:30.125835 | controller | self.decorated.stopTestRun()
2021-11-10 21:38:30.125900 | controller | File "/usr/local/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1529, in stopTestRun
2021-11-10 21:38:30.126206 | controller | return self.decorated.stopTestRun()
2021-11-10 21:38:30.126262 | controller | File "/usr/local/lib/python2.7/dist-packages/os_testr/subunit2html.py", line 517, in stopTestRun
2021-11-10 21:38:30.126405 | controller | report = self._generate_report()
2021-11-10 21:38:30.126459 | controller | File "/usr/local/lib/python2.7/dist-packages/os_testr/subunit2html.py", line 603, in _generate_report
2021-11-10 21:38:30.126603 | controller | self._generate_report_test(rows, cid, tid, n, t, o, e)
2021-11-10 21:38:30.126656 | controller | File "/usr/local/lib/python2.7/dist-packages/os_testr/subunit2html.py", line 685, in _generate_report_test
2021-11-10 21:38:30.126814 | controller | output=saxutils.escape(o + e),
2021-11-10 21:38:30.126921 | controller | UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 121085: ordinal not in range(128)
2021-11-10 21:38:30.473606 | controller | ERROR
2021-11-10 21:38:30.473915 | controller | {
2021-11-10 21:38:30.474026 | controller | "delta": "0:00:00.590579",
2021-11-10 21:38:30.474104 | controller | "end": "2021-11-10 21:38:30.146043",
2021-11-10 21:38:30.474198 | controller | "msg": "non-zero return code",
2021-11-10 21:38:30.474272 | controller | "rc": 1,
2021-11-10 21:38:30.474341 | controller | "start": "2021-11-10 21:38:29.555464"
2021-11-10 21:38:30.474408 | controller | }

But I think that real issue is that some tests are failing earlier in all those jobs.

[Bernard Cafarelli (bcafarel)]

Yes, most of the recent failures are from failed tests - I check for "- output below" or "Generate testr_results.html" in job-output.txt

I think we had checked before to fix the POST_FAILURE result but it was not an easy one (failing when trying to generate the HTML page of failed tests). Anyway to fix these, with the age of the branches I think we can expand the filter out list, like I did recently for stein:
https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/815458

Similar change would be OK for rocky/queens: regroup all unstable tests in single regex and use it in all jobs, with at least:
neutron_tempest_plugin.scenario.test_mtu.NetworkWritableMtuTest.test_connectivity_min_max_mtu
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity
(these are already out in stein)
plus the existing ones of course

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/817933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/817933
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/32cb9c7a131cdf92eb2c653407d02ed075fa795b
Submitter: "Zuul (22348)"
Branch: master

commit 32cb9c7a131cdf92eb2c653407d02ed075fa795b
Author: Slawek Kaplonski <email address hidden>
Date: Mon Nov 15 12:51:11 2021 +0100

    Don't use Ubuntu minimal in the Rocky and Queens scenario jobs

    It seems that tests like test_subport_connectivity and
    test_connectivity_min_max_mtu aren't running fine with Ubuntu minimal
    image used as guest OS on the old branches stable/Rocky and
    stable/Queens.
    As both of those branches are now in the EM phase, let's simply use
    regular Ubuntu cloud image in those jobs.

    Closes-Bug: #1950795
    Change-Id: I3c4297a5df1b30c83050a47a3c08b6623bb00abb

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 1.9.0 release.