periodic featureset 002 failing tempest "Non subunit content"

[Marios Andreou]

At [1][2][3] the periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby is failing during tempest log collection with trace like:

        * 2021-06-10 01:32:11.354657 | primary | TASK [os_tempest : Collecting data from tempest run] ***************************
 2021-06-10 01:32:12.135093 | primary | STDERR:
 2021-06-10 01:32:12.135101 | primary |
 2021-06-10 01:32:12.135110 | primary | Traceback (most recent call last):
 2021-06-10 01:32:12.135141 | primary | File "/home/zuul/stackviz_venv/bin/stackviz-export", line 8, in <module>
 2021-06-10 01:32:12.135161 | primary | sys.exit(main())
 2021-06-10 01:32:12.135170 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 277, in main
 2021-06-10 01:32:12.135197 | primary | artifacts.extend(export_tempest(provider, args.path, args.gzip))
 2021-06-10 01:32:12.135210 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 182, in export_tempest
 2021-06-10 01:32:12.135238 | primary | output_dir, prefix, compress)
 2021-06-10 01:32:12.135264 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 85, in export_tempest_raw
 2021-06-10 01:32:12.135282 | primary | converted = tempest_subunit.convert_stream(subunit, strip_details=True)
 2021-06-10 01:32:12.135332 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/parser/tempest_subunit.py", line 274, in convert_stream
 2021-06-10 01:32:12.135350 | primary | result_stream.run(result)
 2021-06-10 01:32:12.135374 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/subunit/v2.py", line 313, in run
 2021-06-10 01:32:12.135391 | primary | raise Exception("Non subunit content", content)
 2021-06-10 01:32:12.135400 | primary | Exception: ('Non subunit content', b"'")

This was also seen earlier in master at [4]

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby/77e57e3/job-output.txt
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby/02bf600/job-output.txt
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby/5c9bd62/job-output.txt
[4] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/853893e/job-output.txt

[Marios Andreou (marios-b)]

i suspect this may be because we only have one test in the fs2 tempest tests [1]

tempest_whitelist:
  - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'

but we are skipping that since [2] and as a result there are no tempest tests specified ?

I will try reverting part of [2] and posting testproject to confirm

[1] https://github.com/openstack/tripleo-quickstart/blob/141194c402da021975ffe6e93c4d5acfa966dfe8/config/general_config/featureset002.yml#L211-L212
[2] https://opendev.org/openstack/openstack-tempest-skiplist/commit/95558196ac593141c66430bcfafe6b85135738ee

[Marios Andreou (marios-b)]

 Remove TestNetworkBasicOps.test_network_basic_ops from skips https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742

testproject @ https://review.rdoproject.org/r/c/testproject/+/34057

[Marios Andreou (marios-b)]

seeing same for a recent tripleo-ci-testing tag for fs2 and fs39

        * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/280581e/job-output.txt

        * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/b21ba0f/job-output.txt

the fix at https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742 should clear those too but testing with v2 @ https://review.rdoproject.org/r/c/testproject/+/34057/2/.zuul.yaml ... chasing the promotion for the tag 882777ccfc662a70d1c3ada62992ffb4 (buildset @ [1][2])

[1] https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=882777ccfc662a70d1c3ada62992ffb4
[2] https://review.rdoproject.org/zuul/buildset/284e75b4d1ba42429b752a006af0ab1f

[Marios Andreou (marios-b)]

moving fix released couple green runs now that https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795742 merged

        * https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby
          * green @ https://review.rdoproject.org/zuul/build/2bd9081daa474d58a28ea2d0b69e53c4
          * green @ https://review.rdoproject.org/zuul/build/db88b58bd37e4193b462925d0015ee72