tripleo-validations; unable to discover unit tests, breaking the gate

[wes hayutin]

+ cd validations-common-1.1.0.dev3
+ PYTHON=/usr/bin/python3
+ /usr/bin/python3 setup.py test
running test
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/subunit/run.py", line 149, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/subunit/run.py", line 145, in main
    stdout=stdout, exit=False)
  File "/usr/lib/python3.6/site-packages/testtools/run.py", line 171, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.6/site-packages/unittest2/main.py", line 113, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python3.6/site-packages/testtools/run.py", line 211, in _do_discovery
    super(TestProgram, self)._do_discovery(argv, Loader=Loader)
  File "/usr/lib/python3.6/site-packages/unittest2/main.py", line 223, in _do_discovery
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/lib/python3.6/site-packages/unittest2/loader.py", line 367, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python3.6/site-packages/unittest2/loader.py", line 433, in _find_tests
    for test in path_tests:
  File "/usr/lib/python3.6/site-packages/unittest2/loader.py", line 433, in _find_tests
    for test in path_tests:
  File "/usr/lib/python3.6/site-packages/unittest2/loader.py", line 424, in _find_tests
    full_path, pattern, namespace)
TypeError: 'NoneType' object is not iterable
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \

https://da9b434303bc6026f4f7-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf2.rackcdn.com/722274/1/gate/tripleo-ci-centos-8-standalone/598dbb0/logs/delorean_logs/component/tripleo/32/7e/327e7a9b5625dd433b0b1e1d86b704eee05fdbfd_dev/build.log.txt.gz

[wes hayutin (weshayutin)]

Disabling tripleo-validations in the tripleo pipeline atm.. changes will not be integrated until this patch is reverted.

https://review.opendev.org/#/c/722485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722485
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=cc71de4c4126977d38ac982f98733b5a871c1a08
Submitter: Zuul
Branch: master

commit cc71de4c4126977d38ac982f98733b5a871c1a08
Author: Sagi Shnaidman <email address hidden>
Date: Thu Apr 23 23:07:10 2020 +0300

    Disable build of validations

    Validations build always fail on tests discovery
    Related-Bug: #1874535

    Change-Id: I2f398463b18d1fff53ba774a538a7e9879f989c0

[Javier Peña (jpena-c)]

This ringed a bell. We had the same issue on the RDO Trunk builder on Apr 17 [1]. It only happened once, and later builds worked fine. So I suspect there might be something in the validations-common code that triggers this, but not sure about what.

[1] - https://trunk.rdoproject.org/centos8-ussuri/component/tripleo/99/05/99050941f49a8752c8b2a7f75af12039f21c1afe_26f41efa/rpmbuild.log

[Javier Peña (jpena-c)]

s/ringed/rang/

[Marios Andreou (marios-b)]

10:39 < jpena> hm, validations-common... aha, I knew I remembered something
10:39 < jpena> https://trunk.rdoproject.org/centos8-ussuri/component/tripleo/99/05/99050941f49a8752c8b2a7f75af12039f21c1afe_26f41efa/rpm
build.log

[mathieu bultel (mat-bultel)]

So after chatting with CI squad, looks like the commit that breaks "sometimes" the build is:

< jpena> matbu: the issue in RDO Trunk happened with commit 99050941f49a8752c8b2a7f75af12039f21c1afe
09:52 < matbu> jpena: so this commit is breaking "sometimes" or every build ?
09:52 < jpena> matbu: sometimes
09:52 < matbu> erf
09:52 < jpena> https://trunk.rdoproject.org/api-centos8-ussuri/api/report.html?package=validations-common

[Javier Peña (jpena-c)]

I just found out the issue happened once on a tox-py36 job for validations-common: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_183/719598/3/gate/openstack-tox-py36/183a9aa/job-output.txt

[wes hayutin (weshayutin)]

Patch up https://review.opendev.org/#/c/723529/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723529
Committed: https://git.openstack.org/cgit/openstack/validations-common/commit/?id=07d9c9621ef28b492ca093cdd110bfe951ede90d
Submitter: Zuul
Branch: master

commit 07d9c9621ef28b492ca093cdd110bfe951ede90d
Author: danpawlik <email address hidden>
Date: Mon Apr 27 15:16:57 2020 +0200

    Move unit tests execution to stestr

    From time to time, unit tests execution fails with error:
    "TypeError: 'NoneType' object is not iterable". It seems that
    problem is in unittest2 library, which is used by testtools for
    executing the tests.
    Stestr lib is usinig unittest by default and the problem does not appear
    for several hundred launches.

    Change-Id: Ic1053353a87ebd44f20c607651772928020b9e66
    Closes-Bug: #1874535