[Queens] tripleo-ci-centos-7-undercloud-oooq is failing on tempest tests with "tempest run: error: unrecognized arguments: --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt --include-list /home/zuul/tempest/etc/tempest_includelist.txt"

[Sandeep Yadav]

Description:-

tripleo-ci-centos-7-undercloud-oooq is failing on tempest tests with "tempest run: error: unrecognized arguments: --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt --include-list /home/zuul/tempest/etc/tempest_includelist.txt"

First failure noticed on 28th June, but there is no build from 10th june and 28th June
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-undercloud-oooq&project=openstack/tripleo-common

Error Snippet:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_834/794814/1/check/tripleo-ci-centos-7-undercloud-oooq/83456e0/job-output.txt
~~~
2021-06-29 02:41:19.643721 | primary | TASK [os_tempest : Execute tempest tests] **************************************
2021-06-29 02:41:19.643971 | primary | Tuesday 29 June 2021 02:41:19 +0000 (0:00:00.054) 0:02:02.555 **********
2021-06-29 02:41:22.721690 | primary | fatal: [undercloud]: FAILED! => {
2021-06-29 02:41:22.722070 | primary | "changed": false,
2021-06-29 02:41:22.722622 | primary | "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run --serial --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt --include-list /home/zuul/tempest/etc/tempest_includelist.txt > /var/log/tempest/tempest_run.log\n",
2021-06-29 02:41:22.722674 | primary | "delta": "0:00:02.394184",
2021-06-29 02:41:22.722741 | primary | "end": "2021-06-29 02:41:22.688214",
2021-06-29 02:41:22.722797 | primary | "rc": 2,
2021-06-29 02:41:22.722858 | primary | "start": "2021-06-29 02:41:20.294030"
2021-06-29 02:41:22.722875 | primary | }
2021-06-29 02:41:22.722890 | primary |
2021-06-29 02:41:22.722912 | primary | STDERR:
2021-06-29 02:41:22.722926 | primary |
2021-06-29 02:41:22.722986 | primary | usage: tempest run [-h] [--workspace WORKSPACE]
2021-06-29 02:41:22.723051 | primary | [--workspace-path WORKSPACE_PATH]
2021-06-29 02:41:22.723153 | primary | [--config-file CONFIG_FILE] [--smoke | --regex REGEX]
2021-06-29 02:41:22.723211 | primary | [--black-regex BLACK_REGEX]
2021-06-29 02:41:22.723284 | primary | [--whitelist-file WHITELIST_FILE]
2021-06-29 02:41:22.723371 | primary | [--blacklist-file BLACKLIST_FILE] [--load-list LOAD_LIST]
2021-06-29 02:41:22.723444 | primary | [--list-tests] [--concurrency CONCURRENCY]
2021-06-29 02:41:22.723525 | primary | [--parallel | --serial] [--save-state] [--subunit]
2021-06-29 02:41:22.723575 | primary | [--combine]
2021-06-29 02:41:22.723757 | primary | tempest run: error: unrecognized arguments: --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt --include-list /home/zuul/tempest/etc/tempest_includelist.txt
~~~

* Affected job is using include list vs last working job that was using whitelist:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_834/794814/1/check/tripleo-ci-centos-7-undercloud-oooq/83456e0/job-output.txt

~~~
2021-06-29 02:41:12.810648 | primary | TASK [os_tempest : Generate tempest test include list] *************************
2021-06-29 02:41:12.810942 | primary | Tuesday 29 June 2021 02:41:12 +0000 (0:00:07.697) 0:01:55.722 **********
2021-06-29 02:41:16.193473 | primary | changed: [undercloud]
~~~

https://80f4a2c65933273c5cd8-98d07378d3a9c1cb5c4813ce14635919.ssl.cf1.rackcdn.com/794814/1/check/tripleo-ci-centos-7-undercloud-oooq/e17bedc/job-output.txt
~~~
2021-06-10 15:11:44.581259 | primary | TASK [os_tempest : Generate tempest test whitelist] ****************************
2021-06-10 15:11:44.581433 | primary | Thursday 10 June 2021 15:11:44 +0000 (0:00:07.727) 0:01:50.049 *********
2021-06-10 15:11:47.993564 | primary | changed: [undercloud]
~~~

Affected and last working tempest packages are same:-

Affected job

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_834/794814/1/check/tripleo-ci-centos-7-undercloud-oooq/83456e0/logs/undercloud/var/log/extra/package-list-installed.txt

~~~
openstack-tempest.noarch 1:18.0.0-0.20180308115755.b9f9f89.el7.centos
puppet-tempest.noarch 12.5.0-0.20200413052828.b3d03b2.el7
python2-keystone-tests-tempest.noarch
python2-tempest.noarch 1:18.0.0-0.20180308115755.b9f9f89.el7.centos
python2-tempestconf.noarch 2.4.0-0.20191219090050.6a616e5.el7
~~~

last good job result from 10th June

https://80f4a2c65933273c5cd8-98d07378d3a9c1cb5c4813ce14635919.ssl.cf1.rackcdn.com/794814/1/check/tripleo-ci-centos-7-undercloud-oooq/e17bedc/logs/undercloud/var/log/extra/package-list-installed.txt
~~~
openstack-tempest.noarch 1:18.0.0-0.20180308115755.b9f9f89.el7.centos
puppet-tempest.noarch 12.5.0-0.20200413052828.b3d03b2.el7
python2-keystone-tests-tempest.noarch
python2-tempest.noarch 1:18.0.0-0.20180308115755.b9f9f89.el7.centos
python2-tempestconf.noarch 2.4.0-0.20191219090050.6a616e5.el7
~~~

[Brendan Shephard (bshephar)]

Problem seems to come from here:
https://github.com/openstack/openstack-ansible-os_tempest/commit/0e7133665178e84c8401024c267071045a6b876e

Queens doesn't have a version of Tempest that supports the new, inclusive language options. So it always fails. Looks like we're always using the master branch of openstack-ansible-os_tempest?

https://github.com/openstack/tripleo-ci/blob/master/zuul.d/base.yaml#L30-L31

As we discussed today though. Is it worth saving the Queens branch at this stage? Or do we all collectively agree to let it be unsupported and end of life?

[Purandhar Sairam Mannidi (sai438)]

I think we're using Train version of tempest with openstack-ansible-os_tempest if i'm correct.

https://review.opendev.org/plugins/gitiles/openstack/tempest/+/refs/tags/22.1.0

[Purandhar Sairam Mannidi (sai438)]

Do we need master branch of Tempest?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/798827

[yatin (yatinkarel)]

stein jobs are also impacted https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-stein

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/798827
Reason: https://review.rdoproject.org/r/q/topic:%22backport_768583%22+(status:open%20OR%20status:merged) solves the issue.

[Martin Kopec (mkopec)]

please verify the bug is fixed as these backports are merged:
https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/34335
https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/34336
https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/34337

[chandan kumar (chkumar246)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-undercloud-oooq is pretty much green as the tempest spec file backports https://bugs.launchpad.net/tripleo/+bug/1933922/comments/7 are merged.

Moving this bug to fixed released.