jobs are failing in ci.centos when installing tempest-skiplist - The task includes an option with an undefined variable. The error was: 'zuul' is undefined

[Ronelle Landy]

Master and Ussuri jobs in ci.centos are failing with the following error:

10:47:00 TASK [install_skiplist : Install tempest-skiplist] *****************************
10:47:00 task path: /home/jenkins/workspace/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/install_skiplist/tasks/main.yaml:9
10:47:00 Monday 20 July 2020 14:47:00 +0000 (0:00:05.035) 1:50:38.286 ***********
10:47:01 fatal: [undercloud]: FAILED! => {}
10:47:01
10:47:01 MSG:
10:47:01
10:47:01 The task includes an option with an undefined variable. The error was: 'zuul' is undefined
10:47:01
10:47:01 The error appears to be in '/home/jenkins/workspace/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/install_skiplist/tasks/main.yaml': line 9, column 5, but may
10:47:01 be elsewhere in the file depending on the exact syntax problem.
10:47:01
10:47:01 The offending line appears to be:
10:47:01
10:47:01
10:47:01 - name: Install tempest-skiplist
10:47:01 ^ here

Full log is below:
https://ci.centos.org/job/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/34/console

---

https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/playbooks/tempest_skip_validate.yaml#L7 needs to support non-zuul workflows.

[Ronelle Landy (rlandy)]

https://review.opendev.org/741977 Support non-zuul workflow for list_skipped_yaml_file

[yatin (yatinkarel)]

Now after https://review.opendev.org/741977 is merged, it fails with different error on same task:-
TASK [install_skiplist : Install tempest-skiplist] *****************************
task path: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/install_skiplist/tasks/main.yaml:9
Wednesday 22 July 2020 22:00:54 +0000 (0:00:08.327) 2:38:38.200 ********
fatal: [undercloud]: FAILED! => {
    "changed": false,
    "cmd": [
        "/usr/bin/pip3",
        "install",
        "--user",
        "."
    ]
}

MSG:

:stderr: Directory '.' is not installable. File 'setup.py' not found.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743721
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=99f4e977336f932c0db59cc0d327a0303c24e73c
Submitter: Zuul
Branch: master

commit 99f4e977336f932c0db59cc0d327a0303c24e73c
Author: Arx Cruz <email address hidden>
Date: Wed Jul 29 12:28:25 2020 +0200

    Add delegate_to in skiplist tasks

    The skiplist requires python module from openstack-tempest-skiplist
    available in the python path. This works when quickstart is being
    executed from the undercloud, but in some systems where quickstart is
    executed in a different host, it fails. Use delegate_to ensure it
    will be executed from the host quickstart is being executed.

    Partial-Bug: #1888244

    Change-Id: I433e39c212e4ab887b6ed0776715e344bfa404d7

[wes hayutin (weshayutin)]

04:07:01 TASK [list_skipped : List skipped tests] ***************************************
04:07:01 task path: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/list_skipped/tasks/main.yaml:2
04:07:01 Tuesday 28 July 2020 10:07:01 +0000 (0:00:00.119) 2:36:07.251 **********
04:07:02 fatal: [undercloud]: FAILED! => {
04:07:02 "changed": false,
04:07:02 "rc": 127
04:07:02 }
04:07:02
04:07:02 MSG:
04:07:02
04:07:02 The module failed to execute correctly, you probably need to set the interpreter.
04:07:02 See stdout/stderr for the exact error
04:07:02
04:07:02
04:07:02 MODULE_STDERR:
04:07:02
04:07:02 Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
04:07:02 /bin/sh: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/bin/python: No such file or directory
04:07:02

https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/782/console

[yatin (yatinkarel)]

Last run worked for skiplist https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/783/consoleText, now new Error for missing virtualenv

TASK [python_venv_build : Check for virtualenv command being available on system path] ***
task path: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/python_venv_build/tasks/python_venv_preflight.yml:24
Thursday 30 July 2020 23:10:39 +0000 (0:00:00.050) 3:15:56.966 *********
fatal: [undercloud]: FAILED! => {
    "changed": false,
    "cmd": [
        "which",
        "virtualenv"
    ],
    "delta": "0:00:00.007896",
    "end": "2020-07-30 23:10:40.399046",
    "rc": 1,
    "start": "2020-07-30 23:10:40.391150"
}

STDERR:

which: no virtualenv in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin/)

MSG:

non-zero return code
...ignoring

TASK [python_venv_build : Fail when required virtualenv is not present] ********
task path: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/usr/local/share/ansible/roles/python_venv_build/tasks/python_venv_preflight.yml:30
Thursday 30 July 2020 23:10:40 +0000 (0:00:00.946) 3:15:57.913 *********
fatal: [undercloud]: FAILED! => {
    "changed": false
}

MSG:

The command 'virtualenv' version is not present.

Happening as required tasks were removed with https://review.opendev.org/#/c/726773/10/playbooks/tempest.yml@98

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744097

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744119

[OpenStack Infra (hudson-openstack)]

Change abandoned by Arx Cruz (<email address hidden>) on branch: master
Review: https://review.opendev.org/744119
Reason: https://review.opendev.org/#/c/744097/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744097
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=c1e192cf51909a81dfbd0345ea1ef6028640ba3a
Submitter: Zuul
Branch: master

commit c1e192cf51909a81dfbd0345ea1ef6028640ba3a
Author: yatinkarel <email address hidden>
Date: Fri Jul 31 11:52:12 2020 +0530

    Re add virtualenv detection tasks

    These tasks were cleaned up with [1], but are needed for
    cases like where overcloud image is used as undercloud as
    pointed in #1888244.

    [1] https://review.opendev.org/#/c/726773/

    Partial-Bug: #1888244
    Change-Id: Ifddb7a8dda94b094f230dfb95c4bcd9c72c65a17

[yatin (yatinkarel)]

Fixed with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/744097