openstack-tox-tht failing test_tht_ansible_syntax "assert run.rc == 0"

[Marios Andreou]

At [1][2][3] the openstack-tox-tht job is failing and blocking the tripleo-heat-templates gate with trace like:

        * 2022-05-05 12:11:25.173619 | ubuntu-focal | E assert 250 == 0
2022-05-05 12:11:25.173629 | ubuntu-focal | E + where 250 = <ansible_runner.runner.Runner object at 0x7f5d9d5b1f70>.rc
2022-05-05 12:11:25.173639 | ubuntu-focal |
2022-05-05 12:11:25.173650 | ubuntu-focal | tripleo_heat_templates/tests/test_tht_ansible_syntax.py:64: AssertionError
2022-05-05 12:11:25.173660 | ubuntu-focal | ----------------------------- Captured stdout call -----------------------------
2022-05-05 12:11:25.173681 | ubuntu-focal | [WARNING]: Unable to parse /home/zuul/src/opendev.org/openstack/tripleo-heat-
2022-05-05 12:11:25.173692 | ubuntu-focal | templates/test/hosts.ini as an inventory source
2022-05-05 12:11:25.173703 | ubuntu-focal | [WARNING]: No inventory was parsed, only implicit localhost is available
2022-05-05 12:11:25.173712 | ubuntu-focal | [WARNING]: provided hosts list is empty, only localhost is available. Note that
2022-05-05 12:11:25.173722 | ubuntu-focal | the implicit localhost does not match 'all'
2022-05-05 12:11:25.173732 | ubuntu-focal | ERROR! Unexpected Exception, this is probably a bug: type 'NoneType' is not an acceptable base type

Also seen on stable/wallaby with [4]

[1] https://e89447c68a615e3657ee-af871b2bb5f1ea3994d6f36e4c6112be.ssl.cf2.rackcdn.com/839589/1/gate/openstack-tox-tht/b2eb48b/job-output.txt
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7dc/839497/3/gate/openstack-tox-tht/7dc0691/job-output.txt
[3] https://4226f2ecc073dd321ca4-58e8a805a242f8a07eac2fd1c3f6b11b.ssl.cf5.rackcdn.com/839589/1/gate/openstack-tox-tht/6b48875/job-output.txt
[4] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1ee/840581/1/check/openstack-tox-tht/1ee7357/job-output.txt

[Marios Andreou (marios-b)]

the failing assertion is there https://opendev.org/openstack/tripleo-heat-templates/src/commit/13ec11d7aed4e5af0b5dcc870a1bb87b1609380a/tripleo_heat_templates/tests/test_tht_ansible_syntax.py#L64

this is the playbook it is trying to run https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/tripleo_heat_templates/tests/test_tht_ansible_syntax.yml

[Marios Andreou (marios-b)]

confirmed that the 'WARNING' in the trace is not fatal

I added explicit inventory=localhost in the invocation with

58 run = ansible_runner.run(
59 playbook=play_path,
60 extravars={'tht_root': tht_root},
61 inventory='localhost',
62 )

and it squashed that warning. The ERROR remains so this must be the root/issue:

        * (Pdb) out.readlines()
["\x1b[0;31mERROR! Unexpected Exception, this is probably a bug: type 'NoneType' is not an acceptable base type\x1b[0m\n", 'to see the full traceback, use -vvv\n']

[Rabi Mishra (rabi)]

Though I can't reproduce it locally, it maybe some regression with the latest ansible-runner==2.2.0 released a few hours back (failure timelines somewhat match). We can try to pin it in https://github.com/openstack/tripleo-heat-templates/blob/master/test-ansible-requirements.txt#L6

[Rabi Mishra (rabi)]

OK, pinning resolves the issue[1] with can be a temp workaround to unblock.

[1] https://zuul.opendev.org/t/openstack/build/6a776163c74e479b9b70097051dcb8a4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840683
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9f028a9a04edc96e298e51b71e24914ac5f9bb6e
Submitter: "Zuul (22348)"
Branch: master

commit 9f028a9a04edc96e298e51b71e24914ac5f9bb6e
Author: rabi <email address hidden>
Date: Thu May 5 20:16:09 2022 +0530

    Pin ansible-runner <2.2.0

    ansible-runner==2.2.0 has broken openstack-tox-tht
    jobs. We can merge this to unblock till we find out
    what's causing the regression.

    Related-Bug: #1971703
    Change-Id: Ic0cc98b2d92801c222ca1280ca5e0485c7d603f1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840812

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840812
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/0da3db3b9451a7f56c48834681c115169d76f554
Submitter: "Zuul (22348)"
Branch: master

commit 0da3db3b9451a7f56c48834681c115169d76f554
Author: rabi <email address hidden>
Date: Fri May 6 09:43:03 2022 +0530

    Install ansible.posix.debug stdout callback plugin

    ansible.builtin.debug is redirected to ansible.posix.debug.
    We need to install the collection explicitly.

    This needs to be fixed for stdout callback plugin related changes
    in ansible-runner >=2.2.0.

    Also, increases ansible verbosity to track these issues better in
    the future.

    Closes-Bug: #1971703
    Change-Id: Ibe883e19c25926ab302db741afae70821209a350

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840735
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/73c0b05111f65626352b842f1f31dfae4d55a38d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 73c0b05111f65626352b842f1f31dfae4d55a38d
Author: rabi <email address hidden>
Date: Fri May 6 09:43:03 2022 +0530

    Install ansible.posix.debug stdout callback plugin

    ansible.builtin.debug is redirected to ansible.posix.debug.
    We need to install the collection explicitly.

    This needs to be fixed for stdout callback plugin related changes
    in ansible-runner >=2.2.0.

    Also, increases ansible verbosity to track these issues better in
    the future.

    Closes-Bug: #1971703
    Change-Id: Ibe883e19c25926ab302db741afae70821209a350
    (cherry picked from commit 0da3db3b9451a7f56c48834681c115169d76f554)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 17.0.0 release.