Deployment fails due to inventory_path set to empty string on centos stream 9

[Michele Baldessari]

So on my cs9 deployments I need to do the following in order to be able to deploy:
--- a/tripleo_ansible/playbooks/cli-config-download.yaml
+++ b/tripleo_ansible/playbooks/cli-config-download.yaml
@@ -51,7 +51,7 @@
         ansible_ssh_user: "{{ ansible_ssh_user }}"
         ansible_python_interpreter: "{{ python_interpreter }}"
         ansible_ssh_private_key_file: "{{ ansible_ssh_private_key_file }}"
- when: inventory_path is not defined or inventory_path != None
+ #when: inventory_path is not defined or inventory_path != None

Otherwise I will get the following error:
2021-09-08 19:30:54.086721 | 52540035-d961-0b9a-3d36-00000000000a | TIMING | Download config | localhost | 0:00:10.959871 | 10.77s
2021-09-08 19:30:54.089837 | 52540035-d961-0b9a-3d36-00000000000b | TASK | Generate ansible inventory
2021-09-08 19:30:54.104235 | 52540035-d961-0b9a-3d36-00000000000b | SKIPPED | Generate ansible inventory | localhost
2021-09-08 19:30:54.104966 | 52540035-d961-0b9a-3d36-00000000000b | TIMING | Generate ansible inventory | localhost | 0:00:10.978116 | 0.01s
2021-09-08 19:30:54.107643 | 52540035-d961-0b9a-3d36-00000000000d | TASK | Create inventory directory
2021-09-08 19:30:54.358310 | 52540035-d961-0b9a-3d36-00000000000d | OK | Create inventory directory | localhost
2021-09-08 19:30:54.359238 | 52540035-d961-0b9a-3d36-00000000000d | TIMING | Create inventory directory | localhost | 0:00:11.232383 | 0.25s^
2021-09-08 19:30:54.362723 | 52540035-d961-0b9a-3d36-00000000000e | TASK | Copy generated inventory
2021-09-08 19:30:54.633848 | 52540035-d961-0b9a-3d36-00000000000e | FATAL | Copy generated inventory | localhost | error={"changed": false, "msg": "Source /home/stack/overcloud-deploy/overcloud/config-download/overcloud/tripl
eo-ansible-inventory.yaml not found"}

If I dump the extra_vars inside tripleoclient/utils.py:run_ansible_playbook() I see the following:
2021-09-22 16:20:59.682 524975 WARNING tripleoclient.utils.utils [-] {'plan': 'overcloud', 'output_dir': '/home/stack/overcloud-deploy/overcloud/config-download', 'ansible_ssh_user': 'tripleo-admin', 'ansible_ssh_private_key_file': '/home/stack/.ssh/id_rsa_tripleo', 'ssh_network': 'ctlplane', 'python_interpreter': None, 'inventory_path': None}

So both inventory_path and python_interpreter are None (which is expected). But then I notice two
things:
A) If I print out inventory_path inside /usr/share/ansible/tripleo-playbooks/cli-config-download.yaml is is set to the empty string and that is why it will fail to match the when
condition: - when: inventory_path is not defined or inventory_path != None

B) If I look at the generated .sh file I see 'null' being passed around:
[stack@undercloud-0 ~]$ tail -n10 ./overcloud-deploy/overcloud/cli-config-download/ansible-playbook-command.sh
export BASH_FUNC_which%%="() { ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
}"
export _="/usr/bin/openstack"
export OS_HEAT_TYPE="ephemeral"
export OS_HEAT_HOST="192.168.24.1"
export OS_HEAT_PORT="8006"
export ANSIBLE_CONFIG="/home/stack/overcloud-deploy/overcloud/cli-config-download/ansible.cfg"
echo -e "Running Ansible command"
ansible-playbook -i /home/stack/overcloud-deploy/overcloud/cli-config-download/hosts.yaml -e '{"plan":"overcloud","output_dir":"/home/stack/overcloud-deploy/overcloud/config-download","ansible_ssh_user":"tripleo-admin","ansible_ssh_private_key_file":"/home/stack/.ssh/id_rsa_tripleo","ssh_network":"ctlplane","python_interpreter":null,"inventory_path":null}' /usr/share/ansible/tripleo-playbooks/cli-config-download.yaml "$@"

The simple fix would be to add inventory_path == '' to the when condition, but we wouldn't be root-causing it fully?

[Michele Baldessari (michele)]

Ansible versions I get on this systems are:
[stack@undercloud-0 ~]$ rpm -qa |grep ansible
ansible-macros-2021.1.2-1.el9.noarch
ansible-2.9.21-1.el9.noarch
ansible-core-dummy-2.11.0-0.1.el9.noarch
ansible-role-container-registry-1.3.1-0.20210914193400.c17c316.el9.noarch
ansible-tripleo-ipsec-11.0.1-0.20210902153307.b5559c8.el9.noarch
ansible-role-collectd-config-0.0.2-0.20210921104552.1992666.el9.noarch
ansible-role-qdr-config-0.0.1-0.20210902155052.b456651.el9.noarch
ansible-config_template-1.1.2-0.20210921104318.8a9b962.el9.noarch
ansible-role-atos-hsm-1.0.1-0.20210902150239.c9f757d.el9.noarch
ansible-role-chrony-1.0.4-0.20210902150459.15ad507.el9.noarch
ansible-role-lunasa-hsm-1.1.1-0.20210902153521.26da379.el9.noarch
ansible-role-thales-hsm-1.0.0-0.20210902150033.e82db3b.el9.noarch
ansible-freeipa-0.3.8-3.el9.noarch
ansible-collection-ansible-posix-1.2.0-1.el9.noarch
ansible-collection-ansible-utils-2.3.0-2.el9.noarch
ansible-collection-ansible-netcommon-2.2.0-1.el9.noarch
ansible-collection-community-general-3.0.0-1.el9.noarch
ansible-collection-containers-podman-1.6.1-2.el9.noarch
ansible-collections-openstack-1.5.2-0.20210922083845.011515d.el9.noarch
ansible-pacemaker-1.0.4-0.20210902125851.666f706.el9.noarch
ansible-role-openstack-operations-0.0.1-0.20210914192806.2ab288f.el9.noarch
ansible-role-metalsmith-deployment-1.5.1-0.20210921115814.2a39acc.el9.noarch
python3-ansible-runner-2.0.0a1-1.el9.noarch
ansible-tripleo-ipa-0.2.2-0.20210521132335.9159108.el9.noarch
ansible-role-tripleo-modify-image-1.2.3-0.20210902155610.0b9fdcc.el9.noarch
python3-heat-agent-ansible-2.3.0-0.20210601030414.ba45ab4.el9.noarch
tripleo-ansible-4.0.1-0.20210921112826.79949de.el9.noarch

[Michele Baldessari (michele)]

So I think I made a wrong assumption in A) above. When I add this:
     debug:
        msg: "{{ inventory_path|type_debug }}"

Before 'Generate ansible inventory' I get the following output:
2021-09-23 07:48:47.616457 | 525400b7-c9dd-47ec-5595-00000000000c | OK | DEBUG | localhost | result={
    "changed": false,
    "msg": "NoneType"
}

So inventory_path is None. I am not entirely sure why we have this when condition in the first place?

[Michele Baldessari (michele)]

https://review.opendev.org/c/openstack/tripleo-ansible/+/808013 mainly to start a discussion

[Michele Baldessari (michele)]

As Alex pointed out this has been fixed in master via https://review.opendev.org/q/topic:%22stack-only%22+status:merged

[Michele Baldessari (michele)]

I also tested it and am not hitting this issue on master any longer

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Michele Baldessari <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/808013