Unable to upgrade Antelope to Bobcat due to missing openstack.osa.haproxy_service_config file

[zrsolis]

When attempting to run setup-infrastructure during an upgrade from Antelope (stable/2023.1) to Bobcat (stable/2023.2), play immediately fails with the following error message:

root@infra-01:/opt/openstack-ansible/playbooks# openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
[DEPRECATION WARNING]: DEFAULT_GATHER_SUBSET option, the module_defaults keyword is a more generic version and can apply to all calls to the M(ansible.builtin.gather_facts) or M(ansible.builtin.setup) actions, use
module_defaults instead. This feature will be removed from ansible-core in version 2.18. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Unable to parse /etc/openstack_deploy/inventory.ini as an inventory source
ERROR! Unable to retrieve file contents
Could not find or access '/opt/openstack-ansible/playbooks/openstack.osa.haproxy_service_config' on the Ansible Controller.
If you are using a module and expect the file to exist on the remote, see the remote_src option

Contents of the /opt/openstack-ansible/playbooks directory:
root@infra-01:/opt/openstack-ansible/playbooks# ls -1
ceph-install.yml
ceph-nfs-install.yml
ceph-rgw-install.yml
ceph-rgw-keystone-setup.yml
certificate-authority.yml
certificate-generate.yml
certificate-ssh-authority.yml
common-playbooks
common-tasks
containers-deploy.yml
containers-lxc-create.yml
containers-lxc-destroy.yml
containers-lxc-host.yml
defaults
galera-install.yml
haproxy-install.yml
healthcheck-hosts.yml
healthcheck-infrastructure.yml
healthcheck-openstack.yml
infra-journal-remote.yml
listening-port-report.yml
lxc-containers-create.yml
lxc-containers-destroy.yml
lxc-hosts-setup.yml
memcached-install.yml
openstack-hosts-setup.yml
os-adjutant-install.yml
os-aodh-install.yml
os-barbican-install.yml
os-blazar-install.yml
os-ceilometer-install.yml
os-cinder-install.yml
os-cloudkitty-install.yml
os-designate-install.yml
os-glance-install.yml
os-gnocchi-install.yml
os-heat-install.yml
os-horizon-install.yml
os-ironic-install.yml
os-keystone-install.yml
os-magnum-install.yml
os-manila-install.yml
os-masakari-install.yml
os-mistral-install.yml
os-murano-install.yml
os-neutron-install.yml
os-nova-install.yml
os-octavia-install.yml
os-placement-install.yml
os-rally-install.yml
os-sahara-install.yml
os-senlin-install.yml
os-swift-install.yml
os-swift-sync.yml
os-tacker-install.yml
os-tempest-install.yml
os-trove-install.yml
os-zun-install.yml
qdrouterd-install.yml
rabbitmq-install.yml
repo-install.yml
security-hardening.yml
setup-everything.yml
setup-hosts.yml
setup-infrastructure.yml
setup-openstack.yml
unbound-install.yml
utility-install.yml
zookeeper-install.yml

I can see the change here but no inclusion of this above mentioned file:
https://github.com/openstack/openstack-ansible/commit/2302883430b43e5c26388a058909a20a003ccf04

This was following the steps in the following document:
https://docs.openstack.org/openstack-ansible/2023.2/admin/upgrades/major-upgrades.html

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi,

It feels that bootstrap-ansible.sh was either incomplete or failed during execution.

This playbook is part of openstack-ansible-plugins repo which is installed as a collection.

Thus, content and the playbook should be in /etc/ansible/ansible_collections/openstack/osa/playbooks

Potentially, try moving/deleting /etc/ansible folder and re-run bootstrap-ansible.sh

[zrsolis (zrsolis)]

I deleted the /etc/ansible directory and re-ran bootstrap-ansible.sh . It did not create the /etc/ansible/asnbile_collections/openstack/osa/playbooks directory and it looks like the openstack-ansible-plugins collection was skipped:

skipping: [localhost] => (item={'name': 'openstack.osa', 'source': 'https://opendev.org/openstack/openstack-ansible-plugins', 'type': 'git', 'version': 'dee122e2e972f98eac8dd5a717725db88c8f4f80'})

[Dmitriy Rabotyagov (noonedeadpunk)]

So, from what I can tell from this output - the SHA is correct and should include the patch containing the change:
https://opendev.org/openstack/openstack-ansible-plugins/commit/a62ff6732ce2605aad5fcb4a279fc887b3e10d91

But it's hard to tell if collection installation failed or not - would be good to have slightly more context around bootstrap-ansible.sh command.

Also, do you have `openstack.osa` in output of `ansible-galaxy collection list -p /etc/ansible`?

[zrsolis (zrsolis)]

d

ansible-galaxy collection list -p /etc/ansible

# /etc/ansible/ansible_collections
Collection Version
------------------------- -------
ansible.netcommon 5.2.0
ansible.posix 1.5.4
ansible.utils 2.11.0
community.crypto 2.15.1
community.general 7.3.0
community.mysql 3.7.2
community.rabbitmq 1.2.3
gluster.gluster 1.0.2
openstack.cloud 2.2.0
openstack.config_template 2.1.0
openstack.osa 1.0.0
openvswitch.openvswitch 2.1.1

Output of bootstrap-ansible.sh attached as txt file

[Dmitriy Rabotyagov (noonedeadpunk)]

So, it looks like collection is successfully installed based on this output.

Just out of interest, what would be the output of running

openstack-ansible openstack.osa.haproxy_service_config ?

[zrsolis (zrsolis)]

openstack-ansible openstack.osa.haproxy_service_config
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
[DEPRECATION WARNING]: DEFAULT_GATHER_SUBSET option, the module_defaults keyword is a more generic version and can apply
to all calls to the M(ansible.builtin.gather_facts) or M(ansible.builtin.setup) actions, use module_defaults instead.
This feature will be removed from ansible-core in version 2.18. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
ERROR! the playbook: openstack.osa.haproxy_service_config could not be found

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

[Dmitriy Rabotyagov (noonedeadpunk)]

Ok, that is really weird right now.
One thing that I can assume, is that you might have or collection version in some other path of the environment. So potentially running same with -vvv may disclose that.

And then I might have really nasty hack for you to try:

rm -rf /etc/ansible/ansible_collections/openstack/osa
git clone https://opendev.org/openstack/openstack-ansible-plugins /etc/ansible/ansible_collections/openstack/osa
cd /etc/ansible/ansible_collections/openstack/osa
git checkout stable/2023.2

And try to run openstack-ansible openstack.osa.haproxy_service_config -vvv after that....

But I believe there is some weirdness in the environment rather then generic bug.

[zrsolis (zrsolis)]

Well we got a bit farther with the hack:

PLAYBOOK: haproxy_service_config.yml *************************************************************************************
2 plays in /etc/ansible/ansible_collections/openstack/osa/playbooks/haproxy_service_config.yml
ERROR! The field 'hosts' has an invalid value, which includes an undefined variable. The error was: 'service_group' is undefined. 'service_group' is undefined

The error appears to be in '/etc/ansible/ansible_collections/openstack/osa/playbooks/haproxy_service_config.yml': line 25, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- hosts: "{{ service_group }}"
  ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

[Dmitriy Rabotyagov (noonedeadpunk)]

Ok, so playbook now in place as it's supposed to be. Like theoretically you should be able to proceed, but I'm pretty much uncertain if other collection and roles are in expected shape/versions either.

Don't you accidentally have some collection versions overrides in /etc/openstack_deploy/user-collection-requirements.yml?

I will also check log file you've provided one more time as I could miss smth there as was reading it from the phone.

[zrsolis (zrsolis)]

Turns out I did have that file ... as an override you had me create to resolve a different issue: https://bugs.launchpad.net/openstack-ansible/+bug/2024407

[Dmitriy Rabotyagov (noonedeadpunk)]

Yes, so since this issue was fixed and you are doing major upgrade, all such overrides, as well as different variables, must be reviewed carefully.

You should be able to remove the override and re-run bootstrap safely.