master undercloud upgrade fails "not an openstack command"

[Marios Andreou]

At [1] the master undercloud upgrade job is failing at the start of the upgrade with error like:

 2021-05-31 14:31:06 | openstack: 'tripleo deploy --standalone --standalone-role Undercloud --stack undercloud --local-domain=localdomain --local-ip=192.168.24.1/24 --templates=/usr/share/openstack-tripleo-heat-templates --networks-file=/usr/share/openstack-tripleo-heat-templates/network_data_undercloud.yaml -y --upgrade -e /usr/share/openstack-tripleo-heat-templates/environments/lifecycle/undercloud-upgrade-prepare.yaml --heat-native -e /usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml -e /home/zuul/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/undercloud-remove-novajoin.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/public-tls-undercloud.yaml --public-virtual-ip 192.168.24.2 --control-virtual-ip 192.168.24.3 -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/undercloud-haproxy.yaml --deployment-user zuul --output-dir=/home/zuul/tripleo-deploy/undercloud -e /home/zuul/tripleo-deploy/undercloud/tripleo-config-generated-env-files/undercloud_parameters.yaml --hieradata-override=/home/zuul/./hieradata-overrides-t-h-t-undercloud.yaml -e /home/zuul/skip_rhel_release.yaml -e /usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml' is not an openstack command. See 'openstack --help'.
 2021-05-31 14:31:06 | Did you mean one of these?

This job is being moved out of deprecated and back into the master gate with [2]

[1] http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_927/793393/2/check/tripleo-ci-centos-8-undercloud-upgrade/927d492/logs/undercloud/home/zuul/undercloud_upgrade.log
[2] https://review.opendev.org/c/openstack/tripleo-ci/+/793393/

[Marios Andreou (marios-b)]

maybe duplicate https://bugs.launchpad.net/tripleo/+bug/1921679 not sure yet

[wes hayutin (weshayutin)]

You may need to make a similar change for wallaby as what was done below

https://review.opendev.org/c/openstack/tripleo-image-elements/+/788003

[Marios Andreou (marios-b)]

thanks for the pointer weshay but looks like that one is already on wallaby there https://opendev.org/openstack/tripleo-image-elements/src/commit/74acb5493fe9da1e5063e00cd82f38dcf9e54d63/elements/os-apply-config/install.d/os-apply-config-source-install/10-os-apply-config#L20

the master fix merged before we branched wallaby

[Marios Andreou (marios-b)]

not sure we can mark this duplicate of https://bugs.launchpad.net/tripleo/+bug/1921679 because the fix from 1921679 is already present here and the problem persists.

In 1921679 the versions of pbr and importlib-metadata in particular are seen to be causing the problem. In the failing undercloud master upgrade job the versions are pbr 5.5.1 and importlib-metadata 1.7.0:

        * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_927/793393/2/check/tripleo-ci-centos-8-undercloud-upgrade/927d492/logs/undercloud/var/log/extra/pip.txt
        * importlib-metadata (1.7.0)
        * pbr (5.5.1)

        * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_927/793393/2/check/tripleo-ci-centos-8-undercloud-upgrade/927d492/logs/undercloud/var/log/extra/package-list-installed.txt
        * python3-importlib-metadata.noarch 1.7.0-1.el8 @delorean-wallaby-testing
        * python3-pbr.noarch 5.5.1-1.el8 @delorean-wallaby-testing

This is enforced on rdo build side with https://github.com/redhat-openstack/rdoinfo/blob/3df44d36985fb6ccbd349239d6fddbd281b0d3f5/buildsys-tags/cloud8s-openstack-xena-testing.yml#L559

These are the same versions used by wallaby/victoria.

[Marios Andreou (marios-b)]

periodic line is green on this (last few builds at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-undercloud-upgrade-master )

[Marios Andreou (marios-b)]

moving fix released - we are no longer seeing the issue (master undercloud upgrade is now failing on something else :/ but gets passed this point)