Victoria -> Wallaby Undercloud upgrade fails - The template version is invalid: "heat_template_version: wallaby"

[Harald Jensås]

I updated with: (Pretty much what docs says: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/upgrade/undercloud.html)
 sudo yum update python3-tripleoclient openstack-tripleo-common openstack-tripleo-heat-templates

2021-05-18 18:11:17.759 121577 INFO tripleoclient.v1.undercloud.UpgradeUndercloud [ ] User confirmed action.
2021-05-18 18:11:17.760 121577 WARNING tripleoclient.v1.undercloud.UpgradeUndercloud [ ] Updating necessary packages: python3-tripleoclient openstack-tripleo-common openstack-tripleo-heat-templates openstack-tripleo-validations tripleo-ansible
2021-05-18 18:11:21.627 121577 WARNING tripleoclient.v1.undercloud.UpgradeUndercloud [ ] Last metadata expiration check: 0:00:31 ago on Tue 18 May 2021 06:10:47 PM CEST.
Dependencies resolved.
=================================================================================================================
 Package Arch Version Repository Size
=================================================================================================================
Upgrading:
 openstack-tripleo-validations noarch 14.1.1-0.20210517130847.adf99a7.el8 delorean-component-validation 197 k
 tripleo-ansible noarch 3.1.1-0.20210518015826.8fd3e17.el8 delorean-component-tripleo 538 k

Transaction Summary
=================================================================================================================
Upgrade 2 Packages

Total download size: 735 k
Downloading Packages:
(1/2): openstack-tripleo-validations-14.1.1-0.2 204 kB/s | 197 kB 00:00
(2/2): tripleo-ansible-3.1.1-0.20210518015826.8 461 kB/s | 538 kB 00:01
--------------------------------------------------------------------------------
Total 626 kB/s | 735 kB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing : 1/1
  Upgrading : openstack-tripleo-validations-14.1.1-0.2021051713084 1/4
  Upgrading : tripleo-ansible-3.1.1-0.20210518015826.8fd3e17.el8.n 2/4
  Cleanup : openstack-tripleo-validations-13.2.1-0.2021051203540 3/4
  Cleanup : tripleo-ansible-2.2.1-0.20210510133906.ac6cc2a.el8.n 4/4
  Running scriptlet: tripleo-ansible-2.2.1-0.20210510133906.ac6cc2a.el8.n 4/4
  Verifying : tripleo-ansible-3.1.1-0.20210518015826.8fd3e17.el8.n 1/4
  Verifying : tripleo-ansible-2.2.1-0.20210510133906.ac6cc2a.el8.n 2/4
  Verifying : openstack-tripleo-validations-14.1.1-0.2021051713084 3/4
  Verifying : openstack-tripleo-validations-13.2.1-0.2021051203540 4/4
Installed products updated.

Upgraded:
  openstack-tripleo-validations-14.1.1-0.20210517130847.adf99a7.el8.noarch
  tripleo-ansible-3.1.1-0.20210518015826.8fd3e17.el8.noarch

Complete!

<------------- SNIP --------------->

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-auw_z2lq

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

2021-05-18 18:11:35.912 122770 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ admin] Exception: ERROR: The template version is invalid: "heat_template_version: wallaby". "heat_template_version" should be one of: 2013-05-23, 2014-10-16, 2015-04-30, 2015-10-15, 2016-04-08, 2016-10-14, 2017-02-24, 2017-09-01, 2018-03-02, 2018-08-31, newton, ocata, pike, queens, rocky
None
2021-05-18 18:11:35.914 122770 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ admin] None
2021-05-18 18:11:35.969 122770 ERROR openstack [ admin] ERROR: The template version is invalid: "heat_template_version: wallaby". "heat_template_version" should be one of: 2013-05-23, 2014-10-16, 2015-04-30, 2015-10-15, 2016-04-08, 2016-10-14, 2017-02-24, 2017-09-01, 2018-03-02, 2018-08-31, newton, ocata, pike, queens, rocky
None
2021-05-18 18:11:36.124 121577 ERROR tripleoclient.v1.undercloud.UpgradeUndercloud [ ]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

An error has occured while deploying the Undercloud.

See the previous output for details about what went wrong.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

2021-05-18 18:11:36.124 121577 ERROR tripleoclient.v1.undercloud.UpgradeUndercloud [ ] Command '['sudo', '--preserve-env', 'openstack', 'tripleo', 'deploy', '--standalone', '--standalone-role', 'Undercloud', '--stack', 'undercloud', '--local-domain=lab.example.com', '--local-ip=172.20.0.1/26', '--templates=/usr/share/openstack-tripleo-heat-templates/', '--networks-file=/usr/share/openstack-tripleo-heat-templates/network_data_undercloud.yaml', '--upgrade', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/lifecycle/undercloud-upgrade-prepare.yaml', '--heat-native', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/undercloud.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/use-dns-for-vips.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/masquerade-networks.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/undercloud-remove-novajoin.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/public-tls-undercloud.yaml', '--public-virtual-ip', '172.20.0.3', '--control-virtual-ip', '172.20.0.2', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/services/undercloud-haproxy.yaml', '--deployment-user', 'stack', '--output-dir=/home/stack/tripleo-deploy/undercloud', '--cleanup', '-e', '/home/stack/tripleo-deploy/undercloud/tripleo-config-generated-env-files/undercloud_parameters.yaml', '-e', '/usr/share/openstack-tripleo-heat-templates/environments/tripleo-validations.yaml', '--log-file=install-undercloud.log', '-e', '/usr/share/openstack-tripleo-heat-templates/undercloud-stack-vstate-dropin.yaml']' returned non-zero exit status 1.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/792008

[Rabi Mishra (rabi)]

When we update openstack-tripleo-heat-templates to wallaby, we need to update openstack-heat-monolith as well. We don't need it for victoria, needs to be backported to wallaby only.

[Rabi Mishra (rabi)]

Proposed https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33770 which we would backport to wallaby.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Harald Jensås <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/792008
Reason: Setting up the deps in the rpm spec is the better solution. https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33770

[Sandeep Yadav (sandeepyadav93)]

V->W standalone upgrade job is still failing,

Logs:-
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/da2fea1/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

~~~
2021-05-26 17:15:55 | heatclient.exc.HTTPBadRequest: ERROR: The template version is invalid: "heat_template_version: wallaby". "heat_template_version" should be one of: 2013-05-23, 2014-10-16, 2015-04-30, 2015-10-15, 2016-04-08, 2016-10-14, 2017-02-24, 2017-09-01, 2018-03-02, 2018-08-31, newton, ocata, pike, queens, rocky
2021-05-26 17:15:55 | None
~~~

rpm version on standalone host:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/da2fea1/logs/undercloud/var/log/extra/package-list-installed.txt.gz
~~~
python3-tripleoclient.noarch 16.2.1-0.20210522155922.2006c0f.el8 @delorean-current
~~~

I have extracted the rpm locally, its spec file contain changes from https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33783/2/python-tripleoclient.spec

$ wget https://trunk.rdoproject.org/centos8-wallaby/component/tripleo/tripleo-ci-testing/python-tripleoclient-16.2.1-0.20210522155922.2006c0f.el8.src.rpm

$ rpm2cpio python-tripleoclient-16.2.1-0.20210522155922.2006c0f.el8.src.rpm | cpio -civ '*.spec'
python-tripleoclient.spec
659 blocks

$ view python-tripleoclient.spec
~~~
.
117 Requires: openstack-heat-api >= 16.0.0
118 Requires: openstack-heat-engine >= 16.0.0
119 Requires: openstack-heat-monolith >= 16.0.0
~~~

[Marios Andreou (marios-b)]

started seeing this in centos-8-undercloud-upgrade-wallaby job being added now and I filed a duplicate bug there https://bugs.launchpad.net/tripleo/+bug/1930018

basically we are using 'old' version of heat packages for victoria instaed of wallaby pointers in bug/1930018

[Marios Andreou (marios-b)]

trying a sanity check with explicit upgrade of the heat packages there https://review.opendev.org/c/openstack/python-tripleoclient/+/794216 and tested (depends-on) at v3 https://review.opendev.org/c/openstack/puppet-tripleo/+/793124

[Sandeep Yadav (sandeepyadav93)]

Heat related packages are not upgrading as dependency when "dnf update -y python*-tripleoclient" is ran

Today's failing job:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/8879de4/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

~~~
ERROR: The template version is invalid: "heat_template_version: wallaby". "heat_template_version" should be one of: 2013-05-23, 2014-10-16, 2015-04-30, 2015-10-15, 2016-04-08, 2016-10-14, 2017-02-24, 2017-09-01, 2018-03-02, 2018-08-31, newton, ocata, pike, queens, rocky
~~~

Rpm version of python3-tripleoclient:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/8879de4/logs/undercloud/var/log/extra/package-list-installed.txt.gz

~~~
python3-tripleoclient.noarch 16.2.1-0.20210603174615.c5b1e0c.el8 @delorean-current
~~~

Locally extracted python-tripleoclient-16.2.1-0.20210603174615.c5b1e0c.el8.src.rpm,

Contains code from both https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33786/1/python-tripleoclient.spec and https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33783/2/python-tripleoclient.spec

~~~
$ wget https://trunk.rdoproject.org/centos8-wallaby/component/tripleo/tripleo-ci-testing/python-tripleoclient-16.2.1-0.20210603174615.c5b1e0c.el8.src.rpm

2021-06-07 21:06:55 (168 KB/s) - ‘python-tripleoclient-16.2.1-0.20210603174615.c5b1e0c.el8.src.rpm’ saved [343370/343370]
~~~
$ rpm2cpio python-tripleoclient-16.2.1-0.20210603174615.c5b1e0c.el8.src.rpm | cpio -civ '*.spec'

view:-

~~~
105 Requires: openstack-tripleo-common >= 15.2.0
106 Requires: python3-tripleo-common >= 15.2.0
107 Requires: os-net-config
108 Requires: rsync
109
110 # Dependency for correct validations
111 Requires: openstack-tripleo-validations
112
113 Requires: buildah
114 Requires: podman
115 Requires: %{ovs_dep}
116 Requires: openstack-heat-agents >= 1.6.0
117 Requires: openstack-heat-api >= 16.0.0
118 Requires: openstack-heat-engine >= 16.0.0
119 Requires: openstack-heat-monolith >= 16.0.0
120 Requires: openstack-tripleo-heat-templates >= 14.1.1
121 Requires: puppet-tripleo >= 14.1.0
~~~

Seem, Heat related packages are not upgrading as dependency when "dnf update -y python*-tripleoclient" is ran

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/8879de4/logs/undercloud/var/log/dnf.log.txt.gz

~~~
dnf update -y python*-tripleoclient openstack-tripleo-common tripleo-ansible openstack-tripleo-heat-templates

Upgrading:
 openstack-tripleo-common noarch 15.2.1-0.20210606104239.d39a328.el8 delorean-current 41 k
 openstack-tripleo-common-containers noarch 15.2.1-0.20210606104239.d39a328.el8 delorean-current 69 k
 openstack-tripleo-heat-templates noarch 14.1.2-0.20210605053403.fc352f9.el8 de...

[Alex Schultz (alex-schultz)]

https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/34024

[Marios Andreou (marios-b)]

I didn't test the distgit update from comment #9 yet (thanks Alex) however for the record, the test from comment #7 above was successful with the explicit upgrade of heat-* logs there it upgraded the heat packages and got further (but now fails on a new bug :/)

        * https://7e1855dd4f4cae214e00-4bc1af96845a4b2efb646674c5f1b41c.ssl.cf2.rackcdn.com/793124/4/check/tripleo-ci-centos-8-undercloud-upgrade-wallaby/e86b362/logs/undercloud/home/zuul/undercloud_upgrade.log
        * 2021-06-04 15:53:01 | Upgraded:
2021-06-04 15:53:01 | openstack-heat-agents-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | openstack-heat-api-1:16.0.1-0.20210425080922.7b9ffc5.el8.noarch
2021-06-04 15:53:01 | openstack-heat-common-1:16.0.1-0.20210425080922.7b9ffc5.el8.noarch
2021-06-04 15:53:01 | openstack-heat-engine-1:16.0.1-0.20210425080922.7b9ffc5.el8.noarch
2021-06-04 15:53:01 | openstack-heat-monolith-1:16.0.1-0.20210425080922.7b9ffc5.el8.noarch
2021-06-04 15:53:01 | openstack-tripleo-validations-14.1.2-0.20210531055400.8821e60.el8.noarch
2021-06-04 15:53:01 | python-oslo-policy-lang-3.7.0-0.20210315125528.d853485.el8.noarch
2021-06-04 15:53:01 | python3-croniter-0.3.35-1.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-ansible-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-apply-config-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-docker-cmd-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-hiera-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-json-file-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-heat-agent-puppet-2.2.1-0.20210404092834.a0be75b.el8.noarch
2021-06-04 15:53:01 | python3-oslo-policy-3.7.0-0.20210315125528.d853485.el8.noarch
2021-06-04 15:53:01 | tripleo-ansible-3.1.2-0.20210603083915.7062ca0.el8.noarch

(compared to log from bug where it couldn't even start the heat stack create @ https://b32d03500643014759d0-c3c850383d3c4b677d9bd09df1596ea2.ssl.cf1.rackcdn.com/793135/1/check/tripleo-ci-centos-8-undercloud-upgrade-wallaby/5b9c83d/logs/undercloud/home/zuul/undercloud_upgrade.log)

BUT now hitting new bug?

        * https://7e1855dd4f4cae214e00-4bc1af96845a4b2efb646674c5f1b41c.ssl.cf2.rackcdn.com/793124/4/check/tripleo-ci-centos-8-undercloud-upgrade-wallaby/e86b362/logs/undercloud/home/zuul/undercloud_upgrade.log
        * 2021-06-04 15:59:58 | 2021-06-04 15:59:58.017108 | fa163ede-1521-137e-8395-00000000029c | FATAL | migrate existing introspection data | undercloud | error={"changed": true, "cmd": "podman exec -u root ironic_inspector ironic-inspector-migrate-data --from swift --to database --config-file /etc/ironic-inspector/inspector.conf\n", "delta": "0:01:42.288521", "end": "2021-06-04 15:59:57.982282", "...

[Marios Andreou (marios-b)]

the periodic standalone-upgrade job (which was also hitting this issue e.g. [1]) is now green after the change at [2] merged. The last three runs at [3] are now green eg [4] is the latest example

I will try a recheck @ [5] to confirm if this is also now fixed for check/gate as well (and to confirm the undercloud upgrade is fixed too, not just standalone-upgrade). Then we can close out this bug as fix-released.

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby/43b3ec4/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
[2] https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/34035
[3] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-wallaby
[4] https://review.rdoproject.org/zuul/build/b8d92959530e4b6cba773fb5c8fa676e
[5] https://review.opendev.org/c/openstack/tripleo-common/+/793123