periodic-tripleo-ci-centos-8-standalone-upgrade-master failing, tripleo upgrade not identifed as an openstack command.

[Sandeep Yadav]

Description:_

periodic-tripleo-ci-centos-8-standalone-upgrade-master is failing with below error snippet:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/9bd4b10/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

~~~
2021-06-01 18:47:06 | Could not load 'tripleo_config_generate_ansible': No module named 'tripleo_common.actions'
2021-06-01 18:47:06 | Could not load 'tripleo_deploy': No module named 'tripleo_common.actions'
2021-06-01 18:47:06 | Could not load 'tripleo_upgrade': No module named 'tripleo_common.actions'
2021-06-01 18:47:08 | Could not load 'tripleo_config_generate_ansible': No module named 'tripleo_common.actions'
2021-06-01 18:47:08 | Could not load 'tripleo_deploy': No module named 'tripleo_common.actions'
2021-06-01 18:47:08 | Could not load 'tripleo_upgrade': No module named 'tripleo_common.actions'
2021-06-01 18:47:09 | openstack: 'tripleo upgrade --templates --local-ip=192.168.24.1/24 --control-virtual-ip=192.168.24.3 -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml -e /home/zuul/containers-prepare-parameters-upgrade.yaml -e /home/zuul/standalone_parameters_upgrade.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml --output-dir /home/zuul --standalone' is not an openstack command. See 'openstack --help'.
2021-06-01 18:47:09 | Did you mean one of these?
2021-06-01 18:47:09 | tripleo container image build
2021-06-01 18:47:09 | tripleo container image delete
2021-06-01 18:47:09 | tripleo container image hotfix
2021-06-01 18:47:09 | tripleo container image list
2021-06-01 18:47:09 | tripleo container image prepare
2021-06-01 18:47:09 | tripleo container image prepare default
2021-06-01 18:47:09 | tripleo container image push
2021-06-01 18:47:09 | tripleo container image show
2021-06-01 18:47:09 | tripleo launch heat
2021-06-01 18:47:09 | tripleo validator group info
2021-06-01 18:47:09 | tripleo validator list
2021-06-01 18:47:09 | tripleo validator run
2021-06-01 18:47:09 | tripleo validator show
2021-06-01 18:47:09 | tripleo validator show history
2021-06-01 18:47:09 | tripleo validator show parameter
2021-06-01 18:47:09 | tripleo validator show run
2021-06-01 18:47:09 | tsigkey create
2021-06-01 18:47:09 | tsigkey delete
2021-06-01 18:47:09 | tsigkey list
2021-06-01 18:47:09 | tsigkey set
2021-06-01 18:47:09 | tsigkey show
~~~

Job is failing since 29th May, build history:-
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master&project=openstack/tripleo-ci

[Rabi Mishra (rabi)]

Using newer tripleo-common (29th May) and python3-tripleoclient(27th May) is missing https://review.opendev.org/c/openstack/python-tripleoclient/+/792137

Should be fixed with newer python3-tripleoclient

python3-tripleo-common-16.1.0-0.20210529122603.9026107.el8.noarch
python3-tripleoclient-16.2.1-0.20210527122904.fea5819.el8.noarch

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/9bd4b10/logs/undercloud/var/log/extra/rpm-list.txt.gz

[Sandeep Yadav (sandeepyadav93)]

Hello Rabi,

We are hitting a similar issue in the Master tripleo component, which has the latest python-tripleoclient

https://logserver.rdoproject.org/86/31586/64/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-master/c08a30d/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
~~~
2021-06-08 21:04:42 | openstack: 'tripleo upgrade --templates --local-ip=192.168.24.1/24 --control-virtual-ip=192.168.24.3 -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml -e /home/zuul/containers-prepare-parameters-upgrade.yaml -e /home/zuul/standalone_parameters_upgrade.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml --output-dir /home/zuul --standalone' is not an openstack command. See 'openstack --help'.
~~~

https://logserver.rdoproject.org/86/31586/64/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-master/c08a30d/logs/undercloud/var/log/extra/package-list-installed.txt.gz
~~~
openstack-tripleo-common.noarch 16.1.0-0.20210605151902.7e1436b.el8 @tripleo
python3-tripleoclient.noarch 16.2.1-0.20210608151413.c8870f5.el8 @delorean-current
~~~

[Sandeep Yadav (sandeepyadav93)]

Setting promotion blocker flag as this is blocking tripleo component promotion in Master.

[Rabi Mishra (rabi)]

It's using master tripleo-common vs wallaby tripleoclient. I think because of the tag 16.2 in wallaby it's not upgrading tripleoclient.

[Sandeep Yadav (sandeepyadav93)]

I agree with Rabi observation:-

job is still using python3-tripleoclient.noarch version from wallaby

https://logserver.rdoproject.org/86/31586/64/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-master/c08a30d/logs/undercloud/var/log/extra/package-list-installed.txt.gz
~~~
python3-tripleoclient.noarch 16.2.1-0.20210608151413.c8870f5.el8 @delorean-current
~~~

Wallaby:-
https://trunk.rdoproject.org/centos8-wallaby/component/tripleo/component-ci-testing/
~~~
python-tripleoclient-16.2.1-0.20210608151413.c8870f5.el8.src.rpm
~~~
Master:-
https://trunk.rdoproject.org/centos8-master/component/tripleo/component-ci-testing/
~~~
python-tripleoclient-16.1.0-0.20210608041413.c4261c4.el8.src.rpm
~~~

[Sandeep Yadav (sandeepyadav93)]

https://bugs.launchpad.net/tripleo/+bug/1931223 might be a duplicate

[Marios Andreou (marios-b)]

also possibly duplicate of https://bugs.launchpad.net/tripleo/+bug/1930416 (and similar to earlier https://bugs.launchpad.net/tripleo/+bug/1921679 )

strangely undercloud-upgrade is not hitting this [1] but standalone-upgrade still is [2]

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-undercloud-upgrade-master
[2] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master

[Marios Andreou (marios-b)]

the (failing) standalone-upgrade is using 16.2.2 tripleoclient but the (passing) undercloud-upgrade is still with 16.1.0

        * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/dd2ddd0/logs/undercloud/var/log/extra/package-list-installed.txt.gz
        * python3-tripleoclient.noarch 16.2.2-0.20210609035220.48e5e28.el8 @delorean-current

        * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-undercloud-upgrade-master/cafe190/logs/undercloud/var/log/extra/package-list-installed.txt.gz
        * python3-tripleoclient.noarch 16.1.0-0.20210609013907.6def2b9.el8 @delorean-current

[Rabi Mishra (rabi)]

To me it looks like periodic-tripleo-ci-centos-8-undercloud-upgrade-master is deploying standalone and from the package versions it seems to be upgrading from master to master.

Install
-------
2021-06-10T01:32:23+0000 DEBUG ---> Package python3-tripleo-common.noarch 16.1.0-0.20210608215442.e7a4de4.el8 will be installed <<<<< latest master hash
2021-06-10T01:32:23+0000 DEBUG ---> Package python3-tripleoclient.noarch 16.1.0-0.20210609013907.6def2b9.el8 will be installed

Upgrade:
--------
2021-06-10T02:18:04+0000 DDEBUG Command: dnf update -y python*-tripleoclient openstack-tripleo-common tripleo-ansible openstack-tripleo-heat-templates
2021-06-10T02:18:04+0000 DDEBUG Installroot: /
....
2021-06-10T02:18:07+0000 DEBUG --> Starting dependency resolution
2021-06-10T02:18:07+0000 DEBUG --> Finished dependency resolution
2021-06-10T02:18:07+0000 DDEBUG timer: depsolve: 209 ms
2021-06-10T02:18:07+0000 INFO Dependencies resolved.
2021-06-10T02:18:07+0000 INFO Nothing to do. <<<< Not updating anything
2021-06-10T02:18:07+0000 INFO Complete!

[Marios Andreou (marios-b)]

I suspect this might be solved with https://review.opendev.org/c/openstack/releases/+/795948 via https://bugs.launchpad.net/tripleo/+bug/1931223/comments/12 i.e. we need a tag for master so it will try to install tripleoclient 17.x from/for master

[Marios Andreou (marios-b)]

We need a promotion of the tripleo component and it should clear the bug.

Now that https://review.opendev.org/c/openstack/releases/+/795948 merged we have a newer tripleoclient tagged in the tripleo component consistent repo there:

https://trunk.rdoproject.org/centos8-master/component/tripleo/consistent/
python-tripleoclient-17.0.1-0.20210615072306.7797524.el8.src.rpm 2021-06-15 07:24 338K

We need the component tripleo promotion as https://trunk.rdoproject.org/centos8-master/component/tripleo/current-tripleo/ - current-tripleo is still on python-tripleoclient-16.0.1-0.20210520134358.e86bccd.el8.src.rpm at time of writing (and we want the latest version from consistent)

[Marios Andreou (marios-b)]

Confirming comment #11 the tripleo component promotion will help us.

In today's run the standalone-upgrade job [1] went passed this error but failed on tempest and it is using the newer tripleoclient [2]

        * 2021-06-15T20:18:55+0000 DEBUG ---> Package python3-tripleoclient.noarch 17.0.1-0.20210615072306.7797524.el8 will be an upgrade

I am trying to get a re-run of that job with https://review.rdoproject.org/r/c/testproject/+/34130 to force that promotion let's see.

[1] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-master/5b497e2/job-output.txt
[2] https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-master/5b497e2/logs/undercloud/var/log/dnf.log.txt.gz

[Marios Andreou (marios-b)]

managed to get a promotion on the tripleo component (with https://review.rdoproject.org/r/c/testproject/+/34194 ) so we now have the right version of tripleoclient in the promoted-components:

        * https://trunk.rdoproject.org/centos8-master/component/tripleo/promoted-components/
        * python-tripleoclient-17.0.1-0.20210616113250.15f2539.el8.src.rpm 2021-06-16 11:34 338K

So now we need a master promotion so we can get that ^^^ into current-tripleo and we can close the bug