tripleo-ci-centos-9-undercloud-upgrade is failing check/gate - Depsolve Error occurred - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.39-2.el9s.x86_64

[Ronelle Landy]

tripleo-ci-centos-9-undercloud-upgrade is now failing undercloud upgrade with the following error:

 FATAL | Update all packages | undercloud | error={"changed": false, "failures": [], "msg": "Depsolve Error occurred: \n Problem: package python3-oslo-db-12.1.0-0.20220929080853.7f3647b.el9.noarch requires python3-sqlalchemy >= 1.4.0, but none of the providers can be installed\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.23-1.1.el9s.x86_64\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.31-1.el9s.x86_64\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.36-1.el9s.x86_64\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.39-2.el9s.x86_64\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.40-1.el9s.x86_64\n - package python3-sqlalchemy13-1.3.24-2.el9s.x86_64 conflicts with python3-sqlalchemy >= 1.4 provided by python3-sqlalchemy-1.4.37-3.el9.x86_64\n - cannot install the best update candidate for package python3-oslo-db-8.5.2-0.20220905103333.26fd6fb.el9.noarch\n - problem with installed package python3-sqlalchemy13-1.3.24-2.el9s.x86_64", "rc": 1, "results": []}

Full log is included below:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b4d/854178/11/check/tripleo-ci-centos-9-undercloud-upgrade/b4d0fed/logs/undercloud/home/zuul/undercloud_upgrade.log

This failure started on 10/07:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade+&skip=0

Other example logs:

https://4dba791ef4de9ad6be32-8f8891a779b543135fbad241baccf135.ssl.cf2.rackcdn.com/854337/11/check/tripleo-ci-centos-9-undercloud-upgrade/e3f28d6/logs/undercloud/home/zuul/undercloud_install.log

https://0926a9cee607a43da607-50773721ab99e0a8e92d78c395ea477f.ssl.cf2.rackcdn.com/849024/15/check/tripleo-ci-centos-9-undercloud-upgrade/f6785d5/logs/undercloud/home/zuul/undercloud_upgrade.log

[Ronelle Landy (rlandy)]

https://bugs.launchpad.net/tripleo/+bug/1989795 with related CIX card:

https://trello.com/c/GQHuQTmi/2711-cixlp1989795tripleociproa-periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby-is-failing-deploy-failed-containers-design

show the details of why python3-sqlalchemy13-1.3.24-2.el9s.x86_64.rpm was added.

https://review.opendev.org/c/openstack/tripleo-quickstart/+/858030 excludes python3-sqlalchemy-1.4.37-3.el9.x86_64 from wallaby installs.

[Jakob Meng (jm1337)]

As per analysis from @marios and @amorelej, we are excluding python3-sqlalchemy-1.4.37 but now there is 1.4.40, so our exclude is not doing anything. But [2] should fix it which got merged on 2022-10-07. So we need a new c9 wallaby promotion to fix this issue, last promotion was on 2022-10-06.

[1] https://opendev.org/openstack/tripleo-quickstart/src/commit/dc26b849018e3d5f2ecfe625fca7264877fe75ae/config/release/tripleo-ci/CentOS-9/wallaby.yml#L191
[2] https://review.rdoproject.org/r/c/openstack/oslo-db-distgit/+/45589

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810

[Marios Andreou (marios-b)]

proper fix for https://bugs.launchpad.net/tripleo/+bug/1989795 and also the issue seen here is at [1].

To get that however we also need promotion for wallaby/9.

For now we can expand the regex for the package exclude with [2] " Exclude python3-sqlalchemy-1.4.*x86_64 temporarily"

[1] https://review.rdoproject.org/r/q/topic:sqlalchemy13
[2] https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810

[Alfredo Moralejo (amoralej)]

The issue is related to the introduction of python-sqlalchemy13. When doing the full update python3-sqlalchemy-1.4 is not seen as an update of python3-sqlalchemy13 but as the installation of a new package that requires removal of python3-sqlalchemy13.

dnf can manage these situation with the --allowerasing option which is disabled by default.

This option is in ansible dnf module since 2.10:

https://docs.ansible.com/ansible/latest/collections/ansible/builtin/dnf_module.html#parameter-allowerasing

So we should add it in the task "Update all packages" in upgrade_tasks_step3.yaml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/860824

[Marios Andreou (marios-b)]

update - as discussed in irc with amoralej & jm1

we cannot use the temp/workaround with master since we actually need 1.4+ there

commented at the proposed workaround https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810/2#message-ce0249b7d214c9bb26985b5351904f48ec5b88ea (we may yet keep some of that? the wallaby bits? tbd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/860824
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/665391f63f7ada02941d5dd21a66b29a513f716d
Submitter: "Zuul (22348)"
Branch: master

commit 665391f63f7ada02941d5dd21a66b29a513f716d
Author: Alfredo Moralejo <email address hidden>
Date: Mon Oct 10 13:07:01 2022 +0200

    Enable allowerasing in undercloud upgrade from wallaby

    After adding python-sqlalchemy13 in Wallaby [1], the upgrade process
    need to remove python3-sqlalchemy13 package to install
    python3-sqlalchemy-1.4. This can be handled by dnf but requires to
    enable the option --allowerasing which is disabled by default:

    [1] https://review.rdoproject.org/r/c/rdoinfo/+/45546

    Closes-Bug: #1992305
    Change-Id: I4070723741fdf8e5dc27e93f620d7206b6e7893c

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Marios Andreou <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/860810
Reason: nope! that one instead https://review.opendev.org/c/openstack/tripleo-heat-templates/+/860824 :)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 17.0.0 release.