Tracker Bug: Tempest fails with packaging error - python-oslo-db-tests

[Ronelle Landy]

validate-tempest role is failing in Phase 1 tests running Pike due to a packaging error regarding python-oslo-db-tests:

19:46:35 TASK [validate-tempest : Install openstack services tempest plugins] ***********
...
19:46:57 failed: [undercloud] (item=[u'python-keystone-tests', u'python-neutron-tests']) => {"changed": true, "failed": true, "item": ["python-keystone-tests", "python-neutron-tests"], "msg": "Error: Package: python-oslo-db-tests-4.25.1-0.20171121155051.71607d5.el7.centos.noarch (delorean)\n Requires: python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos\n Installed: python2-oslo-db-4.25.1-1.el7.noarch (@delorean-deps)\n python-oslo-db = 4.25.1-1.el7\n Available: python2-oslo-db-4.25.1-0.20171121155051.71607d5.el7.centos.noarch (delorean)\n python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos\n", "rc": 1, "results": ["Loaded plugins: fastestmirror, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror4.ci.centos.org\n * extras: mirror4.ci.centos.org\n * updates: mirror4.ci.centos.org\n1263 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package python-keystone-tests.noarch 1:12.0.1-0.20171116210858.d0721d7.el7.centos will be installed\n--> Processing Dependency: python-keystone-tests-tempest for package: 1:python-keystone-tests-12.0.1-0.20171116210858.d0721d7.el7.centos.noarch\n---> Package python-neutron-tests.noarch 1:11.0.3-0.20171217171436.adc344c.el7.centos will be installed\n--> Processing Dependency: python-ddt >= 1.0.1 for package: 1:python-neutron-tests-11.0.3-0.20171217171436.adc344c.el7.centos.noarch\n--> Processing Dependency: python-oslo-db-tests >= 4.10.0 for package: 1:python-neutron-tests-11.0.3-0.20171217171436.adc344c.el7.centos.noarch\n--> Processing Dependency: python-oslotest >= 1.10.0 for package: 1:python-neutron-tests-11.0.3-0.20171217171436.adc344c.el7.centos.noarch\n--> Processing Dependency: python-testresources >= 0.2.4 for package: 1:python-neutron-tests-11.0.3-0.20171217171436.adc344c.el7.centos.noarch\n--> Processing Dependency: python-testscenarios >= 0.4 for package: 1:python-neutron-tests-11.0.3-0.20171217171436.adc344c.el7.centos.noarch\n--> Running transaction check\n---> Package python-oslo-db-tests.noarch 0:4.25.1-0.20171121155051.71607d5.el7.centos will be installed\n--> Processing Dependency: python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos for package: python-oslo-db-tests-4.25.1-0.20171121155051.71607d5.el7.centos.noarch\n---> Package python-testresources.noarch 0:0.2.7-6.el7 will be installed\n---> Package python-testscenarios.noarch 0:0.4-4.el7 will be installed\n---> Package python2-ddt.noarch 0:1.0.1-2.el7 will be installed\n---> Package python2-keystone-tests-tempest.noarch 0:0.0.1-0.20170824151656.fe269f2.el7.centos will be installed\n---> Package python2-oslotest.noarch 0:2.17.1-0.20171121160645.aea2b5c.el7.centos will be installed\n--> Processing Dependency: python-mox3 >= 0.7.0 for package: python2-oslotest-2.17.1-0.20171121160645.aea2b5c.el7.centos.noarch\n--> Running transaction check\n---> Package python-oslo-db-tests.noarch 0:4.25.1-0.20171121155051.71607d5.el7.centos will be installed\n--> Processing Dependency: python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos for package: python-oslo-db-tests-4.25.1-0.20171121155051.71607d5.el7.centos.noarch\n---> Package python2-mox3.noarch 0:0.23.0-0.20170811075950.812983a.el7.centos will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}

The full log can be found at:
https://ci.centos.org/job/tripleo-quickstart-promote-pike-rdo_trunk-minimal/104/console

[Ronelle Landy (rlandy)]

The error is first recorded in tests on Friday 29 December 2017

[Alex Schultz (alex-schultz)]

For clarity,
rror: Package: python-oslo-db-tests-4.25.1-0.20171121155051.71607d5.el7.centos.noarch (delorean)
           Requires: python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos
           Installed: python2-oslo-db-4.25.1-1.el7.noarch (@delorean-deps)
               python-oslo-db = 4.25.1-1.el7
           Available: python2-oslo-db-4.25.1-0.20171121155051.71607d5.el7.centos.noarch (delorean)
               python-oslo-db = 4.25.1-0.20171121155051.71607d5.el7.centos

Seems to be a version issue with 4.25.1-0 vs 4.25.1-1

[Alan Pevec (apevec)]

python-oslo-db-4.25.1-1.el7 is in deplorean-deps repo since Nov 24th
and since Oslo are pinned in rdoinfo, trunk RPMs stay on Version-0.* pre-release versioning.
But something else must have changed, we always had RPM-wise newer NVRs in deps repo with release RPM builds, in the past trunk repo had Yum priority set so it was always winning.

[Alan Pevec (apevec)]

Priority is set for the trunk repo:
https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-pike-rdo_trunk-minimal-104/undercloud/etc/yum.repos.d/delorean.repo.gz
and yum-plugin-priorities is installed:
https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-pike-rdo_trunk-minimal-104/undercloud/var/log/extra/rpm-list.txt.gz

We need to figure out why python2-oslo-db-4.25.1-1.el7.noarch from delorean-deps is installed, I'd guess yum priorities are not taken into account during image build?

[yatin (yatinkarel)]

Reference log for build image:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/360ed8d/undercloud/home/jenkins/overcloud_image_build.log.txt.gz

Looks like we are affected by: https://bugs.launchpad.net/tripleo/+bug/1727985 during image build. Trying to see if that's the actual cause.

[yatin (yatinkarel)]

I checked locally using the patch(https://review.openstack.org/#/c/516765/) packages from trunk repo are installed during image build.
Cherry-picked it and one another(Fix for rhel) to stable/pike:- https://review.openstack.org/531107

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/531107
Reason: The gate is currently timeouting, we need https://review.openstack.org/#/c/531352/ to improve the situation. I'll restore the patch once the gate is stable again. Please do not recheck or restore this patch, I'll take care of it. Thanks for your patience.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531107
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=a2422e5832baf0c86ff1e1b27d23418f8c87b57b
Submitter: Zuul
Branch: stable/pike

commit a2422e5832baf0c86ff1e1b27d23418f8c87b57b
Author: Alfredo Moralejo <email address hidden>
Date: Tue Oct 31 19:36:07 2017 +0100

    Install yum-plugin-priorities in pre-install phase

    In some cases yum repositories are using priorities configuration
    to force OpenStack packages to be installed from the desired
    repository.

    To work properly, priorities configuration requires the package
    yum-plugin-priorities to be installed before doing the actual
    OpenStack packages installation.

    This patch adds yum-plugin-priorities in pre-install phase for
    overcloud-base element wich is always used when creating
    tripleo images.

    Change-Id: I9c9b50d35d411f85a4cbe33ffab4c71d40dfb0f1
    Closes-Bug: #1740934
    Closes-Bug: #1727985
    (cherry picked from commit cc606ef52757adb438639f736bb505d4ffd2c56b)

    yum-plugin-priorities should not be enabled on RHEL

    Change-Id: Ic3721f0fbd35eb421d439e53ff8418293f83d4af
    (cherry picked from commit 82d0a62d85fafae218414d551191dc1fd3f639d5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-puppet-elements 7.0.4 release.