puppet package versions break ocata deployments and upgrades to pike

[Steven Hardy]

Local testing with tripleo.sh --repo-setup shows we have issues with a number of puppet modules where packages installed aren't the latest so deployment/upgrade fails, here's an example:

[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-openstacklib
puppet-openstacklib-10.3.0-1.el7.noarch

[root@overcloud-controller-0 ~]# repoquery puppet-openstacklib --repoid=delorean-ocata-testing
puppet-openstacklib-0:10.3.0-1.el7.noarch

[root@overcloud-controller-0 ~]# repoquery puppet-openstacklib --repoid=delorean-current
puppet-openstacklib-0:10.3.0-0.20170223102909.b3d0590.el7.centos.noarch

Here we can see we've installed the version from delorean-deps, instead of the correct delorean-current version, which breaks upgrade like this:

  deploy_stderr: |
    exception: connect failed
    Error: Evaluation Error: Error while evaluating a Function Call, os_transport_url(): port should be a String at /etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp:102:37 on node overcloud-controller-0.localdomain

because we don't have https://review.openstack.org/#/c/436401/

We can see this set priorities but the newer release is installed anyway:

[root@overcloud-controller-0 ~]# rpm -qa | grep yum-plugin-priorities
yum-plugin-priorities-1.1.31-40.el7.noarch

[root@overcloud-controller-0 ~]# cat /etc/yum.repos.d/delorean.repo | grep -e priority -e url
baseurl=https://trunk.rdoproject.org/centos7/d1/a9/d1a9d912723614364e8ce807437dd9503bd30879_09945cb4
priority=20
[root@overcloud-controller-0 ~]# cat /etc/yum.repos.d/delorean-deps.repo | grep -e priority -e url
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-ocata/
baseurl=http://cbs.centos.org/repos/cloud7-openstack-common-pending/$basearch/os/
baseurl=http://mirror.centos.org/centos/7/virt/$basearch/kvm-common/

[Steven Hardy (shardy)]

The same problem exists for deployment as puppet-tripleo is outdated, ref https://review.openstack.org/#/c/439759/

Discussion with apevec in #rdo indicates we need to do some rework to the packaging to fix this, and I'm still not clear why we don't see the same issues in CI

[Steven Hardy (shardy)]

This is now impacting CI, as upgrade testing ignores the delorean-ci built package:

http://logs.openstack.org/15/442315/2/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/5130095/logs/yum_mirror_error.txt.gz

We built puppet-tripleo-6.2.0-0.20170307123624.8493b34.el7.centos.noarch.rpm due to a Depends-On, but on the node we have:

puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch

Which I think comes from delorean-deps, and means we ignore the Depends-On package.

[Alfredo Moralejo (amoralej)]

Problem is in the logic used by DLRN to create NVR for puppet modules, as it uses version from git tags instead of PBR + pre-release release:

https://github.com/openstack-packages/DLRN/blob/master/scripts/build_rpm.sh#L33

For openstack puppet modules which implements PBR i think we could move to use setup.py --version as we do for python modules.

For non-openstack puppet modules we'd need to figure out something different.

We'll uppdate this LP with any progress in fixing it in DLRN.

[Javier Peña (jpena-c)]

https://review.rdoproject.org/r/5642 contains a potential fix for NVR generation in Puppet OpenStack modules (or any other with PBR support).

Once it is merged, we will need to rebuild the puppet-* packages, but that can be automated.

[Steven Hardy (shardy)]

Thanks for the update - I think fixing this for puppet-* OpenStack modules will be sufficient at this point.

[Alfredo Moralejo (amoralej)]

I think we may be mixing several issues. Right now, master branch is behind stable/ocata in terms of upstream release:

[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ python setup.py --version
6.2.1.dev46
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ git checkout stable/ocata
Branch stable/ocata set up to track remote branch stable/ocata from origin.
Switched to a new branch 'stable/ocata'
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ python setup.py --version
6.3.1.dev3
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$

This is something to be fixed upstream.

Note that puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch is not coming from delorean-deps but from delorean or delorean-current.

Anyway, I'm still trying to understand why priorities are not working as expected.

[Javier Peña (jpena-c)]

It turns out https://review.rdoproject.org/r/5642 broke something else, so it had to be reverted. https://review.rdoproject.org/r/5646 should be a permanent fix, but I'll take it through a full round of checks to be 100% sure it is not creating an additional issue.

[Alfredo Moralejo (amoralej)]

Regarding issue reported in comment #2

After analysing logs in http://logs.openstack.org/15/442315/2/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/5130095/logs/subnode-2/var/log/journal-text.txt.gz this is what is happening in node2:

1. At around Mar 07 12:38:10, delorean-ci repo is configured in subnode2:

COMMAND=/usr/bin/wget http://10.29.153.134:8766/current/delorean-ci.repo -O /etc/yum.repos.d/delorean-ci.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=1% /etc/yum.repos.d/delorean-ci.repo

2. At Mar 07 12:38:13, delorean-ci repo is disabled:

COMMAND=/usr/bin/sed -i -e s/enabled=1/enabled=0/ /etc/yum.repos.d/delorean-ci.repo

My guess is that, it's https://review.openstack.org/#/c/441166/. It's preparing the system for initial deployment from previous release (ocata), and patch is for master, so delorean-ci shouldn't be needed.

3. At Mar 07 12:38:17 it configures ocata repos as expected to deploy ocata release:

COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-deps.repo http://trunk.rdoproject.org/centos7-ocata/delorean-deps.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=30% /etc/yum.repos.d//delorean-deps.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean.repo https://trunk.rdoproject.org/centos7-ocata/current//delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=20% /etc/yum.repos.d//delorean.repo
COMMAND=/usr/bin/sh -c > /etc/yum.repos.d//delorean-current.repo
COMMAND=/usr/bin/yum -y install yum-plugin-priorities

4. At Mar 07 12:40:14 it installs puppet-tripleo from ocata delorean repo as expected:

yum[23519]: Installed: puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch

5. At Mar 07 13:27:25 it enables delorean-ci again:

COMMAND=/usr/bin/sed -i -e s/enabled=0/enabled=1/ /etc/yum.repos.d/delorean-ci.repo

6. At Mar 07 13:31:29 it enables master repos:

COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-deps.repo http://trunk.rdoproject.org/centos7/delorean-deps.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=30% /etc/yum.repos.d//delorean-deps.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean.repo http://trunk.rdoproject.org/centos7/current-tripleo//delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=20% /etc/yum.repos.d//delorean.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-current.repo http://trunk.rdoproject.org/centos7/current/delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=10% /etc/yum.repos.d//delorean-current.repo

But as version in delorean-ci (puppet-tripleo-6.2.0-0.20170307123624.8493b34.el7.centos.noarch) is < that the installed one, it doesn't get installed.

The recommended fix for this issue is to push high pre-release tag in master branch so that version in master is > that ocata.

[Alfredo Moralejo (amoralej)]

Steven, about issue in comment #1, could i get full deployment logs?. I'd like to understand why priorities didn't help. Was it an upgrade?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/442668

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442668
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ffe1cd5c62a6581671f1907f4ad84e3aec3d9d8f
Submitter: Jenkins
Branch: master

commit ffe1cd5c62a6581671f1907f4ad84e3aec3d9d8f
Author: Alex Schultz <email address hidden>
Date: Tue Mar 7 11:27:12 2017 -0700

    Update version for Pike

    The current version information is behind that of stable/ocata. In order
    to address some version generation issues in packaging, we need to bump
    the version numbers for in preparation for the next version.

    Change-Id: I586811d9623c4bb03b1b234eaed2b3b365ba6e3e
    Releated-Bug: #1669462

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/442807

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alan Pevec (<email address hidden>) on branch: master
Review: https://review.openstack.org/442807

[Ben Nemec (bnemec)]

Is this actually still critical? I see at least one patch has merged related to it.

[Steven Hardy (shardy)]

I think upgrades are now proven in CI so we can close this - the version issue was addressed when we tagged releases for pike-1