puppet package versions break ocata deployments and upgrades to pike

Bug #1669462 reported by Steven Hardy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

Local testing with tripleo.sh --repo-setup shows we have issues with a number of puppet modules where packages installed aren't the latest so deployment/upgrade fails, here's an example:

[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-openstacklib
puppet-openstacklib-10.3.0-1.el7.noarch

[root@overcloud-controller-0 ~]# repoquery puppet-openstacklib --repoid=delorean-ocata-testing
puppet-openstacklib-0:10.3.0-1.el7.noarch

[root@overcloud-controller-0 ~]# repoquery puppet-openstacklib --repoid=delorean-current
puppet-openstacklib-0:10.3.0-0.20170223102909.b3d0590.el7.centos.noarch

Here we can see we've installed the version from delorean-deps, instead of the correct delorean-current version, which breaks upgrade like this:

  deploy_stderr: |
    exception: connect failed
    Error: Evaluation Error: Error while evaluating a Function Call, os_transport_url(): port should be a String at /etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp:102:37 on node overcloud-controller-0.localdomain

because we don't have https://review.openstack.org/#/c/436401/

We can see this set priorities but the newer release is installed anyway:

[root@overcloud-controller-0 ~]# rpm -qa | grep yum-plugin-priorities
yum-plugin-priorities-1.1.31-40.el7.noarch

[root@overcloud-controller-0 ~]# cat /etc/yum.repos.d/delorean.repo | grep -e priority -e url
baseurl=https://trunk.rdoproject.org/centos7/d1/a9/d1a9d912723614364e8ce807437dd9503bd30879_09945cb4
priority=20
[root@overcloud-controller-0 ~]# cat /etc/yum.repos.d/delorean-deps.repo | grep -e priority -e url
baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-ocata/
baseurl=http://cbs.centos.org/repos/cloud7-openstack-common-pending/$basearch/os/
baseurl=http://mirror.centos.org/centos/7/virt/$basearch/kvm-common/

Tags: ci
Revision history for this message
Steven Hardy (shardy) wrote :

The same problem exists for deployment as puppet-tripleo is outdated, ref https://review.openstack.org/#/c/439759/

Discussion with apevec in #rdo indicates we need to do some rework to the packaging to fix this, and I'm still not clear why we don't see the same issues in CI

Changed in tripleo:
status: New → Triaged
milestone: none → pike-1
importance: Undecided → Critical
Revision history for this message
Steven Hardy (shardy) wrote :

This is now impacting CI, as upgrade testing ignores the delorean-ci built package:

http://logs.openstack.org/15/442315/2/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/5130095/logs/yum_mirror_error.txt.gz

We built puppet-tripleo-6.2.0-0.20170307123624.8493b34.el7.centos.noarch.rpm due to a Depends-On, but on the node we have:

puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch

Which I think comes from delorean-deps, and means we ignore the Depends-On package.

tags: added: alert ci
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Problem is in the logic used by DLRN to create NVR for puppet modules, as it uses version from git tags instead of PBR + pre-release release:

https://github.com/openstack-packages/DLRN/blob/master/scripts/build_rpm.sh#L33

For openstack puppet modules which implements PBR i think we could move to use setup.py --version as we do for python modules.

For non-openstack puppet modules we'd need to figure out something different.

We'll uppdate this LP with any progress in fixing it in DLRN.

Revision history for this message
Javier Peña (jpena-c) wrote :

https://review.rdoproject.org/r/5642 contains a potential fix for NVR generation in Puppet OpenStack modules (or any other with PBR support).

Once it is merged, we will need to rebuild the puppet-* packages, but that can be automated.

Revision history for this message
Steven Hardy (shardy) wrote :

Thanks for the update - I think fixing this for puppet-* OpenStack modules will be sufficient at this point.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

I think we may be mixing several issues. Right now, master branch is behind stable/ocata in terms of upstream release:

[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ python setup.py --version
6.2.1.dev46
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ git checkout stable/ocata
Branch stable/ocata set up to track remote branch stable/ocata from origin.
Switched to a new branch 'stable/ocata'
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$ python setup.py --version
6.3.1.dev3
[amoralej@faemino puppet-tripleo(os_amoralej_oslab)]$

This is something to be fixed upstream.

Note that puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch is not coming from delorean-deps but from delorean or delorean-current.

Anyway, I'm still trying to understand why priorities are not working as expected.

Revision history for this message
Javier Peña (jpena-c) wrote :

It turns out https://review.rdoproject.org/r/5642 broke something else, so it had to be reverted. https://review.rdoproject.org/r/5646 should be a permanent fix, but I'll take it through a full round of checks to be 100% sure it is not creating an additional issue.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Regarding issue reported in comment #2

After analysing logs in http://logs.openstack.org/15/442315/2/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/5130095/logs/subnode-2/var/log/journal-text.txt.gz this is what is happening in node2:

1. At around Mar 07 12:38:10, delorean-ci repo is configured in subnode2:

COMMAND=/usr/bin/wget http://10.29.153.134:8766/current/delorean-ci.repo -O /etc/yum.repos.d/delorean-ci.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=1% /etc/yum.repos.d/delorean-ci.repo

2. At Mar 07 12:38:13, delorean-ci repo is disabled:

COMMAND=/usr/bin/sed -i -e s/enabled=1/enabled=0/ /etc/yum.repos.d/delorean-ci.repo

My guess is that, it's https://review.openstack.org/#/c/441166/. It's preparing the system for initial deployment from previous release (ocata), and patch is for master, so delorean-ci shouldn't be needed.

3. At Mar 07 12:38:17 it configures ocata repos as expected to deploy ocata release:

COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-deps.repo http://trunk.rdoproject.org/centos7-ocata/delorean-deps.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=30% /etc/yum.repos.d//delorean-deps.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean.repo https://trunk.rdoproject.org/centos7-ocata/current//delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=20% /etc/yum.repos.d//delorean.repo
COMMAND=/usr/bin/sh -c > /etc/yum.repos.d//delorean-current.repo
COMMAND=/usr/bin/yum -y install yum-plugin-priorities

4. At Mar 07 12:40:14 it installs puppet-tripleo from ocata delorean repo as expected:

yum[23519]: Installed: puppet-tripleo-6.3.0-0.20170307041058.0cad4e0.el7.centos.noarch

5. At Mar 07 13:27:25 it enables delorean-ci again:

COMMAND=/usr/bin/sed -i -e s/enabled=0/enabled=1/ /etc/yum.repos.d/delorean-ci.repo

6. At Mar 07 13:31:29 it enables master repos:

COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-deps.repo http://trunk.rdoproject.org/centos7/delorean-deps.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=30% /etc/yum.repos.d//delorean-deps.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean.repo http://trunk.rdoproject.org/centos7/current-tripleo//delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=20% /etc/yum.repos.d//delorean.repo
COMMAND=/usr/bin/curl -Lvo /etc/yum.repos.d//delorean-current.repo http://trunk.rdoproject.org/centos7/current/delorean.repo
COMMAND=/usr/bin/sed -i -e s%priority=.*%priority=10% /etc/yum.repos.d//delorean-current.repo

But as version in delorean-ci (puppet-tripleo-6.2.0-0.20170307123624.8493b34.el7.centos.noarch) is < that the installed one, it doesn't get installed.

The recommended fix for this issue is to push high pre-release tag in master branch so that version in master is > that ocata.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Steven, about issue in comment #1, could i get full deployment logs?. I'd like to understand why priorities didn't help. Was it an upgrade?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/442668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/442668
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ffe1cd5c62a6581671f1907f4ad84e3aec3d9d8f
Submitter: Jenkins
Branch: master

commit ffe1cd5c62a6581671f1907f4ad84e3aec3d9d8f
Author: Alex Schultz <email address hidden>
Date: Tue Mar 7 11:27:12 2017 -0700

    Update version for Pike

    The current version information is behind that of stable/ocata. In order
    to address some version generation issues in packaging, we need to bump
    the version numbers for in preparation for the next version.

    Change-Id: I586811d9623c4bb03b1b234eaed2b3b365ba6e3e
    Releated-Bug: #1669462

tags: removed: alert
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/442807

Changed in tripleo:
milestone: pike-1 → pike-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (master)

Change abandoned by Alan Pevec (<email address hidden>) on branch: master
Review: https://review.openstack.org/442807

Changed in tripleo:
milestone: pike-2 → pike-3
Revision history for this message
Ben Nemec (bnemec) wrote :

Is this actually still critical? I see at least one patch has merged related to it.

Revision history for this message
Steven Hardy (shardy) wrote :

I think upgrades are now proven in CI so we can close this - the version issue was addressed when we tagged releases for pike-1

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers