rdo jobs ignore depends-on tripleo-common

[Michele Baldessari]

On change https://review.openstack.org/#/c/507963/ we see that the rdo jobs are failing.
For example: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/8280/

Now the above review has a depends-on this tripleo-common review: https://review.openstack.org/#/c/508259

It seems that this depends-on is ignored because on the undercloud I see a tripleo-common rpm package which I verified does not have https://review.openstack.org/#/c/508259

This can also be seen here https://logs.rdoproject.org/63/507963/46/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Zf0ae2e4678494c709042d82afcf6f89f/undercloud/home/jenkins/install_built_repo.log.txt.gz where we see the following:
2018-03-17 16:18:10 | Updating:
2018-03-17 16:18:10 | openstack-tripleo-heat-templates
2018-03-17 16:18:10 | noarch 8.0.0-0.20180317161309.45dcc5f.el7.centos gating-repo 494 k
2018-03-17 16:18:10 | puppet-tripleo
2018-03-17 16:18:10 | noarch 8.3.1-0.20180317161714.79ccad4.el7.centos gating-repo 223 k

I.e. no tripleo-common.

I presume this is unexpected?

[Alex Schultz (alex-schultz)]

Found that tripleo-common is being build but the version is older than what was available.

Wrote: /builddir/build/RPMS/openstack-tripleo-common-8.4.1-0.20180317161516.ce4a549.el7.centos.noarch.rpm

2018-03-17 16:06:19 | ---> Package openstack-tripleo-common.noarch 0:8.5.1-0.20180317115024.6dd8171.el7.centos will be installed

[Michele Baldessari (michele)]

So indeed rebasing the tripleo-common dependency fixes stuff and now the rdo jobs pass as expected. Wasn't aware of this. Not sure we should tweak things to have the gating-repo higher prio or simply educate folks on this?

[Arx Cruz (arxcruz)]

Marking as invalid, please feel free to change if you think it's still a issue

[Sagi (Sergey) Shnaidman (sshnaidm)]

Bumping priority and severity.
After https://review.openstack.org/#/c/592577/ is merged it's almost impossible to test patches with dependencies.
Every commit in delorean-current has higher version than built package usually, except rare case when dependent patch is rebased to newest HEAD.
So rebasing every time we merge something is not a solution at all.

What expected:
host should install all built packages from gating_repo and ignore same packages in delorean-current repo even if they have higher version in delorean-current.

[Javier Peña (jpena-c)]

After an IRC conversation with Sagi, I think I understand the issue. It happens when we try to run "yum update" with packages generated on the gate, but their version number is lower than the one already installed. This is only caused if:

- The patch being tested is based on an old commit, with a lower tag number than the current one (for example, a patch to tripleo-common based on tag 8.1.0 when the current tag is 8.3.0).
- A package with the current tag is already built on RDO Trunk (following the same example, we have tripleo-common-8.3.0-xxx in the repo).

We could try to have a switch to make DLRN build packages with a high enough Epoch (eg. 999) that it always prevails, but that is ugly/hacky. On the other hand, in those cases we're not testing some patches that are already in place for the repo, so if we need a recent patch to tripleo-common but our commit is based on top of an older patch, it would fail and make us be very confused.

I think the cleanest fix would be to not use the Zuul-provided project source, but instead cherry-pick the patch being tested on top of the current HEAD for the branch. However, I'm not sure if that is possible with Zuul or it would require some custom stuff in the jobs.

[Alex Schultz (alex-schultz)]

Or we could just set the priority of the local repo higher than the delorean-current

[Quique Llorente (quiquell)]

Sagi has already merge a fix for it https://review.openstack.org/#/c/598089/.