Problem: package python3-tripleoclient-13.1.1-0.20200404091914.624a61f.el8.noarch requires validations-common

[wes hayutin]

2020-04-05 18:18:03 | Last metadata expiration check: 0:00:08 ago on Sun 05 Apr 2020 06:17:55 PM UTC.
2020-04-05 18:18:04 | Error:
2020-04-05 18:18:04 | Problem: package python3-tripleoclient-13.1.1-0.20200404091914.624a61f.el8.noarch requires validations-common, but none of the providers can be installed
2020-04-05 18:18:04 | - cannot install the best candidate for the job
2020-04-05 18:18:04 | - package validations-common-0.0.1-0.20200403122940.99af090.el8.noarch is excluded
2020-04-05 18:18:04 | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

all master periodic failing here.

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/eca8858/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz

[Sandeep Yadav (sandeepyadav93)]

Hello,

Comparing good run[1] found validations-common was coming from delorean-component-tripleo

~~~
2020-04-04 17:28:19 | validations-common noarch0.0.1-0.20200403122940.99af090.el8 delorean-component-tripleo 99 k
~~

For affected job[2]:-

Mirror - http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos8/component/tripleo/f3/83/f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f was used.

validations-common package was missing here https://trunk.rdoproject.org/centos8-master/component/tripleo/f3/83/f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f/ .

Last run passed[3] successfully and back to green.

Monitoring further while trying to debug probable root cause of last failure so it can be avoided in future.

[1] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/382c84e/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz

[2] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/eca8858/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz

~~~
[delorean-component-tripleo]
name=delorean-tripleo-ansible-f38338fb2ce36e278adcbb636377428b7190e456
baseurl=http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos8/component/tripleo/f3/83/f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f ---> This mirror was used
enabled=1
gpgcheck=0
priority=20
~~~

[3] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/d18dc30/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz

Thank You
Sandeep

[yatin (yatinkarel)]

I checked and found following for the issue in promotion pipeline:-
So the hash(f3/83/f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f) used for tripleo component is too old and promoted on 10th March, 2020[1] and since then there are continuous promotions for tripleo-component, second last[2] promotion for this hash was on 29th March to tripleo-ci-testing, And since the hash is not referenced by any exclude symlink[3] it's likely purged by DLRN.

The aggregate hash used in job is 86ec1b18725f0e49dd0fb2066d1d2174. Wrong tripleo component hash used in job is coming from promoted-components[4], this promoted-components hash was promoted to tripleo-ci-testing in [3] which is used in those promotion jobs. At current stage tripleo-ci-testing repo is good so last run passed.

So the question is how that wrong old hash got into delorean.repo even if good hashes exist for it[6]. Finding and resolving this will help in avoiding future such issues. So either delorean.repo was manually changed or promotions are done wrongly or something is wrong or DLRN side. Will involve @jpena into it as he would have more idea.

I also noticed that there are too many promotions being reported for same hashes.

[1] dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc promotion-get --commit-hash f38338fb2ce36e278adcbb636377428b7190e456 --distro-hash abb1ed0ffbec1e78a0a552f88ec5ef04b7df2596 --promote-name promoted-components
[
  {
    "aggregate_hash": "a7d119e80ff66d63f264e2ec45111b23",
    "commit_hash": "f38338fb2ce36e278adcbb636377428b7190e456",
    "component": "tripleo",
    "distro_hash": "abb1ed0ffbec1e78a0a552f88ec5ef04b7df2596",
    "promote_name": "promoted-components",
    "repo_hash": "f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f",
    "repo_url": "https://trunk.rdoproject.org/centos8/component/tripleo/f3/83/f38338fb2ce36e278adcbb636377428b7190e456_abb1ed0f",
    "timestamp": 1583877750,
    "user": "review_rdoproject_org"
  }
]

[2] dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc promotion-get --commit-hash f38338fb2ce36e278adcbb636377428b7190e456 --distro-hash abb1ed0ffbec1e78a0a552f88ec5ef04b7df2596
[3] https://github.com/rdo-infra/puppet-dlrn/blob/master/files/run-purge.sh#L30
[4] https://trunk.rdoproject.org/centos8-master/promoted-components/86/ec/86ec1b18725f0e49dd0fb2066d1d2174/delorean.repo
[5] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing/3d9a43b/job-output.txt
[6] dlrnapi --url https://trunk-centos8.rdoproject.org/api-centos8-master-uc promotion-get --promote-name promoted-components --component tripleo --limit 5
[7] dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc promotion-get --agg-hash 86ec1b18725f0e49dd0fb2066d1d2174

[yatin (yatinkarel)]

<< So the question is how that wrong old hash got into delorean.repo even if good hashes exist for it[6]. Finding and resolving this will help in avoiding future such issues. So either delorean.repo was manually changed or promotions are done wrongly or something is wrong or DLRN side. Will involve @jpena into it as he would have more idea.

So we found the issue, issue was in dlrn prune script[1], it was not synching symlinks for promotion hashes(component-ci-testing, promoted-components,tripleo-ci-testing etc). and then old data(repos and delorean.repo) was pushed to trunk.rdoproject.org making delorean.repo inconsistent with dlrnapi promoted hashes. Patch https://review.rdoproject.org/r/#/c/26294/ should fix and avoid such issue in future.

<< I also noticed that there are too many promotions being reported for same hashes.
This didn't caused the issue, but good to fix unnecessary promotions, example of multiple promotions:-

dlrnapi --url https://trunk-centos8.rdoproject.org/api-centos8-master-uc promotion-get --promote-name promoted-components --component clients --limit 10 | jq .[].repo_hash

[1] https://github.com/rdo-infra/puppet-dlrn/blob/master/files/run-purge.sh#L42-L44