CI reproducer fails to parse dlrn hash tag

[Bogdan Dobrelya]

I deploy zuul-based libvirt CI reproducer with:

 mirror_path: mirror.regionone.rdo-cloud.rdoproject.org

this ultimately points subnodes to http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-master/

but apparently the further URI leads to 404, see the trace in the attached quickstart.log

The repro tarball:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4b5/730325/11/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/4b55c9b/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar

The hash e961461fc3de7c29860ddacfcf697aaf written in the launcher playbook is faily one, the one calculated by quickstart (3371c543036a0f9f7b485489c490de4bd8cb52d1_f935e212) is OK. So that should be some issue with the former one stored in the reproducer tarball.

[Sagi (Sergey) Shnaidman (sshnaidm)]

Please paste a link to script you run and how you reproduce it.

[Bogdan Dobrelya (bogdando)]

Sagi, the script is here https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4b5/730325/11/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/4b55c9b/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar

The command ran on Centos 8 host:

./reproducer-zuul-based-quickstart.sh -w /var/tmp/reproduce -e @extra.yaml -l --ssh-key-path ~/.ssh/id_rsa_gh -e os_autohold_node=true -e zuul_build_sshkey_cleanup=false -e container_mode=docker -e upstream_gerrit_user=donkey -e rdo_gerrit_user=donkey

and extra.yaml is:

ansible_python_interpreter: "/usr/bin/python3"
mirror_fqdn: mirror.regionone.rdo-cloud.rdoproject.org
pypi_fqdn: mirror01.ord.rax.opendev.org
images:
  - name: undercloud
    url: file:///var/lib/libvirt/images/centos8.qcow2
    md5sum: xxx
    type: qcow2
  - name: overcloud
    url: file:///var/lib/libvirt/images/centos8.qcow2
    md5sum: xxx
    type: qcow2
custom_nameserver: 8.8.8.8
compute_memory: 14096
compute_vcpu: 4
control_memory: 18192
control_vcpu: 4
undercloud_vcpu: 4
undercloud_memory: 18192
force_cached_images: true
image_cache_expire_days: 30
vxlan_networking: false
toci_vxlan_networking: false
modify_image_vc_root_password: r00tme
mergers: 2

The centos8 image is from https://nb04.opendev.org/images (it has yum and python installed in it)

And the preparation steps on host are described here: https://github.com/bogdando/oooq-warp/blob/master/docs/CI-reproducer.md#running-on-host-without-the-wrapper-container

[Bogdan Dobrelya (bogdando)]

note, the workaround is to manually hach the used hash, like:

              export UNDERCLOUD_INSTALL_RELEASE="master" │·················
              export UNDERCLOUD_INSTALL_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export UNDERCLOUD_TARGET_RELEASE="master" │·················
              export UNDERCLOUD_TARGET_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export OVERCLOUD_DEPLOY_RELEASE="master" │·················
              export OVERCLOUD_DEPLOY_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export OVERCLOUD_TARGET_RELEASE="master" │·················
              export OVERCLOUD_TARGET_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export STANDALONE_DEPLOY_RELEASE="master" │·················
              export STANDALONE_DEPLOY_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export STANDALONE_DEPLOY_NEWEST_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export STANDALONE_TARGET_RELEASE="master" │·················
              export STANDALONE_TARGET_NEWEST_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465" │·················
              export STANDALONE_TARGET_HASH="b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465"

[Bogdan Dobrelya (bogdando)]

12:52:18 PM - ykarel: bogdando, i checked logs for that hash
12:52:34 PM - ykarel: and that hash is correct as per yesterday
12:52:43 PM - ykarel: post that there are two new promotions
12:52:57 PM - ykarel: "aggregate_hash": "1fcd094313791317563b22f5dcf54d3b",
12:52:58 PM - ykarel: "aggregate_hash": "53d2c73068becf11b7eb9f3662e666ee",
12:52:58 PM - ykarel: "aggregate_hash": "e961461fc3de7c29860ddacfcf697aaf",
12:53:03 PM - ykarel: latest on top

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/741587

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/741593

[yatin (yatinkarel)]

Related fix proposed to branch: master
Review: https://review.opendev.org/#/c/741633/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741593
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=0571ff7f5516d59891cc208d88e8bf46905a301f
Submitter: Zuul
Branch: master

commit 0571ff7f5516d59891cc208d88e8bf46905a301f
Author: yatinkarel <email address hidden>
Date: Fri Jul 17 13:18:06 2020 +0530

    Set dlrn_baseurl default based on distro

    master release is centos8 only since ussuri
    release, so set dlrn_baseurl based on
    distribution version instead of 7.

    Related-Bug: #1887637
    Change-Id: I4c24e19dc3853f71d6e5f842d5441393cf50e583

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741587
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=8cfedbcf7ef8962d9bdd2c9a4e5683eb68fa620a
Submitter: Zuul
Branch: master

commit 8cfedbcf7ef8962d9bdd2c9a4e5683eb68fa620a
Author: yatinkarel <email address hidden>
Date: Fri Jul 17 12:46:41 2020 +0530

    Use dnf/yum for customizing libvirt nodepool images

    yum is not installed by default in CentOS8 images,
    so use dnf for it and fallback to yum when dnf not
    available.

    Related-Bug: #1887637
    Change-Id: I4ddf957f677ba8cc584a155444b27d43b7d27e5c

[OpenStack Infra (hudson-openstack)]

Change abandoned by yatin (<email address hidden>) on branch: master
Review: https://review.opendev.org/741633
Reason: Failing in gates, will revisit

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741633
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=11c2bb7cdfa401b9e3e2f65ec459e1b7a452ed76
Submitter: Zuul
Branch: master

commit 11c2bb7cdfa401b9e3e2f65ec459e1b7a452ed76
Author: yatinkarel <email address hidden>
Date: Fri Jul 17 18:29:28 2020 +0530

    Allow CentOS8 jobs to run with custom dlrn hashes

    DLRN repo symlinks(ex. current-tripleo, current etc)
    keeps on changing with promotions, in order to reproduce/rerun
    jobs with older dlrn hashes CentOS8 release files were missing
    support, this patch adds it.

    Related-Bug: #1887637
    Depends-On: https://review.opendev.org/#/c/747121/
    Change-Id: Idffb7de57d5f2debb8213b11c5ce3e41c2ece4f3