DLRN not configured to use regional mirrors

[Paul Belanger]

http://logs.openstack.org/09/488509/2/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ffbef54/logs/delorean_logs/2f/a7/2fa7a2ba6dc7f3e0e71695aab8a7f25e0475bd1a_dev/root.log.txt.gz

DLRN, specifically mock, is not setup to use regional mirrors for building packages. This needs to be fixed, as it will have the same failure rates that jobs were having when directly accessing mirrors.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/494262

[Attila Darazs (adarazs)]

Changing the URLs in rdo.yml didn't work according to my local test, it still uses the mirrorlist for CentOS. Alfredo, let me know if you happen to know where to override this.

[Alfredo Moralejo (amoralej)]

This is how DLRN works:

When running the dlrn command it needs to create a mock configuration with all required repos to build packages. This configuration file is created on the fly by DLRN by concatenating several files:

1. First part is taken from scripts/<target>.cfg where target is an option in projects.ini[1]. By default this is centos.cfg [2]
2. Appends content of <baseurl>/delorean-deps.repo [3] where baseurl is an option in projects.ini.
3. If --dev option is used (as in tripleo gate jobs), content of <baseurl>/current/delorean.repo is appended [4]

I think the best option would be:

1. Create a file scripts/custom.cfg with all required content using repo configurations from /etc/yum.repos.d/ as at this point we have config files with openstaci-infra regional mirrors.

2. Create empty /tmp/fake-repo/delorean-deps.repo and /tmp/fake-repo/current/delorean.repo

3. Modifying projects.ini file with values:

baseurl="file:///tmp/fake/repo"
target=custom

urllopen can work with file:/// urls, so this theoretically would work.

[1] https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/build.py#L109
[2] https://github.com/softwarefactory-project/DLRN/blob/master/scripts/centos.cfg
[3] https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/build.py#L151-L162
[4] https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/build.py#L170-L181

[wes hayutin (weshayutin)]

Looking at the old scripts we appear to missing some things in the build/test role
https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L436-L527

* The variables for the delorean build should be overridden in the config/environments in tripleo-ci

* Need to use: baseurl=$NODEPOOL_RDO_PROXY/centos7

- name: Override projects.ini settings
  lineinfile:
    dest: '{{ build_repo_dir }}/DLRN/projects.ini'
    regexp: '{{ project_item.regexp }}'
    line: '{{ project_item.line }}'
  with_items:
    - { regexp: 'baseurl=.*', line: 'baseurl=https://trunk.rdoproject.org/centos7-{{ artg_release }}' }
    - { regexp: 'distro=.*', line: 'distro={{ artg_distro }}' }
    - { regexp: 'source=.*', line: 'source={{ artg_change.branch }}' }
  loop_control:
      loop_var: project_item

[wes hayutin (weshayutin)]

The git checkouts could also just use git.openstack vs. github

https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L481-L489
vs.
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/dlrn-build.yml#L61

[Alfredo Moralejo (amoralej)]

Note that using "baseurl=$NODEPOOL_RDO_PROXY/centos7" will not fis this as in that case the delorean-deps.repo and current/delorean.repo will be downloaded from the mirror but the content of those files is still using trunk.rdoproject.org, so it's not big help.

[Paul Belanger (pabelanger)]

Yes, it will be a big help. Since rdoproject.org is unreliable. I understand it is not the complete fix, but we cannot be hitting rdoproject.org directly for anything. Even repo files.

I have to look into it, but if you could just grab or reference the repo files in yum.repos.d, they are likely setup properly.

[Paul Belanger (pabelanger)]

If you could also update rdoinfo to now use http://mirror.ord.rax.openstack.org:8080/rdo/rdoinfo/ (will be RDO reverse proxy cache) that will also help too.

[Attila Darazs (adarazs)]

Alfredo, thanks for the description of DLRN repo handling. I solved it a bit differently than how you suggested, but it was a big help.

For now I'm waiting for https://review.openstack.org/494056 to merge (that will change the repos to start with quickstart-*).

A previous version (without depends-on on the change above) of this quickstart-extras change seems to be passing: https://review.openstack.org/494262

I enable the local mirrors with this change in tripleo-ci: https://review.openstack.org/494537

And testing the changes with https://review.openstack.org/494538

[Attila Darazs (adarazs)]

It should be now working as expected (without the DLRN or rdoinfo mirroring), we just need to merge https://review.openstack.org/494262 and https://review.openstack.org/494537

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494262
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=92a1bf7f051ef031407eaee27f282fcf7aaff794
Submitter: Jenkins
Branch: master

commit 92a1bf7f051ef031407eaee27f282fcf7aaff794
Author: Attila Darazs <email address hidden>
Date: Wed Aug 16 18:22:55 2017 +0200

    Add options to use local DLRN and CentOS mirrors

    Add the dlrn_baseurl option that allows overriding the DLRN server,
    making possible to use a proxy upstream. Also add
    dlrn_use_local_mirrors option. When true, the package building will
    use the already defined CentOS and DLRN repos from the machine where
    the packages are built.

    Also move the rdoinfo repo fetching in the main part of the role to
    save time and bandwidth.

    Change-Id: I0b440895267fcd8027394504649372bd923128d6
    Closes-bug: #1710678

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.