DLRN builds fail intermittently (network errors)

[Jiří Stránský]

Jobs sometimes fail to build the RPM with the change in question. In console.html one can see something like

2017-08-31 05:13:40.239644 | TASK [build-test-packages : Run DLRN] ******************************************
2017-08-31 05:13:40.239722 | task path: /home/jenkins/workspace/gate-tripleo-ci-centos-7-ovb-ha-oooq/.quickstart/usr/local/share/ansible/roles/build-test-packages/tasks/dlrn-build.yml:66
2017-08-31 05:13:40.260717 | Thursday 31 August 2017 05:13:40 +0000 (0:00:00.068) 0:03:04.352 *******
2017-08-31 05:14:29.172562 | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set +e; source /home/jenkins/dlrn-venv/bin/activate; while true; do\n dlrn --config-file projects.ini --head-only --package-name openstack-tripleo-heat-templates --local --info-repo rdoinfo --dev;\n if [ $? -eq 0 ]; then\n # SUCCESS\n break;\n elif [ $? -eq 1 ]; then\n # FAILED\n exit 1;\n elif [ $? -eq 2 ]; then\n # RETRY\n continue;\n fi;\n # Unexpected DLRN return code\n exit $?;\n done;", "delta": "0:00:48.112420", "end": "2017-08-31 05:14:29.117117", "failed": true, "rc": 1, "start": "2017-08-31 05:13:41.004697", "stderr": "2017-08-31 05:13:43,619 INFO:dlrn-repositories:Getting https://github.com/rdo-packages/tripleo-heat-templates-distgit.git to ./data/openstack-tripleo-heat-templates_distro (pike-rdo)\n2017-08-31 05:13:45,046 INFO:dlrn-repositories:Getting git://git.openstack.org/openstack/tripleo-heat-templates to ./data/openstack-tripleo-heat-templates (stable/pike)\n2017-08-31 05:13:45,211 INFO:dlrn-build:Processing openstack-tripleo-heat-templates 41e80ca24960e24098bdd7d2d9194864a38eda34\n2017-08-31 05:14:21,146 ERROR:dlrn-build:cmd failed. See logs at: /home/jenkins/DLRN/data/repos/41/e8/41e80ca24960e24098bdd7d2d9194864a38eda34_dev/\n2017-08-31 05:14:21,148 INFO:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/41/e8/41e80ca24960e24098bdd7d2d9194864a38eda34_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-pike /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-08-31 05:14:21,152 ERROR:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/41/e8/41e80ca24960e24098bdd7d2d9194864a38eda34_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-pike /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-08-31 05:14:21,160 ERROR:dlrn:Known error building packages for openstack-tripleo-heat-templates, will retry later\nNone", "stdout": "", "stdout_lines": [], "warnings": []}

Looking in delorean logs, the cause is:

DEBUG: https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2607:1680:0:1::2: Network is unreachable"

[Alan Pevec (apevec)]

This should be resolved, buildlogs.centos.org is now accessed via caching proxy in openstack-infra.

[Sorin Sbarnea (ssbarnea)]

This happens again and the failure is not identified by elastic-recheck which means that the original fix was not complete.

[Sorin Sbarnea (ssbarnea)]

LogStash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3Adlrn%3AReceived%20exception%20Error%20in%20build_rpm_wrapper%5C%22%20%20%20AND%20tags%3A%5C%22console%5C%22

[Clark Boylan (cboylan)]

http://logs.openstack.org/66/594566/8/check/tripleo-ci-centos-7-standalone/dd8b0d8/logs/delorean_logs/9e/86/9e8612fe401331a324fb491dd9a6b4c6740aaa02_dev/rpmbuild.log.txt.gz seems to be an example of this failing against a local repomd.xml file:

DEBUG: failure: repodata/repomd.xml from local: [Errno 256] No more mirrors to try.
DEBUG: file:///home/zuul/DLRN/data/repos/9e/86/9e8612fe401331a324fb491dd9a6b4c6740aaa02_dev/repodata/repomd.xml: [Errno 14] curl#37 - "Couldn't open file /home/zuul/DLRN/data/repos/9e/86/9e8612fe401331a324fb491dd9a6b4c6740aaa02_dev/repodata/repomd.xml"

Possible this isn't entirely a network issue?

[Clark Boylan (cboylan)]

For the url in the top comment openstack infra does proxy cache centos buildlogs. https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n172

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?