Baremetal hp_dl360_envD jobs are failing - stalling when installing libvirt rpms

[Ronelle Landy]

Jobs running ocata, pike and master on HP env D are failing since November 26, 2017.
The jobs stall on installing the libvirt-related rpms:

04:45:25 TASK [prep-internal-host : install libvirt] ************************************
04:45:25 task path: /home/rhos-ci/jenkins/workspace/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envD-single_nic_vlans/usr/local/share/ansible/roles/prep-internal-host/tasks/main.yml:39
04:45:25 Thursday 30 November 2017 04:45:25 +0000 (0:03:29.775) 0:21:31.807 *****
04:45:25 Using module file /home/rhos-ci/jenkins/workspace/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envD-single_nic_vlans/lib/python2.7/site-packages/ansible/modules/core/packaging/os/yum.py
...
09:04:15 Build timed out (after 360 minutes). Marking the build as aborted.

Full logs are:

https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-ocata-rdo_trunk-baremetal-hp_dl360_envD-single_nic_vlans/47/consoleFull

https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envD-single_nic_vlans/43/consoleFull

Logging on to the box and running
>> yum install -y libvirt

finishes but is very slow.

We need to determine what happened on Nov 26 to make the rpm install so slow.

[Matt Young (halcyondude)]

Additional instances of this have been observed in rdophase2. Patching internally to bring these configs offline for diagnoses.

[Ronelle Landy (rlandy)]

env E is passing so I would guess the undercloud box of env D needs to be reprovisioned.

[Matt Young (halcyondude)]

should we move this to a card on tripleo-ci trello board? there's no tripleo or quickstart 'bug' here...

[Alex Schultz (alex-schultz)]

The env is no longer available, closing the issue. Feel free to reopen if it occurs again or additional information is available.

[Ronelle Landy (rlandy)]

closing this out - no action for some time. Will reopen if need be.