Repositories setup inside the image in TripleO CI by quickstart may take 50 min

[Sagi (Sergey) Shnaidman]

Sometimes virt-customize task which should install repos runs a lot of time.

50 mins:
http://logs.openstack.org/28/468628/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/9c2e5d8/console.html#_2017-05-27_16_51_49_721548

http://logs.openstack.org/77/430277/112/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/8c1e689/console.html#_2017-05-27_16_15_08_591752

Seems like the main delay is here: (23 mins)

[ 57.770167] hrtimer: interrupt took 4135442 ns
[ 1467.032187] dracut[614] No '/dev/log' or 'logger' included for syslog logging

and then it finishes in 25 additional minutes:
[ 2931.553977] dracut[614] *** Creating initramfs image file '/boot/initramfs-3.10.0-514.21.1.el7.x86_64.img' done ***

http://logs.openstack.org/28/468628/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/9c2e5d8/logs/undercloud/home/jenkins/repo_setup.sh.log.txt.gz

All this causes to job failures. See also https://bugs.launchpad.net/tripleo/+bug/1693945

[Sagi (Sergey) Shnaidman (sshnaidm)]

Working on this: https://review.openstack.org/#/c/468975/1

[John Trowbridge (trown)]

I have been unable to figure out why libguestfs suddenly started having this issue, but the patches from Sagi do help:

https://review.openstack.org/468975
https://review.openstack.org/468976/

[Emilien Macchi (emilienm)]

I'll close it for now as we merged the patches to restore the old way of building images.

[Matt Young (halcyondude)]

Emilien: are you referring to the 2 patches above, or other patches?

thx

[Alan Pevec (apevec)]

> I have been unable to figure out why libguestfs suddenly started having this issue

For posterity, infra team member clarified this:
<pabelanger> right, not all clouds support nested virt, so we disable it across the board

http://eavesdrop.openstack.org/irclogs/%23rdo/%23rdo.2017-05-31.log.html#t2017-05-31T14:13:32

[Emilien Macchi (emilienm)]

Matt: yes, the 2 patches were workarounds so we can move forward with image builds without timeouts. The problem itself sounds related to the fact we don't have nested virt on the nodes where we build the images.

[John Trowbridge (trown)]

In fact, it is not even image building, but image modification. We use DIB for image building everywhere, but we need to inject our gated changes into the cached images. That is what we now have two different methods for. We use virt-customize where we are able, and use the mounted raw image method where we arent.