periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload is timeout in virt-customize step

[Arx Cruz]

Logs:
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/bf98214/console.txt.gz

This task should take only a few minutes, however is taking more than 2 hours and make the job timeout
The job usually takes 2.5 hours to finish.

Please notice that this runs after the tempest, so we do have tempest results, it's not related.

We don't have logs because the log collection is done in toc-test builder, and virt-customize is running in images-upload-and-label builder (another bug will be opened for this)

[yatin (yatinkarel)]

So this is not happening always and something is to do with nested virtualization:-

We hold the node as logs were not available in job to see what's in the virt-customize log, below is what we found:-
[jenkins@undercloud ~]$ tail -f .__repo_setup.sh.log
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 66 89 d8 66 e8 e5 af ff ff 66 83 c4 0c 66 5b 66 5e 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^C
[jenkins@undercloud ~]$ vi .__repo_setup.sh.log
[jenkins@undercloud ~]$ cat .__repo_setup.sh.log
[ 0.0] Examining the guest ...
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_network true
libguestfs: trace: set_network = 0
libguestfs: trace: add_drive "overcloud-full.qcow2" "readonly:false" "protocol:file" "discard:besteffort"
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 36, release: 3, extra: rhel=7,release=6.el7_4.3,libvirt, >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=virt-customize
libguestfs: launch: version=1.36.3rhel=7,release=6.el7_4.3,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfs15JLpA
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin5
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.16
supermin: rpm: detected RPM version 4.11
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed exclu...

[Rafael Folco (rafaelfolco)]

cpu flags seem to be identical on failing / successful runs... perhaps we could enforce tcg mode until we find the root cause.

[yatin (yatinkarel)]

<< cpu flags seem to be identical on failing / successful runs... perhaps we could enforce tcg mode until we find the root cause.

Proposed https://review.rdoproject.org/r/13327 until we find the root cause.

[Rafael Folco (rafaelfolco)]

tagging tech-debt for RCA

[Matt Young (halcyondude)]

triage note: this job has passed for the past 5 runs

[Matt Young (halcyondude)]

(and we've had a promotion)

[Arx Cruz (arxcruz)]

It’s not invalid, we have a promotion because a workaround https://review.rdoproject.org/r/#/c/13327/

[Quique Llorente (quiquell)]

Bugzilla for the root cause https://bugzilla.redhat.com/show_bug.cgi?id=1565179

[Matt Young (halcyondude)]

(triage) holding pattern waiting on BZ

[Sagi (Sergey) Shnaidman (sshnaidm)]

07 May, no updates in BZ

[Quique Llorente (quiquell)]

No updates on the BZ

[Quique Llorente (quiquell)]

Testing if it's still an issue with https://review.openstack.org/601254