vexx: OVB master job running on vexxhost show some nodes failing introspection step

[Ronelle Landy]

One or two nodes are failing introspection in master jobs running on vexxhost.

Example logs are below:

https://logserver.rdoproject.org/79/735179/6/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/384e111/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

2020-06-26 15:04:32 | 2020-06-26 15:04:32.218612 | fa163e10-2375-5c60-d0ee-000000000015 | OK | Nodes that passed introspection | localhost | result={
2020-06-26 15:04:32 | "changed": false,
2020-06-26 15:04:32 | "msg": " b85731ef-c061-43bd-a098-7f8d0b2e85c0 1e88a786-eb8e-47cc-8d7f-0ff27f0570b6"
2020-06-26 15:04:32 | }
2020-06-26 15:04:32 | 2020-06-26 15:04:32.226220 | fa163e10-2375-5c60-d0ee-000000000016 | TASK | Nodes that failed introspection
2020-06-26 15:04:32 | 2020-06-26 15:04:32.227229 | fa163e10-2375-5c60-d0ee-000000000015 | TIMING | Nodes that passed introspection | 0:20:21.214 | 0.12s
2020-06-26 15:04:32 | 2020-06-26 15:04:32.284986 | fa163e10-2375-5c60-d0ee-000000000016 | FATAL | Nodes that failed introspection | localhost | error={
2020-06-26 15:04:32 | "msg": " 726f58b3-59d6-48af-baf3-33af7f12f296 0d94ec59-3049-44b8-a634-596d8f423a95"
2020-06-26 15:04:32 | }

*****************

https://logserver.rdoproject.org/23/737423/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/c064767/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

2020-06-26 14:55:32 | TASK [Nodes that passed introspection] *****************************************
2020-06-26 14:55:32 | Friday 26 June 2020 14:55:32 +0000 (0:20:19.713) 0:20:20.405 ***********
2020-06-26 14:55:32 | ok: [localhost] =>
2020-06-26 14:55:32 | msg: ' a93bf3e8-e0ff-48ab-8a81-86f794b1086a 6147b65d-3674-4979-b965-77f082d562a7 5b35041e-7dce-42db-8550-0aed31d8f3b5'
2020-06-26 14:55:32 |
2020-06-26 14:55:32 | TASK [Nodes that failed introspection] *****************************************
2020-06-26 14:55:32 | Friday 26 June 2020 14:55:32 +0000 (0:00:00.117) 0:20:20.522 ***********
2020-06-26 14:55:32 | fatal: [localhost]: FAILED! =>
2020-06-26 14:55:32 | msg: ' 32e3cb1b-2cf0-4989-b43e-2936d738b0a3'

Possibly we need to increase time outs here.

[Ronelle Landy (rlandy)]

https://review.opendev.org/#/c/737568/ may help

[Ronelle Landy (rlandy)]

Introspection is looking better - for the stacks that get to that point.
Putting this on hold until vexx is in better shape

[Marios Andreou (marios-b)]

 introspection/resource issues continue - examples periodic/traincentos8 from today

    (500 error) https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/b986df3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

    https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/c9d6b11/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

[Marios Andreou (marios-b)]

and more there from ussuri component pipeline

https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-ussuri/5e9b943/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-ussuri/1f6d3ab/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-ussuri/3b4ef2f/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-ussuri/a164688/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

[chandan kumar (chkumar246)]

On RDO-cloud, we are also seeing introspection issue
https://logserver.rdoproject.org/62/744762/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ecd8a2/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/081a430/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

But it is less. Since it is happening more on vexxhost. We need to find out the reason why it is happening irrespective of the cloud. May be Ironic team can help here.

[Bob Fournier (bfournie)]

Patch to increase default DHCP timeout - https://review.opendev.org/#/c/744939/.

[Marios Andreou (marios-b)]

OK looks like this is done now at least we haven't seen it in a few days.

Moving fix released please move back if you see this again or file a new issue thanks.