openstack overcloud node introspect --all-manageable timing out and failing periodic tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master

[Bhagyashri Shewale]

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master

2020-05-18 01:50:39.423518 | primary | TASK [overcloud-prep-images : Introspect overcloud images] *********************
2020-05-18 01:50:39.423526 | primary | Monday 18 May 2020 01:50:39 +0000 (0:00:00.063) 0:01:26.760 ************
2020-05-18 05:57:46.241605 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2020-05-18 05:57:46.242135 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]

Affected jobs:

1. periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master
2. periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
3. periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master

Reference link:

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/57e5627/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/57e5627/logs/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log.txt.gz

[yatin (yatinkarel)]

The issue happened due to incompatibility in python-construct and python-ironic-inspector.
[1] updated required python-construct version by ironic-inspector and caused FTBFS[2]. To unclear it python-construct was updated to u-c version 2.10.56 with [3].

So if containers have latest ironic-inspector but old python-construct there will be issue or if containers have latest python-construct but old python-ironic-inspector then also there will be an issue. So both needs to be either latest or old in container fo job to work.

Now in component pipeline, since containers from current-tripleo are used, python-construct is older and ironic-inspector is latest so issue got hit and blocked component promotion for baremetal(component-ci-testing to promoted-components)[5]

And in integration pipeline, containers are rebuild, so python-construct is latest there but due to blocked baremetal component promotion, ironic-inspector was older so issue got hit here as well.

To unblock we forced promote baremetal component using [4], with this latest ironic-inspector got into integration pipeline. And this unblocked integration pipeline and it promoted, next run in baremetal-component should also work fine as current-tripleo containers contains both latest ironic-inspector and python-construct.

[1] https://review.opendev.org/#/c/724822/
[2] https://review.rdoproject.org/r/#/c/27549/1
[3] https://review.rdoproject.org/r/#/c/27552/
[4] https://review.rdoproject.org/r/#/c/27585/
[5] https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/6daffc1/logs/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log.txt.gz

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi All,

As master gets promoted recently so i am closing this bug.

Thanks Yatin!