overcloud-prep-images failes at Introspect overcloud images (fs001, fs002, fs020and fs039 is failing)

[Bhagyashri Shewale]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/0ca0cb9/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz

2020-09-28 02:01:34 | /usr/lib64/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
2020-09-28 02:01:34 | return f(*args, **kwds)
2020-09-28 02:01:34 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
2020-09-28 02:01:34 | Exception occured while running the command
2020-09-28 02:01:34 | Traceback (most recent call last):
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-09-28 02:01:34 | super(Command, self).run(parsed_args)
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2020-09-28 02:01:34 | return super(Command, self).run(parsed_args)
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-09-28 02:01:34 | return_code = self.take_action(parsed_args) or 0
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_node.py", line 190, in take_action
2020-09-28 02:01:34 | verbosity=oooutils.playbook_verbosity(self=self)
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 237, in introspect_manageable_nodes
2020-09-28 02:01:34 | verbosity=verbosity
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 191, in introspect
2020-09-28 02:01:34 | "retry_timeout": retry_timeout,
2020-09-28 02:01:34 | File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 719, in run_ansible_playbook
2020-09-28 02:01:34 | raise RuntimeError(err_msg)
2020-09-28 02:01:34 | RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
2020-09-28 02:01:34 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2

-----------------------ironic inspector logs---------------------

2020-09-28 02:01:28.145 8 ERROR ironic_inspector.utils [-] Failed to set power state for node 6ffc264c-02be-4952-81b3-34e24c621310 Error: Failed to set power state for bare metal node 6ffc264c-02be-4952-81b3-34e24c621310 to power off: Client Error for url: http://192.168.24.3:6385/v1/nodes/6ffc264c-02be-4952-81b3-34e24c621310/states/power, Node 6ffc264c-02be-4952-81b3-34e24c621310 is locked by host undercloud.localdomain, please retry after the current operation is completed.: openstack.exceptions.ConflictException: Failed to set power state for bare metal node 6ffc264c-02be-4952-81b3-34e24c621310 to power off: Client Error for url: http://192.168.24.3:6385/v1/nodes/6ffc264c-02be-4952-81b3-34e24c621310/states/power, Node 6ffc264c-02be-4952-81b3-34e24c621310 is locked by host undercloud.localdomain, please retry after the current operation is completed.
2020-09-28 02:01:28.145 8 ERROR ironic_inspector.node_cache [-] [node: 6ffc264c-02be-4952-81b3-34e24c621310 state starting] Processing the error event because of an exception <class 'ironic_inspector.utils.Error'>: Failed to set power state for node 6ffc264c-02be-4952-81b3-34e24c621310 Error: Failed to set power state for bare metal node 6ffc264c-02be-4952-81b3-34e24c621310 to power off: Client Error for url: http://192.168.24.3:6385/v1/nodes/6ffc264c-02be-4952-81b3-34e24c621310/states/power, Node 6ffc264c-02be-4952-81b3-34e24c621310 is locked by host undercloud.localdomain, please retry after the current operation is completed. raised by ironic_inspector.introspect._do_introspect: ironic_inspector.utils.Error: Failed to set power state for node 6ffc264c-02be-4952-81b3-34e24c621310 Error: Failed to set power state for bare metal node 6ffc264c-02be-4952-81b3-34e24c621310 to power off: Client Error for url: http://192.168.24.3:6385/v1/nodes/6ffc264c-02be-4952-81b3-34e24c621310/states/power, Node 6ffc264c-02be-4952-81b3-34e24c621310 is locked by host undercloud.localdomain, please retry after the current operation is completed.

Refence link:

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/0ca0cb9/logs/bmc-console.log

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/0ca0cb9/logs/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log.txt.gz

[wes hayutin (weshayutin)]

https://review.opendev.org/#/c/754783/ is the fix required.. revert

[wes hayutin (weshayutin)]

This change https://review.opendev.org/#/c/749845/ was causing this issue.
Revert the change here : https://review.opendev.org/#/c/754783/

After revert https://review.opendev.org/#/c/754783/ got merged, Now the jobs are passing.