Introspection failing for OVB jobs - No nodes are manageable at this time.

[Sandeep Yadav]

Introspection failing for OVB jobs in master component pipeline with error - No nodes are manageable at this time.

Error Snippet
=============

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/04e70b0/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz

~~~
2020-04-06 08:14:18 | PLAY [Baremetal Introspection for multiple Ironic Nodes] ***********************
2020-04-06 08:14:18 |
2020-04-06 08:14:18 | TASK [Check for required inputs] ***********************************************
2020-04-06 08:14:18 | Monday 06 April 2020 08:14:18 +0000 (0:00:00.123) 0:00:00.123 **********
2020-04-06 08:14:18 | skipping: [localhost] => (item=node_uuids)
2020-04-06 08:14:18 |
2020-04-06 08:14:18 | TASK [Set node_uuids_intro fact] ***********************************************
2020-04-06 08:14:18 | Monday 06 April 2020 08:14:18 +0000 (0:00:00.065) 0:00:00.189 **********
2020-04-06 08:14:18 | ok: [localhost]
2020-04-06 08:14:18 |
2020-04-06 08:14:18 | TASK [Notice] ******************************************************************
2020-04-06 08:14:18 | Monday 06 April 2020 08:14:18 +0000 (0:00:00.062) 0:00:00.252 **********
2020-04-06 08:14:18 | fatal: [localhost]: FAILED! => changed=false
2020-04-06 08:14:18 | msg: No nodes are manageable at this time.
2020-04-06 08:14:18 |
2020-04-06 08:14:18 | NO MORE HOSTS LEFT *************************************************************
2020-04-06 08:14:18 |
2020-04-06 08:14:18 | PLAY RECAP *********************************************************************
2020-04-06 08:14:18 | localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
2020-04-06 08:14:18 | Monday 06 April 2020 08:14:18 +0000 (0:00:00.059) 0:00:00.312 **********
2020-04-06 08:14:18 | ===============================================================================
2020-04-06 08:14:18 | Check for required inputs ----------------------------------------------- 0.07s
2020-04-06 08:14:18 | Set node_uuids_intro fact ----------------------------------------------- 0.06s
2020-04-06 08:14:18 | Notice ------------------------------------------------------------------ 0.06s
2020-04-06 08:14:18 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
2020-04-06 08:14:18 | Exception occured while running the command
2020-04-06 08:14:18 | Traceback (most recent call last):
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-04-06 08:14:18 | super(Command, self).run(parsed_args)
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-04-06 08:14:18 | return super(Command, self).run(parsed_args)
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-04-06 08:14:18 | return_code = self.take_action(parsed_args) or 0
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_node.py", line 177, in take_action
2020-04-06 08:14:18 | verbosity=oooutils.playbook_verbosity(self=self)
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 222, in introspect_manageable_nodes
2020-04-06 08:14:18 | verbosity=verbosity
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 190, in introspect
2020-04-06 08:14:18 | "concurrency": concurrency,
2020-04-06 08:14:18 | File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 640, in run_ansible_playbook
2020-04-06 08:14:18 | raise RuntimeError(err_msg)
2020-04-06 08:14:18 | RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
2020-04-06 08:14:18 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
~~~

[Sandeep Yadav (sandeepyadav93)]

Hi,

On further check found metadata issue, detailed logs[1].

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master last run was on 2020-04-06. This issue seems duplicate of bug[1] which was fixed already yesterday.

~~~
[ 137.351520] cloud-init[857]: 2020-04-06 07:21:38,783 - url_helper.py[WARNING]: Calling 'http://192.168.100.1/latest/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='192.168.100.1', port=80): Max retries exceeded with url: /latest/meta-data/instance-id (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f16541b9350>: Failed to establish a new connection: [Errno 111] Connection refused',))]
~~~

[1] https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/04e70b0/logs/bmc-console.log
[2] https://bugs.launchpad.net/tripleo/+bug/1871076

[Sandeep Yadav (sandeepyadav93)]

This bug is duplicate of https://bugs.launchpad.net/tripleo/+bug/1871076, FTR admins did the same as in https://bugs.launchpad.net/tripleo/+bug/1790127/comments/2 - working now :)