openstack overcloud node introspect --all-manageable --provide no longer works

[Alex Schultz]

--all-manageable no longer picks up on the nodes in 'available' and using a uuid errors.

(undercloud) [centos@undercloud ~]$ openstack overcloud node import nodes.rdo.json
Waiting for messages on queue 'tripleo' with no timeout.

3 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID ebc0d52f-1c93-4c72-8b93-5b9e052ddc60
Successfully registered node UUID 0b3f9b16-3027-49f5-994a-bdf830f8837f
Successfully registered node UUID 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd
(undercloud) [centos@undercloud ~]$ openstack overcloud node introspect --all-manageable --provide

PLAY [Baremetal Introspection for multiple Ironic Nodes] ***********************

TASK [Check for required inputs] ***********************************************
Thursday 05 March 2020 19:48:25 +0000 (0:00:00.103) 0:00:00.103 ********
skipping: [localhost] => (item=node_uuids)

TASK [Set node_uuids_intro fact] ***********************************************
Thursday 05 March 2020 19:48:25 +0000 (0:00:00.064) 0:00:00.168 ********
ok: [localhost]

TASK [Notice] ******************************************************************
Thursday 05 March 2020 19:48:25 +0000 (0:00:00.064) 0:00:00.233 ********
ok: [localhost] =>
  msg: No nodes are manageable at this time.

PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Thursday 05 March 2020 19:48:25 +0000 (0:00:00.068) 0:00:00.301 ********
===============================================================================
Notice ------------------------------------------------------------------ 0.07s
Check for required inputs ----------------------------------------------- 0.07s
Set node_uuids_intro fact ----------------------------------------------- 0.06s
Waiting for messages on queue 'tripleo' with no timeout.
3 node(s) successfully moved to the "available" state.

(undercloud) [centos@undercloud ~]$ openstack baremetal node list
+--------------------------------------+----------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------------+---------------+-------------+--------------------+-------------+
| ebc0d52f-1c93-4c72-8b93-5b9e052ddc60 | aschultz-baremetal-0 | None | power off | available | False |
| 0b3f9b16-3027-49f5-994a-bdf830f8837f | aschultz-baremetal-1 | None | power off | available | False |
| 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd | aschultz-baremetal-2 | None | power off | available | False |
+--------------------------------------+----------------------+---------------+-------------+--------------------+-------------+
(undercloud) [centos@undercloud ~]$ openstack overcloud node introspect --help
usage: openstack overcloud node introspect [-h] [--all-manageable] [--provide]
                                           [--run-validations]
                                           [--concurrency CONCURRENCY]
                                           [<node_uuid> [<node_uuid> ...]]

Introspect specified nodes or all nodes in 'manageable' state.

positional arguments:
  <node_uuid> Baremetal Node UUIDs for the node(s) to be
                        introspected

optional arguments:
  -h, --help show this help message and exit
  --all-manageable Introspect all nodes currently in 'manageable' state
  --provide Provide (make available) the nodes once introspected
  --run-validations Run the pre-deployment validations. These external
                        validations are from the TripleO Validations project.
  --concurrency CONCURRENCY
                        Maximum number of nodes to introspect at once.

This command is provided by the python-tripleoclient plugin.
(undercloud) [centos@undercloud ~]$ openstack overcloud node introspect --provide ebc0d52f-1c93-4c72-8b93-5b9e052ddc60 0b3f9b16-3027-49f5-994a-bdf830f8837f 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd

PLAY [Baremetal Introspection for multiple Ironic Nodes] ***********************

TASK [Check for required inputs] ***********************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.085) 0:00:00.085 ********
skipping: [localhost] => (item=node_uuids)

TASK [Set node_uuids_intro fact] ***********************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.065) 0:00:00.150 ********
ok: [localhost]

TASK [Notice] ******************************************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.066) 0:00:00.217 ********
skipping: [localhost]

TASK [Run Validations] *********************************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.071) 0:00:00.289 ********
skipping: [localhost]

TASK [Start baremetal introspection] *******************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.061) 0:00:00.351 ********
changed: [localhost]

TASK [Viewing introspection progress] ******************************************
Thursday 05 March 2020 19:50:41 +0000 (0:00:00.568) 0:00:00.919 ********
ok: [localhost] =>
  msg: Introspection progress can been viewed in the syslog

TASK [Wait for introspection to complete] **************************************
Thursday 05 March 2020 19:50:42 +0000 (0:00:00.067) 0:00:00.987 ********
FAILED - RETRYING: Wait for introspection to complete (1000 retries left).
fatal: [localhost]: FAILED! => changed=true
  ansible_job_id: '528155422113.260463'
  attempts: 2
  failed_nodes:
  - ebc0d52f-1c93-4c72-8b93-5b9e052ddc60
  - 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd
  - 0b3f9b16-3027-49f5-994a-bdf830f8837f
  finished: 1
  introspection_data:
    0b3f9b16-3027-49f5-994a-bdf830f8837f:
      error: 'Error for introspection node 0b3f9b16-3027-49f5-994a-bdf830f8837f: BadRequestException: 400: Client Error for url: https://192.168.24.2:13050/v1/introspection/0b3f9b16-3027-49f5-994a-bdf830f8837f, Invalid provision state for introspection: "available", valid states are "[''enroll'', ''manageable'', ''inspect wait'', ''inspecting'', ''inspect failed'']" '
      failed: true
      status: ''
    0cacc47d-7bf4-447c-a6d1-ef637ed05dbd:
      error: 'Error for introspection node 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd: BadRequestException: 400: Client Error for url: https://192.168.24.2:13050/v1/introspection/0cacc47d-7bf4-447c-a6d1-ef637ed05dbd, Invalid provision state for introspection: "available", valid states are "[''enroll'', ''manageable'', ''inspect wait'', ''inspecting'', ''inspect failed'']" '
      failed: true
      status: ''
    ebc0d52f-1c93-4c72-8b93-5b9e052ddc60:
      error: 'Error for introspection node ebc0d52f-1c93-4c72-8b93-5b9e052ddc60: BadRequestException: 400: Client Error for url: https://192.168.24.2:13050/v1/introspection/ebc0d52f-1c93-4c72-8b93-5b9e052ddc60, Invalid provision state for introspection: "available", valid states are "[''enroll'', ''manageable'', ''inspect wait'', ''inspecting'', ''inspect failed'']" '
      failed: true
      status: ''
  msg: Introspection completed with failures. 3 node(s) failed.
  passed_nodes: []
...ignoring

TASK [Nodes that passed introspection] *****************************************
Thursday 05 March 2020 19:50:47 +0000 (0:00:05.468) 0:00:06.455 ********
ok: [localhost] =>
  msg: ' No nodes completed introspection successfully!'

TASK [Nodes that failed introspection] *****************************************
Thursday 05 March 2020 19:50:47 +0000 (0:00:00.063) 0:00:06.519 ********
fatal: [localhost]: FAILED! =>
  msg: ' ebc0d52f-1c93-4c72-8b93-5b9e052ddc60 0cacc47d-7bf4-447c-a6d1-ef637ed05dbd 0b3f9b16-3027-49f5-994a-bdf830f8837f'

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost : ok=5 changed=2 unreachable=0 failed=1 skipped=3 rescued=0 ignored=1
Thursday 05 March 2020 19:50:47 +0000 (0:00:00.062) 0:00:06.582 ********
===============================================================================
Wait for introspection to complete -------------------------------------- 5.47s
Start baremetal introspection ------------------------------------------- 0.57s
Notice ------------------------------------------------------------------ 0.07s
Viewing introspection progress ------------------------------------------ 0.07s
Check for required inputs ----------------------------------------------- 0.07s
Set node_uuids_intro fact ----------------------------------------------- 0.07s
Nodes that passed introspection ----------------------------------------- 0.06s
Nodes that failed introspection ----------------------------------------- 0.06s
Run Validations --------------------------------------------------------- 0.06s
Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v2/overcloud_node.py", line 86, in take_action
    extra_vars=extra_vars
  File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 620, in run_ansible_playbook
    raise RuntimeError(err_msg)
RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
(undercloud) [centos@undercloud ~]$ ^C
(undercloud) [centos@undercloud ~]$ openstack overcloud node introspect --provide ebc0d52f-1c93-4c72-8b93-5b9e052ddc60

PLAY [Baremetal Introspection for multiple Ironic Nodes] ***********************

TASK [Check for required inputs] ***********************************************
Thursday 05 March 2020 19:53:17 +0000 (0:00:00.084) 0:00:00.084 ********
skipping: [localhost] => (item=node_uuids)

TASK [Set node_uuids_intro fact] ***********************************************
Thursday 05 March 2020 19:53:17 +0000 (0:00:00.063) 0:00:00.147 ********
ok: [localhost]

TASK [Notice] ******************************************************************
Thursday 05 March 2020 19:53:17 +0000 (0:00:00.063) 0:00:00.211 ********
skipping: [localhost]

TASK [Run Validations] *********************************************************
Thursday 05 March 2020 19:53:17 +0000 (0:00:00.073) 0:00:00.284 ********
skipping: [localhost]

TASK [Start baremetal introspection] *******************************************
Thursday 05 March 2020 19:53:17 +0000 (0:00:00.062) 0:00:00.347 ********
changed: [localhost]

TASK [Viewing introspection progress] ******************************************
Thursday 05 March 2020 19:53:18 +0000 (0:00:00.515) 0:00:00.862 ********
ok: [localhost] =>
  msg: Introspection progress can been viewed in the syslog

TASK [Wait for introspection to complete] **************************************
Thursday 05 March 2020 19:53:18 +0000 (0:00:00.065) 0:00:00.928 ********
FAILED - RETRYING: Wait for introspection to complete (1000 retries left).
fatal: [localhost]: FAILED! => changed=true
  ansible_job_id: '120849419049.262626'
  attempts: 2
  failed_nodes:
  - ebc0d52f-1c93-4c72-8b93-5b9e052ddc60
  finished: 1
  introspection_data:
    ebc0d52f-1c93-4c72-8b93-5b9e052ddc60:
      error: 'Error for introspection node ebc0d52f-1c93-4c72-8b93-5b9e052ddc60: BadRequestException: 400: Client Error for url: https://192.168.24.2:13050/v1/introspection/ebc0d52f-1c93-4c72-8b93-5b9e052ddc60, Invalid provision state for introspection: "available", valid states are "[''enroll'', ''manageable'', ''inspect wait'', ''inspecting'', ''inspect failed'']" '
      failed: true
      status: ''
  msg: Introspection completed with failures. 1 node(s) failed.
  passed_nodes: []
...ignoring

TASK [Nodes that passed introspection] *****************************************
Thursday 05 March 2020 19:53:23 +0000 (0:00:05.453) 0:00:06.381 ********
ok: [localhost] =>
  msg: ' No nodes completed introspection successfully!'

TASK [Nodes that failed introspection] *****************************************
Thursday 05 March 2020 19:53:23 +0000 (0:00:00.064) 0:00:06.446 ********
fatal: [localhost]: FAILED! =>
  msg: ' ebc0d52f-1c93-4c72-8b93-5b9e052ddc60'

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost : ok=5 changed=2 unreachable=0 failed=1 skipped=3 rescued=0 ignored=1
Thursday 05 March 2020 19:53:23 +0000 (0:00:00.061) 0:00:06.507 ********
===============================================================================
Wait for introspection to complete -------------------------------------- 5.45s
Start baremetal introspection ------------------------------------------- 0.52s
Notice ------------------------------------------------------------------ 0.07s
Viewing introspection progress ------------------------------------------ 0.07s
Nodes that passed introspection ----------------------------------------- 0.06s
Check for required inputs ----------------------------------------------- 0.06s
Run Validations --------------------------------------------------------- 0.06s
Set node_uuids_intro fact ----------------------------------------------- 0.06s
Nodes that failed introspection ----------------------------------------- 0.06s
Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 34, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v2/overcloud_node.py", line 86, in take_action
    extra_vars=extra_vars
  File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 620, in run_ansible_playbook
    raise RuntimeError(err_msg)
RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2

[Alex Schultz (alex-schultz)]

two issues:

1) --all-manageable no longer works

2) --provide seems to be running prior to the introspection which breaks things

[Alex Schultz (alex-schultz)]

The bug is that --all-manageable is currently a noop, so --provide will move the nodes into the available state anyway. https://review.opendev.org/#/c/711543/ should address the --all-manageable flag not properly functioning. In the mean time openstack overcloud node introspect --provide <uuid> <uuid> should still work

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711543
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=c27075ba70cf316d07dfcc4c391652b50e2abf30
Submitter: Zuul
Branch: master

commit c27075ba70cf316d07dfcc4c391652b50e2abf30
Author: Kevin Carter (cloudnull) <email address hidden>
Date: Thu Mar 5 20:09:10 2020 +0000

    Revert "Modify the introspection playbook to simplify client interactions"

    This reverts commit d2b745cf4be5292d54b7715290e093788e355d8c.

    Closes-Bug: #1866233
    Change-Id: I1f8e2e548754bd64440e2f98543c0af3ba70b1ea

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 1.3.0 release.