Comment 1 for bug 2016307

INFO ironic.conductor.task_manager [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Node cdef5b7c-2650-4175-9ebd-1a10686bcac2 moved to provision state "deleting" from state "active"; target provision state is "available"
INFO eventlet.wsgi.server [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] ::ffff:10.255.106.2 "POST / HTTP/1.1" status: 200 len: 211 time: 0.4214077
INFO ironic.conductor.utils [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Node cdef5b7c-2650-4175-9ebd-1a10686bcac2 current power state is 'power on', requested state is 'power off'.
INFO ironic.conductor.utils [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Successfully set node cdef5b7c-2650-4175-9ebd-1a10686bcac2 power state to power off by power off.
INFO ironic.conductor.manager [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Successfully unprovisioned node cdef5b7c-2650-4175-9ebd-1a10686bcac2 with instance None.
INFO ironic.conductor.task_manager [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Node cdef5b7c-2650-4175-9ebd-1a10686bcac2 moved to provision state "cleaning" from state "deleting"; target provision state is "available"
ERROR oslo.service.loopingcall [-] Dynamic backoff interval looping call 'ironic.conductor.utils.node_wait_for_power_state.<locals>._wait' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 69.25 seconds
ERROR oslo.service.loopingcall Traceback (most recent call last):
ERROR oslo.service.loopingcall File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop
ERROR oslo.service.loopingcall idle = idle_for_func(result, self._elapsed(watch))
ERROR oslo.service.loopingcall File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_service/loopingcall.py", line 349, in _idle_for
ERROR oslo.service.loopingcall raise LoopingCallTimeOut(
ERROR oslo.service.loopingcall oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 69.25 seconds
ERROR oslo.service.loopingcall
ERROR ironic.conductor.utils [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Timed out after 60 secs waiting for power on on node cdef5b7c-2650-4175-9ebd-1a10686bcac2.: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 69.25 seconds
ERROR ironic.conductor.utils [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Failed to prepare node cdef5b7c-2650-4175-9ebd-1a10686bcac2 for cleaning: Failed to set node power state to power on.: ironic.common.exception.PowerStateFailure: Failed to set node power state to power on.
ERROR ironic.conductor.utils Traceback (most recent call last):
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/utils.py", line 183, in node_wait_for_power_state
ERROR ironic.conductor.utils return timer.start(initial_delay=1, timeout=retry_timeout).wait()
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/eventlet/event.py", line 125, in wait
ERROR ironic.conductor.utils result = hub.switch()
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
ERROR ironic.conductor.utils return self.greenlet.switch()
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop
ERROR ironic.conductor.utils idle = idle_for_func(result, self._elapsed(watch))
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_service/loopingcall.py", line 349, in _idle_for
ERROR ironic.conductor.utils raise LoopingCallTimeOut(
ERROR ironic.conductor.utils oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 69.25 seconds
ERROR ironic.conductor.utils
ERROR ironic.conductor.utils During handling of the above exception, another exception occurred:
ERROR ironic.conductor.utils
ERROR ironic.conductor.utils Traceback (most recent call last):
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/cleaning.py", line 91, in do_node_clean
ERROR ironic.conductor.utils prepare_result = task.driver.deploy.prepare_cleaning(task)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic_lib/metrics.py", line 59, in wrapped
ERROR ironic.conductor.utils result = f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/drivers/modules/agent_base.py", line 726, in prepare_cleaning
ERROR ironic.conductor.utils result = deploy_utils.prepare_inband_cleaning(
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/drivers/modules/deploy_utils.py", line 732, in prepare_inband_cleaning
ERROR ironic.conductor.utils manager_utils.node_power_action(task, states.REBOOT)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/task_manager.py", line 153, in wrapper
ERROR ironic.conductor.utils return f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/utils.py", line 350, in node_power_action
ERROR ironic.conductor.utils notify_utils.emit_power_set_notification(
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR ironic.conductor.utils self.force_reraise()
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR ironic.conductor.utils raise self.value
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/utils.py", line 338, in node_power_action
ERROR ironic.conductor.utils task.driver.power.reboot(task, timeout=timeout)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/task_manager.py", line 153, in wrapper
ERROR ironic.conductor.utils return f(*args, **kwargs)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/drivers/modules/redfish/power.py", line 166, in reboot
ERROR ironic.conductor.utils _set_power_state(task, system, next_state, timeout=timeout)
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/drivers/modules/redfish/power.py", line 68, in _set_power_state
ERROR ironic.conductor.utils cond_utils.node_wait_for_power_state(task, target_state,
ERROR ironic.conductor.utils File "/usr/venv-ironic/lib64/python3.8/site-packages/ironic/conductor/utils.py", line 189, in node_wait_for_power_state
ERROR ironic.conductor.utils raise exception.PowerStateFailure(pstate=new_state)
ERROR ironic.conductor.utils ironic.common.exception.PowerStateFailure: Failed to set node power state to power on.
ERROR ironic.conductor.utils
ERROR ironic.conductor.task_manager [None req-01c83d87-b8e2-469c-bdc0-3c5d51edb050 a1336539b1204341a2cc47d59cda0837 8299bb6b4a4b4af5a7ca24d31a9cc42d - - default default] Node cdef5b7c-2650-4175-9ebd-1a10686bcac2 moved to provision state "clean failed" from state "cleaning"; target provision state is "available": ironic.common.exception.PowerStateFailure: Failed to set node power state to power on.
WARNING ironic.conductor.manager [None req-cf674b97-b95e-4110-8e8b-20e7be343d0c - - - - - -] During sync_power_state, node cdef5b7c-2650-4175-9ebd-1a10686bcac2 state 'None' does not match expected state. Changing hardware state to 'power off'.
INFO ironic.conductor.utils [None req-cf674b97-b95e-4110-8e8b-20e7be343d0c - - - - - -] Node cdef5b7c-2650-4175-9ebd-1a10686bcac2 current power state is 'None', requested state is 'power off'.
ERROR ironic.drivers.modules.redfish.power [None req-cf674b97-b95e-4110-8e8b-20e7be343d0c - - - - - -] Setting power state to power off failed for node cdef5b7c-2650-4175-9ebd-1a10686bcac2. Error: HTTP POST https://10.219.199.205/redfish/v1/Systems/1/Actions/ComputerSystem.Reset returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. Extended information: [{'MessageArgs': ['Power Management Error'], 'MessageId': 'iLO.2.15.InvalidOperationForSystemState'}]: sushy.exceptions.BadRequestError: HTTP POST https://10.219.199.205/redfish/v1/Systems/1/Actions/ComputerSystem.Reset returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. Extended information: [{'MessageArgs': ['Power Management Error'], 'MessageId': 'iLO.2.15.InvalidOperationForSystemState'}]
ERROR ironic.conductor.manager [None req-cf674b97-b95e-4110-8e8b-20e7be343d0c - - - - - -] Failed to change power state of node cdef5b7c-2650-4175-9ebd-1a10686bcac2 to 'power off', attempt 1 of 3.: ironic.common.exception.RedfishError: Redfish exception occurred. Error: Setting power state to power off failed for node cdef5b7c-2650-4175-9ebd-1a10686bcac2. Error: HTTP POST https://10.219.199.205/redfish/v1/Systems/1/Actions/ComputerSystem.Reset returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. Extended information: [{'MessageArgs': ['Power Management Error'], 'MessageId': 'iLO.2.15.InvalidOperationForSystemState'}]