> I think the ages to boot is not quite accurrate,
OK. I just noticed the following in conductor logs which made me think that probably some issue with powering on the nodes...
2021-10-19 08:18:11.128 8 DEBUG ironic.common.utils [-] Command stderr is: "" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:79
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall [-] Dynamic backoff interval looping call 'ironic.conductor.utils.node_wait_for_power_state.<locals>._wait' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 20.40 seconds
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall Traceback (most recent call last):
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall File "/usr/lib/python3.6/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall idle = idle_for_func(result, self._elapsed(watch))
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall File "/usr/lib/python3.6/site-packages/oslo_service/loopingcall.py", line 351, in _idle_for
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall % self._error_time)
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 20.40 seconds
2021-10-19 08:18:11.129 8 ERROR oslo.service.loopingcall
2021-10-19 08:18:11.133 8 ERROR ironic.conductor.utils [req-06824ad3-b2ed-4557-a5b0-b8bfa0b3b061 4d8927747a344c9590e907366812a57a f0136f62e9b04cc6bee7ed60d56dd2c5 - default default] Timed out after 30 secs waiting for power off on node 102752b7-61e1-4a34-98a6-fe4e38b475ce.: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 20.40 seconds
....
2021-10-19 08:19:28.677 8 WARNING ironic.conductor.manager [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] During sync_power_state, node b981374c-3f42-4e99-8f85-c0b3109f4e38 state does not match expected state 'power on'. Updating recorded state to 'power off'.
2021-10-19 08:19:28.682 8 DEBUG ironic.conductor.task_manager [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] Node 102752b7-61e1-4a34-98a6-fe4e38b475ce successfully reserved for power state sync (took 0.04 seconds) reserve_node /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:339
2021-10-19 08:19:28.682 8 WARNING ironic.conductor.manager [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] During sync_power_state, node 102752b7-61e1-4a34-98a6-fe4e38b475ce state does not match expected state 'power on'. Updating recorded state to 'power off'.
2021-10-19 08:19:28.703 8 DEBUG ironic.conductor.task_manager [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] Node e2b3a3d2-93aa-43a5-bc10-7c6a3559e765 successfully reserved for power state sync (took 0.03 seconds) reserve_node /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:339
2021-10-19 08:19:28.704 8 WARNING ironic.conductor.manager [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] During sync_power_state, node e2b3a3d2-93aa-43a5-bc10-7c6a3559e765 state does not match expected state 'power on'. Updating recorded state to 'power off'.
...
2021-10-19 08:19:28.603 8 DEBUG ironic.common.utils [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] Execution completed, command line is "ipmitool -I lanplus -H 10.0.1.89 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpp1y8kdm5 power status" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:77
2021-10-19 08:19:28.604 8 DEBUG ironic.common.utils [req-0daec60d-e154-48e1-8ca8-4fafcb28c015 - - - - -] Command stdout is: "Chassis Power is off
" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:78
Then the nodes state was powered off till an hour or so....
> I think the ages to boot is not quite accurrate,
OK. I just noticed the following in conductor logs which made me think that probably some issue with powering on the nodes...
2021-10-19 08:18:11.128 8 DEBUG ironic.common.utils [-] Command stderr is: "" execute /usr/lib/ python3. 6/site- packages/ ironic/ common/ utils.py: 79 loopingcall [-] Dynamic backoff interval looping call 'ironic. conductor. utils.node_ wait_for_ power_state. <locals> ._wait' failed: oslo_service. loopingcall. LoopingCallTime Out: Looping call timed out after 20.40 seconds loopingcall Traceback (most recent call last): loopingcall File "/usr/lib/ python3. 6/site- packages/ oslo_service/ loopingcall. py", line 154, in _run_loop loopingcall idle = idle_for_ func(result, self._elapsed( watch)) loopingcall File "/usr/lib/ python3. 6/site- packages/ oslo_service/ loopingcall. py", line 351, in _idle_for loopingcall % self._error_time) loopingcall oslo_service. loopingcall. LoopingCallTime Out: Looping call timed out after 20.40 seconds loopingcall conductor. utils [req-06824ad3- b2ed-4557- a5b0-b8bfa0b3b0 61 4d8927747a344c9 590e907366812a5 7a f0136f62e9b04cc 6bee7ed60d56dd2 c5 - default default] Timed out after 30 secs waiting for power off on node 102752b7- 61e1-4a34- 98a6-fe4e38b475 ce.: oslo_service. loopingcall. LoopingCallTime Out: Looping call timed out after 20.40 seconds
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.129 8 ERROR oslo.service.
2021-10-19 08:18:11.133 8 ERROR ironic.
....
2021-10-19 08:19:28.677 8 WARNING ironic. conductor. manager [req-0daec60d- e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] During sync_power_state, node b981374c- 3f42-4e99- 8f85-c0b3109f4e 38 state does not match expected state 'power on'. Updating recorded state to 'power off'. conductor. task_manager [req-0daec60d- e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] Node 102752b7- 61e1-4a34- 98a6-fe4e38b475 ce successfully reserved for power state sync (took 0.04 seconds) reserve_node /usr/lib/ python3. 6/site- packages/ ironic/ conductor/ task_manager. py:339 conductor. manager [req-0daec60d- e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] During sync_power_state, node 102752b7- 61e1-4a34- 98a6-fe4e38b475 ce state does not match expected state 'power on'. Updating recorded state to 'power off'. conductor. task_manager [req-0daec60d- e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] Node e2b3a3d2- 93aa-43a5- bc10-7c6a3559e7 65 successfully reserved for power state sync (took 0.03 seconds) reserve_node /usr/lib/ python3. 6/site- packages/ ironic/ conductor/ task_manager. py:339 conductor. manager [req-0daec60d- e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] During sync_power_state, node e2b3a3d2- 93aa-43a5- bc10-7c6a3559e7 65 state does not match expected state 'power on'. Updating recorded state to 'power off'. e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] Execution completed, command line is "ipmitool -I lanplus -H 10.0.1.89 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpp1y8kdm5 power status" execute /usr/lib/ python3. 6/site- packages/ ironic/ common/ utils.py: 77 e154-48e1- 8ca8-4fafcb28c0 15 - - - - -] Command stdout is: "Chassis Power is off python3. 6/site- packages/ ironic/ common/ utils.py: 78
2021-10-19 08:19:28.682 8 DEBUG ironic.
2021-10-19 08:19:28.682 8 WARNING ironic.
2021-10-19 08:19:28.703 8 DEBUG ironic.
2021-10-19 08:19:28.704 8 WARNING ironic.
...
2021-10-19 08:19:28.603 8 DEBUG ironic.common.utils [req-0daec60d-
2021-10-19 08:19:28.604 8 DEBUG ironic.common.utils [req-0daec60d-
" execute /usr/lib/
Then the nodes state was powered off till an hour or so....