I do have this error looping every minute or so.
2019-05-17 21:33:39.463 8 DEBUG ironicclient.common.http [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] No version header in response, requesting from server negotiate_version /usr/lib/python3.6/site-packages/ironicclient/common/http.py:157 2019-05-17 21:33:39.473 8 ERROR nova.virt.ironic.driver [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: 'StrictVersion' object has no attribute 'version' 2019-05-17 21:33:39.473 8 WARNING nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Virt driver is not ready.: nova.exception.VirtDriverNotReady: Virt driver is not ready. 2019-05-17 21:33:40.473 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._heal_instance_info_cache run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:40.473 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Starting heal instance info cache _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7247 2019-05-17 21:33:40.474 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Rebuilding the list of instances to heal _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7251 2019-05-17 21:33:40.487 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Didn't find any instances for network info cache update. _heal_instance_info_cache /usr/lib/python3.6/site-packages/nova/compute/manager.py:7333 2019-05-17 21:33:40.487 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:40.488 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._reclaim_queued_deletes run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:40.488 8 DEBUG nova.compute.manager [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] CONF.reclaim_instance_interval <= 0, skipping... _reclaim_queued_deletes /usr/lib/python3.6/site-packages/nova/compute/manager.py:7946 2019-05-17 21:33:41.422 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:41.423 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_rescued_instances run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:43.422 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:46.415 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._sync_scheduler_instance_info run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:47.429 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._check_instance_build_time run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217 2019-05-17 21:33:49.422 8 DEBUG oslo_service.periodic_task [req-f1d152cb-d19c-4946-b0b0-baaf2f9e4804 - - - - -] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
and some error in mistral when trying to introspect overcloud nodes:
2019-05-17 21:21:09.091 8 INFO workflow_trace [req-d462b3f8-acde-4224-9977-877545feed2e 022b15c9e0b14450b55c71d23bce3253 561ad084af1941b5931dc317db563e12 - default default] Task 'cell_v2_discover_hosts' (ff79c16c-b48e-495a-b2cc-38d9e92df08f) [RUNNING -> ERROR, msg=Failure caused by error in tasks: wait_for_nova_resources
wait_for_nova_resources [task_ex_id=ec65be73-4f36-425c-9bd4-6684456c3b32] -> One or more actions had failed. [action_ex_id=15a7c7e2-558b-4548-b73b-a78d4b7be594, idx=0]: The action raised an exception [action_ex_id=15a7c7e2-558b-4548-b73b-a78d4b7be594, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '005d8935-9991-40bc-95be-dd0eb0e3830e'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '005d8935-9991-40bc-95be-dd0eb0e3830e'}. (HTTP 404) [action_ex_id=266a8a2b-4218-441d-902c-3f7da984d912, idx=1]: The action raised an exception [action_ex_id=266a8a2b-4218-441d-902c-3f7da984d912, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '4b0e84d7-7810-4a42-81d5-6b14579362c8'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '4b0e84d7-7810-4a42-81d5-6b14579362c8'}. (HTTP 404) [action_ex_id=4f6e7bea-55ef-4b92-8539-76f2d2decfa4, idx=2]: The action raised an exception [action_ex_id=4f6e7bea-55ef-4b92-8539-76f2d2decfa4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': 'c586ead4-07b0-4e3c-983e-ad1ece3ffc1e'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': 'c586ead4-07b0-4e3c-983e-ad1ece3ffc1e'}. (HTTP 404) [action_ex_id=64e5320d-fb64-49b0-8cb0-eb62f04f4170, idx=3]: The action raised an exception [action_ex_id=64e5320d-fb64-49b0-8cb0-eb62f04f4170, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '6bf2c92f-8eaf-4596-982c-87cef89337b7'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '6bf2c92f-8eaf-4596-982c-87cef89337b7'}. (HTTP 404) [action_ex_id=83e97b9c-2775-4752-962d-d19cc728ff0c, idx=4]: The action raised an exception [action_ex_id=83e97b9c-2775-4752-962d-d19cc728ff0c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '6f539c86-3905-4014-8973-f515343c6c4b'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '6f539c86-3905-4014-8973-f515343c6c4b'}. (HTTP 404) [action_ex_id=b7881d3d-78c7-4898-a8cd-36a8b25112a5, idx=5]: The action raised an exception [action_ex_id=b7881d3d-78c7-4898-a8cd-36a8b25112a5, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': '82cdb49e-a248-4221-8206-62813b6deb26'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': '82cdb49e-a248-4221-8206-62813b6deb26'}. (HTTP 404) [action_ex_id=c0aca431-e692-45e0-98ef-2f5fcf8fa623, idx=6]: The action raised an exception [action_ex_id=c0aca431-e692-45e0-98ef-2f5fcf8fa623, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{'client_method_name': 'hypervisors.find'}', params='{'hypervisor_hostname': 'd1e608f4-9e95-4d85-8d8b-ad8ee73c01d5'}'] NovaAction.hypervisors.find failed: No Hypervisor matching {'hypervisor_hostname': 'd1e608f4-9e95-4d85-8d8b-ad8ee73c01d5'}. (HTTP 404) ] (execution_id=e0b1ab58-9b03-4088-a008-a92614dcd892) (END)
I do have this error looping every minute or so.
2019-05-17 21:33:39.463 8 DEBUG ironicclient. common. http [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] No version header in response, requesting from server negotiate_version /usr/lib/ python3. 6/site- packages/ ironicclient/ common/ http.py: 157 ironic. driver [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: 'StrictVersion' object has no attribute 'version' manager [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Virt driver is not ready.: nova.exception. VirtDriverNotRe ady: Virt driver is not ready. periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _heal_instance_ info_cache run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 manager [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Starting heal instance info cache _heal_instance_ info_cache /usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py:7247 manager [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Rebuilding the list of instances to heal _heal_instance_ info_cache /usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py:7251 manager [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Didn't find any instances for network info cache update. _heal_instance_ info_cache /usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py:7333 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _poll_unconfirm ed_resizes run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _reclaim_ queued_ deletes run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 manager [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] CONF.reclaim_ instance_ interval <= 0, skipping... _reclaim_ queued_ deletes /usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py:7946 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _poll_rebooting _instances run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _poll_rescued_ instances run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _instance_ usage_audit run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _sync_scheduler _instance_ info run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _check_ instance_ build_time run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217 periodic_ task [req-f1d152cb- d19c-4946- b0b0-baaf2f9e48 04 - - - - -] Running periodic task ComputeManager. _poll_volume_ usage run_periodic_tasks /usr/lib/ python3. 6/site- packages/ oslo_service/ periodic_ task.py: 217
2019-05-17 21:33:39.473 8 ERROR nova.virt.
2019-05-17 21:33:39.473 8 WARNING nova.compute.
2019-05-17 21:33:40.473 8 DEBUG oslo_service.
2019-05-17 21:33:40.473 8 DEBUG nova.compute.
2019-05-17 21:33:40.474 8 DEBUG nova.compute.
2019-05-17 21:33:40.487 8 DEBUG nova.compute.
2019-05-17 21:33:40.487 8 DEBUG oslo_service.
2019-05-17 21:33:40.488 8 DEBUG oslo_service.
2019-05-17 21:33:40.488 8 DEBUG nova.compute.
2019-05-17 21:33:41.422 8 DEBUG oslo_service.
2019-05-17 21:33:41.423 8 DEBUG oslo_service.
2019-05-17 21:33:43.422 8 DEBUG oslo_service.
2019-05-17 21:33:46.415 8 DEBUG oslo_service.
2019-05-17 21:33:47.429 8 DEBUG oslo_service.
2019-05-17 21:33:49.422 8 DEBUG oslo_service.
and some error in mistral when trying to introspect overcloud nodes:
2019-05-17 21:21:09.091 8 INFO workflow_trace [req-d462b3f8- acde-4224- 9977-877545feed 2e 022b15c9e0b1445 0b55c71d23bce32 53 561ad084af1941b 5931dc317db563e 12 - default default] Task 'cell_v2_ discover_ hosts' (ff79c16c- b48e-495a- b2cc-38d9e92df0 8f) [RUNNING -> ERROR, msg=Failure caused by error in tasks: wait_for_ nova_resources
wait_ for_nova_ resources [task_ex_ id=ec65be73- 4f36-425c- 9bd4-6684456c3b 32] -> One or more actions had failed. ex_id=15a7c7e2- 558b-4548- b73b-a78d4b7be5 94, idx=0]: The action raised an exception [action_ ex_id=15a7c7e2- 558b-4548- b73b-a78d4b7be5 94, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : '005d8935- 9991-40bc- 95be-dd0eb0e383 0e'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : '005d8935- 9991-40bc- 95be-dd0eb0e383 0e'}. (HTTP 404) ex_id=266a8a2b- 4218-441d- 902c-3f7da984d9 12, idx=1]: The action raised an exception [action_ ex_id=266a8a2b- 4218-441d- 902c-3f7da984d9 12, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : '4b0e84d7- 7810-4a42- 81d5-6b14579362 c8'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : '4b0e84d7- 7810-4a42- 81d5-6b14579362 c8'}. (HTTP 404) ex_id=4f6e7bea- 55ef-4b92- 8539-76f2d2decf a4, idx=2]: The action raised an exception [action_ ex_id=4f6e7bea- 55ef-4b92- 8539-76f2d2decf a4, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : 'c586ead4- 07b0-4e3c- 983e-ad1ece3ffc 1e'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : 'c586ead4- 07b0-4e3c- 983e-ad1ece3ffc 1e'}. (HTTP 404) ex_id=64e5320d- fb64-49b0- 8cb0-eb62f04f41 70, idx=3]: The action raised an exception [action_ ex_id=64e5320d- fb64-49b0- 8cb0-eb62f04f41 70, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : '6bf2c92f- 8eaf-4596- 982c-87cef89337 b7'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : '6bf2c92f- 8eaf-4596- 982c-87cef89337 b7'}. (HTTP 404) ex_id=83e97b9c- 2775-4752- 962d-d19cc728ff 0c, idx=4]: The action raised an exception [action_ ex_id=83e97b9c- 2775-4752- 962d-d19cc728ff 0c, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : '6f539c86- 3905-4014- 8973-f515343c6c 4b'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : '6f539c86- 3905-4014- 8973-f515343c6c 4b'}. (HTTP 404) ex_id=b7881d3d- 78c7-4898- a8cd-36a8b25112 a5, idx=5]: The action raised an exception [action_ ex_id=b7881d3d- 78c7-4898- a8cd-36a8b25112 a5, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : '82cdb49e- a248-4221- 8206-62813b6deb 26'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : '82cdb49e- a248-4221- 8206-62813b6deb 26'}. (HTTP 404) ex_id=c0aca431- e692-45e0- 98ef-2f5fcf8fa6 23, idx=6]: The action raised an exception [action_ ex_id=c0aca431- e692-45e0- 98ef-2f5fcf8fa6 23, action_cls='<class 'mistral. actions. action_ factory. NovaAction' >', attributes= '{'client_ method_ name': 'hypervisors. find'}' , params= '{'hypervisor_ hostname' : 'd1e608f4- 9e95-4d85- 8d8b-ad8ee73c01 d5'}'] hypervisors. find failed: No Hypervisor matching {'hypervisor_ hostname' : 'd1e608f4- 9e95-4d85- 8d8b-ad8ee73c01 d5'}. (HTTP 404) id=e0b1ab58- 9b03-4088- a008-a92614dcd8 92)
[action_
NovaAction.
[action_
NovaAction.
[action_
NovaAction.
[action_
NovaAction.
[action_
NovaAction.
[action_
NovaAction.
[action_
NovaAction.
] (execution_
(END)