StarlingX

Bug #1824994
Comment #3

Comment 3 for bug 1824994

Revision history for this message

Frank Miller (sensfan22) wrote on 2019-04-17: Re: 2+3 system host-lock failed after swact during lab setup

From the logs provided by Peng, nfv-vim is failing the lock due to task disable_host_services timing out:

2019-04-16T07:49:37.365 controller-1 2019-04-16T07:49:37.365 controller-1 2019-04-16T07:49:37.366 controller-1 2019-04-16T07:49:37.370 controller-1 2019-04-16T07:49:37.370 controller-1 2019-04-16T07:49:37.370 controller-1 2019-04-16T07:49:37.373 controller-1 2019-04-16T07:49:37.374 controller-1 2019-04-16T07:49:37.374 controller-1 2019-04-16T07:49:37.374 controller-1 2019-04-16T07:49:37.375 controller-1 2019-04-16T07:49:37.377 controller-1 2019-04-16T07:49:37.379 controller-1 2019-04-16T07:49:57.421 controller-1 2019-04-16T07:49:58.425 controller-1 2019-04-16T07:49:58.425 controller-1 2019-04-16T07:49:58.426 controller-1 2019-04-16T07:49:58.426 controller-1 2019-04-16T07:49:58.427 controller-1 2019-04-16T07:49:58.427 controller-1 2019-04-16T07:49:58.427 controller-1 2019-04-16T07:49:58.427 controller-1 2019-04-16T07:49:58.428 controller-1 2019-04-16T07:49:58.428 controller-1 2019-04-16T07:49:58.428 controller-1 2019-04-16T07:49:58.428 controller-1 2019-04-16T07:49:58.429 controller-1 2019-04-16T07:49:58.435 controller-1 VIM_Thread[110384] DEBUG _vim_nfvi_events.py.63 Host action, host_uuid=5766bad1-fc00-42bf-b7ee-534677538e95, host_name=controller-0, do_action=lock.
VIM_Thread[110384] INFO _host_state_enabled.py.33 Exiting state (enabled) for controller-0.
VIM_Thread[110384] INFO _host_state_disabling.py.27 Entering state (disabling) for controller-0.
VIM_Thread[110384] INFO _host_director.py.395 Notify other directors that the host controller-0 services are disabling.
VIM_Thread[110384] INFO _instance_director.py.1339 Host controller-0 services disabling.
VIM_Thread[110384] INFO _instance_director.py.495 No hypervisors available, can't evacuate instances off of host controller-0.
VIM_Thread[110384] INFO _host_director.py.408 Notify other directors that the host controller-0 services are disabled.
VIM_Thread[110384] INFO _instance_director.py.1392 Host controller-0 services disabled.
VIM_Thread[110384] DEBUG _instance_director.py.1408 Canceling previous host operation host-lock, before continuing with host operation host-lock for controller-0.
VIM_Thread[110384] INFO _instance_director.py.495 No hypervisors available, can't evacuate instances off of host controller-0.
VIM_Thread[110384] INFO _host.py.820 Host controller-0 FSM State-Change: prev_state=enabled, state=disabling, event=lock.
VIM_Thread[110384] INFO _host_director.py.475 Host controller-0 state change notification.
VIM_Thread[110384] DEBUG nfvi_infrastructure_api.py.2260 Host rest-api patch path: /nfvi-plugins/v1/hosts/5766bad1-fc00-42bf-b7ee-534677538e95.
VIM_Thread[110384] INFO _task_worker_pool.py.73 Timeout worker Infrastructure-Worker-0
VIM_Thread[110384] ERROR _task.py.200 Task(disable_host_services) work (taint_node) timed out, id=22.
VIM_Thread[110384] ERROR nfvi_infrastructure_api.py.955 Kubernetes taint_node failed, operation did not complete, host_uuid=5766bad1-fc00-42bf-b7ee-534677538e95, host_name=controller-0.
VIM_Thread[110384] INFO _host_director.py.464 Notify other directors that a host controller-0 abort is inprogress.
VIM_Thread[110384] INFO _instance_director.py.1332 Canceling host operation host-lock for host controller-0.
VIM_Thread[110384] INFO _host_director.py.464 Notify other directors that a host controller-0 abort is inprogress.
VIM_Thread[110384] INFO _host_director.py.421 Notify other directors that the host controller-0 is disabled.
VIM_Thread[110384] INFO _instance_director.py.1427 Host controller-0 disabled.
VIM_Thread[110384] DEBUG _host_tasks.py.276 Task (disable-host_controller-0) complete.
VIM_Thread[110384] INFO _host_state_disabling.py.81 Disable failed for controller-0.
VIM_Thread[110384] INFO _host_state_disabling.py.83 Lock failed for controller-0.
VIM_Thread[110384] INFO _host_state_disabling.py.36 Exiting state (disabling) for controller-0.
VIM_Thread[110384] INFO _host_state_disabling_failed.py.27 Entering state (disabling-failed) for controller-0.
VIM_Thread[110384] INFO _host.py.820 Host controller-0 FSM State-Change: prev_state=disabling, state=disabling-failed, event=task-failed.
VIM_Thread[110384] INFO _host_director.py.475 Host controller-0 state change notification.

From the logs provided by Peng, nfv-vim is failing the lock due to task disable_host_services timing out:

2019-04-16T07:49:37.365 controller-1                           VIM_Thread[110384]    DEBUG                        _vim_nfvi_events.py.63    Host action, host_uuid=5766bad1-fc00-42bf-b7ee-534677538e95, host_name=controller-0, do_action=lock.
2019-04-16T07:49:37.365 controller-1                           VIM_Thread[110384]     INFO                     _host_state_enabled.py.33    Exiting state (enabled) for controller-0.
2019-04-16T07:49:37.366 controller-1                           VIM_Thread[110384]     INFO                   _host_state_disabling.py.27    Entering state (disabling) for controller-0.
2019-04-16T07:49:37.370 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.395   Notify other directors that the host controller-0 services are disabling.
2019-04-16T07:49:37.370 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.1339  Host controller-0 services disabling.
2019-04-16T07:49:37.370 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.495   No hypervisors available, can't evacuate instances off of host controller-0.
2019-04-16T07:49:37.373 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.408   Notify other directors that the host controller-0 services are disabled.
2019-04-16T07:49:37.374 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.1392  Host controller-0 services disabled.
2019-04-16T07:49:37.374 controller-1                           VIM_Thread[110384]    DEBUG                      _instance_director.py.1408  Canceling previous host operation host-lock, before continuing with host operation host-lock for controller-0.
2019-04-16T07:49:37.374 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.495   No hypervisors available, can't evacuate instances off of host controller-0.
2019-04-16T07:49:37.375 controller-1                           VIM_Thread[110384]     INFO                                   _host.py.820   Host controller-0 FSM State-Change: prev_state=enabled, state=disabling, event=lock.
2019-04-16T07:49:37.377 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.475   Host controller-0 state change notification.
2019-04-16T07:49:37.379 controller-1                           VIM_Thread[110384]    DEBUG                 nfvi_infrastructure_api.py.2260  Host rest-api patch path: /nfvi-plugins/v1/hosts/5766bad1-fc00-42bf-b7ee-534677538e95.
2019-04-16T07:49:57.421 controller-1                           VIM_Thread[110384]     INFO                       _task_worker_pool.py.73    Timeout worker Infrastructure-Worker-0
2019-04-16T07:49:58.425 controller-1                           VIM_Thread[110384]    ERROR                                   _task.py.200   Task(disable_host_services) work (taint_node) timed out, id=22.
2019-04-16T07:49:58.425 controller-1                           VIM_Thread[110384]    ERROR                 nfvi_infrastructure_api.py.955   Kubernetes taint_node failed, operation did not complete, host_uuid=5766bad1-fc00-42bf-b7ee-534677538e95, host_name=controller-0.
2019-04-16T07:49:58.426 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.464   Notify other directors that a host controller-0 abort is inprogress.
2019-04-16T07:49:58.426 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.1332  Canceling host operation host-lock for host controller-0.
2019-04-16T07:49:58.427 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.464   Notify other directors that a host controller-0 abort is inprogress.
2019-04-16T07:49:58.427 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.421   Notify other directors that the host controller-0 is disabled.
2019-04-16T07:49:58.427 controller-1                           VIM_Thread[110384]     INFO                      _instance_director.py.1427  Host controller-0 disabled.
2019-04-16T07:49:58.427 controller-1                           VIM_Thread[110384]    DEBUG                             _host_tasks.py.276   Task (disable-host_controller-0) complete.
2019-04-16T07:49:58.428 controller-1                           VIM_Thread[110384]     INFO                   _host_state_disabling.py.81    Disable failed for controller-0.
2019-04-16T07:49:58.428 controller-1                           VIM_Thread[110384]     INFO                   _host_state_disabling.py.83    Lock failed for controller-0.
2019-04-16T07:49:58.428 controller-1                           VIM_Thread[110384]     INFO                   _host_state_disabling.py.36    Exiting state (disabling) for controller-0.
2019-04-16T07:49:58.428 controller-1                           VIM_Thread[110384]     INFO            _host_state_disabling_failed.py.27    Entering state (disabling-failed) for controller-0.
2019-04-16T07:49:58.429 controller-1                           VIM_Thread[110384]     INFO                                   _host.py.820   Host controller-0 FSM State-Change: prev_state=disabling, state=disabling-failed, event=task-failed.
2019-04-16T07:49:58.435 controller-1                           VIM_Thread[110384]     INFO                          _host_director.py.475   Host controller-0 state change notification.