Activity log for bug #1730556

Date Who What changed Old value New value Message
2017-11-07 03:41:56 xhzhf bug added bug
2017-11-07 03:42:18 xhzhf nova: assignee xhzhf (guoyongxhzhf)
2017-11-07 03:45:57 xhzhf affects nova oslo.service
2017-11-07 03:49:07 xhzhf description Description =========== all periodic tasks of nova-compute don't work report_state works. Steps to reproduce ================== after some error of our virtulization product happened. At that time, get_available_nodes of the driver can not fulfil and throw a exception that does nto inherit from Excetipn Expected result =============== all periodic task should keep going Actual result ============= all periodic tasks stop Analysis ============= I found that the coroutine has disappear and run_periodic_tasks in periodic_task.py does not catch BaseException Solution ============= run_periodic_tasks should catch all exception. This is a major problem. All periodic_task died even those tasks which work well and don't throw exception. Because they belong to a same coroutine. We should keep nova-compute robust. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ openstack-nova-common-16.0.1-1.el7.noarch python2-novaclient-9.1.0-1.noarch python-nova-16.0.1-1.el7.noarch openstack-nova-compute-16.0.1-1.noarch 2. Which hypervisor did you use? Our own virtulization product 2. Which storage type did you use? None 3. Which networking type did you use? Neutron with our sdn Description =========== all periodic tasks of nova-compute don't work in environment of our customers report_state works. Steps to reproduce ================== after some error of our virtulization product happened. At that time, get_available_nodes of the driver can not fulfil and throw a exception that does nto inherit from Excetipn Analysis ============= I found that the coroutine has disappear and run_periodic_tasks in periodic_task.py does not catch BaseException Solution ============= run_periodic_tasks should catch all exception. This is a major problem. All periodic_task died even those tasks which work well and don't throw exception. Because they belong to a same coroutine. We should keep nova-compute robust. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    openstack-nova-common-16.0.1-1.el7.noarch    python2-novaclient-9.1.0-1.noarch    python-nova-16.0.1-1.el7.noarch    openstack-nova-compute-16.0.1-1.noarch 2. Which hypervisor did you use?    Our own virtulization product 2. Which storage type did you use?    None 3. Which networking type did you use?    Neutron with our sdn
2017-11-07 06:50:26 OpenStack Infra oslo.service: status New In Progress
2017-11-13 06:06:33 OpenStack Infra oslo.service: status In Progress Fix Released