all periodic_task died due to uncaught exceptions

Bug #1730556 reported by xhzhf on 2017-11-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
Undecided
xhzhf

Bug Description

Description
===========
all periodic tasks of nova-compute don't work in environment of our customers
report_state works.

Steps to reproduce
==================
after some error of our virtulization product happened.
At that time, get_available_nodes of the driver can not fulfil and throw a exception that does nto inherit from Excetipn

Analysis
=============
I found that the coroutine has disappear and run_periodic_tasks in periodic_task.py does not catch BaseException

Solution
=============
run_periodic_tasks should catch all exception.
This is a major problem. All periodic_task died even those tasks which work well and don't throw exception.
Because they belong to a same coroutine.
We should keep nova-compute robust.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/
   openstack-nova-common-16.0.1-1.el7.noarch
   python2-novaclient-9.1.0-1.noarch
   python-nova-16.0.1-1.el7.noarch
   openstack-nova-compute-16.0.1-1.noarch

2. Which hypervisor did you use?
   Our own virtulization product

2. Which storage type did you use?
   None

3. Which networking type did you use?
   Neutron with our sdn

xhzhf (guoyongxhzhf) on 2017-11-07
Changed in nova:
assignee: nobody → xhzhf (guoyongxhzhf)
affects: nova → oslo.service
xhzhf (guoyongxhzhf) on 2017-11-07
description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/518249

Changed in oslo.service:
status: New → In Progress

Reviewed: https://review.openstack.org/518249
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=b96538fcb46ba9b5656ed07b5b75874d3125eb9e
Submitter: Zuul
Branch: master

commit b96538fcb46ba9b5656ed07b5b75874d3125eb9e
Author: xhzhf <guoyongxhzhf@163.com>
Date: Tue Nov 7 11:56:04 2017 +0800

    change periodic_task to catch all exceptions including BaseException

    the function of periodic_task does not catch all exception so that
    all periodic task will die if one periodic task throw BaseException
    Closes-Bug: #1730556

    Change-Id: I2819cd3644aacac86bc6d12c980d8902f1a71823

Changed in oslo.service:
status: In Progress → Fix Released

This issue was fixed in the openstack/oslo.service 1.27.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers