all periodic_task died due to uncaught exceptions

Bug #1730556 reported by xhzhf
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
Fix Released
Undecided
xhzhf

Bug Description

Description
===========
all periodic tasks of nova-compute don't work in environment of our customers
report_state works.

Steps to reproduce
==================
after some error of our virtulization product happened.
At that time, get_available_nodes of the driver can not fulfil and throw a exception that does nto inherit from Excetipn

Analysis
=============
I found that the coroutine has disappear and run_periodic_tasks in periodic_task.py does not catch BaseException

Solution
=============
run_periodic_tasks should catch all exception.
This is a major problem. All periodic_task died even those tasks which work well and don't throw exception.
Because they belong to a same coroutine.
We should keep nova-compute robust.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/
   openstack-nova-common-16.0.1-1.el7.noarch
   python2-novaclient-9.1.0-1.noarch
   python-nova-16.0.1-1.el7.noarch
   openstack-nova-compute-16.0.1-1.noarch

2. Which hypervisor did you use?
   Our own virtulization product

2. Which storage type did you use?
   None

3. Which networking type did you use?
   Neutron with our sdn

xhzhf (guoyongxhzhf)
Changed in nova:
assignee: nobody → xhzhf (guoyongxhzhf)
affects: nova → oslo.service
xhzhf (guoyongxhzhf)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.service (master)

Fix proposed to branch: master
Review: https://review.openstack.org/518249

Changed in oslo.service:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.service (master)

Reviewed: https://review.openstack.org/518249
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=b96538fcb46ba9b5656ed07b5b75874d3125eb9e
Submitter: Zuul
Branch: master

commit b96538fcb46ba9b5656ed07b5b75874d3125eb9e
Author: xhzhf <guoyongxhzhf@163.com>
Date: Tue Nov 7 11:56:04 2017 +0800

    change periodic_task to catch all exceptions including BaseException

    the function of periodic_task does not catch all exception so that
    all periodic task will die if one periodic task throw BaseException
    Closes-Bug: #1730556

    Change-Id: I2819cd3644aacac86bc6d12c980d8902f1a71823

Changed in oslo.service:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.service 1.27.0

This issue was fixed in the openstack/oslo.service 1.27.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.