ceph driver deletes can cause excessive exceptions

Bug #1843074 reported by Walt Boring
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Walt Boring

Bug Description

When issuing lots of delete calls in parallel for the ceph driver, it can issue lots of ImageNotFound exceptions. When cinder does a delete_volume it will ask the driver to get_volume_stats, which for the rbd driver can result in scans of every volume in the cluster.
There is a race condition where parallel deletes can cause an image in the pool to be removed after another delete call has gotten it in it's list for scans.

_get_usage_info() gets a list of all volumes to scan for their size here:
https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/drivers/rbd.py#L384-L403

The exception is Logged here:
https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/drivers/rbd.py#L144

In the case of getting volume stats, the exception is later ignored here:
https://github.com/openstack/cinder/blob/stable/pike/cinder/volume/drivers/rbd.py#L402-L403

We should add a new param to the RBDVolumeProxy to silence the logging of the ImageNotFound exception, since it's going to be debug logged and ignored for this case.

Revision history for this message
Walt Boring (walter-boring) wrote :

This was already addressed.

Changed in cinder:
status: New → Invalid
Changed in cinder:
assignee: nobody → Walt Boring (walter-boring)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/680764

Revision history for this message
Walt Boring (walter-boring) wrote :
Changed in cinder:
status: Invalid → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.opendev.org/680764
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d9b4a9af821e77057d890151003014e68ddcf7ca
Submitter: Zuul
Branch: stable/pike

commit d9b4a9af821e77057d890151003014e68ddcf7ca
Author: Jon Bernard <email address hidden>
Date: Wed May 9 14:43:56 2018 -0400

    RBD: remove redundant exception log to reduce noise

    If RBDVolumeProxy fails to open the underlying image it reraises the
    exception, so there is no need to log it unconditionally. The caller
    can gracefully handle this case without alarming the admin.

    Closes-Bug: #1843074
    Change-Id: If5a9d014dbdd63a6e4659d700671c6ac6496b96a
    (cherry picked from commit 71584d6852a6dec8d8d04c8f3cbddf0f114777af)

tags: added: in-stable-pike
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.