Cinder

(ceph) ImageNotFound: error opening image volume-cf89079d-8c06-4fbe-8e79-ee38066b2226 at snapshot None

Bug #1645851 reported by Matt Riedemann on 2016-11-29

This bug report is a duplicate of: Bug #1698786: [SRU] cinder-volume fails on start when rbd pool contains partially deleted images. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Medium	Unassigned

Bug Description

I've been seeing this a lot lately in ceph jobs:

http://logs.openstack.org/88/403988/3/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/03858ac/logs/screen-c-vol.txt.gz?level=TRACE#_2016-11-29_20_04_23_003

It appears to be a race when deleting a volume and the underlying image is gone from the backing ceph store:

2016-11-29 20:04:23.003 ERROR oslo_messaging.rpc.server [req-3e3046e9-73a0-4ee0-beab-6b44e2799d27 tempest-VolumesListAdminV2TestJSON-352565107] Exception during message handling
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 225, in dispatch
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 195, in _do_dispatch
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "<decorator-gen-238>", line 2, in delete_volume
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/coordination.py", line 286, in _synchronized
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server return f(*a, **k)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "<decorator-gen-237>", line 2, in delete_volume
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/objects/cleanable.py", line 191, in wrapper
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/manager.py", line 786, in delete_volume
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server self.publish_service_capabilities(context)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/manager.py", line 2022, in publish_service_capabilities
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server self._report_driver_status(context)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1978, in _report_driver_status
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server volume_stats = self.driver.get_volume_stats(refresh=True)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/rbd.py", line 350, in get_volume_stats
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server self._update_volume_stats()
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/rbd.py", line 336, in _update_volume_stats
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server self._get_usage_info()
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/rbd.py", line 295, in _get_usage_info
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server with RBDVolumeProxy(self, t, read_only=True) as v:
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/rbd.py", line 116, in __init__
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server read_only=read_only)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server File "rbd.pyx", line 1042, in rbd.Image.__init__ (/build/ceph-XmVvyr/ceph-10.2.2/src/build/rbd.c:9862)
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server ImageNotFound: error opening image volume-cf89079d-8c06-4fbe-8e79-ee38066b2226 at snapshot None
2016-11-29 20:04:23.003 3510 ERROR oslo_messaging.rpc.server

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ImageNotFound%3A%20error%20opening%20image%20volume-%5C%22%20AND%20message%3A%5C%22at%20snapshot%20None%5C%22%20AND%20message%3A%5C%22delete_volume%5C%22%20AND%20message%3A%5C%22_update_volume_stats%5C%22%20AND%20tags%3A%5C%22screen-c-vol.txt%5C%22&from=7d

Tags:

int32bit (int32bit) on 2016-11-30

Changed in cinder:
assignee:	nobody → int32bit (int32bit)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-30: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/404821

Changed in cinder:
assignee:	int32bit (int32bit) → Jon Bernard (jbernard)
status:	New → In Progress

Jon Bernard (jbernard) on 2016-11-30

Changed in cinder:
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-09: Change abandoned on cinder (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/404821
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.