Cinder tries to delete a volume that has been deleted by Cinder on another node

[Yuriy Nesenenko]

cinder-volume on the node (427) has deleted volume-3a02d9ff-...

Jan 12 03:16:59 node-427 cinder-volume: 2016-01-12 03:16:59.187 32799 DEBUG cinder.volume.drivers.rbd [req-7fba16c9-995f-42d3-8a61-640d9f775807 6bb3862fc2d048e6b39b88e6b35361cc 3aadc6ebfa1a4c6
6aecdad76c70c7941 - - -] Volume volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd is not a clone. _get_clone_info /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:619
Jan 12 03:16:59 node-427 cinder-volume: 2016-01-12 03:16:59.201 32799 DEBUG cinder.volume.drivers.rbd [req-7fba16c9-995f-42d3-8a61-640d9f775807 6bb3862fc2d048e6b39b88e6b35361cc 3aadc6ebfa1a4c6
6aecdad76c70c7941 - - -] deleting rbd volume volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd delete_volume /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:715
Running cmd (subprocess): rbd import --pool volumes --order 22 /var/lib/cinder/False/tmpzROmkR volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd --new-format --id volumes --conf /etc/ceph/ceph.conf --cluster ceph execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:230

But on another node (588) cinder-volume also tries to delete the same volume volume-3a02d9ff-... and got:

/var/log/cinder-all.log.5.gz:37245-<159>Jan 12 03:48:20 node-588 cinder-volume: 2016-01-12 03:48:20.204 24797 DEBUG cinder.volume.drivers.rbd [req-ffb51a9f-ac32-4a15-8301-2e49d3687710 b99f9b41ae4f4f9ba7a4a457ae0f896c 3aadc6ebfa1a4c66aecdad76c70c7941 - - -]
Volume volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd is not a clone. _get_clone_info /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:619
/var/log/cinder-all.log.5.gz:37246-<159>Jan 12 03:48:20 node-588 cinder-volume: 2016-01-12 03:48:20.218 24797 DEBUG cinder.volume.drivers.rbd [req-ffb51a9f-ac32-4a15-8301-2e49d3687710 b99f9b41ae4f4f9ba7a4a457ae0f896c 3aadc6ebfa1a4c6
6aecdad76c70c7941 - - -] deleting rbd volume volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd delete_volume /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:715
/var/log/cinder-all.log.5.gz:37247-<8>Jan 12 03:48:20 node-588 cinder-volume: 2016-01-12 03:48:20.260067 7f45eeaae740 -1 librbd::ImageCtx: error finding header: (2) No such file or directory

[Roman Podoliaka (rpodolyaka)]

Hmm, request ids are different:

req-7fba16c9-995f-42d3-8a61-640d9f775807

and

req-ffb51a9f-ac32-4a15-8301-2e49d3687710

so either Cinder does not preserve the request context for RPC calls. Or these are two different API calls.

Could you provide more logs, Yuriy?

[Roman Podoliaka (rpodolyaka)]

Yuriy, have you made any progress on this?

[Yuriy Nesenenko (ynesenenko)]

Hi, Roman

It's not clear of the logs why the request is made to remove the same
volume half
an hour.

On Fri, Jan 29, 2016 at 3:23 PM, Roman Podoliaka <
<email address hidden>> wrote:

> Yuriy, have you made any progress on this?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1535816
>
> Title:
> Cinder tries to delete a volume that has been deleted by Cinder on
> another node
>
> Status in Mirantis OpenStack:
> Incomplete
>
> Bug description:
> cinder-volume on the node (427) has deleted volume-3a02d9ff-...
>
> Jan 12 03:16:59 node-427 cinder-volume: 2016-01-12 03:16:59.187 32799
> DEBUG cinder.volume.drivers.rbd [req-7fba16c9-995f-42d3-8a61-640d9f775807
> 6bb3862fc2d048e6b39b88e6b35361cc 3aadc6ebfa1a4c6
> 6aecdad76c70c7941 - - -] Volume
> volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd is not a clone. _get_clone_info
> /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:619
> Jan 12 03:16:59 node-427 cinder-volume: 2016-01-12 03:16:59.201 32799
> DEBUG cinder.volume.drivers.rbd [req-7fba16c9-995f-42d3-8a61-640d9f775807
> 6bb3862fc2d048e6b39b88e6b35361cc 3aadc6ebfa1a4c6
> 6aecdad76c70c7941 - - -] deleting rbd volume
> volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd delete_volume
> /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:715
> Running cmd (subprocess): rbd import --pool volumes --order 22
> /var/lib/cinder/False/tmpzROmkR volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd
> --new-format --id volumes --conf /etc/ceph/ceph.conf --cluster ceph execute
> /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:230
>
> But on another node (588) cinder-volume also tries to delete the same
> volume volume-3a02d9ff-... and got:
>
> /var/log/cinder-all.log.5.gz:37245-<159>Jan 12 03:48:20 node-588
> cinder-volume: 2016-01-12 03:48:20.204 24797 DEBUG
> cinder.volume.drivers.rbd [req-ffb51a9f-ac32-4a15-8301-2e49d3687710
> b99f9b41ae4f4f9ba7a4a457ae0f896c 3aadc6ebfa1a4c66aecdad76c70c7941 - - -]
> Volume volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd is not a clone.
> _get_clone_info
> /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:619
> /var/log/cinder-all.log.5.gz:37246-<159>Jan 12 03:48:20 node-588
> cinder-volume: 2016-01-12 03:48:20.218 24797 DEBUG
> cinder.volume.drivers.rbd [req-ffb51a9f-ac32-4a15-8301-2e49d3687710
> b99f9b41ae4f4f9ba7a4a457ae0f896c 3aadc6ebfa1a4c6
> 6aecdad76c70c7941 - - -] deleting rbd volume
> volume-3a02d9ff-c51f-4bb6-948f-ce1dfdbd22fd delete_volume
> /usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py:715
> /var/log/cinder-all.log.5.gz:37247-<8>Jan 12 03:48:20 node-588
> cinder-volume: 2016-01-12 03:48:20.260067 7f45eeaae740 -1 librbd::ImageCtx:
> error finding header: (2) No such file or directory
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1535816/+subscriptions
>

[Timur Nurlygayanov (tnurlygayanov)]

Hi Yuriy,

could you please describe detailed Steps To Reproduce and attach a diagnostic snapshot to the description of the issue?
It is not clear how to reproduce the issue and what is the root of the issue, so, status Incomplete means that Dev team needs more information.

The status will be changed to Invalid if we don't have additional information.

[Roman Podoliaka (rpodolyaka)]

No longer fixing Medium bugs in 8.0

[Roman Podoliaka (rpodolyaka)]

No feedback in a month, closing as Invalid. Please re-open the bug, if you think it's still valid.