Unable to force-delete if volume group is missing

[Joe D'Andrea]

When the cinder-volumes LVM volume group can't be located (e.g., due to a disk array failure), force-delete does not succeed.

Per IRC chat with jgriffith, this appears to be a regression WRT "require init" and that this used to succeed. (Perhaps recheck unit tests as well?)

See http://paste.openstack.org/show/147910/ for full context.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 144, in lvo_inner1
    return lvo_inner2(inst, context, volume_id, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/lockutils.py", line 233, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 143, in lvo_inner2
    return f(*_args, **_kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 416, in delete_volume
    {'status': 'error_deleting'})
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 405, in delete_volume
    self.driver.delete_volume(volume_ref)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 224, in delete_volume
    if self._volume_not_present(volume['name']):
  File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 122, in _volume_not_present
    return self.vg.get_volume(volume_name) is None
  File "/usr/lib/python2.7/dist-packages/cinder/brick/local_dev/lvm.py", line 278, in get_volume
    ref_list = self.get_volumes()
  File "/usr/lib/python2.7/dist-packages/cinder/brick/local_dev/lvm.py", line 269, in get_volumes
    self.lv_list = self.get_all_volumes(self._root_helper, self.vg_name)
  File "/usr/lib/python2.7/dist-packages/cinder/brick/local_dev/lvm.py", line 253, in get_all_volumes
    run_as_root=True)
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py", line 179, in execute
    cmd=sanitized_cmd)
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix cinder-volumes
Exit code: 5
Stdout: u''
Stderr: u' Volume group "cinder-volumes" not found\n Skipping volume group cinder-volumes\n'

[Eric Harney (eharney)]

I think this is arguably reasonable behavior. All force-delete does is lets you delete things that are in states other than "available", "error", etc.

Allowing force-delete to proceed and delete the volume if it can't find the VG means that you leak LVs when the VG is brought back online. This is different semantics of what force-delete is currently intended to do, and probably falls more under the "unmanage" area rather than "delete".

[Joe D'Andrea (jdandrea)]

That's a really good point. Semantic integrity matters. You can't delete something that isn't there to begin with.

Is there an "unmanage" or equivalent action that I'm not aware of? Is it in the works? Meanwhile, is there anything I can do in the present moment?

[Eric Harney (eharney)]

unmanage exists in the v2 API now. However it will currently fail in this case in the same way, because it requires the driver to be initialized. (Which it won't be if the VG is missing when cinder-volume is started.)

I was mostly noting that if we wanted to pursue being able to do something more useful in this case, that would be the place to start rather than in delete.

There isn't a useful Cinder action to handle this at the moment outside of restoring the VG so that the delete succeeds.

[Joe D'Andrea (jdandrea)]

Got it, and agreed: Start with unmanage. (Understood WRT next steps.)

[Duncan Thomas (duncan-thomas)]

I can see a strong argument that the semantics of 'force delete' are/should
be 'make me stop paying for this volume', so the correct thing to do if the
backend is not initialised might be to put the volume in error and let the
admin clean up later.

[Joe D'Andrea (jdandrea)]

Duncan: I should mention that, in the scenario I describe, volumes *do* end up in "error_deleting" status.

[Duncan Thomas (duncan-thomas)]

Thanks for that detail Joe. In which case I think that behaviour is fine.

Duncan Thomas

[Joe D'Andrea (jdandrea)]

You're welcome!

Q: Ok to rename this to align with 'unmanage' instead, or file a new one?

[Sean McGinnis (sean-mcginnis)]

Anyone know if this is still an issue?

[Brian Rosmaita (brian-rosmaita)]

Discussion indicates that this is no longer considered a bug.