Cinder node can not detach volume

[Alisa Tselovalnikova]

Detailed bug description:

  The issue was found by
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/134/testReport/(root)/ha_neutron_delete_vips/ha_neutron_delete_vips/

Steps to reproduce:

  1. Delete 10 time public and management VIPs
  2. Wait while it is being restored
  3. Verify it is restored
  4. Run OSTF

Expected results:

  All steps are passed.

Actual result:

  The OSTF test "Create volume and attach it to instance" is failed.

  Take a look at part of logs http://paste.openstack.org/show/589906/

Description of the environment:

 snapshot #537

[Roman Podoliaka (rpodolyaka)]

We've seen this error for some time now in different situations. My current understanding is that we call and retry https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainDetachDeviceFlags , but the call is asynchronous: libvirt API docs suggest to periodically check for the changes in a domain XML or to subscribe for a event to be dispatched by libvirtd event loop. The problem is that this never happens and we time out waiting.

libvirtd itself is waiting for DEVICE_DELETE event from qemu monitor after issuing a request to "hot unplug" a device (device_del), but looks like it may never arrive, if something is wrong with the guest. Curiously, libvirt folks think that we should not try to forcefully disconnect a device even in that case (https://www.redhat.com/archives/libvir-list/2016-March/msg00482.html). In other words, libvirtd does not try to send "drive_del" (forcefully disconnect an attached device) to a qemu process.

We'll take a closer a look at libvirtd API and see what we can do about it. Anyway, this must not be a big deal: if this happened, your instance is most likely in some weird state by this point and you need to delete/hard reboot it anyway (which would allow us to properly detach such a volume).