Mismatch happens between BDM and domain XML If instance does not respond to ACPI hotplug during detach/attach.

[git-harry]

tempest.api.compute.servers.test_server_rescue_negative:ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume

This test passes however it fails to properly cleanup after itself - the detach completes but without running the necessary iscsiadm commands.

In nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver.disconnect_volume the list returned by self.connection._get_all_block_devices includes the host_device which means that self._disconnect_from_iscsi_portal is never run.

You can see evidence of this in /etc/iscsi/nodes as well as errors logged in /var/log/syslog

I'm guessing there is a race between the unrescue and the detach within libvirt. In nova.virt.libvirt.driver.LibvirtDriver.detach_volume if I put in a sleep before virt_dom.detachDeviceFlags(xml, flags) the detach appears to work properly however if I sleep after that line it does not appear to have any effect.

[Boden R (boden)]

potential dup of https://bugs.launchpad.net/openstack-ci/+bug/1317298

I have an e-r query out for review here: https://review.openstack.org/#/c/124456/

[git-harry (git-harry)]

I don't see how this is a duplicate of #1317298 - that bug looks unrelated. This is not an issue with tempest timing out.

[git-harry (git-harry)]

I've removed the duplicate link.

[Mitsuhiro Tanino (mitsuhiro-tanino)]

I added https://bugs.launchpad.net/cinder/+bug/1440201 as a duplicate of this bug.

[Mitsuhiro Tanino (mitsuhiro-tanino)]

Further investigation there are some differences between normal volume detachment and this error case of volume detachment.

(1) libvirtd.log of Normal volume detach
http://paste.openstack.org/show/202077/

(2) libvirtd.log of this error case during volume detach
http://paste.openstack.org/show/202078/

In the log (1), I can see the following logs which mentions removing the virtio-disk. However, I couldn't see same logs in the log (2).
--- snip ---
2015-04-10 14:46:33.098+0000: 787: debug : qemuProcessHandleDeviceDeleted:1349 : Device virtio-disk1 removed from domain 0x7f7da0026b60 instance-0000002e
2015-04-10 14:46:33.098+0000: 787: debug : qemuDomainRemoveDiskDevice:2444 : Removing disk virtio-disk1 from domain 0x7f7da0026b60 instance-0000002e

Also I confirmed following two points.
(3) Login the instance via ssh after reboot to confirm device list of instance.
(4) Check virsh dumpxml result to confirm the device list of instance.

As for (3), I couldn't see the device file(/dev/vdb) for the Cinder volume. This means that the device was already detached from the viewpoint of qemu-kvm.
As for (4), the Cinder volume configuration still remained in the XML file of the instance.So libvirt kept Cinder volume information even though there was no device in the VM.

Very odd status...

[Jay Bryant (jsbryant)]

Mitsuhiro,

I have now been able to recreate this in devstack. I think I understand why I was initially having trouble recreating. On the system where we originally saw this the VM was running RHEL instead of Cirros, so the reboot was taking much longer. The window to hit the timing issue is much smaller in devstack with Cirros. Anyway, now that I can recreate I plan to spend some time working on this next week.

Thanks for your continued assistance on this as well!
Jay

[Tomoki Sekiyama (tsekiyama)]

This issue ( self._disconnect_from_iscsi_portal is not called ) also happens when the guest kernel is crashed.

(1) In the guest, crash the kernel.
  echo 1 > /proc/sys/kernel/sysrq
  echo c > /proc/sysrq-trigger

(2) Rnu "nova volume-detach <server> <volume-id>"

This might be because guest doesn't prospectively respond to ACPI hotplug event.
While (re)-booting the guest OS, there also is the window the guest doesn't respond to the event.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175067

[Ryan McNair (rdmcnair)]

Able to reproduce this issue by issuing soft reboot in Horizon and immediately detaching a volume using "nova detach". On the Cinder side, the volume gets detached from the instance and is no longer "in-use".

However, the VM still has the mapping to the device on "/dev/vdb" and trying to write to the device results in a "write error: No space on disk". Also, libvirt still has this device mapping in "libvirt.driver._get_all_block_devices()".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227851

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mitsuhiro Tanino (<email address hidden>) on branch: master
Review: https://review.openstack.org/175067
Reason: https://review.openstack.org/227851 covers this problem.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227851
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3a3fb3cfb2c41ad182545e47649ff12a4f3a743e
Submitter: Jenkins
Branch: master

commit 3a3fb3cfb2c41ad182545e47649ff12a4f3a743e
Author: Ryan McNair <email address hidden>
Date: Thu Sep 24 22:20:23 2015 +0000

    Add retry logic for detaching device using LibVirt

    Add retry logic for removing a disk device from the LibVirt
    guest domain XML. This is needed because immediately after a guest
    reboot, libvirtmod.virDomainDetachDeviceFlags() will silently fail
    to remove the mapping from the guest domain. The async retry
    behavior is done in Guest and is generic so it can be re-used by any other
    detaches which hit this same race condition.

    Change-Id: I983f80822a5c210929f33e1aa348a0fef91e890b
    Closes-Bug: #1374508

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/259685

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/260127

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/260127
Reason: Not appropriate for stable/kilo.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/260127
Reason: This doesn't help on https://review.openstack.org/#/c/218355/ so dropping it again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/259685
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9701921f1e921885d7c65bfde9b3cc384389fe74
Submitter: Jenkins
Branch: stable/liberty

commit 9701921f1e921885d7c65bfde9b3cc384389fe74
Author: Ryan McNair <email address hidden>
Date: Thu Sep 24 22:20:23 2015 +0000

    Add retry logic for detaching device using LibVirt

    Add retry logic for removing a disk device from the LibVirt
    guest domain XML. This is needed because immediately after a guest
    reboot, libvirtmod.virDomainDetachDeviceFlags() will silently fail
    to remove the mapping from the guest domain. The async retry
    behavior is done in Guest and is generic so it can be re-used by any other
    detaches which hit this same race condition.

    Change-Id: I983f80822a5c210929f33e1aa348a0fef91e890b
    Closes-Bug: #1374508
    (cherry picked from commit 3a3fb3cfb2c41ad182545e47649ff12a4f3a743e)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.2 release.