Comment 5 for bug 1998148

So I reenabled the test cases in the lvm job, bumped the detach timeout to 60 sec from the default 20 sec and the test still fails. The nova-compute log shows the same symptom that that the first detach request to qemu never succeeds and later retries are rejected by qemu:

Apr 24 16:31:47.197069 np0033831695 nova-compute[81721]: WARNING nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 is timed out.
Apr 24 16:31:47.199043 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Failed to detach device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 from the live domain config. Libvirt did not report any error but the device is still in the config. {{(pid=81721) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2512}}
Apr 24 16:31:47.199290 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] (2/8): Attempting to detach device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 from the live domain config. {{(pid=81721) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2494}}
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.guest [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] detach device xml: <disk type="block" device="disk">
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: <driver name="qemu" type="raw" cache="none" io="native"/>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: <source dev="/dev/sdb"/>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: <target dev="vdb" bus="virtio"/>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: <serial>95041152-35bf-4352-9228-9acdac3aa034</serial>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: </disk>
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: {{(pid=81721) detach_device /opt/stack/nova/nova/virt/libvirt/guest.py:465}}
Apr 24 16:31:47.202783 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Libvirt returned error while detaching device vdb from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56. Libvirt error code: 1, error message: internal error: unable to execute QEMU command 'device_del': Device virtio-disk1 is already in the process of unplug. {{(pid=81721) _detach_sync /opt/stack/nova/nova/virt/libvirt/driver.py:2638}}
Apr 24 16:31:47.203293 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Ignoring QEMU rejecting our request to detach device vdb from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 as it is caused by a previous request still being in progress. {{(pid=81721) _detach_sync /opt/stack/nova/nova/virt/libvirt/driver.py:2675}}
Apr 24 16:31:47.204055 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Start waiting for the detach event from libvirt for device vdb with device alias virtio-disk1 for instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 {{(pid=81721) _detach_from_live_and_wait_for_event /opt/stack/nova/nova/virt/libvirt/driver.py:2570}}

I will do one more trial with a lot longer detach timeout value but I afraid this not a timeout problem but the well known qemu + guest problem that attach and detach request can be lost and cannot be retried.