So I reenabled the test cases in the lvm job, bumped the detach timeout to 60 sec from the default 20 sec and the test still fails. The nova-compute log shows the same symptom that that the first detach request to qemu never succeeds and later retries are rejected by qemu: Apr 24 16:31:47.197069 np0033831695 nova-compute[81721]: WARNING nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 is timed out. Apr 24 16:31:47.199043 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Failed to detach device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 from the live domain config. Libvirt did not report any error but the device is still in the config. {{(pid=81721) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2512}} Apr 24 16:31:47.199290 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] (2/8): Attempting to detach device vdb with device alias virtio-disk1 from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 from the live domain config. {{(pid=81721) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2494}} Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.guest [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] detach device xml: Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: 95041152-35bf-4352-9228-9acdac3aa034 Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]:
Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: Apr 24 16:31:47.199707 np0033831695 nova-compute[81721]: {{(pid=81721) detach_device /opt/stack/nova/nova/virt/libvirt/guest.py:465}} Apr 24 16:31:47.202783 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Libvirt returned error while detaching device vdb from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56. Libvirt error code: 1, error message: internal error: unable to execute QEMU command 'device_del': Device virtio-disk1 is already in the process of unplug. {{(pid=81721) _detach_sync /opt/stack/nova/nova/virt/libvirt/driver.py:2638}} Apr 24 16:31:47.203293 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Ignoring QEMU rejecting our request to detach device vdb from instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 as it is caused by a previous request still being in progress. {{(pid=81721) _detach_sync /opt/stack/nova/nova/virt/libvirt/driver.py:2675}} Apr 24 16:31:47.204055 np0033831695 nova-compute[81721]: DEBUG nova.virt.libvirt.driver [None req-49426422-4e77-407c-8207-8cee80cec58d tempest-AttachVolumeTestJSON-1677813526 tempest-AttachVolumeTestJSON-1677813526-project-member] Start waiting for the detach event from libvirt for device vdb with device alias virtio-disk1 for instance e3a5dcba-2acf-4f71-96c9-fd723d438f56 {{(pid=81721) _detach_from_live_and_wait_for_event /opt/stack/nova/nova/virt/libvirt/driver.py:2570}} I will do one more trial with a lot longer detach timeout value but I afraid this not a timeout problem but the well known qemu + guest problem that attach and detach request can be lost and cannot be retried.