Need to rollback persistent domain xml after DeviceDetachFailed exception

[Fan Zhang]

Description
===========
Volume status rollbacks to "in-use" after a failure to detach a volume due to DeviceDetachFailed, which is fixed in https://bugs.launchpad.net/nova/+bug/1786318.
But the next detaching action will fail because of DeviceNotFound exception.

Steps to reproduce
==================
Attempt to detach a volume while it is mounted and in-use within an instance.
Wait volume status rollbacks to "in-use".
Attempt to detach the volume again.

Expected result
===============
Volume detach succeeds.

Actual result
=============
Volume detach fails and the volume returns to in-use after the second detach.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   Queens-17.0.3

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Logs of second detach after DeviceDetachFailed

2019-10-30 13:17:29.401 2109249 INFO nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Attempting to driver detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from mountpoint /dev/vdb
2019-10-30 13:17:29.407 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] Attempting initial detach for device vdb detach_device_with_retry /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:435
2019-10-30 13:17:29.408 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] detach device xml: <disk type="network" device="disk">
  <driver name="qemu" type="raw" cache="writethrough" discard="unmap"/>
  <source protocol="rbd" name="volumes/volume-355d2619-cd59-4c97-80a9-b65ff1870961">
    <host name="10.114.198.109" port="6789"/>
    <host name="10.114.198.110" port="6789"/>
    <host name="10.114.198.111" port="6789"/>
    <host name="10.114.198.116" port="6789"/>
    <host name="10.114.198.117" port="6789"/>
  </source>
  <target bus="virtio" dev="vdb"/>
  <serial>355d2619-cd59-4c97-80a9-b65ff1870961</serial>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
</disk>
 detach_device /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:486
2019-10-30 13:17:29.417 2109249 ERROR nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Failed to detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from /dev/vdb: libvirtError: device not found: no target device vdb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692370