In my case, we had a chain of patches from https://review.openstack.org/#/q/topic:bug/1686116 backported to ocata downstream. Then, when detaching a ceph volume from a node, the following happens:
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] Traceback (most recent call last): nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4835, in _driver_detach_volume nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] encryption=encryption) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1393, in detach_volume nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] live=live) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 413, in detach_device_with_retry nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] _try_detach_device(conf, persistent, live) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 407, in _try_detach_device nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] raise exception.DeviceNotFound(device=device) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in _exit_ nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.force_reraise() nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(self.type_, self.value, self.tb) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 392, in _try_detach_device nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.detach_device(conf, persistent=persistent, live=live) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 449, in detach_device nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self._domain.detachDeviceFlags(device_xml, flags=flags) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] result = proxy_call(self._autowrap, f, *args, **kwargs) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = execute(f, *args, **kwargs) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(c, e, tb) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = meth(*args, **kwargs) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1190, in detachDeviceFlags nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of domain, bus, or slot must be > 0 nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
Here is the instance xml bit for the volume:
<disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='volumes'> <secret type='ceph' uuid='ce6d1549-4d63-476b-afb6-88f0b196414f'/> </auth> <source protocol='rbd' name='volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc'> <host name='192.168.90.22' port='6789'/> </source> <backingStore/> <target dev='vdb' bus='virtio'/> <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk>
After inserting some debug logs, the following xml gets passed to the libvirt:
<disk type="network" device="disk"> <driver name="qemu" type="raw" cache="none"/> <source protocol="rbd" name="volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc"> <host name="192.168.90.22" port="6789"/> </source> <target bus="virtio" dev="vdb"/> <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial> <address type="pci"/> </disk>
After introducing a proper format_dom method for LibvirtConfigGuestDeviceAddressPCI, things seem to be back to normal.
In my case, we had a chain of patches from https:/ /review. openstack. org/#/q/ topic:bug/ 1686116 backported to ocata downstream. Then, when detaching a ceph volume from a node, the following happens:
nova/nova- compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] Traceback (most recent call last): compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 4835, in _driver_ detach_ volume compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] encryption= encryption) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ driver. py", line 1393, in detach_volume compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] live=live) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ guest.py" , line 413, in detach_ device_ with_retry compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] _try_detach_ device( conf, persistent, live) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ guest.py" , line 407, in _try_detach_device compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] raise exception. DeviceNotFound( device= device) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in _exit_ compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] self.force_ reraise( ) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] six.reraise( self.type_ , self.value, self.tb) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ guest.py" , line 392, in _try_detach_device compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] self.detach_ device( conf, persistent= persistent, live=live) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ guest.py" , line 449, in detach_device compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] self._domain. detachDeviceFla gs(device_ xml, flags=flags) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ eventlet/ tpool.py" , line 186, in doit compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] result = proxy_call( self._autowrap, f, *args, **kwargs) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ eventlet/ tpool.py" , line 144, in proxy_call compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] rv = execute(f, *args, **kwargs) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ eventlet/ tpool.py" , line 125, in execute compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] six.reraise(c, e, tb) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ eventlet/ tpool.py" , line 83, in tworker compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] rv = meth(*args, **kwargs) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] File "/usr/lib/ python2. 7/dist- packages/ libvirt. py", line 1190, in detachDeviceFlags compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] if ret == -1: raise libvirtError ('virDomainDeta chDeviceFlags( ) failed', dom=self) compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04] libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of domain, bus, or slot must be > 0 compute. log.1:2017- 07-31 00:21:24.261 341396 ERROR nova.compute. manager [instance: 43304a1b- bfcf-4e78- a9a0-eec1c6eff6 04]
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
nova/nova-
Here is the instance xml bit for the volume:
<disk type='network' device='disk'> 4d63-476b- afb6-88f0b19641 4f'/> volume- 49bd8d30- 96cb-455c- 9b73-8dc67bf0e7 fc'> 168.90. 22' port='6789'/> backingStore/ > serial> 49bd8d30- 96cb-455c- 9b73-8dc67bf0e7 fc</serial> disk1'/ >
<driver name='qemu' type='raw' cache='none'/>
<auth username='volumes'>
<secret type='ceph' uuid='ce6d1549-
</auth>
<source protocol='rbd' name='volumes/
<host name='192.
</source>
<
<target dev='vdb' bus='virtio'/>
<
<alias name='virtio-
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
After inserting some debug logs, the following xml gets passed to the libvirt:
<disk type="network" device="disk"> volume- 49bd8d30- 96cb-455c- 9b73-8dc67bf0e7 fc"> 168.90. 22" port="6789"/> 49bd8d30- 96cb-455c- 9b73-8dc67bf0e7 fc</serial>
<driver name="qemu" type="raw" cache="none"/>
<source protocol="rbd" name="volumes/
<host name="192.
</source>
<target bus="virtio" dev="vdb"/>
<serial>
<address type="pci"/>
</disk>
After introducing a proper format_dom method for LibvirtConfigGu estDeviceAddres sPCI, things seem to be back to normal.