OpenStack Compute (nova)

LibvirtConfigGuestDeviceAddressPCI missing format_dom method

  1. Series ocata
  2. Bug #1709319
Bug #1709319 reported by Vladyslav Drok
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vladyslav Drok
Ocata
Fix Committed
Medium
Vladyslav Drok

Bug Description

In my case, we had a chain of patches from https://review.openstack.org/#/q/topic:bug/1686116 backported to ocata downstream. Then, when detaching a ceph volume from a node, the following happens:

nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] Traceback (most recent call last):
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4835, in _driver_detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] encryption=encryption)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1393, in detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 413, in detach_device_with_retry
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] _try_detach_device(conf, persistent, live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 407, in _try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] raise exception.DeviceNotFound(device=device)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in _exit_
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.force_reraise()
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(self.type_, self.value, self.tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 392, in _try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self.detach_device(conf, persistent=persistent, live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 449, in detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] self._domain.detachDeviceFlags(device_xml, flags=flags)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] result = proxy_call(self._autowrap, f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = execute(f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] six.reraise(c, e, tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv = meth(*args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1190, in detachDeviceFlags
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of domain, bus, or slot must be > 0
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]

Here is the instance xml bit for the volume:

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='volumes'>
        <secret type='ceph' uuid='ce6d1549-4d63-476b-afb6-88f0b196414f'/>
      </auth>
      <source protocol='rbd' name='volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc'>
        <host name='192.168.90.22' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

After inserting some debug logs, the following xml gets passed to the libvirt's virDomainDetachDeviceFlags:

    <disk type="network" device="disk">
      <driver name="qemu" type="raw" cache="none"/>
      <source protocol="rbd" name="volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc">
        <host name="192.168.90.22" port="6789"/>
      </source>
      <target bus="virtio" dev="vdb"/>
      <serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
      <address type="pci"/>
    </disk>

After introducing a proper format_dom method for LibvirtConfigGuestDeviceAddressPCI, things seem to be back to normal.

See original description
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/491822

Changed in nova:
assignee: nobody → Vladyslav Drok (vdrok)
status: New → In Progress
Vladyslav Drok (vdrok)
description: updated
description: updated
Jay Pipes (jaypipes)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/491822
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=376a902cabd548f1bcc7f7f7f8f98bd2dfa87c89
Submitter: Jenkins
Branch: master

commit 376a902cabd548f1bcc7f7f7f8f98bd2dfa87c89
Author: Vladyslav Drok <email address hidden>
Date: Tue Aug 8 17:18:37 2017 +0300

    Add format_dom for PCI device addresses

    In case of having a PCI device, its address can not be output
    properly in the instance XML because of the missing format_dom
    method. This change adds this method.

    Closes-Bug: #1709319
    Change-Id: I1a8023ee6e8c85eed1c7c55a21f996371a0dd80a

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0rc1

This issue was fixed in the openstack/nova 16.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/495814

Vladyslav Drok (vdrok)
no longer affects: nova (Ubuntu)
Matt Riedemann (mriedem)
no longer affects: nova/pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/495814
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bcf110fe7ca2d989a956303313f413614e7a6548
Submitter: Jenkins
Branch: stable/ocata

commit bcf110fe7ca2d989a956303313f413614e7a6548
Author: Vladyslav Drok <email address hidden>
Date: Tue Aug 8 17:18:37 2017 +0300

    Add format_dom for PCI device addresses

    In case of having a PCI device, its address can not be output
    properly in the instance XML because of the missing format_dom
    method. This change adds this method.

    Closes-Bug: #1709319
    Change-Id: I1a8023ee6e8c85eed1c7c55a21f996371a0dd80a
    (cherry picked from commit 376a902cabd548f1bcc7f7f7f8f98bd2dfa87c89)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.8

This issue was fixed in the openstack/nova 15.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.