detaching a direct-physical port fails with "AttributeError: 'LibvirtConfigGuestHostdevPCI' object has no attribute 'alias'"

Bug #1942345 reported by Balazs Gibizer
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Balazs Gibizer

Bug Description

Since the refactor of the device detach logic in nova libvirt virt driver[1] detaching a port that is backed by a PF hostdev is not possible.

Probably [1] missed extending the LibvirtConfigGuestHostdevPCI class with an alias field.

[1] https://review.opendev.org/q/topic:bug/1882521

Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server [None req-08fdb8e4-94e6-4cdd-98a4-736715cbc2b1 admin admin] Exception during message handling: AttributeError: 'LibvirtConfigGuestHostdevPCI' object has no attribute 'alias'
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self.force_reraise()
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server raise self.value
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1433, in decorated_function
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 211, in decorated_function
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self.force_reraise()
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server raise self.value
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 7893, in detach_interface
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server do_detach_interface(context, instance, port_id)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 7891, in do_detach_interface
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self._detach_interface(context, instance, port_id)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 7934, in _detach_interface
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self.driver.detach_interface(context, instance, condemned)
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2793, in detach_interface
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self._detach_with_retry(
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2334, in _detach_with_retry
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server self._detach_from_live_with_retry(
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2387, in _detach_from_live_with_retry
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server attempt + 1, max_attempts, device_name, live_dev.alias,
Sep 01 14:26:40 master0 nova-compute[3382639]: ERROR oslo_messaging.rpc.server AttributeError: 'LibvirtConfigGuestHostdevPCI' object has no attribute 'alias'

Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
tags: added: sriov
tags: added: compute
tags: added: xena-rc-potential
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/806943

Changed in nova:
status: New → In Progress
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I marked this as xena-rc-potential as we landed the detach logic change in early Xena regressing SRIOV PF detach

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/806943
Committed: https://opendev.org/openstack/nova/commit/b67b928c381511f6101778ab50f281ecd689d283
Submitter: "Zuul (22348)"
Branch: master

commit b67b928c381511f6101778ab50f281ecd689d283
Author: Balazs Gibizer <email address hidden>
Date: Wed Sep 1 17:32:56 2021 +0200

    Parse alias from domain hostdev

    In I16e7df6932bb7dff243706ee49338ba6b3782085 we missed that
    LibvirtConfigGuestHostdevPCI is not a child class of
    LibvirtConfigGuestInterface and therefore we missed parsing out the
    alias field from the domain xml for hostdevs. The new libvirt driver
    device detach logic[1] uses the alias as the identifier towards libvirt
    so now hostdevs cannot be detached. This patch parses out the alias
    field to fix the issue.

    Closes-Bug: #1942345
    Related-Bug: #1882521

    [1] https://review.opendev.org/q/topic:bug/1882521

    Change-Id: I30d30a772475cb82d0fd088f14a54a35646bd1dc

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/807767

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/807767
Committed: https://opendev.org/openstack/nova/commit/0cd4ef2dcd482e6ac2f86a345e0874970cce0bdd
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0cd4ef2dcd482e6ac2f86a345e0874970cce0bdd
Author: Balazs Gibizer <email address hidden>
Date: Wed Sep 1 17:32:56 2021 +0200

    Parse alias from domain hostdev

    In I16e7df6932bb7dff243706ee49338ba6b3782085 we missed that
    LibvirtConfigGuestHostdevPCI is not a child class of
    LibvirtConfigGuestInterface and therefore we missed parsing out the
    alias field from the domain xml for hostdevs. The new libvirt driver
    device detach logic[1] uses the alias as the identifier towards libvirt
    so now hostdevs cannot be detached. This patch parses out the alias
    field to fix the issue.

    Closes-Bug: #1942345
    Related-Bug: #1882521

    [1] https://review.opendev.org/q/topic:bug/1882521

    Change-Id: I30d30a772475cb82d0fd088f14a54a35646bd1dc
    (cherry picked from commit b67b928c381511f6101778ab50f281ecd689d283)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/816486

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/victoria)

Change abandoned by "Balazs Gibizer <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/816486
Reason: issue is not in stable/victoria

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Balazs Gibizer <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/816486

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.2.0

This issue was fixed in the openstack/nova 23.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.