Interface detach fails for VFHostUser vif

Bug #1807340 reported by Maria Luisa Arches
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Maria Luisa Arches
Queens
Medium
Maria Luisa Arches
Rocky
Medium
Matt Riedemann

Bug Description

Description
===========
Nova skips removing VHostUser from libvirt, thinking that it
already deleted the interface

Error happened here:
https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/virt/libvirt/driver.py#L1709

Interface was not present because target_dev does not match between libvirt and cfg
https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/virt/libvirt/guest.py#L247

target_dev did not match, because nova does not fill value for target_dev
https://github.com/openstack/nova/blob/5bf6f6304e185c4af2c1b19bc11631793acdfe02/nova/virt/libvirt/vif.py#L433

Steps to reproduce
==================
1. Launch VM with 2 vnics with type vhostuser
$ openstack port list --server test-vm
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 7dc38cc5-7c04-4018-96ff-8975b357771e | | fa:16:3e:25:a0:c4 | ip_address='10.10.12.10', subnet_id='417b8d8b-cb0f-4764-96ca-e716006ee8d1' | ACTIVE |
| ac02bb45-c913-4fed-aa9b-8448106a91d3 | | fa:16:3e:54:de:9b | ip_address='10.10.13.9', subnet_id='d64b8217-22a4-475a-b323-bd03e39cc65e' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
2. Detach one port from the VM:
$ nova detach-interface test-vm ac02bb45-c913-4fed-aa9b-8448106a91d3
$ openstack port list --server test-vm
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 7dc38cc5-7c04-4018-96ff-8975b357771e | | fa:16:3e:25:a0:c4 | ip_address='10.10.12.10', subnet_id='417b8d8b-cb0f-4764-96ca-e716006ee8d1' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
3. Check VM interface. Interface is still visible in the VM
[root@test-vm ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc fq_codel state DOWN group default qlen 1000
    link/ether fa:16:3e:54:de:9b brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:25:a0:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.10.12.10/24 brd 10.10.12.255 scope global dynamic ens4
       valid_lft 85986sec preferred_lft 85986sec
    inet6 fe80::e7df:16b0:a13d:df37/64 scope link
       valid_lft forever preferred_lft forever

Expected result
===============
Interface ens3 should be removed

Environment
===========
Libvirt+KVM
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0
$ virsh --version
3.9.0
Neutron+OVS DPDK
Openstack Queens openstack-nova-compute-17.0.7-1

summary: - Interface detach fails for VFHosUser vif
+ Interface detach fails for VFHostUser vif
tags: added: dpdk libvirt
Revision history for this message
sean mooney (sean-k-mooney) wrote :

ill try an reproduce this tommorow.
if you want to address this yourself feel free to reassign.
otherwise ill try and resolve this in the next week or two.

Changed in nova:
assignee: nobody → sean mooney (sean-k-mooney)
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Maria Luisa Arches (arches) wrote :

Hi,

Thanks.

Additional info, it seems that when interface was attached runtime, target dev is not present in libvirt. In this case, detach was successful.

First interface is from vm creation. Second one, was attached run-time.
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/vhu5b00f2b1-02' mode='server'/>
      <target dev='vhu5b00f2b1-02'/>
      <model type='virtio'/>
      <alias name='net0'/>
    </interface>
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/vhu70a4f277-36' mode='server'/>
      <model type='virtio'/>
      <alias name='net1'/>
    </interface>

Possible solution that I can think of:
1. Skip checking of target dev for vhostuser
2. Update vhostuser to check if target dev is empty, if not, assign a target dev. I need to investigate this still how this can be done.

I'm thinking robust solution is #2, but still need to figure out how to do this.

Revision history for this message
Maria Luisa Arches (arches) wrote :

Hi Sean,

I will re-assign this to myself. I will upload proposed fix in review.

Basically fix was to just set target_dev to vif_name in _set_config_vifvhostuser

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/627537

Changed in nova:
assignee: sean mooney (sean-k-mooney) → Maria Luisa Arches (arches)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Maria Luisa Arches (<email address hidden>) on branch: master
Review: https://review.openstack.org/627537
Reason: rebranch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/627540

Changed in nova:
assignee: Maria Luisa Arches (arches) → sean mooney (sean-k-mooney)
Changed in nova:
assignee: sean mooney (sean-k-mooney) → Maria Luisa Arches (arches)
Changed in nova:
assignee: Maria Luisa Arches (arches) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Maria Luisa Arches (arches)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/627540
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Submitter: Zuul
Branch: master

commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/633734

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/633734
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1ac56a403876fa109385ef6281c525335a613b34
Submitter: Zuul
Branch: stable/rocky

commit 1ac56a403876fa109385ef6281c525335a613b34
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/634236

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/634236
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6d38ba1589e58d13113b1a2e3549927c091ee17c
Submitter: Zuul
Branch: stable/queens

commit 6d38ba1589e58d13113b1a2e3549927c091ee17c
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)
    (cherry picked from commit 1ac56a403876fa109385ef6281c525335a613b34)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.10

This issue was fixed in the openstack/nova 17.0.10 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.0

This issue was fixed in the openstack/nova 18.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/702231

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.opendev.org/702231
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b599bacbcd1f915acf9712662254b2f84baab279
Submitter: Zuul
Branch: stable/pike

commit b599bacbcd1f915acf9712662254b2f84baab279
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)
    (cherry picked from commit 1ac56a403876fa109385ef6281c525335a613b34)
    (cherry picked from commit 6d38ba1589e58d13113b1a2e3549927c091ee17c)

tags: added: in-stable-pike
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers