Interface detach fails for VFHostUser vif

[Maria Luisa Arches]

Description
===========
Nova skips removing VHostUser from libvirt, thinking that it
already deleted the interface

Error happened here:
https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/virt/libvirt/driver.py#L1709

Interface was not present because target_dev does not match between libvirt and cfg
https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/virt/libvirt/guest.py#L247

target_dev did not match, because nova does not fill value for target_dev
https://github.com/openstack/nova/blob/5bf6f6304e185c4af2c1b19bc11631793acdfe02/nova/virt/libvirt/vif.py#L433

Steps to reproduce
==================
1. Launch VM with 2 vnics with type vhostuser
$ openstack port list --server test-vm
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 7dc38cc5-7c04-4018-96ff-8975b357771e | | fa:16:3e:25:a0:c4 | ip_address='10.10.12.10', subnet_id='417b8d8b-cb0f-4764-96ca-e716006ee8d1' | ACTIVE |
| ac02bb45-c913-4fed-aa9b-8448106a91d3 | | fa:16:3e:54:de:9b | ip_address='10.10.13.9', subnet_id='d64b8217-22a4-475a-b323-bd03e39cc65e' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
2. Detach one port from the VM:
$ nova detach-interface test-vm ac02bb45-c913-4fed-aa9b-8448106a91d3
$ openstack port list --server test-vm
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 7dc38cc5-7c04-4018-96ff-8975b357771e | | fa:16:3e:25:a0:c4 | ip_address='10.10.12.10', subnet_id='417b8d8b-cb0f-4764-96ca-e716006ee8d1' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
3. Check VM interface. Interface is still visible in the VM
[root@test-vm ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc fq_codel state DOWN group default qlen 1000
    link/ether fa:16:3e:54:de:9b brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:25:a0:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.10.12.10/24 brd 10.10.12.255 scope global dynamic ens4
       valid_lft 85986sec preferred_lft 85986sec
    inet6 fe80::e7df:16b0:a13d:df37/64 scope link
       valid_lft forever preferred_lft forever

Expected result
===============
Interface ens3 should be removed

Environment
===========
Libvirt+KVM
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0
$ virsh --version
3.9.0
Neutron+OVS DPDK
Openstack Queens openstack-nova-compute-17.0.7-1

[sean mooney (sean-k-mooney)]

ill try an reproduce this tommorow.
if you want to address this yourself feel free to reassign.
otherwise ill try and resolve this in the next week or two.

[Maria Luisa Arches (arches)]

Hi,

Thanks.

Additional info, it seems that when interface was attached runtime, target dev is not present in libvirt. In this case, detach was successful.

First interface is from vm creation. Second one, was attached run-time.
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/vhu5b00f2b1-02' mode='server'/>
      <target dev='vhu5b00f2b1-02'/>
      <model type='virtio'/>
      <alias name='net0'/>
    </interface>
    <interface type='vhostuser'>
      <source type='unix' path='/var/run/openvswitch/vhu70a4f277-36' mode='server'/>
      <model type='virtio'/>
      <alias name='net1'/>
    </interface>

Possible solution that I can think of:
1. Skip checking of target dev for vhostuser
2. Update vhostuser to check if target dev is empty, if not, assign a target dev. I need to investigate this still how this can be done.

I'm thinking robust solution is #2, but still need to figure out how to do this.

[Maria Luisa Arches (arches)]

Hi Sean,

I will re-assign this to myself. I will upload proposed fix in review.

Basically fix was to just set target_dev to vif_name in _set_config_vifvhostuser

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/627537

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maria Luisa Arches (<email address hidden>) on branch: master
Review: https://review.openstack.org/627537
Reason: rebranch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/627540

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627540
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Submitter: Zuul
Branch: master

commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/633734

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633734
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1ac56a403876fa109385ef6281c525335a613b34
Submitter: Zuul
Branch: stable/rocky

commit 1ac56a403876fa109385ef6281c525335a613b34
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/634236

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634236
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6d38ba1589e58d13113b1a2e3549927c091ee17c
Submitter: Zuul
Branch: stable/queens

commit 6d38ba1589e58d13113b1a2e3549927c091ee17c
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)
    (cherry picked from commit 1ac56a403876fa109385ef6281c525335a613b34)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/702231

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702231
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b599bacbcd1f915acf9712662254b2f84baab279
Submitter: Zuul
Branch: stable/pike

commit b599bacbcd1f915acf9712662254b2f84baab279
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340
    (cherry picked from commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121)
    (cherry picked from commit 1ac56a403876fa109385ef6281c525335a613b34)
    (cherry picked from commit 6d38ba1589e58d13113b1a2e3549927c091ee17c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova pike-eol release.