Device tagging does not work for PF passthrough

[Artom Lifshitz]

Description
===========

When booting an instance with a PF passed through, device tags applied to the PF do not appear in the metadata.

Steps to reproduce
==================

Create a PF neutron port:

$ neutron port-create sriov --binding:vnic-type direct-physical --name pf

Boot a VM with that port:

$ nova test-sriov_vf_2 [...] --nic port-id=88691de9-1a20-4f86-9931-e694786b5bdf,tag=ticky

From inside the guest, access the metadata:

$ curl http://169.254.169.254/openstack/latest/meta_data.json

Expected result
===============

A device with role tag 'ticky' appears in the metadata.

Actual result
=============

No device with role tag 'ticky' in the metadata.

Environment
===========

1. Exact version of OpenStack you are running.

This was originally reported in Red Hat OpenStack 10 (Newton), but I believe the problem still exists on master.

2. Which hypervisor did you use?

libvirt+kvm

3. Which networking type did you use?

neutron

[Artom Lifshitz (notartom)]

I believe this is happening because in _build_device_metadata [1], we only consider <interface> elements in the libvirt XML, whereas PF passthrough appears as a <hostdev> element.

[1] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8041

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/533805

[Stephen Finucane (stephenfinucane)]

It appears this also affects VLAN tags, which should be exposed through the metadata service but are not.

https://bugzilla.redhat.com/show_bug.cgi?id=1367786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533804
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d16d0d553b20fd45a83cf7c68bc477b97aece9a1
Submitter: Zuul
Branch: master

commit d16d0d553b20fd45a83cf7c68bc477b97aece9a1
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:36:40 2018 -0500

    Refactor _build_device_metadata

    This patch splits out interface and disk metadata generation into its
    own function each, in preparation for handling hostdev metadata
    generation in a subsequent patch. It also adds some debug logging.

    Related-bug: 1743458
    Change-Id: I318b9118c97d7bf2b61bc45698443e8d0255ff6f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533805
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=abd01a757b8a83d714a6ab8b5085e7cc1b4a2c5c
Submitter: Zuul
Branch: master

commit abd01a757b8a83d714a6ab8b5085e7cc1b4a2c5c
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:52:11 2018 -0500

    Consider hostdev devices when building metadata

    Previously, device role tagging metadata would only be built for
    LibvirtConfigGuestInterface and LibvirtConfigGuestDisk. This didn't
    account for passed through PFs, which can also be tagged and appear as
    <hostdev> in the libvirt XML. This caused device role tagging to not
    work for PFs. This patch adds support for <hostdev> devices to
    _build_device_metadata.

    Change-Id: I24e5ff1b446f2ac41d589c026ce8ee8adad4bcbc
    Closes-bug: 1743458
    Closes-bug: 1694183

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/581398

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/581399

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581398
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0f253ea9c5f3f3d97bc049f00d1bcdf2761f0736
Submitter: Zuul
Branch: stable/queens

commit 0f253ea9c5f3f3d97bc049f00d1bcdf2761f0736
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:36:40 2018 -0500

    Refactor _build_device_metadata

    This patch splits out interface and disk metadata generation into its
    own function each, in preparation for handling hostdev metadata
    generation in a subsequent patch. It also adds some debug logging.

    NOTE(artom): Conflicts in nova/virt/libvirt/driver.py due to trusted
    VFs not being present in Queens.

    Related-bug: 1743458
    Change-Id: I318b9118c97d7bf2b61bc45698443e8d0255ff6f
    (cherry picked from commit d16d0d553b20fd45a83cf7c68bc477b97aece9a1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581399
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e57f66a26daf858d1348c8406108a33a323b7b2e
Submitter: Zuul
Branch: stable/queens

commit e57f66a26daf858d1348c8406108a33a323b7b2e
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:52:11 2018 -0500

    Consider hostdev devices when building metadata

    Previously, device role tagging metadata would only be built for
    LibvirtConfigGuestInterface and LibvirtConfigGuestDisk. This didn't
    account for passed through PFs, which can also be tagged and appear as
    <hostdev> in the libvirt XML. This caused device role tagging to not
    work for PFs. This patch adds support for <hostdev> devices to
    _build_device_metadata.

    NOTE(artom): Conflicts in nova/tests/unit/virt/libvirt/test_driver.py
    due to trusted VFs not being present in Queens.

    Change-Id: I24e5ff1b446f2ac41d589c026ce8ee8adad4bcbc
    Closes-bug: 1743458
    Closes-bug: 1694183
    (cherry picked from commit abd01a757b8a83d714a6ab8b5085e7cc1b4a2c5c)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/587248

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/587249

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587248
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2460d0ef9438026b1691697022a86030b9f2d21e
Submitter: Zuul
Branch: stable/pike

commit 2460d0ef9438026b1691697022a86030b9f2d21e
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:36:40 2018 -0500

    Refactor _build_device_metadata

    This patch splits out interface and disk metadata generation into its
    own function each, in preparation for handling hostdev metadata
    generation in a subsequent patch. It also adds some debug logging.

    NOTE(artom): Queens -> Pike backport is clean, conflicts in
    nova/virt/libvirt/driver.py in Queens due to trusted VFs not being
    present.

    Related-bug: 1743458
    Change-Id: I318b9118c97d7bf2b61bc45698443e8d0255ff6f
    (cherry picked from commit d16d0d553b20fd45a83cf7c68bc477b97aece9a1)
    (cherry picked from commit 0f253ea9c5f3f3d97bc049f00d1bcdf2761f0736)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587249
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ddcda02a8500ef88ee22b2cabf6d23527ad8de8
Submitter: Zuul
Branch: stable/pike

commit 5ddcda02a8500ef88ee22b2cabf6d23527ad8de8
Author: Artom Lifshitz <email address hidden>
Date: Mon Jan 15 15:52:11 2018 -0500

    Consider hostdev devices when building metadata

    Previously, device role tagging metadata would only be built for
    LibvirtConfigGuestInterface and LibvirtConfigGuestDisk. This didn't
    account for passed through PFs, which can also be tagged and appear as
    <hostdev> in the libvirt XML. This caused device role tagging to not
    work for PFs. This patch adds support for <hostdev> devices to
    _build_device_metadata.

    NOTE(artom): Queens -> Pike backport is clean, conflicts in
    nova/tests/unit/virt/libvirt/test_driver.py in Queens due to trusted
    VFs not being present.

    Change-Id: I24e5ff1b446f2ac41d589c026ce8ee8adad4bcbc
    Closes-bug: 1743458
    Closes-bug: 1694183
    (cherry picked from commit abd01a757b8a83d714a6ab8b5085e7cc1b4a2c5c)
    (cherry picked from commit e57f66a26daf858d1348c8406108a33a323b7b2e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.7 release.