Using macvtap vnic_type is not working with vif_type=hw_veb

[Itzik Brown]

When trying to boot an instance with a port using vnic_type=macvtap and vif_type=hw_veb I get this error in Compute log:

TRACE nova.compute.manager mlibvirtError: unsupported configuration: an interface of type 'direct' is requesting a vlan tag, but that is not supported for this type of connection

[Daniel Berrange (berrange)]

Can you say which version of libvirt you are using here.

[Daniel Berrange (berrange)]

So this Nova code appears to be just be plain broken & can't ever have worked in the config you request

Libvirt code has this validation check

    if (virDomainNetGetActualVlan(iface)) {
        /* vlan configuration via libvirt is only supported for
         * PCI Passthrough SR-IOV devices and openvswitch bridges.
         * otherwise log an error and fail
         */
        if (!(actualType == VIR_DOMAIN_NET_TYPE_HOSTDEV ||
              (actualType == VIR_DOMAIN_NET_TYPE_BRIDGE &&
               virtport && virtport->virtPortType
               == VIR_NETDEV_VPORT_PROFILE_OPENVSWITCH))) {
            if (netdef) {
                virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
                               _("an interface connecting to network '%s' "
                                 "is requesting a vlan tag, but that is not "
                                 "supported for this type of network"),
                               netdef->name);
            } else {
                virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
                               _("an interface of type '%s' "
                                 "is requesting a vlan tag, but that is not "
                                 "supported for this type of connection"),
                               virDomainNetTypeToString(iface->type));
            }
            goto error;

The 'hw_veb' VIF type has two ways to be configured, either using 'hostdev' mode or 'direct' mode. Only the 'hostdev' mode will support use of a vlan, but Nova configures it unconditionally even for 'direct' mode.

One possible fix is to change

def set_vif_host_backend_hw_veb(conf, net_type, devname, vlan,
                                tapname=None):
    """Populate a LibvirtConfigGuestInterface instance
    with host backend details for an device that supports hardware
    virtual ethernet bridge.
    """

    conf.net_type = net_type
    if net_type == 'direct':
        conf.source_mode = 'passthrough'
        conf.source_dev = pci_utils.get_ifname_by_pci_address(devname)
        conf.driver_name = 'vhost'
    else:
        conf.source_dev = devname
        conf.model = None
    conf.vlan = vlan
    if tapname:
        conf.target_dev = tapname

To be

def set_vif_host_backend_hw_veb(conf, net_type, devname, vlan,
                                tapname=None):
    """Populate a LibvirtConfigGuestInterface instance
    with host backend details for an device that supports hardware
    virtual ethernet bridge.
    """

    conf.net_type = net_type
    if net_type == 'direct':
        conf.source_mode = 'passthrough'
        conf.source_dev = pci_utils.get_ifname_by_pci_address(devname)
        conf.driver_name = 'vhost'
       conf.vlan = vlan
    else:
        conf.source_dev = devname
        conf.model = None

    if tapname:
        conf.target_dev = tapname

ie, only set the vlan attribute for the hostdev mode.

I've no idea if such a fix is *semantically* correct though in terms of what Neutron expects.

Could you try such a change though and report back if it works correctly.

[Itzik Brown (itzikb1)]

libvirt version is 1.2.2

Changed to:
def set_vif_host_backend_hw_veb(conf, net_type, devname, vlan,
                                tapname=None):
    """Populate a LibvirtConfigGuestInterface instance
    with host backend details for an device that supports hardware
    virtual ethernet bridge.
    """

    conf.net_type = net_type
    if net_type == 'direct':
        conf.source_mode = 'passthrough'
        conf.source_dev = pci_utils.get_ifname_by_pci_address(devname)
        conf.driver_name = 'vhost'
    else:
        conf.source_dev = devname
        conf.model = None
        conf.vlan = vlan
    if tapname:
        conf.target_dev = tapname

And it works.
Can I push this fix?

There is a need to add a the VLAN set in the Neutron side (i.e. Neutron Agent).

[Baodong (Robert) Li (baoli)]

Yes, I think moving 'conf.vlan = vlan' inside the else: is correct.

In the case of macvtap, the vlan interface needs to be created. This doesn't have to be done by neutron l2 agent, and can be achieved inside the vif driver. Check nova/network/linux_net.py, in which you can find how vlan interfaces are created for linux bridge (ensure_vlan()). However, it's not written as a generic routine such as create_tap_dev() that is inside the same module.

Can we create a method in linux_net.py: create_vlan_interface(vlan_num, parent_interface, mac_addr, mtu), which then can be called by ensure_vlan() as well. This routine can be called by the vif driver when setting up macvtap for hw_veb.

[Irena Berezovsky (irenab)]

I am not sure if solution proposed by Robert is aligned with nova/ neutron separation of concerns. AFAIK neutron is expected to apply networking configuration to vnics created by nova. But following this, neutron agent will be required, while today there is an option to use SR-IOV without running neutron agent, i.e. Cisco VM-FEX. Please advise.

[Baodong (Robert) Li (baoli)]

Irena, if you check the method plug_ovs_hybrid() in vif.py, you will see that the vif driver creates a linux bridge, and adds ports in the integration bridge. Neutron later will add flows in the integration bridge and the downstream bridge to complete the data path setup (as done by the neutron agent).

Certainly, there is no integration bridge anymore for SR-IOV. But it doesn't mean that nova shouldn't participate in the data path setup any more, IMO. macvtap requires a host networking interface (be it regular ethernet interface, or vlan interface) to be connected with, and such an interface either already exists or can be created/setup by nova.

[Daniel Berrange (berrange)]

I don't see a particular problem with nova creating the vlan device if that's what's needed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/126357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126357
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=386e38198d63cf2dc45507ca7e4dc82f0bcd9bb9
Submitter: Jenkins
Branch: master

commit 386e38198d63cf2dc45507ca7e4dc82f0bcd9bb9
Author: Robert Li <email address hidden>
Date: Mon Oct 6 12:48:07 2014 -0400

    Support macvtap for vif_type being hw_veb

    This patch programs VLAN into a VF pci device, and properly
    plug/unplug it into the instance.

    Change-Id: I85eb3c1347d057d1f292e747f950065b8f394147
    Closes-Bug: 1370348
    Co-Authored-By: Itzik Brown <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/151391

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151391
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4ea3c18b2a430d136a9826a086376526a85f4430
Submitter: Jenkins
Branch: stable/juno

commit 4ea3c18b2a430d136a9826a086376526a85f4430
Author: Robert Li <email address hidden>
Date: Mon Oct 6 12:48:07 2014 -0400

    Support macvtap for vif_type being hw_veb

    This patch programs VLAN into a VF pci device, and properly
    plug/unplug it into the instance.

    Change-Id: I85eb3c1347d057d1f292e747f950065b8f394147
    Closes-Bug: 1370348
    Co-Authored-By: Itzik Brown <email address hidden>
    (cherry picked from commit 386e38198d63cf2dc45507ca7e4dc82f0bcd9bb9)