MTUs are not set for VIFs if using kernel ovs + hybrid plug = false

Bug #1747496 reported by sean mooney
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
sean mooney

Bug Description

Description
===========
over the last few cyles supprot for mtu other then the default of 1500 has
gernerally been improved in both nova and neutron.
At the same time it was decided to remove the responsibility for VIF plugging from
the virt drivers and centrailse it in os-vif.

over the last few cycles os-vif has been enhanced to support setting the mtu on all codepaths
and this work was completed in pike, however there are stills codepaths in the nova libvirt driver where os-vif is not used to plug the VIF and instead it is done by libvirt.

when the VIF_TYPE is ovs and hybrid_plug=False libvirt plug the VM's VIFs iteself and os-vif is only responcible for creating the bridge it will be plugged into. in this case as the mtu is not set in teh libvirt xml and since os-vif is not respocible for pluggin the vif nothing set the mtu on the tap device that is added to ovs. This scenario arrises whenever libvirt is the nova virt driver and the no-op or openvswitch security group drivers are used.

the end result is that in the vm the quest correctly recives the non default(e.g. jumbo frame) mtu form the neutron dhcp server and configures the mtu in its kernel but the mtu of the tap device added to the ovs bridge is left at the default of 1500 preventing jumboframes from being used by the guest.

Steps to reproduce
==================
using a host with a non default mtu

deploy devstack normally useing libvirt + kvm/qemu
and enable the openvsiwtch or no-op neutron security group driver

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]
[securitygroup]
firewall_driver = openvswitch

or

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]
[securitygroup]
firewall_driver = noop

spawn a singel vm via nova.

and retrive the name of the interface for ovsdb or
via virsh dumpxml.

then run ifconfig <interface name> and check the mtu.

note if openvsiwtch direver is used you will need to allow icmp/ssh
in the security groups to be able to validate network conncetivity.

Expected result
===============
tap should have same mtu as is set on neutron network.
and a ping of max mtu e.g. ping -s 9000 ... for a network mtu of 9000 should work.

Actual result
=============
tap mtu will be 1500
it is not possible to ping the vm with a packet larger then

Environment
===========
1. it was seen on pike but this effect all versions of openstack.
   before the introduction of os-vif we did not support neutron network mtus
   and after we started to use os-vif we enable neutron mtu support only for
   the os-vif codepath so this never worked.

2. Which hypervisor did you use?
   libvirt with kvm. this is not libvirt version
   specific as we do not generate the libvirt xml to set the mtu
   https://libvirt.org/formatdomain.html#mtu

2. Which storage type did you use?
   N/a but i used ceph

3. Which networking type did you use?
   neutron with kernel ovs and noop or openvsiwtch security group driver.
   not this will not happen with the iptables driver as that set hybrid_plug=True
   so os-vif is used to plug the VIF and it sets the mtu correctly.
   to

Revision history for this message
sean mooney (sean-k-mooney) wrote :

the simple fix is to add a new fucntion to the desinger.py and invoke it here
https://github.com/openstack/nova/blob/58328627f56379f1fd80ffbfca186262d578bc9f/nova/virt/libvirt/vif.py#L239

that will add the mtu to the xml fragment if libvirt version is > 3.2

alternitive fix is to use os-vif for all code paths where vif_type is ovs and change libvirt to interface type ethernet instead of bridge.

Revision history for this message
sean mooney (sean-k-mooney) wrote :

ill try an fix this week or before the ptg if i get delayed.

Changed in nova:
assignee: nobody → sean mooney (sean-k-mooney)
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
tags: added: libvirt network osvif
Revision history for this message
sean mooney (sean-k-mooney) wrote :

FYI i am just getting around to starting this. i have set up a dev env to reporduce and will be working on the libvirt driver xml change first which is back portable followed possibly by delegating the plugin to os-vif which is not backportable.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/553072

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: sean mooney (sean-k-mooney) → sahid (sahid-ferdjaoui)
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

Please Sean, reassign yourself to the bug. It seems that I can't do that for you.

Changed in nova:
assignee: sahid (sahid-ferdjaoui) → nobody
Changed in nova:
assignee: nobody → sean mooney (sean-k-mooney)
Changed in nova:
assignee: sean mooney (sean-k-mooney) → sahid (sahid-ferdjaoui)
Changed in nova:
assignee: sahid (sahid-ferdjaoui) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/553072
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f02b3800051234ecc14f3117d5987b1a8ef75877
Submitter: Zuul
Branch: master

commit f02b3800051234ecc14f3117d5987b1a8ef75877
Author: Sean Mooney <email address hidden>
Date: Wed Mar 14 21:00:33 2018 +0000

    add mtu to libvirt xml for ethernet and bridge types

    Where libvirt is creating the tap device e.g: OVS bridge
    (hybrid=False). os-vif will just ensure that the OVS bridge exists. We
    want libvirt to create the tap, configure the MTU and then plug the
    tap into the right bridge.

    - This change adds the mtu element to libvirt guest
      interface xml for vifs of type ethernet and bridge.
    - This change adds a libvirt version check
      to enable setting the interface mtu on hosts with
      libvirt 3.3.0+

    Change-Id: Iecc265fb25e88fa00a66f1fd38e215cad53e7669
    Partial-bug: #1747496

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/578010

tags: added: queens-backport-potential
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → sean mooney (sean-k-mooney)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/578010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=127dd738c0ec67f3559774bae83dd8872e616991
Submitter: Zuul
Branch: stable/queens

commit 127dd738c0ec67f3559774bae83dd8872e616991
Author: Sean Mooney <email address hidden>
Date: Wed Mar 14 21:00:33 2018 +0000

    add mtu to libvirt xml for ethernet and bridge types

    Where libvirt is creating the tap device e.g: OVS bridge
    (hybrid=False). os-vif will just ensure that the OVS bridge exists. We
    want libvirt to create the tap, configure the MTU and then plug the
    tap into the right bridge.

    - This change adds the mtu element to libvirt guest
      interface xml for vifs of type ethernet and bridge.
    - This change adds a libvirt version check
      to enable setting the interface mtu on hosts with
      libvirt 3.3.0+

    Conflicts:
            nova/virt/libvirt/vif.py

    NOTE(lyarwood): This conflict is due to the removal of IVS plug/unplug
    via dIcf948f6ee1c0da2327fb5eac61fec6e89ac30531 during Rocky, resulting
    in the eventual master change having to reintroduce get_config_bridge
    that is already present in stable/queens.

    Change-Id: Iecc265fb25e88fa00a66f1fd38e215cad53e7669
    Partial-bug: #1747496
    (cherry picked from commit f02b3800051234ecc14f3117d5987b1a8ef75877)

tags: added: in-stable-queens
Changed in nova:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.