[X(L)710] MTU for private network on DPDK-enabled interface is configured incorrectly

[Mikhail Chernik]

Environment:
MOS 9.0, ISO 400, NIC: XL710, dpdk driver igb_uio.

Detailed description

Due to wrong MTU setting network is unstable if a VM on DPDK private network is booted on image other than Cirros. MTU does not take into account 4 bytes VLAN tag.

#ps aux | grep dnsmasq | grep -o -P 'option:mtu[^ ]+'
option:mtu,1500

Steps to reproduce:
1. Deploy an env with Private network placed on a DPDK-enabled interface on a compute node
2. Create Ubuntu Cloud image
3. Set hw:mem_page_size=2048 metadata option for m1.smal flavor
4. Boot a VM from Ubuntu Cloud image on the compute with DPDK-enabled interface
5. When the VM is fully booted, ping it from controller and try to log into it via ssh

Expected result:
Both ping and ssh works

Actual result:
Ping works well, but ssh connection is closed after timeout.

Workaround:
On controller:
NS=$(ip netns list | grep qdhcp)
ip netns exec $NS ifconfig
DEV=$(ifconfig | grep tap | awk '{print $1}')
ip netns exec $NS i plink set mtu 1496 dev $DEV

On VM:
ip link set mtu 1496 dev eth0

[Mikhail Chernik (mchernik)]

Environment: MOS 9.0, ISO 400

[Atsuko Ito (yottatsa)]

MTU doesn't needed to be adjusted for 802.1Q AFAIK

[Atsuko Ito (yottatsa)]

Looks like dpdk driver and/or NIC specifics:
http://dpdk.org/ml/archives/dev/2015-August/023089.html
http://dpdk.org/dev/patchwork/patch/12918/

[Sergey Matov (smatov)]

According to neutron dhcp/ml2 configuration we need to check out 1 parameter for MTU.

In file
/etc/neutron/plugins/ml2/ml2_conf.ini
Parameter
physical_network_mtus

[Atsuko Ito (yottatsa)]

@smatov it should be noted as a workaround.

Also, it could be done by setting up MTU 1496 on Private interfaces on controllers and computes to apply this workaround out of the box.

[Mikhail Chernik (mchernik)]

Request for adding this issue to release notes: https://bugs.launchpad.net/fuel/+bug/1587867
Closed for 9.0.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Olena Logvinova <email address hidden>
Review: https://review.fuel-infra.org/22403

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/22403
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: 9bfce94c4aec46c05e744432fd307a020d4c62c7
Author: Olena Logvinova <email address hidden>
Date: Wed Jun 22 13:20:42 2016

[RN 9.0] [NFV] [Intel XL710 40] known issues

This patch adds the following bugs and their workarounds
to the RN 9.0 Known issues section:

https://bugs.launchpad.net/fuel/+bug/1583077
https://bugs.launchpad.net/fuel/+bug/1587310

Change-Id: Ie15d0888afd46599a0ed421bd757bf1d471504a5
Related-Bug: #1583077
Related-Bug: #1587310
Closes-Bug: #1587867

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Sergey Matov <email address hidden>
Review: https://review.fuel-infra.org/28628

[Sergey Matov (smatov)]

Issue can be fixed with OVS 2.6.1 and DPDK 16.07 by configuring L2 MTU on dpdkX physical interface.
E.g. if we are setting MTU as default (1500 for layer 3) next command required for port named dpdk0

ovs-vsctl set Interface dpdk0 mtu_request=1518

Where 1518 is l3 mtu + eth header size + vlan tag size.

Appropriate changes will be applied for MOS 9.2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/420350

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/420914

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422724

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423629

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423629
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3bdad751d0c46ed964f1dcacffb701d59d7c9958
Submitter: Jenkins
Branch: stable/mitaka

commit 3bdad751d0c46ed964f1dcacffb701d59d7c9958
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422724
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=747e897ae0238251066f7bd3f20c45ad49f7b20a
Submitter: Jenkins
Branch: stable/mitaka

commit 747e897ae0238251066f7bd3f20c45ad49f7b20a
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>
    (cherry picked from commit 737556d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423631
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4a64028d70d492afc0bc631f120e2a54ef076d48
Submitter: Jenkins
Branch: stable/newton

commit 4a64028d70d492afc0bc631f120e2a54ef076d48
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420350
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4e8c035dc4c177f644171a1ea823fdaf14763992
Submitter: Jenkins
Branch: master

commit 4e8c035dc4c177f644171a1ea823fdaf14763992
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424031

[Dmitry Pyzhov (dpyzhov)]

Reopening for master and newton because not all patches landed in these branches: https://review.openstack.org/#/q/topic:bug/1587310

[Mikhail Chernik (mchernik)]

Deploy failed on MOS9.2 RC2 (snapshot #822) with message:

Parameter mtu failed on L23_stored_config[ens11f1]: Validate method failed for class mtu: undefined method `to_i' for {"value"=>{"value"=>"1504"}}:Hash at /etc/puppet/modules/l23network/manifests/l2/port.pp:119

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/427773

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30384

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Evgeny Konstantinov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/30384
Reason: will do a new commit

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/428214

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30442

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/428615

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/428616

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30442
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: ec2096af34a329ece25984db03c0e6c4b614af1e
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:44:26 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/9.2
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30468

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30468
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: a60eafdc71ff08ba3633f68a266694ed7e09998c
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:54:09 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310
(cherry picked from commit ec2096af34a329ece25984db03c0e6c4b614af1e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420914
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Submitter: Jenkins
Branch: master

commit 9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447955

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/428615
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/427773
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428616
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f199c0ba29de6364c67140bfa48ff11c4bb33ded
Submitter: Jenkins
Branch: stable/mitaka

commit f199c0ba29de6364c67140bfa48ff11c4bb33ded
Author: Mikhail <email address hidden>
Date: Thu Feb 2 19:50:29 2017 +0400

    Fix vsctl call

    Change-Id: I700ff89d0ff90fabc1843f0e740ab38d7fbaa45a
    Closes-Bug: #1587310

[Ilya Bumarskov (ibumarskov)]

Can't reproduce bug on our test environment due to lack of appropriate HW. In accordance with our policy, fix should be verified on the customer side.

[Daniel Forsberg (daniel-forsberg1)]

Hi!

I can confirm that this still is a issue in mitaka-9.2.

Fix from here: https://review.openstack.org/#/c/427773/ is not applied

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Matov <email address hidden> on branch: 9.0
Review: https://review.fuel-infra.org/28628

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/428214
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/447955
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/424031
Reason: This repo is retired now, no further work will get merged.