[X(L)710] MTU for private network on DPDK-enabled interface is configured incorrectly

Bug #1587310 reported by Mikhail Chernik on 2016-05-31
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Fedor Zhadaev
Mitaka
High
MOS Maintenance
Newton
High
Fedor Zhadaev
Ocata
High
Fedor Zhadaev

Bug Description

Environment:
MOS 9.0, ISO 400, NIC: XL710, dpdk driver igb_uio.

Detailed description

Due to wrong MTU setting network is unstable if a VM on DPDK private network is booted on image other than Cirros. MTU does not take into account 4 bytes VLAN tag.

#ps aux | grep dnsmasq | grep -o -P 'option:mtu[^ ]+'
option:mtu,1500

Steps to reproduce:
1. Deploy an env with Private network placed on a DPDK-enabled interface on a compute node
2. Create Ubuntu Cloud image
3. Set hw:mem_page_size=2048 metadata option for m1.smal flavor
4. Boot a VM from Ubuntu Cloud image on the compute with DPDK-enabled interface
5. When the VM is fully booted, ping it from controller and try to log into it via ssh

Expected result:
Both ping and ssh works

Actual result:
Ping works well, but ssh connection is closed after timeout.

Workaround:
On controller:
NS=$(ip netns list | grep qdhcp)
ip netns exec $NS ifconfig
DEV=$(ifconfig | grep tap | awk '{print $1}')
ip netns exec $NS i plink set mtu 1496 dev $DEV

On VM:
ip link set mtu 1496 dev eth0

tags: added: area-library
Mikhail Chernik (mchernik) wrote :

Environment: MOS 9.0, ISO 400

description: updated
Vladimir Eremin (yottatsa) wrote :

MTU doesn't needed to be adjusted for 802.1Q AFAIK

Vladimir Eremin (yottatsa) wrote :
tags: added: area-linux
removed: area-library
tags: added: team-network
Sergey Matov (smatov) wrote :

According to neutron dhcp/ml2 configuration we need to check out 1 parameter for MTU.

In file
/etc/neutron/plugins/ml2/ml2_conf.ini
Parameter
physical_network_mtus

Vladimir Eremin (yottatsa) wrote :

@smatov it should be noted as a workaround.

Also, it could be done by setting up MTU 1496 on Private interfaces on controllers and computes to apply this workaround out of the box.

summary: - MTU for private network on DPDK-enabled interface is configured
- incorrectly
+ [X(L)710] MTU for private network on DPDK-enabled interface is
+ configured incorrectly
Ilya Kutukov (ikutukov) on 2016-05-31
Changed in fuel:
importance: Undecided → High
status: New → Confirmed
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
no longer affects: fuel/newton
Dmitry Pyzhov (dpyzhov) on 2016-06-01
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Networking (l23-network)
Mikhail Chernik (mchernik) wrote :

Request for adding this issue to release notes: https://bugs.launchpad.net/fuel/+bug/1587867
Closed for 9.0.

Related fix proposed to branch: master
Change author: Olena Logvinova <email address hidden>
Review: https://review.fuel-infra.org/22403

tags: added: release-notes
Dmitry Pyzhov (dpyzhov) on 2016-06-22
Changed in fuel:
assignee: Registry Administrators (registry) → l23network (l23network)

Reviewed: https://review.fuel-infra.org/22403
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: 9bfce94c4aec46c05e744432fd307a020d4c62c7
Author: Olena Logvinova <email address hidden>
Date: Wed Jun 22 13:20:42 2016

[RN 9.0] [NFV] [Intel XL710 40] known issues

This patch adds the following bugs and their workarounds
to the RN 9.0 Known issues section:

https://bugs.launchpad.net/fuel/+bug/1583077
https://bugs.launchpad.net/fuel/+bug/1587310

Change-Id: Ie15d0888afd46599a0ed421bd757bf1d471504a5
Related-Bug: #1583077
Related-Bug: #1587310
Closes-Bug: #1587867

tags: added: release-notes-done
removed: release-notes

Fix proposed to branch: 9.0
Change author: Sergey Matov <email address hidden>
Review: https://review.fuel-infra.org/28628

Sergey Matov (smatov) wrote :

Issue can be fixed with OVS 2.6.1 and DPDK 16.07 by configuring L2 MTU on dpdkX physical interface.
E.g. if we are setting MTU as default (1500 for layer 3) next command required for port named dpdk0

ovs-vsctl set Interface dpdk0 mtu_request=1518

Where 1518 is l3 mtu + eth header size + vlan tag size.

Appropriate changes will be applied for MOS 9.2

Dmitry Pyzhov (dpyzhov) on 2016-12-05
Changed in fuel:
assignee: Registry Administrators (registry) → Fuel Sustaining (fuel-sustaining-team)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Sergey Matov (smatov)
Sergey Matov (smatov) on 2016-12-17
Changed in fuel:
status: Confirmed → In Progress
Dmitry Klenov (dklenov) on 2016-12-19
Changed in fuel:
assignee: Sergey Matov (smatov) → Mikhail Zhnichkov (mzhnichkov)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
Changed in fuel:
assignee: Fedor Zhadaev (fzhadaev) → Mikhail Zhnichkov (mzhnichkov)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
Changed in fuel:
assignee: Fedor Zhadaev (fzhadaev) → Mikhail Zhnichkov (mzhnichkov)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)

Reviewed: https://review.openstack.org/423629
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3bdad751d0c46ed964f1dcacffb701d59d7c9958
Submitter: Jenkins
Branch: stable/mitaka

commit 3bdad751d0c46ed964f1dcacffb701d59d7c9958
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Reviewed: https://review.openstack.org/422724
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=747e897ae0238251066f7bd3f20c45ad49f7b20a
Submitter: Jenkins
Branch: stable/mitaka

commit 747e897ae0238251066f7bd3f20c45ad49f7b20a
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>
    (cherry picked from commit 737556d)

tags: added: in-stable-mitaka

Reviewed: https://review.openstack.org/423631
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4a64028d70d492afc0bc631f120e2a54ef076d48
Submitter: Jenkins
Branch: stable/newton

commit 4a64028d70d492afc0bc631f120e2a54ef076d48
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/420350
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4e8c035dc4c177f644171a1ea823fdaf14763992
Submitter: Jenkins
Branch: master

commit 4e8c035dc4c177f644171a1ea823fdaf14763992
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Dmitry Pyzhov (dpyzhov) wrote :

Reopening for master and newton because not all patches landed in these branches: https://review.openstack.org/#/q/topic:bug/1587310

Mikhail Chernik (mchernik) wrote :

Deploy failed on MOS9.2 RC2 (snapshot #822) with message:

Parameter mtu failed on L23_stored_config[ens11f1]: Validate method failed for class mtu: undefined method `to_i' for {"value"=>{"value"=>"1504"}}:Hash at /etc/puppet/modules/l23network/manifests/l2/port.pp:119

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30384

Change abandoned by Evgeny Konstantinov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/30384
Reason: will do a new commit

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30442

Reviewed: https://review.fuel-infra.org/30442
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: ec2096af34a329ece25984db03c0e6c4b614af1e
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:44:26 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310

Related fix proposed to branch: stable/9.2
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30468

Reviewed: https://review.fuel-infra.org/30468
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: a60eafdc71ff08ba3633f68a266694ed7e09998c
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:54:09 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310
(cherry picked from commit ec2096af34a329ece25984db03c0e6c4b614af1e)

Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Georgy Kibardin (gkibardin)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Georgy Kibardin (gkibardin)

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)

Reviewed: https://review.openstack.org/420914
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Submitter: Jenkins
Branch: master

commit 9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/428615
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/427773
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Reviewed: https://review.openstack.org/428616
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f199c0ba29de6364c67140bfa48ff11c4bb33ded
Submitter: Jenkins
Branch: stable/mitaka

commit f199c0ba29de6364c67140bfa48ff11c4bb33ded
Author: Mikhail <email address hidden>
Date: Thu Feb 2 19:50:29 2017 +0400

    Fix vsctl call

    Change-Id: I700ff89d0ff90fabc1843f0e740ab38d7fbaa45a
    Closes-Bug: #1587310

Ilya Bumarskov (ibumarskov) wrote :

Can't reproduce bug on our test environment due to lack of appropriate HW. In accordance with our policy, fix should be verified on the customer side.

Hi!

I can confirm that this still is a issue in mitaka-9.2.

Fix from here: https://review.openstack.org/#/c/427773/ is not applied

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers