GRE - fragmentation packages on big size MTU

[Egor Kotko]

{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis", "experimental"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Steps to reproduce:
1. Deploy cluster (hardware) HA Ubuntu LACP bond 3 (2x10G) Controllers, 2 Computes, 2 Cephs
2. Measure the speed:
 - Create instance, on it execute: nice -n -20 iperf -s --print_mss --nodelay
 - On the controller: #ip netns exec qdhcp-6ea4143c-ba7c-4e39-a541-c58eec1d209c /bin/bash
#nice -n -20 iperf -c <iperf_server__internal_IP> --format m --len 96K --print_mss --nodelay --time 180
3. The speed is ~1 Mbitps

See the traffic on the controller on with executed iperf client:
#tcpdump -i br-mgmt '((ip[6:2] > 0) and (not ip[6] = 64))'

15:49:32.820589 IP node-28.domain.tld > node-27.domain.tld: GREv0, key=0x2, length 1480: IP truncated-ip - 42 bytes missing! 192.168.111.2.37223 > 192.168.111.5.rfe: Flags [.], seq 152040:153488, ack 1, win 229, options [nop,nop,TS val 5992114 ecr 3494352], length 1448
15:49:32.820603 IP node-28.domain.tld > node-27.domain.tld: gre
15:49:33.024643 IP node-28.domain.tld > node-27.domain.tld: GREv0, key=0x2, length 1480: IP truncated-ip - 42 bytes missing! 192.168.111.2.37223 > 192.168.111.5.rfe: Flags [.], seq 153488:154936, ack 1, win 229, options [nop,nop,TS val 5992165 ecr 3494403], length 1448
15:49:33.024658 IP node-28.domain.tld > node-27.domain.tld: gre
The packages will be fragmented because of the GRE + to MTU ~42 bytes

If change the mss size:
#nice -n -20 iperf -c 192.168.111.5 --format m --mss 500 --nodelay --time 180 -p 5002
The speed is 775 Mbits/sec

[Jay Pipes (jaypipes)]

Please also see: https://bugs.launchpad.net/neutron/+bug/1419069

[Jay Pipes (jaypipes)]

Relevant openstack[-dev] ML discussions here:

http://lists.openstack.org/pipermail/openstack/2015-January/011207.html

http://lists.openstack.org/pipermail/openstack-dev/2015-February/056248.html

[Andrey Maximov (maximov)]

I've discussed this bug with Alexander Ignatov and we agreed to check if it is possible to backport fix from kilo release of neutron (https://blueprints.launchpad.net/neutron/+spec/mtu-selection-and-advertisement)

[Ilya Shakhat (shakhat)]

It's proposed to configure MTU via dnsmasq extended config and not to introduce any new parameters and changes into Neutron code-base. The how-to is available at https://www.rdoproject.org/forum/discussion/302/quantumneutron-and-ip-packet-mtus-questions-answers-and-suggestions/p1

The value should be calculated as following:
 - if segmentation type VLAN then set MTU = MTU of physical interface where VLAN specific network is deployed
 - if segmentation type GRE then set MTU = MTU of physical interface with management network - 42 bytes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/170424

[Alexander Ignatov (aignatov)]

According to ILya's comment above moving this bug to Won't fix for mos

[Sergey Kolekonov (skolekonov)]

It was decided to postpone the fix (https://review.openstack.org/#/c/170424) till experiments with mtu over 65k

[Alexander Ignatov (aignatov)]

Some discussions around this issue were in comments of https://bugs.launchpad.net/mos/+bug/1425933
Put it here and will close bug/1425933 as dublicate of this issue

[Victor Ryzhenkin (vryzhenkin)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2-6.1"
  api: "1.0"
  build_number: "337"
  build_id: "2015-04-21_22-54-31"
  nailgun_sha: "3928c5aee6e7aabad37cf0665acc6790ef220141"
  python-fuelclient_sha: "08640730176591a3818f24e75b710f8c95846e6e"
  astute_sha: "bf1751a4fe0d912325e3b4af629126a59c1b2b51"
  fuel-library_sha: "af4eee78a4dd6e6606079f5515bac91c45b04114"
  fuel-ostf_sha: "df8db1f48f03b18126ce5ec65317a1eb83a5a95f"
  fuelmain_sha: "aad0a7ad97a5660f5d53fa830b168d36fa9694eb"

Steps:
1. Deploy OS, Ubuntu, neutron-gre, Murano, Ceph All, HA (3 KVM Controllers, 1 Hardware compute)
2. Create Murano env and add any app
3. Deploy fails by timeout.

Actual:
Network connection on deployed vms very poor, ~3 kb/s and because of this, deployment of Murano fails with timeout exception.

[Timur Nurlygayanov (tnurlygayanov)]

So, looks like it its critical issue for OpenStack clouds with Ubuntu will have significant issues with network performance, it is critical for customers.

[Vladimir Kuklin (vkuklin)]

Victor, please consider cherry-picking of this commit https://review.openstack.org/#/c/170424/ as a workaround. We are workin on the better solution right now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170424
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0d73bfe19c405b186ae2e86ead351ca51db0b22a
Submitter: Jenkins
Branch: master

commit 0d73bfe19c405b186ae2e86ead351ca51db0b22a
Author: Sergey Kolekonov <email address hidden>
Date: Fri Apr 3 13:14:38 2015 +0300

    Set mtu for internal networks according to segmentation type

    Currently MTU for cloud networks is always set to 1500, which is not correct
    for GRE segmentation and causes performance degradation. MTU value should also
    depend on MTU settings of physical interfaces.
    The only way to change MTU for Neutron private network is to use an additional
    dnsmasq configuration file.

    Change-Id: I1fa979639c187e6f6d416100efc2a6548e141c50
    Closes-bug: #1411732

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/180510

[Bogdan Dobrelya (bogdando)]

The commit https://review.openstack.org/170424 should be reverted as the 'vendor_specific' path should not be used for mgmt interface evaluation

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180510
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=40f77208b88e23fdadf13c52c4e91b8a69df6aaf
Submitter: Jenkins
Branch: master

commit 40f77208b88e23fdadf13c52c4e91b8a69df6aaf
Author: Bogdan Dobrelya <email address hidden>
Date: Wed May 6 14:25:53 2015 +0200

    Revert "Set mtu for internal networks according to segmentation type"

    This reverts commit 0d73bfe19c405b186ae2e86ead351ca51db0b22a.
    The path 'vendor_specific' is wrong and should not be used for
    the management interface evaluation.

    Closes-bug: #1452256
    Related-bug: #1411732

    Change-Id: I577f60363f5dd443755c6d0ace514b08ebc77b76

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/181134

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/181134
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0f9e5f9b29fdecaa194df6442d94d85b6f8d2c5a
Submitter: Jenkins
Branch: master

commit 0f9e5f9b29fdecaa194df6442d94d85b6f8d2c5a
Author: Sergey Kolekonov <email address hidden>
Date: Fri Apr 3 13:14:38 2015 +0300

    Set mtu for internal networks according to segmentation type

    Currently MTU for Neutron private networks is always set to 1500,
    which is not correct for GRE segmentation and causes performance degradation.
    MTU value should also depend on MTU settings of physical interfaces.
    The only way to change MTU for Neutron private network is to use an additional
    dnsmasq configuration file.

    Co-Authored-By: Sergey Vasilenko <email address hidden>
    Change-Id: Iaa9631b58f268ec414c58d1b7c47de3573772fc5
    Closes-bug: #1411732

[Egor Kotko (ykotko)]

Verified on:
{"build_id": "2015-05-17_15-56-42", "build_number": "429", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-17_15-56-42", "build_number": "429", "api": "1.0", "fuel-library_sha": "0caef655535a602bc3a9eb72fb4fb402e46dd70d", "nailgun_sha": "076566b5df37f681c3fd5b139c966d680d81e0a5", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "38765563e1a7f14f45201fd47cf507393ff5d673", "astute_sha": "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15", "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33", "release": "6.1", "fuelmain_sha": "0e970647a83d9a7d336c4cc253606d4dd0d59a60"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "0caef655535a602bc3a9eb72fb4fb402e46dd70d", "nailgun_sha": "076566b5df37f681c3fd5b139c966d680d81e0a5", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "38765563e1a7f14f45201fd47cf507393ff5d673", "astute_sha": "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15", "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33", "release": "6.1", "fuelmain_sha": "0e970647a83d9a7d336c4cc253606d4dd0d59a60"}

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.0-updates as there is no channel to deliver Fuel fixes in 6.0