OpenvSwitch agent sets DF flag for GRE tunnels breaking instances connectivity

[Vladimir Kuklin]

Currently, we are facing issues with neutron ovs agent and GRE tunnels. At least for our Ubuntu version which has 2.0.2 version of kernel module and userspace we can see that packets more than 1458 bytes cannot get through. This happens because for some reason gre tunnels are created with df_inherit and df_default flags set to true.

Disabling them fixes the issue completely.

It seems we need to apply the patch that sets dont_fragment option to false into agent section of neutron plugins config along with setting df_inherit to false in neutron code.

I am not sure if we need to set Centos df_inherit flag to false (as long as I am not sure it exists for OVS 1.10), but it seems we also need to set df_default to false for centos installation.

For ubuntu we need to add the same string in

/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovs_lib.py

232: vsctl_command.append(("options:df_inherit=%s" %
                                 bool(dont_fragment)).lower())

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141487

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/141488

[Alexei Sheplyakov (asheplyakov)]

> Currently, we are facing issues with neutron ovs agent and GRE tunnels. At least for our Ubuntu version which has
> 2.0.2 version of kernel module and userspace we can see that packets more than 1458 bytes cannot get through.

I think this behavior is correct: ovs tries to avoid fragmentation of the outer IPv4 packets. The packets larger than
GRE tunnel MTU (1456 == 1500 - 24 - 20) must be fragmented by the guest OS itself. To make this work neutron
should pass the correct MTU value (<= 1456) to instances attached to GRE tunnels.

[Sergii Golovatiuk (sgolovatiuk)]

Usually, GRE tunnel MTU should be (Host MTU - 24 - 20) to allow clients to
use Jumbo frames. If HOST MTU=9000, then guest MTU should be = 8956

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Sun, Dec 14, 2014 at 7:36 PM, Alexei Sheplyakov <<email address hidden>
> wrote:
>
> > Currently, we are facing issues with neutron ovs agent and GRE tunnels.
> At least for our Ubuntu version which has
> > 2.0.2 version of kernel module and userspace we can see that packets
> more than 1458 bytes cannot get through.
>
> I think this behavior is correct: ovs tries to avoid fragmentation of the
> outer IPv4 packets. The packets larger than
> GRE tunnel MTU (1456 == 1500 - 24 - 20) must be fragmented by the guest OS
> itself. To make this work neutron
> should pass the correct MTU value (<= 1456) to instances attached to GRE
> tunnels.
>
> --
> You received this bug notification because you are a member of Fuel
> Library Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1402046
>
> Title:
> OpenvSwitch agent sets DF flag for GRE tunnels breaking instances
> connectivity
>
> Status in Fuel: OpenStack installer that works:
> In Progress
> Status in Fuel for OpenStack 6.0.x series:
> In Progress
> Status in Fuel for OpenStack 6.1.x series:
> Triaged
>
> Bug description:
> Currently, we are facing issues with neutron ovs agent and GRE
> tunnels. At least for our Ubuntu version which has 2.0.2 version of
> kernel module and userspace we can see that packets more than 1458
> bytes cannot get through. This happens because for some reason gre
> tunnels are created with df_inherit and df_default flags set to true.
>
> Disabling them fixes the issue completely.
>
> It seems we need to apply the patch that sets dont_fragment option to
> false into agent section of neutron plugins config along with setting
> df_inherit to false in neutron code.
>
> I am not sure if we need to set Centos df_inherit flag to false (as
> long as I am not sure it exists for OVS 1.10), but it seems we also
> need to set df_default to false for centos installation.
>
> For ubuntu we need to add the same string in
>
> /usr/lib/python2.7/dist-packages/neutron/agent/linux/ovs_lib.py
>
> 232: vsctl_command.append(("options:df_inherit=%s" %
> bool(dont_fragment)).lower())
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1402046/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141487
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a57c156550ebe769119df2e011a1135e8ea4598c
Submitter: Jenkins
Branch: master

commit a57c156550ebe769119df2e011a1135e8ea4598c
Author: Vladimir Kuklin <email address hidden>
Date: Fri Dec 12 23:36:49 2014 +0300

    Disable DF bit for tunneled installations

    Due to the way how OVS handles GRE tunnels
    it tries to copy DF flag to wrapping GRE
    packet header, thus making it impossible
    for packets of more than 1430 bytes size
    to be transfered through regular interfaces
    this breaks almost all applications. Thus
    we are setting dont_fragment option to false
    and we will also need to make ovs agent
    set df_inherit flag to this option for
    OVS versions that support df_inherit flag

    We will also need to document that user should
    extend his MTU size to reasonable one according
    to http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron-ml2-network-node.html
    in order to not get performance issues.

    DocImpact
    Change-Id: I031351b36f56881f21986b412bfae79fba0e884b
    Partial-bug: #1402046

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141488
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=631853551ca8996bab70a84e2fadf082301509a8
Submitter: Jenkins
Branch: stable/6.0

commit 631853551ca8996bab70a84e2fadf082301509a8
Author: Vladimir Kuklin <email address hidden>
Date: Fri Dec 12 23:36:49 2014 +0300

    Disable DF bit for tunneled installations

    Due to the way how OVS handles GRE tunnels
    it tries to copy DF flag to wrapping GRE
    packet header, thus making it impossible
    for packets of more than 1430 bytes size
    to be transfered through regular interfaces
    this breaks almost all applications. Thus
    we are setting dont_fragment option to false
    and we will also need to make ovs agent
    set df_inherit flag to this option for
    OVS versions that support df_inherit flag

    We will also need to document that user should
    extend his MTU size to reasonable one according
    to http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron-ml2-network-node.html
    in order to not get performance issues.

    DocImpact
    Change-Id: I031351b36f56881f21986b412bfae79fba0e884b
    Partial-bug: #1402046

[Vladimir Kuklin (vkuklin)]

Partial fix was merged for 6.0. Anyway, this issue needs to be resolved by hypervisor and investigated further.

[Alexander Kurenyshev (akurenyshev)]

Verified on
{"build_id": "2014-12-18_01-32-01",
"ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
"build_number": "56",
"auth_required": true, "api": "1.0",
"nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
"production": "docker",
"fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9",
"astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
"feature_groups": ["mirantis"], "release": "6.0",
"release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01",
"ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
"build_number": "56",
"api": "1.0",
"nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
"production": "docker",
"fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9",
"astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
"feature_groups": ["mirantis"], "release": "6.0",
"fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Packets with length more than 1458 can get through tunnels:
ubuntu@test:~$ sudo tcpdump
09:31:49.186357 IP 10.108.3.1 > 10.108.3.3: ICMP echo request, id 7831, seq 1, length 1480
09:31:49.186648 IP 10.108.3.1 > 10.108.3.3: icmp

[Bogdan Dobrelya (bogdando)]

@Vladimir, does merged related fix resolve the issue for 6.1?