[ovn] mtu for metadata veth interface is not set

Bug #2053274 reported by Mohammed Naser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Rodolfo Alonso

Bug Description

When using OVN, the `veth` interfaces which get created inside the network namespace (and the other half that goes into the OVS bridge) both do not get an MTU configured for them when they are provisioned.

https://github.com/openstack/neutron/blob/stable/zed/neutron/agent/ovn/metadata/agent.py#L589-L594

This can cause some unknown/annoying errors with packets being dropped if a user is hitting large requests on the metadata service, the ideal solution would be to configure the correct MTU for the interface to avoid this issue.

tags: added: ovn
Revision history for this message
Brian Haley (brian-haley) wrote :

Can you add the failure/errors just so they are documented?

And have you tried setting it manually in the namespace and verified it helps?

It seems there is already a key for this, OVN_NETWORK_MTU_EXT_ID_KEY, but it's never set when a port is created in OVN NB from what I can tell. If that happened then the OVN metadata agent should be able to pull it out of the port binding in OVN SB when it gets the port info for its datapaths.

Most likely place to put that is in the ovn_client code in get_external_ids_from_port() and related functions, then modify the agent accordingly. Just my guess.

Changed in neutron:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
Mohammed Naser (mnaser) wrote :

Thanks for that feedback Brian.

Reference from here: https://github.com/vexxhost/atmosphere/issues/949

We're seeing the following error:

```
2024-02-08T16:24:36Z|271237|netdev_linux|WARN|tap6e394a6b-30: Dropped packet: Too big. GRO/TSO enabled?
```

It disappears once the MTU on the `veth` interfaces (prefixed `tap`) is set to the correct one.

Revision history for this message
Brian Haley (brian-haley) wrote :

Thanks for the info, does seem like an issue and setting the mtu would help.

Changed in neutron:
status: Incomplete → Confirmed
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Mohammed:

Please check [1]. The message you are receiving is because most probably you have TSO enabled in the kernel but not in netdev (you mentioned this is happening when using DPDK). Please check disabling TSO in the kernel.

Regards.

[1]https://<email address hidden>/msg57091.html

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/909684

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/910717

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/910718

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/909684
Committed: https://opendev.org/openstack/neutron/commit/47b4d14955451c320ce612e4e2e7d0a896e2fe36
Submitter: "Zuul (22348)"
Branch: master

commit 47b4d14955451c320ce612e4e2e7d0a896e2fe36
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Feb 21 15:34:13 2024 +0000

    [OVN] Set MTU of the VETH interfaces between OVS and metadata

    The VETH pair between the metadata namespace and the local OVS now
    has the same MTU as the network associated to this metadata service.
    The "LSP.external_ids" field has a new key defined: "neutron:mtu".
    This is the value of the network MTU.

    This patch does not update the previous metadata datapaths nor the
    existing LSP. This change will affect only to new created ports
    and the corresponding metadata datapaths.

    Closes-Bug: #2053274

    Change-Id: I7ff300e9634e5e3fc68d70540392109fd8b9babc

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910717
Committed: https://opendev.org/openstack/neutron/commit/101898fde869a5f530580d1c69c52cdc4a33cd98
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 101898fde869a5f530580d1c69c52cdc4a33cd98
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Feb 21 15:34:13 2024 +0000

    [OVN] Set MTU of the VETH interfaces between OVS and metadata

    The VETH pair between the metadata namespace and the local OVS now
    has the same MTU as the network associated to this metadata service.
    The "LSP.external_ids" field has a new key defined: "neutron:mtu".
    This is the value of the network MTU.

    This patch does not update the previous metadata datapaths nor the
    existing LSP. This change will affect only to new created ports
    and the corresponding metadata datapaths.

    Conflicts:
        neutron/agent/ovn/metadata/agent.py

    Closes-Bug: #2053274
    Change-Id: I7ff300e9634e5e3fc68d70540392109fd8b9babc
    (cherry picked from commit 47b4d14955451c320ce612e4e2e7d0a896e2fe36)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910718
Committed: https://opendev.org/openstack/neutron/commit/5ffec5d17a0aa03afb6353c39865b42290b678c1
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 5ffec5d17a0aa03afb6353c39865b42290b678c1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Feb 21 15:34:13 2024 +0000

    [OVN] Set MTU of the VETH interfaces between OVS and metadata

    The VETH pair between the metadata namespace and the local OVS now
    has the same MTU as the network associated to this metadata service.
    The "LSP.external_ids" field has a new key defined: "neutron:mtu".
    This is the value of the network MTU.

    This patch does not update the previous metadata datapaths nor the
    existing LSP. This change will affect only to new created ports
    and the corresponding metadata datapaths.

    Conflicts:
        neutron/agent/ovn/metadata/agent.py

    Closes-Bug: #2053274
    Change-Id: I7ff300e9634e5e3fc68d70540392109fd8b9babc
    (cherry picked from commit 47b4d14955451c320ce612e4e2e7d0a896e2fe36)
    (cherry picked from commit 101898fde869a5f530580d1c69c52cdc4a33cd98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/911065

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/911065
Committed: https://opendev.org/openstack/neutron/commit/cf21a4d9791281605ab3cebbbf3de3cb8631b5bc
Submitter: "Zuul (22348)"
Branch: stable/zed

commit cf21a4d9791281605ab3cebbbf3de3cb8631b5bc
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Feb 21 15:34:13 2024 +0000

    [OVN] Set MTU of the VETH interfaces between OVS and metadata

    The VETH pair between the metadata namespace and the local OVS now
    has the same MTU as the network associated to this metadata service.
    The "LSP.external_ids" field has a new key defined: "neutron:mtu".
    This is the value of the network MTU.

    This patch does not update the previous metadata datapaths nor the
    existing LSP. This change will affect only to new created ports
    and the corresponding metadata datapaths.

    Conflicts:
        neutron/agent/ovn/metadata/agent.py
        neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py

    Closes-Bug: #2053274
    Change-Id: I7ff300e9634e5e3fc68d70540392109fd8b9babc
    (cherry picked from commit 47b4d14955451c320ce612e4e2e7d0a896e2fe36)
    (cherry picked from commit 101898fde869a5f530580d1c69c52cdc4a33cd98)
    (cherry picked from commit 5ffec5d17a0aa03afb6353c39865b42290b678c1)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 24.0.0.0rc1

This issue was fixed in the openstack/neutron 24.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/914697

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.2.1

This issue was fixed in the openstack/neutron 21.2.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.