MTU mismatch between o-hm0 and lb-mgmt-net

[Giuseppe Petralia]

lb-mgmt-net is created by the charm with mtu 8942 while on o-hm0 charm sets mtu 1500. This is causing heartbeats of certain LBs to be dropped forcing octavia to fail over them.

This seem related to https://storyboard.openstack.org/#!/story/2009734

Some LBs, because of the number of members/listeners/pools, may have an health status message longer then 1500.

i.e.
12:06:56.706585 fa:16:3e:56:a6:05 > fa:16:3e:cb:ba:9e, ethertype IPv6 (0x86dd), length 1596: fc00:51be:95ec:ac07:f816:3eff:fe56:a605.63575 > fc00:51be:95ec:ac07:f816:3eff:fecb:ba9e.5555: UDP, length 1534

See 1534 > 1500 and o-hm0 never receives them.

We worked around that running on all octavia:

ovs-vsctl set interface o-hm0 mtu_request=8942

[Pedro Victor Lourenço Fragola (pedrovlf)]

I had the same situation when the lb-mgmt-net was set to use 9042 MTU and o-hm0 is set to use 1500 MTU.

The Octavia units will show the drop packets:

[2065317.692684] o-hm0: dropped over-mtu packet: 1744 > 1500
[2065339.573539] o-hm0: dropped over-mtu packet: 1654 > 1500
[2065347.747633] o-hm0: dropped over-mtu packet: 1744 > 1500
[2065375.092206] o-hm0: dropped over-mtu packet: 1653 > 1500
[2065405.176370] o-hm0: dropped over-mtu packet: 1655 > 1500

If the lb-mgmt-net is set to use the same MTU as o-hm0, for example, 1500 it will work using defragment.
During the problem, Octavia Health starts the failover of the amphoras after 60 seconds.

In this case, the LB has more than 60 members.

[Workaround]
The workaround described on this LP works, but the final solution would be the charm check the MTU config lb-mgmt-net and set the same config for o-hm0 interface.

[Edward Hope-Morley (hopem)]

I think this is reasonably high priority given that it is not uncommon for a k8s deployment to have sufficient members to hit this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/883868

[Felipe Reyes (freyes)]

Please, document this new behavior in the charm-guide's release note.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/883868
Committed: https://opendev.org/openstack/charm-octavia/commit/f326ef267aa3e671b608b823bfcaf938868cba45
Submitter: "Zuul (22348)"
Branch: master

commit f326ef267aa3e671b608b823bfcaf938868cba45
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886091

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886261

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia/+/886262

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/886091
Committed: https://opendev.org/openstack/charm-octavia/commit/3ebb523f89cda314b52805443fb4782a16888c66
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 3ebb523f89cda314b52805443fb4782a16888c66
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/886261
Committed: https://opendev.org/openstack/charm-octavia/commit/1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-octavia/+/898957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-octavia/+/898958

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-octavia/+/898959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/898960

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-guide/+/900416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-guide/+/900416
Committed: https://opendev.org/openstack/charm-guide/commit/5c69cfdd46247fc37f6ac901ebf11d4622f8dd81
Submitter: "Zuul (22348)"
Branch: master

commit 5c69cfdd46247fc37f6ac901ebf11d4622f8dd81
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 8 16:25:06 2023 +0200

    Release note: Enforce MTU size between Octavia networks

    Add release note for change in behaviour of MTU with the Octavia charm
    for health manager port and load balancer network.

    Change-Id: I47e0209ad9b6be4b626fdd137a547a46c99fe6a5
    Related-Bug: #2018998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/898957
Committed: https://opendev.org/openstack/charm-octavia/commit/31cb8fe1eb6df48c1b2a8aa2486a699bc544a96f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 31cb8fe1eb6df48c1b2a8aa2486a699bc544a96f
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)
    (cherry picked from commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0)
    (cherry picked from commit 5747bcad9386e30a63e7f2bbb56f81b55979785a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/898958
Committed: https://opendev.org/openstack/charm-octavia/commit/52f34c5ca6f43cb5e0da1f93a1e13e5de6e43b53
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 52f34c5ca6f43cb5e0da1f93a1e13e5de6e43b53
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)
    (cherry picked from commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0)
    (cherry picked from commit 5747bcad9386e30a63e7f2bbb56f81b55979785a)
    (cherry picked from commit 31cb8fe1eb6df48c1b2a8aa2486a699bc544a96f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/898959
Committed: https://opendev.org/openstack/charm-octavia/commit/45dad04e0b05e67b4b79515cc8243249b46b359e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 45dad04e0b05e67b4b79515cc8243249b46b359e
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)
    (cherry picked from commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0)
    (cherry picked from commit 5747bcad9386e30a63e7f2bbb56f81b55979785a)
    (cherry picked from commit 31cb8fe1eb6df48c1b2a8aa2486a699bc544a96f)
    (cherry picked from commit 52f34c5ca6f43cb5e0da1f93a1e13e5de6e43b53)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/898960
Committed: https://opendev.org/openstack/charm-octavia/commit/9c548de6f0e967d860d4280413dbeaf1d33ab8e5
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 9c548de6f0e967d860d4280413dbeaf1d33ab8e5
Author: Edward Hope-Morley <email address hidden>
Date: Mon May 22 20:05:29 2023 +0100

    Ensure mgmt network mtu

    This will ensure that mtu is set whenever setup_hm_port()
    is called regardless of if the port has just been created
    or not. Also checks mtu in update-status hook.

    Change-Id: I80b5cea812becd1724fcfe2a8a232253eac76735
    Closes-Bug: #2018998
    (cherry picked from commit f326ef267aa3e671b608b823bfcaf938868cba45)
    (cherry picked from commit 3ebb523f89cda314b52805443fb4782a16888c66)
    (cherry picked from commit 1fa839826f5009d1cbf8bcc97234afa1fbd5dfb0)
    (cherry picked from commit 5747bcad9386e30a63e7f2bbb56f81b55979785a)
    (cherry picked from commit 31cb8fe1eb6df48c1b2a8aa2486a699bc544a96f)
    (cherry picked from commit 52f34c5ca6f43cb5e0da1f93a1e13e5de6e43b53)
    (cherry picked from commit 45dad04e0b05e67b4b79515cc8243249b46b359e)