MTU problem for external network access

[Chris L]

In a 2-node Microstack deployment, I followed the instructions to get a proper external network connectivity from: https://connection.rnascimento.com/2021/03/08/openstack-single-node-microstack/

When a VM is trying to access the external network, a ping of 1470 byte fails, while only around 1400 bytes work. It seems that there is something prevents a standard MTU from working between the VM and the external network.

I can ping fine between different VM with a full frame, the issue is only to the external network.

Should the network_type be "geneve" for the private network? Following the traffic with a tcpdump, the VM is running on 1st node, but traffic leaves to the network on br-ex on the secondary node for some reason.

Could the issue be the local network encapsulation adds extra headers and there's no room for a full 1500 MTU ?

clambert@microstack1:/var/snap/microstack/common/log$ openstack network show public
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2021-07-01T20:40:10Z |
| description | |
| dns_domain | None |
| id | 9c40d4b4-e708-477c-9057-0d98fd68dc63 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| location | cloud='', project.domain_id=, project.domain_name='default', project.id='6100575b7f95425ab8017d5cb0b8f6ce', project.name='admin', region_name='', zone= |
| mtu | 1500 |
| name | public |
| port_security_enabled | True |
| project_id | 6100575b7f95425ab8017d5cb0b8f6ce |
| provider:network_type | flat |
| provider:physical_network | physnet1 |
| provider:segmentation_id | None |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | External |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 19c5c393-d015-41a0-9b9f-d1d5b9a56ffb |
| tags | |
| updated_at | 2021-07-01T20:40:11Z |
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
clambert@microstack1:/var/snap/microstack/common/log$ openstack network show private
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2021-07-01T20:40:47Z |
| description | |
| dns_domain | None |
| id | 5edbde4b-149a-4431-83c1-c790c3b00971 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| location | cloud='', project.domain_id=, project.domain_name='default', project.id='6100575b7f95425ab8017d5cb0b8f6ce', project.name='admin', region_name='', zone= |
| mtu | 1500 |
| name | private |
| port_security_enabled | True |
| project_id | 6100575b7f95425ab8017d5cb0b8f6ce |
| provider:network_type | local |
| provider:physical_network | None |
| provider:segmentation_id | None |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 680b3913-4f44-45eb-bb92-dcc1797399dd |
| tags | |
| updated_at | 2021-07-01T20:40:48Z |
+---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+

clambert@microstack1:/var/snap/microstack/common/log$ openstack router show router
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2021-07-01T20:41:06Z |
| description | |
| external_gateway_info | {"network_id": "9c40d4b4-e708-477c-9057-0d98fd68dc63", "external_fixed_ips": [{"subnet_id": "19c5c393-d015-41a0-9b9f-d1d5b9a56ffb", "ip_address": "172.16.6.254"}], "enable_snat": true} |
| flavor_id | None |
| id | c9017708-eb86-481f-a349-269f9fe8922c |
| interfaces_info | [{"port_id": "3dd3a07a-0df7-44f4-bb5f-570460e00194", "ip_address": "10.20.20.1", "subnet_id": "680b3913-4f44-45eb-bb92-dcc1797399dd"}] |
| location | cloud='', project.domain_id=, project.domain_name='default', project.id='6100575b7f95425ab8017d5cb0b8f6ce', project.name='admin', region_name='', zone= |
| name | router |
| project_id | 6100575b7f95425ab8017d5cb0b8f6ce |
| revision_number | 4 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2021-07-01T20:41:19Z |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

clambert@microstack1:/var/snap/microstack/common/log$ ip -o addr
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever
3: docker0 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\ valid_lft forever preferred_lft forever
5: br-ex inet 10.20.20.1/24 scope global br-ex\ valid_lft forever preferred_lft forever
5: br-ex inet 172.16.6.111/24 scope global br-ex\ valid_lft forever preferred_lft forever
5: br-ex inet6 fe80::cce:3ff:fed4:8043/64 scope link \ valid_lft forever preferred_lft forever
6: br-int inet6 fe80::8c9a:7ff:fe68:14f/64 scope link \ valid_lft forever preferred_lft forever
8: genev_sys_6081 inet6 fe80::702d:b6ff:fe4e:4b28/64 scope link \ valid_lft forever preferred_lft forever

clambert@microstack1:/var/snap/microstack/common/log$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:a4:38:bf brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:e8:c4:df:33 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:9a:eb:16:85:4a brd ff:ff:ff:ff:ff:ff
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:a4:38:bf brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 8e:9a:07:68:01:4f brd ff:ff:ff:ff:ff:ff
8: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 72:2d:b6:4e:4b:28 brd ff:ff:ff:ff:ff:ff

clambert@microstack1:/var/snap/microstack/common/log$ sudo ovs-vsctl show
8043fc61-ba0b-45f9-8f8e-492033d97709
    Bridge br-int
        fail_mode: secure
        Port ovn-1f92da-0
            Interface ovn-1f92da-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.16.6.113"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port br-int
            Interface br-int
                type: internal
        Port patch-br-int-to-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70
            Interface patch-br-int-to-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70
                type: patch
                options: {peer=patch-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70-to-br-int}
    Bridge br-ex
        datapath_type: system
        Port ens160
            Interface ens160
        Port br-ex
            Interface br-ex
                type: internal
        Port patch-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70-to-br-int
            Interface patch-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-439cc84d-fde3-42e2-90d7-0fb0e6af0d70}
    ovs_version: "2.14.0"

[Chris L (onyx4)]

I did more research on this issue... I saw that the default private network created by the Microstack init had a MTU of 1442 instead of 1500. I assume that this is due to the overhead of the Geneve encapsulation between the compute nodes.

Is it possible due to OVN (in a 2-node cluster) that any of the 2 nodes could be the network gateway? Meaning traffic destined for the external network (type flat) with a floating IP, may have to cross to the other node first through the Geneve tunnel in order to exit out on the physical interface?

Is there a supported method with Microstack to be able to support the full 1500 MTU for virtual machines by having a private network interface with jumbo frames? I see 2 possible solutions, decrease the private network MTU or increase the physical MTU, but I think this would require inserting some parameters in the neutron config files to recognize the higher MTU.