[22.03] ICMP to network forward listen address not forwarded for instance colocated with chassis bound to chassisredirect port

[Francesco De Simone]

Hi,
I am currently facing a bug during a microcloud deployment.
If I bootstrap three nodes and establish an uplink connection, then try to configure network forward ip, one of the machines becomes unreachable. This issue does not occur with the latest/edge version of microovn

snap list:
```
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical✓ base
core22 20240111 1122 latest/stable canonical✓ base
lxd 5.20-f3dd836 27049 latest/stable canonical✓ in-cohort
microceph 0+git.4a608fc 793 quincy/stable canonical✓ in-cohort
microcloud 1.1-04a1c49 734 latest/stable canonical✓ in-cohort
microovn 22.03.3+snap0e23a0e4f5 395 22.03/stable canonical✓ in-cohort
snapd 2.61.2 21184 latest/stable canonical✓ snapd
```
lxc network show default:
```
config:
  bridge.mtu: "1442"
  ipv4.address: 10.99.49.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:42b0:f235:53cb::1/64
  ipv6.nat: "true"
  network: UPLINK
  volatile.network.ipv4.address: 10.205.17.135
description: ""
name: default
type: ovn
used_by:
- /1.0/instances/u1
- /1.0/instances/u2
- /1.0/instances/u3
- /1.0/profiles/default
managed: true
status: Created
locations:
- microcloudp01
- microcloudp03
- microcloudp02
```
lxc network show UPLINK:
```
config:
  ipv4.gateway: 10.205.17.129/25
  ipv4.ovn.ranges: 10.205.17.135-10.205.17.199
  ipv4.routes: 10.205.17.200/29
  volatile.last_state.created: "false"
description: ""
name: UPLINK
type: physical
used_by:
- /1.0/networks/default
managed: true
status: Created
locations:
- microcloudp01
- microcloudp03
- microcloudp02
```
how to reproduce:
add a forward ip to a container:
```
lxc network forward create default 10.205.17.201 target_address=10.99.49.4
```
Move the container around the cluster and test its reachability until one node is not reachable:
```
ping 10.205.17.201
PING 10.205.17.201 (10.205.17.201) 56(84) bytes of data.
64 bytes from 10.205.17.201: icmp_seq=1 ttl=63 time=4.12 ms
64 bytes from 10.205.17.201: icmp_se: 3 time=1.60 ms
--- 10.205.17.201 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
$ lxc stop u1
$ lxc move u1 --target microcloudp02
$ lxc start u1
$ ping 10.205.17.201

PING 10.205.17.201 (10.205.17.201) 56(84) bytes of data.

--- 10.205.17.201 ping statistics ---

4 packets transmitted, 0 received, 100% packet loss, time 3076ms
```

UPDATE:

adding some additional logs: https://pastebin.canonical.com/p/fr6mNG6gDz/

[Francesco De Simone (fdesi)]

subscribed ~field-high

[Frode Nordahl (fnordahl)]

Beginning with commit [0], ICMP to network forward listen address is not forwarded for instance colocated with chassis bound to chassisredirect port.

The issue is fixed in commit [1] which was backported down to branch-23.03.

It is unfortunately not feasible to directly backport it to the 22.x branches as it has many dependencies on commits that refactors parts of the northd code base.

Next action is for us to seek upstream advice if there is some path to a 22.x specific fix for this condition.

I would also like to mention that the 24.03 release which fixes this will become stable within a month or two.

0: https://github.com/ovn-org/ovn/commit/3f360a49058c62ddb12258d39427f7d8b815d09a
1: https://github.com/ovn-org/ovn/commit/ce46a1bacf69140e64689a066a507471ba72d80f

[Frode Nordahl (fnordahl)]

Upstream concurs that fixing this for 22.x is non-trivial and suggest upgrade:
https://mail.openvswitch.org/pipermail/ovs-discuss/2024-April/053074.html

[Edward Hope-Morley (hopem)]

@fnordahl what upgrade options exist currently for microovn? iiuc only 22.03/stable is GA:

channels:
  22.03/stable: 22.03.3+snap0e23a0e4f5 2024-03-13 (395) 23MB -
  22.03/candidate: ↑
  22.03/beta: ↑
  22.03/edge: 22.03.3+snap0e23a0e4f5 2024-03-13 (395) 23MB -
  latest/stable: –
  latest/candidate: –
  latest/beta: –
  latest/edge: 24.03.1+snap707732e1c2 2024-04-03 (403) 24MB -