Ubuntu
ovn package

Hairpin traffic does not work with centralized NAT gw

Bug #1967856 reported by Frode Nordahl
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Status tracked in Kinetic
Focal
Confirmed
Undecided
Unassigned
Jammy
Confirmed
Undecided
Unassigned
Kinetic
Confirmed
Undecided
Unassigned
openvswitch (Ubuntu)
Status tracked in Kinetic
Focal
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Kinetic
Invalid
High
Unassigned
ovn (Ubuntu)
Status tracked in Kinetic
Focal
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Kinetic
Invalid
Undecided
Unassigned

Bug Description

If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2.

On instance on hv2 hairpin traffic works for the first session, but not for the next:

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7

We made an attempt at using OVN built with [0], but that did unfortunately not help.

If we however revert [1] it works again:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1

0: https://patchwork<email address hidden>/
1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq
 crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 virtio_gpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-37-generic N/A
 linux-backports-modules-5.15.0-37-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.board.name: LXD
dmi.board.vendor: Canonical Ltd.
dmi.board.version: pc-q35-7.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-7.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-7.0
dmi.sys.vendor: QEMU

See original description
Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Numan Siddique (numansiddique) wrote :

Is it possible to attach the OVN dbs ?

I'm not able to reproduce it locally. For me a different zone for snat is used on the gateway chassis for the hairpin traffic.

Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Sure thing!

In this DB the active gateway chassis is `deep-ferret.maas` and the instance on `comic-perch.maas` is unable to have two ping sessions to itself using non-distributed FIP 10.78.95.196.

Revision history for this message
Numan Siddique (numansiddique) wrote :
Download full text (4.8 KiB)

It works fine for me

---------------------

[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.18 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.651 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.102 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.141 ms
^C
--- 10.78.95.196 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3044ms
rtt min/avg/max/mdev = 0.102/0.518/1.179/0.438 ms
[root@ovn-chassis-1 data]#
[root@ovn-chassis-1 data]#
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.113 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.339 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.242 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.110 ms
64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.251 ms
64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.213 ms
64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms
64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=9 ttl=62 time=0.259 ms
64 bytes from 10.78.95.196: icmp_seq=10 ttl=62 time=0.257 ms
64 bytes from 10.78.95.196: icmp_seq=11 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=12 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=13 ttl=62 time=0.311 ms
64 bytes from 10.78.95.196: icmp_seq=14 ttl=62 time=0.257 ms
64 bytes from 10.78.95.196: icmp_seq=15 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=16 ttl=62 time=0.253 ms
64 bytes from 10.78.95.196: icmp_seq=17 ttl=62 time=0.249 ms
64 bytes from 10.78.95.196: icmp_seq=18 ttl=62 time=0.286 ms
64 bytes from 10.78.95.196: icmp_seq=19 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=20 ttl=62 time=0.252 ms
64 bytes from 10.78.95.196: icmp_seq=21 ttl=62 time=0.239 ms
^C
--- 10.78.95.196 ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20515ms
rtt min/avg/max/mdev = 0.110/0.247/0.339/0.050 ms
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.816 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.265 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.269 ms
64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.256 ms
64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.273 ms
64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms
64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.239 ms
^C
--- 10.78.95.196 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7165ms
rtt min/avg/max/mdev = 0.239/0.329/0.816/0.184 ms
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.41 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=2.10 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.27...

Read more...

Revision history for this message
Frode Nordahl (fnordahl) wrote (last edit ):

Updated OVN to main and it unfortunately made no difference.

The combination of stateless on the NAT rule and the allow-related ACLs does indeed look strange, but this is how OpenStack sets it up. Have not looked into whether that makes sense or not yet.

To ensure we're looking at the same thing I made this modification to the `DNAT LR hairpin IPv4` system test [2]

And executed it like this:

    sudo make check-kernel TESTSUITEFLAGS="337"

It fails consistently here. If I either revert [1] or remove the check for the second ping from the test it succeeds.

2: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch

Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :

The current line of thought is that the change in OVN has uncovered a conntrack related bug in either OVS, the OVS kernel datapath or kernel CT in general ref [3].

3: https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393426.html

Revision history for this message
Frode Nordahl (fnordahl) wrote :

A update on some findings.

If we either revert OVS commit [4], OR change a open vswitch kernel data path function [5] to always return 'false' (credits to Numan), the problem goes away.

This also appears to be a root of a different issue previously reported to the ovs-discuss list [6].

4: https://github.com/openvswitch/ovs/commit/355fef6f2
5: https://elixir.bootlin.com/linux/latest/source/net/openvswitch/conntrack.c#L683
6: https://mail.openvswitch.org/pipermail/ovs-discuss/2022-March/051771.html

Revision history for this message
Frode Nordahl (fnordahl) wrote :

A possible fix is being discussed in [7].

7: https://mail.openvswitch.org/pipermail/ovs-dev/2022-May/393981.html

Frode Nordahl (fnordahl)
Changed in openvswitch (Ubuntu):
status: New → Triaged
importance: Undecided → High
Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
status: Triaged → Invalid
Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
importance: High → Undecided
Frode Nordahl (fnordahl)
Changed in openvswitch (Ubuntu):
status: Triaged → Invalid
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1967856

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Frode Nordahl (fnordahl) wrote :

This issue has been fixed by a patch to the openvswitch datapath code in the kernel [0].

The patched kernel would be required when used in conjunction with OVN 21.12 or newer, which translates to Focal (yoga UCA), Jammy and Kinetic.

0: https://<email address hidden>/T/#u

Changed in ovn (Ubuntu Jammy):
status: New → Invalid
Changed in ovn (Ubuntu Focal):
status: New → Invalid
Changed in openvswitch (Ubuntu Jammy):
status: New → Invalid
Changed in openvswitch (Ubuntu Focal):
status: New → Invalid
Revision history for this message
Frode Nordahl (fnordahl) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected jammy uec-images
description: updated
Revision history for this message
Frode Nordahl (fnordahl) wrote : Lspci.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : Lspci-vt.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : Lsusb-v.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcModules.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : UdevDb.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : WifiSyslog.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu Kinetic):
status: Incomplete → Confirmed
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Patches have found their way into the various stable kernels too:
https://<email address hidden>/
https://<email address hidden>/
https://<email address hidden>/T/#u

Frode Nordahl (fnordahl)
Changed in linux (Ubuntu Jammy):
status: New → Confirmed
Changed in linux (Ubuntu Focal):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers