Hairpin traffic does not work with centralized NAT gw

Bug #1967856 reported by Frode Nordahl
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Fix Released
Medium
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Released
Medium
Unassigned
Kinetic
Invalid
Undecided
Unassigned
openvswitch (Ubuntu)
Invalid
High
Unassigned
Focal
Invalid
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Kinetic
Invalid
High
Unassigned
ovn (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Invalid
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Kinetic
Invalid
Undecided
Unassigned

Bug Description

[Impact]
Users of Open vSwitch on Focal will not be able to upgrade to v2.16.0 or newer until this long standing kernel bug has been fixed.

Users of Open vSwitch on Jammy will be affected by this bug and
have no user space fix available. This bug currently blocks the
OpenStack Engineering team's charm product gate.

[Test Plan]
Execute the OVN system testsuite utilizing the kernel data path with the test synthesis patch in comment #7 applied.

In addition to that validating that the OpenStack charm test gate is unblocked would be valuable.

[Regression Potential]
The regression potential can be considered as low because:
- The calls added in the openvswitch kernel datapath code would
  prior to Open vSwitch 2.16.0 have been initiated from the
  userspace code and by chance concealed this bug.
- After an optimization done in 2.16.0 the kernel bug was
  revealed and these calls now must be made from the kernel
  datapath to retain functionality in use in the wild.

[Original Bug Description]
If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2.

On instance on hv2 hairpin traffic works for the first session, but not for the next:

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7

We made an attempt at using OVN built with [0], but that did unfortunately not help.

If we however revert [1] it works again:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1

$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms

--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms

$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1

0: https://patchwork<email address hidden>/
1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq
 crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 virtio_gpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-37-generic N/A
 linux-backports-modules-5.15.0-37-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.board.name: LXD
dmi.board.vendor: Canonical Ltd.
dmi.board.version: pc-q35-7.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-7.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-7.0
dmi.sys.vendor: QEMU

Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Numan Siddique (numansiddique) wrote :

Is it possible to attach the OVN dbs ?

I'm not able to reproduce it locally. For me a different zone for snat is used on the gateway chassis for the hairpin traffic.

Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Sure thing!

In this DB the active gateway chassis is `deep-ferret.maas` and the instance on `comic-perch.maas` is unable to have two ping sessions to itself using non-distributed FIP 10.78.95.196.

Revision history for this message
Numan Siddique (numansiddique) wrote :
Download full text (4.8 KiB)

It works fine for me

---------------------

[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.18 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.651 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.102 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.141 ms
^C
--- 10.78.95.196 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3044ms
rtt min/avg/max/mdev = 0.102/0.518/1.179/0.438 ms
[root@ovn-chassis-1 data]#
[root@ovn-chassis-1 data]#
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.113 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.339 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.242 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.110 ms
64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.251 ms
64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.213 ms
64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms
64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=9 ttl=62 time=0.259 ms
64 bytes from 10.78.95.196: icmp_seq=10 ttl=62 time=0.257 ms
64 bytes from 10.78.95.196: icmp_seq=11 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=12 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=13 ttl=62 time=0.311 ms
64 bytes from 10.78.95.196: icmp_seq=14 ttl=62 time=0.257 ms
64 bytes from 10.78.95.196: icmp_seq=15 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=16 ttl=62 time=0.253 ms
64 bytes from 10.78.95.196: icmp_seq=17 ttl=62 time=0.249 ms
64 bytes from 10.78.95.196: icmp_seq=18 ttl=62 time=0.286 ms
64 bytes from 10.78.95.196: icmp_seq=19 ttl=62 time=0.264 ms
64 bytes from 10.78.95.196: icmp_seq=20 ttl=62 time=0.252 ms
64 bytes from 10.78.95.196: icmp_seq=21 ttl=62 time=0.239 ms
^C
--- 10.78.95.196 ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20515ms
rtt min/avg/max/mdev = 0.110/0.247/0.339/0.050 ms
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.816 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.258 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.265 ms
64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.269 ms
64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.256 ms
64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.273 ms
64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms
64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.239 ms
^C
--- 10.78.95.196 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7165ms
rtt min/avg/max/mdev = 0.239/0.329/0.816/0.184 ms
[root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196
PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data.
64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.41 ms
64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=2.10 ms
64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.27...

Read more...

Revision history for this message
Frode Nordahl (fnordahl) wrote (last edit ):

Updated OVN to main and it unfortunately made no difference.

The combination of stateless on the NAT rule and the allow-related ACLs does indeed look strange, but this is how OpenStack sets it up. Have not looked into whether that makes sense or not yet.

To ensure we're looking at the same thing I made this modification to the `DNAT LR hairpin IPv4` system test [2]

And executed it like this:

    sudo make check-kernel TESTSUITEFLAGS="337"

It fails consistently here. If I either revert [1] or remove the check for the second ping from the test it succeeds.

2: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch

Revision history for this message
Frode Nordahl (fnordahl) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :

The current line of thought is that the change in OVN has uncovered a conntrack related bug in either OVS, the OVS kernel datapath or kernel CT in general ref [3].

3: https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393426.html

Revision history for this message
Frode Nordahl (fnordahl) wrote :

A update on some findings.

If we either revert OVS commit [4], OR change a open vswitch kernel data path function [5] to always return 'false' (credits to Numan), the problem goes away.

This also appears to be a root of a different issue previously reported to the ovs-discuss list [6].

4: https://github.com/openvswitch/ovs/commit/355fef6f2
5: https://elixir.bootlin.com/linux/latest/source/net/openvswitch/conntrack.c#L683
6: https://mail.openvswitch.org/pipermail/ovs-discuss/2022-March/051771.html

Revision history for this message
Frode Nordahl (fnordahl) wrote :

A possible fix is being discussed in [7].

7: https://mail.openvswitch.org/pipermail/ovs-dev/2022-May/393981.html

Frode Nordahl (fnordahl)
Changed in openvswitch (Ubuntu):
status: New → Triaged
importance: Undecided → High
Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
status: Triaged → Invalid
Frode Nordahl (fnordahl)
Changed in ovn (Ubuntu):
importance: High → Undecided
Frode Nordahl (fnordahl)
Changed in openvswitch (Ubuntu):
status: Triaged → Invalid
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1967856

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Frode Nordahl (fnordahl) wrote :

This issue has been fixed by a patch to the openvswitch datapath code in the kernel [0].

The patched kernel would be required when used in conjunction with OVN 21.12 or newer, which translates to Focal (yoga UCA), Jammy and Kinetic.

0: https://<email address hidden>/T/#u

Changed in ovn (Ubuntu Jammy):
status: New → Invalid
Changed in ovn (Ubuntu Focal):
status: New → Invalid
Changed in openvswitch (Ubuntu Jammy):
status: New → Invalid
Changed in openvswitch (Ubuntu Focal):
status: New → Invalid
Revision history for this message
Frode Nordahl (fnordahl) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected jammy uec-images
description: updated
Revision history for this message
Frode Nordahl (fnordahl) wrote : Lspci.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : Lspci-vt.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : Lsusb-v.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : ProcModules.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : UdevDb.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : WifiSyslog.txt

apport information

Revision history for this message
Frode Nordahl (fnordahl) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu Kinetic):
status: Incomplete → Confirmed
Revision history for this message
Frode Nordahl (fnordahl) wrote :

Patches have found their way into the various stable kernels too:
https://<email address hidden>/
https://<email address hidden>/
https://<email address hidden>/T/#u

Frode Nordahl (fnordahl)
Changed in linux (Ubuntu Jammy):
status: New → Confirmed
Changed in linux (Ubuntu Focal):
status: New → Confirmed
Frode Nordahl (fnordahl)
Changed in linux (Ubuntu Impish):
status: New → Confirmed
Stefan Bader (smb)
Changed in linux (Ubuntu Jammy):
importance: Undecided → Medium
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Changed in linux (Ubuntu Impish):
status: Confirmed → Won't Fix
Changed in linux (Ubuntu Kinetic):
status: Confirmed → Invalid
Stefan Bader (smb)
Changed in linux (Ubuntu Jammy):
status: Confirmed → Fix Committed
Changed in linux (Ubuntu Focal):
status: Confirmed → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-123.139 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Download full text (3.5 KiB)

Control test:

ubuntu@actual-beagle:~/src/ovn$ uname -a
Linux actual-beagle 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@actual-beagle:~/src/ovn$ git -C ./ovs status
HEAD detached at 6f24c2bc7
nothing to commit, working tree clean
ubuntu@actual-beagle:~/src/ovn$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
ubuntu@actual-beagle:~/src/ovn$ wget -qO - https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch|patch -p1
patching file tests/system-ovn.at
Hunk #1 succeeded at 6699 (offset 15 lines).
Hunk #2 succeeded at 6724 (offset 15 lines).
Hunk #3 succeeded at 6750 (offset 15 lines).
ubuntu@actual-beagle:~/src/ovn$ make check-kernel SUDO=sudo TESTSUITEFLAGS=169
make all-am
make[1]: Entering directory '/home/ubuntu/src/ovn'
/bin/bash /home/ubuntu/src/ovn/build-aux/missing autom4te --language=autotest -I '.' -o tests/system-kmod-testsuite.tmp tests/system-kmod-testsuite.at
mv tests/system-kmod-testsuite.tmp tests/system-kmod-testsuite
/bin/bash /home/ubuntu/src/ovn/build-aux/missing autom4te --language=autotest -I '.' -o tests/system-userspace-testsuite.tmp tests/system-userspace-testsuite.at
mv tests/system-userspace-testsuite.tmp tests/system-userspace-testsuite
make[1]: Leaving directory '/home/ubuntu/src/ovn'
set /bin/bash './tests/system-kmod-testsuite' -C tests AUTOTEST_PATH='/home/ubuntu/src/ovn/ovs/utilities:/home/ubuntu/src/ovn/ovs/vswitchd:/home/ubuntu/src/ovn/ovs/ovsdb:/home/ubuntu/src/ovn/ovs/vtep:tests:::controller-vtep:northd:utilities:controller:ic'; \
sudo "$@" 169 -j1 || (test X'' = Xyes && sudo "$@" --recheck)
## ------------------------ ##
## ovn 22.06.90 test suite. ##
## ------------------------ ##
169: DNAT LR hairpin IPv4 -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes FAILED (system-ovn.at:6661)

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: 1 test was run,
1 failed unexpectedly.

Verification:

ubuntu@actual-beagle:~/src/ovn$ uname -a
Linux actual-beagle 5.4.0-123-generic #139-Ubuntu SMP Mon Jul 11 16:02:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@actual-beagle:~/src/ovn$ git -C ovs status
HEAD detached at 6f24c2bc7
nothing to commit, working tree clean
ubuntu@actual-beagle:~/src/ovn$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
 modified: tests/system-ovn.at

Untracked files:
  (use "git add <file>..." to include in what will be committed)
 tests/system-ovn.at.orig

no changes added to commit (use "git add" and/or "git commit -a")
ubuntu@actual-beagle:~/src/ovn$ make check-kernel SUDO=sudo TESTSUITEFLAGS=169
make all-am
make[1]: Entering directory '/home/ubuntu/src/ovn'
make[1]: Leaving directory '/home/ubuntu/src/ovn'
set /bin/bash './tests/system-kmod-testsuite' -C tests AUTOTEST_PATH='/home/ubuntu/src/ovn/ovs/utilities:/home/ubuntu/src/ovn/ovs/vswitchd:/home/ubuntu/src/ovn/ovs/ovsdb:/home/ubuntu/src/ovn/ovs/vtep:test...

Read more...

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.15.0-43.46 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-jammy
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Download full text (3.5 KiB)

Control test:

ubuntu@game-cicada:~/src/ovn$ uname -a
Linux game-cicada 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@game-cicada:~/src/ovn$ git -C ./ovs status
HEAD detached at 6f24c2bc7
nothing to commit, working tree clean
ubuntu@game-cicada:~/src/ovn$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
ubuntu@game-cicada:~/src/ovn$ wget -qO - https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch|patch -p1
patching file tests/system-ovn.at
Hunk #1 succeeded at 6699 (offset 15 lines).
Hunk #2 succeeded at 6724 (offset 15 lines).
Hunk #3 succeeded at 6750 (offset 15 lines).
ubuntu@game-cicada:~/src/ovn$ make check-kernel SUDO=sudo TESTSUITEFLAGS=169
make all-am
make[1]: Entering directory '/home/ubuntu/src/ovn'
/bin/bash '/home/ubuntu/src/ovn/build-aux/missing' autom4te --language=autotest -I '.' -o tests/system-kmod-testsuite.tmp tests/system-kmod-testsuite.at
mv tests/system-kmod-testsuite.tmp tests/system-kmod-testsuite
/bin/bash '/home/ubuntu/src/ovn/build-aux/missing' autom4te --language=autotest -I '.' -o tests/system-userspace-testsuite.tmp tests/system-userspace-testsuite.at
mv tests/system-userspace-testsuite.tmp tests/system-userspace-testsuite
make[1]: Leaving directory '/home/ubuntu/src/ovn'
set /bin/bash './tests/system-kmod-testsuite' -C tests AUTOTEST_PATH='/home/ubuntu/src/ovn/ovs/utilities:/home/ubuntu/src/ovn/ovs/vswitchd:/home/ubuntu/src/ovn/ovs/ovsdb:/home/ubuntu/src/ovn/ovs/vtep:tests:::controller-vtep:northd:utilities:controller:ic'; \
sudo "$@" 169 -j1 || (test X'' = Xyes && sudo "$@" --recheck)
## ------------------------ ##
## ovn 22.06.90 test suite. ##
## ------------------------ ##
169: DNAT LR hairpin IPv4 -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes FAILED (system-ovn.at:6661)

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: 1 test was run,
1 failed unexpectedly.

Verification:

ubuntu@game-cicada:~/src/ovn$ uname -a
Linux game-cicada 5.15.0-43-generic #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@game-cicada:~/src/ovn$ git -C ./ovs status
HEAD detached at 6f24c2bc7
nothing to commit, working tree clean
ubuntu@game-cicada:~/src/ovn$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
 modified: tests/system-ovn.at

Untracked files:
  (use "git add <file>..." to include in what will be committed)
 tests/system-ovn.at.orig

no changes added to commit (use "git add" and/or "git commit -a")
ubuntu@game-cicada:~/src/ovn$ make check-kernel SUDO=sudo TESTSUITEFLAGS=169
make all-am
make[1]: Entering directory '/home/ubuntu/src/ovn'
make[1]: Leaving directory '/home/ubuntu/src/ovn'
set /bin/bash './tests/system-kmod-testsuite' -C tests AUTOTEST_PATH='/home/ubuntu/src/ovn/ovs/utilities:/home/ubuntu/src/ovn/ovs/vswitchd:/home/ubuntu/src/ovn/ovs/ovsdb:/home/ubuntu/src/ovn/ovs/vtep:tests:::controller-vtep...

Read more...

Frode Nordahl (fnordahl)
description: updated
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Download full text (6.4 KiB)

In addition to the verification provided in comment #26 and #28 we've taken the neutron-api-plugin-ovn e2e charm gate with an added hairpin test for a spin for both Focal and Jammy where we found the control to fail as expected and successful test with the updated kernel.

$ juju run --application nova-compute 'uname -a'
- Stdout: |
    Linux civil-ibex 5.4.0-123-generic #139-Ubuntu SMP Mon Jul 11 16:02:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  UnitId: nova-compute/0
- Stdout: |
    Linux pumped-pika 5.4.0-123-generic #139-Ubuntu SMP Mon Jul 11 16:02:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  UnitId: nova-compute/1
- Stdout: |
    Linux moved-toucan 5.4.0-123-generic #139-Ubuntu SMP Mon Jul 11 16:02:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  UnitId: nova-compute/2
...
2022-07-19 16:25:09 [INFO] XXX HAIRPIN
2022-07-19 16:25:09 [INFO] Attempting to ssh to instance-1(10.78.95.25)
2022-07-19 16:25:09 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:09 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:09 [INFO] Running ping -M do -s 1414 -c 1 10.78.95.25 on instance-1
2022-07-19 16:25:10 [INFO] Attempting to ssh to instance-1(10.78.95.25)
2022-07-19 16:25:10 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:10 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:10 [INFO] Running ping -M do -s 1414 -c 1 10.78.95.25 on instance-1
2022-07-19 16:25:10 [INFO] Attempting to ssh to instance-1(10.78.95.25)
2022-07-19 16:25:10 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:10 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:10 [INFO] Running ping -M do -s 1414 -c 1 10.78.95.25 on instance-1
2022-07-19 16:25:11 [INFO] Attempting to ssh to instance(10.78.95.71)
2022-07-19 16:25:11 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:11 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:11 [INFO] Running ping -c 1 192.168.0.1 on instance
2022-07-19 16:25:11 [INFO] Attempting to ssh to instance(10.78.95.71)
2022-07-19 16:25:11 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:11 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:11 [INFO] Running ping -M do -s 1414 -c 1 192.168.0.1 on instance
2022-07-19 16:25:12 [INFO] Attempting to ssh to instance(10.78.95.25)
2022-07-19 16:25:12 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:12 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:12 [INFO] Running ping -c 1 192.168.0.1 on instance
2022-07-19 16:25:12 [INFO] Attempting to ssh to instance(10.78.95.25)
2022-07-19 16:25:12 [INFO] Connected (version 2.0, client OpenSSH_7.6p1)
2022-07-19 16:25:12 [INFO] Authentication (publickey) successful!
2022-07-19 16:25:12 [INFO] Running ping -M do -s 1414 -c 1 192.168.0.1 on instance
2022-07-19 16:25:13 [INFO] Running resource cleanup
2022-07-19 16:25:13 [INFO] Removing resources created by test (zaza-neutrontests*)
2022-07-19 16:25:21 [INFO] ok
2022-07-19 16:25:21 [INFO] ----------------------------------------------------------------------
2022-07-19 16:25:21 [INFO] Ran 1 test in 48.934s
2022-07-19 16:25:21 [INFO] OK
2022-...

Read more...

tags: added: verification-done-focal verification-done-jammy
removed: verification-needed-focal verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.6 KiB)

This bug was fixed in the package linux - 5.15.0-43.46

---------------
linux (5.15.0-43.46) jammy; urgency=medium

  * jammy/linux: 5.15.0-43.46 -proposed tracker (LP: #1981243)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.07.11)

  * nbd: requests can become stuck when disconnecting from server with qemu-nbd
    (LP: #1896350)
    - nbd: don't handle response without a corresponding request message
    - nbd: make sure request completion won't concurrent
    - nbd: don't clear 'NBD_CMD_INFLIGHT' flag if request is not completed
    - nbd: fix io hung while disconnecting device

  * Ubuntu 22.04 and 20.04 DPC Fixes for Failure Cases of DownPort Containment
    events (LP: #1965241)
    - PCI/portdrv: Rename pm_iter() to pcie_port_device_iter()
    - PCI: pciehp: Ignore Link Down/Up caused by error-induced Hot Reset
    - [Config] Enable config option CONFIG_PCIE_EDR

  * [SRU] Ubuntu 22.04 Feature Request-Add support for a NVMe-oF-TCP CDC Client
    - TP 8010 (LP: #1948626)
    - nvme: add CNTRLTYPE definitions for 'identify controller'
    - nvme: send uevent on connection up
    - nvme: expose cntrltype and dctype through sysfs

  * [UBUNTU 22.04] Kernel oops while removing device from cio_ignore list
    (LP: #1980951)
    - s390/cio: derive cdev information only for IO-subchannels

  * Jammy Charmed OpenStack deployment fails over connectivity issues when using
    converged OVS bridge for control and data planes (LP: #1978820)
    - net/mlx5e: TC NIC mode, fix tc chains miss table

  * Hairpin traffic does not work with centralized NAT gw (LP: #1967856)
    - net: openvswitch: fix misuse of the cached connection on tuple changes

  * alsa: asoc: amd: the internal mic can't be dedected on yellow carp machines
    (LP: #1980700)
    - ASoC: amd: Add driver data to acp6x machine driver
    - ASoC: amd: Add support for enabling DMIC on acp6x via _DSD

  * AMD ACP 6.x DMIC Supports (LP: #1949245)
    - ASoC: amd: add Yellow Carp ACP6x IP register header
    - ASoC: amd: add Yellow Carp ACP PCI driver
    - ASoC: amd: add acp6x init/de-init functions
    - ASoC: amd: add platform devices for acp6x pdm driver and dmic driver
    - ASoC: amd: add acp6x pdm platform driver
    - ASoC: amd: add acp6x irq handler
    - ASoC: amd: add acp6x pdm driver dma ops
    - ASoC: amd: add acp6x pci driver pm ops
    - ASoC: amd: add acp6x pdm driver pm ops
    - ASoC: amd: enable Yellow carp acp6x drivers build
    - ASoC: amd: create platform device for acp6x machine driver
    - ASoC: amd: add YC machine driver using dmic
    - ASoC: amd: enable Yellow Carp platform machine driver build
    - ASoC: amd: fix uninitialized variable in snd_acp6x_probe()
    - [Config] Enable AMD ACP 6 DMIC Support

  * [UBUNTU 20.04] Include patches to avoid self-detected stall with Secure
    Execution (LP: #1979296)
    - KVM: s390: pv: add macros for UVC CC values
    - KVM: s390: pv: avoid stalls when making pages secure

  * [22.04 FEAT] KVM: Attestation support for Secure Execution (crypto)
    (LP: #1959973)
    - drivers/s390/char: Add Ultravisor io device
    - s390/uv_uapi: depend on CONFIG_S390
    - [Co...

Read more...

Changed in linux (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.9 KiB)

This bug was fixed in the package linux - 5.4.0-124.140

---------------
linux (5.4.0-124.140) focal; urgency=medium

  * CVE-2022-2586
    - SAUCE: netfilter: nf_tables: do not allow SET_ID to refer to another table
    - SAUCE: netfilter: nf_tables: do not allow RULE_ID to refer to another chain

  * CVE-2022-2588
    - SAUCE: net_sched: cls_route: remove from list when handle is 0

  * CVE-2022-34918
    - netfilter: nf_tables: stricter validation of element data

linux (5.4.0-123.139) focal; urgency=medium

  * focal/linux: 5.4.0-123.139 -proposed tracker (LP: #1981284)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.07.11)

  * Hairpin traffic does not work with centralized NAT gw (LP: #1967856)
    - net: openvswitch: fix misuse of the cached connection on tuple changes

  * [UBUNTU 20.04] Include patches to avoid self-detected stall with Secure
    Execution (LP: #1979296)
    - KVM: s390: pv: add macros for UVC CC values
    - KVM: s390: pv: avoid stalls when making pages secure
    - KVM: s390: pv: avoid stalls for kvm_s390_pv_init_vm

  * Focal update: v5.4.195 upstream stable release (LP: #1980407)
    - batman-adv: Don't skb_split skbuffs with frag_list
    - hwmon: (tmp401) Add OF device ID table
    - mac80211: Reset MBSSID parameters upon connection
    - net: Fix features skip in for_each_netdev_feature()
    - ipv4: drop dst in multicast routing path
    - drm/nouveau: Fix a potential theorical leak in nouveau_get_backlight_name()
    - netlink: do not reset transport header in netlink_recvmsg()
    - mac80211_hwsim: call ieee80211_tx_prepare_skb under RCU protection
    - dim: initialize all struct fields
    - hwmon: (ltq-cputemp) restrict it to SOC_XWAY
    - s390/ctcm: fix variable dereferenced before check
    - s390/ctcm: fix potential memory leak
    - s390/lcs: fix variable dereferenced before check
    - net/sched: act_pedit: really ensure the skb is writable
    - net/smc: non blocking recvmsg() return -EAGAIN when no data and
      signal_pending
    - net: sfc: ef10: fix memory leak in efx_ef10_mtd_probe()
    - gfs2: Fix filesystem block deallocation for short writes
    - hwmon: (f71882fg) Fix negative temperature
    - ASoC: max98090: Reject invalid values in custom control put()
    - ASoC: max98090: Generate notifications on changes for custom control
    - ASoC: ops: Validate input values in snd_soc_put_volsw_range()
    - s390: disable -Warray-bounds
    - net: emaclite: Don't advertise 1000BASE-T and do auto negotiation
    - tcp: resalt the secret every 10 seconds
    - tty: n_gsm: fix mux activation issues in gsm_config()
    - usb: cdc-wdm: fix reading stuck on device close
    - usb: typec: tcpci: Don't skip cleanup in .remove() on error
    - USB: serial: pl2303: add device id for HP LM930 Display
    - USB: serial: qcserial: add support for Sierra Wireless EM7590
    - USB: serial: option: add Fibocom L610 modem
    - USB: serial: option: add Fibocom MA510 modem
    - slimbus: qcom: Fix IRQ check in qcom_slim_probe
    - serial: 8250_mtk: Fix UART_EFR register address
    - serial: 8250_mtk: Fix register address for XON/XOFF character
    - dr...

Read more...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Frode Nordahl (fnordahl)
Changed in openvswitch (Ubuntu Impish):
status: New → Won't Fix
Changed in ovn (Ubuntu Impish):
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.