OVS/CT kernel stack trace during boot on Jammy OS

[Itai Levy]

Charmed Openstack deployment (Yoga) with Jammy series
(For testing OVN Hardware Offload with Connection Tracking)

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"

root@node3:/home/ubuntu# uname -a
Linux node3 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@node3:/home/ubuntu# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.17.0
DB Schema 8.3.0

I see the following trace on dmesg every time the server boots up (full trace in the attached file):

[Mon Jun 6 10:51:09 2022] RIP: 0010:__ovs_ct_lookup+0x36c/0x3e0 [openvswitch]

[Mon Jun 6 10:51:09 2022] Call Trace:
[Mon Jun 6 10:51:09 2022] <TASK>
[Mon Jun 6 10:51:09 2022] ovs_ct_execute+0x3a2/0x490 [openvswitch]
[Mon Jun 6 10:51:09 2022] do_execute_actions+0xbb/0xa90 [openvswitch]
[Mon Jun 6 10:51:09 2022] ? __ovs_nla_copy_actions+0x5a0/0x8a0 [openvswitch]
[Mon Jun 6 10:51:09 2022] ? __kmalloc+0x179/0x330
[Mon Jun 6 10:51:09 2022] ovs_execute_actions+0x4c/0x110 [openvswitch]
[Mon Jun 6 10:51:09 2022] ovs_packet_cmd_execute+0x280/0x300 [openvswitch]
[Mon Jun 6 10:51:09 2022] genl_family_rcv_msg_doit+0xe7/0x150
[Mon Jun 6 10:51:09 2022] genl_rcv_msg+0xe2/0x1e0
[Mon Jun 6 10:51:09 2022] ? ovs_vport_cmd_del+0x200/0x200 [openvswitch]

I see the same trace when deploying OpenStack with Focal series and HWE-Edge kernel...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Shane Peters (shaner)]

Has anybody found a workaround for this by chance? I'm getting the same stack trace on 5.15.0-35-generic and 2.17.0-0ubuntu1

[dann frazier (dannf)]

This appears to now be addressed in stable:
  https://<email address hidden>/msg65275.html
And that fix is currently queued in jammy/master-next:
  https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=master-next&id=352a39407e5bab5a5965e5bcb7abf1f3e1e34f7f

So there should be a fixed kernel in jammy-proposed next week, per the SRU cycle schedule:
  https://kernel.ubuntu.com/

If someone who can reproduce this can do a local build of the current master-next branch and verify that the problem is indeed solved, that'd be keen :)

[Mark Beierl (mbeierl)]

I just tried jammy/master-next and got linux-image-unsigned-5.15.0-40-generic_5.15.0-40.43_amd64.deb, which is the same as what is in jammy-proposed now. Current uname:

Linux mark-t7600 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And the stack trace is still present :(

[Mark Beierl (mbeierl)]

Update to jammy-proposed this week:

Linux mark-t7600 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And there is no stack trace present, therefore I think this bug is resolved.