OVS/CT kernel stack trace during boot on Jammy OS

Bug #1977847 reported by Itai Levy
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Committed
Undecided
Unassigned
openvswitch (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

Charmed Openstack deployment (Yoga) with Jammy series
(For testing OVN Hardware Offload with Connection Tracking)

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"

root@node3:/home/ubuntu# uname -a
Linux node3 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@node3:/home/ubuntu# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.17.0
DB Schema 8.3.0

I see the following trace on dmesg every time the server boots up (full trace in the attached file):

[Mon Jun 6 10:51:09 2022] RIP: 0010:__ovs_ct_lookup+0x36c/0x3e0 [openvswitch]

[Mon Jun 6 10:51:09 2022] Call Trace:
[Mon Jun 6 10:51:09 2022] <TASK>
[Mon Jun 6 10:51:09 2022] ovs_ct_execute+0x3a2/0x490 [openvswitch]
[Mon Jun 6 10:51:09 2022] do_execute_actions+0xbb/0xa90 [openvswitch]
[Mon Jun 6 10:51:09 2022] ? __ovs_nla_copy_actions+0x5a0/0x8a0 [openvswitch]
[Mon Jun 6 10:51:09 2022] ? __kmalloc+0x179/0x330
[Mon Jun 6 10:51:09 2022] ovs_execute_actions+0x4c/0x110 [openvswitch]
[Mon Jun 6 10:51:09 2022] ovs_packet_cmd_execute+0x280/0x300 [openvswitch]
[Mon Jun 6 10:51:09 2022] genl_family_rcv_msg_doit+0xe7/0x150
[Mon Jun 6 10:51:09 2022] genl_rcv_msg+0xe2/0x1e0
[Mon Jun 6 10:51:09 2022] ? ovs_vport_cmd_del+0x200/0x200 [openvswitch]

I see the same trace when deploying OpenStack with Focal series and HWE-Edge kernel...

Revision history for this message
Itai Levy (etlvnvda) wrote :
Itai Levy (etlvnvda)
affects: ubuntu → openvswitch (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed
Revision history for this message
Shane Peters (shaner) wrote (last edit ):

Has anybody found a workaround for this by chance? I'm getting the same stack trace on 5.15.0-35-generic and 2.17.0-0ubuntu1

Revision history for this message
dann frazier (dannf) wrote :

This appears to now be addressed in stable:
  https://<email address hidden>/msg65275.html
And that fix is currently queued in jammy/master-next:
  https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=master-next&id=352a39407e5bab5a5965e5bcb7abf1f3e1e34f7f

So there should be a fixed kernel in jammy-proposed next week, per the SRU cycle schedule:
  https://kernel.ubuntu.com/

If someone who can reproduce this can do a local build of the current master-next branch and verify that the problem is indeed solved, that'd be keen :)

Changed in linux (Ubuntu):
status: New → Confirmed
Changed in openvswitch (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Mark Beierl (mbeierl) wrote :

I just tried jammy/master-next and got linux-image-unsigned-5.15.0-40-generic_5.15.0-40.43_amd64.deb, which is the same as what is in jammy-proposed now. Current uname:

Linux mark-t7600 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And the stack trace is still present :(

Revision history for this message
Mark Beierl (mbeierl) wrote :

Update to jammy-proposed this week:

Linux mark-t7600 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And there is no stack trace present, therefore I think this bug is resolved.

dann frazier (dannf)
Changed in linux (Ubuntu):
status: Confirmed → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.