linux: soft lockup - CPU stuck in nft_connlimit

[Piotr Roszatycki]

Kernel reports many soft lockups and after some time totally hangs up.

Before hang up the stack trace in reported in dmesg:

Oct 14 19:07:32 ubuntu kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [swapper/6:0]
Oct 14 19:07:32 ubuntu kernel: Modules linked in: nft_chain_nat_ipv4 nf_log_ipv6 cfg80211 8021q garp mrp stp llc bonding intel_rapl sch_fq_codel sb_edac x86_pkg_temp_thermal i
Oct 14 19:07:32 ubuntu kernel: i2c_algo_bit libahci lpc_ich megaraid_sas dca wmi mdio
Oct 14 19:07:32 ubuntu kernel: CPU: 6 PID: 0 Comm: swapper/6 Not tainted 5.0.0-31-generic #33-Ubuntu
Oct 14 19:07:32 ubuntu kernel: Hardware name: Dell Inc. PowerEdge R620/0VV3F2, BIOS 2.7.0 05/23/2018
Oct 14 19:07:32 ubuntu kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x136/0x1a0
Oct 14 19:07:32 ubuntu kernel: Code: 74 41 c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 40 3e 02 00 48 03 04 cd 20 98 3d b4 48 89 10 8b 42 08 85 c0 75 09 <f3> 90 8b 4
Oct 14 19:07:32 ubuntu kernel: RSP: 0018:ffff8c1b7b6c3888 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Oct 14 19:07:32 ubuntu kernel: RAX: 0000000000000000 RBX: ffff8c1aba3d4c88 RCX: 0000000000000012
Oct 14 19:07:32 ubuntu kernel: RDX: ffff8c1b7b6e3e40 RSI: 00000000001c0000 RDI: ffff8c1aba3d4c88
Oct 14 19:07:32 ubuntu kernel: RBP: ffff8c1b7b6c3888 R08: 0000000000000000 R09: 0000000000000000
Oct 14 19:07:32 ubuntu kernel: R10: 0000000000000010 R11: 0000000000000001 R12: ffffffffb47d5e40
Oct 14 19:07:32 ubuntu kernel: R13: ffff8c1abfda35e0 R14: ffff8c1abfda35ca R15: ffff8c1a9f94cb18
Oct 14 19:07:32 ubuntu kernel: FS: 0000000000000000(0000) GS:ffff8c1b7b6c0000(0000) knlGS:0000000000000000
Oct 14 19:07:32 ubuntu kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 19:07:32 ubuntu kernel: CR2: 00007ff14631e560 CR3: 00000000b860e001 CR4: 00000000001606e0
Oct 14 19:07:32 ubuntu kernel: Call Trace:
Oct 14 19:07:32 ubuntu kernel: <IRQ>
Oct 14 19:07:32 ubuntu kernel: _raw_spin_lock_bh+0x2a/0x30
Oct 14 19:07:32 ubuntu kernel: nf_conncount_add+0x24/0x50 [nf_conncount]
Oct 14 19:07:32 ubuntu kernel: nft_connlimit_eval+0x50/0xe0 [nft_connlimit]
Oct 14 19:07:32 ubuntu kernel: nft_dynset_eval+0xba/0x110 [nf_tables]
Oct 14 19:07:32 ubuntu kernel: nft_do_chain+0xea/0x420 [nf_tables]
Oct 14 19:07:32 ubuntu kernel: ? nft_do_chain+0xea/0x420 [nf_tables]
Oct 14 19:07:32 ubuntu kernel: ? fib4_rule_action+0x66/0x70
Oct 14 19:07:32 ubuntu kernel: ? fib_rules_lookup+0x150/0x1e0
Oct 14 19:07:32 ubuntu kernel: ? __fib_lookup+0x6a/0xb0
Oct 14 19:07:32 ubuntu kernel: ? ip_route_input_slow+0x438/0xbb0
Oct 14 19:07:32 ubuntu kernel: nft_do_chain_inet+0x7f/0xf0 [nf_tables]
Oct 14 19:07:32 ubuntu kernel: ? ip_route_input_rcu+0x15f/0x2b0
Oct 14 19:07:32 ubuntu kernel: nf_hook_slow+0x49/0xd0
Oct 14 19:07:32 ubuntu kernel: ip_forward+0x45b/0x480
Oct 14 19:07:32 ubuntu kernel: ? ip_defrag.cold.12+0x3e/0x3e
Oct 14 19:07:32 ubuntu kernel: ip_rcv_finish+0x84/0xa0
Oct 14 19:07:32 ubuntu kernel: ip_rcv+0xbc/0xd0
Oct 14 19:07:32 ubuntu kernel: ? ip_rcv_finish_core.isra.22+0x370/0x370
Oct 14 19:07:32 ubuntu kernel: __netif_receive_skb_one_core+0x57/0x80
Oct 14 19:07:32 ubuntu kernel: __netif_receive_skb+0x18/0x60
Oct 14 19:07:32 ubuntu kernel: netif_receive_skb_internal+0x45/0xf0
Oct 14 19:07:32 ubuntu kernel: napi_gro_receive+0xfa/0x160
Oct 14 19:07:32 ubuntu kernel: ixgbe_clean_rx_irq+0x2c8/0xb70 [ixgbe]
Oct 14 19:07:32 ubuntu kernel: ixgbe_poll+0x105/0x750 [ixgbe]
Oct 14 19:07:32 ubuntu kernel: net_rx_action+0x130/0x350
Oct 14 19:07:32 ubuntu kernel: __do_softirq+0xdc/0x2f3
Oct 14 19:07:32 ubuntu kernel: irq_exit+0xc0/0xd0
Oct 14 19:07:32 ubuntu kernel: do_IRQ+0x84/0xe0
Oct 14 19:07:32 ubuntu kernel: common_interrupt+0xf/0xf
Oct 14 19:07:32 ubuntu kernel: </IRQ>
Oct 14 19:07:32 ubuntu kernel: RIP: 0010:cpuidle_enter_state+0xbd/0x450
Oct 14 19:07:32 ubuntu kernel: Code: ff e8 a7 9f 86 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 63 03 00 00 31 ff e8 5a d0 8c ff fb 66 0f 1f 44 00 00 <45> 85 ed 0
Oct 14 19:07:32 ubuntu kernel: RSP: 0018:ffffa79780d0fe60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd8
Oct 14 19:07:32 ubuntu kernel: RAX: ffff8c1b7b6e3200 RBX: ffffffffb4753e00 RCX: 000000000000001f
Oct 14 19:07:32 ubuntu kernel: RDX: 000002fcd83bc92e RSI: 000000004b4b4b4b RDI: 0000000000000000
Oct 14 19:07:32 ubuntu kernel: RBP: ffffa79780d0fea0 R08: 0000000000000002 R09: 0000000000022ac0
Oct 14 19:07:32 ubuntu kernel: R10: 0009fb24c180afd3 R11: ffff8c1b7b6e2084 R12: ffffc7967f8c0b00
Oct 14 19:07:32 ubuntu kernel: R13: 0000000000000004 R14: ffffffffb4753f98 R15: ffffffffb4753f80
Oct 14 19:07:32 ubuntu kernel: cpuidle_enter+0x17/0x20
Oct 14 19:07:32 ubuntu kernel: call_cpuidle+0x23/0x40
Oct 14 19:07:32 ubuntu kernel: do_idle+0x23a/0x280
Oct 14 19:07:32 ubuntu kernel: cpu_startup_entry+0x1d/0x20
Oct 14 19:07:32 ubuntu kernel: start_secondary+0x1ab/0x200
Oct 14 19:07:32 ubuntu kernel: secondary_startup_64+0xa4/0xb0

Machine does nothing except forwarding traffic between intefaces using nftables and conntrack.

I believe this bug is already resolved in this commit: https://github.com/torvalds/linux/commit/34a4c95abd25ab41fb390b985a08a651b1fa0b0f