Comment 0 for bug 1885696

Revision history for this message
Antti Jaakkola (remodantti) wrote :

Ubuntu 20.04 server with latest kernel 5.4.0-39-generic panics under load when cachefilesd is used.

This bug is known and fixed in upstream kernel 5.4.44

https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.44

cachefiles: Fix race between read_waiter and read_copier involving op->to_do

[ Upstream commit 7bb0c5338436dae953622470d52689265867f032 ]

https://github.com/torvalds/linux/commit/7bb0c5338436dae953622470d52689265867f032#diff-14a48d14f7cd98c97d57211cfd758797

package: linux-image-5.4.0-39-generic, linux-image-generic

Dmesg output:

[450023.072377] FS-Cache:
[450023.075150] FS-Cache: Assertion failed
[450023.079496] FS-Cache: 4 == 5 is false
[450023.083749] kernel BUG at fs/fscache/operation.c:69!
[450023.089468] invalid opcode: 0000 [#1] SMP PTI
[450023.094500] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 5.4.0-37-generic #41-Ubuntu
[450023.104057] Hardware name: Dell Inc. PowerEdge R430/0CN7X8, BIOS 2.11.0 12/23/2019
[450023.112660] RIP: 0010:fscache_enqueue_operation.cold+0xb7/0xb9 [fscache]
[450023.120279] Code: a8 96 c0 e8 3f dc 1a e1 48 c7 c7 14 a8 96 c0 e8 33 dc 1a e1 41 8b 74 24 40 ba 05 00 00 00 48 c7 c7 28 98 96 c0 e8 1d dc 1a e1 <0f> 0b 48 c7 c7 06 a8 96 c0 e8 0f dc 1a e1 48 c7 c7 14 a8 96 c0 e8
[450023.141515] RSP: 0018:ffffbaca8cb88b98 EFLAGS: 00010046
[450023.147490] RAX: 0000000000000019 RBX: ffff98535b7f6540 RCX: 0000000000000006
[450023.156712] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff9872bf3978c0
[450023.165774] RBP: ffffbaca8cb88bb0 R08: 000000000000337d R09: ffffbaca8f984020
[450023.175116] R10: ffff9852b7ebc2a0 R11: 0000000000000001 R12: ffff9872b870f740
[450023.184384] R13: ffff9838bd287580 R14: 0000000000000000 R15: ffffbaca8cb88c48
[450023.193659] FS: 0000000000000000(0000) GS:ffff9872bf380000(0000) knlGS:0000000000000000
[450023.204016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[450023.211346] CR2: 000055eb5afaf2d8 CR3: 000000359180a004 CR4: 00000000003626e0
[450023.220602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[450023.229668] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[450023.238621] Call Trace:
[450023.242266] <IRQ>
[450023.245426] cachefiles_read_waiter+0xe5/0x100 [cachefiles]
[450023.252592] __wake_up_common+0x7e/0x140
[450023.257909] __wake_up_locked_key_bookmark+0x1b/0x20
[450023.264404] wake_up_page_bit+0xa7/0x100
[450023.269733] unlock_page+0x25/0x30
[450023.274491] __read_end_io+0xb6/0x110
[450023.279530] mpage_end_io+0x2f/0x40
[450023.284389] bio_endio+0xfb/0x160
[450023.289052] dec_pending+0x11c/0x210
[450023.294006] clone_endio+0x77/0x170
[450023.298874] bio_endio+0xfb/0x160
[450023.303540] blk_update_request+0x18a/0x2e0
[450023.309201] scsi_end_request+0x30/0x160
[450023.314571] scsi_io_completion+0x7c/0x540
[450023.320129] ? __update_load_avg_cfs_rq+0x212/0x2f0
[450023.326580] scsi_finish_command+0xe7/0x120
[450023.332259] scsi_softirq_done+0x14a/0x170
[450023.337840] blk_done_softirq+0x92/0xc0
[450023.343141] __do_softirq+0xe1/0x2d6
[450023.348145] irq_exit+0xae/0xb0
[450023.352670] smp_call_function_single_interrupt+0x48/0xd0
[450023.359735] call_function_single_interrupt+0xf/0x20
[450023.366323] </IRQ>
[450023.369712] RIP: 0010:cpuidle_enter_state+0xc5/0x450
[450023.376326] Code: ff e8 5f 0c 81 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 65 03 00 00 31 ff e8 72 6e 87 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8f 02 00 00 49 63 cd 4c 8b 7d d0 4c 2b 7d c8 48 8d
[450023.399419] RSP: 0018:ffffbaca8017be38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[450023.409020] RAX: ffff9872bf3aad00 RBX: ffffffffa3159c00 RCX: 000000000000001f
[450023.418148] RDX: 0000000000000000 RSI: 000000003a3fe5c1 RDI: 0000000000000000
[450023.427270] RBP: ffffbaca8017be78 R08: 0001994b295dad0d R09: 0000000000000001
[450023.436383] R10: ffff9872bf3a9a00 R11: ffff9872bf3a99e0 R12: ffffdaca7f583400
[450023.445496] R13: 0000000000000003 R14: 0000000000000003 R15: ffffdaca7f583400
[450023.454584] ? cpuidle_enter_state+0xa1/0x450
[450023.460544] cpuidle_enter+0x2e/0x40
[450023.465593] call_cpuidle+0x23/0x40
[450023.470525] do_idle+0x1dd/0x270
[450023.475147] cpu_startup_entry+0x20/0x30
[450023.480547] start_secondary+0x167/0x1c0
[450023.485934] secondary_startup_64+0xa4/0xb0
[450023.491620] Modules linked in: vhost_net vhost tap cpuid xt_state br_netfilter xt_physdev ip_set xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter bridge cachefiles rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache 8021q garp mrp stp llc bonding nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ipmi_ssif intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel dcdbas kvm intel_cstate intel_rapl_perf ipmi_si mei_me ipmi_devintf mei ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel mgag200 drm_vram_helper
[450023.491667] i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect mxm_wmi aesni_intel crypto_simd sysimgblt cryptd fb_sys_fops glue_helper bnx2x drm ahci lpc_ich mdio libahci libcrc32c wmi
[450023.619482] ---[ end trace 250fd62ab6d68a23 ]---
[450023.705655] RIP: 0010:fscache_enqueue_operation.cold+0xb7/0xb9 [fscache]
[450023.714191] Code: a8 96 c0 e8 3f dc 1a e1 48 c7 c7 14 a8 96 c0 e8 33 dc 1a e1 41 8b 74 24 40 ba 05 00 00 00 48 c7 c7 28 98 96 c0 e8 1d dc 1a e1 <0f> 0b 48 c7 c7 06 a8 96 c0 e8 0f dc 1a e1 48 c7 c7 14 a8 96 c0 e8
[450023.737252] RSP: 0018:ffffbaca8cb88b98 EFLAGS: 00010046
[450023.744197] RAX: 0000000000000019 RBX: ffff98535b7f6540 RCX: 0000000000000006
[450023.753289] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff9872bf3978c0
[450023.762403] RBP: ffffbaca8cb88bb0 R08: 000000000000337d R09: ffffbaca8f984020
[450023.771518] R10: ffff9852b7ebc2a0 R11: 0000000000000001 R12: ffff9872b870f740
[450023.780643] R13: ffff9838bd287580 R14: 0000000000000000 R15: ffffbaca8cb88c48
[450023.789760] FS: 0000000000000000(0000) GS:ffff9872bf380000(0000) knlGS:0000000000000000
[450023.799961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[450023.807545] CR2: 000055eb5afaf2d8 CR3: 000000359180a004 CR4: 00000000003626e0
[450023.816699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[450023.825847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[450023.834993] Kernel panic - not syncing: Fatal exception in interrupt
[450024.873211] Shutting down cpus with NMI
[450024.878695] Kernel Offset: 0x20a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[450024.970837] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---