linux 5.15.0: repeated warnings from fs/inode.c:265 __destroy_inode+0x143/0x180

[James Dingwall]

The kernel is a rebuild of the Ubuntu source tree with a small number of xen related patches. On a small percentage of boots we encounter a repeated kernel warning which comes from fs/inode.c:

        if (!inode->i_nlink) {
                WARN_ON(atomic_long_read(&inode->i_sb->s_remove_count) == 0);
                atomic_long_dec(&inode->i_sb->s_remove_count);
        }

Using strace to attach to the pid logged in the warning the filesystem is a tmpfs (either separate, e.g. /run or as the writeable layer of an aufs overlay for /). This can be raised for different filesystem operations, in this example do_unlinkat, others do_renameat2. The problem does not seem to be hardware specific as we have encountered on five different systems.

[Wed Nov 29 10:51:47 2023] ------------[ cut here ]------------
[Wed Nov 29 10:51:47 2023] WARNING: CPU: 2 PID: 3295469 at fs/inode.c:265 __destroy_inode+0x143/0x180
[Wed Nov 29 10:51:47 2023] Modules linked in: tcp_diag udp_diag inet_diag cls_cgroup sch_htb drbd_transport_tcp(OE) handshake(OE) drbd(OE) lru_cache bridge stp llc xt_owner xt_multiport xt_hl xt_tcpudp xt_state xt_conntrack xt_NFLOG ip6
table_raw iptable_raw ip6table_mangle iptable_mangle ip6table_nat iptable_nat nf_nat ip6table_filter ip6_tables iptable_filter bpfilter dell_rbu nfnetlink_log nfnetlink i40e dcdbas igb ipmi_ssif joydev input_leds mei_me mei ioatdma inte
l_pch_thermal dca acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ramoops reed_solomon pstore_blk pstore_zone xenfs ip_tables x_
tables autofs4 raid10 raid0 multipath linear xen_privcmd zstd zstd_compress zram zfs(POE) zunicode(POE) zzstd(O) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 aufs raid1 dm_raid
 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
[Wed Nov 29 10:51:47 2023] hid_generic usbhid hid raid6_pq libcrc32c ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec rc_core ahci i2c_i801 xhci_pci efi_pstore drm i2c
_smbus libahci xhci_pci_renesas wmi
[Wed Nov 29 10:51:47 2023] CPU: 2 PID: 3295469 Comm: nano Tainted: P W OE 5.15.0-91-generic #101~20.04.1zynstra1
[Wed Nov 29 10:51:47 2023] Hardware name: Supermicro SYS-5019D-FN8TP-5-NC041/X11SDV-4C-TP8F, BIOS 1.2 11/14/2019
[Wed Nov 29 10:51:47 2023] RIP: e030:__destroy_inode+0x143/0x180
[Wed Nov 29 10:51:47 2023] Code: 83 c7 08 e8 5f 3f da ff eb 93 be 08 00 00 00 48 83 c7 08 e8 4f 3f da ff eb 92 0f 0b 0f 0b e9 25 ff ff ff 0f 0b e9 f0 fe ff ff <0f> 0b e9 55 ff ff ff 48 8b 90 60 02 00 00 f0 48 83 2a 01 0f 85 00
[Wed Nov 29 10:51:47 2023] RSP: e02b:ffffc9004c4a3d80 EFLAGS: 00010246
[Wed Nov 29 10:51:47 2023] RAX: ffff88812d892000 RBX: ffff88816be64b18 RCX: 0000000000004e01
[Wed Nov 29 10:51:47 2023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88816be64b18
[Wed Nov 29 10:51:47 2023] RBP: ffffc9004c4a3d88 R08: 0000000000000001 R09: 0000000000000064
[Wed Nov 29 10:51:47 2023] R10: ffffffff8261d3de R11: 0000000000000003 R12: ffffffffc031a340
[Wed Nov 29 10:51:47 2023] R13: ffff88816be64ba0 R14: ffff88816be64b18 R15: ffff8881727c8180
[Wed Nov 29 10:51:47 2023] FS: 00007fdee76d0040(0000) GS:ffff889046c80000(0000) knlGS:0000000000000000
[Wed Nov 29 10:51:47 2023] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[Wed Nov 29 10:51:47 2023] CR2: 00007fbce67eeff0 CR3: 0000000fc6a84000 CR4: 0000000000050660
[Wed Nov 29 10:51:47 2023] Call Trace:
[Wed Nov 29 10:51:47 2023] <TASK>
[Wed Nov 29 10:51:47 2023] ? show_trace_log_lvl+0x1d6/0x2ea
[Wed Nov 29 10:51:47 2023] ? show_trace_log_lvl+0x1d6/0x2ea
[Wed Nov 29 10:51:47 2023] ? destroy_inode+0x2f/0x80
[Wed Nov 29 10:51:47 2023] ? show_regs.part.0+0x23/0x29
[Wed Nov 29 10:51:47 2023] ? show_regs.cold+0x8/0xd
[Wed Nov 29 10:51:47 2023] ? __destroy_inode+0x143/0x180
[Wed Nov 29 10:51:47 2023] ? __warn+0x8c/0x100
[Wed Nov 29 10:51:47 2023] ? __destroy_inode+0x143/0x180
[Wed Nov 29 10:51:47 2023] ? report_bug+0xa4/0xd0
[Wed Nov 29 10:51:47 2023] ? handle_bug+0x39/0x90
[Wed Nov 29 10:51:47 2023] ? exc_invalid_op+0x19/0x70
[Wed Nov 29 10:51:47 2023] ? asm_exc_invalid_op+0x1b/0x20
[Wed Nov 29 10:51:47 2023] ? __destroy_inode+0x143/0x180
[Wed Nov 29 10:51:47 2023] destroy_inode+0x2f/0x80
[Wed Nov 29 10:51:47 2023] evict+0x142/0x190
[Wed Nov 29 10:51:47 2023] iput_final+0xa5/0x150
[Wed Nov 29 10:51:47 2023] iput.part.0+0xa0/0xb0
[Wed Nov 29 10:51:47 2023] iput+0x1c/0x30
[Wed Nov 29 10:51:47 2023] do_unlinkat+0x1c7/0x2c0
[Wed Nov 29 10:51:47 2023] __x64_sys_unlink+0x42/0x70
[Wed Nov 29 10:51:47 2023] do_syscall_64+0x59/0xc0
[Wed Nov 29 10:51:47 2023] ? ptrace_notify+0x68/0x80
[Wed Nov 29 10:51:47 2023] ? exit_to_user_mode_prepare+0x96/0xb0
[Wed Nov 29 10:51:47 2023] ? syscall_exit_to_user_mode+0x35/0x50
[Wed Nov 29 10:51:47 2023] ? do_syscall_64+0x69/0xc0
[Wed Nov 29 10:51:47 2023] entry_SYSCALL_64_after_hwframe+0x62/0xcc
[Wed Nov 29 10:51:47 2023] RIP: 0033:0x7fdee77e6cdb
[Wed Nov 29 10:51:47 2023] Code: f0 ff ff 73 01 c3 48 8b 0d b2 c1 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 c1 0d 00 f7 d8 64 89 01 48
[Wed Nov 29 10:51:47 2023] RSP: 002b:00007ffe02bbec38 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
[Wed Nov 29 10:51:47 2023] RAX: ffffffffffffffda RBX: 000055a41ee8b0c0 RCX: 00007fdee77e6cdb
[Wed Nov 29 10:51:47 2023] RDX: 000055a41ee9b7e0 RSI: 000055a41f09f5e0 RDI: 000055a41f0068a0
[Wed Nov 29 10:51:47 2023] RBP: 000055a41f007320 R08: 0000000000000001 R09: 0000000000009600
[Wed Nov 29 10:51:47 2023] R10: 00007ffe02bbea40 R11: 0000000000000202 R12: 000055a41f0068a0
[Wed Nov 29 10:51:47 2023] R13: 0000000000000001 R14: 0000000000000003 R15: 00007ffe02bbf4f8
[Wed Nov 29 10:51:47 2023] </TASK>
[Wed Nov 29 10:51:47 2023] ---[ end trace 7ecd80b467ae428b ]---

$ lsb_release -rd
Description: Ubuntu 20.04.6 LTS
Release: 20.04

[James Dingwall (a-james-launchpad)]

Prior to running 5.15.0-85 we were using 5.15.0-79 where the problem was not observed. Between these two releases it looks like https://github.com/torvalds/linux/commit/36ce9d76b0a93bae799e27e4f5ac35478c676592 was added to the Ubuntu tree. As that relates to tmpfs superblock operations and the warning seems to be coming from that area is there an issue with that backport?