kernel BUG at lib/assoc_array.c:652!

[mjw99]

I am seeing this in Ubuntu 22.04 with kernel 5.15.0-67.74:

Mar 23 12:38:32 box kernel: [13399.080589] ------------[ cut here ]------------
Mar 23 12:38:32 box kernel: [13399.080596] kernel BUG at lib/assoc_array.c:652!
Mar 23 12:38:32 box kernel: [13399.082833] invalid opcode: 0000 [#1] SMP PTI
Mar 23 12:38:32 box kernel: [13399.084912] CPU: 27 PID: 3746279 Comm: kworker/27:3 Tainted: G OE 5.15.0-67-generic #74-Ubuntu
Mar 23 12:38:32 box kernel: [13399.087091] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 07/18/2022
Mar 23 12:38:32 box kernel: [13399.089195] Workqueue: cifsiod cifs_resolve_server [cifs]
Mar 23 12:38:32 box kernel: [13399.091379] RIP: 0010:assoc_array_insert_into_terminal_node.isra.0+0x806/0xc20
Mar 23 12:38:32 box kernel: [13399.093480] Code: 4c 63 e2 44 8d 6a 01 49 83 fc 10 0f 87 d7 01 00 00 4c 8b 4d b8 44 89 ea 4b 8d 4c e1 10 e9 a7 fc ff ff 0f 0b 0f 0b 0f 0b 0f 0b <0f> 0b 0f 0b 48 c7 c7 e0 cf 4b 9c e8 ba 37 03 00 e9 53 f8 ff ff 0f
Mar 23 12:38:32 box kernel: [13399.097735] RSP: 0018:ffffc08e7a66fa20 EFLAGS: 00010286
Mar 23 12:38:32 box kernel: [13399.099790] RAX: 0000000000000002 RBX: 00000000ffffffff RCX: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.101871] RDX: ffff9d2275968000 RSI: 000000000000000f RDI: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.103963] RBP: ffffc08e7a66faa0 R08: ffff9d31e7cbda00 R09: ffff9d2275969b00
Mar 23 12:38:32 box kernel: [13399.106111] R10: ffff9cb6cfc93800 R11: 000000000000000c R12: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.108209] R13: 000000000000000f R14: 0000000000000000 R15: ffff9d2275969b00
Mar 23 12:38:32 box kernel: [13399.110292] FS: 0000000000000000(0000) GS:ffff9d14bfc40000(0000) knlGS:0000000000000000
Mar 23 12:38:32 box kernel: [13399.112355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 12:38:32 box kernel: [13399.114411] CR2: 00007f3a9b9e4a58 CR3: 00000012c8810005 CR4: 00000000001706e0
Mar 23 12:38:32 box kernel: [13399.116461] Call Trace:
Mar 23 12:38:32 box kernel: [13399.118452] <TASK>
Mar 23 12:38:32 box kernel: [13399.120455] assoc_array_insert+0x153/0x170
Mar 23 12:38:32 box kernel: [13399.122422] ? _raw_spin_lock+0x22/0x30
Mar 23 12:38:32 box kernel: [13399.124357] ? key_alloc+0x2e6/0x550
Mar 23 12:38:32 box kernel: [13399.126273] __key_link_begin+0x4c/0xb0
Mar 23 12:38:32 box kernel: [13399.128153] construct_alloc_key+0xdb/0x270
Mar 23 12:38:32 box kernel: [13399.130032] request_key_and_link+0x22c/0x320
Mar 23 12:38:32 box kernel: [13399.131867] ? dns_resolver_read+0x30/0x30
Mar 23 12:38:32 box kernel: [13399.133689] ? key_default_cmp+0x30/0x30
Mar 23 12:38:32 box kernel: [13399.135469] request_key_tag+0x49/0x90
Mar 23 12:38:32 box kernel: [13399.137241] dns_query+0x13d/0x250
Mar 23 12:38:32 box kernel: [13399.138989] dns_resolve_server_name_to_ip+0xb7/0x310 [cifs]
Mar 23 12:38:32 box kernel: [13399.141130] ? scnprintf+0x4d/0x90
Mar 23 12:38:32 box kernel: [13399.142866] reconn_set_ipaddr_from_hostname+0x86/0x2b0 [cifs]
Mar 23 12:38:32 box kernel: [13399.144654] ? finish_task_switch.isra.0+0x7e/0x280
Mar 23 12:38:32 box kernel: [13399.146412] cifs_resolve_server+0x2c/0x90 [cifs]
Mar 23 12:38:32 box kernel: [13399.148176] process_one_work+0x22b/0x3d0
Mar 23 12:38:32 box kernel: [13399.149845] worker_thread+0x53/0x420
Mar 23 12:38:32 box kernel: [13399.151459] ? process_one_work+0x3d0/0x3d0
Mar 23 12:38:32 box kernel: [13399.153061] kthread+0x12a/0x150
Mar 23 12:38:32 box kernel: [13399.154631] ? set_kthread_struct+0x50/0x50
Mar 23 12:38:32 box kernel: [13399.156181] ret_from_fork+0x22/0x30
Mar 23 12:38:32 box kernel: [13399.157716] </TASK>
Mar 23 12:38:32 box kernel: [13399.159565] Modules linked in: xt_REDIRECT xt_state ip_vs_rr xt_ipvs ip_vs vxlan ip6_udp_tunnel udp_tunnel xt_policy xt_mark xt_u32 veth xt_nat xt_tcpudp xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc cmac nls_utf8 cifs cifs_arc4 cifs_md4 binfmt_misc nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache netfs cpuid eset_rtp(OE-) overlay hpwdt nls_iso8859_1 intel_rapl_msr intel_rapl_common sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate mgag200 drm_kms_helper cec rc_core i2c_algo_bit fb_sys_fops syscopyarea sysfillrect ioatdma input_leds joydev hpilo sysimgblt dca acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_tad acpi_power_meter mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua coretemp lp parport
Mar 23 12:38:32 box kernel: [13399.159693] ramoops reed_solomon pstore_blk pstore_zone efi_pstore drm nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq hid_generic bnx2x usbhid nvme i2c_i801 hpsa ahci xhci_pci mdio crc32_pclmul hid libahci lpc_ich tg3 i2c_smbus nvme_core xhci_pci_renesas scsi_transport_sas libcrc32c wmi
Mar 23 12:38:32 box kernel: [13399.179701] ---[ end trace 2ef4625ad06f1cf9 ]---
Mar 23 12:38:32 box kernel: [13399.194447] RIP: 0010:assoc_array_insert_into_terminal_node.isra.0+0x806/0xc20
Mar 23 12:38:32 box kernel: [13399.195804] Code: 4c 63 e2 44 8d 6a 01 49 83 fc 10 0f 87 d7 01 00 00 4c 8b 4d b8 44 89 ea 4b 8d 4c e1 10 e9 a7 fc ff ff 0f 0b 0f 0b 0f 0b 0f 0b <0f> 0b 0f 0b 48 c7 c7 e0 cf 4b 9c e8 ba 37 03 00 e9 53 f8 ff ff 0f
Mar 23 12:38:32 box kernel: [13399.198454] RSP: 0018:ffffc08e7a66fa20 EFLAGS: 00010286
Mar 23 12:38:32 box kernel: [13399.199789] RAX: 0000000000000002 RBX: 00000000ffffffff RCX: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.201169] RDX: ffff9d2275968000 RSI: 000000000000000f RDI: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.202543] RBP: ffffc08e7a66faa0 R08: ffff9d31e7cbda00 R09: ffff9d2275969b00
Mar 23 12:38:32 box kernel: [13399.203923] R10: ffff9cb6cfc93800 R11: 000000000000000c R12: 0000000000000001
Mar 23 12:38:32 box kernel: [13399.205327] R13: 000000000000000f R14: 0000000000000000 R15: ffff9d2275969b00
Mar 23 12:38:32 box kernel: [13399.206740] FS: 0000000000000000(0000) GS:ffff9d14bfc40000(0000) knlGS:0000000000000000
Mar 23 12:38:32 box kernel: [13399.208166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 12:38:32 box kernel: [13399.209633] CR2: 00007f3a9b9e4a58 CR3: 00000012c8810005 CR4: 00000000001706e0

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2012661

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[mjw99 (mark-williamson)]

I am unable to run this command.

[mjw99 (mark-williamson)]

Insight via personal communication with Paulo Alcantara:

>By looking at the call trace, the bug is triggered from a worker
>(fs/cifs/connect.c::cifs_resolve_server) that periodically upcalls[*]
>(/sbin/key.dns_resolver) to resolve hostnames (10 minutes by default)
>per tcp connection in cifs.ko.
>
>So, such dns resolve worker was introduced in v5.14 by
>
> 506c1da44fee ("cifs: use the expiry output of dns_query to schedule
>next resolution")
>
>and then was removed in v6.3-rc1 by
>
> 0e9bd27b2a63 ("cifs: get rid of dns resolve worker")
>
>My guess would be an use-after-free bug in any of the fields in
>TCP_Server_Info structure -- most likely the hostname one.
>
>I see that it is v5.15 but I don't know what Ubuntu kernel has in
>terms of cifs.ko backports, so a potential commit for that would be
>
> 153695d36ead ("cifs: fix use-after-free caused by invalidpointer
>`hostname`")
>
>[*] upcall is basically a kernel mechanism to call into userspace -- in
>which case, cifs.ko usually ends up calling /sbin/key.dns_resolver
>(dns_resolver key type) to resolve hostnames in the kernel. Make sure
>you have it installed and for more information, look at
>request-key.conf(5).

[mjw99 (mark-williamson)]

The "cifs: fix use-after-free caused by invalid pointer
>`hostname`" patch made it into 5.15.0-65.72 ( http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_5.15.0-69.76/changelog )

[mjw99 (mark-williamson)]

Still seeing this with linux-image-5.15.0-71-generic 5.15.0-71.78

[mjw99 (mark-williamson)]

Still seeing this with 5.15.0-72-generic

[mjw99 (mark-williamson)]

Additional personal communication with Paulo Alcantara:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d55901522f96082a43b9842d34867363c0cdbac5