rtmutex bug causing kernel state corruption issues on arm64

[Jacob Martin]

Running the latest version in -next, 5.15.0-1032-realtime #35

There is a bug in the current rtmutex implementation, causing locking to not work correctly. The bug seems to have always existed, but is observed much more frequently on the arm64 Ampere Altra CPU. I have observed this issue on the bizzy, kopter, and howzit machines, accessible via the server enablement MAAS. As it affects rtmutex, it will also affect spinlocks on RT kernels.

The bug is causing some spurious test deployment failures for kernels based on linux-realtime.

This has recently been patched in the kernel tip repo, under the locking/urgent branch. However, it has not yet made it to the current (5.15.86-rt56) RT patchset.

Commit in kernel tip repo:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=1c0908d8e441631f5b8ba433523cf39339ee2ba0

Some LKML discussion:
https://lore.kernel.org/lkml/20221103115444.m2rjglbkubydidts@quack3/
https://<email address hidden>/

Steps to reliably reproduce (on the Ampere Altra CPU):
1. Run the following:
$ dd if=/dev/urandom bs=1 count=50000000 | xxd | less

2. Use the Shift + G shortcut of less to skip to the bottom

3. Before less loads the bottom, an oops message should be seen in the console or dmesg, and the above command should be stalled and uninterruptible.

[Jacob Martin (jacobmartin)]

An example of an oops resulting from this bug:

[ 384.960099] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 384.960104] Mem abort info:
[ 384.960105] ESR = 0x96000004
[ 384.960107] EC = 0x25: DABT (current EL), IL = 32 bits
[ 384.960109] SET = 0, FnV = 0
[ 384.960110] EA = 0, S1PTW = 0
[ 384.960111] FSC = 0x04: level 0 translation fault
[ 384.960113] Data abort info:
[ 384.960114] ISV = 0, ISS = 0x00000004
[ 384.960116] CM = 0, WnR = 0
[ 384.960116] user pgtable: 4k pages, 48-bit VAs, pgdp=0000400006947000
[ 384.960118] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[ 384.960121] Internal error: Oops: 96000004 [#1] PREEMPT_RT SMP
[ 384.960123] Modules linked in: binfmt_misc nls_iso8859_1 input_leds arm_spe_pmu joydev acpi_ipmi ipmi_ssif ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu xgene_hwmon arm_dsu_pmu cppc_cpufreq acpi_tad sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid cdc_ether hid usbnet mlx5_ib ib_uverbs ib_core uas usb_storage ast drm_vram_helper drm_ttm_helper ttm i2c_algo_bit drm_kms_helper crct10dif_ce syscopyarea ghash_ce sysfillrect sha2_ce sysimgblt fb_sys_fops sha256_arm64 sha1_ce cec rc_core nvme mlx5_core xhci_pci drm nvme_core xhci_pci_renesas mlxfw psample tls aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
[ 384.960228] CPU: 107 PID: 3946 Comm: dd Not tainted 5.15.0-1032-realtime #35
[ 384.960230] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0007/Mt.Jade Motherboard, BIOS 1.6.20210526 (SCP: 1.06.20210526) 2021/05/26
[ 384.960231] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 384.960233] pc : pipe_write+0x540/0x754
[ 384.960237] lr : pipe_write+0x50/0x754
[ 384.960238] sp : ffff80003e1cbbc0
[ 384.960239] x29: ffff80003e1cbbc0 x28: ffff3fff8a931a00 x27: ffff3fff983e4800
[ 384.960242] x26: 0000000000000003 x25: 0000000000000000 x24: ffff3fff983e4990
[ 384.960244] x23: 0000000000000190 x22: ffff3fff8a92cdc0 x21: ffff80003e1cbca0
[ 384.960246] x20: ffffffffffffffee x19: 0000000000000001 x18: 0000000000000000
[ 384.960248] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 384.960250] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 384.960252] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffa5ccdcffe188
[ 384.960254] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
[ 384.960257] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000001f74a
[ 384.960259] x2 : 0000000000000010 x1 : 0000000000000004 x0 : 0000000000000000
[ 384.960261] Call trace:
[ 384.960262] pipe_write+0x540/0x754
[ 384.960263] new_sync_write+0x17c/0x18c
[ 384.960266] vfs_write+0x278/0x2e4
[ 384.960268] ksys_write+0xe4/0x100
[ 384.960270] __arm64_sys_write+0x24/0x30
[ 384.960272] invoke_syscall+0x78/0x100
[ 384.960276] el0_svc_common.constprop.0+0x54...

[Joseph Salisbury (jsalisbury)]

I built a test kernel with commit:
1c0908d8e441 ("rtmutex: Add acquire semantics for rtmutex lock acquisition slow path")

If anyone would like to test this kernel, it can be downloaded from:
https://people.canonical.com/~jsalisbury/lp2004582/

Commit 1c0908d8e441 will be picked up by the Ubuntu real-time kernel within the next SRU cycle or two. It first needs to land in upstream stable, and this commit will be picked up in the usual stable update process.

[Joseph Salisbury (jsalisbury)]

Commit 1c0908d8e441 has made it to the upstream stable v5.15.109 kernel. The real-time kernel will pick up this commit once Jammy is rebased to 5.15.109.

[Joseph Salisbury (jsalisbury)]

This issue is fixed as of the 5.15.0-1043.48 real-time kernel version.