io_uring regression in the Ubuntu kernel (deadlock)

Bug #2020901 reported by Stéphane Graber
272
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Triaged
Critical
Unassigned
Kinetic
Won't Fix
High
Unassigned

Bug Description

Whenever using io_uring on the Ubuntu 5.15 or 5.19 kernel, one gets:
```
[ 123.226074] BUG: kernel NULL pointer dereference, address: 000000000000001d
[ 123.226160] #PF: supervisor read access in kernel mode
[ 123.226201] #PF: error_code(0x0000) - not-present page
[ 123.226241] PGD 0 P4D 0
[ 123.226272] Oops: 0000 [#1] PREEMPT SMP PTI
[ 123.226310] CPU: 2 PID: 4326 Comm: qemu-system-x86 Tainted: P O 5.19.0-42-generic #43~22.04.1-Ubuntu
[ 123.226381] Hardware name: /D33217GKE, BIOS GKPPT10H.86A.0069.2019.1104.1340 11/04/2019
[ 123.228698] RIP: 0010:__blk_queue_split+0x53/0x1f0
[ 123.231029] Code: 00 00 83 f8 09 0f 84 e7 00 00 00 83 f8 03 0f 84 15 01 00 00 48 89 d1 4c 89 c6 4c 89 ca e8 b5 f2 ff ff 48 89 c3 48 85 db 74 5f <44> 8b 63 28 81 4b 10 00 40 00 00 49 be 00 00 00 00 00 00 00 80 4c
[ 123.235909] RSP: 0018:ffff9bb3414779e8 EFLAGS: 00010286
[ 123.238328] RAX: fffffffffffffff5 RBX: fffffffffffffff5 RCX: 0000000000000000
[ 123.240737] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 123.243093] RBP: ffff9bb341477a08 R08: 0000000000000000 R09: 0000000000000000
[ 123.245435] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e095d629ac0
[ 123.247735] R13: ffff9bb341477a18 R14: ffff8e0940df2040 R15: 0000000001400000
[ 123.250024] FS: 00007fa1cff602c0(0000) GS:ffff8e0a57300000(0000) knlGS:0000000000000000
[ 123.252306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 123.254591] CR2: 000000000000001d CR3: 0000000111ccc006 CR4: 00000000001726e0
[ 123.256899] Call Trace:
[ 123.259174] <TASK>
[ 123.261406] blk_mq_submit_bio+0x8c/0x440
[ 123.263626] __submit_bio+0x109/0x1a0
[ 123.265795] __submit_bio_noacct+0x81/0x1f0
[ 123.267922] submit_bio_noacct_nocheck+0x91/0x120
[ 123.270016] ? blk_cgroup_bio_start+0xac/0x130
[ 123.272076] ? recalibrate_cpu_khz+0x10/0x10
[ 123.274114] ? ktime_get+0x46/0xc0
[ 123.276126] submit_bio_noacct+0x209/0x590
[ 123.278132] submit_bio+0x40/0xf0
[ 123.280121] __blkdev_direct_IO_async+0x146/0x1f0
[ 123.282108] blkdev_direct_IO.part.0+0x40/0xa0
[ 123.284097] blkdev_read_iter+0x9f/0x1a0
[ 123.286065] io_read+0xea/0x510
[ 123.288080] ? fget+0x83/0xc0
[ 123.290031] io_issue_sqe+0x61/0x440
[ 123.291960] ? io_init_req+0xfa/0x2f0
[ 123.293847] io_submit_sqes+0x141/0x4a0
[ 123.295703] ? __fget_light+0xb5/0x160
[ 123.297537] __do_sys_io_uring_enter+0x316/0x670
[ 123.299345] ? __secure_computing+0x9b/0x110
[ 123.301153] __x64_sys_io_uring_enter+0x22/0x40
[ 123.302900] do_syscall_64+0x5c/0x90
[ 123.304608] ? do_syscall_64+0x69/0x90
[ 123.306286] ? exit_to_user_mode_prepare+0x3b/0xd0
[ 123.307969] ? syscall_exit_to_user_mode+0x2a/0x50
[ 123.309605] ? do_syscall_64+0x69/0x90
[ 123.311176] ? do_syscall_64+0x69/0x90
[ 123.312717] ? sysvec_reschedule_ipi+0x7b/0x120
[ 123.314252] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 123.315791] RIP: 0033:0x7fa1d28855e1
[ 123.317314] Code: 89 55 e4 89 4d e0 4c 89 45 d8 4c 89 4d d0 44 8b 55 e0 4c 8b 45 d8 4c 8b 4d d0 b8 aa 01 00 00 8b 7d ec 8b 75 e8 8b 55 e4 0f 05 <48> 89 45 f8 48 8b 45 f8 5d c3 55 48 89 e5 48 83 ec 18 89 7d fc 89
[ 123.320664] RSP: 002b:00007fa17550ae68 EFLAGS: 00000216 ORIG_RAX: 00000000000001aa
[ 123.322364] RAX: ffffffffffffffda RBX: 00005603c0418a28 RCX: 00007fa1d28855e1
[ 123.324060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000002d
[ 123.325684] RBP: 00007fa17550ae68 R08: 0000000000000000 R09: 0000000000000008
[ 123.327225] R10: 0000000000000000 R11: 0000000000000216 R12: 00005603c0418b10
[ 123.328734] R13: 00005603bdc48948 R14: 00005603bdc48988 R15: 0000000000000000
[ 123.330247] </TASK>
[ 123.331740] Modules linked in: nft_masq nft_chain_nat zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock unix_diag tls bridge stp llc binfmt_misc intel_rapl_msr mei_pxp mei_hdcp intel_rapl_common x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp snd_hda_intel kvm_intel snd_intel_dspcfg kvm snd_intel_sdw_acpi snd_hda_codec rapl intel_cstate snd_hda_core joydev snd_hwdep input_leds at24 mei_me snd_pcm snd_timer mei snd soundcore mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy
[ 123.331923] async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear i915 drm_buddy i2c_algo_bit ttm hid_generic drm_display_helper cec usbhid hid rc_core drm_kms_helper crct10dif_pclmul syscopyarea sysfillrect crc32_pclmul sysimgblt fb_sys_fops ghash_clmulni_intel cryptd ahci drm i2c_i801 e1000e i2c_smbus lpc_ich libahci video
[ 123.350700] CR2: 000000000000001d
[ 123.352644] ---[ end trace 0000000000000000 ]---
[ 123.354014] RIP: 0010:__blk_queue_split+0x53/0x1f0
[ 123.355051] Code: 00 00 83 f8 09 0f 84 e7 00 00 00 83 f8 03 0f 84 15 01 00 00 48 89 d1 4c 89 c6 4c 89 ca e8 b5 f2 ff ff 48 89 c3 48 85 db 74 5f <44> 8b 63 28 81 4b 10 00 40 00 00 49 be 00 00 00 00 00 00 00 80 4c
[ 123.357377] RSP: 0018:ffff9bb3414779e8 EFLAGS: 00010286
[ 123.358553] RAX: fffffffffffffff5 RBX: fffffffffffffff5 RCX: 0000000000000000
[ 123.359798] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 123.361170] RBP: ffff9bb341477a08 R08: 0000000000000000 R09: 0000000000000000
[ 123.362410] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e095d629ac0
[ 123.363544] R13: ffff9bb341477a18 R14: ffff8e0940df2040 R15: 0000000001400000
[ 123.364704] FS: 00007fa1cff602c0(0000) GS:ffff8e0a57300000(0000) knlGS:0000000000000000
[ 123.365949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 123.367059] CR2: 000000000000001d CR3: 0000000111ccc006 CR4: 00000000001726e0
```

This is due to a bad backport in the Ubuntu kernel:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?id=13f7058f1bd06c78775305cc0b16f0bcb0510eb6

As that can be triggered by an unprivileged user and causes a NULL pointer deref, this may be exploitable either as a way to DoS the system or even panic it in some cases.

Revision history for this message
Stéphane Graber (stgraber) wrote :

We first saw this hit maybe a month or 6 weeks ago in 5.15. As we didn't have time to debug it at the time, we just moved our CI to the 5.19 kernel and moved on, but now it's hitting 5.19 as well...

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
information type: Private Security → Public Security
Changed in linux (Ubuntu Kinetic):
status: New → In Progress
Stefan Bader (smb)
Changed in linux (Ubuntu Kinetic):
importance: Undecided → High
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.19.0-47.49 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux verification-needed-kinetic
Revision history for this message
Binoy Jayan (binoyjayan) wrote :

Can someone validate this fix, please? The original reporter said that he left the company. Thank you.

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :

Fix was commited already and will get into release. We just need to wait.

If you want you can validate it by yourself it will be good. But as far as I know it's not a requirement.

Revision history for this message
Binoy Jayan (binoyjayan) wrote :

For some reason, I wasn't able to get the -proposed images. However, I was able to validate this patch on the following kernel branch by running it through our I/O workloads.

git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/kinetic: Ubuntu-5.19.0-43.44

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

Changed in linux (Ubuntu Kinetic):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.