bt_iter() crash due to NULL pointer
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
Artful |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
SRU Justification:
[Impact]
The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9):
[ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
[ 10.980708] IP: bt_iter+0x31/0x50
[ 10.984310] PGD 0
[ 10.984310] P4D 0
[ 10.986439]
[ 10.990190] Oops: 0000 [#1] SMP PTI
[ 11.016282] Workqueue: kblockd blk_mq_timeout_work
[ 11.021196] task: ffff8e7c2e700000 task.stack: ffffb8d4c67a8000
[ 11.027234] RIP: 0010:bt_
[ 11.031187] RSP: 0018:ffffb8d4c6
[ 11.037730] RAX: ffffb8d4c67abdd0 RBX: 0000000000000180 RCX: 0000000000000000
[ 11.045172] RDX: ffff8e7c34c8d280 RSI: 0000000000000000 RDI: ffff8e7c32dd8000
[ 11.053321] RBP: ffffb8d4c67abe20 R08: 0000000000000000 R09: 0000200000000100
[ 11.060582] R10: 0000000000000130 R11: 00000000fffee5bf R12: ffff8e7c3572c790
[ 11.068094] R13: ffff8e7c3572c780 R14: 0000000000000008 R15: ffff8e7c35e7c180
[ 11.075522] FS: 000000000000000
[ 11.083721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 11.089593] CR2: 0000000000000030 CR3: 000000009e20a003 CR4: 00000000001606e0
[ 11.096871] Call Trace:
[ 11.099468] ? blk_mq_
[ 11.104558] ? blk_mq_
[ 11.109130] ? blk_mq_
[ 11.114933] blk_mq_
[ 11.119408] process_
[ 11.123641] worker_
[ 11.127827] kthread+0x109/0x140
[ 11.131186] ? process_
[ 11.135499] ? kthread_
[ 11.140408] ret_from_
[ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff
[ 11.167573] RIP: bt_iter+0x31/0x50 RSP: ffffb8d4c67abda0
[ 11.173028] CR2: 0000000000000030
[ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]---
[ 11.182589] Kernel panic - not syncing: Fatal exception
Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section).
[Fix]
The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions.
The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe.
(https:/
[Testcase]
Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs.
CVE References
no longer affects: | linux-gcp (Ubuntu) |
no longer affects: | linux (Ubuntu Bionic) |
Changed in linux (Ubuntu Xenial): | |
status: | New → Fix Released |
Changed in linux (Ubuntu Artful): | |
status: | New → In Progress |
Changed in linux (Ubuntu Artful): | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-artful removed: verification-needed-artful |
Changed in linux (Ubuntu): | |
status: | Incomplete → Fix Released |
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1744300
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.