NULL pointer dereference in nvme reset work-queue when VMD raid mode and SecureBoot turned on simultaneously on TigerLake

Bug #1876707 reported by You-Sheng Yang on 2020-05-04
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
HWE Next
Undecided
Unassigned
linux (Ubuntu)
Status tracked in Groovy
Focal
Undecided
Unassigned
Groovy
Critical
You-Sheng Yang
linux-oem-5.6 (Ubuntu)
Status tracked in Groovy
Focal
Critical
You-Sheng Yang
Groovy
Undecided
Unassigned

Bug Description

[SRU Justfication]

[Impact]

On platforms with NVMe attached to VMD controller, enable SecureBoot
would also force enable iommu:

  DMAR: Intel-IOMMU force enabled due to platform opt in

While devices behind the VMD controller, also a PCI bridge, maybe forced
to use a DMA domain by current Intel-IOMMU driver, this may break some
relationships between sub devices behind VMD controller and between the
VMD controller and its children devices, and finally caused undefined
system behavior.

On devices at hand, this results in kernel NULL dereference at
__intel_map_single called from nvme_reset_work and fails root device
lookup at boot.

  kernel: BUG: kernel NULL pointer dereference, address: 0000000000000018
  kernel: #PF: supervisor read access in kernel mode
  kernel: #PF: error_code(0x0000) - not-present page
  kernel: PGD 0 P4D 0
  kernel: Oops: 0000 [#2] SMP NOPTI
  kernel: CPU: 1 PID: 254 Comm: kworker/u8:4 Tainted: G D W
  5.7.0-050700rc3-generic #202004262131
  kernel: Hardware name: Dell Inc. Vostro 5402/, BIOS 0.1.2 04/13/2020
  kernel: Workqueue: nvme-reset-wq nvme_reset_work [nvme]
  kernel: RIP: 0010:__intel_map_single+0xa3/0x1a0
  ...

[Fix]

Patchset[1] currently landed in iommu/next beginning with commit
327d5b2fee91 ("iommu/vt-d: Allow 32bit devices to uses DMA domain")
gives the solution to this problem. However, it's based on a massive
subsystem rewrite in patchset[2], currently in iommu/next beginning with
commit 441ff2ff8327 ("Move default domain allocation to separate
function").

On v5.6, it also depends on yet a few more patch series landed in
v5.7-rc1 beginning with commit 098accf2da94 ("iommu: Use C99 flexible
array in fwspec") that rewrote private data access, changed struct
names, etc.

Yet a few additional patches included as fixes to above changes.

[1]: https://<email address hidden>/
[2]: https://lore.kernel.org/linux-iommu/20200429133712.31431-1-joro@8bytes.org/

[Test Case]

Test on platforms with VMD/NVMe and enable SecureBoot. System should
boot normally rather than into initramfs emergency shell.

[Regression Potential]

For unstable, all the patches are from iommu-next and will probably be
merged in next few -rc releases, so should be safe to place a LOW here.

For oem-5.6, the fixing patchset is depending on iommu group setup
refactoring that touched almost every architecture/platform uses iommu
although we would only care amd64 among them. Even with follow-up fixes
included, this is still a 60-patches change and deserves some more
attention. Medium.

========== Original Bug Description ==========

This is found on a Dell TigerLake platform that when VMD raid mode is turned on along with SecureBoot, either deploy mode or audit mode, kernel dumps warnings and null pointer deref errors at boot. While it happens, it blocks systemd-udevd worker processes until killed due to timeout. System still boots to multi-users.target.

Kernel bisect shows commit e3560ee4cfb2 ("iommu/vt-d: Remove VMD child device sanity check") merged in v5.6-rc1 is the first commit to fail, and is still reproducible on v5.7-rc3.

kernel: Secure boot disabled
...
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 1 PID: 8 at drivers/iommu/intel-iommu.c:625 domain_get_iommu+0x4b/0x60
kernel: Modules linked in: rc_core r8169(+) intel_lpss nvme crc32_pclmul(+) psmouse intel_ish_ipc(+) i2c_hid i2c_i801(+) realtek idma64 drm virt_dma intel_ishtp vmd(+) nvme_core hid video wmi pinctrl_tigerlake pinctrl_intel
kernel: CPU: 1 PID: 8 Comm: kworker/u8:0 Not tainted 5.7.0-050700rc3-generic #202004262131
kernel: Hardware name: Dell Inc. Vostro 5402/, BIOS 0.1.2 04/13/2020
kernel: Workqueue: nvme-reset-wq nvme_reset_work [nvme]
kernel: RIP: 0010:domain_get_iommu+0x4b/0x60
kernel: Code: eb 22 48 8d 50 01 48 39 c8 74 1b 48 89 d0 8b 74 87 04 48 63 d0 85 f6 74 e9 48 8b 05 ef 63 63 01 48 8b 04 d0 5d c3 31 c0 5d c3 <0f> 0b 31 c0 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f
kernel: RSP: 0018:ffffb5f5c00fbcf8 EFLAGS: 00010202
kernel: RAX: ffff9ebdaf100b00 RBX: 0000000000000000 RCX: 0000000000000000
kernel: RDX: 0000000000001000 RSI: 000000036dd41000 RDI: ffff9ebdaf100b00
kernel: RBP: ffffb5f5c00fbcf8 R08: ffffffffffffffff R09: ffff9ebdadd41000
kernel: R10: ffffffff8d069060 R11: 0000000000004879 R12: ffff9ebdadd810b0
kernel: R13: 000000036dd41000 R14: ffffffffffffffff R15: ffff9ebdaf100b00
kernel: FS: 0000000000000000(0000) GS:ffff9ebdc1680000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007f900d20e660 CR3: 000000036e05a003 CR4: 0000000000760ee0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: __intel_map_single+0x47/0x1a0
kernel: intel_alloc_coherent+0xab/0x120
kernel: dma_alloc_attrs+0x4d/0x60
kernel: nvme_alloc_queue+0x63/0x180 [nvme]
kernel: nvme_reset_work+0x31a/0xa64 [nvme]
kernel: ? wake_up_process+0x15/0x20
kernel: ? swake_up_locked.part.0+0x17/0x30
kernel: process_one_work+0x1e8/0x3b0
kernel: worker_thread+0x4d/0x400
kernel: kthread+0x104/0x140
kernel: ? process_one_work+0x3b0/0x3b0
kernel: ? kthread_park+0x90/0x90
kernel: ret_from_fork+0x1f/0x40
kernel: ---[ end trace caf06459a58aa8d4 ]---
....
kernel: BUG: kernel NULL pointer dereference, address: 0000000000000018
kernel: #PF: supervisor read access in kernel mode
kernel: #PF: error_code(0x0000) - not-present page
kernel: PGD 0 P4D 0
kernel: Oops: 0000 [#2] SMP NOPTI
kernel: CPU: 1 PID: 254 Comm: kworker/u8:4 Tainted: G D W 5.7.0-050700rc3-generic #202004262131
kernel: Hardware name: Dell Inc. Vostro 5402/, BIOS 0.1.2 04/13/2020
kernel: Workqueue: nvme-reset-wq nvme_reset_work [nvme]
kernel: RIP: 0010:__intel_map_single+0xa3/0x1a0
kernel: Code: 89 d2 4c 89 55 d0 e8 ec b3 ff ff 4c 8b 55 d0 48 85 c0 49 89 c6 0f 84 e9 00 00 00 41 b9 01 00 00 00 83 fb 01 76 14 48 8b 45 c0 <4c> 8b 48 18 49 c1 e9 16 49 83 f1 01 41 83 e1 01 44 89 c8 4c 89 e9
kernel: RSP: 0018:ffffb5f5c050f878 EFLAGS: 00010202
kernel: RAX: 0000000000000000 RBX: 0000000000000002 RCX: ffff9ebdbf205140
kernel: RDX: ffff9ebdbf205bc0 RSI: 0000000000000257 RDI: ffff9ebdaf100e30
kernel: RBP: ffffb5f5c050f8c0 R08: ffff9ebdae266f00 R09: 0000000000000001
kernel: R10: 0000000000000001 R11: 0000000000000022 R12: ffff9ebdadd860b0
kernel: R13: 000000036ef0d000 R14: 00000000000ffffa R15: ffff9ebdaf100b00
kernel: FS: 0000000000000000(0000) GS:ffff9ebdc1680000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000018 CR3: 000000036efd2001 CR4: 0000000000760ee0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: intel_map_page+0x86/0xa0
kernel: nvme_map_data+0x486/0x990 [nvme]
kernel: ? fbcon_cursor+0x128/0x180
kernel: ? bit_putcs+0x5a0/0x5a0
kernel: nvme_queue_rq+0xa2/0x1d0 [nvme]
kernel: blk_mq_dispatch_rq_list+0x93/0x5d0
kernel: ? __alloc_pages_nodemask+0x161/0x2f0
kernel: ? _find_next_bit.constprop.0+0x20/0x80
kernel: blk_mq_sched_dispatch_requests+0xfe/0x180
kernel: __blk_mq_run_hw_queue+0x5a/0x110
kernel: __blk_mq_delay_run_hw_queue+0x15b/0x160
kernel: blk_mq_run_hw_queue+0x70/0x110
kernel: blk_mq_sched_insert_request+0xce/0x190
kernel: ? blk_rq_append_bio+0x28/0x180
kernel: blk_execute_rq_nowait+0x61/0x70
kernel: blk_execute_rq+0x50/0xb0
kernel: __nvme_submit_sync_cmd+0x92/0x1e0 [nvme_core]
kernel: ? __cpuhp_state_add_instance_cpuslocked+0xe8/0x110
kernel: nvme_identify_ctrl.isra.0+0x7e/0xc0 [nvme_core]
kernel: nvme_init_identify+0x97/0x6d0 [nvme_core]
kernel: nvme_reset_work+0x422/0xa64 [nvme]
kernel: ? try_to_wake_up+0x65/0x690
kernel: process_one_work+0x1e8/0x3b0
kernel: worker_thread+0x4d/0x400
kernel: kthread+0x104/0x140
kernel: ? process_one_work+0x3b0/0x3b0
kernel: ? kthread_park+0x90/0x90
kernel: ret_from_fork+0x1f/0x40
kernel: Modules linked in: cec(+) intel_lpss_pci(+) rc_core fjes(-) r8169(+) intel_lpss nvme crc32_pclmul psmouse intel_ish_ipc(+) i2c_hid i2c_i801(+) realtek idma64 drm virt_dma intel_ishtp vmd nvme_core hid video wmi pinctrl_tigerlake pinctrl_intel
kernel: CR2: 0000000000000018
kernel: ---[ end trace caf06459a58aa8db ]---
kernel: RIP: 0010:__intel_map_single+0xa3/0x1a0
kernel: Code: 89 d2 4c 89 55 d0 e8 ec b3 ff ff 4c 8b 55 d0 48 85 c0 49 89 c6 0f 84 e9 00 00 00 41 b9 01 00 00 00 83 fb 01 76 14 48 8b 45 c0 <4c> 8b 48 18 49 c1 e9 16 49 83 f1 01 41 83 e1 01 44 89 c8 4c 89 e9
kernel: RSP: 0018:ffffb5f5c00fb878 EFLAGS: 00010202
kernel: RAX: 0000000000000000 RBX: 0000000000000002 RCX: ffff9ebdae2665c0
kernel: RDX: ffff9ebdbf205581 RSI: 0000000000000257 RDI: ffff9ebdaf100e30
kernel: RBP: ffffb5f5c00fb8c0 R08: ffff9ebdaf100e38 R09: 0000000000000001
kernel: R10: 0000000000000001 R11: 0000000000000022 R12: ffff9ebdadd810b0
kernel: R13: 000000036ef0e000 R14: 00000000000ffffd R15: ffff9ebdaf100b00
kernel: FS: 0000000000000000(0000) GS:ffff9ebdc1680000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000018 CR3: 000000036efd2001 CR4: 0000000000760ee0
kernel: PKRU: 55555554

You-Sheng Yang (vicamo) wrote :
You-Sheng Yang (vicamo) wrote :
You-Sheng Yang (vicamo) wrote :
tags: added: oem-priority originate-from-1873426 somerville
Changed in linux (Ubuntu Focal):
status: New → Invalid
Changed in linux-oem-5.6 (Ubuntu Groovy):
status: New → Invalid

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1876707

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
You-Sheng Yang (vicamo) wrote :

Verified linux-5.7=5.7.0-2.3+lp1876707 and linux-oem-5.6=5.6.0-1010.10+lp1876707 in my ppa.

You-Sheng Yang (vicamo) wrote :

Per https://bugzilla.kernel.org/show_bug.cgi?id=207575#c9, this has been superceded by Baolu's set in https://<email address hidden>/T/#t . To backport and verify again.

You-Sheng Yang (vicamo) wrote :

Backported iommu fixes in previous comment to Groovy in https://launchpad.net/~vicamo/+archive/ubuntu/ppa-1876707 as version 5.7.0-4.5+lp1876707.20200506015947.28662.1 . Working on oem-5.6 and locating a device for verification.

Changed in linux (Ubuntu Groovy):
status: Incomplete → In Progress
assignee: nobody → You-Sheng Yang (vicamo)
Changed in linux-oem-5.6 (Ubuntu Focal):
assignee: nobody → You-Sheng Yang (vicamo)
status: New → In Progress
You-Sheng Yang (vicamo) wrote :

Pushed proposed build to ppa and verified both (unstable, oem-5.6) with SecureBoot ON+DeployMode.

You-Sheng Yang (vicamo) wrote :
Changed in linux (Ubuntu Groovy):
importance: Undecided → Critical
Changed in linux-oem-5.6 (Ubuntu Focal):
importance: Undecided → Critical
Timo Aaltonen (tjaalton) on 2020-06-12
Changed in linux-oem-5.6 (Ubuntu Focal):
status: In Progress → Fix Committed

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
You-Sheng Yang (vicamo) wrote :

Verified oem-5.6 1012.

tags: added: verification-done-focal
removed: verification-needed-focal
Launchpad Janitor (janitor) wrote :
Download full text (30.4 KiB)

This bug was fixed in the package linux-oem-5.6 - 5.6.0-1018.18

---------------
linux-oem-5.6 (5.6.0-1018.18) focal; urgency=medium

  * focal/linux-oem-5.6: 5.6.0-1018.18 -proposed tracker (LP: #1884496)

  * Focal update: v5.6.18 upstream stable release (LP: #1883304)
    - devinet: fix memleak in inetdev_init()
    - l2tp: add sk_family checks to l2tp_validate_socket
    - l2tp: do not use inet_hash()/inet_unhash()
    - net: check untrusted gso_size at kernel entry
    - net/mlx5: Fix crash upon suspend/resume
    - net: stmmac: enable timestamp snapshot for required PTP packets in dwmac
      v5.10a
    - net: usb: qmi_wwan: add Telit LE910C1-EUX composition
    - NFC: st21nfca: add missed kfree_skb() in an error path
    - nfp: flower: fix used time of merge flow statistics
    - sctp: check assoc before SCTP_ADDR_{MADE_PRIM, ADDED} event
    - virtio_vsock: Fix race condition in virtio_transport_recv_pkt
    - vsock: fix timeout in vsock_accept()
    - net: be more gentle about silly gso requests coming from user
    - net: dsa: felix: send VLANs on CPU port as egress-tagged
    - mptcp: fix unblocking connect()
    - net/sched: fix infinite loop in sch_fq_pie
    - net/mlx5e: replace EINVAL in mlx5e_flower_parse_meta()
    - USB: serial: qcserial: add DW5816e QDL support
    - USB: serial: usb_wwan: do not resubmit rx urb on fatal errors
    - USB: serial: option: add Telit LE910C1-EUX compositions
    - USB: serial: ch341: add basis for quirk detection
    - USB: serial: ch341: fix lockup of devices with limited prescaler
    - iio:chemical:sps30: Fix timestamp alignment
    - iio: vcnl4000: Fix i2c swapped word reading.
    - iio:chemical:pms7003: Fix timestamp alignment and prevent data leak.
    - iio: adc: stm32-adc: fix a wrong error message when probing interrupts
    - usb: musb: start session in resume for host port
    - usb: musb: Fix runtime PM imbalance on error
    - serial: 8250: Enable 16550A variants by default on non-x86
    - vt: keyboard: avoid signed integer overflow in k_ascii
    - tty: hvc_console, fix crashes on parallel open/close
    - staging: rtl8712: Fix IEEE80211_ADDBA_PARAM_BUF_SIZE_MASK
    - CDC-ACM: heed quirk also in error handling
    - nvmem: qfprom: remove incorrect write support
    - x86/cpu: Add a steppings field to struct x86_cpu_id
    - x86/cpu: Add 'table' argument to cpu_matches()
    - x86/speculation: Add Special Register Buffer Data Sampling (SRBDS)
      mitigation
    - x86/speculation: Add SRBDS vulnerability and mitigation documentation
    - x86/speculation: Add Ivy Bridge to affected list
    - uprobes: ensure that uprobe->offset and ->ref_ctr_offset are properly
      aligned
    - Revert "net/mlx5: Annotate mutex destroy for root ns"
    - Linux 5.6.18

  * NULL pointer dereference in nvme reset work-queue when VMD raid mode and
    SecureBoot turned on simultaneously on TigerLake (LP: #1876707)
    - iommu: Use C99 flexible array in fwspec
    - iommu: Define dev_iommu_fwspec_get() for !CONFIG_IOMMU_API
    - ACPI/IORT: Remove direct access of dev->iommu_fwspec
    - drm/msm/mdp5: Remove direct access of dev->iommu_fwspec
    - iommu/tegra-gart: Remove direct access of dev->i...

Changed in linux-oem-5.6 (Ubuntu Focal):
status: Fix Committed → Fix Released
You-Sheng Yang (vicamo) wrote :

Landed to unstable already. Wait groovy kernel to be uplifted to v5.7 or v5.8 from unstable to
automatically include this fix.

Changed in linux (Ubuntu Groovy):
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.