device hotplug of vfio devices can lead to deadlock in vfio_pci_release
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Critical
|
Andy Whitcroft | ||
Bionic |
Fix Released
|
Critical
|
Andy Whitcroft |
Bug Description
[Impact]
Attempts to hotplug devices shared to userspace (qemu) via vfio triggers a deadlock in the kernel. A reboot is required to resolve this.
[Test Case]
Set up a KVM instance with attached devices, attempt to hotplug those using ipmitool.
[Regression Potential]
The change is to an uncommonly used driver. There is common code changes, but these are a noop in the normal case and should be easy to confirm basic operation.
[Other Info]
This fix has been verified by the reporter as fixing the deadlock.
===
We are seeing deadlocks during hotplug of devices under vfio.
As per the Linux kernel source code, there is a deadlock situation between vfio_pci_remove() and vfio_pci_release() on PCIe hotplug events. This issue can be avoided either by skipping the PCIe reset functionality or do device_unlock() in vfio_pci_remove() beforfe calling the function vfio_del_
Code flow on PCIe hotplug event:
Execution flow 1:
device_
device_
device_
vfio_
vfio_
send event request to user and wait for VFIO_PCI_DEVICE release in vfio_pci_release() ( https:/
Execution flow 2 triggered by above step "send event request to user":
vfio_pci_releas() ( https:/
vfio_
vfio_
CVE References
Changed in linux (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → Critical |
assignee: | nobody → Andy Whitcroft (apw) |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → Critical |
assignee: | nobody → Andy Whitcroft (apw) |
status: | New → In Progress |
description: | updated |
summary: |
- vfio_pci_release hotplug deadlock + device hotplug of vfio devices can lead to deadlock in vfio_pci_release |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Fix Committed → Fix Released |
tags: |
added: kernel-fixup-verification-needed-bionic removed: verification-needed-bionic |
tags: | added: cscc |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- bionic' to 'verification- done-bionic' . If the problem still exists, change the tag 'verification- needed- bionic' to 'verification- failed- bionic' .
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!