| 2020-12-09 08:09:22 |
bugproxy |
bug |
|
|
added bug |
| 2020-12-09 08:09:24 |
bugproxy |
tags |
|
architecture-s39064 bugnameltc-190211 severity-high targetmilestone-inin2104 |
|
| 2020-12-09 08:09:25 |
bugproxy |
ubuntu: assignee |
|
Skipper Bug Screeners (skipper-screen-team) |
|
| 2020-12-09 08:09:30 |
bugproxy |
affects |
ubuntu |
linux (Ubuntu) |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
nominated for series |
|
Ubuntu Hirsute |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
bug task added |
|
linux (Ubuntu Hirsute) |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
bug task added |
|
linux (Ubuntu Groovy) |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
nominated for series |
|
Ubuntu Focal |
|
| 2020-12-09 08:10:55 |
Frank Heimes |
bug task added |
|
linux (Ubuntu Focal) |
|
| 2020-12-09 08:11:15 |
Frank Heimes |
bug task added |
|
ubuntu-z-systems |
|
| 2020-12-09 09:48:35 |
Frank Heimes |
linux (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2020-12-09 09:48:57 |
Frank Heimes |
ubuntu-z-systems: assignee |
|
Skipper Bug Screeners (skipper-screen-team) |
|
| 2020-12-09 09:49:03 |
Frank Heimes |
linux (Ubuntu Hirsute): assignee |
Skipper Bug Screeners (skipper-screen-team) |
|
|
| 2020-12-09 09:49:11 |
Frank Heimes |
linux (Ubuntu Groovy): assignee |
|
Frank Heimes (fheimes) |
|
| 2020-12-09 09:49:14 |
Frank Heimes |
linux (Ubuntu Focal): assignee |
|
Frank Heimes (fheimes) |
|
| 2020-12-09 09:49:18 |
Frank Heimes |
ubuntu-z-systems: status |
New |
Triaged |
|
| 2020-12-09 14:46:06 |
Frank Heimes |
linux (Ubuntu Focal): status |
New |
Incomplete |
|
| 2020-12-09 14:46:09 |
Frank Heimes |
linux (Ubuntu Groovy): status |
New |
Incomplete |
|
| 2020-12-09 14:46:14 |
Frank Heimes |
ubuntu-z-systems: status |
Triaged |
Incomplete |
|
| 2021-01-20 09:41:56 |
Frank Heimes |
linux (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-01-20 09:42:20 |
Frank Heimes |
linux (Ubuntu Focal): status |
Incomplete |
Confirmed |
|
| 2021-01-20 09:42:27 |
Frank Heimes |
ubuntu-z-systems: status |
Incomplete |
Confirmed |
|
| 2021-01-20 15:35:15 |
Frank Heimes |
linux (Ubuntu Groovy): status |
Incomplete |
Confirmed |
|
| 2021-01-20 20:08:50 |
Frank Heimes |
description |
Description: vfio: pass DMA availability information to userspace
Symptom: vfio-pci device on s390 enters error state
Problem: Commit 492855939bdb added a limit to the number of concurrent
DMA requests for a vfio container. However, lazy unmapping in
s390 can in fact cause quite a large number of outstanding DMA
requests to build up prior to being purged, potentially the
entire guest DMA space. This results in unexpected errors seen
in qemu such as 'VFIO_MAP_DMA failed: No space left on device'
Solution: The solution requires a change to both kernel and qemu - For
the kernel, add the ability to provide the number of allowable
DMA requests via the VFIO_IOMMU_GET_INFO ioctl.
Reproduction: Put a vfio-pci device on s390 under I/O load
Upstream-ID: a717072007e8aedd3f951726d8cf55454860b30d
7d6e1329652ed971d1b6e0e7bea66fba5044e271
Need also to be integrated into 20.10 and 20.04.
OK, just to clarify we don't need to fix bionic for this one, but rather focal (20.04) and groovy (20.10). Furthermore, for 20.04, 20.10 and 21.04 ONLY commit 7d6e1329652ed971d1b6e0e7bea66fba5044e271 is needed, the other was a pre-req that is already present. |
Description: vfio: pass DMA availability information to userspace
Symptom: vfio-pci device on s390 enters error state
Problem: Commit 492855939bdb added a limit to the number of concurrent
DMA requests for a vfio container. However, lazy unmapping in
s390 can in fact cause quite a large number of outstanding DMA
requests to build up prior to being purged, potentially the
entire guest DMA space. This results in unexpected errors seen
in qemu such as 'VFIO_MAP_DMA failed: No space left on device'
Solution: The solution requires a change to both kernel and qemu - For
the kernel, add the ability to provide the number of allowable
DMA requests via the VFIO_IOMMU_GET_INFO ioctl.
Reproduction: Put a vfio-pci device on s390 under I/O load
Upstream-ID: a717072007e8aedd3f951726d8cf55454860b30d
7d6e1329652ed971d1b6e0e7bea66fba5044e271
Need also to be integrated into 20.10 and 20.04.
OK, just to clarify we don't need to fix bionic for this one, but rather focal (20.04) and groovy (20.10). Furthermore, for 20.04, 20.10 and 21.04 ONLY commit 7d6e1329652ed971d1b6e0e7bea66fba5044e271 is needed, the other was a pre-req that is already present.
__________
SRU Justification:
[Impact]
* In case a vfio-pci device on s390x is under I/O load, vfio-pci device may end up in error state.
* The commit 492855939bdb added a limit to the number of concurrent DMA requests for a vfio container.
* However, lazy unmapping in s390x can in fact cause quite a large number of outstanding DMA requests
to build up prior to being purged - potentially the entire guest DMA space.
* This results in unexpected errors seen in qemu such as 'VFIO_MAP_DMA failed: No space left on device'.
* The solution requires a change to both kernel and qemu.
* The kernel side of things is addressed by this SRU.
* The fix adds the ability to provide the number of allowable DMA requests via VFIO_IOMMU_GET_INFO ioctl.
* The actual fix comes with commit 7d6e1329652e, but another fix ccd59dce1a21 is needed to get it cleanly applied.
[Fix]
* ccd59dce1a21f473518bf273bdf5b182bab955b3 ccd59dce1a21 "vfio/type1: Refactor vfio_iommu_type1_ioctl()"
* 7d6e1329652ed971d1b6e0e7bea66fba5044e271 7d6e1329652e "vfio iommu: Add dma available capability"
[Test Case]
* IBM Z or LinuxONE hardware with Ubuntu Server 20.10 installed.
* PCIe adapters in place that provide vfio, like RoCE Express 2.
* A KVM host needs to be setup and a KVM guest (use again 20.10) that uses vfio.
* Generate I/O that flows through the vf and watch out for error like 'VFIO_MAP_DMA failed: No space left on device' in the log.
[Regression Potential]
* The first patch ccd59dce1a21 modifies the common code file drivers/vfio/vfio_iommu_type1.c quity significantly.
* But the reason is not that it introduces a lot of new things, it's a refactoring patch.
* Nevertheless if done in a bad way it can significantly harm the IO memory management of virtual function adapters.
* In worst case it may break them entirely, instead of 'just' exeeding the entire DMA space.
* Things could also go wrong while doing the mapping and unmapping of DMA, that may even have an impact beyond vf adapters - harming other DMA devices.
* The handling of dirty pages is also touched and the ioctl itself - which is important to keep the control of the devices.
* But as said before, it re-factoring work, it's upstream accepted since 5.9 and the provenance shows that many engineers had an eye on these changes.
* The second patch 7d6e1329652e - that inclides the needed fix - comes with far less modifications.
* It also tounches drivers/vfio/vfio_iommu_type1.c but in a way that it mainly add one new function (vfio_iommu_dma_avail_build_caps) and an add. if statement in vfio_iommu_type1_get_info.
* That allows to provide the current allowable number of DMA mappings to the userspace via the IOMMU info chain,
so that the userspace can take appropriate mitigation.
* Potential problems here can be that the current allowable number of DMA mappings are wrong and in best case just mappings are wasted and in worst case there are more reported than available in reality, which could have a severe impact.
* What happens in such a case is a bit depending on the userspace.
* This patch got upstream accepted with kernel 5.10 and pre-tested by IBM.
* In addition a PPA was created with a patched groovy kernel that was shared for further testing.
[Other]
* The patch got upstream accepted with kernel v5.10, hence it will land in Hirsute once the target kernel 5.10 is in.
* For 5.4 this will come as upstream stable update 5.4.90/91.
* For 5.8 (and 5.9) upstream stable support already ended, hence this SRU is only needed for groovy. |
|
| 2021-01-20 20:09:04 |
Frank Heimes |
linux (Ubuntu Groovy): status |
Confirmed |
In Progress |
|
| 2021-01-22 19:48:51 |
Kelsey Steele |
linux (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2021-01-29 07:39:33 |
Ubuntu Kernel Bot |
tags |
architecture-s39064 bugnameltc-190211 severity-high targetmilestone-inin2104 |
architecture-s39064 bugnameltc-190211 severity-high targetmilestone-inin2104 verification-needed-groovy |
|
| 2021-01-29 09:29:05 |
Frank Heimes |
tags |
architecture-s39064 bugnameltc-190211 severity-high targetmilestone-inin2104 verification-needed-groovy |
architecture-s39064 bugnameltc-190211 severity-high targetmilestone-inin2104 verification-done-groovy |
|
| 2021-02-08 07:46:20 |
Frank Heimes |
linux (Ubuntu Focal): status |
Confirmed |
In Progress |
|
| 2021-02-08 07:46:24 |
Frank Heimes |
ubuntu-z-systems: status |
Confirmed |
In Progress |
|
| 2021-02-19 12:32:41 |
Frank Heimes |
linux (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-02-19 12:32:46 |
Frank Heimes |
ubuntu-z-systems: status |
In Progress |
Fix Committed |
|
| 2021-02-22 07:25:44 |
Frank Heimes |
linux (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-02-22 12:09:57 |
Cemirtan Igor Grigore |
bug |
|
|
added subscriber Cemirtan Igor Grigore |
| 2021-02-23 16:22:36 |
Launchpad Janitor |
linux (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2021-02-23 16:22:36 |
Launchpad Janitor |
cve linked |
|
2020-27777 |
|
| 2021-02-23 16:22:36 |
Launchpad Janitor |
cve linked |
|
2020-28974 |
|
| 2021-03-15 16:08:47 |
Frank Heimes |
linux (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-03-15 16:08:52 |
Frank Heimes |
ubuntu-z-systems: status |
Fix Committed |
Fix Released |
|
