[UBUNTU 20.04] Lost virtio host --> guest notifications cause devices to cease normal operation
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on IBM z Systems |
Fix Released
|
High
|
Skipper Bug Screeners | ||
qemu (Ubuntu) |
Fix Released
|
High
|
Canonical Server | ||
Xenial |
Fix Released
|
Low
|
Unassigned | ||
Bionic |
Fix Released
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Canonical Server |
Bug Description
[Impact]
* Host -> Guest notifications can be lost and kill I/O due to that,
see below at the original bug report for more details.
* Backport the fix that ensures that the generated code has to re-load
variables properly avoiding the issue.
[Test Case]
* Set up iperf in the host and run the server "iperf -s"
* get a guest using driver=qemu like:
<interface type='network'>
<source network='default'/>
<model type='virtio'/>
<driver name='qemu'/>
<interface/>
* In the guest run a loop of iperf runs connecting to the
server on the host.
#!/bin/bash
for i in $(seq 1 1000);
do
echo Try $i
iperf -c 192.168.122.1 || break
done
* Depending on the HW model, the machine saturation and such it seems
the above test either is rather reproducible or not-at-all.
That is bad, but we haven't found a much better repro, gladly IBM
who reported this issue (and created the fix) can recreate this on
their end and are willing to do so again for the SRU verification.
[Regression Potential]
* The changed code path is s390x only and there on the virtio-ccw
handling. Therefore regressions - if any - would be isolated to s390x
only and there manifest on virtio-ccw based I/O.
[Other Info]
* n/a
----
Problem Description:
When irqfds are not used setting of the adapter interruption host-->guest notifier bit is accomplished by the QEMU function virtio_
The atomic_cmpxchg() loop in virtio_
The direct consequence of the problem is that host --> guest notifications can get lost. The indirect consequence is that queues may get stuck and the devices may cease operate normally. We stumbled on debugging a choked virtio-net interface (one that used the qemu driver and not vhost). But it can affect other virtio-ccw devices as well.
If irqfds are used for host->guest notifications, then we are safe because notifier bit manipulation is done in the kernel (and it's done correctly).
The problem described above is fixed upstream by commit.
1a8242f7c3 ("virtio-ccw: fix virtio_
All upstream versions since v2.0.0 are (potentially) affected.
The same mistake was made in QEMU in another place, and is fixed by:
45175361f1 ("s390x/pci: fix set_ind_atomic")
We can file a separate BZ for it if necessary.
Related branches
- Lucas Kanashiro (community): Approve
- Canonical Server: Pending requested
- Canonical Server packageset reviewers: Pending requested
-
Diff: 173 lines (+145/-0)4 files modifieddebian/changelog (+6/-0)
debian/patches/series (+2/-0)
debian/patches/ubuntu/lp-1894942-s390x-pci-fix-set_ind_atomic.patch (+67/-0)
debian/patches/ubuntu/lp-1894942-virtio-ccw-fix-virtio_set_ind_atomic.patch (+70/-0)
- Lucas Kanashiro (community): Approve
- Canonical Server packageset reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 173 lines (+145/-0)4 files modifieddebian/changelog (+6/-0)
debian/patches/series (+2/-0)
debian/patches/ubuntu/lp-1894942-s390x-pci-fix-set_ind_atomic.patch (+67/-0)
debian/patches/ubuntu/lp-1894942-virtio-ccw-fix-virtio_set_ind_atomic.patch (+70/-0)
- Lucas Kanashiro (community): Approve
- Canonical Server packageset reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 171 lines (+143/-0)4 files modifieddebian/changelog (+6/-0)
debian/patches/series (+2/-0)
debian/patches/ubuntu/lp-1894942-s390x-pci-fix-set_ind_atomic.patch (+67/-0)
debian/patches/ubuntu/lp-1894942-virtio-ccw-fix-virtio_set_ind_atomic.patch (+68/-0)
- Andreas Hasenack: Approve
- Canonical Server: Pending requested
- git-ubuntu developers: Pending requested
-
Diff: 183 lines (+155/-0)4 files modifieddebian/changelog (+6/-0)
debian/patches/series (+2/-0)
debian/patches/ubuntu/lp-1894942-s390x-pci-fix-set_ind_atomic.patch (+72/-0)
debian/patches/ubuntu/lp-1894942-virtio-ccw-fix-virtio_set_ind_atomic.patch (+75/-0)
Changed in ubuntu-z-systems: | |
assignee: | nobody → Skipper Bug Screeners (skipper-screen-team) |
Changed in qemu (Ubuntu): | |
assignee: | Skipper Bug Screeners (skipper-screen-team) → Canonical Server Team (canonical-server) |
Changed in ubuntu-z-systems: | |
importance: | Undecided → High |
Changed in ubuntu-z-systems: | |
status: | New → Triaged |
Changed in qemu (Ubuntu Bionic): | |
status: | Incomplete → Triaged |
Changed in qemu (Ubuntu Xenial): | |
status: | Incomplete → Triaged |
importance: | Undecided → Low |
Changed in qemu (Ubuntu Bionic): | |
importance: | Undecided → Medium |
description: | updated |
Changed in qemu (Ubuntu Xenial): | |
status: | Triaged → In Progress |
Changed in qemu (Ubuntu Bionic): | |
status: | Triaged → In Progress |
Changed in qemu (Ubuntu Focal): | |
status: | Triaged → In Progress |
Changed in ubuntu-z-systems: | |
status: | Triaged → In Progress |
Changed in ubuntu-z-systems: | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-focal removed: verification-needed-focal |
Changed in ubuntu-z-systems: | |
status: | Fix Committed → Fix Released |
Default Comment by Bridge