QEMU 2.11.1 VM crash when performing block pull. bdrv_co_do_copy_on_readv

Bug #1818264 reported by Matt Fleming on 2019-03-01
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

[Impact]

 * During qemu-io VM disk read, there is a possibility for qemu-io to
   abort crashing the process running. This also crashes the VM the read
   is running against.

 * Backport upstream fix from

[Test Case]

 * Steps to reproduce outside of a block pull:
    $ qemu-img create -f qcow2 test 1024 2>&1
    $ qemu-io -f qcow2 -C -c 'read 0 1024' test

    Without the fix:
    Error:
    qemu-io: /build/qemu-Eap4uc/qemu-2.11+dfsg/block/io.c:1042:
    bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
    Aborted (core dumped)

    With fix:
    read 1024/1024 bytes at offset 0
    1 KiB, 1 ops; 0.0045 sec (220.604 KiB/sec and 220.6045 ops/sec)

[Regression Potential]

 * The change is small and unchanged since two releases which indicates it
   is rather stable. The only thing it adds is a safety check on ret&&pnum
   being zero to leave the loop.
   The only related regression I could think of is leaving the loop too
   early, but when pnum==0 it has nothing more to write, so there should
   be no issue to leave.

[Other Info]

 * n/a

---

When attempting to do a blockpull the following error occurs and the VM crashes.

qemu-system-x86_64: /build/qemu-Eap4uc/qemu-2.11+dfsg/block/io.c:1042: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
2019-02-17 17:38:27.820+0000: shutting down, reason=crashed

This appears to be fixed upstream.
https://patchwork.kernel.org/patch/10512135/
https://github.com/qemu/qemu/commit/b0ddcbbb36a66a605eb232b905cb49b1cc72e74e

Matt Fleming (devpump) on 2019-03-01
affects: qemu-kvm (Ubuntu) → qemu (Ubuntu)

@DevPump - I think I can work on that along a general 2.11.2 upgrade which is comming sooner or later.
But I'd need you to provide some good steps on how to recreate and test/verify this to make [1] look usable on this bugs description.

Do you think you could provide as much as you can of [1] but at least a detailed howto, how to trigger the issue?

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

Changed in qemu (Ubuntu):
status: New → Incomplete
Matt Fleming (devpump) wrote :

[Impact]

During qemu-io VM disk read, there is a possibility for qemu-io to abort crashing the process running. This also crashes the VM the read is running against.

[Test Case]
Steps to reproduce outside of a block pull:

# qemu-img create -f qcow2 test 1024 2>&1
# qemu-io -f qcow2 -C -c 'read 0 1024' test

Error:
qemu-io: /build/qemu-Eap4uc/qemu-2.11+dfsg/block/io.c:1042: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
Aborted (core dumped)

QEMU Test: https://github.com/qemu/qemu/blob/bf22957309369cf6f642e715ff6c470671920e7e/tests/qemu-iotests/197

Changed in qemu (Ubuntu):
status: Incomplete → Triaged
status: Triaged → Fix Released
Changed in qemu (Ubuntu Bionic):
status: New → Triaged
Changed in qemu (Ubuntu Cosmic):
status: New → Triaged

Thanks, that looks good.
I can confirm the test and updated the bug description.
I'll check backportability for 2.11 and 2.12 later today after I have added 2.11.2

description: updated

Applies fine, tomorrow I'll create PPAs to test

There is a test PPA [1] available that we can use to pre-check this before the actual SRU.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/qemu-bionic-1815477-1818264

Tested and working fine from PPA.

@SRU Team - FYI Cosmic up for review in -unapproved, Bionic intentionally not yet - but that way around ordering should be fine.
- For Cosmic this can already be uploaded (where it is the only change).
- for Bionic the fix is bundled with the 2.11.2 changes which take much longer in regard to verification and preparation before it will goe to -unapproved.

Changed in qemu (Ubuntu Cosmic):
status: Triaged → In Progress

Hello DevPump, or anyone else affected,

Accepted qemu into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.12+dfsg-3ubuntu8.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic

Before upgrade:

root@c:~# qemu-img create -f qcow2 test 1024 2>&1
Formatting 'test', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
root@c:~# qemu-io -f qcow2 -C -c 'read 0 1024' test
qemu-io: /build/qemu-w2bpg6/qemu-2.12+dfsg/block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
Aborted (core dumped)

Installing the upgrade
apt install qemu-utils
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  qemu-block-extra qemu-system-common
Suggested packages:
  debootstrap
The following packages will be upgraded:
  qemu-block-extra qemu-system-common qemu-utils
3 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.
Need to get 1057 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 qemu-utils amd64 1:2.12+dfsg-3ubuntu8.4 [987 kB]
Get:2 http://archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 qemu-system-common amd64 1:2.12+dfsg-3ubuntu8.4 [31.0 kB]
Get:3 http://archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 qemu-block-extra amd64 1:2.12+dfsg-3ubuntu8.4 [39.9 kB]
Fetched 1057 kB in 0s (3445 kB/s)
(Reading database ... 55838 files and directories currently installed.)
Preparing to unpack .../qemu-utils_1%3a2.12+dfsg-3ubuntu8.4_amd64.deb ...
Unpacking qemu-utils (1:2.12+dfsg-3ubuntu8.4) over (1:2.12+dfsg-3ubuntu8.3) ...
Preparing to unpack .../qemu-system-common_1%3a2.12+dfsg-3ubuntu8.4_amd64.deb ...
Unpacking qemu-system-common (1:2.12+dfsg-3ubuntu8.4) over (1:2.12+dfsg-3ubuntu8.3) ...
Preparing to unpack .../qemu-block-extra_1%3a2.12+dfsg-3ubuntu8.4_amd64.deb ...
Unpacking qemu-block-extra:amd64 (1:2.12+dfsg-3ubuntu8.4) over (1:2.12+dfsg-3ubuntu8.3) ...
Setting up qemu-block-extra:amd64 (1:2.12+dfsg-3ubuntu8.4) ...
Setting up qemu-utils (1:2.12+dfsg-3ubuntu8.4) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up qemu-system-common (1:2.12+dfsg-3ubuntu8.4) ...

Then test succeeds:
qemu-io -f qcow2 -C -c 'read 0 1024' test
read 1024/1024 bytes at offset 0
1 KiB, 1 ops; 0.0001 sec (7.570 MiB/sec and 7751.9380 ops/sec)

Along the bigger SRU for Bionic I have already run this code through some more regression checks which all were fine.
Due to the above, setting verified for the cosmic release

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.12+dfsg-3ubuntu8.4

---------------
qemu (1:2.12+dfsg-3ubuntu8.4) cosmic; urgency=medium

  * fix crash when performing block pull on partial cluster (LP: #1818264)
    - d/p/ubuntu/lp-1818264-block-Fix-copy-on-read-crash-with-partial.patch

 -- Christian Ehrhardt <email address hidden> Tue, 05 Mar 2019 16:56:51 +0100

Changed in qemu (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

After long back and forth on the 2.11.2 bug it is dead now :-/

I decoupled this fix and put it for SRU Team review into bionic-unapproved.

Hello Matt, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done

I already verified it since that is very easy to do.
Setting verified.

But the actual release might be done through a security update which we bundled with this change.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.12

---------------
qemu (1:2.11+dfsg-1ubuntu7.12) bionic-security; urgency=medium

  [ Marc Deslauriers ]
  * SECURITY UPDATE: TOCTTOU in MTP
    - debian/patches/CVE-2018-16872.patch: use O_NOFOLLOW and O_CLOEXEC in
      hw/usb/dev-mtp.c.
    - CVE-2018-16872
  * SECURITY UPDATE: race during file renaming in v9fs_wstat
    - debian/patches/CVE-2018-19489.patch: add locks to hw/9pfs/9p.c.
    - CVE-2018-19489
  * SECURITY UPDATE: out-of-bounds read via i2 commands
    - debian/patches/CVE-2019-3812.patch: add bounds check to
      hw/i2c/i2c-ddc.c.
    - CVE-2019-3812
  * SECURITY UPDATE: heap based buffer overflow in slirp
    - debian/patches/CVE-2019-6778.patch: check data length while emulating
      ident function in slirp/tcp_subr.c.
    - CVE-2019-6778

  [ Christian Ehrhardt ]
  * fix crash when performing block pull on partial cluster (LP: #1818264)
    - d/p/ubuntu/lp-1818264-block-Fix-copy-on-read-crash-with-partial.patch
  * qemu-guest-agent: fix path of fsfreeze-hook (LP: #1820291)
    - d/qemu-guest-agent.install: use correct path for fsfreeze-hook
    - d/qemu-guest-agent.pre{rm|inst}/.postrm: special handling for
      mv_conffile since the new path is a directory in the old package
      version which can not be handled by mv_conffile

 -- Marc Deslauriers <email address hidden> Mon, 25 Mar 2019 08:32:58 -0400

Changed in qemu (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers