QEMU hits assertion when virtual disk is stored on NFS server and is not 4 kib byte aligned
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
qemu (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Matthew Ruffell | ||
Groovy |
Fix Released
|
Medium
|
Matthew Ruffell |
Bug Description
[Impact]
QEMU can hit an assertion and crash when attempting to write to a virtual disk image when the following conditions are met:
1. disk type is "raw"
2. disk cache type set to "none"
3. disk is shared over NFS
4. disk size is not a multiple of 4 kiB
In this case, QEMU assumes that the image needs to be aligned to 4kib, and that writing to the disk which is not a multiple of 4kib will lead to writing past the end of the disk image, and will hit the following assert:
qemu-system-x86_64: /build/
Aborted (core dumped)
This is particularly prevalent if you have a disk of size 10000000000 on the NFS server. You can work-around the problem by making the disk image a multiple of 4kib, in order to not hit the assert.
[Test case]
This bug is straightforward to reproduce on Focal and Groovy.
Start with a fresh install of Ubuntu Server, install KVM stack:
$ sudo apt-get install qemu-kvm libvirt-
$ sudo reboot
Next, install and configure a NFS server:
$ sudo -s
$ sudo apt install nfs-kernel-server
$ mkdir -p /export
$ chown libvirt-qemu:kvm /export
$ echo "/export 127.0.0.
$ systemctl restart nfs-server
Create a disk image:
$ truncate -s 10000000000 /export/
$ chown libvirt-qemu:kvm /export/
$ chmod 666 /export/
Mount the NFS server to /mnt:
$ mount 127.0.0.1:/export /mnt -o bg,noacl,
(for Groovy / 5.8 kernel, drop the proto=udp option)
Download the CentOS image:
$ wget https:/
Start the VM:
$ qemu-system-x86_64 -cdrom ./CentOS-
Connect to the VM with Reminna, configured for VNC connection to the server on <ip address>:5900
Click Continue at the language/keyboard selection screen.
Click Installation Destination.
Click Done (no changes needed on that screen).
Click Begin Installation.
It will crash after displaying "Setting up the installation environment" (which is the second thing printed) or the message about creating the disk label. If it gets any farther than that (i.e. starts creating filesystems), it's going to work and you can stop the test.
This is the error I see:
qemu-system-x86_64: /build/
Aborted (core dumped)
Test packages are available for Focal and Groovy in the following PPA:
https:/
When you use the QEMU packages from this PPA, the issue is fixed and the CentOS installation works properly.
[Where problems could occur]
There are two places where problems could occur.
The first, is around the handling of byte aligned disk images, in the block subsystem of QEMU.A new check is added to see if we have the write permission, and not the resize permission, and if that is the case, then we throw an error. This error is more graceful than hitting an assert, but it introduces error handling and other risks for regression.
The second, is that NFS is explicitly cleared for byte aligned writes, and this is enforced by testing for the NFS magic bytes on the start of the filesystem the disk image is being loaded from. By checking magic bytes, we ensure no other filesystem type could get mixed up and allow byte aligned writes when they would not be supported, which reduces the risk of regression.
If a regression were to occur, it would likely only affect users with non 4kib aligned disk images, and a workaround would be to resize the virtual disk image to 4kib alignment, or create new VMs with disk images as a multiple of 4kib.
[Other]
The commits which fix the problem landed in QEMU 5.1, and are:
commit 9c60a5d1978e6dc
From: Kevin Wolf <email address hidden>
Date: Thu, 16 Jul 2020 16:26:00 +0200
Subject: block: Require aligned image size to avoid assertion failure
Link: https:/
commit 5edc85571e7b726
From: Kevin Wolf <email address hidden>
Date: Thu, 16 Jul 2020 16:26:01 +0200
Subject: file-posix: Allow byte-aligned O_DIRECT with NFS
Link: https:/
Mailing list discussion:
https://<email address hidden>
Related branches
- Utkarsh Gupta (community): Approve
- Sergio Durigan Junior (community): Needs Information
- Canonical Server: Pending requested
- git-ubuntu developers: Pending requested
-
Diff: 587 lines (+530/-0)8 files modifieddebian/changelog (+14/-0)
debian/patches/series (+5/-0)
debian/patches/ubuntu/lp-1887535-configure-replace-enable-disable-git-update-with-wit.patch (+277/-0)
debian/patches/ubuntu/lp-1921468-pc-bios-s390-ccw-break-loop-if-a-null-block-number-i.patch (+40/-0)
debian/patches/ubuntu/lp-1921468-pc-bios-s390-ccw-fix-off-by-one-error.patch (+41/-0)
debian/patches/ubuntu/lp-1921665-1-block-Require-aligned-image-size-to-avoid-assert.patch (+66/-0)
debian/patches/ubuntu/lp-1921665-2-file-posix-Allow-byte-aligned-O_DIRECT-with-NFS.patch (+86/-0)
debian/rules (+1/-0)
- Utkarsh Gupta (community): Approve
- Canonical Server: Pending requested
- Canonical Server packageset reviewers: Pending requested
- git-ubuntu developers: Pending requested
-
Diff: 586 lines (+529/-0)8 files modifieddebian/changelog (+14/-0)
debian/patches/series (+5/-0)
debian/patches/ubuntu/lp-1887535-configure-replace-enable-disable-git-update-with-wit.patch (+276/-0)
debian/patches/ubuntu/lp-1921468-pc-bios-s390-ccw-break-loop-if-a-null-block-number-i.patch (+40/-0)
debian/patches/ubuntu/lp-1921468-pc-bios-s390-ccw-fix-off-by-one-error.patch (+41/-0)
debian/patches/ubuntu/lp-1921665-1-block-Require-aligned-image-size-to-avoid-assert.patch (+66/-0)
debian/patches/ubuntu/lp-1921665-2-file-posix-Allow-byte-aligned-O_DIRECT-with-NFS.patch (+86/-0)
debian/rules (+1/-0)
Changed in qemu (Ubuntu): | |
status: | New → Fix Released |
Changed in qemu (Ubuntu Focal): | |
status: | New → In Progress |
Changed in qemu (Ubuntu Groovy): | |
status: | New → In Progress |
Changed in qemu (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in qemu (Ubuntu Groovy): | |
importance: | Undecided → Medium |
Changed in qemu (Ubuntu Focal): | |
assignee: | nobody → Matthew Ruffell (mruffell) |
Changed in qemu (Ubuntu Groovy): | |
assignee: | nobody → Matthew Ruffell (mruffell) |
tags: | added: focal groovy sts |
Attached is a debdiff for Focal which fixes this bug.