[ Impact ]
QEMU users who rely on virtiofs for mounting external paths will face a connection refused error while trying to accessing such mountpoints. There is no workaround for this problem other than using a previous version of QEMU.
[ Test Plan ]
We'll need to create a VM on libvirt and edit its XML domain definition in order to make it use virtiofsd to access a path on the host.
Inside a Jammy system:
$ sudo apt install -y libvirt-daemon-system uvtool-libvirt
$ uvt-simplestreams-libvirt sync release=jammy arch=amd64
$ uvt-kvm create j release=jammy --memory 1024
$ virsh destroy j
$ virsh edit j
Inside the editor, add the following snippets:
<domain type='kvm'>
...
<memoryBacking>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
...
<devices>
...
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/tmp/test'/>
<target dir='mytag'/>
</filesystem>
...
</devices>
</domain>
Save and exit.
$ mkdir -p /tmp/test
$ touch /tmp/test/a
$ virsh start j
$ virsh wait j
$ virsh ssh j
Now, while inside the VM (as the ubuntu user):
$ sudo mount -t virtiofs mytag /mnt
$ ls -la /mnt
You should see the file 'a'.
[ Where problems could occur ]
This bug is a regression caused by one of the patches backported to address bug bug #1853307. The problem happens because the Linux kernel headers have been updated, and that caused a change in the size of "struct fuse_init_in" that wasn't accounted for. Upstream's fix for this was to initially limit the function that reads such struct in a way that only the 16 initial bytes are considered. This fix, albeit correct in theory, wasn't part of any release because https://gitlab.com/qemu-project/qemu/-/commit/242f2cae782d433d69d195e14564b6437ec9f7e6 was merged right after which implemented new features for virtiofsd including the support for the extra bytes coming from "struct fuse_init_in". I chose not to backport any of the commits that are part of the merge aforementioned exactly because they fall under the category "new features", which is not acceptable for our SRUs.
A regression, if it were to occur, would likely manifest in the form of problems with the code responsible for parsing the first 16 bytes of the struct. I'm not entirely sure whether the struct can be affected by compiler optimizations which add paddings to speed better align the struct fields; it doesn't that it is.
Note that a regression here would only affect virtiofsd users, who are already completely unable to use the feature.
[ Original Description ]
After installing the upgrades
qemu-system-x86:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-user-static:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-utils:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-common:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-block-extra:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-data:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-gui:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13)
and a reboot of the server, all virtual machines are getting an "connection refused" error and cannot access the host folders via virtiofs anymore.
Exact Error message on the client:
cannot access '/var/www-lib': Connection refused
Nothing else in the Logs.
Client OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux 5.15.0-82-generic #91-Ubuntu SMP
FileSystem: EXT4
Server OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Ubuntu 22.04, 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
FileSystem: EXT4
I scanned all logs, in the kern.log, I found the following message:
virtiofs virtio0: virtio_fs_setup_dax: No cache capability
the XML-Code from the guest machine looks like this:
...
<memoryBacking>
<hugepages>
<page size='2' unit='M'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/Da/n/W/w'/>
<target dir='W'/>1
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</filesystem>
...
I did a little more trying to find the issue. The change in the file /usr/lib/qemu/virtiofsd with the update 1:6.2+dfsg-2ubuntu6.13 is causing the problem.
I did an upgrade to the new rust based virtiofsd and modified my virtual machine to be loaded without flock and posix on.
Voila, mapping is working.
Now I changed it back to the original version - access denied
flock and posix still not configured, changing back to version 1:6.2+dfsg-2ubuntu6.12 is also working.
Therefore I assume we have an bug in the new virtiofsd version.
Status changed to 'Confirmed' because the bug affects multiple users.