| 2023-09-02 18:12:03 |
Steffen McPrivacy |
bug |
|
|
added bug |
| 2023-09-03 04:06:22 |
Launchpad Janitor |
qemu (Ubuntu): status |
New |
Confirmed |
|
| 2023-09-04 01:07:09 |
JohnJay |
bug |
|
|
added subscriber JohnJay |
| 2023-09-04 08:47:22 |
Solb |
bug |
|
|
added subscriber Øyvind Ursin |
| 2023-09-04 17:15:34 |
Trebacz |
bug |
|
|
added subscriber Trebacz |
| 2023-09-05 10:08:01 |
Christian Ehrhardt |
qemu (Ubuntu): assignee |
|
Sergio Durigan Junior (sergiodj) |
|
| 2023-09-05 17:44:30 |
Nick Groenen |
bug |
|
|
added subscriber Nick Groenen |
| 2023-09-05 22:10:35 |
Sergio Durigan Junior |
qemu (Ubuntu): importance |
Undecided |
High |
|
| 2023-09-05 22:10:37 |
Sergio Durigan Junior |
qemu (Ubuntu): status |
Confirmed |
Triaged |
|
| 2023-09-05 22:10:46 |
Sergio Durigan Junior |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-09-05 22:10:46 |
Sergio Durigan Junior |
bug task added |
|
qemu (Ubuntu Jammy) |
|
| 2023-09-05 22:10:50 |
Sergio Durigan Junior |
qemu (Ubuntu Jammy): status |
New |
Triaged |
|
| 2023-09-05 22:10:53 |
Sergio Durigan Junior |
qemu (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2023-09-05 22:10:55 |
Sergio Durigan Junior |
qemu (Ubuntu Jammy): assignee |
|
Sergio Durigan Junior (sergiodj) |
|
| 2023-09-05 22:10:58 |
Sergio Durigan Junior |
qemu (Ubuntu): importance |
High |
Undecided |
|
| 2023-09-05 22:11:00 |
Sergio Durigan Junior |
qemu (Ubuntu): assignee |
Sergio Durigan Junior (sergiodj) |
|
|
| 2023-09-05 22:11:07 |
Sergio Durigan Junior |
qemu (Ubuntu): status |
Triaged |
Invalid |
|
| 2023-09-05 22:11:16 |
Sergio Durigan Junior |
tags |
|
server-todo |
|
| 2023-09-06 03:05:36 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~sergiodj/ubuntu/+source/qemu/+git/qemu/+merge/450752 |
|
| 2023-09-06 03:08:04 |
Sergio Durigan Junior |
description |
After installing the upgrades
qemu-system-x86:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-user-static:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-utils:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-common:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-block-extra:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-data:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-gui:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13)
and a reboot of the server, all virtual machines are getting an "connection refused" error and cannot access the host folders via virtiofs anymore.
Exact Error message on the client:
cannot access '/var/www-lib': Connection refused
Nothing else in the Logs.
Client OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux 5.15.0-82-generic #91-Ubuntu SMP
FileSystem: EXT4
Server OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Ubuntu 22.04, 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
FileSystem: EXT4
I scanned all logs, in the kern.log, I found the following message:
virtiofs virtio0: virtio_fs_setup_dax: No cache capability
the XML-Code from the guest machine looks like this:
...
<memoryBacking>
<hugepages>
<page size='2' unit='M'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/Da/n/W/w'/>
<target dir='W'/>1
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</filesystem>
...
I did a little more trying to find the issue. The change in the file /usr/lib/qemu/virtiofsd with the update 1:6.2+dfsg-2ubuntu6.13 is causing the problem.
I did an upgrade to the new rust based virtiofsd and modified my virtual machine to be loaded without flock and posix on.
Voila, mapping is working.
Now I changed it back to the original version - access denied
flock and posix still not configured, changing back to version 1:6.2+dfsg-2ubuntu6.12 is also working.
Therefore I assume we have an bug in the new virtiofsd version.
|
[ Impact ]
TBD.
[ Test Plan ]
TBD.
[ Where problems could occur ]
TBD.
[ Original Description ]
After installing the upgrades
qemu-system-x86:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-user-static:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-utils:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-common:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-block-extra:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-data:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-gui:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13)
and a reboot of the server, all virtual machines are getting an "connection refused" error and cannot access the host folders via virtiofs anymore.
Exact Error message on the client:
cannot access '/var/www-lib': Connection refused
Nothing else in the Logs.
Client OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux 5.15.0-82-generic #91-Ubuntu SMP
FileSystem: EXT4
Server OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Ubuntu 22.04, 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
FileSystem: EXT4
I scanned all logs, in the kern.log, I found the following message:
virtiofs virtio0: virtio_fs_setup_dax: No cache capability
the XML-Code from the guest machine looks like this:
...
<memoryBacking>
<hugepages>
<page size='2' unit='M'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/Da/n/W/w'/>
<target dir='W'/>1
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</filesystem>
...
I did a little more trying to find the issue. The change in the file /usr/lib/qemu/virtiofsd with the update 1:6.2+dfsg-2ubuntu6.13 is causing the problem.
I did an upgrade to the new rust based virtiofsd and modified my virtual machine to be loaded without flock and posix on.
Voila, mapping is working.
Now I changed it back to the original version - access denied
flock and posix still not configured, changing back to version 1:6.2+dfsg-2ubuntu6.12 is also working.
Therefore I assume we have an bug in the new virtiofsd version. |
|
| 2023-09-07 18:12:39 |
Sergio Durigan Junior |
bug |
|
|
added subscriber Ubuntu Server |
| 2023-09-07 18:30:37 |
Sergio Durigan Junior |
description |
[ Impact ]
TBD.
[ Test Plan ]
TBD.
[ Where problems could occur ]
TBD.
[ Original Description ]
After installing the upgrades
qemu-system-x86:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-user-static:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-utils:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-common:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-block-extra:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-data:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-gui:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13)
and a reboot of the server, all virtual machines are getting an "connection refused" error and cannot access the host folders via virtiofs anymore.
Exact Error message on the client:
cannot access '/var/www-lib': Connection refused
Nothing else in the Logs.
Client OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux 5.15.0-82-generic #91-Ubuntu SMP
FileSystem: EXT4
Server OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Ubuntu 22.04, 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
FileSystem: EXT4
I scanned all logs, in the kern.log, I found the following message:
virtiofs virtio0: virtio_fs_setup_dax: No cache capability
the XML-Code from the guest machine looks like this:
...
<memoryBacking>
<hugepages>
<page size='2' unit='M'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/Da/n/W/w'/>
<target dir='W'/>1
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</filesystem>
...
I did a little more trying to find the issue. The change in the file /usr/lib/qemu/virtiofsd with the update 1:6.2+dfsg-2ubuntu6.13 is causing the problem.
I did an upgrade to the new rust based virtiofsd and modified my virtual machine to be loaded without flock and posix on.
Voila, mapping is working.
Now I changed it back to the original version - access denied
flock and posix still not configured, changing back to version 1:6.2+dfsg-2ubuntu6.12 is also working.
Therefore I assume we have an bug in the new virtiofsd version. |
[ Impact ]
QEMU users who rely on virtiofs for mounting external paths will face a connection refused error while trying to accessing such mountpoints. There is no workaround for this problem other than using a previous version of QEMU.
[ Test Plan ]
We'll need to create a VM on libvirt and edit its XML domain definition in order to make it use virtiofsd to access a path on the host.
Inside a Jammy system:
$ sudo apt install -y libvirt-daemon-system uvtool-libvirt
$ uvt-simplestreams-libvirt sync release=jammy arch=amd64
$ uvt-kvm create j release=jammy --memory 1024
$ virsh destroy j
$ virsh edit j
Inside the editor, add the following snippets:
<domain type='kvm'>
...
<memoryBacking>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
...
<devices>
...
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/tmp/test'/>
<target dir='mytag'/>
</filesystem>
...
</devices>
</domain>
Save and exit.
$ mkdir -p /tmp/test
$ touch /tmp/test/a
$ virsh start j
$ virsh wait j
$ virsh ssh j
Now, while inside the VM (as the ubuntu user):
$ sudo mount -t virtiofs mytag /mnt
$ ls -la /mnt
You should see the file 'a'.
[ Where problems could occur ]
This bug is a regression caused by one of the patches backported to address bug bug #1853307. The problem happens because the Linux kernel headers have been updated, and that caused a change in the size of "struct fuse_init_in" that wasn't accounted for. Upstream's fix for this was to initially limit the function that reads such struct in a way that only the 16 initial bytes are considered. This fix, albeit correct in theory, wasn't part of any release because https://gitlab.com/qemu-project/qemu/-/commit/242f2cae782d433d69d195e14564b6437ec9f7e6 was merged right after which implemented new features for virtiofsd including the support for the extra bytes coming from "struct fuse_init_in". I chose not to backport any of the commits that are part of the merge aforementioned exactly because they fall under the category "new features", which is not acceptable for our SRUs.
A regression, if it were to occur, would likely manifest in the form of problems with the code responsible for parsing the first 16 bytes of the struct. I'm not entirely sure whether the struct can be affected by compiler optimizations which add paddings to speed better align the struct fields; it doesn't that it is.
Note that a regression here would only affect virtiofsd users, who are already completely unable to use the feature.
[ Original Description ]
After installing the upgrades
qemu-system-x86:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-user-static:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-utils:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-common:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-block-extra:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-data:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13),
qemu-system-gui:amd64 (1:6.2+dfsg-2ubuntu6.12, 1:6.2+dfsg-2ubuntu6.13)
and a reboot of the server, all virtual machines are getting an "connection refused" error and cannot access the host folders via virtiofs anymore.
Exact Error message on the client:
cannot access '/var/www-lib': Connection refused
Nothing else in the Logs.
Client OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Linux 5.15.0-82-generic #91-Ubuntu SMP
FileSystem: EXT4
Server OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Ubuntu 22.04, 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
FileSystem: EXT4
I scanned all logs, in the kern.log, I found the following message:
virtiofs virtio0: virtio_fs_setup_dax: No cache capability
the XML-Code from the guest machine looks like this:
...
<memoryBacking>
<hugepages>
<page size='2' unit='M'/>
</hugepages>
<access mode='shared'/>
</memoryBacking>
....
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs' queue='1024'/>
<binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
<cache mode='always'/>
<lock posix='on' flock='on'/>
</binary>
<source dir='/Da/n/W/w'/>
<target dir='W'/>1
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</filesystem>
...
I did a little more trying to find the issue. The change in the file /usr/lib/qemu/virtiofsd with the update 1:6.2+dfsg-2ubuntu6.13 is causing the problem.
I did an upgrade to the new rust based virtiofsd and modified my virtual machine to be loaded without flock and posix on.
Voila, mapping is working.
Now I changed it back to the original version - access denied
flock and posix still not configured, changing back to version 1:6.2+dfsg-2ubuntu6.12 is also working.
Therefore I assume we have an bug in the new virtiofsd version. |
|
| 2023-09-07 18:30:39 |
Sergio Durigan Junior |
qemu (Ubuntu Jammy): status |
Triaged |
In Progress |
|
| 2023-09-07 18:32:50 |
Sergio Durigan Junior |
tags |
server-todo |
regression-update server-todo |
|
| 2023-09-07 21:43:03 |
Ubuntu Archive Robot |
bug |
|
|
added subscriber Sergio Durigan Junior |
| 2023-09-15 12:23:14 |
Timo Aaltonen |
qemu (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2023-09-15 12:23:15 |
Timo Aaltonen |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-09-15 12:23:17 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2023-09-15 12:23:22 |
Timo Aaltonen |
tags |
regression-update server-todo |
regression-update server-todo verification-needed verification-needed-jammy |
|
| 2023-09-15 20:28:41 |
Steffen McPrivacy |
tags |
regression-update server-todo verification-needed verification-needed-jammy |
regression-update server-todo verification-done-jammy verification-needed verification-needed-jammy |
|
| 2023-09-15 20:29:46 |
Steffen McPrivacy |
tags |
regression-update server-todo verification-done-jammy verification-needed verification-needed-jammy |
regression-update server-todo verification-needed |
|
| 2023-09-15 20:30:32 |
Steffen McPrivacy |
tags |
regression-update server-todo verification-needed |
regression-update server-todo verification-done-jammy verification-needed |
|
| 2023-09-18 06:18:04 |
Christian Ehrhardt |
tags |
regression-update server-todo verification-done-jammy verification-needed |
regression-update server-todo verification-done verification-done-jammy |
|
| 2023-09-21 04:26:22 |
Brian Turek |
bug |
|
|
added subscriber Brian Turek |
| 2023-09-21 05:29:29 |
Tony Phan |
bug |
|
|
added subscriber Tony Phan |
| 2023-09-25 20:21:54 |
David Hedlund |
bug |
|
|
added subscriber David Hedlund |
| 2023-09-28 00:36:07 |
Launchpad Janitor |
qemu (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-09-28 00:36:17 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2023-09-28 22:41:43 |
David Hedlund |
removed subscriber David Hedlund |
|
|
|
