virt-manager local directory storage failing

[Mauro Gaspari]

[Impact]

 * An upstream change in libvirt 6.6 [1] broke the logic of creating a
   pool without an explicit yes/no for COW handling.

 * Another upstream patch in libvirt 6.7 [2] fixes the issue.

 * Backport the patch to avoid issues with tools like virt-manager but
   also with any other automation that uses define+build via api or
   cmdline

[1]: https://gitlab.com/libvirt/libvirt/-/commit/7230bc95aa78379c9ee20cf59394c5fc4305b75b
[2]: https://gitlab.com/libvirt/libvirt/-/commit/2edd63a0dbd445112db23596ee0128521e8f1ff5

[Test Case]

 * Define a dir pool and call pool-build:

$ virsh pool-define-as --name home --type dir --target /home --print-xml > home.xml
$ virsh pool-define home.xml
$ virsh pool-build home
error: Failed to build pool home
error: unable to control COW flag on '/home', not btrfs: Function not implemented
$ virsh pool-start home

With the fix (and with e.g. 6.0 in Focal) this works, but with libvirt 6.6 missing the fix it fails as shown above.

[Regression Potential]

 * Gladly a one line change there aren't many side effects to consider.
   It will help a certain set of use-cases and if anything could influence
   actual execution on filesystems with COW support in that case btrfs.
   Now one might think "but hey this influences BTRFS behavior", but no it
   does not. The change is inside an "if btrfs magic" section. Therefore
   BTRFS will not change, only the non-btrfs cases (formerly broken) will.
   This is one of the rare cases I really fail to see an explicit
   regression - implicit ones e.g. due to needing a rebuild could still
   happen.

[Other Info]

 * This is upstream issue:
   https://bugzilla.redhat.com/show_bug.cgi?id=1866157

--- ---

Installed KVM and Virt-Manager successfully.
When trying to add storage pool as local directory on my ext4 file system, I get this error:

Error creating pool: Could not build storage pool: unable to control COW flag on '/var/lib/libvirt/images/pool', not btrfs: Function not implemented

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createpool.py", line 378, in _async_pool_create
    poolobj = pool.install(create=True, meter=meter, build=build)
  File "/usr/share/virt-manager/virtinst/storage.py", line 415, in install
    raise RuntimeError(errmsg)
RuntimeError: Could not build storage pool: unable to control COW flag on '/var/lib/libvirt/images/pool', not btrfs: Function not implemented

It seems that virt-manager is trying to disable the COW flag and returns an error because my file system is not BTRFS.

Tested on 2 fresh installs of Ubuntu Studio 20.10
Tested on 1 install of Ubuntu 20.04.1 and virt-manager has no issues with local storage on ext4 file system.

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: virt-manager 1:2.2.1-4ubuntu2
ProcVersionSignature: Ubuntu 5.8.0-25.26-lowlatency 5.8.14
Uname: Linux 5.8.0-25-lowlatency x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu50
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: KDE
Date: Sat Oct 24 02:15:08 2020
InstallationDate: Installed on 2020-10-07 (16 days ago)
InstallationMedia: Ubuntu-Studio 20.10 "Groovy Gorilla" - Beta amd64 (20200930)
PackageArchitecture: all
SourcePackage: virt-manager
UpgradeStatus: No upgrade log present (probably fresh install)

[Mauro Gaspari (ilvipero)]

Interesting workaround found to keep using virt-manager:

1. Install cockpit-machines
2. Login to https://localhost:9090
3. Create the storage pool from cockpit, on same ext4 file system

The storage pool is added successfully, and immediately available on virt-manager too.

[Christian Ehrhardt  (paelzer)]

Confirmed:
Okt 26 08:49:04 Keschdeichel libvirtd[931584]: unable to control COW flag on '/home', not btrfs: Function not implemented

[Christian Ehrhardt  (paelzer)]

The XML virt-manager creates is listed as (in virt-manager XML view when creating it):
<pool type='dir'>
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>

That is the same as:

$ virsh pool-create-as --print-xml --name home --type dir --target /home
<pool type='dir'>
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>

But interestingly the command to create it works fine:
$ virsh pool-create-as --name home --type dir --target /home
Pool home created

[Christian Ehrhardt  (paelzer)]

Tracing libvirtd and checking what really "appears int he request" I see that:

the console request (that works) has:
debug : virStoragePoolCreateXML:564 : conn=0x7f39640040b0,
xmlDesc=<pool type='dir'>
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>
, flags=0x0

The request from virt-manager for what should be the same has:

debug : virStoragePoolDefineXML:607 : conn=0x7f39640042b0, xml=<pool type="dir">
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>
, flags=0x0

The difference is not in the XML content, but the function called.
virStoragePoolCreateXML vs virStoragePoolDefineXML

[Christian Ehrhardt  (paelzer)]

I was switching my commandline to use a more equivalent sequence which is using define+start instead of a direct create:

$ virsh pool-define-as --name home --type dir --target /home --print-xml
$ virsh pool-start home
$ virsh pool-destroy home
$ virsh pool-undefine home

But now I get it still working:
debug : virStoragePoolDefineXML:607 : conn=0x7efd580040a0, xml=<pool type='dir'>
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>
, flags=0x0

[Christian Ehrhardt  (paelzer)]

I found that with libvirt 6.0 this works and libvirt 6.6 fails (when driven from virt-manager). This isn't a virt-manager but a libvirt issue

[Christian Ehrhardt  (paelzer)]

Good case (commandline):

2020-10-26 08:08:15.210+0000: 941251: debug : virStoragePoolDefineXML:607 : conn=0x7efd580040a0, xml=<pool type='dir'>
  <name>home</name>
  <target>
    <path>/home</path>
  </target>
</pool>
, flags=0x0
2020-10-26 08:08:15.210+0000: 941251: debug : virFileMakePathHelper:3038 : path=/etc/libvirt/storage mode=0777
2020-10-26 08:08:15.220+0000: 941251: debug : virFileClose:135 : Closed fd 24
2020-10-26 08:08:15.220+0000: 941251: debug : virThreadJobClear:118 : Thread 941251 (rpc-worker) finished job remoteDispatchStoragePoolDefineXML with ret=0
2020-10-26 08:08:15.221+0000: 941253: debug : virThreadJobSet:93 : Thread 941253 (rpc-worker) is now running job remoteDispatchConnectUnregisterCloseCallback
2020-10-26 08:08:15.221+0000: 941253: debug : virConnectUnregisterCloseCallback:1476 : conn=0x7efd580040a0
2020-10-26 08:08:15.221+0000: 941253: debug : virThreadJobClear:118 : Thread 941253 (rpc-worker) finished job remoteDispatchConnectUnregisterCloseCallback with ret=0
2020-10-26 08:08:15.221+0000: 941252: debug : virThreadJobSet:93 : Thread 941252 (rpc-worker) is now running job remoteDispatchConnectClose
2020-10-26 08:08:15.221+0000: 941252: debug : virThreadJobClear:118 : Thread 941252 (rpc-worker) finished job remoteDispatchConnectClose with ret=0

Bad case (the same from virt-manager):

2020-10-26 08:01:06.215+0000: 935606: debug : virStoragePoolDefineXML:607 : conn=0x7f3958005130, xml=<pool type="dir">
  <name>home</name>
  <target>
    <path>/home/</path>
  </target>
</pool>
, flags=0x0
2020-10-26 08:01:06.216+0000: 935606: debug : virFileMakePathHelper:3038 : path=/etc/libvirt/storage mode=0777
2020-10-26 08:01:06.219+0000: 935606: debug : virFileClose:135 : Closed fd 25
2020-10-26 08:01:06.220+0000: 935606: debug : virThreadJobClear:118 : Thread 935606 (rpc-worker) finished job remoteDispatchStoragePoolDefineXML with ret=0
2020-10-26 08:01:06.221+0000: 935605: debug : virThreadJobSet:93 : Thread 935605 (rpc-worker) is now running job remoteDispatchStoragePoolBuild
2020-10-26 08:01:06.221+0000: 935605: debug : virStoragePoolBuild:648 : pool=0x7f395000a530, flags=0x0
2020-10-26 08:01:06.221+0000: 935605: debug : virFileSetCOW:4536 : Setting COW flag on '/home' to 'default'
2020-10-26 08:01:06.221+0000: 935605: error : virFileSetCOW:4554 : unable to control COW flag on '/home', not btrfs: Function not implemented
2020-10-26 08:01:06.221+0000: 935605: debug : virFileClose:135 : Closed fd 25

In the virt-manager case it goes into remoteDispatchStoragePoolBuild and then fails

[Christian Ehrhardt  (paelzer)]

Ok, now that I knew what to look for I guess I found it.

#1 Workaround (other than installing cockpit) for now can be a commandline call like:
$ virsh pool-create-as --name home --type dir --target /home

#2 testing without virt-manager
$ virsh pool-define-as --name home --type dir --target /home --print-xml > home.xml
$ virsh pool-define home.xml
$ virsh pool-build home
error: Failed to build pool home
error: unable to control COW flag on '/home', not btrfs: Function not implemented
$ virsh pool-start home

This is upstream issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1866157

Which has a fix at:
https://gitlab.com/libvirt/libvirt/-/commit/2edd63a0dbd445112db23596ee0128521e8f1ff5

This is fixed in libvirt >=6.7

I'll need Ubuntu 21.04 be "fully open" for development first, then fix it in 21.04 (libvirt changed to meson, that can be a bit of work).
Once fixed in 21.04 we can consider an SRU of the change for 20.10 (Groovy).

Thanks @ Mauro for the report!

[Christian Ehrhardt  (paelzer)]

Already added an SRU Template to the bug description to be ready for SRU processing asap then.

[Mauro Gaspari (ilvipero)]

Thanks @ Christian for the thorough investigation.

I also tried virsh commands and found exactly the same issue as you described.
It is good that we at least for now have workarounds to create and enabled storage repository.

[Christian Ehrhardt  (paelzer)]

I have prepared merge-requests for the required packaging changes and a PPA [1].
But right now builders are all kind of stuck so the PPA doesn't have much to try yet :-/

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4320/+packages

[Christian Ehrhardt  (paelzer)]

FYI - Uploaded the fix to Hirsute (a bit slow due to the test queue being rather full atm) and to groovy-unapproved for the SRU process.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.6.0-1ubuntu4

---------------
libvirt (6.6.0-1ubuntu4) hirsute; urgency=medium

  * d/p/ubuntu/lp-1901242-util-Fix-logic-in-virFileSetCOW.patch: fix dir pool
    handling on non BTRFS affecting virt-manager, api and commandline pool
    handling (LP: #1901242)

 -- Christian Ehrhardt <email address hidden> Wed, 28 Oct 2020 07:47:53 +0100

[Łukasz Zemczak (sil2100)]

Hello Mauro, or anyone else affected,

Accepted libvirt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.6.0-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauro Gaspari (ilvipero)]

Hello Lukasz,

Thank you for your great work. I already tested this, enabled groovy-proposed, updated and restarted system. I saw libvirt updates and apt-cache confirmed version 6.6.0-1ubuntu3.1

After reboot, I created a local storage repository with virt-manager and it worked right away. No more error messages. Is there anything else I can do to help?

[Christian Ehrhardt  (paelzer)]

Prio to the fix

root@g:~# dpkg -l libvirt-daemon-system
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================-==============-============-==================================
ii libvirt-daemon-system 6.6.0-1ubuntu3 amd64 Libvirt daemon configuration files
root@g:~# virsh pool-define-as --name home --type dir --target /home --print-xml > home.xml
root@g:~# virsh pool-define home.xml
Pool home defined from home.xml

root@g:~# virsh pool-build home
error: Failed to build pool home
error: unable to control COW flag on '/home', not btrfs: Function not implemented

Post upgrade
root@g:~# virsh pool-define-as --name home --type dir --target /home --print-xml > home.xml
root@g:~# virsh pool-define home.xml
Pool home defined from home.xml

root@g:~# virsh pool-build home
Pool home built

root@g:~# virsh pool-start home
Pool home started

Works fine, setting to verified

FYI Upgrade to proposed worked fine:
root@g:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libffi7 libllvm10 libnvpair1linux libuutil1linux libzfs2linux libzpool2linux
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  apt apt-utils iptables libapt-pkg6.0 libip4tc2 libip6tc2 libvirt-clients libvirt-daemon libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu libvirt-daemon-driver-storage-zfs
  libvirt-daemon-system libvirt-daemon-system-systemd libvirt0 libxtables12 linux-libc-dev
16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 7138 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libapt-pkg6.0 amd64 2.1.11 [848 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 apt amd64 2.1.11 [1305 kB]
Get:3 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 apt-utils amd64 2.1.11 [213 kB]
Get:4 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 iptables amd64 1.8.5-3ubuntu2.20.10.1 [432 kB]
Get:5 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libxtables12 amd64 1.8.5-3ubuntu2.20.10.1 [28.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libip6tc2 amd64 1.8.5-3ubuntu2.20.10.1 [19.1 kB]
Get:7 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libip4tc2 amd64 1.8.5-3ubuntu2.20.10.1 [18.7 kB]
Get:8 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libvirt-daemon-system amd64 6.6.0-1ubuntu3.1 [67.1 kB]
Get:9 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libvirt-clients amd64 6.6.0-1ubuntu3.1 [355 kB]
Get:10 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64 libvirt-daemon-driver-storage-zfs amd64 6.6.0-1ubuntu3.1 [10.7 kB]
Get:11 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64 libvirt-daemon-dr...

[prasket (eight-edge)]

Wanted to confirm that I have installed 6.6.0-1ubuntu3.1 from groovy-proposed and can now create pools!

Thank you everyone for the work to fix this!

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.6.0-1ubuntu3.1

---------------
libvirt (6.6.0-1ubuntu3.1) groovy; urgency=medium

  * d/p/ubuntu/lp-1901242-util-Fix-logic-in-virFileSetCOW.patch: fix dir pool
    handling on non BTRFS affecting virt-manager, api and commandline pool
    handling (LP: #1901242)

 -- Christian Ehrhardt <email address hidden> Wed, 28 Oct 2020 07:47:53 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.