Unable to remove disk metadata on vm

[Riccardo Pittau]

[Impact]

 * The impact is log flooding with warnings for every guest start/stop

 * Maybe the impact also has in some cases worse effects due to the bad RC
   of the function e.g. blocking other actions - that could be happening in
   the CI run it was reported with.

 * Upstream has added a fix that fixes the warning.
   We worked on and upstreamed another fix that also corrects the internal
   error we were seeing.
   Both are easily backportd and apply as-is.

[Test Case]

 * Run this in one console
     $ journalctl -f -u libvirtd | grep -i meta
 * In the other console spawn a guest (e.g. via uvtool-libvirt or
   any other and shut it down. Shutdown will pass:
    qemuBlockRemoveImageMetadata ->
      qemuSecurityMoveImageMetadata
   ... which will trigger the issue.

   Without a fix the log will all the time get an entry like:

Mai 20 07:10:12 Keschdeichel libvirtd[2638]: this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: internal error: child reported (status=125): this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/focal.qcow (disk target vda)
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: internal error: child reported (status=125): this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjAuMDQ6YW1kNjQgMjAyMDAzMjk= (disk target vda)
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: internal error: child reported (status=125): this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:10:12 Keschdeichel libvirtd[2638]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/focal-ds.qcow (disk target vdb)

[Regression Potential]

 * It is a check on a function that doesn't need to exist for apparmor.
   Hence making that check not a fail does not have a huge regression
   potential - it is not that "now" it would do more. It just no more
   complains about it and thereby avoids log flooding.
   Regressions could happen if we'd have silenced other warnings by that,
   but I don't see that in code or tests.
 * The other change converts a bad RC into a good RC for a given set of
   condition that applies either when built without libattr or when working
   on an FS that does not support XATTR.
   The same change of behavior we have in Ubuntu (built without libattr)
   would now also be dependent on the FS type (no error on such FS
   anymore). But that isn't true for Ubuntu builds and therefore doesn't
   matter for the SRU considerations.
   A regression could be if there would be another low level fail that is
   mis-detected and masked to be "ok" by the code. But the API is rather
   clear on -1 = fail, -2 = the kind we mask 0 = ok. So this seems not to
   be an issue.

[Other Info]

 * n/a

---

We can reproduce this 100% in our CI (upstream Openstack Ironic)
using Ubuntu 20.04 LTS Focal Fossa with libvirt 6.0.0

2020-05-15 10:05:50.626+0000: 96089: error : virSecurityManagerMoveImageMetadata:476 : this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
2020-05-15 10:05:50.628+0000: 96089: error : virProcessRunInFork:1159 : internal error: child reported (status=125): this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
2020-05-15 10:05:50.628+0000: 96089: warning : qemuBlockRemoveImageMetadata:2629 : Unable to remove disk metadata on vm node-0 from /var/lib/libvirt/images/node-0.qcow2 (disk target vda)

complete libvirt logs:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_688/716889/26/check/ironic-tempest-ipa-partition-uefi-pxe-grub2/688814a/controller/logs/libvirt/libvirtd_log.txt

This was filed upstream and fixed by:
https://gitlab.com/libvirt/libvirt/-/commit/cc8c297e473afd55e5d8e35e18345d8df176059d

[Christian Ehrhardt  (paelzer)]

Note: related upstream issue https://gitlab.com/libvirt/libvirt/-/issues/25

Hi Riccardo,
thanks for the report and the discussion until there was a fix.

I have prepared PPAs for Groovy and Focal including that fix for you to try if that change alone is really sufficient for your case as well. If you could give these a try and let me know?

[Christian Ehrhardt  (paelzer)]

Added an SRU template in preparation.

@Riccardo - in addition to trying the PPA, would you let me know if in your environment this is "just the warning flooding the log" or does it also have more severe impact like a function failing?
If so does the PPA only fix the warning or also the formerly failing function?

[Christian Ehrhardt  (paelzer)]

Incomplete for now waiting on user reply in regard to the PPA fix.

[Christian Ehrhardt  (paelzer)]

I tested the PPA and it works to eliminate the warnings.

No more appear:
Mai 20 07:24:40 Keschdeichel libvirtd[2638]: this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata
Mai 20 07:24:40 Keschdeichel libvirtd[2638]: internal error: child reported (status=125): this function is not supported by the connection driver: virSecurityManagerMoveImageMetadata

But matching the patch description, that doesn't change much as the actual metadata removal still isn't done (which might be fine)

So on shutdown I still see:
Mai 20 11:33:27 Keschdeichel libvirtd[981209]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/focal.qcow (disk target vda)
Mai 20 11:33:27 Keschdeichel libvirtd[981209]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjAuMDQ6YW1kNjQgMjAyMDAzMjk= (disk target vda)
Mai 20 11:33:27 Keschdeichel libvirtd[981209]: Unable to remove disk metadata on vm focal from /var/lib/uvtool/libvirt/images/focal-ds.qcow (disk target vdb)

The questions now are:
- did it remove the warning you as well?
- is this enough to help?

We might need to split the actual issue to remove metadata (not needed I guess) to another bug.
But lets complete this one first to be sure.

@Ricardo - for your checks the PPA is at
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4068

MPs (preliminary):
Groovy: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/384243
Focal: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/384244

[Riccardo Pittau (rpittau)]

Hi Christian,
thanks for the quick answer and fix, I'll test that in our CI as soon as possible.

What actually worries me is the message "unable to remove disk metadata", but I guess we'll have to wait for the CI jobs to complete to see if there was an improvement on the related error that we're seeing in our application, that is actually modifying the disk partitions.
It might be a coincidence (the time stamps coincide) or hopefully it will help.

I'll post an update as soon as I'm done testing.

[Riccardo Pittau (rpittau)]

Hi Christian,
thank you for your patience, I just finished testing the new package in our ci and unfortunately I'm still seeing the same message about disk metadata:
2020-05-22 09:21:40.896+0000: 96720: error : virProcessRunInFork:1159 : internal error: child reported (status=125):
2020-05-22 09:21:40.896+0000: 96720: warning : qemuBlockRemoveImageMetadata:2629 : Unable to remove disk metadata on vm node-0 from /var/lib/libvirt/images/node-0.qcow2 (disk target vda)

Here you can find the complete libvirt logs:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f9b/716889/29/check/ironic-tempest-ipa-partition-uefi-pxe-grub2/f9b3f6c/controller/logs/libvirt/libvirtd_log.txt

These are the new installed packages:
2020-05-22 08:56:30.550878 | controller | Get:1 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-dev amd64 6.0.0-0ubuntu8.1 [161 kB]
2020-05-22 08:56:31.041548 | controller | Get:2 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-driver-qemu amd64 6.0.0-0ubuntu8.1 [605 kB]
2020-05-22 08:56:31.707272 | controller | Get:3 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-system amd64 6.0.0-0ubuntu8.1 [67.5 kB]
2020-05-22 08:56:32.121898 | controller | Get:4 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-clients amd64 6.0.0-0ubuntu8.1 [344 kB]
2020-05-22 08:56:32.712179 | controller | Get:5 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt0 amd64 6.0.0-0ubuntu8.1 [1,445 kB]
2020-05-22 08:56:33.484721 | controller | Get:6 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon amd64 6.0.0-0ubuntu8.1 [404 kB]
2020-05-22 08:56:34.094783 | controller | Get:7 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-system-systemd amd64 6.0.0-0ubuntu8.1 [12.5 kB]

the libvirtd process has been restarted after the installation

These are the complete job logs:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f9b/716889/29/check/ironic-tempest-ipa-partition-uefi-pxe-grub2/f9b3f6c/job-output.txt

Thank you again for your help!

[Christian Ehrhardt  (paelzer)]

Yeah those two you still see, that was expected.
The fix you referred to only eliminated the former messages in regard to "this function is not supported"

I agree in my repro that the following two kind of messages stay (for now):
  internal error: child reported (status=125):
  Unable to remove disk metadata on vm testguest from /var/lib/uvtool/libvirt/images/testguest.qcow (disk target vda)

The error message buffer gets extended, so in my case I have three images so eventually it is:
  internal error: child reported (status=125): internal error: child reported (status=125): internal error: child reported (status=125):

Therefore applying the fix you pointed me do was step #1 of a yet unknown journey.

[Christian Ehrhardt  (paelzer)]

I was debugging the case in gdb to check what exactly fail
qemuBlockRemoveImageMetadata (the one that emits the warning)
 -> qemuSecurityMoveImageMetadata
   -> virSecurityManagerMoveImageMetadata (where the former function was missing)

The first hit is the stacked security, which has an implementation:

Thread 18 "libvirtd" hit Breakpoint 3, virSecurityManagerMoveImageMetadata (mgr=0x7f8c1800fe10, pid=-1, src=src@entry=0x7f8c4c044440, dst=dst@entry=0x0)
    at ../../../src/security/security_manager.c:467
warning: Source file is more recent than executable.
467 {
(gdb) p mgr->drv->domainMoveImageMetadata
$1 = (virSecurityDomainMoveImageMetadata) 0x7f8c638a47e0 <virSecurityStackMoveImageMetadata>

This essentially calls itself in the lower stack (=apparmor) and there it is 0x0 as expected.

Thread 18 "libvirtd" hit Breakpoint 3, virSecurityManagerMoveImageMetadata (mgr=0x7f8c18047f10, pid=pid@entry=-1, src=src@entry=0x7f8c4c044440, dst=dst@entry=0x0)
    at ../../../src/security/security_manager.c:467
467 {
(gdb) n
468 if (mgr->drv->domainMoveImageMetadata) {
(gdb) p mgr->drv->domainMoveImageMetadata
$2 = (virSecurityDomainMoveImageMetadata) 0x0

Then there is DAC

(gdb) p mgr->drv->domainMoveImageMetadata
$3 = (virSecurityDomainMoveImageMetadata) 0x7f8c638a5570 <virSecurityDACMoveImageMetadata>

Calling into:
virSecurityDACMoveImageMetadata (mgr=0x7f8c1805cf10, pid=-1, src=0x7f8c4c044440, dst=0x0) at ../../../src/security/security_dac.c:1122
warning: Source file is more recent than executable.

This is more promising as we see in the errors "subprocess fail" and DAC uses those.

(gdb) bt
#0 virSecurityDACMoveImageMetadata (mgr=0x7f8c1805cf10, pid=-1, src=0x7f8c4c044440, dst=0x0) at ../../../src/security/security_dac.c:1122
#1 0x00007f8c638a8ecf in virSecurityManagerMoveImageMetadata (mgr=0x7f8c1805cf10, pid=pid@entry=-1, src=src@entry=0x7f8c4c044440, dst=dst@entry=0x0)
    at ../../../src/security/security_manager.c:471
#2 0x00007f8c638a4820 in virSecurityStackMoveImageMetadata (mgr=<optimized out>, pid=-1, src=0x7f8c4c044440, dst=0x0) at ../../../src/security/security_stack.c:705
#3 0x00007f8c638a8ecf in virSecurityManagerMoveImageMetadata (mgr=0x7f8c1800fe10, pid=-1, src=src@entry=0x7f8c4c044440, dst=dst@entry=0x0) at ../../../src/security/security_manager.c:471
#4 0x00007f8c45938708 in qemuSecurityMoveImageMetadata (driver=driver@entry=0x7f8c18052c30, vm=vm@entry=0x7f8c180c4110, src=src@entry=0x7f8c4c044440, dst=dst@entry=0x0)
    at ../../../src/qemu/qemu_security.c:182
#5 0x00007f8c458630d0 in qemuBlockRemoveImageMetadata (driver=driver@entry=0x7f8c18052c30, vm=vm@entry=0x7f8c180c4110, diskTarget=0x7f8c4c043cd0 "vda", src=<optimized out>)
    at ../../../src/qemu/qemu_block.c:2628
#6 0x00007f8c458ca13b in qemuProcessStop (driver=driver@entry=0x7f8c18052c30, vm=vm@entry=0x7f8c180c4110, reason=reason@entry=VIR_DOMAIN_SHUTOFF_SHUTDOWN,
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, flags=flags@entry=0) at ../../../src/qemu/qemu_process.c:7568
#7 0x00007f8c45935fd9 in processMonitorEOFEvent (vm=0x7f8c180c4110, driver=0x7f8c18052c30) at ../../../src/qemu/qemu_driver.c:4792
#8 qemuProcessEventHandler (data=0x562f8f...

[Christian Ehrhardt  (paelzer)]

It calls:
virProcessRunInFork for virSecurityDACMoveImageMetadataHelper

And I was able to confirm that this is the "remaining" source of a negative return code that was formerly hidden behind the missing function.

Thread 18 "libvirtd" hit Breakpoint 4, virSecurityDACMoveImageMetadata (mgr=0x7f8c1805cf10, pid=-1, src=0x7f8c4803fbf0, dst=0x0) at ../../../src/security/security_dac.c:1122
1122 {
(gdb) fin
Run till exit from #0 virSecurityDACMoveImageMetadata (mgr=0x7f8c1805cf10, pid=-1, src=0x7f8c4803fbf0, dst=0x0) at ../../../src/security/security_dac.c:1122
[Detaching after fork from child process 14055]
0x00007f8c638a8ecf in virSecurityManagerMoveImageMetadata (mgr=0x7f8c1805cf10, pid=pid@entry=-1, src=src@entry=0x7f8c4803fbf0, dst=dst@entry=0x0)
    at ../../../src/security/security_manager.c:471
471 ret = mgr->drv->domainMoveImageMetadata(mgr, pid, src, dst);
Value returned is $11 = -1

Checking for behavior in virProcessRunInFork and virSecurityDACMoveImageMetadataHelper next ...

[Christian Ehrhardt  (paelzer)]

1129 if (!priv->dynamicOwnership)
is set

1132 if (src && virStorageSourceIsLocalStorage(src))
is true and sets
(gdb) p data.src
$18 = 0x7f8c500271e0 "/var/lib/uvtool/libvirt/images/testguest.qcow"

1135 if (dst && virStorageSourceIsLocalStorage(dst))
isn't true as the destination is a NULL pointer.

For anyone wondering if dst=0x0 is correct. That is matchign the shutdown process, since
virSecurityManagerMoveImageMetadata defines it as:

 451 * If @dst is NULL then metadata is removed from @src and not
 452 * stored anywhere.

In virProcessRunInFork I see
1155 ret = virProcessWait(child, &status, false);
placing the
(gdb) p status
$23 = 125

[Christian Ehrhardt  (paelzer)]

Since that is a fork this needs
(gdb) set detach-on-fork off
(gdb) set follow-fork-mode child
To be able to track the virSecurityDACMoveImageMetadataHelper

Inside of virSecurityDACMoveImageMetadataHelper we have
(gdb) p *data
$27 = {mgr = 0x7f8c1805cf10, src = 0x7f8c50031a10 "/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjAuMDQ6YW1kNjQgMjAyMDA1MjA=", dst = 0x0}

That in turn runs virSecurityMoveRememberedLabel with
Thread 2.1 "libvirtd" hit Breakpoint 9, virSecurityMoveRememberedLabel (name=name@entry=0x7f8c63a2504f "dac",
    src=0x7f8c50031a10 "/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjAuMDQ6YW1kNjQgMjAyMDA1MjA=", dst=0x0) at ../../../src/security/security_util.c:448

[Christian Ehrhardt  (paelzer)]

At -O2 the function seems to be inlined which makes debugging a bit harder.

This is the function returning the bad RC, but not only itself but also most of its callees are inlined virSecurityGetRefCountAttrName virSecurityGetAttrName virSecurityGetTimestampAttrName.

461 if (virFileGetXAttrQuiet(src, ref_name, &ref_value) < 0) {
(gdb) n
462 if (errno == ENOSYS || errno == ENOTSUP) {
(gdb) p errno
$32 = 38

Which matches ENOSYS
#define ENOSYS 38 /* Function not implemented */

So either DAC (in the apparmor context) doesn't use XATTR either an fails on still trying (similar to [1] which removed trying to do so for apparmor) or we have an issue with XATTR in the DAC implementation.

[1]: https://gitlab.com/libvirt/libvirt/-/commit/cc8c297e473afd55e5d8e35e18345d8df176059d

[Christian Ehrhardt  (paelzer)]

I reach the upper virFileGetXAttrQuiet

Thread 3.1 "libvirtd" hit Breakpoint 10, virFileGetXAttrQuiet (path=path@entry=0x7f8c50041390 "/var/lib/uvtool/libvirt/images/testguest-ds.qcow",
    name=name@entry=0x7f8c1803dc40 "trusted.libvirt.security.ref_dac", value=value@entry=0x7f8c29b605e0) at ../../../src/util/virfile.c:4418

And it is the implementation of

4412 #else /* !HAVE_LIBATTR */
4413
4414 int
4415 virFileGetXAttrQuiet(const char *path G_GNUC_UNUSED,
4416 const char *name G_GNUC_UNUSED,
4417 char **value G_GNUC_UNUSED)
4418 {
4419 errno = ENOSYS;
4420 return -1;
4421 }

which is our ENOSYS.

There are two issues with that.
#1 - why do we have !HAVE_LIBATTR
#2 - I think never having set any XATTR the driver shouldn't bail out unable to remove them

[Christian Ehrhardt  (paelzer)]

I've checked some releases back and we build with:
  --without-attr
since what seems forever.
Maybe the code can't handle !HAVE_LIBATTR as good as the old did.

And configure confirms showing
  configure: attr: no
throughout all the builds.

That theory LGTM as the metadata handling that now fails didn't exist back in e.g. Bionic.
[1][2] are >=5.6 which matches >=Focal where we see these issues.

I'll need to report that to Michael who authored the initial offending change as well as the first fix you referred to me initially. I'd prefer a proper upstream fix by him to a spontaneous change at only the Ubuntu code.

[1]: https://gitlab.com/libvirt/libvirt/-/commit/706e68237f5
[2]: https://gitlab.com/libvirt/libvirt/-/commit/d73f3f58360

[Christian Ehrhardt  (paelzer)]

To be clear on the severity, the change won't make the meta data now being applied. We never did and we don't need that. But that failing un-labelling does flood the error logs and it's bad return code might cause other steps on e.g. shutdown to be skipped.

[Christian Ehrhardt  (paelzer)]

Replied to the referred fix describing what happens here outlining that the overall case isn't fixed yet.
=> https://www.redhat.com/archives/libvir-list/2020-May/msg01067.html

[Christian Ehrhardt  (paelzer)]

Discussion went on until https://www.redhat.com/archives/libvir-list/2020-May/msg01088.html
Fix submitted as https://www.redhat.com/archives/libvir-list/2020-May/msg01089.html and waiting for feedback.

[Christian Ehrhardt  (paelzer)]

5243d34b4f (paelzer/fix-lp-1879325-metadata-FOCAL, fix-lp-1879325-metadata-FOCAL) changelog: avoid issues with apparmor metadata labeling (LP: #1879325)
ad0478c226

I have put a version with that fix in the PPA [1] for
Focal: 6.0.0-0ubuntu8.2~ppa1
Groovy: 6.0.0-0ubuntu10~ppa1

That way I can retest while waiting for upstream feedback on the change.

@Riccardo - could you also retest from that PPA please?

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4068

[Christian Ehrhardt  (paelzer)]

FYI - The commits mentioned above were a copy&paste accident - they are not important to be mentioned.

[Christian Ehrhardt  (paelzer)]

Got a Review-by already, will wait another 24h or so before pushing and preparing a build with it.

[Christian Ehrhardt  (paelzer)]

In my testing all errors of the following kind are gone now:
- internal error: child reported (status=125)
- Unable to remove disk metadata on vm testguest from /var/lib/uvtool/libvirt/images/testguest.qcow (disk target vda)

@Riccardo - does this make your CI happy as well?

[Christian Ehrhardt  (paelzer)]

The extra change we need is now pushed upstream
=> https://libvirt.org/git/?p=libvirt.git;a=commit;h=55029d93150e33d70b02b6de2b899c05054c5d3a

My PPA builds include this final version of the change and the initially referred fix for the warning.
Waiting for MP review by the team now.

@Riccardo - this certainly fixes "an issue" I'd still be happy to hear from you if that also fixes your CI fails.

[Riccardo Pittau (rpittau)]

Hi Christian,

I ran a test in our CI with the packages:
2020-05-26 16:52:02.258571 | controller | Get:1 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-dev amd64 6.0.0-0ubuntu8.2~ppa1 [161 kB]
2020-05-26 16:52:02.486631 | controller | Get:2 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-driver-qemu amd64 6.0.0-0ubuntu8.2~ppa1 [606 kB]
2020-05-26 16:52:02.806459 | controller | Get:3 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-system amd64 6.0.0-0ubuntu8.2~ppa1 [67.5 kB]
2020-05-26 16:52:03.206779 | controller | Get:4 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-clients amd64 6.0.0-0ubuntu8.2~ppa1 [344 kB]
2020-05-26 16:52:03.473402 | controller | Get:5 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt0 amd64 6.0.0-0ubuntu8.2~ppa1 [1,447 kB]
2020-05-26 16:52:03.857247 | controller | Get:6 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon amd64 6.0.0-0ubuntu8.2~ppa1 [404 kB]
2020-05-26 16:52:04.151046 | controller | Get:7 http://ppa.launchpad.net/ci-train-ppa-service/4068/ubuntu focal/main amd64 libvirt-daemon-system-systemd amd64 6.0.0-0ubuntu8.2~ppa1 [12.5 kB]

The message "unable to remove disk metadata" is gone from the logs, but unfortunately I'm still seeing the same error as before in our app, so it's probably something else that is going sideways, I'll have to dig deeper on that.
We're going from bionic to focal and there are more things that probably need to be changed/adjusted.

I think this can be solved anyway as the original issue related to libvirt is gone.

Thanks for the time spent and the help!

[Christian Ehrhardt  (paelzer)]

Ok Riccardo, could you - once you find the other issue - file a new bug then.
I'll take this bug here to drive the issue(s) identified to far to a conclusion.

[Christian Ehrhardt  (paelzer)]

FYI: Fix uploaded to groovy

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu9

---------------
libvirt (6.0.0-0ubuntu9) groovy; urgency=medium

  * d/p/ubuntu/lp-1879325-*: avoid issues with apparmor metadata labeling
    (LP: #1879325)

 -- Christian Ehrhardt <email address hidden> Wed, 20 May 2020 06:59:57 +0200

[Christian Ehrhardt  (paelzer)]

Now that the fix migrated in groovy I uploaded it to focal-unapproved for the consideration by the SRU Team.

[Brian Murray (brian-murray)]

Hello Riccardo, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Riccardo Pittau (rpittau)]

Hey Brian,

I tested the new packages in our CI and I can confirm the original issues are both gone.
This is a recent libvirt log file:
https://1a06a6c5517c6a746e50-e807054499ce854694349e023b0ab289.ssl.cf5.rackcdn.com/716889/38/check/ironic-tempest-ipa-partition-uefi-pxe-grub2/a34d5b6/controller/logs/libvirt/libvirtd_log.txt

I used the focal proposed repo:
deb http://archive.ubuntu.com/ubuntu/ focal-proposed restricted main multiverse universe

Upgrade command:
sudo apt-get -y upgrade libvirt-daemon-system libvirt-dev libvirt-clients libvirt-daemon libvirt-daemon-driver-qemu libvirt-daemon-system-systemd libvirt0

List of upgraded packages including dependencies with versions:
liblzma5 amd64 5.2.4-1ubuntu1
libapt-pkg6.0 amd64 2.0.3
apt amd64 2.0.3
gir1.2-freedesktop amd64 1.64.1-1~ubuntu20.04.1
libgirepository-1.0-1 amd64 1.64.1-1~ubuntu20.04.1
gir1.2-glib-2.0 amd64 1.64.1-1~ubuntu20.04.1
xz-utils amd64 5.2.4-1ubuntu1
apt-transport-https all 2.0.3
busybox amd64 1:1.30.1-4ubuntu6.1
initramfs-tools-core all 0.136ubuntu6.1
initramfs-tools all 0.136ubuntu6.1
initramfs-tools-bin amd64 0.136ubuntu6.1
busybox-initramfs amd64 1:1.30.1-4ubuntu6.1
grub-efi-amd64-signed amd64 1.142.1+2.04-1ubuntu26
libproxy1v5 amd64 0.4.15-10ubuntu1
libvirt-dev amd64 6.0.0-0ubuntu8.1
libvirt-daemon-driver-qemu amd64 6.0.0-0ubuntu8.1
libvirt-daemon-system amd64 6.0.0-0ubuntu8.1
libvirt-clients amd64 6.0.0-0ubuntu8.1
libvirt0 amd64 6.0.0-0ubuntu8.1
libvirt-daemon amd64 6.0.0-0ubuntu8.1
libvirt-daemon-system-systemd amd64 6.0.0-0ubuntu8.1
linux-libc-dev amd64 5.4.0-34.38
qemu amd64 1:4.2-3ubuntu6.2
qemu-utils amd64 1:4.2-3ubuntu6.2
qemu-system-common amd64 1:4.2-3ubuntu6.2
qemu-block-extra amd64 1:4.2-3ubuntu6.2
qemu-system-data all 1:4.2-3ubuntu6.2
qemu-kvm amd64 1:4.2-3ubuntu6.2
qemu-system-x86 amd64 1:4.2-3ubuntu6.2
qemu-system-arm amd64 1:4.2-3ubuntu6.2
qemu-system-mips amd64 1:4.2-3ubuntu6.2
qemu-system-ppc amd64 1:4.2-3ubuntu6.2
qemu-system-sparc amd64 1:4.2-3ubuntu6.2
qemu-system-s390x amd64 1:4.2-3ubuntu6.2
qemu-system-misc amd64 1:4.2-3ubuntu6.2
qemu-system amd64 1:4.2-3ubuntu6.2
rabbitmq-server all 3.8.2-0ubuntu1.1

I noticed another issue but I'm sure it was not introduced by the upgrades, it looks like no suitable block device to install an operating system can be found.
I'm going to open a different bug for that as soon as I have more concrete info.

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.1

---------------
libvirt (6.0.0-0ubuntu8.1) focal; urgency=medium

  * d/p/ubuntu/lp-1879325-*: avoid issues with apparmor metadata labeling
    (LP: #1879325)

 -- Christian Ehrhardt <email address hidden> Wed, 20 May 2020 06:59:57 +0200

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.