After upgrade: "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified"

[Peter Sabaini]

[Impact]

 * New libvirt got more strict in regard to file format specification.
   While this is generally the right approach it causes some issues for
   upgraders that have old image chains now failing.

 * Upstream has added code to relax those checks under a set of conditions
   which will allow to go forward with stricter conditions as planned but
   at the same time not break/block upgrades.

[Test Plan]

 * Thanks to Brett Milford for sharing his test steps for this

sudo apt-get update
sudo apt-get install libvirt-daemon-system cloud-image-utils virtinst -y

IMG="focal-server-cloudimg-amd64.img"
IMG_PATH="/var/lib/libvirt/images/base/$IMG"
INSTANCE_NAME=testinst
[ -f $IMG_PATH ] || {
sudo mkdir -p /var/lib/libvirt/images/base
sudo wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img \
    -O $IMG_PATH
}
sudo mkdir -p /var/lib/libvirt/images/$INSTANCE_NAME
sudo qemu-img convert -O raw $IMG_PATH ${IMG_PATH%.*}
sudo qemu-img create -f qcow2 -o backing_file=${IMG_PATH%.*} /var/lib/libvirt/images/$INSTANCE_NAME/root.img
sudo qemu-img resize /var/lib/libvirt/images/$INSTANCE_NAME/root.img 5G

virt-install --connect qemu:///system --name $INSTANCE_NAME --cpu host --os-type linux --os-variant generic --graphics vnc --console pty,target_type=serial --disk path=/var/lib/libvirt/images/$INSTANCE_NAME/root.img,bus=virtio,format=qcow2 --network default,model=virtio --noautoconsole --vcpus 1 --memory 1024 --import

[Where problems could occur]

 * Of the many things that qemu/libvirt do this changes only the format
   probing. So issues (hopefully not) would be expected to appear mostly
   around complex scenarios of image files.
   We've had a look at image files and image file chains, and so far all
   were good. But there are more obscure (and not supported) cases like
   image backed by real-disk that might misbehave. But still it would
   fix Focal to be the outlier as the past was ok (didn't care) and the
   future (relaxed check) and only focal is left broken in between.

[Other Info]

 * A lot has changes in that area, but instead of pulling in a vast set
   of changes a smaller set was identified to suite the SRU needs. It
   was so far found not found regressing anything and OTOH fixed the issue
   (tested form PPA) for affected people.

----

In a site upgraded to Ussuri we are getting faults starting instances

2020-11-30 13:41:40.586 232871 ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: format of backing image '/var/lib/nova/instances/_base/xxx' of image '/var/lib/nova/instances/xxx' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)

Bug #1864020 reports similar symptoms, where due to an upstream change in Libvirt v6.0.0+ images need the backing format specified.

The fix for Bug #1864020 handles the case for new instances. However, for upgraded instances we're hitting the same problem, as those still don't have backing format specified.

[Lee Yarwood (lyarwood)]

I had thought this failure case had been relaxed in later versions of libvirt 6.

Which version of libvirt are you using?

[Lee Yarwood (lyarwood)]

Ah apologies it's still there in master:

https://github.com/libvirt/libvirt/blob/84dc367e2a6febfcc4aeea4c52e44c52aea298fc/src/util/virstoragefile.c#L5265-L5270

So we will need to workaround this within the libvirt driver using the following:

https://libvirt.org/kbase/backing_chains.html#vm-refuses-to-start-due-to-misconfigured-backing-store-format

This should be easy enough to wire up in the qcow2 imagebackend during cache() assuming it doesn't impact already running domains/instances using the backing file.

[Lee Yarwood (lyarwood)]

Ignore c#2, stephenfin correctly points out that the following landed in 6.1.0 relaxing this:

https://github.com/libvirt/libvirt/commit/ae9e6c2a2b75d958995c661f7bb64ed4353a6404

We shouldn't need to rebase existing disks without a backing file format as a result.

[Stephen Finucane (stephenfinucane)]

Given the above, the solution here seems to be to update your version of libvirt to >= 6.1.0. I'm going to mark this as WONTFIX. If this does not resolve the issue, please reset to new and provide information on the version of libvirt you've tested with and detailed logs from nova-compute showing the error.

[Peter Sabaini (peter-sabaini)]

It appears ubuntu bionic/focal ship with libvirt < 6.1.0, marking nova as affected in those distribs

[Felipe Reyes (freyes)]

Adding a task for libvirt at Ubuntu/focal since this patch[0] might need to be backported.

[0] https://github.com/libvirt/libvirt/commit/ae9e6c2a2b75d958995c661f7bb64ed4353a6404

[Christian Ehrhardt  (paelzer)]

Thanks everyone, I agree that we'd want to consider this for Focals libvirt.
But there is a quite excessive series of changes in v6.0.0 -> v6.1.0 before the referenced change took place. So it will either be a massive backport (picking much more related changes to cherry-pick) or have a rather big set of backporting changes.

I wanted to ask if out of all your testing above there is already a simpler testcase that one can use without nova and all such. Just a) how to create a broken image + b) how to make libvirt stumble over it.
We will need this for the SRU process anyway if we want to bring it to Focal.

Also for severity - since didn't have this show up in any other place yet I wonder how common this case would be? Any comment on that?

For the time being (while discussing) I have tried to rather "bluntly" backport the change.
If it builds would you be able to try this PPA [1] in your existing test environment?
If it works then MP [2] could be a simpler change than what I was afraid it could be.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4388/+packages
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778

[Alex Kavanagh (ajkavanagh)]

Hi Christian,

I've tested the patch in your [2] ([2] below) and it worked perfectly. I got a colleague to rebuild it for me against the current focal libvirt [1] as I couldn't find your PPA mentioned. I'm upgrading an OpenStack system from bionic-ussuri -> focal-ussuri, and confirmed that the instance wouldn't start with 6.0.0-0ubuntu8.9. I then upgraded to 6.0.0-0ubuntu8.10.1~ubuntu20.04.1~ppa202106240958 from the PPA [1] and the instanced started.

Is this sufficient to get it included in the focal archives?

Many thanks for your work on this.

[1] https://launchpad.net/~chris.macnaughton/+archive/ubuntu/focal-ussuri/
[2] https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778

[Christian Ehrhardt  (paelzer)]

Thanks Alex,
I'll rebase and polish it and hand it over to a team member for review then.

But what I'd still need is some written out test steps for the SRU process.
Since you have just verified it would you mind adding a statement here that I could copy into the SRU template then?

[Christian Ehrhardt  (paelzer)]

I have rebased and finalized the MP for this:
https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/395778
This is now open for review by the Team.

Furthermore I have opened a new PPA to have up-to-date test builds.
new PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4591/+packages

@Alex - we will need a proper SRU Template and I could only talk about the patch, but not to much about the testing. I'd appreaciate if you could add an SRU template (https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template) to the bug description.

[Christian Ehrhardt  (paelzer)]

FYI it seems bug 1864020 talks about the same case.

[Brett Milford (brettmilford)]

Ran the test plan on focal which produced the following error:

ERROR Requested operation is not valid: format of backing image '/var/lib/libvirt/images/base/focal-server-cloudimg-amd64' of image '/var/lib/libvirt/images/testinst/root.img' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start testinst
otherwise, please restart your installation.

Ran the test plan on focal with libvirt packages from ppa:ci-train-ppa-service/4591.
The domain booted successfully.

[Christian Ehrhardt  (paelzer)]

Thanks Brett, that confirms they are the same root cause and fixed by the same solution.
Thanks.
And it also confirms that the test steps I've got from you will apply here.

[Robie Basak (racb)]

MP approved for upload and subsequent SRU accept.

Given this relaxes the previous commit https://libvirt.org/git/?p=libvirt.git;a=commit;h=3615e8b39bad, would there be value in additionally testing during SRU verification that libvirt correctly still rejects what it should, as well as that this bug is fixed as you've already documented?

[Christian Ehrhardt  (paelzer)]

FYI: I uploaded the fix to Focal-unapproved for verification by the SRU Team.

[Christian Ehrhardt  (paelzer)]

"additionally testing during SRU verification that libvirt correctly still rejects what it should"

I can try then, but I'm not 100% sure I can easily recreate that case.
If I can then sure, if not we should not block on it IMHO as the backported code matches the >=Groovy behavior anyway.

[Robie Basak (racb)]

Hello Peter, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brett Milford (brettmilford)]

Hi @racb,

I have tested:

libvirt-daemon-system:
  Installed: 6.0.0-0ubuntu8.11
  Candidate: 6.0.0-0ubuntu8.11
  Version table:
 *** 6.0.0-0ubuntu8.11 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

As per the test plan.
This package fixed the bug for me.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.11

---------------
libvirt (6.0.0-0ubuntu8.11) focal; urgency=medium

  * d/p/ubuntu/lp-1906266-virStorageFileGetMetadataRecurse-Allow-format-probin:
    relax restrictions on format probing for compat with older images
    (LP: #1906266)

 -- Christian Ehrhardt <email address hidden> Tue, 05 Jan 2021 13:48:48 +0100

[Corey Bryant (corey.bryant)]

Hello Peter, or anyone else affected,

Accepted libvirt into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

[Brett Milford (brettmilford)]

Hi @corey.bryant

I have tested:

libvirt-daemon-system:
  Installed: 6.0.0-0ubuntu8.11~cloud0
  Candidate: 6.0.0-0ubuntu8.11~cloud0
  Version table:
 *** 6.0.0-0ubuntu8.11~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status

As per the test plan.
This package fixed the bug for me.

[Joris Vleminckx (jvleminc)]

I can confirm that installing a newer version of libvirt fixes the issue.

We just did this for an nova-libvirt container in the Openstack Victoria release, wich we had upgraded from Rocky.

We used the latest available libvirt version for Ubuntu 20.04: 6.0.0-0ubuntu8.12