[Ubuntu 20.04] Libvirtd will not pre-create images on the target host during migration

[Dmitry Balashov]

[Impact]

 * On copy-storage-all migration for certain combinations one
   expects the creation of the target storage (it is known to not
   work in all cases)

 * That regressed in Focal and was fixed later via upstream bug
   https://bugzilla.redhat.com/show_bug.cgi?id=1793263

 * This is backporting the same changes to Focal's libvirt

[Test Plan]

1. .The same pool in the both source and target host:
# virsh pool-list
 Name State Autostart
-------------------------------
 default active yes
 pool0 active yes

# virsh pool-dumpxml pool0
<pool type='dir'>
  <name>pool0</name>
  <uuid>a66e307a-cd3c-4164-95f5-7809392912cb</uuid>
  <capacity unit='bytes'>13886460059648</capacity>
  <allocation unit='bytes'>1855581720576</allocation>
  <available unit='bytes'>12030878339072</available>
  <source>
  </source>
  <target>
    <path>/opt/kvm/pool0</path>
    <permissions>
      <mode>0775</mode>
      <owner>64055</owner>
      <group>116</group>
    </permissions>
  </target>
</pool>

2.Show the image of the guest:
# virsh domblklist ubuntu-minimal-18.04
 Target Source
-------------------------------------------------------------
 vda /opt/kvm/pool0/ubuntu-minimal-18.04.hd-system.img

3.Do storage migration:
# virsh migrate --live --persistent --copy-storage-all --verbose ubuntu-minimal-18.04 qemu+ssh://10.10.10.10/system
error: Cannot access storage file '/opt/kvm/pool0/ubuntu-minimal-18.04.hd-system.img': No such file or directory

[Where problems could occur]

 * This is a series that also does some cleanups. It is upstream
   and in EL8 for a while now and seems to be stable, but still
   it changes plenty of code around qemu migration.
   Therefore the area to look out for are all kind of migrations
   done with qemu/kvm.

 * I have considered to backport a smaller subset with less cleanups, but
   it becomes a mess and will eventually be more error prone and less
   reliable.

[Other Info]

 * TBD: I need to check myself how that really looked in Bionic (out
   of curiosity)
 * @SRU Team - I'd appreciate if we could take this into -proposed early
   but release it late.
   Late in this case means the two weeks of my PTO and the time for me
   to run the full regression tests. That extra exposure in -proposed
   should further help to cover potential issues.

---

[System Info]
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Linux hv1 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[Package Info]
libvirt0:
  Installed: 6.0.0-0ubuntu8.8
  Candidate: 6.0.0-0ubuntu8.8
  Version table:
 *** 6.0.0-0ubuntu8.8 500
        500 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     6.0.0-0ubuntu8.3 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     6.0.0-0ubuntu8 500
        500 http://gb.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Steps to Reproduce:
1.The same pool in the both source and target host:
# virsh pool-list
 Name State Autostart
-------------------------------
 default active yes
 pool0 active yes

# virsh pool-dumpxml pool0
<pool type='dir'>
  <name>pool0</name>
  <uuid>a66e307a-cd3c-4164-95f5-7809392912cb</uuid>
  <capacity unit='bytes'>13886460059648</capacity>
  <allocation unit='bytes'>1855581720576</allocation>
  <available unit='bytes'>12030878339072</available>
  <source>
  </source>
  <target>
    <path>/opt/kvm/pool0</path>
    <permissions>
      <mode>0775</mode>
      <owner>64055</owner>
      <group>116</group>
    </permissions>
  </target>
</pool>

2.Show the image of the guest:
# virsh domblklist ubuntu-minimal-18.04
 Target Source
-------------------------------------------------------------
 vda /opt/kvm/pool0/ubuntu-minimal-18.04.hd-system.img

3.Do storage migration:
# virsh migrate --live --persistent --copy-storage-all --verbose ubuntu-minimal-18.04 qemu+ssh://10.10.10.10/system
error: Cannot access storage file '/opt/kvm/pool0/ubuntu-minimal-18.04.hd-system.img': No such file or directory

Actual results:
As step 3,migration failed.

Expected results:
Migrate successfully with pre-creating the image on target pool.

Similar bug in rhel: https://bugzilla.redhat.com/show_bug.cgi?id=1793263

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Hmm,
I thought that is conceptually this way, from the man page of e.g. virsh:

"--copy-storage-all indicates migration with non-shared storage with full disk copy ... In both cases the disk images have to exist on destination host, the --copy-storage-* options only tell libvirt to transfer data from the images on source host to the images found at the same place on the destination host."

This was later improved to work "a bit better" in some cases, and that is the patch series
  https://listman.redhat.com/archives/libvir-list/2020-January/msg01356.html
That the referenced bugzilla refers to.

That patch series is in v6.1.0 and thereby released in Ubuntu >=20.10 which has v6.6.0.

I'm a bit unsure if we should backport this to Focal as it is:
a) AFAIK not a regression more like a new feature
b) the functionality is available in Ubuntu (Groovy and later)
c) If needed for the LTS there also is [1] which some prefer to a full upgrade to a non LTS
d) I haven't looked at the patches yet, but seeing that it is quite a few of them there will certainly be some churn and risk to add them.

To be clear I'm not saying this is undoable, I'm saying maybe the risk outweights the benefit here.
On the positive side, this is from v6.1.0 so there should not be too much noise compared to things that got added later.

I think to give this a chance someone will need to make a compelling case why this, being somewhat of a new feature and the option to upgrade being available will still apply to be a valid SRU [2] under all the SRU rules.

[1]: https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports
[2]: https://wiki.ubuntu.com/StableReleaseUpdates

[Christian Ehrhardt  (paelzer)]

As I assumed the backporting (to make it apply, not in regard to potential regressions) is trivial and was already done here
  https://git.centos.org/rpms/libvirt/blob/c8s-stream-rhel/f/SPECS/libvirt.spec#_286

Maybe we can spin a PPA for you to give it a try if it even achieves what you expect.
And reconsider from there - what do you think?

[Dmitry Balashov (fixed77)]

PPA might be as solution.
but this functionality worked in 18.04, but does not work in 20.04

[Christian Ehrhardt  (paelzer)]

> On Tue, Jul 20, 2021 at 12:55 PM Balashov Dmitry <email address hidden> wrote:
> ...
> but this functionality worked in 18.04, but does not work in 20.04

Really, I didn't expect that - thanks for the info.
I'll need to try that myself - if true then this clearly can be
considered a regression and the backports would fulfill the needs of
the SRU policy.
But I'm super-busy this week and won't get to my own tests, and then
I'm on PTO for a while.

So I beg your pardon in advance to have some delay on this.

[Christian Ehrhardt  (paelzer)]

I found no follow up fixes to those commits in git.
So I have prepared a (very quick and untested) PPA:
  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4629

Could you please let me know if that would for for your case?

[copacule (copacule)]

the ppa works for me & i can confirm it used to work on 18.04

thanks

[Christian Ehrhardt  (paelzer)]

Added an SRU template to this bug and an MP here
  https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/405967

[Christian Ehrhardt  (paelzer)]

@SRU Team - I'd appreciate if we could take this into -proposed early but release it late.
Late in this case means the two weeks of my PTO and the time for me to run the full regression tests. That extra exposure in -proposed should further help to cover potential issues.

[Christian Ehrhardt  (paelzer)]

Uploaded to Focal-unapproved for evaluation by the SRU Team

[Robie Basak (racb)]

Hello Balashov, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dmitry Balashov (fixed77)]

i have tested package from proposed repository. problem is solved.
version: 6.0.0-0ubuntu8.12

[Łukasz Zemczak (sil2100)]

Thank you for the verification fixed77! I will revert the tags to verification-needed as per comment #9, waiting for Christian to perform additional regression testing prior to release of this into -updates.

[Christian Ehrhardt  (paelzer)]

I also had none of the usually tested migrations failing nor did in the time anyone file a bug that proposed would be regressing anything.
Setting verification-done

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.12

---------------
libvirt (6.0.0-0ubuntu8.12) focal; urgency=medium

  * d/p/u/lp-1929202-*: fix pre-creation of images during migration
    (LP: #1929202)

 -- Christian Ehrhardt <email address hidden> Tue, 20 Jul 2021 14:13:56 +0200