I thank you for all your work.

I have since moved away from this block architecture and am no longer able
to verify with an existing configuration.



On Fri, Nov 22, 2019 at 6:36 AM Timo Aaltonen <email address hidden> wrote:

> Hello Patrick, or anyone else affected,
>
> Accepted libvirt into xenial-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.29 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package.  See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed.  Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested and change the tag from
> verification-needed-xenial to verification-done-xenial. If it does not
> fix the bug for you, please add a comment stating that, and change the
> tag to verification-failed-xenial. In either case, without details of
> your testing we will not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: libvirt (Ubuntu Xenial)
>        Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed verification-needed-xenial
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1681839
>
> Title:
>   libvirt: blockcommit fails - disk not ready for pivot yet
>
> Status in libvirt package in Ubuntu:
>   Fix Released
> Status in libvirt source package in Xenial:
>   Fix Committed
> Status in libvirt source package in Artful:
>   Fix Released
> Status in libvirt source package in Bionic:
>   Fix Released
>
> Bug description:
>   [Impact]
>
>   On xenial, if you manually invoke blockcommit through virsh in
>   libvirt, the command immediately fails with blockcommit supposedly
>   being 100%, and that the disk is not ready for pivot yet:
>
>   root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
> --pivot --wait
>   Block commit: [100 %]
>   error: failed to pivot job for disk vda
>   error: block copy still active: disk 'vda' not ready for pivot yet
>
>   However, if you look at the status of the active blockjob, we see that
>   the blockcommit is still active in the background:
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [0 %]
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [2 %]
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [6 %]
>
>   This happens until it reaches 100%, where it gets stuck. To un-stick
>   things, you must then manually --abort the blockjob.
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [100 %]
>
>   This happens in VMs which are experiencing load, and is caused by a
>   race condition in libvirt. Users are not able to commit their
>   snapshots to disk.
>
>   [Test Case]
>
>   Credit goes to Fabio Martins, who determined how to reproduce this
>   issue.
>
>   On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:
>
>   1) Create a VG and define a LVM pool:
>
>   root@xenial-apparmor:~# cat lvmpool.xml
>   <pool type="logical">
>   <name>LVMpool_vg</name>
>   <source>
>   <device path="/dev/sdb"/>
>   </source>
>   <target>
>   <path>/dev/LVMpool_vg</path>
>   </target>
>   </pool>
>
>   # virsh pool-define lvmpool.xml
>   # virsh pool-start LVMpool_vg
>   # virsh pool-autostart LVMpool_vg
>
>   2) Create a config file to use as a cdrom device with the new VM (will
>   be created in next steps), just to inject a password with cloud-init:
>
>   # cat > config <<EOF
>   > #cloud-config
>   > password: passw0rd
>   > chpasswd: { expire: False }
>   > ssh_pwauth: True
>   > EOF
>
>   # apt install cloud-image-utils
>
>   # cloud-localds config.img config
>
>   # mv config.img /var/lib/libvirt/images/
>   # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
>   # chmod 664 /var/lib/libvirt/images/config.img
>
>   3) Create one VM using this pool:
>
>   # virt-install --connect=qemu:///system --name snapvm --ram 2048
>   --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio
>   --disk /var/lib/libvirt/images/config.img,device=cdrom --network
>   network=kvm-br0 --graphics none --import --noautoconsole
>
>   4) Stop the VM
>
>   # virsh destroy snapvm
>
>   5) Download a Ubuntu cloud image, convert to raw and restore it into
>   the LV used as a disk by our VM:
>
>   # wget
> https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
>   # qemu-img convert ./bionic-server-cloudimg-amd64.img
> ./bionic-server-cloudimg-amd64.raw
>   # dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm
> bs=8M conv=sparse
>
>   6) Start the VM and connect to it in another window
>
>   # virsh start snapvm
>
>   7) Check that the VM is using the LV as the disk:
>
>   root@xenial-apparmor:~# virsh domblklist snapvm
>   Target Source
>   ------------------------------------------------
>   vda /dev/LVMpool_vg/snapvm
>   hda /var/lib/libvirt/images/config.img
>
>   8) Create a snapshot and check that the new domblklist points to the
>   snapshot file:
>
>   # virsh snapshot-create-as --domain snapvm --diskspec
>   vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external
>   --disk-only --atomic
>
>   root@xenial-apparmor:~# virsh domblklist snapvm
>   Target Source
>   ------------------------------------------------
>   vda /var/lib/libvirt/images/xenial-snapvm.qcow2
>   hda /var/lib/libvirt/images/config.img
>
>   9) Connect to your VM and start an I/O intensive job. In this case I'm
>   starting a 'dd' writing zeroes to a file until it gets to 10GBs:
>
>   ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=10240000
>
>   10) Back to the host, monitor the snapshot file and let it grow until
>   at list a bit more than 1GB, as in the example below (where we can see
>   the file with 3.9G):
>
>   root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/
>   total 5.2G
>   -rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18
> bionic-server-cloudimg-amd64.img
>   -rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw
>   -rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img
>   -rw------- 1 libvirt-qemu kvm 3.9G Sep 3 04:41 xenial-snapvm.qcow2
>
>   11) Start a blockcommit job with --active --verbose --pivot --wait and
>   we'll hit the error when the job gets to 100%:
>
>   root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
> --pivot --wait
>   Block commit: [100 %]
>   error: failed to pivot job for disk vda
>   error: block copy still active: disk 'vda' not ready for pivot yet
>
>   12) The blkjob will continue in the background, and status increments:
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [0 %]
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [2 %]
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [6 %]
>
>   13) The blkjob will show it is stuck at 100% until you --abort the
>   blkjob:
>
>   root@xenial-apparmor:~# virsh blockjob snapvm vda --info
>   Active Block Commit: [100 %]
>
>   I have created a test package with the commits needed to solve the
>   problem, and it is available here:
>
>   https://launchpad.net/~mruffell/+archive/ubuntu/sf242822-test
>
>   What should happen:
>
>   If you install the test libvirt-bin and libvirt0 packages from the
>   above ppa, and run through the test case, when blockcommit is invoked,
>   it will not fail immediately, and instead, will continue on until it
>   reaches 100%. Once 100% is reached, the blockjob will complete
>   successfully.
>
>   [Regression Potential]
>
>   While there are four commits which are required to fix this issue, all
>   of them are fairly minor and only modify the way the current status
>   percentage is counted, and how states are being changed, upon reaching
>   100% blockcommit. All changes are localised to one file.
>
>   Most of the commits are limited to blockcommit, and in event of
>   regression, only blockcommit and by extension, some blockjobs would be
>   impacted.
>
>   The commits have been present in upstream for a long time, have been
>   well tested by the community, and are from a release of libvirt with
>   very small delta to the one in xenial (1.3.2 versus 1.3.1 in xenial),
>   I believe there is little risk of regression.
>
>   [Other Info]
>
>   The following commits were identified in the upstream bug:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1197592
>
>   which are also listed in comment #6.
>
>   commit 86c4df83b913dd73b79caeed2038291374384dc5
>   Author: Michael Chapman <email address hidden>
>   Date:   Wed Jan 27 13:24:54 2016 +1100
>   Subject: virsh: improve waiting for block job readiness
>
>   commit 8fa216bbb40df33e7fce5d727aa3dc334480878a
>   Author: Michael Chapman <email address hidden>
>   Date:   Wed Jan 27 13:24:53 2016 +1100
>   Subject: virsh: ensure SIGINT action is reset on all errors
>
>   commit 15dee2ef24f2f19f6dcd30d997b81c8a14582361
>   Author: Michael Chapman <email address hidden>
>   Date:   Wed Jan 27 13:24:52 2016 +1100
>   Subject: virsh: be consistent with style of loop exit
>
>   commit 704dfd6b0fafe7eafca93a03793389239f8ab869
>   Author: Michael Chapman <email address hidden>
>   Date:   Wed Jan 27 13:24:51 2016 +1100
>   Subject: virsh: avoid unnecessary progress updates
>
>   These fix the problem, and were introduced in libvirt 1.3.2 upstream.
>   All commits are clean cherry picks, and the code is still present in
>   B, D, E and F.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions
>