| 2017-04-11 13:05:43 |
Patrick Best |
bug |
|
|
added bug |
| 2017-04-11 13:06:18 |
Patrick Best |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=1197592 |
|
| 2017-04-12 18:31:22 |
Joshua Powers |
bug |
|
|
added subscriber Joshua Powers |
| 2017-04-12 18:31:24 |
Joshua Powers |
libvirt (Ubuntu): status |
New |
Incomplete |
|
| 2017-04-12 18:31:35 |
Joshua Powers |
libvirt (Ubuntu): status |
Incomplete |
New |
|
| 2017-04-18 10:46:05 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Incomplete |
|
| 2017-04-18 10:46:07 |
Christian Ehrhardt |
libvirt (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-04-18 10:46:15 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
| 2017-11-05 21:37:42 |
Dominik Psenner |
bug |
|
|
added subscriber Dominik Psenner |
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Xenial |
|
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Xenial) |
|
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Bionic) |
|
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Artful |
|
| 2018-04-27 05:50:14 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Artful) |
|
| 2018-04-27 05:50:30 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
Incomplete |
Fix Released |
|
| 2018-04-27 05:50:35 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): status |
New |
Fix Released |
|
| 2018-04-27 05:50:42 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
New |
Won't Fix |
|
| 2019-09-12 00:35:45 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2019-10-31 23:50:49 |
Matthew Ruffell |
summary |
libvirt - disk not ready for pivot yet |
libvirt: blockcommit fails - disk not ready for pivot yet |
|
| 2019-10-31 23:51:59 |
Matthew Ruffell |
description |
root@thewind:/home/bestpa/scripts# virsh blockcommit mail vda --active --verbose --pivot
Block commit: [100 %]error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
found related bugfix at redhat... can i get 1.3.2 pushed into ubuntu 16.04 release?
bestpa@thewind:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
bestpa@thewind:~$ libvirtd --version
libvirtd (libvirt) 1.3.1 |
[Impact]
On xenial, if you manually invoke blockcommit through virsh in libvirt, the command immediately fails with blockcommit supposedly being 100%, and that the disk is not ready for pivot yet:
root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose --pivot --wait
Block commit: [100 %]
error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
However, if you look at the status of the active blockjob, we see that the blockcommit is still active in the background:
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [0 %]
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [2 %]
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [6 %]
This happens until it reaches 100%, where it gets stuck. To un-stick things, you must then manually --abort the blockjob.
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [100 %]
This happens in VMs which are experiencing load, and is caused by a race condition in libvirt. Users are not able to commit their snapshots to disk.
[Test Case]
Credit goes to Fabio Martins, who determined how to reproduce this issue.
On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:
1) Create a VG and define a LVM pool:
root@xenial-apparmor:~# cat lvmpool.xml
<pool type="logical">
<name>LVMpool_vg</name>
<source>
<device path="/dev/sdb"/>
</source>
<target>
<path>/dev/LVMpool_vg</path>
</target>
</pool>
# virsh pool-define lvmpool.xml
# virsh pool-start LVMpool_vg
# virsh pool-autostart LVMpool_vg
2) Create a config file to use as a cdrom device with the new VM (will be created in next steps), just to inject a password with cloud-init:
# cat > config <<EOF
> #cloud-config
> password: passw0rd
> chpasswd: { expire: False }
> ssh_pwauth: True
> EOF
# apt install cloud-image-utils
# cloud-localds config.img config
# mv config.img /var/lib/libvirt/images/
# chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
# chmod 664 /var/lib/libvirt/images/config.img
3) Create one VM using this pool:
# virt-install --connect=qemu:///system --name snapvm --ram 2048 --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio --disk /var/lib/libvirt/images/config.img,device=cdrom --network network=kvm-br0 --graphics none --import --noautoconsole
4) Stop the VM
# virsh destroy snapvm
5) Download a Ubuntu cloud image, convert to raw and restore it into the LV used as a disk by our VM:
# wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
# qemu-img convert ./bionic-server-cloudimg-amd64.img ./bionic-server-cloudimg-amd64.raw
# dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm bs=8M conv=sparse
6) Start the VM and connect to it in another window
# virsh start snapvm
7) Check that the VM is using the LV as the disk:
root@xenial-apparmor:~# virsh domblklist snapvm
Target Source
------------------------------------------------
vda /dev/LVMpool_vg/snapvm
hda /var/lib/libvirt/images/config.img
8) Create a snapshot and check that the new domblklist points to the snapshot file:
# virsh snapshot-create-as --domain snapvm --diskspec vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external --disk-only --atomic
root@xenial-apparmor:~# virsh domblklist snapvm
Target Source
------------------------------------------------
vda /var/lib/libvirt/images/xenial-snapvm.qcow2
hda /var/lib/libvirt/images/config.img
9) Connect to your VM and start an I/O intensive job. In this case I'm starting a 'dd' writing zeroes to a file until it gets to 10GBs:
ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=10240000
10) Back to the host, monitor the snapshot file and let it grow until at list a bit more than 1GB, as in the example below (where we can see the file with 3.9G):
root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/
total 5.2G
-rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18 bionic-server-cloudimg-amd64.img
-rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw
-rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img
-rw------- 1 libvirt-qemu kvm 3.9G Sep 3 04:41 xenial-snapvm.qcow2
11) Start a blockcommit job with --active --verbose --pivot --wait and we'll hit the error when the job gets to 100%:
root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose --pivot --wait
Block commit: [100 %]
error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
12) The blkjob will continue in the background, and status increments:
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [0 %]
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [2 %]
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [6 %]
13) The blkjob will show it is stuck at 100% until you --abort the blkjob:
root@xenial-apparmor:~# virsh blockjob snapvm vda --info
Active Block Commit: [100 %]
I have created a test package with the commits needed to solve the problem, and it is available here:
https://launchpad.net/~mruffell/+archive/ubuntu/sf242822-test
What should happen:
If you install the test libvirt-bin and libvirt0 packages from the above ppa, and run through the test case, when blockcommit is invoked, it will not fail immediately, and instead, will continue on until it reaches 100%. Once 100% is reached, the blockjob will complete successfully.
[Regression Potential]
While there are four commits which are required to fix this issue, all of them are fairly minor and only modify the way the current status percentage is counted, and how states are being changed, upon reaching 100% blockcommit. All changes are localised to one file.
Most of the commits are limited to blockcommit, and in event of regression, only blockcommit and by extension, some blockjobs would be impacted.
The commits have been present in upstream for a long time, have been well tested by the community, and are from a release of libvirt with very small delta to the one in xenial (1.3.2 versus 1.3.1 in xenial), I believe there is little risk of regression.
[Other Info]
The following commits were identified in the upstream bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1197592
which are also listed in comment #6.
commit 86c4df83b913dd73b79caeed2038291374384dc5
Author: Michael Chapman <mike@very.puzzling.org>
Date: Wed Jan 27 13:24:54 2016 +1100
Subject: virsh: improve waiting for block job readiness
commit 8fa216bbb40df33e7fce5d727aa3dc334480878a
Author: Michael Chapman <mike@very.puzzling.org>
Date: Wed Jan 27 13:24:53 2016 +1100
Subject: virsh: ensure SIGINT action is reset on all errors
commit 15dee2ef24f2f19f6dcd30d997b81c8a14582361
Author: Michael Chapman <mike@very.puzzling.org>
Date: Wed Jan 27 13:24:52 2016 +1100
Subject: virsh: be consistent with style of loop exit
commit 704dfd6b0fafe7eafca93a03793389239f8ab869
Author: Michael Chapman <mike@very.puzzling.org>
Date: Wed Jan 27 13:24:51 2016 +1100
Subject: virsh: avoid unnecessary progress updates
These fix the problem, and were introduced in libvirt 1.3.2 upstream. All commits are clean cherry picks, and the code is still present in B, D, E and F. |
|
| 2019-10-31 23:52:23 |
Matthew Ruffell |
tags |
|
sts |
|
| 2019-10-31 23:52:53 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): status |
Won't Fix |
In Progress |
|
| 2019-10-31 23:52:57 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2019-10-31 23:53:01 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2019-11-01 02:00:01 |
Matthew Ruffell |
attachment added |
|
libvirt debdiff for xenial https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+attachment/5301911/+files/lp1681839_xenial.debdiff |
|
| 2019-11-22 11:25:54 |
Timo Aaltonen |
libvirt (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2019-11-22 11:25:58 |
Timo Aaltonen |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-11-22 11:26:00 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2019-11-22 11:26:04 |
Timo Aaltonen |
tags |
sts |
sts verification-needed verification-needed-xenial |
|
| 2020-01-08 21:33:23 |
Matthew Ruffell |
tags |
sts verification-needed verification-needed-xenial |
sts verification-done-xenial |
|
| 2020-01-09 21:27:06 |
Eric Desrochers |
bug |
|
|
added subscriber Eric Desrochers |
| 2020-01-13 13:58:25 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-01-13 13:58:44 |
Launchpad Janitor |
libvirt (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
