libvirtError: block copy still active: disk 'vdb' not ready for pivot yet

[Lee Yarwood]

This is an additional corner case for swap_volume not covered by bug#1630600.

The following failure is taken from the devstack change enabling the new swap_volume tempest test :

tempest: configure compute-feature-enabled.swap_volume if libvirt
https://review.openstack.org/#/c/374373/

http://logs.openstack.org/73/374373/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/961698c/logs/screen-n-cpu.txt.gz#_2016-11-18_16_11_39_759

2016-11-18 16:11:39.759 4511 ERROR root [req-f40054e1-f18d-478a-ae14-9c738bb961f2 tempest-TestVolumeSwap-1057055027 tempest-TestVolumeSwap-1057055027] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1228, in _swap_volume\n dev.abort_job(pivot=True)\n', ' File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 647, in abort_job\n self._guest._domain.blockJobAbort(self._disk, flags=flags)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call\n rv = execute(f, *args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute\n six.reraise(c, e, tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker\n rv = meth(*args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 739, in blockJobAbort\n if ret == -1: raise libvirtError (\'virDomainBlockJobAbort() failed\', dom=self)\n', "libvirtError: block copy still active: disk 'vdb' not ready for pivot yet\n"]

As discussed in bug#1630600 and at length in RHBZ#1382165 the issue here is that QEMU reports the progress of the block job as complete (status.cur == status.end) _before_ updating the ready field of the block job. This causes the following attempt to pivot to fail.

Nova should attempt to use the ready field of the block job when available (libvirt >1.2.18) to ensure that we are able to pivot.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/399735

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/402162

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/402163

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: master
Review: https://review.openstack.org/402162

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/399735
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ed782641970dd371adc151e7d3913a402369dd3
Submitter: Jenkins
Branch: master

commit 9ed782641970dd371adc151e7d3913a402369dd3
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 21 11:30:01 2016 +0000

    libvirt: Use the mirror element to detect job completion

    The mirror element was introduced in Libvirt 0.10.0 [1] and provides a
    ready attribute that helps to determine when a blockjob is complete.
    This is now used in addition to checking the progress of a given
    blockjob to ensure Nova waits until these jobs complete fully.

    [1] https://libvirt.org/git/?p=libvirt.git;a=commit;h=ae6aa8c3965e9aaa245b8e669c6324d44312ac1b

    Closes-bug: #1643017
    Change-Id: I0c52917a5555a70c4973f37dea1aebf878dd73b4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/427290

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/433793

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/427290
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a2b886dcc81ba00949653a32a8f851d6e61e44a4
Submitter: Jenkins
Branch: stable/newton

commit a2b886dcc81ba00949653a32a8f851d6e61e44a4
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 21 11:30:01 2016 +0000

    libvirt: Use the mirror element to detect job completion

    The mirror element was introduced in Libvirt 0.10.0 [1] and provides a
    ready attribute that helps to determine when a blockjob is complete.
    This is now used in addition to checking the progress of a given
    blockjob to ensure Nova waits until these jobs complete fully.

    [1] https://libvirt.org/git/?p=libvirt.git;a=commit;h=ae6aa8c3965e9aaa245b8e669c6324d44312ac1b

    Closes-bug: #1643017
    Change-Id: I0c52917a5555a70c4973f37dea1aebf878dd73b4
    (cherry picked from commit 9ed782641970dd371adc151e7d3913a402369dd3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.4 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/433793