resize fails with volume multiattach using with libvirt 4.0.0 (and qemu 2.11.1): Failed to get shared "write" lock

[Matt Riedemann]

Seeing this in a patch https://review.openstack.org/#/c/554317/ that runs the nova-multiattach job with the Queens Ubuntu Cloud Archive which has libvirt 4.0.0 and qemu 2.11.1:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/libvirt/qemu/instance-00000066.txt.gz

2018-03-19T19:48:16.175548Z qemu-system-x86_64: -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1: Failed to get shared "write" lock
Is another process using the image?

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_19_19_48_16_261051

Mar 19 19:48:17.132940 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: ERROR nova.compute.manager [None req-3a092a4b-7ae7-4f29-9f78-97bf1dc0d46d service nova] [instance: 0eed0237-245e-4a18-9e30-9e72accd36c6] Setting instance vm_state to ERROR: libvirtError: internal error: process exited while connecting to monitor: 2018-03-19T19:48:16.147136Z qemu-system-x86_64: -drive file=/dev/sdb,format=raw,if=none,id=drive-virtio-disk1,serial=652600d5-f6dc-4089-ba95-d71d7640cafa,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead
Mar 19 19:48:17.133724 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: 2018-03-19T19:48:16.155115Z qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Mar 19 19:48:17.134022 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: 2018-03-19T19:48:16.175548Z qemu-system-x86_64: -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1: Failed to get shared "write" lock

That last error likely means the 'shareable' element isn't in the disk config xml, and it's not:

<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native'/>
    <source dev='/dev/sdb'/>
    <target dev='vdb' bus='virtio'/>
    <serial>652600d5-f6dc-4089-ba95-d71d7640cafa</serial>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>

[Matt Riedemann (mriedem)]

This is a bit confusing because I'm seeing the disk 'shareable' flag set in the guests in other multiattach tests:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz#_Mar_19_19_54_59_543776

[Matt Riedemann (mriedem)]

The test_resize_server_with_multiattached_volume test creates two servers and a single multiattach volume, then attaches the volume to the two servers, resizes both servers and then detaches the volume from each.

This is when we attach the multiattach volume to the first instance:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz#_Mar_19_19_48_00_429470

And the shareable flag is set:

Mar 19 19:48:00.429470 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: DEBUG nova.virt.libvirt.guest [None req-4a992c4c-d85a-4cd8-9896-569924500d5a tempest-AttachVolumeMultiAttachTest-1944074929 tempest-AttachVolumeMultiAttachTest-1944074929] attach device xml: <disk type="block" device="disk">
Mar 19 19:48:00.429807 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <driver name="qemu" type="raw" cache="none" io="native"/>
Mar 19 19:48:00.430097 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <source dev="/dev/sdb"/>
Mar 19 19:48:00.430383 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <target bus="virtio" dev="vdb"/>
Mar 19 19:48:00.430670 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <serial>652600d5-f6dc-4089-ba95-d71d7640cafa</serial>
Mar 19 19:48:00.430969 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <shareable/>
Mar 19 19:48:00.431268 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: </disk>
Mar 19 19:48:00.431544 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: {{(pid=27735) attach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:302}}

That's instance 0eed0237-245e-4a18-9e30-9e72accd36c6.

This is when we attach the multiattach volume to the second instance:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz#_Mar_19_19_48_04_769855

And the shareable flag is set:

Mar 19 19:48:04.769855 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: DEBUG nova.virt.libvirt.guest [None req-445edbca-bb87-489e-bf14-eb646e570690 tempest-AttachVolumeMultiAttachTest-1944074929 tempest-AttachVolumeMultiAttachTest-1944074929] attach device xml: <disk type="block" device="disk">
Mar 19 19:48:04.770106 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <driver name="qemu" type="raw" cache="none" io="native"/>
Mar 19 19:48:04.770301 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <source dev="/dev/sdb"/>
Mar 19 19:48:04.770483 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <target bus="virtio" dev="vdb"/>
Mar 19 19:48:04.770667 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <serial>652600d5-f6dc-4089-ba95-d71d7640cafa</serial>
Mar 19 19:48:04.770845 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: <shareable/>
Mar 19 19:48:04.771027 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: </disk>
Mar 19 19:48:04.771217 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: {{(pid=27735) attach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:302}}

Then we start to resize the servers. We correctly count the number of servers on the same host and don't disconnect the volume from the host when detaching the first instance:

http://logs.o...

[Matt Riedemann (mriedem)]

I can tell that during migrate_disk_and_power_off from the virt driver on the source host, that the bdm.connection_info has the multiattach flag set because we get this message when disconnecting the volume from the first instance on the first host:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz#_Mar_19_19_48_12_057443

Mar 19 19:48:12.057443 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: INFO nova.virt.libvirt.driver [None req-f044373a-53dd-4de7-b881-ff511ecb3a70 tempest-AttachVolumeMultiAttachTest-1944074929 tempest-AttachVolumeMultiAttachTest-1944074929] [instance: 0eed0237-245e-4a18-9e30-9e72accd36c6] Detected multiple connections on this host for volume: 652600d5-f6dc-4089-ba95-d71d7640cafa, skipping target disconnect.

Which comes from the _should_disconnect_target code.

Then I eventually see _terminate_volume_connections which creates a new empty attachment between the server and volume:

http://logs.openstack.org/17/554317/1/check/nova-multiattach/8e97832/logs/screen-n-cpu.txt.gz#_Mar_19_19_48_12_307926

Mar 19 19:48:12.307926 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: DEBUG cinderclient.v3.client [None req-f044373a-53dd-4de7-b881-ff511ecb3a70 tempest-AttachVolumeMultiAttachTest-1944074929 tempest-AttachVolumeMultiAttachTest-1944074929] REQ: curl -g -i -X POST http://198.72.124.95/volume/v3/774fc170bdb24f20959a551dd666ef06/attachments -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "OpenStack-API-Version: volume 3.44" -H "X-Auth-Token: {SHA1}6ba4939df9b73a2d082ac56165bd1e04d7f33e6d" -H "Content-Type: application/json" -H "X-OpenStack-Request-ID: req-f044373a-53dd-4de7-b881-ff511ecb3a70" -d '{"attachment": {"instance_uuid": "0eed0237-245e-4a18-9e30-9e72accd36c6", "connector": null, "volume_uuid": "652600d5-f6dc-4089-ba95-d71d7640cafa"}}' {{(pid=27735) _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:372}}
...
Mar 19 19:48:12.412178 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: RESP BODY: {"attachment": {"status": "reserved", "detached_at": "", "connection_info": {}, "attached_at": "", "attach_mode": null, "instance": "0eed0237-245e-4a18-9e30-9e72accd36c6", "volume_id": "652600d5-f6dc-4089-ba95-d71d7640cafa", "id": "c33fe43e-ac88-48df-93bd-3293746213b6"}}

And deletes the existing attachment:

Mar 19 19:48:12.418026 ubuntu-xenial-inap-mtl01-0003062768 nova-compute[27735]: DEBUG cinderclient.v3.client [None req-f044373a-53dd-4de7-b881-ff511ecb3a70 tempest-AttachVolumeMultiAttachTest-1944074929 tempest-AttachVolumeMultiAttachTest-1944074929] REQ: curl -g -i -X DELETE http://198.72.124.95/volume/v3/774fc170bdb24f20959a551dd666ef06/attachments/448fc228-0e4c-49b4-b0a9-00f3b17468dd -H "OpenStack-API-Version: volume 3.44" -H "User-Agent: python-cinderclient" -H "X-OpenStack-Request-ID: req-f044373a-53dd-4de7-b881-ff511ecb3a70" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}6ba4939df9b73a2d082ac56165bd1e04d7f33e6d" {{(pid=27735) _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:372}}

Later once we're in _finish_resize, we call _update_volume_attachments. I can see that...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/554667

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/555029

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554667
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5f3cca205581d45d92714ce1a909d4394b7812ff
Submitter: Zuul
Branch: master

commit 5f3cca205581d45d92714ce1a909d4394b7812ff
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 20 15:04:27 2018 -0400

    Preserve multiattach flag when refreshing connection_info

    When we attach a multiattach-capable volume, we do something
    dirty and stash a "multiattach" boolean flag in the
    BlockDeviceMapping.connection_info dict. This is used by the
    virt driver to determine how to connect the volume (for the
    libvirt driver, it sets the "shareable" element on the disk
    config xml).

    When resizing an instance, ComputeManager._finish_resize on
    the destination host refreshes the block device mapping list
    along with the connection_info for each BDM. Because of this,
    it would overwrite the BDM.connection_info along with the stashed
    "multiattach" flag which is later used to connect the volumes
    on the destination host via the virt driver.finish_migration
    method. This leads to failures with multiattach volumes because
    the disk config is wrong.

    To fix this, when refreshing BDM connection_info, we preserve
    the multiattach flag in the connection_info, similar to the
    serial (volume ID) and multipath_id.

    Interestingly enough, the nova-multiattach job does not fail
    the volume multiattach resize test with libvirt 1.3.1 and qemu
    2.5. This failure was only noticed once the nova-multiattach
    job was tested with libvirt 4.0.0 and qemu 2.11.1. So maybe there
    was something in the older package versions that masked this
    obvious bug in the nova code.

    Change-Id: Iaee13478212cc04e6d1a1249f33822369d94d41d
    Closes-Bug: #1757190

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555029
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8706dc3f4fe44622d459b020b1c5adef392faa0c
Submitter: Zuul
Branch: stable/queens

commit 8706dc3f4fe44622d459b020b1c5adef392faa0c
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 20 15:04:27 2018 -0400

    Preserve multiattach flag when refreshing connection_info

    When we attach a multiattach-capable volume, we do something
    dirty and stash a "multiattach" boolean flag in the
    BlockDeviceMapping.connection_info dict. This is used by the
    virt driver to determine how to connect the volume (for the
    libvirt driver, it sets the "shareable" element on the disk
    config xml).

    When resizing an instance, ComputeManager._finish_resize on
    the destination host refreshes the block device mapping list
    along with the connection_info for each BDM. Because of this,
    it would overwrite the BDM.connection_info along with the stashed
    "multiattach" flag which is later used to connect the volumes
    on the destination host via the virt driver.finish_migration
    method. This leads to failures with multiattach volumes because
    the disk config is wrong.

    To fix this, when refreshing BDM connection_info, we preserve
    the multiattach flag in the connection_info, similar to the
    serial (volume ID) and multipath_id.

    Interestingly enough, the nova-multiattach job does not fail
    the volume multiattach resize test with libvirt 1.3.1 and qemu
    2.5. This failure was only noticed once the nova-multiattach
    job was tested with libvirt 4.0.0 and qemu 2.11.1. So maybe there
    was something in the older package versions that masked this
    obvious bug in the nova code.

    Change-Id: Iaee13478212cc04e6d1a1249f33822369d94d41d
    Closes-Bug: #1757190
    (cherry picked from commit 5f3cca205581d45d92714ce1a909d4394b7812ff)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.