Fails to extend in-use (non LUKS v1) encrypted volumes

[Gorka Eguileor]

Patch fixing bug #1861071 resolved the issue of extending LUKS v1 volumes when nova connects them via libvirt instead of through os-brick, but nova side still fails to extend in-use volumes when they don't go through libvirt (i.e., LUKS v2).

The logs will show a very similar error, but the user won't know that his has happened and Cinder will show the new size:

Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [req-100471fa-c198-40ac-b713-adc395e480f1 req-3a1ea13e-916b-4851-be67-6d849bf4aa3a service nova] [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] resizing block device failed.: libvirt.libvirtError: internal error: unable to execut>
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] Traceback (most recent call last):
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2809, in extend_volume
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] connection_info, encryption)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2763, in _resize_attached_encrypted_volume
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] decrypted_device_new_size, block_device, instance)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2712, in _resize_attached_volume
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] block_device.resize(new_size)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 789, in resize
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] self._guest._domain.blockResize(self._disk, size, flags=flags)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 193, in doit
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] result = proxy_call(self._autowrap, f, *args, **kwargs)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 151, in proxy_call
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] rv = execute(f, *args, **kwargs)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 132, in execute
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] six.reraise(c, e, tb)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] raise value
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 86, in tworker
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] rv = meth(*args, **kwargs)
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1207, in blockResize
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] raise libvirtError('virDomainBlockResize() failed')
Mar 29 21:25:39 ssmc.localdomain nova-compute[1376242]: ERROR nova.virt.libvirt.driver [instance: 3f206ec4-fad5-48b8-9cb2-c3e6f00f30c9] libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/836064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/836064
Committed: https://opendev.org/openstack/nova/commit/8fbaeba11f445bcf6c6be1f5f7b7aeeb6995c9cd
Submitter: "Zuul (22348)"
Branch: master

commit 8fbaeba11f445bcf6c6be1f5f7b7aeeb6995c9cd
Author: Gorka Eguileor <email address hidden>
Date: Wed Mar 30 19:49:18 2022 +0200

    Fix extending non LUKSv1 encrypted volumes

    Patch fixing bug #1861071 resolved the issue of extending LUKS v1
    volumes when nova connects them via libvirt instead of through os-brick,
    but nova side still fails to extend in-use volumes when they don't go
    through libvirt (i.e., LUKS v2).

    The logs will show a very similar error, but the user won't know that
    this has happened and Cinder will show the new size:

     libvirt.libvirtError: internal error: unable to execute QEMU command
     'block_resize': Cannot grow device files

    There are 2 parts to this problem:

    - The device mapper device is not automatically extended.
    - Nova tries to use the encrypted block device size as the size of the
      decrypted device.

    This patch leverages the "extend_volume" method in os-brick connectors
    to extend the device mapper device, after the encrypted device has been
    extended, and use the size of the decrypted volume for the block_resize
    operation.

    Related change: I351f1a7769c9f915e4cd280f05a8b8b87f40df84
    Closes-Bug: #1967157
    Change-Id: Ia1411f11ec4bf44af6a42d5f96c8a0903846ed66

[melanie witt (melwitt)]

Note: this fix will not be complete in nova until https://review.opendev.org/c/openstack/os-brick/+/836059 merges and a new version of os-brick is released and nova requirements.txt is updated to consume it.

[melanie witt (melwitt)]

https://review.opendev.org/c/openstack/os-brick/+/836059 is proposed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/836059
Committed: https://opendev.org/openstack/os-brick/commit/a9a53f9b504b21d41cb0ca8c9f8739bc9444e07f
Submitter: "Zuul (22348)"
Branch: master

commit a9a53f9b504b21d41cb0ca8c9f8739bc9444e07f
Author: Gorka Eguileor <email address hidden>
Date: Wed Mar 30 12:23:46 2022 +0200

    LUKS: Support extending host attached volumes

    Patch fixing bug #1861071 resolved the issue of extending LUKS v1
    volumes when nova connects them via libvirt instead of through os-brick,
    but nova side still fails to extend LUKSv2 in-use volumes when they
    don't go through libvirt.

    The logs will show a very similar error, but the user won't know that
    this has happened and Cinder will show the new size:

     libvirt.libvirtError: internal error: unable to execute QEMU command
     'block_resize': Cannot grow device files

    There are 2 parts to this problem:

    - The device mapper device is not automatically extended.
    - Nova tries to use the encrypted block device size as the size of the
      decrypted device.

    This patch adds new functionality to the encryptors so that they can
    extend decrypted volumes to match the size of the encrypted device.

    New method added to encryptors is called "extend_volume", and should be
    called after the homonymous method in the connector has been called, and
    the value returned by the encryptor's extend_volume method is the real
    size of the decrypted volume (encrypted volume - headers).

    The patch only adds functionality for LUKS and LUKSv2 volumes, not to
    cryptsetup volumes.

    Related-Bug: #1967157
    Change-Id: I351f1a7769c9f915e4cd280f05a8b8b87f40df84

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/843700
Committed: https://opendev.org/openstack/nova/commit/a669f9150a4bfe27abbda3f0b13d98c7acbfc74f
Submitter: "Zuul (22348)"
Branch: master

commit a669f9150a4bfe27abbda3f0b13d98c7acbfc74f
Author: melanie witt <email address hidden>
Date: Sat May 28 03:27:37 2022 +0000

    Test attached volume extend actions in the nova-next job

    Currently we don't have test coverage for volume extend of attached
    volumes and we recently had a bug around extending attached LUKSv2
    encrypted volumes.

    This adjusts the test regex to include volume extend tests and enables
    the corresponding features in the volume service so the tests will run.

    Depends-On: https://review.opendev.org/c/openstack/tempest/+/836065

    Related-Bug: #1967157

    Change-Id: I13cd01fe15c935a235ec1abb7cfb62de1f15065f

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

[Gorka Eguileor (gorka)]

Fix available in Zed