Retype of non-encrypted root volume to encrypted fails

[Jan Wasilewski]

Description
===========
I tried to retype volume from unencrypted to encrypted one. That works fine for non-root volumes, however it doesn't work when volume was/is root volume.

Steps to reproduce
==================
1. Create an instance with unencrypted root volume(select do not delete volume on termination)
2. Delete instance
3. Retype left volume from deleted instance to encrypted one and wait till retype will finish successfully
4. Launch an instance from this encrypted volume(increase original size by +2GB)

Expected result
===============
New instance is started from encrypted volume

Actual result
=============
Instance fails to start. At horizon we can see an error message:
Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 610, in build_instances filter_properties, instances[0].uuid) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 680, in populate_retry raise exception.MaxRetriesExceeded(reason=msg) MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance d19504b4-8c05-423b-abaa-b1b8c0658fa8. Last exception: Unexpected error while running command. Command: cryptsetup --batch-mode luksFormat --key-file=- --cipher aes-xts-plain64 --key-size 256 /dev/dm-19 Exit code: 5 Stdout: u'' Stderr: u'Cannot use device /dev/dm-19 which is in use (already mapped or mounted).\n'

Environment
===========
1. Version of cinder:
ii cinder-api 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - API server
ii cinder-backup 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Scheduler server
ii cinder-common 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - common files
ii cinder-scheduler 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Scheduler server
ii cinder-volume 2:13.0.9-0ubuntu1~cloud1.1 all Cinder storage service - Volume server
ii python-cinder 2:13.0.9-0ubuntu1~cloud1.1 all Cinder Python 2 libraries
ii python-cinderclient 1:3.5.0-0ubuntu1 all Python bindings to the OpenStack Volume API - Python 2.x

Version of barbican:

ii barbican-api 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - API Server
ii barbican-common 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - common files
ii barbican-keystone-listener 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Keystone Listener
ii barbican-worker 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Worker Node
ii python-barbican 1:6.0.1-0ubuntu1 all OpenStack Key Management Service - Python files
ii python3-barbican 1:10.0.0-0ubuntu0.20.04.1~cloud0 all OpenStack Key Management Service - Python 3 files
ii python3-barbicanclient 4.10.0-0ubuntu1~cloud0 all OpenStack Key Management API client - Python 3.x

Version of nova:
ii nova-api 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - API frontend
ii nova-common 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - common files
ii nova-conductor 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - conductor service
ii nova-consoleauth 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - NoVNC proxy
ii nova-placement-api 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - placement API frontend
ii nova-scheduler 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:18.3.0-0ubuntu1~cloud1 all OpenStack Compute Python 2 libraries
ii python-novaclient 2:11.0.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - Python 2.7

2. We use iSCSI storage(Huawei dorado)

Logs & Configs
==============
Cinder volume logs:
2021-04-23 16:33:43.288 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-19f3af30-022a-4c96-ab28-2c3aaa91764c 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.
2021-04-23 16:34:05.883 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-969c4e48-91a7-4c46-824d-63b924503c34 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:3a7188e5b55f', u'ip': u'10.10.0.248', u'platform': u'x86_64', u'host': u'comp-c08', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

2021-04-23 16:34:12.452 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-314b1423-2650-4a06-9f2e-945bd648e70f 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Initialize iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:a1edf5552e8', u'ip': u'10.10.0.253', u'platform': u'x86_64', u'host': u'comp-a09', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

2021-04-23 16:34:58.111 17298 INFO cinder.volume.drivers.huawei.huawei_driver [req-77231901-21bf-42c3-99a3-5a2e0b13a066 7d52a2a770e4402982390eb658425083 ab5b748ab5e04fa58c8b4aad9f744e2b - default default] Terminate iscsi connection for volume de2c38b7-bf67-42b7-ae51-76fca6c7c05b, connector info {u'initiator': u'iqn.1993-08.org.debian:01:36b34c8cbdd8', u'ip': u'10.10.0.148', u'platform': u'x86_64', u'host': u'comp-a03', u'do_local_attach': False, u'mountpoint': u'/dev/vda', u'os_type': u'linux2', u'multipath': True}.

In case some additional logs are needed, just let me know.

[Jan Wasilewski (janwasilewski)]

I'm attaching error code from nova-compute logs. It is a bit weird, before attachment dm-55 is not in-use, so this error message doesn't make a sense.

[Lee Yarwood (lyarwood)]

I've added cinder and os-brick here as AFAICT this isn't a Nova issue given the description in c#0 and logs in c#1.

The attempt by Cinder to retype the detached volume doesn't appear to have encrypted anything.

This might be the result of another issue in os-brick when connecting the volume to the underlying host but that isn't obvious from the compute logs.

Can you reproduce this with DEBUG logs on the compute so we can see what os-brick is doing?

[Jan Wasilewski (janwasilewski)]

I changed nova logging.conf to below values:

[logger_nova]
level = DEBUG
handlers = stderr
qualname = nova

But I do not know if there is something os-brick specific. If there is another way to provide you requested logs - please let me know, I will try to deliver the best possible logs.

You will find new sort of logs in provided attachment.

[Sofia Enriquez (lsofia-enriquez)]

Hi Jan Wasilewski, hope this msg finds you well.

Looks like this is duplicate of https://bugs.launchpad.net/cinder/+bug/1687880

Just to double check, this works with fine with non-root volumes iSCSI storage(Huawei dorado). Have you tried Huawei without iSCI?

This is a known issue and Cinder team discussed this topic last month in Xena PTG, for more context of the problem please check: https://wiki.openstack.org/wiki/CinderXenaPTGSummary#How_to_handle_retyping.2Fmigrating_nonencrypted_volumes_to_encrypted_volumes_of_the_same_size

Cheers,
Sofi

[Jan Wasilewski (janwasilewski)]

Hi Sofi,

thanks for your input here. Unfortunately I have to try with iSCSI storage(Huawei dorado), as I do not have anything else in my environment.
In the meantime when I'm trying to deeply looking to materials shared by you, I can disagree that it looks like duplicate, because retype of volume finished successfully(at least from what I can see in OS), but attaching this volume to an instance fails with the error message shared by me.

I was already informed at #openstack-cinder channel that problem with retype the same size partition is known, so from the beginning I was adding a bit more(around +2GB) of extra space to avoid such issue.

After all I see that topic of this bug can sounds a bit misleading, as it is related to problem with attaching encrypted volume which was retyped from non-encrypted volume to an instance(just to mention: volume has to be filled with some data before, in my use case it was around 50% of partition size. Empty partition was always fine).

Anyway, if there is any logs which can be useful here, I'm more than happy to collect this and try to help with solution.

Best regards,
Jan

[Sofia Enriquez (lsofia-enriquez)]

Hi Jan,
Do you mind adding c-vol logs to this bug report?
Thanks

[Jan Wasilewski (janwasilewski)]

Hi,

please find attached. At first try it failed with too less space during retype, but second try finished successfully, howerver attachment failed with the same story as previously. If anything else is needed - just let me know.

[Jan Wasilewski (janwasilewski)]

Hi,

any news related to such failure, as I'm still hitting this issue. I believe problem is a bit serious, as there is no nice way to make unencrypted volume easily encrypted.