Volume retype fails from non-encrypted to encrypted volume type

[Oleksandr Kozachenko]

Openstack version: wallaby
Cinder storage backend: ceph
keymanager for nova and cinder: barbican

Reproduce step:

1. Create the volume type LUKS
```
openstack volume type create --encryption-provider luks \
  --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LUKS
```
2. Create unencrypted volume
```
openstack volume create --size 1 'unencrypted volume' -c id -f value
```
3. Retype
```
cinder retype --migration-policy on-demand cca3582f-c8b4-4175-bb44-ae4bf49be566 LUKS
```

The output error log of c-vol is;
```
2021-10-29 19:29:16.436 233 INFO oslo.privsep.daemon [-] privsep daemon starting
2021-10-29 19:29:16.443 233 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2021-10-29 19:29:16.446 233 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
2021-10-29 19:29:16.447 233 INFO oslo.privsep.daemon [-] privsep daemon running as pid 233
2021-10-29 19:29:16.670 9 WARNING os_brick.initiator.connectors.iscsi [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Could not find the iSCSI Initiator File /etc/iscsi/initiatorname.iscsi: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:16.687 9 WARNING os_brick.initiator.connectors.nvmeof [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Process execution error in _get_host_uuid: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:16.699 9 WARNING os_brick.initiator.connectors.nvmeof [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Could not generate host nqn: Unexpected error while running command.
Command: mkdir -m 755 -p /etc/nvme
Exit code: 1
Stdout: ''
Stderr: 'mkdir: cannot create directory ‘/etc/nvme’: Read-only file system\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-10-29 19:29:17.247 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Initialize volume connection completed successfully.
2021-10-29 19:29:17.319 9 ERROR os_brick.encryptors [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Error instantiating os_brick.encryptors.luks.LuksEncryptor: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.319 9 ERROR cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Failed to attach volume encryptor 92ec3065-ed0a-4742-8f1b-1d30cd416684.
2021-10-29 19:29:17.346 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Terminate volume connection completed successfully.
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Failed to copy volume cca3582f-c8b4-4175-bb44-ae4bf49be566 to 92ec3065-ed0a-4742-8f1b-1d30cd416684: TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager Traceback (most recent call last):
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2344, in _migrate_volume_generic
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2217, in _copy_volume_data
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager dest_attach_info = self._attach_volume(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2161, in _attach_volume
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self._detach_volume(ctxt, attach_info, volume, properties,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self.force_reraise()
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager raise self.value
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2154, in _attach_volume
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager volume_utils.brick_attach_volume_encryptor(ctxt,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1360, in brick_attach_volume_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager encryptor = brick_get_encryptor(connection_info,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1350, in brick_get_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager return encryptors.get_volume_encryptor(root_helper=root_helper,
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/__init__.py", line 88, in get_volume_encryptor
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager encryptor = importutils.import_object(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/importutils.py", line 44, in import_object
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager return import_class(import_str)(*args, **kwargs)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/luks.py", line 57, in __init__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager super(LuksEncryptor, self).__init__(
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/cryptsetup.py", line 62, in __init__
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager File "/usr/lib/python3.8/posixpath.py", line 142, in basename
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager p = os.fspath(p)
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.377 9 ERROR cinder.volume.manager
2021-10-29 19:29:17.415 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Remove volume export completed successfully.
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Exception during message handling: TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 3040, in retype
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server _retype_error(context, volume, old_reservations,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 3036, in retype
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2619, in migrate_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server volume.save()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2612, in migrate_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2365, in _migrate_volume_generic
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._clean_temporary_volume(ctxt, volume,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2344, in _migrate_volume_generic
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2217, in _copy_volume_data
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server dest_attach_info = self._attach_volume(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2161, in _attach_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self._detach_volume(ctxt, attach_info, volume, properties,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server raise self.value
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/manager.py", line 2154, in _attach_volume
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server volume_utils.brick_attach_volume_encryptor(ctxt,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1360, in brick_attach_volume_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server encryptor = brick_get_encryptor(connection_info,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/volume_utils.py", line 1350, in brick_get_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return encryptors.get_volume_encryptor(root_helper=root_helper,
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/__init__.py", line 88, in get_volume_encryptor
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server encryptor = importutils.import_object(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/importutils.py", line 44, in import_object
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server return import_class(import_str)(*args, **kwargs)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/luks.py", line 57, in __init__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server super(LuksEncryptor, self).__init__(
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/var/lib/openstack/lib/python3.8/site-packages/os_brick/encryptors/cryptsetup.py", line 62, in __init__
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/posixpath.py", line 142, in basename
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server p = os.fspath(p)
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server
2021-10-29 19:29:17.642 9 INFO cinder.volume.manager [req-b8e226cf-cf35-4969-bae0-de3a2deda8fd e08aeb12768b4021a04002b6a1cbf3c5 00cc2ee8b0d24b4799b2319d8cc315da - - -] Deleted volume successfully.
```

[Oleksandr Kozachenko (okozachenko)]

So I think the main error is the following;
```
2021-10-29 19:29:17.524 9 ERROR oslo_messaging.rpc.server TypeError: expected str, bytes or os.PathLike object, not RBDVolumeIOWrapper
```
https://opendev.org/openstack/os-brick/src/branch/stable/wallaby/os_brick/initiator/connectors/rbd.py#L230-L245

All `os_brick.initiator.connectors` have `connect_volume` and `disconnect_volume` methods which returns `device_path`. While retyping, the `device_path` is used to attach the volume. But as you can see the above link, `connect_volume` of rbd connector returns `linuxrbd.RBDVolumeIOWrapper` and returns str when only `do_local_attach` is set true.
But do_local_attach is set false.

[Sofia Enriquez (lsofia-enriquez)]

https://opendev.org/openstack/cinder/commit/09ad89b7eecd6ea654a17690996c2ec4a63dd31e

[Oleksandr Kozachenko (okozachenko)]

The current error is only for rbd os_brick.initiator.connectors type because of the type mismatch.
I can say it is not the duplicated one. Maybe the final result is the same with the known issue after this bug fixed but not for now.

Other connector types are returning correct device_path in connect_volume methods.

[Noel Ashford (nashford77)]

I hit this as well in 2023.2, I try to create a LUKS volume which worked prior, and on the re-type from an non encrypted image, it dies with this error.