can't migrate plain encrypted volume - keymanager error

Bug #1929128 reported by Tobias Gurtzick
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Invalid
Undecided
Unassigned

Bug Description

I was trying to test the volume migrate functionality with a cinder lvm backend from one to another host together with barbican for encryption activated. Luks is not supported for migration so the setting used was plain.

However when the migration is started, it complains about access permissions, which should be there. This was a test installation with no custom configs at all, so it was just a default install with kolla-ansible and nova cells activated.

```
=> /var/log/kolla/cinder/cinder-volume.log <==
2021-05-20 22:08:03.479 31 INFO cinder.volume.manager [req-db72f343-c0f7-48d7-8bc2-beab7e39f4c8 acc8abe8567645db97f47c37c93a223b e25d9689fc43434ab8d7513c6ab3586c - - -] migrate_volume_completion is cleaning up an error for volume 05f522fb-7ef3-4063-9a10-446d756a5ed2 (temporary volume 2444e342-3911-47fd-8c13-88cd5c026fab

==> /var/log/kolla/nova/nova-compute.log <==
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server [req-d7ac090d-4235-4dd7-9025-2b83ec3674c9 acc8abe8567645db97f47c37c93a223b e25d9689fc43434ab8d7513c6ab3586c - default default] Exception during message handling: castellan.common.exception.KeyManagerError: Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/castellan/key_manager/barbican_key_manager.py", line 572, in get
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return self._get_castellan_object(secret, metadata_only)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/castellan/key_manager/barbican_key_manager.py", line 497, in _get_castellan_object
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server if barbican_type == secret.secret_type:
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/barbicanclient/v1/secrets.py", line 34, in wrapper
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self._fill_lazy_properties()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/barbicanclient/v1/secrets.py", line 417, in _fill_lazy_properties
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server result = self._api.get(uuid_ref)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/barbicanclient/client.py", line 70, in get
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return super(_HTTPClient, self).get(*args, **kwargs).json()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 395, in get
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/barbicanclient/client.py", line 63, in request
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self._check_status_code(resp)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/barbicanclient/client.py", line 105, in _check_status_code
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise exceptions.HTTPClientError(
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise self.value
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise self.value
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7309, in swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server _do_locked_swap_volume(context, old_volume_id, new_volume_id, instance,
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7307, in _do_locked_swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self._do_swap_volume(context, old_volume_id, new_volume_id,
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7351, in _do_swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server comp_ret, new_cinfo = self._swap_volume(context,
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7232, in _swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.volume_api.attachment_delete(
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise self.value
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7190, in _swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.driver.swap_volume(context, old_cinfo, new_cinfo, instance,
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2154, in swap_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self._connect_volume(context, new_connection_info, instance)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1790, in _connect_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise self.value
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1784, in _connect_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server self._attach_encryptor(context, connection_info, encryption)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1950, in _attach_encryptor
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server encryptor.attach_volume(context, **encryption)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/encryptors/cryptsetup.py", line 160, in attach_volume
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server key = self._get_key(context).get_encoded()
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/encryptors/base.py", line 48, in _get_key
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server return self._key_manager.get(context, self.encryption_key_id)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/castellan/key_manager/barbican_key_manager.py", line 581, in get
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server raise exception.KeyManagerError(reason=e)
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server castellan.common.exception.KeyManagerError: Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server

```

Revision history for this message
Tobias Gurtzick (wzrdtales) wrote :

Latest official release btw.

Revision history for this message
Tobias Gurtzick (wzrdtales) wrote :

what i noticed, the new volume created seems to be project less, so that might be related.

description: updated
Revision history for this message
Tobias Gurtzick (wzrdtales) wrote :

version 22.2.0

i tried applying a different policy found here (https://bugs.launchpad.net/nova/+bug/1895848), however that won't work as well, it throws still the same error. It is like nova is accessing without any auth at all.

Revision history for this message
Tobias Gurtzick (wzrdtales) wrote :

experimenting more with it, it seems to be related with an online migration. this works fine for unencrypted drives. but fails for encrypted ones. luks fails with an error before it does anything, plain ends with the error above.

deleting the machine, migrating and then starting works. which is of course not really an option. seems like I hit functionality that is just not there?

Changed in cinder:
importance: Undecided → High
tags: added: encryption migration volume
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Greetings,

From the bug description 'Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges
2021-05-20 22:08:03.606 7 ERROR oslo_messaging.rpc.server'.
It doesn't loooks like a Cinder problem but a credentials problems.

Changed in cinder:
status: New → Invalid
importance: High → Undecided
tags: added: kolla lvm
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.