Scenario:
1. user creates encrypted volume (https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html)
2. user creates backup from this encrypted volume
3. admin user (logged into admin project) tries to delete that backup
This results in backup left in deleting state, with the following trace in the cinder-backup service logs
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: DEBUG barbicanclient.client [req-9e287909-ff2a-4be4-8e2b-8a3c6107b8f6 req-941d2fa1-4b83-4008-851e-fb5f3e2f59da admin None] Response status 403 {{(pid=26635) _che
ck_status_code /usr/local/lib/python3.6/dist-packages/barbicanclient/client.py:87}}
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR barbicanclient.client [req-9e287909-ff2a-4be4-8e2b-8a3c6107b8f6 req-941d2fa1-4b83-4008-851e-fb5f3e2f59da admin None] 4xx Client error: Forbidden: Secret de
letion attempt not allowed - please review your user/project privileges
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR castellan.key_manager.barbican_key_manager [req-9e287909-ff2a-4be4-8e2b-8a3c6107b8f6 req-941d2fa1-4b83-4008-851e-fb5f3e2f59da admin None] Error deleting ob
ject: Forbidden: Secret deletion attempt not allowed - please review your user/project privileges: barbicanclient.exceptions.HTTPClientError: Forbidden: Secret deletion attempt not allowed - please review your use
r/project privileges
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: INFO cinder.volume.volume_utils [req-9e287909-ff2a-4be4-8e2b-8a3c6107b8f6 req-941d2fa1-4b83-4008-851e-fb5f3e2f59da admin None] First attempt to delete key id f38
2a11c-46fc-42a6-b0b3-a91e51735053 failed, retrying with cinder's service context.
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server [req-9e287909-ff2a-4be4-8e2b-8a3c6107b8f6 req-941d2fa1-4b83-4008-851e-fb5f3e2f59da admin None] Exception during message handling:
oslo_config.cfg.NoSuchOptError: no such option password in group [keystone_authtoken]
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/castellan/key_manager/barbican_key_manager.py", line 597, in delete
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server barbican_client.secrets.delete(secret_ref)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/barbicanclient/v1/secrets.py", line 540, in delete
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server self._api.delete(uuid_ref)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/keystoneauth1/adapter.py", line 410, in delete
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server return self.request(url, 'DELETE', **kwargs)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/barbicanclient/client.py", line 63, in request
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server self._check_status_code(resp)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/barbicanclient/client.py", line 107, in _check_status_code
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server status
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server barbicanclient.exceptions.HTTPClientError: Forbidden: Secret deletion attempt not allowed - please review your user/project privileges
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/volume_utils.py", line 947, in delete_encryption_key
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server key_manager.delete(context, encryption_key_id)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/castellan/key_manager/barbican_key_manager.py", line 606, in delete
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server raise exception.KeyManagerError(reason=e)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server castellan.common.exception.KeyManagerError: Key manager error: Forbidden: Secret deletion attempt not allowed - please review yo$r user/project privileges
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 772, in delete_backup
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server backup.encryption_key_id)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/volume_utils.py", line 956, in delete_encryption_key
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server password=conf.keystone_authtoken.password,
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 3161, in __getattr__
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server return self._conf._get(name, self._group)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2653, in _get
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server value, loc = self._do_get(name, group, namespace)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2671, in _do_get
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server info = self._get_opt_info(name, group)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2876, in _get_opt_info
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server raise NoSuchOptError(opt_name, group)
Oct 08 12:52:53 victoria-dsvm cinder-backup[26628]: ERROR oslo_messaging.rpc.server oslo_config.cfg.NoSuchOptError: no such option password in group [keystone_authtoken]
AFAICT this is related to this place in delete_encryption_key function:
https://github.com/openstack/cinder/blob/05d397bed11bdb02ba80bb147e87bd4b5c561e28/cinder/volume/volume_utils.py#L1045
the method register_auth_conf_options from keystoneauth1.loading only registers two common config options - auth_type and auth_section
https://github.com/openstack/keystoneauth/blob/112bcae1fbec355fcb58da07ae1d9f1adf8b77ba/keystoneauth1/loading/conf.py#L66
The other options related to specific auth plugin (like password for Password plugin) are dynamically registered on actual plugin loading.
I presume that in other places this code is executed when such loading has already happened (like in cinder-api on volume delete, where [keystone_authtoken] section was naturally processed already and the required plugin was already properly loaded in order to validate the incoming request in the first place), however with backups, this function is called from the cinder-backup that has not loaded anything (yet?).
Note that even if fixed, it will probably still fail the actual deletion of the secret if the user defined in the keystone_authtoken section lacks permisisons in Barbican to delete secrets created by another user (which is the case in the default setup of Barbican API policies and ACLs), but at least the error will be much more readable and proper (and it will actually work if barbican api policies are properly adjusted, same as when admin deletes encrypted volume created by another user in another project).