Unable to delete encrypted volume (barbican)

[Benny Kopilov]

Devstack isntallation , local cinder LVM + babbican.

Managed to create an encrypted volume but when trying to delete the volume with admin tenants, action fails with Error :
Delete for volume a1bc0685-81e9-44e8-8965-323c402139e7 failed: Invalid volume: Unable to delete encryption key for volume. (HTTP 400) (Request-ID: req-1b715dad-c9ce-4422-a506-191f84bc7d9e)
ERROR: Unable to delete any of the specified volumes.

Here are the errors:
/opt/stack/logs/barbican.log:66:2018-06-06 08:29:51.683 ERROR barbican.api.controllers [req-101bb298-6d7c-485d-91ad-3c185ece5e04 demo admin] Secret deletion attempt not allowed - please review your user/project privileges: PolicyNotAuthorized: secret:delete is disallowed by policy
/opt/stack/logs/barbican.log:70:2018-06-06 08:29:51.861 ERROR barbican.api.controllers [req-850f78ef-c7ed-416a-8042-c1f0fd80c233 service cinder] Secret deletion attempt not allowed - please review your user/project privileges: PolicyNotAuthorized: secret:delete is disallowed by policy

[Benny Kopilov (bkopilov)]

Latest devstack queens.

[Benny Kopilov (bkopilov)]

Since the volume created as encrypted , we expect that admin tenant will be able to delete the volume.

[Benny Kopilov (bkopilov)]

Steps to reproduce:
# create a volume type (admin tenant)
# create encryption type - > (admin tenant)
encryption_prov = nova.volume.encryptors.luks.LuksEncryptor
key_size = 256
cipher = aes-xts-plain64
control_location = front-end

# add non-admin tenant to the role creator

# create a volume (non-admin) tenant.

Here are some shows:
Tenants:
89517260b9504830a0d3154cf061f36d | tempest-TestEncryptedCinderVolumes-57557031

cinder backend:
[root@localhost devstack]# cinder get-pools --detail
+-----------------------------+------------------------------------------------------------------------------+
| Property | Value |
+-----------------------------+------------------------------------------------------------------------------+
| QoS_support | False |
| allocated_capacity_gb | 1 |
| backend_state | up |
| driver_version | 3.0.0 |
| filter_function | None |
| free_capacity_gb | 22.8 |
| goodness_function | None |
| location_info | LVMVolumeDriver:localhost.localdomain.localdomain:stack-volumes-lvm_2:thin:0 |
| max_over_subscription_ratio | 20.0 |
| multiattach | True |
| name | localhost.localdomain.localdomain@lvm_2#lvm_2 |
| pool_name | lvm_2 |
| provisioned_capacity_gb | 1.0 |
| reserved_percentage | 0 |
| storage_protocol | iSCSI |
| thick_provisioning_support | False |
| thin_provisioning_support | True |
| timestamp | 2018-06-07T05:12:27.189877 |
| total_capacity_gb | 22.8 |
| total_volumes | 2 |
| vendor_name | Open Source |
| volume_backend_name | lvm_2 ...

[Alan Bishop (alan-bishop)]

This is probably not a cinder bug, but is a side effect of using barbican. Cinder has its own policies for who can delete a resource (such as a volume), and barbican has its own policies that can be more restrictive. Barbican has stricter rules about deleting another user's secrets than cinder has about deleting another user's volumes. Here, cinder is refusing to delete the volume because the user is not authorized to delete the secret associated with the volume. This is done in order to avoid "leaking" an encryption key.

[Alex Walender (awalende)]

Would it be really an issue if cinder would delete the volume anyways? The encryption key is generated on volume create, so the key should be worthless after the volume is gone?
Maybe a temporary solution until the admin user is able to delete user secrets.
As far as I know, it is not possible to delete barbican user secrets with an admin user, making the management of encrypted volumes really annoying.

[Mark Goddard (mgoddard)]

Agreed this is quite annoying behaviour. Would it be acceptable to return a 403 to the cinder API caller if we get a 403 from barbican?

[Alan Bishop (alan-bishop)]

@awalende, the concern is that "abandoned" encryption keys are still consuming a resource, notably in the HSM where they might be stored. Cinder's behavior is intended to highlight the fact that there's an issue deleting the key in order to avoid it becoming abandoned. I don't know about the specifics of barbican's policies regarding whether an admin can delete a user secret.

@mgoddard, I agree the behavior does not make for a good user experience. The problem is volume deletion is inherently an asynchronous operation, and at the cinder-api layer it can only do so much validation before it returns 202 when it passes the deletion request on to the cinder-volume service. And in this situation, cinder-volume deletes the key.

[Mark Goddard (mgoddard)]

Alan, as far as I can tell, the key is deleted in the API service synchronously, so it would be possible to alter the return code.

https://opendev.org/openstack/cinder/src/commit/9abb00521849616b4b8662d65480e67c5848a530/cinder/volume/api.py#L522

[Pavlo Shchelokovskyy (pshchelo)]

we are hitting that as well

IMO it all boils down to default API access policies in Barbican are generally more strict than policies in other services using Barbican in respect for (global) admin actions.

I see at least 3 possible solutions, but we need to discuss consequences of each of them for the security model and use cases this functionality is intended to solve:

1) (as already mentioned) just ignore key deletion errors and orphan them with appropriate WARNING in the logs
2) change (default) api policies in Barbican to match its consumers and allow the same global admin to delete whatever secret it has to delete
3) store these auto-generated secrets under service user, at least make this choice available to operators via config option for new deployments. These secrets are basically transparent to the user, who may be oblivious to them being created and used in the first place, and user has no option to pass her own secret to the volume create / attach call, so why store them in the user's project at all? This of course highly depends on what use cases this encryption is actually trying to solve, and while it seems to me the best option, I may be missing something here...