Creation of encrypted volume fails when barbican backend is dogtag

[Vladyslav Drok]

When following https://docs.openstack.org/cinder/pike/configuration/block-storage/volume-encryption.html, with having barbican setup with dogtag backend, the following issue appears:

openstack volume create --size 1 --type LUKS 'encrypted volume'

The volume gets error status afterwards. Related logs:

Key manager error (HTTP 400) (Request-ID: req-f87b90e4-c209-421b-9cf4-fd36d6b042fd)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 113, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/volume/v2/volume.py", line 222, in take_action
    scheduler_hints=parsed_args.hint,
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 290, in create
    return self._create('/volumes', body, 'volume')
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 324, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 180, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
BadRequest: Key manager error (HTTP 400) (Request-ID: req-f87b90e4-c209-421b-9cf4-fd36d6b042fd)
clean_up CreateVolume: Key manager error (HTTP 400) (Request-ID: req-f87b90e4-c209-421b-9cf4-fd36d6b042fd)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 113, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/volume/v2/volume.py", line 222, in take_action
    scheduler_hints=parsed_args.hint,
  File "/usr/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 290, in create
    return self._create('/volumes', body, 'volume')
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 324, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 180, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 168, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 154, in request
    raise exceptions.from_response(resp, body)
BadRequest: Key manager error (HTTP 400) (Request-ID: req-f87b90e4-c209-421b-9cf4-fd36d6b042fd)

root@ctl01:~# tail -f /var/log/apache2/barbican.error.log
2018-03-28 14:32:45.427 24096 ERROR barbican.tasks.resources File "/usr/lib/python2.7/dist-packages/barbican/tasks/resources.py", line 68, in process_and_suppress_exceptions
2018-03-28 14:32:45.427 24096 ERROR barbican.tasks.resources return self.process(*args, **kwargs)
2018-03-28 14:32:45.427 24096 ERROR barbican.tasks.resources File "/usr/lib/python2.7/dist-packages/barbican/tasks/resources.py", line 113, in process
2018-03-28 14:32:45.427 24096 ERROR barbican.tasks.resources raise e_orig
2018-03-28 14:32:45.427 24096 ERROR barbican.tasks.resources MissingArgumentError: Must supply non-None value argument for SecretStoreMetadatum entry.

When adding mode to the create_key method in castellan, and explicitly passing mode in the cinder/volume/flows/api/create_volume.py to this method (by specifying it in 'barbican:secret_mode' extra_spec of the LUKS volume type), volume gets created successfully.

Ideally, this secret mode should be added to the volume type POST method as a separate argument, with a corresponding field addition to VolumeTypeEncryption database model.

[Eric Harney (eharney)]

Can you explain a bit more why Cinder needs code to supply this dogtag flag? Can the right value not be supplied by Castellan by default?

[Vladyslav Drok (vdrok)]

Hey Eric, thanks for taking a look at this. Several things:

1. mode is not a dogtag flag, this is an argument in barbican in the create_key method -- https://github.com/openstack/python-barbicanclient/blob/4.6.0/barbicanclient/v1/orders.py#L442-L443
2. castellan is not a service, so can only provide default, which means there will be no way to specify which mode to use. we need to add mode argument here -- https://github.com/openstack/castellan/blob/0.17.0/castellan/key_manager/barbican_key_manager.py#L216-L217, that's why this bug also affects castellan
3. other parameters provided to the key creation method come from VolumeTypeEncryption -- https://github.com/openstack/cinder/blob/a6e79968ed237a7f0982cdc0d8fcf231d63b38fc/cinder/volume/utils.py#L900-L903

Why the mode is required in case of using dogtag backend but is not required when using eg simplecrypt backend might be a bug in barbican, but still would be good to allow providing mode through cinder.

[Vladyslav Drok (vdrok)]

It seems like here https://github.com/openstack/barbican/blob/d2ce5952ddaa6fd659b0dcc0a0f8f0eba6f3988c/barbican/plugin/dogtag.py#L588 dogtag does not return mode if it was not requested explicitly. So here it is returned https://github.com/openstack/barbican/blob/d2ce5952ddaa6fd659b0dcc0a0f8f0eba6f3988c/barbican/plugin/dogtag.py#L253, then from here https://github.com/openstack/barbican/blob/baf5edfbf9eff60ede03fd167e124960c126e525/barbican/plugin/resources.py#L279 it gets to here https://github.com/openstack/barbican/blob/baf5edfbf9eff60ede03fd167e124960c126e525/barbican/plugin/resources.py#L111, and then at https://github.com/openstack/barbican/blob/baf5edfbf9eff60ede03fd167e124960c126e525/barbican/plugin/resources.py#L348 barbican tries to save it into db.

Then during save this https://github.com/openstack/barbican/blob/0539a28ee95479a707b7dc3da635308a682e2ecc/barbican/model/repositories.py#L845 happens, and value is defined as not nullable here https://github.com/openstack/barbican/blob/0539a28ee95479a707b7dc3da635308a682e2ecc/barbican/model/models.py#L372

[Vladyslav Drok (vdrok)]

OTOH https://github.com/openstack/barbican/blob/d2ce5952ddaa6fd659b0dcc0a0f8f0eba6f3988c/barbican/plugin/dogtag.py#L588 it is checked that mode is added into metadata only if it is not None. So will need to debug this a bit more on the barbican side.

This is happening on pike setup BTW, but did not see much difference in code with master.

[Eric Harney (eharney)]

This sounds like an issue that needs to be fixed outside of Cinder.

[Vladyslav Drok (vdrok)]

This is valid only in pike.