test_boot_server_from_encrypted_volume_luks failing with u'Key manager error on stable/stein scenario 2

[chandan kumar]

http://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario002-standalone&pipeline=check&branch=stable%2Fstein

standalone scenario 2 is failing at following tempest tests
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_214/705628/1/check/tripleo-ci-centos-7-scenario002-standalone/2144804/logs/undercloud/home/zuul/tempest.log

2020-02-04 23:40:34 |
2020-02-04 23:40:34 | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks[compute,id-cb78919a-e553-4bab-b73b-10cf4d2eb125,volume]
2020-02-04 23:40:34 | -------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-02-04 23:40:34 |
2020-02-04 23:40:34 | Captured traceback:
2020-02-04 23:40:34 | ~~~~~~~~~~~~~~~~~~~
2020-02-04 23:40:34 | Traceback (most recent call last):
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
2020-02-04 23:40:34 | return f(*func_args, **func_kwargs)
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 286, in test_boot_server_from_encrypted_volume_luks
2020-02-04 23:40:34 | volume_type='luks')
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 1361, in create_encrypted_volume
2020-02-04 23:40:34 | return self.create_volume(volume_type=volume_type['name'])
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 319, in create_volume
2020-02-04 23:40:34 | volume = self.volumes_client.create_volume(**kwargs)['volume']
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/v3/volumes_client.py", line 97, in create_volume
2020-02-04 23:40:34 | resp, body = self.post('volumes', post_body)
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 283, in post
2020-02-04 23:40:34 | return self.request('POST', url, extra_headers, headers, body, chunked)
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/volume/base_client.py", line 38, in request
2020-02-04 23:40:34 | method, url, extra_headers, headers, body, chunked)
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 679, in request
2020-02-04 23:40:34 | self._error_checker(resp, resp_body)
2020-02-04 23:40:34 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 790, in _error_checker
2020-02-04 23:40:34 | raise exceptions.BadRequest(resp_body, resp=resp)
2020-02-04 23:40:34 | tempest.lib.exceptions.BadRequest: Bad request
2020-02-04 23:40:34 | Details: {u'message': u'Key manager error', u'code': 400}
2020-02-04 23:40:34 |

While looking in details

http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_214/705628/1/check/tripleo-ci-centos-7-scenario002-standalone/2144804/logs/undercloud/var/log/containers/cinder/cinder-api.log

2020-02-04 23:23:10.706 19 ERROR castellan.key_manager.barbican_key_manager [req-53e26409-1c9b-422e-81f7-9ef860240d63 832a49e6af594bbab0f459bc5bf6d04b 49cb070db8cb4128a57ed4c13f752196 - default default] Error creating key: Forbidden: Order creation attempt not allowed - please review your user/project privileges: HTTPClientError: Forbidden: Order creation attempt not allowed - please review your user/project privileges
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils [req-53e26409-1c9b-422e-81f7-9ef860240d63 832a49e6af594bbab0f459bc5bf6d04b 49cb070db8cb4128a57ed4c13f752196 - default default] Key manager error: KeyManagerError: Key manager error: Forbidden: Order creation attempt not allowed - please review your user/project privileges
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils Traceback (most recent call last):
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils File "/usr/lib/python2.7/site-packages/cinder/volume/utils.py", line 931, in create_encryption_key
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils length=length)
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 251, in create_key
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils raise exception.KeyManagerError(reason=e)
2020-02-04 23:23:10.707 19 ERROR cinder.volume.utils KeyManagerError: Key manager error: Forbidden: Order creation attempt not allowed - please review your user/project privileges

It appears to be some issue with cinder and barbican communication.

Currently we are going to skip the test and investigate on that.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/706215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706215
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=699355b71b5272832a1a3c4632dbc3ecaeee66ad
Submitter: Zuul
Branch: master

commit 699355b71b5272832a1a3c4632dbc3ecaeee66ad
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 6 15:48:21 2020 +0530

    Updated tempest skip list for stein

    Currently scenario 1 and 4 standalone stable/stein check and gate
    job is broken due to volume encryption and swift related tempest
    tests failure.

    In order to unblock the gates, we are currently moving tests to
    skip list.

    Related-Bug: #1862143
    Related-Bug: #1862142

    Depends-On: https://review.opendev.org/#/c/706196/

    Change-Id: Ieabc0d56e01ced15860da5642c9d62c620e9c0db
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707461

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707461
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=ece4dbc9f4f8e5e37a8b8dd9d2aa2fafa50d255c
Submitter: Zuul
Branch: master

commit ece4dbc9f4f8e5e37a8b8dd9d2aa2fafa50d255c
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 13 01:11:27 2020 +0530

    Unskip test_boot_server_from_encrypted_volume_luks from stein

    Fixed is merged in this review https://review.opendev.org/706452

    Closes-Bug: #1862142

    Change-Id: I23dd5fc3855d1f538267e52f81c66ec11669373d
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>