Cinder backup of en encrypted RBD volume fails

[Tzach Shefi]

Description of problem: When I try to backup an RBD backed Cinder (attached or not) volume backup fails. Backup of a none encrypted volume works fine.

Version-Release number of selected component (if applicable):
rhel 7.5
openstack-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch
python2-cinderclient-3.5.0-1.el7ost.noarch
python-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch

How reproducible:
Every time

Steps to Reproduce:
1. Create an encrypted RBD backed Cinder volume

2. Try to backup that volume resulting in backup error state.

$cinder backup-create 84eae95b-efed-441a-880a-bd35af1aadcd --name BackupOfEncNoneAttVolFails
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | fd55218a-f26c-45a0-abd6-03ccabbd448a |
| name | BackupOfEncNoneAttVolFails |
| volume_id | 84eae95b-efed-441a-880a-bd35af1aadcd |
+-----------+--------------------------------------+

cinder backup-show fd55218a-f26c-45a0-abd6-03ccabbd448a | grep status
| status | error

3. Backup of a none encrypted volume works fine

 cinder backup-create 9a90e8dd-59c2-445c-961c-e2dd3413289b --force --name WorkingBackupOfNoneEncAttVol
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | 781434ee-ade8-4f44-85de-be0cf11b84c4 |
| name | WorkingBackupOfNoneEncAttVol |
| volume_id | 9a90e8dd-59c2-445c-961c-e2dd3413289b |
+-----------+--------------------------------------+

$ cinder backup-show 781434ee-ade8-4f44-85de-be0cf11b84c4 | grep status
| status | available

Actual results:
Backup should complete

Expected results:
Backup status error.

Additional info:

On the same system a fellow coworker Archit helped debug/get Barbican working with Cinder. Was failing to create an encrypted volume, this is what he suggested/did on my system ->
-----
Barbican is running on your setup on tigris01 now. The issue was that the auth_endpoint in cinder.conf [barbican] section was pointing to a versionless auth url and it was going to ip:5000/v2 while v2 is deprecated in RHOS-13.
-----
This here backup bug might be more of the same. As backup log trackback also complains about keystone resource not being found.

2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server [req-70762a72-90d4-42f5-8a60-7b3c160254a4 a2707f862dba4a43a5fb3f5698f0ed64 c662a91a23e44faab47f1b319495a73f - - -] Exception during message handling: NotFound: (http://172.17.1.12:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-0cb44c9a-7a6c-4d74-a313-0eb6a170a0ec)
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-04-15 11:53:25.143 1 ERROR oslo_messaging.rpc.server

More can be see on attached Cinder backup log.
Look towards end of log, as head of the file contains previous attempts, which wont fit above mentioned IDs