v3 credentials client does not honour the configured domain

Bug #1613819 reported by kislotniq on 2016-08-16
This bug affects 10 people
Affects Status Importance Assigned to Milestone
Dmitrii Shcherbakov

Bug Description

I'm running keystone in multi-domain setup:
| ID | Name | Enabled | Description |
| 1a37107e7eed4a73bf7de2c679bb9173 | heat | True | Stack projects and users |
| fd64f342726549e398e6225ffb645fa4 | default | True | Default Domain |

Running tempest causes following exceptions:
{1} setUpClass (tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON) [0.000000s] ... FAILED

Captured traceback:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 273, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 261, in setUpClass
      File "/usr/local/lib/python2.7/dist-packages/tempest/api/compute/base.py", line 61, in setup_credentials
        super(BaseV2ComputeTest, cls).setup_credentials()
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 361, in setup_credentials
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 535, in get_client_manager
        creds = getattr(cred_provider, credentials_method)()
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 305, in get_primary_creds
        return self.get_credentials('primary')
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 286, in get_credentials
        credentials = self._create_creds(admin=is_admin)
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 138, in _create_creds
        username, user_password, project, email)
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/cred_client.py", line 45, in create_user
        user = self.users_client.create_user(**params)
      File "/usr/local/lib/python2.7/dist-packages/tempest/services/identity/v3/json/users_clients.py", line 41, in create_user
        resp, body = self.post('users', post_body)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 273, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 667, in request
        resp, resp_body)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 765, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: {u'message': u'Could not find domain: default', u'code': 404, u'title': u'Not Found'}

I've noticed that the reason to this is in the tempest/services/identity/v3/json/users_clients.py, create_user - it has 'default' by default in the domain id.
As soon as I replace that default value with the id of my domain it works well.
So I see the following two problems, correct me if I'm wrong:
1. calling code does not pass domain id
2. the string 'default' cannot be a default value since id of domains are now normal ids, not human-readable strings

If this is a real issue and not my misconfiguration I can dig into the code and provide a patch.

Jordan Pittier (jordan-pittier) wrote :

There's a config flag named 'default_domain_id' in the 'identity' section. Have you tried it ?

Dmitry Kudyukin (gmorfy) wrote :

I've got the same problem. even using correct default_domain_id in tempest.conf. The cause of bug is tempest don't send domain_id in request to keystone. From keystone.log:

2016-10-06 06:26:08.526 25868 INFO tempest.lib.common.rest_client [req-ce5b8019-4b93-4f54-b6f4-8c6731023b30 ] Request (UsersV3TestJSON:test_user_update): 404 POST 0.050s
2016-10-06 06:26:08.526 25868 DEBUG tempest.lib.common.rest_client [req-ce5b8019-4b93-4f54-b6f4-8c6731023b30 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"user": {"email": "<email address hidden>", "password": "I6~*9wqsP4gPIQI", "enabled": false, "name": "tempest-user-1809623701", "description":"tempest-user-1809623701description"}}
    Response - Headers: {'status': '404', 'content-length': '91', 'content-location': '', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5', 'connection': 'close', 'date': 'Thu, 06 Oct 2016 06:26:08 GMT', 'content-type': 'application/json', 'x-openstack-request-id': 'req-ce5b8019-4b93-4f54-b6f4-8c6731023b30'}
        Body: {"error": {"message": "Could not find domain: default", "code": 404, "title": "Not Found"}} _log_request_full tempest/lib/common/rest_client.py:431

I suppose to tempest use of default_domain_id from config file during this request. There is my patch for tempest, that solves the problem.

diff -uNr tempest.orig/common/cred_client.py tempest/common/cred_client.py
--- tempest.orig/common/cred_client.py 2016-10-05 05:20:12.692336942 +0000
+++ tempest/common/cred_client.py 2016-10-06 06:12:03.615336942 +0000
@@ -39,11 +39,12 @@
         self.projects_client = projects_client
         self.roles_client = roles_client

- def create_user(self, username, password, project, email):
+ def create_user(self, username, password, project, email, domain_id):
         params = {'name': username,
                   'password': password,
                   self.project_id_param: project['id'],
- 'email': email}
+ 'email': email,
+ 'domain_id': domain_id}
         user = self.users_client.create_user(**params)
         if 'user' in user:
             user = user['user']
diff -uNr tempest.orig/common/dynamic_creds.py tempest/common/dynamic_creds.py
--- tempest.orig/common/dynamic_creds.py 2016-10-06 04:51:15.965337000 +0000
+++ tempest/common/dynamic_creds.py 2016-10-06 06:07:15.444336942 +0000
@@ -134,8 +134,9 @@
         username = project_name
         user_password = data_utils.rand_password()
         email = data_utils.rand_name(root) + "@example.com"
+ domain_id = CONF.identity.default_domain_id
         user = self.creds_client.create_user(
- username, user_password, project, email)
+ username, user_password, project, email, domain_id)
         role_assigned = False
         if admin:
             self.creds_client.assign_user_role(user, project, self.admin_role)

information type: Public → Public Security
Dmitry Kudyukin (gmorfy) on 2016-10-06
Changed in tempest:
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/383601

Changed in tempest:
assignee: nobody → Dmitry Kudyukin (gmorfy)
status: Confirmed → In Progress

Besides changing

default_domain_id = default

Did you try checking playing around with default_credentials_domain_name? This looks like a config issue to me

Dmitry Kudyukin (gmorfy) wrote :

Default domain id is not default for me:
openstack domain list
| ID | Name | Enabled | Description |
| 8cb83861ae81409a8f88300e3a3ef237 | default | True | Default Domain |
| c95f6e6234b9483b8311c4b7ecc728b5 | heat | True | Stack projects and users |

I use such configuration option in conf file:
debug = True
log_file = tempest.log
use_stderr = False
default_credentials_domain_name = default
use_dynamic_credentials = True
admin_username = admin
admin_password = ***
admin_project_name = admin
admin_domain_name = default
default_credentials_domain_name = default
region = RegionOne
auth_version = v3
uri =
uri_v3 =
disable_ssl_certificate_validation = True
ca_certificates_file =
default_domain_id = 8cb83861ae81409a8f88300e3a3ef237

The problem is still persist.

I think it because in _create_creds function from dynamic_creds.py the body request to keystone didnt depend any domain feature. From trace:

      File "tempest/common/dynamic_creds.py", line 303, in get_primary_creds
        return self.get_credentials('primary')
      File "tempest/common/dynamic_creds.py", line 284, in get_credentials
        credentials = self._create_creds(admin=is_admin)
      File "tempest/common/dynamic_creds.py", line 138, in _create_creds
        username, user_password, project, email)
      File "tempest/common/cred_client.py", line 47, in create_user
        user = self.users_client.create_user(**params)

The default domain is setup by default as default/Default in devstack.
That can be changed, which is why it's a configuration option.

stack@ubuntu-xenial:~/devstack$ openstack domain list
| ID | Name | Enabled | Description |
| default | Default | True | The default domain |
| f2ea5fb408784e589457c910591169dc | swift_test | True | Used for swift functional testing |

The default_domain_id setting is only used by one test, which checks the existence of a default domain, so it's completely irrelevant in terms of credentials providers:


The default domain_id is not necessarily the domain where you want to provision your test accounts, which is why we have different settings for that, namely:

- when using dynamic credentials, the project domain name of the configured admin user (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/dynamic_creds.py#n103)

- when using pre-provisioned credentials, the domain specified in the YAML file (it loaded from the YAML file into a Credentials object)

- default_credentials_domain_name: this is used if no domain could be found with the methods above. It's passed to the credential providers and init time (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/credentials_factory.py#n44)

- 'Default' if the none of the above worked (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/lib/common/cred_provider.py#n40)

V3 Users are provisioned without a domain ID, which causes keystone to provision them in the 'default' domain. This is wrong because Tempest should honour either the domain name of the admin user or the the default_credentials_domain_name setting.

So this is a bug, but the bug description is a bit misleading.

summary: - v3 identity client has bad default domain id
+ v3 credentials client does not honour the configured domain
Changed in tempest:
importance: Undecided → Medium

I set this as medium priority since we don't have any Tempest gate job running on a non-default domain.

chandan kumar (chkumar246) wrote :

Unassigning this bug as there is no activity from last 6 months. If you are still want to work on this bug, feel free to assign yourself.

Changed in tempest:
assignee: Dmitry Kudyukin (gmorfy) → nobody
Martin Kopec (mkopec) wrote :

Moving back to confirmed as nobody is working on this at the moment.

Changed in tempest:
status: In Progress → Confirmed
Chiawei Xie (dommgifer) wrote :

I got same issue in my mitaka version, but tempest testing ocata version has no problem.

Does anyone have ideas?

Attila Fazekas (afazekas) wrote :

CredsClient create_user still does not passes a domain argument. The issue has high heat so increasing the priority to high.


The id of the domain now a string(64) which contains an uuid `like` string, except the Default domain where the id is 'default'.

Changed in tempest:
importance: Medium → High

Fix proposed to branch: master
Review: https://review.openstack.org/642237

Changed in tempest:
assignee: nobody → Dmitrii Shcherbakov (dmitriis)
status: Confirmed → In Progress
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers