v3 credentials client does not honour the configured domain

[kislotniq]

I'm running keystone in multi-domain setup:
+----------------------------------+---------+---------+--------------------------+
| ID | Name | Enabled | Description |
+----------------------------------+---------+---------+--------------------------+
| 1a37107e7eed4a73bf7de2c679bb9173 | heat | True | Stack projects and users |
| fd64f342726549e398e6225ffb645fa4 | default | True | Default Domain |
+----------------------------------+---------+---------+--------------------------+

Running tempest causes following exceptions:
{1} setUpClass (tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 273, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 261, in setUpClass
        cls.setup_credentials()
      File "/usr/local/lib/python2.7/dist-packages/tempest/api/compute/base.py", line 61, in setup_credentials
        super(BaseV2ComputeTest, cls).setup_credentials()
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 361, in setup_credentials
        credential_type=credentials_type)
      File "/usr/local/lib/python2.7/dist-packages/tempest/test.py", line 535, in get_client_manager
        creds = getattr(cred_provider, credentials_method)()
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 305, in get_primary_creds
        return self.get_credentials('primary')
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 286, in get_credentials
        credentials = self._create_creds(admin=is_admin)
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/dynamic_creds.py", line 138, in _create_creds
        username, user_password, project, email)
      File "/usr/local/lib/python2.7/dist-packages/tempest/common/cred_client.py", line 45, in create_user
        user = self.users_client.create_user(**params)
      File "/usr/local/lib/python2.7/dist-packages/tempest/services/identity/v3/json/users_clients.py", line 41, in create_user
        resp, body = self.post('users', post_body)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 273, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 667, in request
        resp, resp_body)
      File "/usr/local/lib/python2.7/dist-packages/tempest/lib/common/rest_client.py", line 765, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: {u'message': u'Could not find domain: default', u'code': 404, u'title': u'Not Found'}

I've noticed that the reason to this is in the tempest/services/identity/v3/json/users_clients.py, create_user - it has 'default' by default in the domain id.
As soon as I replace that default value with the id of my domain it works well.
So I see the following two problems, correct me if I'm wrong:
1. calling code does not pass domain id
2. the string 'default' cannot be a default value since id of domains are now normal ids, not human-readable strings

If this is a real issue and not my misconfiguration I can dig into the code and provide a patch.

[Jordan Pittier (jordan-pittier)]

There's a config flag named 'default_domain_id' in the 'identity' section. Have you tried it ?

[Dmitry Kudyukin (gmorfy)]

I've got the same problem. even using correct default_domain_id in tempest.conf. The cause of bug is tempest don't send domain_id in request to keystone. From keystone.log:

2016-10-06 06:26:08.526 25868 INFO tempest.lib.common.rest_client [req-ce5b8019-4b93-4f54-b6f4-8c6731023b30 ] Request (UsersV3TestJSON:test_user_update): 404 POST http://192.168.1.200:55357/v3/users 0.050s
2016-10-06 06:26:08.526 25868 DEBUG tempest.lib.common.rest_client [req-ce5b8019-4b93-4f54-b6f4-8c6731023b30 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"user": {"email": "<email address hidden>", "password": "I6~*9wqsP4gPIQI", "enabled": false, "name": "tempest-user-1809623701", "description":"tempest-user-1809623701description"}}
    Response - Headers: {'status': '404', 'content-length': '91', 'content-location': 'http://192.168.1.200:55357/v3/users', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5', 'connection': 'close', 'date': 'Thu, 06 Oct 2016 06:26:08 GMT', 'content-type': 'application/json', 'x-openstack-request-id': 'req-ce5b8019-4b93-4f54-b6f4-8c6731023b30'}
        Body: {"error": {"message": "Could not find domain: default", "code": 404, "title": "Not Found"}} _log_request_full tempest/lib/common/rest_client.py:431

I suppose to tempest use of default_domain_id from config file during this request. There is my patch for tempest, that solves the problem.

diff -uNr tempest.orig/common/cred_client.py tempest/common/cred_client.py
--- tempest.orig/common/cred_client.py 2016-10-05 05:20:12.692336942 +0000
+++ tempest/common/cred_client.py 2016-10-06 06:12:03.615336942 +0000
@@ -39,11 +39,12 @@
         self.projects_client = projects_client
         self.roles_client = roles_client

- def create_user(self, username, password, project, email):
+ def create_user(self, username, password, project, email, domain_id):
         params = {'name': username,
                   'password': password,
                   self.project_id_param: project['id'],
- 'email': email}
+ 'email': email,
+ 'domain_id': domain_id}
         user = self.users_client.create_user(**params)
         if 'user' in user:
             user = user['user']
diff -uNr tempest.orig/common/dynamic_creds.py tempest/common/dynamic_creds.py
--- tempest.orig/common/dynamic_creds.py 2016-10-06 04:51:15.965337000 +0000
+++ tempest/common/dynamic_creds.py 2016-10-06 06:07:15.444336942 +0000
@@ -134,8 +134,9 @@
         username = project_name
         user_password = data_utils.rand_password()
         email = data_utils.rand_name(root) + "@example.com"
+ domain_id = CONF.identity.default_domain_id
         user = self.creds_client.create_user(
- username, user_password, project, email)
+ username, user_password, project, email, domain_id)
         role_assigned = False
         if admin:
             self.creds_client.assign_user_role(user, project, self.admin_role)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/383601

[Daniel Mellado (daniel-mellado)]

Besides changing

default_domain_id = default

Did you try checking playing around with default_credentials_domain_name? This looks like a config issue to me

[Dmitry Kudyukin (gmorfy)]

Default domain id is not default for me:
openstack domain list
+----------------------------------+---------+---------+--------------------------+
| ID | Name | Enabled | Description |
+----------------------------------+---------+---------+--------------------------+
| 8cb83861ae81409a8f88300e3a3ef237 | default | True | Default Domain |
| c95f6e6234b9483b8311c4b7ecc728b5 | heat | True | Stack projects and users |
+----------------------------------+---------+---------+--------------------------+

I use such configuration option in conf file:
[DEFAULT]
debug = True
log_file = tempest.log
use_stderr = False
default_credentials_domain_name = default
[auth]
use_dynamic_credentials = True
admin_username = admin
admin_password = ***
admin_project_name = admin
admin_domain_name = default
default_credentials_domain_name = default
[identity]
region = RegionOne
auth_version = v3
uri = http://192.168.1.200:55357/v2.0
uri_v3 = http://192.168.1.200:55357/v3
disable_ssl_certificate_validation = True
ca_certificates_file =
default_domain_id = 8cb83861ae81409a8f88300e3a3ef237

The problem is still persist.

I think it because in _create_creds function from dynamic_creds.py the body request to keystone didnt depend any domain feature. From trace:

      File "tempest/common/dynamic_creds.py", line 303, in get_primary_creds
        return self.get_credentials('primary')
      File "tempest/common/dynamic_creds.py", line 284, in get_credentials
        credentials = self._create_creds(admin=is_admin)
      File "tempest/common/dynamic_creds.py", line 138, in _create_creds
        username, user_password, project, email)
      File "tempest/common/cred_client.py", line 47, in create_user
        user = self.users_client.create_user(**params)

[Andrea Frittoli (andrea-frittoli)]

The default domain is setup by default as default/Default in devstack.
That can be changed, which is why it's a configuration option.

stack@ubuntu-xenial:~/devstack$ openstack domain list
+----------------------------------+------------+---------+-----------------------------------+
| ID | Name | Enabled | Description |
+----------------------------------+------------+---------+-----------------------------------+
| default | Default | True | The default domain |
| f2ea5fb408784e589457c910591169dc | swift_test | True | Used for swift functional testing |
+----------------------------------+------------+---------+-----------------------------------+

The default_domain_id setting is only used by one test, which checks the existence of a default domain, so it's completely irrelevant in terms of credentials providers:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/identity/admin/v3/test_domains.py#n167

The default domain_id is not necessarily the domain where you want to provision your test accounts, which is why we have different settings for that, namely:

- when using dynamic credentials, the project domain name of the configured admin user (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/dynamic_creds.py#n103)

- when using pre-provisioned credentials, the domain specified in the YAML file (it loaded from the YAML file into a Credentials object)

- default_credentials_domain_name: this is used if no domain could be found with the methods above. It's passed to the credential providers and init time (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/credentials_factory.py#n44)

- 'Default' if the none of the above worked (http://git.openstack.org/cgit/openstack/tempest/tree/tempest/lib/common/cred_provider.py#n40)

V3 Users are provisioned without a domain ID, which causes keystone to provision them in the 'default' domain. This is wrong because Tempest should honour either the domain name of the admin user or the the default_credentials_domain_name setting.

So this is a bug, but the bug description is a bit misleading.

[Andrea Frittoli (andrea-frittoli)]

I set this as medium priority since we don't have any Tempest gate job running on a non-default domain.

[chandan kumar (chkumar246)]

Unassigning this bug as there is no activity from last 6 months. If you are still want to work on this bug, feel free to assign yourself.

[Martin Kopec (mkopec)]

Moving back to confirmed as nobody is working on this at the moment.

[Chiawei Xie (dommgifer)]

I got same issue in my mitaka version, but tempest testing ocata version has no problem.

Does anyone have ideas?

[Attila Fazekas (afazekas)]

CredsClient create_user still does not passes a domain argument. The issue has high heat so increasing the priority to high.

Ps.:

The id of the domain now a string(64) which contains an uuid `like` string, except the Default domain where the id is 'default'.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/642237

[Paras Babbar (pbabbar)]

Are you guys still facing this issue? or is it fixed?

[Martin Kopec (mkopec)]

https://review.opendev.org/c/openstack/tempest/+/642237 got merged.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 26.1.0 release.