Default domain no longer lets keystone tenant-list work

[Kris Lindgren]

We recently upgraded from kilo.0 to kilo.2 in our dev environment and noticed that keystone tenant-list is always failing for the admin user.

Our config is as follows default domain is tied to read-only ldap (AD), a heat domain is created to use for trusts to handle the created heatstack users/passwords. Under kilo.0 everything was happy. Under kilo0.2 we get the following error:

keystone tenant-list
The request you have made requires authentication. (HTTP 401) (Request-ID: req-d30289f0-778d-4577-8150-7ddd5438ad9c)

The main error message is:
2015-12-16 17:07:36.493 20386 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 10.224.48.132

Looking at the differences between kilo.0 and kilo.2 it seems like: https://github.com/openstack/keystone/commit/9dfad21201251364c6d205e8e79813bfe78e6107 is the most likely culprit for this regression. However, I have not yet been able to test if reverting that change fixes the issue.

[Kris Lindgren (klindgren)]

I reverted the above commit and restarted keystone. keystone teant-list works correctly again.

Kilo.2 with the above referenced commit:

$ keystone tenant-list
Invalid OpenStack Identity credentials.

kilo.2 with tht changed reverted:

$ keystone tenant-list
+----------------------------------+----------------------------------+---------+
| id | name | enabled |
+----------------------------------+----------------------------------+---------+
All projects.

Change that I made:
     224 def filter_domain_id(ref):
    225 """Remove domain_id since v2 calls are not domain-aware."""
    226 ref.pop('domain_id', None)
    227 #if 'domain_id' in ref:
    228 # if ref['domain_id'] != CONF.identity.default_domain_id:
    229 # raise exception.Unauthorized(
    230 # _('Non-default domain is not supported'))
    231 # del ref['domain_id']
    232 return ref

We attempted to make sure that the we set the default_domain_id to "default" which is the default - and we had the same unauthorized/invalid credentials error.

[Steve Martinelli (stevemar)]

I was unable to reproduce this problem in the latest kilo stream.

My kilo version is just a handful of commits ahead of the kilo.2 version you are using. None of which seem related to the code path in question.

I'm using version 2015.1.3
>>> import pbr.version
>>> keystone_version = pbr.version.VersionInfo('keystone')
>>> print keystone_version
2015.1.3

I use the following credentials:

$ env | grep OS
OS_REGION_NAME=RegionOne
OS_IDENTITY_API_VERSION=2.0
OS_PASSWORD=openstack
OS_AUTH_URL=http://172.16.240.156:5000/v2.0
OS_USERNAME=admin
OS_TENANT_NAME=admin
OS_VOLUME_API_VERSION=2

I try to list projects and everything turned out alright

$ keystone tenant-list
/usr/local/lib/python2.7/dist-packages/keystoneclient/shell.py:65: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
+----------------------------------+--------------------+---------+
| id | name | enabled |
+----------------------------------+--------------------+---------+
| 03cfc579679947dbba52d1a6e737873a | admin | True |
| 952bac021b14445bb52b27abe90c463b | alt_demo | True |
| 5785286638bf4c368292a5e0aa46ebb1 | demo | True |
| ce376a35838b46d687958e21d643e48b | invisible_to_admin | True |
| fb524d76208f44329e7b62c101115210 | service | True |
+----------------------------------+--------------------+---------+

Even with the new `openstack` CLI, things work.

$ openstack project list
+----------------------------------+--------------------+
| ID | Name |
+----------------------------------+--------------------+
| 03cfc579679947dbba52d1a6e737873a | admin |
| 5785286638bf4c368292a5e0aa46ebb1 | demo |
| 952bac021b14445bb52b27abe90c463b | alt_demo |
| ce376a35838b46d687958e21d643e48b | invisible_to_admin |
| fb524d76208f44329e7b62c101115210 | service |
+----------------------------------+--------------------+

What I'm slightly confused about is that the error you mentioned was "Invalid OpenStack Identity credentials", but the code you reference should produce "Non-default domain is not supported" instead.

    def filter_domain_id(ref):
        """Remove domain_id since v2 calls are not domain-aware."""
        if 'domain_id' in ref:
            if ref['domain_id'] != CONF.identity.default_domain_id:
                raise exception.Unauthorized(
                    _('Non-default domain is not supported'))
            del ref['domain_id']
        return ref

Can you try using --debug to the command that's failing and paste the output: keystone --debug tenant-list ?

It'll also help to see any logs from the server side, and the environment variables you're using with the command (using env | grep OS) so we can further investigate this problem.

[Kris Lindgren (klindgren)]

So I grabbed the wrong error messages:

2015-12-16 17:07:36.365 20386 INFO keystone.common.wsgi [-] GET https://openstack-dev.int.godaddy.com:35357/v2.0/tenants
2015-12-16 17:07:36.376 20386 DEBUG keystone.policy.backends.rules [-] enforce admin_required: {'user_id': u'xxxx', u'is_admin': 0, u'roles': [u'admin'], 'tenant_id': u'f48e57277a7a484290ba9afdc49a21a9'} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76
2015-12-16 17:07:36.377 20386 DEBUG oslo_policy.openstack.common.fileutils [-] Reloading cached file /etc/keystone/policy.json read_cached_file /usr/lib/python2.7/site-packages/oslo_policy/openstack/common/fileutils.py:64
2015-12-16 17:07:36.384 20386 DEBUG oslo_policy.policy [-] Reloaded policy file: /etc/keystone/policy.json _load_policy_file /usr/lib/python2.7/site-packages/oslo_policy/policy.py:425
2015-12-16 17:07:36.493 20386 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 10.224.48.132
2015-12-16 17:07:36.499 20386 INFO eventlet.wsgi.server [-] 10.224.48.132 - - [16/Dec/2015 17:07:36] "GET /v2.0/tenants HTTP/1.1" 401 454 0.236560

RC file we are using:
export OS_USERNAME=xxxx
export OS_TENANT_NAME=openstack
export OS_PROJECT_NAME=openstack
export OS_PASSWORD='xxxx'
export OS_IDENTITY_API_VERSION=2.0
export OS_AUTH_URL=https://openstack-dev.int.godaddy.com:35357/v2.0/
export OS_AUTH_URL_V3=https://openstack-dev.int.godaddy.com:35357/v3/

The default domain is tied to read-only ldap and the admin user is a user in the ldap domain.

[Steve Martinelli (stevemar)]

Hey Kris,

Apologies for not getting back to you sooner, I missed your update during the holidays. Here's my understanding of the situation.

It looks like the only way to hit the issue is if, a) you are authenticating and using v2.0 APIs, and b) the user is coming from the non-default domain. Remember that in v2.0 we do not have the concept of a domain, and as a result, the v2.0 APIs assume the default domain.

So a few more comments so we can narrow down the problem.

You mention that the admin user is tied to the default domain, so you only have a single domain in your deployment?

What's the value of `default_domain_id` in the config file, it's under the [identity] section?

If using v3 of the APIs, I believe this issue should go away, could you try that to make sure? A quick test would be:

export OS_USERNAME=xxxx
export OS_USER_DOMAIN_NAME=xxxx
export OS_PROJECT_NAME=openstack
export OS_PROJECT_DOMAIN_NAME=xxxx
export OS_PASSWORD='xxxx'
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=https://openstack-dev.int.godaddy.com:35357/v3/

then issue: `openstack project list`

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/265002
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=c75f39f3d6e4a2caa37322adcf2e296ec7c573c8
Submitter: Jenkins
Branch: master

commit c75f39f3d6e4a2caa37322adcf2e296ec7c573c8
Author: Morgan Fainberg <email address hidden>
Date: Thu Jan 7 15:18:03 2016 -0800

    Revert "Validate domain ownership for v2 tokens"

    This reverts commit c4723550aa95be403ff591dd132c9024549eff10.

    This revert is being proposed as it breaks behavior that real-world
    deployments rely on. The deployments requested the V2 token with
    user_id and tenantId and then used the V2 token for the
    non-default-domain user to access swift.

    While the deployment is being encouraged to fix their code to use V3,
    this is behavior that was supported and used.

    This revert was done by hand due to the volume of change that has
    occured to the tests since the original patch landed.

    Conflicts (a lot of test refactoring):
        keystone/tests/unit/test_v3_assignment.py
        keystone/tests/unit/test_v3_auth.py
        keystone/tests/unit/test_v3_identity.py

    Change-Id: I4a303a5fcc8c2dacef5960e9e26ad9402f34a790
    Closes-Bug: 1527759

[Steve Martinelli (stevemar)]

patch for liberty: https://review.openstack.org/#/c/265023/

patch for kilo: https://review.openstack.org/#/c/265019/

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/265023
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7724b912b81c0e3d452f9d87f8d32257ced0dec0
Submitter: Jenkins
Branch: stable/liberty

commit 7724b912b81c0e3d452f9d87f8d32257ced0dec0
Author: Morgan Fainberg <email address hidden>
Date: Thu Jan 7 15:18:03 2016 -0800

    Revert "Validate domain ownership for v2 tokens"

    This reverts commit c4723550aa95be403ff591dd132c9024549eff10.

    This revert is being proposed as it breaks behavior that real-world
    deployments rely on. The deployments requested the V2 token with
    user_id and tenantId and then used the V2 token for the
    non-default-domain user to access swift.

    While the deployment is being encouraged to fix their code to use V3,
    this is behavior that was supported and used.

    Conflicts (new file post Liberty):
        keystone/tests/unit/test_v3_resource.py

    Closes-Bug: 1527759
    Change-Id: I4a303a5fcc8c2dacef5960e9e26ad9402f34a790

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/keystone 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Morgan Fainberg (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/265019
Reason: Abandoned, Kilo is near EOL and this is not an easy revert. The people asking for it were all on liberty and beyond. This can be carried locally as needed if a deployer is relying on this in kilo... or upgrade to liberty

[Morgan Fainberg (mdrnstm)]

Wont fix in kilo