keystone role names lack case sensitivity - admin == Admin

[Nicolas Thomas]

I am using 3 keystone units connecting to Percona. The only obvious change from previously working one is move to Juju 1.25

keystone/0 end up in error in shared-db-relation-changed keystone/1 and /2 does not report issues.

Using : cs:trusty/keystone-31

Moving to debug-mode and debug Python I got the following :

Traceback (most recent call last):
  File "/usr/lib/python2.7/pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/usr/lib/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "hooks/shared-db-relation-changed", line 647, in <module>
    main()
  File "hooks/shared-db-relation-changed", line 639, in main
    hooks.execute(sys.argv)
  File "hooks/charmhelpers/core/hookenv.py", line 704, in execute
    self._hooks[hook_name]()
  File "hooks/charmhelpers/core/host.py", line 393, in wrapped_f
    f(*args, **kwargs)
  File "hooks/keystone_utils.py", line 1224, in inner_synchronize_ca_if_changed2
    return f(*args, **kwargs)
  File "hooks/shared-db-relation-changed", line 294, in db_changed
    update_all_identity_relation_units(check_db_ready=False)
  File "hooks/shared-db-relation-changed", line 266, in update_all_identity_relation_units
    identity_changed(relation_id=rid, remote_unit=unit)
  File "hooks/charmhelpers/core/host.py", line 393, in wrapped_f
    f(*args, **kwargs)
  File "hooks/keystone_utils.py", line 1224, in inner_synchronize_ca_if_changed2
    return f(*args, **kwargs)
  File "hooks/shared-db-relation-changed", line 335, in identity_changed
    add_service_to_keystone(relation_id, remote_unit)
  File "hooks/keystone_utils.py", line 1503, in add_service_to_keystone
    new_roles=roles)
  File "hooks/keystone_utils.py", line 1368, in create_service_credentials
    grants=[config('admin-role')])
  File "hooks/keystone_utils.py", line 1351, in create_user_credentials
    create_role(role, user, tenant)
  File "hooks/keystone_utils.py", line 594, in create_role
    manager.api.roles.create(name=name)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/roles.py", line 39, in create
    return self._create('/OS-KSADM/roles', params, "role")
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 150, in _create
    return self._post(url, body, response_key, return_raw, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 164, in _post
    resp, body = self.client.post(url, body=body, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 617, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 603, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 578, in request
    resp = super(HTTPClient, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 339, in request
    raise exceptions.from_response(resp, method, url)
Conflict: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/lib/python2.7/dist-packages/keystoneclient/session.py(339)request()

(Pdb) args
self = <keystoneclient.session.Session object at 0x7f88835c9710>
url = http://localhost:35347/v2.0/OS-KSADM/roles
method = POST
json = {'role': {'name': u'Admin'}}
original_ip = None
user_agent = python-keystoneclient
redirect = 30
authenticated = True
endpoint_filter = {'service_type': 'identity', 'interface': 'admin', 'version': 'v2.0'}
auth = None
requests_auth = None
raise_exc = True
allow_reauth = True
log = True
kwargs = {'headers': {'User-Agent': 'python-keystoneclient', 'Content-Type': 'application/json', 'X-Auth-Token': 'admin'}, 'allow_redirects': False, 'data': '{"role": {"name": "Admin"}}', 'verify': True}

[Nicolas Thomas (thomnico)]

Find attached the juju log of the deployment

[Nicolas Thomas (thomnico)]

Logs from another attempt was not an issue just a few hours before that run

[Nicolas Thomas (thomnico)]

Bundle used to trigger the issue

[David Ames (thedac)]

Nicolas,

While I set up tests to re-create this bug. Can you please test using the min-cluster-size setting for percona-cluster. This setting is required to let percona know it needs to cluster. This setting is missing from the bundle you have provided and may be a factor in the bug.

percona-cluster
min-cluster-size: 3

[Nicolas Thomas (thomnico)]

Testing right now .. looks like the workaround (1 keystone related
then add-unit create issue with nova-control probably linked to
reducing the cluster count then expanding during deployment).

On Wed, Nov 4, 2015 at 7:15 PM, David Ames <email address hidden> wrote:
> Nicolas,
>
> While I set up tests to re-create this bug. Can you please test using
> the min-cluster-size setting for percona-cluster. This setting is
> required to let percona know it needs to cluster. This setting is
> missing from the bundle you have provided and may be a factor in the
> bug.
>
> percona-cluster
> min-cluster-size: 3
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1512984
>
> Title:
> hook shared-db relation fail wiht role already exist
>
> Status in keystone package in Juju Charms Collection:
> New
>
> Bug description:
>
> I am using 3 keystone units connecting to Percona. The only obvious change from previously working one is move to Juju 1.25
>
> keystone/0 end up in error in shared-db-relation-changed keystone/1
> and /2 does not report issues.
>
> Using : cs:trusty/keystone-31
>
> Moving to debug-mode and debug Python I got the following :
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/pdb.py", line 1314, in main
> pdb._runscript(mainpyfile)
> File "/usr/lib/python2.7/pdb.py", line 1233, in _runscript
> self.run(statement)
> File "/usr/lib/python2.7/bdb.py", line 400, in run
> exec cmd in globals, locals
> File "<string>", line 1, in <module>
> File "hooks/shared-db-relation-changed", line 647, in <module>
> main()
> File "hooks/shared-db-relation-changed", line 639, in main
> hooks.execute(sys.argv)
> File "hooks/charmhelpers/core/hookenv.py", line 704, in execute
> self._hooks[hook_name]()
> File "hooks/charmhelpers/core/host.py", line 393, in wrapped_f
> f(*args, **kwargs)
> File "hooks/keystone_utils.py", line 1224, in inner_synchronize_ca_if_changed2
> return f(*args, **kwargs)
> File "hooks/shared-db-relation-changed", line 294, in db_changed
> update_all_identity_relation_units(check_db_ready=False)
> File "hooks/shared-db-relation-changed", line 266, in update_all_identity_relation_units
> identity_changed(relation_id=rid, remote_unit=unit)
> File "hooks/charmhelpers/core/host.py", line 393, in wrapped_f
> f(*args, **kwargs)
> File "hooks/keystone_utils.py", line 1224, in inner_synchronize_ca_if_changed2
> return f(*args, **kwargs)
> File "hooks/shared-db-relation-changed", line 335, in identity_changed
> add_service_to_keystone(relation_id, remote_unit)
> File "hooks/keystone_utils.py", line 1503, in add_service_to_keystone
> new_roles=roles)
> File "hooks/keystone_utils.py", line 1368, in create_service_credentials
> grants=[config('admin-role')])
> File "hooks/keystone_utils.py", line 1351, in create_user_credentials
> create_role(role, user, tenant)
> File "hooks/keystone_utils.py", line 594, in create_role
> manager.api.roles.create(name=name)
> Fil...

[Nicolas Thomas (thomnico)]

Tested with min-size-cluster = 3 .. the initial connection works but now I have the same :

    raise exceptions.from_response(resp, method, url)
keystoneclient.openstack.common.apiclient.exceptions.Conflict: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)

When other services tries to relate like swift, opnestack etc.. (all of them).

[Nicolas Thomas (thomnico)]

After moving to debug-hooks and simply clear the errors (not retry) the resulting cloud works ..

It looks simply annoying. Find attached logs of the last try

[Nicolas Thomas (thomnico)]

This bugs kicks again .. when moving to debug mode and clearing the error for Juju .. the cloud works.

Please add a check before insert asap.

[James Page (james-page)]

There is something wonky going on with regards to role creation:

unit-keystone-0[909]: 2015-11-05 10:44:08 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Created new role 'admin'"]
unit-keystone-0[909]: 2015-11-05 10:44:08 DEBUG unit.keystone/0.juju-log server.go:268 shared-db:38: Created new role 'admin'
unit-keystone-0[909]: 2015-11-05 10:51:57 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Created new role 'Member'"]
unit-keystone-0[909]: 2015-11-05 10:51:57 DEBUG unit.keystone/0.juju-log server.go:268 identity-service:80: Created new role 'Member'
unit-keystone-0[909]: 2015-11-05 13:56:29 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Created new role 'ResellerAdmin'"]
unit-keystone-0[909]: 2015-11-05 13:56:29 DEBUG unit.keystone/0.juju-log server.go:268 identity-service:112: Created new role 'ResellerAdmin'

Note that a lowercase admin role gets created early in unit lifecycle - not quite sure where that comes from.

[James Page (james-page)]

I suspect that keystone sees 'admin' and 'Admin' as the same thing from a role name perspective; the problem is that the role created by default is currently all lowercase, whereas the role requested via swift is not - the code checks but is case sensitive.

We should fix that, but the root cause of the lowercase role creation is bemusing - its default is 'Admin' in config, not 'admin' and that's used raw by the charm.

[James Page (james-page)]

1.24.7 config-get admin-role returns 'Admin'

checking with 1.25.0.

[Nicolas Thomas (thomnico)]

I my environment I set :
ubuntu@maas:~/orange-box-examples/SDN/contrail.thomnico⟫ juju get keystone | grep admin
  admin-password:
    value: admin
  admin-port:
  admin-role:
    description: Admin role to be associated with admin and service users
    value: admin
  admin-token:
    value: admin
  admin-user:
    description: Default admin user to create and manage.
    value: admin
  keystone-admin-role:
    description: Role that allows admin operations (access to all operations).
  keystone-service-admin-role:
    description: Role that allows acting as service admin.
  os-admin-network:
      This network will be used for admin endpoints.
  service-admin-prefix:
      service admin_username in keystone. The name used may be too crude for
      option will be prepended to each service admin_username.

[James Page (james-page)]

Well that's the problem:

  admin-role:
    description: Admin role to be associated with admin and service users
    value: admin

That's not the default value; the charm could handle this better, but I'm pondering how that became 'admin' - I don't see that being set in the bundle attached.

[Nicolas Thomas (thomnico)]

http://paste.ubuntu.com/13123271/ for the full output

[James Page (james-page)]

OK so I missed:

  overrides:
    admin-role: admin

in the bundle - this is the cause of the problem; switching back to 'Admin' or setting the roles in swift-proxy to 'member,admin' will probably resolve this problem.

[James Page (james-page)]

Code snippet:

def create_role(name, user=None, tenant=None):
    """Creates a role if it doesn't already exist. grants role to user"""
    import manager
    manager = manager.KeystoneManager(endpoint=get_local_endpoint(),
                                      token=get_admin_token())
    roles = [r._info for r in manager.api.roles.list()]
    if not roles or name not in [r['name'] for r in roles]:
        manager.api.roles.create(name=name)
        log("Created new role '%s'" % name, level=DEBUG)
    else:
        log("A role named '%s' already exists" % name, level=DEBUG)

Keystoneclient returns role names as created = however it will throw a conflict if you try to create a role with different capitalization:

admin
Admin

will conflict by the looks of things.

[James Page (james-page)]

Reading around this, the error is being thrown by the sql backend for roles in keystone; the name column has a unique key, which by default in mysql is case insensitive, so mysql will see admin == Admin during insertion.

[James Page (james-page)]

Openstack projects should all be comparing role membership without case sensitivity, so we should be good to update the create_role pre-creation check to drop everything to lower, and to create any new roles as lower case.

We should update the configuration defaults in swift-proxy and ceilometer as well to be lower case.

[James Page (james-page)]

Dropping this from Critical to Medium as we have a workaround.

[James Page (james-page)]

All four charms have configuration options related to roles that need to be reviewed.

I think dropping all to lower case works just fine, with some updates to the keystone utils code to always convert things to lowercase for roles and users.

[James Page (james-page)]

We might want todo a one-off update of all roles to lowercase as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293043

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293043
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=3163574413a1739f0a9a4853d27bbe19f4520219
Submitter: Jenkins
Branch: master

commit 3163574413a1739f0a9a4853d27bbe19f4520219
Author: David Ames <email address hidden>
Date: Tue Mar 15 09:48:29 2016 -0700

    Fix case-insensitivity for roles/users/tenants

    When checking for existing roles/users/tenants the charm was case
    sensitive such that admin != Admin. However, when keystone tries to
    create a role/user/tenant that exists but with different case mysql will
    error out. OpenNFV requires that the admin user be named 'admin' with
    lower case but the default is 'Admin' leading to failed deploys of
    OpenStack.

    This change makes the check for existence case insensitive. It does
    *not* change the creation of roles/users/tenants. Therefore,
    roles/users/tenants will be created unchanged but checks for existence
    will still match even when case does not.

    Change-Id: I49c4f5e8d0e79f64fbc8bf412341a93f4a970778
    Closes-Bug: #1512984