User <user_id> is unauthorized for tenant <tenant_id> from keystone during rally test

[Leontii Istomin]

* Create user with name XXX, Create tenant with name YYY
* Do authentication of this user
* Delete recently create user & tenant
* Create user and tenant with the same name as in first step
* Try to authenticate multiple times in parallel
* Some of iterations will fail with error message "Unauthorized: User ...."

Simplest way to reproduce bug:

rally task start <task_file.yaml> --task-args '{"c": 10, "t": 300}'
---
  Authenticate.validate_glance:
    -
      args:
        repetitions: 1
      runner:
        type: "constant"
        times: {{t}}
        concurrency: {{c}}
      context:
        users:
          tenants: 1
          users_per_tenant: 1

* During the task validation (we are creating and deleting user & tenant):
2015-09-09 06:10:26.751 14888 DEBUG keystoneclient.session [-] REQ: curl -g -i -X DELETE http://192.168.0.2:35357/v2.0/tenants/5476f7af15dc493eacb08df12fa91552 -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}596a02bbf15c358f68de5552222f4ad09c24bdce" _http_log_request /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:195

* During the load we are using pre-created user & tenant. And one of iteration failed like:

2015-09-09 06:10:27.774 15091 ERROR rally.benchmark.runners.base Unauthorized: User 84f690c069ce4a4f89e4eb7929c0cd4a is unauthorized for tenant 5476f7af15dc493eacb08df12fa91552 (Disable debug mode to suppress these details.) (HTTP 401) (Request-ID: req-8bfb4abb-42db-43ef-b00a-3891bcd1bc56)

In other words we are referring during load and validation to the same tenant_id which is impossible If we take into account 3 facts:

- Tenant & User during validation & load are different
- Tenant & User during validation & load have the same name
- Keystone authentication is done via user name and tenant name

It becomes clear that this bug is not related to our testing tool & scripts.
We have some issues in keystone or it's configuration.

Related bug from rally project: https://bugs.launchpad.net/rally/+bug/1477156

[Leontii Istomin (listomin)]

This bug hasn't been reproduced with 7.0-182 and has been reproduced with 7.0-259

[Georgy Okrokvertskhov (gokrokvertskhov)]

Hi everybody,

Recently our Scale tests started failing (even Rally light runs).

During todays debugging hackathon I collected bunch of interesting information:

###
Rally light scenarios are run in parallel actually.

###
MOS Scale scripts work properly (at least they are not related to this bug)

###
Rally works properly:
*) I checked that new keystone client release 1.7.0 doesn't affect rally.
    The same result we are getting with 1.3.2 release.
*) As well we should assume that we used this version of rally for > 3 months.

###
The issue is not related to running in parallel multiple Rally tasks

###
Simplest way to reproduce bug:

rally task start <task_file.yaml> --task-args '{"c": 10, "t": 300}'
---
  Authenticate.validate_glance:
    -
      args:
        repetitions: 1
      runner:
        type: "constant"
        times: {{t}}
        concurrency: {{c}}
      context:
        users:
          tenants: 1
          users_per_tenant: 1

###
Bug trigger is next :

* Create user with name XXX, Create tenant with name YYY
* Do authentication of this user
* Delete recently create user & tenant
* Create user and tenant with the same name as in first step
* Try to authenticate multiple times in parallel
* Some of iterations will fail with error message "Unauthorized: User ...."

###
From Rally Logs I saw the next thing:

* During the task validation (we are creating and deleting user & tenant):
2015-09-09 06:10:26.751 14888 DEBUG keystoneclient.session [-] REQ: curl -g -i -X DELETE http://192.168.0.2:35357/v2.0/tenants/5476f7af15dc493eacb08df12fa91552 -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}596a02bbf15c358f68de5552222f4ad09c24bdce" _http_log_request /opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py:195

* During the load we are using pre-created user & tenant. And one of iteration failed like:

2015-09-09 06:10:27.774 15091 ERROR rally.benchmark.runners.base Unauthorized: User 84f690c069ce4a4f89e4eb7929c0cd4a is unauthorized for tenant 5476f7af15dc493eacb08df12fa91552 (Disable debug mode to suppress these details.) (HTTP 401) (Request-ID: req-8bfb4abb-42db-43ef-b00a-3891bcd1bc56)

In other words we are referring during load and validation to the same tenant_id which is impossible If we take into account 3 facts:

- Tenant & User during validation & load are different
- Tenant & User during validation & load have the same name
- Keystone authentication is done via user name and tenant name

It becomes clear that this bug is not related to our testing tool & scripts.
We have some issues in keystone or it's configuration.

The things that we should check are next:
* Are they issues between synchronization of keystone backends between controller nodes?
* Is getting tenant_id by name somehow somewhere cached?
* Is authentication request cached?
* .... maybe something else...

Best regards,
Boris Pavlovic

[Dmitry Kalashnik (dkalashnik)]

Hi, have the same issue with #287

[Dmitry Kalashnik (dkalashnik)]

Keystone logs attached

[Timur Nurlygayanov (tnurlygayanov)]

Priority of the issue changed to Medium because looks like this bug will be reproduced only when we will create / delete users with the same name in parallel and also here is the not large user impact, because the workaround for the issue is to login again and authentication should pass.

Please, feel free to change the priority if my assumptions are wrong.

[Alexander Makarov (amakarov)]

Well, given the fact that "Keystone authentication is done via user name and tenant name"
we have yet another race condition issue:

thread1: get tenant ref by tenant name and extract tenant id
therad2: delete tenant
thread1: try to get the tenant roles for this user, and the tenant ref by user id and tenant id. (FAIL)

[Alexander Makarov (amakarov)]

The proof that tenant name is used rather than id:

https://github.com/openstack/rally/blob/master/rally/osclients.py#L106
https://github.com/openstack/rally/blob/master/rally/common/objects/endpoint.py#L45

[Alexander Makarov (amakarov)]

Created upstream bug: https://bugs.launchpad.net/keystone/+bug/1494398
I doubt the case is valid: if the project/tenant is deleted during authN, the authN should not succeed.

[Boris Bobrov (bbobrov)]

I'm setting it to "fix released" because it was fixed in fuel-library. It is "fix-released" and not "fix-commited" because otherwise anything wouldn't work.