Requests auth issue when there are multiple threads or processes

[Kudaka Poorna Rajesh]

Observed in train:

Steps to reproduce:

1. Setup devstack(stable/train)

2. Make the below config changes and restart keystone.

cat /etc/keystone/policy.json

```
{
  "admin_required": "role:admin or is_admin:1",
  "identity:get_domain_config_default": "rule:admin_required"
}
```

cat /etc/keystone/keystone.conf

```
[identity]
domain_specific_drivers_enabled = true
password_hash_rounds = 4
driver = sql

[assignment]
driver = sql

[role]
driver = sql

[resource]
driver = sql

[cache]
memcache_servers = localhost:11211
backend = dogpile.cache.memcached
enabled = True

[oslo_messaging_notifications]
transport_url = rabbit://stackrabbit:password@localhost:5672/

[DEFAULT]
max_token_size = 16384
debug = True
logging_exception_prefix = ERROR %(name)s %(instance)s
logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}}
admin_endpoint = http://localhost/identity
public_endpoint = http://localhost/identity

[token]
provider = fernet
cache_time = 3600
caching = true
expiration = 43200

[revoke]
cache_time = 3600
caching = true
[role]
cache_time = 3600
caching = true

[database]
connection = mysql+pymysql://root:password@127.0.0.1/keystone?charset=utf8

[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/

[credential]
key_repository = /etc/keystone/credential-keys/

[security_compliance]
unique_last_password_count = 2
lockout_duration = 10
lockout_failure_attempts = 2
```

cat /etc/keystone/keystone-uwsgi-public.ini

```
[uwsgi]
chmod-socket = 666
socket = /var/run/uwsgi/keystone-wsgi-public.socket
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_kill_them_all
thunder-lock = true
plugins = http,python
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
threads = 2
wsgi-file = /usr/local/bin/keystone-wsgi-public
```
3. Create test creds.

openstack domain create --description "Test Domain" test-domain
openstack project create --domain Test-Domain --description "Test Project" test-token
openstack user create --domain Test-Domain --password 123456 test-user

4. Test script, provide the admin, test creds and required ids in the test script.

# test-script.py
http://paste.openstack.org/show/801281/

5. Run the test script, python3 test-script.py <no of test repeations>

Run: python3 test-script.py 10

Expected final outcome: For the above script test-user with 'admin' role should be allowed to get the default domain config and when the role is switched to 'member' role get request response should be '403'. SCRIPT OUTPUT: ACTION EXECUTIONS: 20 PASS: 20 FAIL: 0

Actual final outcome: Random failures, http://paste.openstack.org/show/801283/

NOTE: Please increase the processes and/or threads count in /etc/keystone/keystone-uwsgi-public.ini if issue not reproduced with the above mentioned values.

Analysis: The above provided script yields successful result when there are one or few processes and/or threads configured in /etc/keystone/keystone-uwsgi-public.ini. But, If the processes and/or threads config is higher, and a user role is switched, requests are failing for authorised users or requests are passing to users who are not authorised(even after providing sleep after role switch). After enabling debug_cache_backend = true in keystone.conf, I have observed that keystone is using old request id and processing a part of the request where the token is getting set in cache twice by different processes of keystone but with different roles. I have observed the same in all-in-one openstack-helm setup.

Will try to fetch and provide keystone logs when possible.