Keystone authtoken middleware seems to work wrong with memcached cache

[Dina Belova]

== Abstract ==

During Keystone OSprofiler integration it was a wish to check how does Keystone was changed from Liberty to Mitaka in regarding DB/Caching layers workability. There were lots of changes related to federation support added and because of movement to slo.cache usage.

Ideas of the experiment can be found here: http://docs.openstack.org/developer/performance-docs/test_plans/keystone/plan.html

== What was discovered ==

Preliminary results can be found here - http://docs.openstack.org/developer/performance-docs/test_results/keystone/all-in-one/index.html

In short: two identical Keystone API calls were done to both Liberty and Mitaka env. To make keystone authtoken middleware used let's choose nova boot request. The second call was profiled using OSprofiler - and compared between Liberty and Mitaka env.

Both env had the same Apache config, the same Keystone authtoken cache config in the services. For instance, for Nova:

[keystone_authtoken]
memcached_servers = 10.0.2.15:11211
signing_dir = /var/cache/nova
cafile = /opt/stack/data/ca-bundle.pem
auth_uri = http://10.0.2.15:5000
project_domain_id = default
project_name = service
user_domain_id = default
password = password
username = nova
auth_url = http://10.0.2.15:35357
auth_type = password

On Liberty only the first keystone authmiddleware call from nova is presented in requests tree - that is expected behaviour, as after this in case of memcached backend usage cached authentication values should be used by other services. So we see no more keystone calls in the requests tree. In Mitaka all API calls to nova. glance, neutron are paired with API call to the keystone.

Liberty call example: http://dinabelova.github.io/liberty_server_create.html
Mitaka call example: http://dinabelova.github.io/mitaka_server_create.html

== Note ==

This might (?) be related to the https://bugs.launchpad.net/keystone/+bug/1566835 issue - I'm not sure, this needs to be investigated.

[Dina Belova (dbelova)]

Was affected by the same environmental issue as https://bugs.launchpad.net/keystone/+bug/1566835