Token not found

[Viktoria Efimova]

Two keystone rally tests failed with Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (HTTP 401) and Could not find token: fbe9320a1057442b934d82e3c3357369 (Disable debug mode to suppress these details.) (HTTP 401)

grepping logs using 5731bfea58c64784b9e7e83c29f83c6d token id gives only:
/var/log/keystone-all.log:<12>Jun 24 22:41:01 node-9 keystone-all Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2
/var/log/keystone/keystone-all.log.1:2015-06-24 22:41:01.538 32676 WARNING keystone.common.wsgi [-] Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2

Topology:
CentOS+GRE+525(RC3)
Controllers: 3, Computes: 21

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

Log snapshot at http://mos-scale-share.mirantis.com/fuel-snapshot-2015-06-25_16-05-41.tar.xz

[Boris Bobrov (bbobrov)]

In the logs I see

```
2015-06-24T22:41:01.537318+00:00 debug: Memcached pool 90129936, thread 140052079146752: Marked host 192.168.0.10:11211 dead until 1435185691.53
2015-06-24T22:41:01.542767+00:00 warning: Authorization failed. Could not find token: 5731bfea58c64784b9e7e83c29f83c6d (Disable debug mode to suppress these details.) (Disable debug mode to s
uppress these details.) from 192.168.0.2

```
same for the second token:
```
2015-06-24T22:31:07.226938+00:00 debug: Memcached pool 61707152, thread 140200754067200: Marked host 192.168.0.10:11211 dead until 1435185097.09
2015-06-24T22:31:07.251874+00:00 warning: Authorization failed. Could not find token: fbe9320a1057442b934d82e3c3357369 (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 192.168.0.2
```
which means that memcache host became dead there. Keystone in 6.1 requires some seconds to failover.

Were the tests destructive?

[Dina Belova (dbelova)]

Boris, these tests are not destructive, we process tests like that in manual way, this is simple rally test... May it be the case when memache host was moving to other node because of any issue?

[Boris Bobrov (bbobrov)]

Dina, Viktoria, was it reproduced?

[Alexander Makarov (amakarov)]

Yes, we have an env with the bug

[Alexander Makarov (amakarov)]

The problem is in memcache sharding: it's unstable under high load (somtimes fails to connect).
The solution lies in configuration rather than coding, and problem is about to go away along with memcache token storage.

[Vitaly Sedelnik (vsedelnik)]

Nominated for 8.0 and Won't Fix for 7.0-updates and 6.1-updates because of Medium importance

[Alexander Makarov (amakarov)]

Looks like sometimes keystone fails to connect to memcache holding the token:
https://bugs.launchpad.net/fuel/+bug/1515350

[Boris Bobrov (bbobrov)]

This should be fixed with Fernet tokens.

[Timur Nurlygayanov (tnurlygayanov)]

It looks like the issue was fixed, not reproduced in my environment.
Please reopen it and add more information if the issue will be reproduced again.

Verified on MOS 8.0 RC1.

Thank you!

[Ivan Lozgachev (ilozgachev)]

Cannot verify because of https://bugs.launchpad.net/mos/+bug/1538251. Moving to Fix Released.

[Ivan Lozgachev (ilozgachev)]

Sorry, didn't saw that it's already in Fix Released state...