Mainline&R3.0: HA-2718 & build3(3.1)keystone tenant-list failed-incorrect token driver used.

[shajuvk]

In HA setup with 2718 build -ubuntu kilo image, setup_all failed while listing keystone tenant list.

keystone.conf has memcache instead of sql token
driver = keystone.token.persistence.backends.memcache.Token

log:

http://anamika.englab.juniper.net:8080/job/ubuntu-14-04_Kilo_vcenter_compute_HA_Sanity/10/console

Sanity setup with build 2718- kilo failed due to error while listing keystone tenant-list. Please check the node a6s32- 10.84.13.32

2016-03-07 15:02:01.904 2367 TRACE keystone
2016-03-07 16:02:02.507 20212 DEBUG keystone.common.kvs.core [-] KVS region configuration for token-driver: {'keystone.kvs.backend': 'openstack.kvs.Memcached', 'keystone.kvs.arguments.distributed_lock': T
rue, 'keystone.kvs.arguments.no_expiry_keys': ['revocation-list'], 'keystone.kvs.arguments.url': ['localhost:11211'], 'keystone.kvs.arguments.memcached_expire_time': 3600, 'keystone.kvs.arguments.memcache
d_backend': 'memcached', 'keystone.kvs.arguments.lock_timeout': 6} _configure_region /usr/lib/python2.7/dist-packages/keystone/common/kvs/core.py:240
2016-03-07 16:02:02.510 20212 INFO keystone.common.kvs.core [-] Using default dogpile sha1_mangle_key as KVS region token-driver key_mangler
2016-03-07 16:02:02.510 20212 CRITICAL keystone [-] NotImplemented: The action you have requested has not been implemented.
2016-03-07 16:02:02.510 20212 TRACE keystone Traceback (most recent call last):
2016-03-07 16:02:02.510 20212 TRACE keystone File "/usr/bin/keystone-manage", line 44, in <module>
2016-03-07 16:02:02.510 20212 TRACE keystone cli.main(argv=sys.argv, config_files=config_files)
2016-03-07 16:02:02.510 20212 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 585, in main
2016-03-07 16:02:02.510 20212 TRACE keystone CONF.command.cmd_class.main()
2016-03-07 16:02:02.510 20212 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 243, in main
2016-03-07 16:02:02.510 20212 TRACE keystone token_manager.flush_expired_tokens()
2016-03-07 16:02:02.510 20212 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/token/persistence/backends/kvs.py", line 357, in flush_expired_tokens
2016-03-07 16:02:02.510 20212 TRACE keystone raise exception.NotImplemented()
2016-03-07 16:02:02.510 20212 TRACE keystone NotImplemented: The action you have requested has not been implemented.
2016-03-07 16:02:02.510 20212 TRACE keystone

[shajuvk (shajuvk)]

root@a6s32:~# keystone tenant-list
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-14066684-d7c1-42fb-924f-2e0d2b78f49d)
root@a6s32:~#

Logs are copied to : /cs-shared/bugs/1554810/

[shajuvk (shajuvk)]

issue also occurred on R3.1 build3.

root@a6s32:~/fabric-utils# keystone token-get
2016-07-20 16:42:09.141 10018 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:223
2016-07-20 16:42:09.142 10018 INFO keystone.common.wsgi [-] POST http://10.84.13.32:5000/v2.0/tokens
2016-07-20 16:42:15.494 10018 WARNING keystone.common.wsgi [-] Maximum lock attempts on _lockusertokens-05f000d8f7404e7e9b6883896c8d6772 occurred. (Disable debug mode to suppress these details.)
2016-07-20 16:42:15.497 10018 INFO eventlet.wsgi.server [-] 10.84.13.32 - - [20/Jul/2016 16:42:15] "POST /v2.0/tokens HTTP/1.1" 500 442 6.356517
Authorization Failed: Maximum lock attempts on _lockusertokens-05f000d8f7404e7e9b6883896c8d6772 occurred. (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-87740491-ef68-4cfc-a7a1-bf3a123bb666)
root@a6s32:~/fabric-utils#

[shajuvk (shajuvk)]

please check the node a6s32 -10.84.13.32. provisioning logs are under ~/fabric-utils

[shajuvk (shajuvk)]

R3.1 build 3 logs are update on same location: /cs-shared/bugs/1554810/R3-1-build3

[shajuvk (shajuvk)]

sanity setup aborted while starting casandra, while provisioning manylly didn't export env.HA_TEST=True