[Mainline/Mitaka] Keystone authentication failing

[Suresh]

In latest CB mainline build 16, few services not coming up in controller and analytics.
   In contrail-api.log, could see some keystone authentication failure messages. Keystone password is correct in contrail-keystone-auth.conf file in controller.
  Json file is /root/old_regr.json in nodec54.

setup detail:
==========
nodec54 : controller & analyticsdb
nodec55, nodec56 : analytics & analyticsdb
nodec53 : openstack
nodel1, nodei10 : compute

Error message:
==============
08/09/2017 01:08:57 PM [contrail-api]: Session Event: TCP Connect Fail
08/09/2017 01:08:57 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodec54 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 22.22.22.54:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 22.22.22.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 22.22.22.54:9161, ] status = Initializing description = >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >>
08/09/2017 01:08:57 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodec54 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 22.22.22.54:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 22.22.22.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 22.22.22.54:9161, ] status = Initializing description = >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >>
08/09/2017 01:08:57 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodec54:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1880 http_port = 8084 start_time = 1502264333992313 collector_name = collector_ip = 22.22.22.56:8086 collector_list = [ 22.22.22.56:8086, 22.22.22.55:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >>
08/09/2017 01:09:01 PM [contrail-api]: Session Event: TCP Connect Fail
:$
08/09/2017 02:25:55 PM [contrail-api]: Connect error to contrail api 4300 tries: Authentication Failure
08/09/2017 02:27:41 PM [contrail-api]: Connect error to contrail api 4400 tries: Authentication Failure
08/09/2017 02:29:28 PM [contrail-api]: Connect error to contrail api 4500 tries: Authentication Failure
08/09/2017 02:31:15 PM [contrail-api]: Connect error to contrail api 4600 tries: Authentication Failure

    Tried restarting the keystone service and see the below error messages

2017-08-09 16:53:36.927 13055 DEBUG keystone.middleware.auth [req-a09cdbbd-d850-4685-b1d2-0908ecdc4030 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71
2017-08-09 16:53:36.928 13055 INFO keystone.common.wsgi [req-a09cdbbd-d850-4685-b1d2-0908ecdc4030 - - - - -] POST http://22.22.22.53:35357/v2.0/tokens
2017-08-09 16:53:36.944 13057 WARNING keystone.common.wsgi [req-f0e5bb2b-4581-443b-ba26-b4a586315fd1 - - - - -] Authorization failed. The request you have made requires authentication. from 22.22.22.55

root@nodec54(controller):~# contrail-status
== Contrail Control ==
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 initializing (Generic Connection:Keystone[] connection down)
contrail-config-nodemgr active
contrail-device-manager active
contrail-schema active
contrail-svc-monitor initializing (Database:Cassandra[] connection down)

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

root@nodec54(controller):~#

root@nodec55:~# docker exec -it analytics contrail-status
== Contrail Analytics ==
contrail-alarm-gen initializing (ApiServer:Config[Config sync failed] connection down)
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

root@nodec55:~#

[Nitish Krishna Kaveri (nitishk)]

Waiting to see if issue is due to controller & analyticsdb being on same node

[Dheeraj Gautam (dgautam)]

keystone endpoints were not configured correctly due a ceilometer issue. issue has been fixed by https://review.opencontrail.org/34360 check-in and is available in build #17 (change #51).

So marking this is as fix now.