Contrail Container :: ubuntu 14.04 build 27 mitaka :: provisioning failure :: contrail-api unable to authenticate with keystone

[Ritam Gangopadhyay]

Setup:-

10.204.217.143 nodei31 single node setup

contrail-api.log

07/28/2017 06:21:23 PM [contrail-api]: ModuleCpuStateTrace: data = << name = nodei31 build_info = {"build-info" : [{"build-version" : "4.0.1.0", "build-time" : "2017-07-28 09:26:53.167808", "build-user" : "contrail-builder", "build-hostname" : "ubuntu", "build-id" : "4.0.1.0-27", "build-number" : "27"}]} config_node_ip = [ 10.204.217.143, 172.17.0.1, 192.168.122.1, ] >>
07/28/2017 06:21:23 PM [contrail-api]: SandeshModuleClientTrace: data = << name = nodei31:Config:contrail-api:0 client_info = << status = Established successful_connections = 1 pid = 22697 http_port = 8084 start_time = 1501266079858688 collector_name = nodei31 collector_ip = 10.204.217.143:8086 collector_list = [ 10.204.217.143:8086, ] >> sm_queue_count = 11 max_sm_queue_count = 11 >>
07/28/2017 06:21:23 PM [contrail-api]: NodeStatusUVE: data = << name = nodei31 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.204.217.143:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.204.217.143:8086, ] status = Up description = ClientInit to Established on EvSandeshCtrlMessageRecv >>, << type = Database name = Cassandra server_addrs = [ 10.204.217.143:9161, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.204.217.143:35357/v2.0, ] status = Initializing description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.204.217.143:5672, ] status = Up description = >>, ] description = Generic Connection:Keystone[] connection down >>, ] >>

07/28/2017 05:19:13 PM [contrail-api]: Connect error to contrail api 4700 tries: Authentication Failure
07/28/2017 05:21:01 PM [contrail-api]: Connect error to contrail api 4800 tries: Authentication Failure
07/28/2017 05:22:49 PM [contrail-api]: Connect error to contrail api 4900 tries: Authentication Failure
07/28/2017 05:24:36 PM [contrail-api]: Connect error to contrail api 5000 tries: Authentication Failure
07/28/2017 05:26:24 PM [contrail-api]: Connect error to contrail api 5100 tries: Authentication Failure
07/28/2017 05:28:11 PM [contrail-api]: Connect error to contrail api 5200 tries: Authentication Failure
07/28/2017 05:29:59 PM [contrail-api]: Connect error to contrail api 5300 tries: Authentication Failure
07/28/2017 05:31:47 PM [contrail-api]: Connect error to contrail api 5400 tries: Authentication Failure

[Ritam Gangopadhyay (ritam)]

This is seen in multi interface multi node setup also.
On openstack HA setup contrail-api is able to connect to keystone

[Ritam Gangopadhyay (ritam)]

On HA setup as well there is problem with keystone. Seeing below trace in keystone log

2017-07-28 23:41:20.254 38063 CRITICAL keystone [-] error: [Errno 98] Address already in use
2017-07-28 23:41:20.254 38063 ERROR keystone Traceback (most recent call last):
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/bin/keystone-all", line 10, in <module>
2017-07-28 23:41:20.254 38063 ERROR keystone sys.exit(main())
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/all.py", line 39, in main
2017-07-28 23:41:20.254 38063 ERROR keystone eventlet_server.run(possible_topdir)
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 156, in run
2017-07-28 23:41:20.254 38063 ERROR keystone serve(*servers)
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 88, in serve
2017-07-28 23:41:20.254 38063 ERROR keystone server.launch_with(launcher)
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 54, in launch_with
2017-07-28 23:41:20.254 38063 ERROR keystone self.server.listen()
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/common/environment/eventlet_server.py", line 110, in listen
2017-07-28 23:41:20.254 38063 ERROR keystone backlog=backlog)
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 44, in listen
2017-07-28 23:41:20.254 38063 ERROR keystone sock.listen(backlog)
2017-07-28 23:41:20.254 38063 ERROR keystone File "/usr/lib/python2.7/socket.py", line 224, in meth
2017-07-28 23:41:20.254 38063 ERROR keystone return getattr(self._sock,name)(*args)
2017-07-28 23:41:20.254 38063 ERROR keystone error: [Errno 98] Address already in use
2017-07-28 23:41:20.254 38063 ERROR keystone

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34257
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34257
Committed: http://github.com/Juniper/contrail-puppet/commit/91ab2a5fa1f6be60ca66af10e0233ab4132f4b8f
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 91ab2a5fa1f6be60ca66af10e0233ab4132f4b8f
Author: Dheeraj Gautam <email address hidden>
Date: Wed Aug 2 14:05:33 2017 -0700

remove ceilometer code that is causing multiple other issues.

Change-Id: Iff32a63e730864ca164a8716f8f1a2f6c9329300
Closes-Bug: #1707279

[Dheeraj Gautam (dgautam)]

This issue is not there now in master. Current check-in made was to remove offending code. Proper fix has been made under https://bugs.launchpad.net/juniperopenstack/+bug/1695741. So no changes are needed for master now.