Contrail-kube-manager

01/31/2018 04:08:43 PM [contrail-kube-manager]: Unable to delete the old ampq queue: connection already closed
01/31/2018 04:08:44 PM [contrail-kube-manager]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0]
01/31/2018 04:08:44 PM [contrail-kube-manager]: SANDESH: Trace: PRINT: [None] -> [False]
01/31/2018 04:08:44 PM [contrail-kube-manager]: SANDESH: Flow Logging: [None] -> [False]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@192.168.1.5:5672//', 'pyamqp://guest:guest@192.168.1.8:5672//', 'pyamqp://guest:guest@192.168.1.9:5672//']
01/31/2018 04:08:45 PM [contrail-kube-manager]: Add uve <testbed-1-vm5, 1> in the [ObjectKubernetesManagerNode:NodeStatusUVE] map
01/31/2018 04:08:45 PM [contrail-kube-manager]: RabbitMQ connection down
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvStart] in state[none]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Add uve <testbed-1-vm5:KubernetesMgr:contrail-kube-manager:0, 1> in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map
01/31/2018 04:08:45 PM [contrail-kube-manager]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Initializing  description =   >>,  ]  description = Database:RabbitMQ[] connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Down  description =   >>,  ]  description = Database:RabbitMQ[] connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name =   server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Down  description =   >>,  ]  description = Collector, Database:RabbitMQ[] connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = testbed-1-vm5:KubernetesMgr:contrail-kube-manager:0  client_info = <<  status = Idle  successful_connections = 0  pid = 1959  start_time = 1517395125260735  collector_name =   collector_ip =   collector_list = [ 192.168.1.8:8086, 192.168.1.9:8086, 192.168.1.5:8086,  ]  >>  sm_queue_count = 3  max_sm_queue_count = 3  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: RabbitMQ connection ESTABLISHED <Connection: amqp://guest:**@192.168.1.5:5672// at 0x7f0abb0a5050>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name =   server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Up  description =   >>,  ]  description = Collector connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Removed remained AMQP queue
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name =   server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 192.168.1.5:2181, 192.168.1.8:2181, 192.168.1.9:2181,  ]  status = Initializing  description =   >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Up  description =   >>,  ]  description = Collector, Zookeeper:Zookeeper[] connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: Waiting to be elected as master...
01/31/2018 04:08:45 PM [contrail-kube-manager]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master...
01/31/2018 04:08:45 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:45 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name =   server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 192.168.1.5:2181, 192.168.1.8:2181, 192.168.1.9:2181,  ]  status = Up  description =   >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Up  description =   >>,  ]  description = Collector connection down  >>,  ]  >>
01/31/2018 04:08:45 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Idle]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvIdleHoldTimerExpired] in state[Idle]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Session Event: TCP Connected
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[Connect]
01/31/2018 04:08:49 PM [contrail-kube-manager]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = testbed-1-vm5  process_status = [ <<  module_id = contrail-kube-manager  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Collector  name =   server_addrs = [ 192.168.1.8:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 192.168.1.5:2181, 192.168.1.8:2181, 192.168.1.9:2181,  ]  status = Up  description =   >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 192.168.1.5:5672, 192.168.1.8:5672, 192.168.1.9:5672,  ]  status = Up  description =   >>,  ]  description = Collector connection down  >>,  ]  >>
01/31/2018 04:08:49 PM [contrail-kube-manager]: Discarding event[EvSandeshUVESend] in state[Connect]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvTcpConnected] in state[Connect]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Send sandesh control message. uve type count # 11
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[ClientInit]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvSandeshUVESend] in state[ClientInit]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Send Level [INVALID] -> [SYS_DEBUG]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Send Level [SYS_DEBUG] -> [INVALID]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Extracted sandesh length: 0000000985
01/31/2018 04:08:49 PM [contrail-kube-manager]: Received sandesh control message [SandeshCtrlServerToClient]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Number of uve types in sandesh control message is 0
01/31/2018 04:08:49 PM [contrail-kube-manager]: send sync_uve <testbed-1-vm5: 8> in the [ObjectKubernetesManagerNode:NodeStatusUVE] map
01/31/2018 04:08:49 PM [contrail-kube-manager]: send sync_uve <testbed-1-vm5:KubernetesMgr:contrail-kube-manager:0: 2> in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Send Level [INVALID] -> [SYS_DEBUG]
01/31/2018 04:08:49 PM [contrail-kube-manager]: Sandesh Send Level [SYS_DEBUG] -> [INVALID]

kube-manager-zk.log LOGS:

Owner=99443349941452816, dataLength=4, numChildren=0, pzxid=4294975381)
01/31/2018 04:08:45 PM [kube-manager]: Connecting to 192.168.1.8:2181
01/31/2018 04:08:45 PM [kube-manager]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=400000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None)
01/31/2018 04:08:45 PM [kube-manager]: Zookeeper connection established, state: CONNECTED
01/31/2018 04:08:45 PM [kube-manager]: Sending request(xid=1): Exists(path='/kube-manager', watcher=None)
01/31/2018 04:08:45 PM [kube-manager]: Received response(xid=1): ZnodeStat(czxid=4294970244, mzxid=4294970244, ctime=1517385422347, mtime=1517385422347, version=0, cversion=28, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=4, pzxid=4294975448)
01/31/2018 04:08:45 PM [kube-manager]: Sending request(xid=2): Create(path='/kube-manager/0111a791088347fea7520978c4b1c6d9__lock__', data='1959', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=3)
01/31/2018 04:08:45 PM [kube-manager]: Received response(xid=2): u'/kube-manager/0111a791088347fea7520978c4b1c6d9__lock__0000000016'
01/31/2018 04:08:45 PM [kube-manager]: Sending request(xid=3): GetChildren(path='/kube-manager', watcher=None)
01/31/2018 04:08:45 PM [kube-manager]: Received response(xid=3): [u'bb53a0dd41a444ed99833f73dd56fa3a__lock__0000000013', u'b09083b61d5d45f5815993a85eff99fd__lock__0000000015', u'0111a791088347fea7520978c4b1c6d9__lock__0000000016', u'788f78211c6b4abfb456106739bba35a__lock__0000000012', u'21aeae6d426d415685d21eea7e78b76a__lock__0000000014']
01/31/2018 04:08:45 PM [kube-manager]: Sending request(xid=4): Exists(path=u'/kube-manager/b09083b61d5d45f5815993a85eff99fd__lock__0000000015', watcher=<bound method Lock._watch_predecessor of <kazoo.recipe.lock.Lock object at 0x7f0ab9f57490>>)
01/31/2018 04:08:45 PM [kube-manager]: Received response(xid=4): ZnodeStat(czxid=4294975448, mzxid=4294975448, ctime=1517395125113, mtime=1517395125113, version=0, cversion=0, aversion=0, ephemeralOwner=243558542540865553, dataLength=4, numChildren=0, pzxid=4294975448)