Activity log for bug #1774474

Date Who What changed Old value New value Message
2018-05-31 18:16:06 Venkatesh Velpula bug added bug
2018-05-31 18:19:22 Venkatesh Velpula attachment added instances.yaml https://bugs.launchpad.net/juniperopenstack/+bug/1774474/+attachment/5147109/+files/insntances_yaml.yml
2018-05-31 18:20:07 Venkatesh Velpula description with same latest ansible deployer code ,on build79 ..we don't see this issue. build :ocata-5.0-80 deployer :ansible-deployer hostos :centos7.5 setup: controller/master :nodec19,nodec20,nodec21 compute/minion . :nodei16,nodei18 docker log snippet =================== + mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini + exec /usr/bin/python /usr/bin/contrail-analytics-api -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: CONNECT TO COLLECTOR: True 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-analytics-api.log] Traceback (most recent call last): File "/usr/bin/contrail-analytics-api", line 9, in <module> load_entry_point('opserver==0.1dev', 'console_scripts', 'contrail-analytics-api')() File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 2638, in main opserver = OpServer(args_str) File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 702, in __init__ staticmethod(ConnectionState.get_process_state_cb), AttributeError: type object 'ConnectionState' has no attribute 'get_process_state_cb' ++++ echo 10.204.217.98 +++ local server_address=10.204.217.98 +++ extended_server_list+='10.204.217.98:9092 ' +++ '[' -n '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092 ' ']' +++ echo '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ KAFKA_SERVERS='10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ ANALYTICS_API_VIP= ++ CONFIG_API_VIP= ++ LINKLOCAL_SERVICE_PORT=80 ++ LINKLOCAL_SERVICE_NAME=metadata ++ LINKLOCAL_SERVICE_IP=169.254.169.254 ++ IPFABRIC_SERVICE_PORT=8775 ++ IPFABRIC_SERVICE_HOST= ++ XMPP_SSL_ENABLE=False ++ XMPP_SERVER_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ XMPP_SERVER_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ XMPP_SERVER_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ INTROSPECT_SSL_ENABLE=False ++ INTROSPECT_SSL_INSECURE=True ++ INTROSPECT_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ INTROSPECT_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ INTROSPECT_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ SANDESH_SSL_ENABLE=False ++ SANDESH_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ SANDESH_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ SANDESH_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ read -r -d '' sandesh_client_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ xmpp_certs_config= ++ METADATA_SSL_ENABLE=false ++ METADATA_SSL_CERTFILE= ++ METADATA_SSL_KEYFILE= ++ METADATA_SSL_CA_CERTFILE= ++ METADATA_SSL_CERT_TYPE= ++ RABBITMQ_VHOST=/ ++ RABBITMQ_USER=guest ++ RABBITMQ_PASSWORD=guest ++ RABBITMQ_USE_SSL=False ++ RABBITMQ_SSL_VER= ++ RABBITMQ_CLIENT_SSL_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ RABBITMQ_CLIENT_SSL_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ RABBITMQ_CLIENT_SSL_CACERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ read -r -d '' rabbitmq_config ++ true ++ read -r -d '' rabbit_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ AGENT_MODE=kernel ++ DPDK_UIO_DRIVER=uio_pci_generic ++ CPU_CORE_MASK=0x01 ++ HUGE_PAGES=1024 ++ HUGE_PAGES_DIR=/dev/hugepages ++ DPDK_MEM_PER_SOCKET=1024 ++ VHOST_CONFIG_DIR=/etc/sysconfig/network-scripts ++ TSN_EVPN_MODE=False ++ VROUTER_CRYPT_INTERFACE=crypt0 ++ VROUTER_DECRYPT_INTERFACE=decrypt0 ++ VROUTER_DECRYPT_KEY=15 ++ DIST_SNAT_PROTO_PORT_LIST= ++ FLOW_EXPORT_RATE=0 ++ FABRIC_SNAT_HASH_TABLE_SIZE=4096 ++ PRIORITY_ID= ++ PRIORITY_BANDWIDTH= ++ PRIORITY_SCHEDULING= ++ QOS_QUEUE_ID= ++ QOS_LOGICAL_QUEUES= ++ QOS_DEF_HW_QUEUE=False ++ PRIORITY_TAGGING=True + pre_start_init + wait_certs_if_ssl_enabled + is_ssl_enabled + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + return ++ get_listen_ip_for_node ANALYTICS +++ find_my_ip_and_order_for_node ANALYTICS +++ cut -d ' ' -f 1 +++ local server_typ=ANALYTICS_NODES +++ local server_list= +++ IFS=, +++ read -ra server_list ++++ get_local_ips ++++ tr '\n' , ++++ cat /proc/net/fib_trie ++++ awk '/32 host/ { print f } {f=$2}' +++ local local_ips=,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, +++ local ord=1 +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.52, ]] +++ (( ord+=1 )) +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.71, ]] +++ echo 10.204.217.71 2 +++ return ++ local ip=10.204.217.71 ++ [[ -z 10.204.217.71 ]] ++ echo 10.204.217.71 + host_ip=10.204.217.71 + cat + add_ini_params_from_env ANALYTICS_API /etc/contrail/contrail-analytics-api.conf + local service_name=ANALYTICS_API + local cfg_path=/etc/contrail/contrail-analytics-api.conf + local delim=__ ++ grep '^ANALYTICS_API__.*__.*=.*$' ++ sort ++ cut -d = -f 1 ++ set -o posix ++ set ++ sed 's/^ANALYTICS_API__//g' + local vars= + local section= + set_third_party_auth_config + [[ noauth != \k\e\y\s\t\o\n\e ]] + return + set_vnc_api_lib_ini + local tmp_file=/etc/contrail/vnc_api_lib.ini.tmp + cat + [[ noauth == \k\e\y\s\t\o\n\e ]] [root@nodeg31 ~]# contrail-status Pod Service Original Name State Status analytics alarm-gen contrail-analytics-alarm-gen restarting Restarting (1) 2 minutes ago analytics api contrail-analytics-api restarting Restarting (1) 2 minutes ago analytics collector contrail-analytics-collector running Up 2 hours analytics nodemgr contrail-nodemgr running Up 2 hours analytics query-engine contrail-analytics-query-engine running Up 2 hours analytics snmp-collector contrail-analytics-snmp-collector restarting Restarting (1) 44 minutes ago analytics topology contrail-analytics-topology restarting Restarting (1) 44 minutes ago config api contrail-controller-config-api running Up 2 hours config cassandra contrail-external-cassandra running Up 2 hours config device-manager contrail-controller-config-devicemgr running Up 2 hours config nodemgr contrail-nodemgr running Up About a minute config rabbitmq contrail-external-rabbitmq running Up 2 hours config schema contrail-controller-config-schema running Up 2 hours config svc-monitor contrail-controller-config-svcmonitor running Up 2 hours config zookeeper contrail-external-zookeeper running Up 2 hours control control contrail-controller-control-control running Up 2 hours control dns contrail-controller-control-dns running Up 2 hours control named contrail-controller-control-named running Up 2 hours control nodemgr contrail-nodemgr running Up 2 hours database cassandra contrail-external-cassandra running Up 2 hours database kafka contrail-external-kafka running Up 2 hours database nodemgr contrail-nodemgr running Up 2 hours database zookeeper contrail-external-zookeeper running Up 2 hours kubernetes kube-manager contrail-kubernetes-kube-manager running Up 2 hours webui job contrail-controller-webui-job running Up 2 hours webui web contrail-controller-webui-web running Up 2 hours == Contrail control == control: active nodemgr: active named: active dns: active == Contrail kubernetes == kube-manager: active == Contrail database == kafka: active nodemgr: initializing (Disk for DB is too low. ) zookeeper: active cassandra: active == Contrail analytics == snmp-collector: inactive query-engine: active api: inactive<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< alarm-gen: inactive nodemgr: active collector: active topology: inactive == Contrail webui == web: active job: active == Contrail config == api: active zookeeper: active svc-monitor: backup nodemgr: initializing device-manager: backup cassandra: active rabbitmq: active schema: backup [root@nodeg31 ~]# with same latest ansible deployer code ,on build79 ..we don't see this issue. build :ocata-5.0-80 deployer :ansible-deployer hostos :centos7.5 setup: controller/master :nodec19,nodec20,nodec21(HA with multi innterface) compute/minion . :nodei16,nodei18 docker log snippet ===================     + mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini + exec /usr/bin/python /usr/bin/contrail-analytics-api -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: CONNECT TO COLLECTOR: True 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-analytics-api.log] Traceback (most recent call last):   File "/usr/bin/contrail-analytics-api", line 9, in <module>     load_entry_point('opserver==0.1dev', 'console_scripts', 'contrail-analytics-api')()   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 2638, in main     opserver = OpServer(args_str)   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 702, in __init__     staticmethod(ConnectionState.get_process_state_cb), AttributeError: type object 'ConnectionState' has no attribute 'get_process_state_cb' ++++ echo 10.204.217.98 +++ local server_address=10.204.217.98 +++ extended_server_list+='10.204.217.98:9092 ' +++ '[' -n '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092 ' ']' +++ echo '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ KAFKA_SERVERS='10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ ANALYTICS_API_VIP= ++ CONFIG_API_VIP= ++ LINKLOCAL_SERVICE_PORT=80 ++ LINKLOCAL_SERVICE_NAME=metadata ++ LINKLOCAL_SERVICE_IP=169.254.169.254 ++ IPFABRIC_SERVICE_PORT=8775 ++ IPFABRIC_SERVICE_HOST= ++ XMPP_SSL_ENABLE=False ++ XMPP_SERVER_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ XMPP_SERVER_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ XMPP_SERVER_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ INTROSPECT_SSL_ENABLE=False ++ INTROSPECT_SSL_INSECURE=True ++ INTROSPECT_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ INTROSPECT_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ INTROSPECT_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ SANDESH_SSL_ENABLE=False ++ SANDESH_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ SANDESH_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ SANDESH_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ read -r -d '' sandesh_client_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ xmpp_certs_config= ++ METADATA_SSL_ENABLE=false ++ METADATA_SSL_CERTFILE= ++ METADATA_SSL_KEYFILE= ++ METADATA_SSL_CA_CERTFILE= ++ METADATA_SSL_CERT_TYPE= ++ RABBITMQ_VHOST=/ ++ RABBITMQ_USER=guest ++ RABBITMQ_PASSWORD=guest ++ RABBITMQ_USE_SSL=False ++ RABBITMQ_SSL_VER= ++ RABBITMQ_CLIENT_SSL_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ RABBITMQ_CLIENT_SSL_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ RABBITMQ_CLIENT_SSL_CACERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ read -r -d '' rabbitmq_config ++ true ++ read -r -d '' rabbit_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ AGENT_MODE=kernel ++ DPDK_UIO_DRIVER=uio_pci_generic ++ CPU_CORE_MASK=0x01 ++ HUGE_PAGES=1024 ++ HUGE_PAGES_DIR=/dev/hugepages ++ DPDK_MEM_PER_SOCKET=1024 ++ VHOST_CONFIG_DIR=/etc/sysconfig/network-scripts ++ TSN_EVPN_MODE=False ++ VROUTER_CRYPT_INTERFACE=crypt0 ++ VROUTER_DECRYPT_INTERFACE=decrypt0 ++ VROUTER_DECRYPT_KEY=15 ++ DIST_SNAT_PROTO_PORT_LIST= ++ FLOW_EXPORT_RATE=0 ++ FABRIC_SNAT_HASH_TABLE_SIZE=4096 ++ PRIORITY_ID= ++ PRIORITY_BANDWIDTH= ++ PRIORITY_SCHEDULING= ++ QOS_QUEUE_ID= ++ QOS_LOGICAL_QUEUES= ++ QOS_DEF_HW_QUEUE=False ++ PRIORITY_TAGGING=True + pre_start_init + wait_certs_if_ssl_enabled + is_ssl_enabled + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + return ++ get_listen_ip_for_node ANALYTICS +++ find_my_ip_and_order_for_node ANALYTICS +++ cut -d ' ' -f 1 +++ local server_typ=ANALYTICS_NODES +++ local server_list= +++ IFS=, +++ read -ra server_list ++++ get_local_ips ++++ tr '\n' , ++++ cat /proc/net/fib_trie ++++ awk '/32 host/ { print f } {f=$2}' +++ local local_ips=,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, +++ local ord=1 +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.52, ]] +++ (( ord+=1 )) +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.71, ]] +++ echo 10.204.217.71 2 +++ return ++ local ip=10.204.217.71 ++ [[ -z 10.204.217.71 ]] ++ echo 10.204.217.71 + host_ip=10.204.217.71 + cat + add_ini_params_from_env ANALYTICS_API /etc/contrail/contrail-analytics-api.conf + local service_name=ANALYTICS_API + local cfg_path=/etc/contrail/contrail-analytics-api.conf + local delim=__ ++ grep '^ANALYTICS_API__.*__.*=.*$' ++ sort ++ cut -d = -f 1 ++ set -o posix ++ set ++ sed 's/^ANALYTICS_API__//g' + local vars= + local section= + set_third_party_auth_config + [[ noauth != \k\e\y\s\t\o\n\e ]] + return + set_vnc_api_lib_ini + local tmp_file=/etc/contrail/vnc_api_lib.ini.tmp + cat + [[ noauth == \k\e\y\s\t\o\n\e ]] [root@nodeg31 ~]# contrail-status Pod Service Original Name State Status analytics alarm-gen contrail-analytics-alarm-gen restarting Restarting (1) 2 minutes ago analytics api contrail-analytics-api restarting Restarting (1) 2 minutes ago analytics collector contrail-analytics-collector running Up 2 hours analytics nodemgr contrail-nodemgr running Up 2 hours analytics query-engine contrail-analytics-query-engine running Up 2 hours analytics snmp-collector contrail-analytics-snmp-collector restarting Restarting (1) 44 minutes ago analytics topology contrail-analytics-topology restarting Restarting (1) 44 minutes ago config api contrail-controller-config-api running Up 2 hours config cassandra contrail-external-cassandra running Up 2 hours config device-manager contrail-controller-config-devicemgr running Up 2 hours config nodemgr contrail-nodemgr running Up About a minute config rabbitmq contrail-external-rabbitmq running Up 2 hours config schema contrail-controller-config-schema running Up 2 hours config svc-monitor contrail-controller-config-svcmonitor running Up 2 hours config zookeeper contrail-external-zookeeper running Up 2 hours control control contrail-controller-control-control running Up 2 hours control dns contrail-controller-control-dns running Up 2 hours control named contrail-controller-control-named running Up 2 hours control nodemgr contrail-nodemgr running Up 2 hours database cassandra contrail-external-cassandra running Up 2 hours database kafka contrail-external-kafka running Up 2 hours database nodemgr contrail-nodemgr running Up 2 hours database zookeeper contrail-external-zookeeper running Up 2 hours kubernetes kube-manager contrail-kubernetes-kube-manager running Up 2 hours webui job contrail-controller-webui-job running Up 2 hours webui web contrail-controller-webui-web running Up 2 hours == Contrail control == control: active nodemgr: active named: active dns: active == Contrail kubernetes == kube-manager: active == Contrail database == kafka: active nodemgr: initializing (Disk for DB is too low. ) zookeeper: active cassandra: active == Contrail analytics == snmp-collector: inactive query-engine: active api: inactive<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< alarm-gen: inactive nodemgr: active collector: active topology: inactive == Contrail webui == web: active job: active == Contrail config == api: active zookeeper: active svc-monitor: backup nodemgr: initializing device-manager: backup cassandra: active rabbitmq: active schema: backup [root@nodeg31 ~]#
2018-05-31 18:24:10 Jeba Paulaiyan nominated for series juniperopenstack/r5.0
2018-05-31 18:24:10 Jeba Paulaiyan bug task added juniperopenstack/r5.0
2018-05-31 18:24:10 Jeba Paulaiyan nominated for series juniperopenstack/trunk
2018-05-31 18:24:10 Jeba Paulaiyan bug task added juniperopenstack/trunk
2018-05-31 18:24:16 Jeba Paulaiyan juniperopenstack/r5.0: importance Undecided Critical
2018-05-31 18:24:25 Jeba Paulaiyan juniperopenstack/r5.0: assignee Sundaresan Rajangam (srajanga)
2018-05-31 18:24:33 Jeba Paulaiyan juniperopenstack/trunk: assignee Sundaresan Rajangam (srajanga)
2018-05-31 18:24:36 Jeba Paulaiyan juniperopenstack/trunk: milestone r5.1.0
2018-05-31 18:24:38 Jeba Paulaiyan juniperopenstack/r5.0: milestone r5.0.1
2018-05-31 18:24:49 Jeba Paulaiyan tags analytics ansible sanityblocker analytics ansible beta-blocker fabric sanityblocker
2018-05-31 18:47:39 Venkatesh Velpula description with same latest ansible deployer code ,on build79 ..we don't see this issue. build :ocata-5.0-80 deployer :ansible-deployer hostos :centos7.5 setup: controller/master :nodec19,nodec20,nodec21(HA with multi innterface) compute/minion . :nodei16,nodei18 docker log snippet ===================     + mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini + exec /usr/bin/python /usr/bin/contrail-analytics-api -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: CONNECT TO COLLECTOR: True 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-analytics-api.log] Traceback (most recent call last):   File "/usr/bin/contrail-analytics-api", line 9, in <module>     load_entry_point('opserver==0.1dev', 'console_scripts', 'contrail-analytics-api')()   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 2638, in main     opserver = OpServer(args_str)   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 702, in __init__     staticmethod(ConnectionState.get_process_state_cb), AttributeError: type object 'ConnectionState' has no attribute 'get_process_state_cb' ++++ echo 10.204.217.98 +++ local server_address=10.204.217.98 +++ extended_server_list+='10.204.217.98:9092 ' +++ '[' -n '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092 ' ']' +++ echo '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ KAFKA_SERVERS='10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ ANALYTICS_API_VIP= ++ CONFIG_API_VIP= ++ LINKLOCAL_SERVICE_PORT=80 ++ LINKLOCAL_SERVICE_NAME=metadata ++ LINKLOCAL_SERVICE_IP=169.254.169.254 ++ IPFABRIC_SERVICE_PORT=8775 ++ IPFABRIC_SERVICE_HOST= ++ XMPP_SSL_ENABLE=False ++ XMPP_SERVER_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ XMPP_SERVER_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ XMPP_SERVER_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ INTROSPECT_SSL_ENABLE=False ++ INTROSPECT_SSL_INSECURE=True ++ INTROSPECT_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ INTROSPECT_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ INTROSPECT_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ SANDESH_SSL_ENABLE=False ++ SANDESH_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ SANDESH_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ SANDESH_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ read -r -d '' sandesh_client_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ xmpp_certs_config= ++ METADATA_SSL_ENABLE=false ++ METADATA_SSL_CERTFILE= ++ METADATA_SSL_KEYFILE= ++ METADATA_SSL_CA_CERTFILE= ++ METADATA_SSL_CERT_TYPE= ++ RABBITMQ_VHOST=/ ++ RABBITMQ_USER=guest ++ RABBITMQ_PASSWORD=guest ++ RABBITMQ_USE_SSL=False ++ RABBITMQ_SSL_VER= ++ RABBITMQ_CLIENT_SSL_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ RABBITMQ_CLIENT_SSL_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ RABBITMQ_CLIENT_SSL_CACERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ read -r -d '' rabbitmq_config ++ true ++ read -r -d '' rabbit_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ AGENT_MODE=kernel ++ DPDK_UIO_DRIVER=uio_pci_generic ++ CPU_CORE_MASK=0x01 ++ HUGE_PAGES=1024 ++ HUGE_PAGES_DIR=/dev/hugepages ++ DPDK_MEM_PER_SOCKET=1024 ++ VHOST_CONFIG_DIR=/etc/sysconfig/network-scripts ++ TSN_EVPN_MODE=False ++ VROUTER_CRYPT_INTERFACE=crypt0 ++ VROUTER_DECRYPT_INTERFACE=decrypt0 ++ VROUTER_DECRYPT_KEY=15 ++ DIST_SNAT_PROTO_PORT_LIST= ++ FLOW_EXPORT_RATE=0 ++ FABRIC_SNAT_HASH_TABLE_SIZE=4096 ++ PRIORITY_ID= ++ PRIORITY_BANDWIDTH= ++ PRIORITY_SCHEDULING= ++ QOS_QUEUE_ID= ++ QOS_LOGICAL_QUEUES= ++ QOS_DEF_HW_QUEUE=False ++ PRIORITY_TAGGING=True + pre_start_init + wait_certs_if_ssl_enabled + is_ssl_enabled + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + return ++ get_listen_ip_for_node ANALYTICS +++ find_my_ip_and_order_for_node ANALYTICS +++ cut -d ' ' -f 1 +++ local server_typ=ANALYTICS_NODES +++ local server_list= +++ IFS=, +++ read -ra server_list ++++ get_local_ips ++++ tr '\n' , ++++ cat /proc/net/fib_trie ++++ awk '/32 host/ { print f } {f=$2}' +++ local local_ips=,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, +++ local ord=1 +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.52, ]] +++ (( ord+=1 )) +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.71, ]] +++ echo 10.204.217.71 2 +++ return ++ local ip=10.204.217.71 ++ [[ -z 10.204.217.71 ]] ++ echo 10.204.217.71 + host_ip=10.204.217.71 + cat + add_ini_params_from_env ANALYTICS_API /etc/contrail/contrail-analytics-api.conf + local service_name=ANALYTICS_API + local cfg_path=/etc/contrail/contrail-analytics-api.conf + local delim=__ ++ grep '^ANALYTICS_API__.*__.*=.*$' ++ sort ++ cut -d = -f 1 ++ set -o posix ++ set ++ sed 's/^ANALYTICS_API__//g' + local vars= + local section= + set_third_party_auth_config + [[ noauth != \k\e\y\s\t\o\n\e ]] + return + set_vnc_api_lib_ini + local tmp_file=/etc/contrail/vnc_api_lib.ini.tmp + cat + [[ noauth == \k\e\y\s\t\o\n\e ]] [root@nodeg31 ~]# contrail-status Pod Service Original Name State Status analytics alarm-gen contrail-analytics-alarm-gen restarting Restarting (1) 2 minutes ago analytics api contrail-analytics-api restarting Restarting (1) 2 minutes ago analytics collector contrail-analytics-collector running Up 2 hours analytics nodemgr contrail-nodemgr running Up 2 hours analytics query-engine contrail-analytics-query-engine running Up 2 hours analytics snmp-collector contrail-analytics-snmp-collector restarting Restarting (1) 44 minutes ago analytics topology contrail-analytics-topology restarting Restarting (1) 44 minutes ago config api contrail-controller-config-api running Up 2 hours config cassandra contrail-external-cassandra running Up 2 hours config device-manager contrail-controller-config-devicemgr running Up 2 hours config nodemgr contrail-nodemgr running Up About a minute config rabbitmq contrail-external-rabbitmq running Up 2 hours config schema contrail-controller-config-schema running Up 2 hours config svc-monitor contrail-controller-config-svcmonitor running Up 2 hours config zookeeper contrail-external-zookeeper running Up 2 hours control control contrail-controller-control-control running Up 2 hours control dns contrail-controller-control-dns running Up 2 hours control named contrail-controller-control-named running Up 2 hours control nodemgr contrail-nodemgr running Up 2 hours database cassandra contrail-external-cassandra running Up 2 hours database kafka contrail-external-kafka running Up 2 hours database nodemgr contrail-nodemgr running Up 2 hours database zookeeper contrail-external-zookeeper running Up 2 hours kubernetes kube-manager contrail-kubernetes-kube-manager running Up 2 hours webui job contrail-controller-webui-job running Up 2 hours webui web contrail-controller-webui-web running Up 2 hours == Contrail control == control: active nodemgr: active named: active dns: active == Contrail kubernetes == kube-manager: active == Contrail database == kafka: active nodemgr: initializing (Disk for DB is too low. ) zookeeper: active cassandra: active == Contrail analytics == snmp-collector: inactive query-engine: active api: inactive<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< alarm-gen: inactive nodemgr: active collector: active topology: inactive == Contrail webui == web: active job: active == Contrail config == api: active zookeeper: active svc-monitor: backup nodemgr: initializing device-manager: backup cassandra: active rabbitmq: active schema: backup [root@nodeg31 ~]# Below analytics containers are keeps restarting contrail-analytics-alarm-gen:ocata-5.0-80 contrail-analytics-api:ocata-5.0-80 contrail-analytics-snmp-collector:ocata-5.0-80 contrail-analytics-topology:ocata-5.0-80 With same latest ansible deployer code,on build79,we don't see this issue. build :ocata-5.0-80 deployer :ansible-deployer hostos :centos7.5 regitry :10.204.217.152:5000 setup: controller/master :nodec19,nodec20,nodec21(HA with multi innterface) compute/minion . :nodei16,nodei18 docker log snippet ===================     + mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini + exec /usr/bin/python /usr/bin/contrail-analytics-api -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf -c /etc/contrail/contrail-analytics-api.conf -c /etc/contrail/contrail-keystone-auth.conf 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: CONNECT TO COLLECTOR: True 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG] 05/31/2018 08:00:32 PM [contrail-analytics-api]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-analytics-api.log] Traceback (most recent call last):   File "/usr/bin/contrail-analytics-api", line 9, in <module>     load_entry_point('opserver==0.1dev', 'console_scripts', 'contrail-analytics-api')()   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 2638, in main     opserver = OpServer(args_str)   File "/usr/lib/python2.7/site-packages/opserver/opserver.py", line 702, in __init__     staticmethod(ConnectionState.get_process_state_cb), AttributeError: type object 'ConnectionState' has no attribute 'get_process_state_cb' ++++ echo 10.204.217.98 +++ local server_address=10.204.217.98 +++ extended_server_list+='10.204.217.98:9092 ' +++ '[' -n '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092 ' ']' +++ echo '10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ KAFKA_SERVERS='10.204.217.52:9092 10.204.217.71:9092 10.204.217.98:9092' ++ ANALYTICS_API_VIP= ++ CONFIG_API_VIP= ++ LINKLOCAL_SERVICE_PORT=80 ++ LINKLOCAL_SERVICE_NAME=metadata ++ LINKLOCAL_SERVICE_IP=169.254.169.254 ++ IPFABRIC_SERVICE_PORT=8775 ++ IPFABRIC_SERVICE_HOST= ++ XMPP_SSL_ENABLE=False ++ XMPP_SERVER_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ XMPP_SERVER_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ XMPP_SERVER_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ INTROSPECT_SSL_ENABLE=False ++ INTROSPECT_SSL_INSECURE=True ++ INTROSPECT_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ INTROSPECT_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ INTROSPECT_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ SANDESH_SSL_ENABLE=False ++ SANDESH_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ SANDESH_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ SANDESH_CA_CERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ read -r -d '' sandesh_client_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ xmpp_certs_config= ++ METADATA_SSL_ENABLE=false ++ METADATA_SSL_CERTFILE= ++ METADATA_SSL_KEYFILE= ++ METADATA_SSL_CA_CERTFILE= ++ METADATA_SSL_CERT_TYPE= ++ RABBITMQ_VHOST=/ ++ RABBITMQ_USER=guest ++ RABBITMQ_PASSWORD=guest ++ RABBITMQ_USE_SSL=False ++ RABBITMQ_SSL_VER= ++ RABBITMQ_CLIENT_SSL_CERTFILE=/etc/contrail/ssl/certs/server.pem ++ RABBITMQ_CLIENT_SSL_KEYFILE=/etc/contrail/ssl/private/server-privkey.pem ++ RABBITMQ_CLIENT_SSL_CACERTFILE=/etc/contrail/ssl/certs/ca-cert.pem ++ read -r -d '' rabbitmq_config ++ true ++ read -r -d '' rabbit_config ++ true ++ is_enabled False ++ local val=false ++ [[ false == \t\r\u\e ]] ++ [[ false == \y\e\s ]] ++ [[ false == \e\n\a\b\l\e\d ]] ++ AGENT_MODE=kernel ++ DPDK_UIO_DRIVER=uio_pci_generic ++ CPU_CORE_MASK=0x01 ++ HUGE_PAGES=1024 ++ HUGE_PAGES_DIR=/dev/hugepages ++ DPDK_MEM_PER_SOCKET=1024 ++ VHOST_CONFIG_DIR=/etc/sysconfig/network-scripts ++ TSN_EVPN_MODE=False ++ VROUTER_CRYPT_INTERFACE=crypt0 ++ VROUTER_DECRYPT_INTERFACE=decrypt0 ++ VROUTER_DECRYPT_KEY=15 ++ DIST_SNAT_PROTO_PORT_LIST= ++ FLOW_EXPORT_RATE=0 ++ FABRIC_SNAT_HASH_TABLE_SIZE=4096 ++ PRIORITY_ID= ++ PRIORITY_BANDWIDTH= ++ PRIORITY_SCHEDULING= ++ QOS_QUEUE_ID= ++ QOS_LOGICAL_QUEUES= ++ QOS_DEF_HW_QUEUE=False ++ PRIORITY_TAGGING=True + pre_start_init + wait_certs_if_ssl_enabled + is_ssl_enabled + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + is_enabled False + local val=false + [[ false == \t\r\u\e ]] + [[ false == \y\e\s ]] + [[ false == \e\n\a\b\l\e\d ]] + return ++ get_listen_ip_for_node ANALYTICS +++ find_my_ip_and_order_for_node ANALYTICS +++ cut -d ' ' -f 1 +++ local server_typ=ANALYTICS_NODES +++ local server_list= +++ IFS=, +++ read -ra server_list ++++ get_local_ips ++++ tr '\n' , ++++ cat /proc/net/fib_trie ++++ awk '/32 host/ { print f } {f=$2}' +++ local local_ips=,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, +++ local ord=1 +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.52, ]] +++ (( ord+=1 )) +++ for server in '"${server_list[@]}"' +++ [[ ,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,10.204.217.71,77.77.1.30,127.0.0.1,172.17.0.1,, =~ ,10\.204\.217\.71, ]] +++ echo 10.204.217.71 2 +++ return ++ local ip=10.204.217.71 ++ [[ -z 10.204.217.71 ]] ++ echo 10.204.217.71 + host_ip=10.204.217.71 + cat + add_ini_params_from_env ANALYTICS_API /etc/contrail/contrail-analytics-api.conf + local service_name=ANALYTICS_API + local cfg_path=/etc/contrail/contrail-analytics-api.conf + local delim=__ ++ grep '^ANALYTICS_API__.*__.*=.*$' ++ sort ++ cut -d = -f 1 ++ set -o posix ++ set ++ sed 's/^ANALYTICS_API__//g' + local vars= + local section= + set_third_party_auth_config + [[ noauth != \k\e\y\s\t\o\n\e ]] + return + set_vnc_api_lib_ini + local tmp_file=/etc/contrail/vnc_api_lib.ini.tmp + cat + [[ noauth == \k\e\y\s\t\o\n\e ]] [root@nodeg31 ~]# contrail-status Pod Service Original Name State Status analytics alarm-gen contrail-analytics-alarm-gen restarting Restarting (1) 2 minutes ago analytics api contrail-analytics-api restarting Restarting (1) 2 minutes ago analytics collector contrail-analytics-collector running Up 2 hours analytics nodemgr contrail-nodemgr running Up 2 hours analytics query-engine contrail-analytics-query-engine running Up 2 hours analytics snmp-collector contrail-analytics-snmp-collector restarting Restarting (1) 44 minutes ago analytics topology contrail-analytics-topology restarting Restarting (1) 44 minutes ago config api contrail-controller-config-api running Up 2 hours config cassandra contrail-external-cassandra running Up 2 hours config device-manager contrail-controller-config-devicemgr running Up 2 hours config nodemgr contrail-nodemgr running Up About a minute config rabbitmq contrail-external-rabbitmq running Up 2 hours config schema contrail-controller-config-schema running Up 2 hours config svc-monitor contrail-controller-config-svcmonitor running Up 2 hours config zookeeper contrail-external-zookeeper running Up 2 hours control control contrail-controller-control-control running Up 2 hours control dns contrail-controller-control-dns running Up 2 hours control named contrail-controller-control-named running Up 2 hours control nodemgr contrail-nodemgr running Up 2 hours database cassandra contrail-external-cassandra running Up 2 hours database kafka contrail-external-kafka running Up 2 hours database nodemgr contrail-nodemgr running Up 2 hours database zookeeper contrail-external-zookeeper running Up 2 hours kubernetes kube-manager contrail-kubernetes-kube-manager running Up 2 hours webui job contrail-controller-webui-job running Up 2 hours webui web contrail-controller-webui-web running Up 2 hours == Contrail control == control: active nodemgr: active named: active dns: active == Contrail kubernetes == kube-manager: active == Contrail database == kafka: active nodemgr: initializing (Disk for DB is too low. ) zookeeper: active cassandra: active == Contrail analytics == snmp-collector: inactive query-engine: active api: inactive<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< alarm-gen: inactive nodemgr: active collector: active topology: inactive == Contrail webui == web: active job: active == Contrail config == api: active zookeeper: active svc-monitor: backup nodemgr: initializing device-manager: backup cassandra: active rabbitmq: active schema: backup [root@nodeg31 ~]#
2018-06-01 16:01:57 Sundaresan Rajangam juniperopenstack/r5.0: assignee Sundaresan Rajangam (srajanga) Venkatesh Velpula (vvelpula)
2018-06-01 16:02:04 Sundaresan Rajangam juniperopenstack/trunk: assignee Sundaresan Rajangam (srajanga) Venkatesh Velpula (vvelpula)
2018-06-01 16:02:14 Sundaresan Rajangam juniperopenstack/r5.0: status New Opinion
2018-06-01 16:02:19 Sundaresan Rajangam juniperopenstack/trunk: status New Opinion
2018-06-06 17:52:54 Venkatesh Velpula juniperopenstack/r5.0: status Opinion Invalid
2018-06-06 17:52:57 Venkatesh Velpula juniperopenstack/trunk: status Opinion Invalid