SM: Provision gets stuck with config_started with cassandra issues

[sundarkh]

SM: Provision gets stuck with config_started with cassandra issues

Seen with 3.0 2729, 3.1.0.0 build 2728, 3.0.1.0-17

 contrail-status
== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Discovery:Collector[Subscribe - Status Code 503], Collector connection down)
contrail-config-nodemgr active
contrail-device-manager initializing (Discovery:Collector[Subscribe - Status Code 503], ApiServer:ApiServer[], Collector connection down)
contrail-discovery:0 initializing (Discovery:Collector[Subscribe - Connection Error], Collector, Database:Cassandra[] connection down)
contrail-schema backup
contrail-svc-monitor backup
ifmap active

== Contrail Database ==
contrail-database: inactive
supervisor-database: active
contrail-database-nodemgr initializing (Cassandra state detected DOWN.Disk space for analytics db not retrievable.)
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

Mar 29 05:52:30 nodeg21 puppet-agent[25372]: contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.61" --api_server_port 8082 --host_name_list "nodeg33,nodec58" --host_ip_list "10.204.217.73,10.204.217.98" --router_asn "64512" --mt_options "admin,contrail123,admin" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out
Mar 29 05:52:30 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Exec_provision_control/Notify[contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.61" --api_server_port 8082 --host_name_list "nodeg33,nodec58" --host_ip_list "10.204.217.73,10.204.217.98" --router_asn "64512" --mt_options "admin,contrail123,admin" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out]/message) defined 'message' as 'contrail contrail_exec_provision_control is python exec_provision_control.py --api_server_ip "10.204.217.61" --api_server_port 8082 --host_name_list "nodeg33,nodec58" --host_ip_list "10.204.217.73,10.204.217.98" --router_asn "64512" --mt_options "admin,contrail123,admin" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out'
Mar 29 05:55:01 nodeg21 CRON[27777]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 29 05:56:00 nodeg21 puppet-agent[25372]: python exec_provision_control.py --api_server_ip "10.204.217.61" --api_server_port 8082 --host_name_list "nodeg33,nodec58" --host_ip_list "10.204.217.73,10.204.217.98" --router_asn "64512" --mt_options "admin,contrail123,admin" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]
Mar 29 05:56:00 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Exec_provision_control/Exec[exec-provision-control]/returns) change from notrun to 0 failed: python exec_provision_control.py --api_server_ip "10.204.217.61" --api_server_port 8082 --host_name_list "nodeg33,nodec58" --host_ip_list "10.204.217.73,10.204.217.98" --router_asn "64512" --mt_options "admin,contrail123,admin" && echo exec-provision-control >> /etc/contrail/contrail_config_exec.out returned 1 instead of one of [0]
Mar 29 05:56:00 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Exec_provision_control/Notify[executed exec_provision_control]) Dependency Exec[exec-provision-control] has failures: true
Mar 29 05:56:00 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Exec_provision_control/Notify[executed exec_provision_control]) Skipping because of failed dependencies
Mar 29 05:59:30 nodeg21 puppet-agent[25372]: python /etc/contrail/contrail_setup_utils/setup_external_bgp.py --bgp_params "[('blr-mx2', '10.204.216.245')]" --api_server_ip "10.204.217.61" --api_server_port 8082 --router_asn "64512" --mt_options "admin,contrail123,admin" && echo provision-external-bgp >> /etc/contrail/contrail_config_exec.out returned 255 instead of one of [0]
Mar 29 05:59:30 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Setup_external_bgp/Exec[provision-external-bgp]/returns) change from notrun to 0 failed: python /etc/contrail/contrail_setup_utils/setup_external_bgp.py --bgp_params "[('blr-mx2', '10.204.216.245')]" --api_server_ip "10.204.217.61" --api_server_port 8082 --router_asn "64512" --mt_options "admin,contrail123,admin" && echo provision-external-bgp >> /etc/contrail/contrail_config_exec.out returned 255 instead of one of [0]
Mar 29 05:59:30 nodeg21 puppet-agent[25372]: executed setup_external_bgp
Mar 29 05:59:30 nodeg21 puppet-agent[25372]: (/Stage[main]/Contrail::Setup_external_bgp/Notify[executed setup_external_bgp]/message) defined 'message' as 'executed setup_external_bgp'
Mar 29 06:00:01 nodeg21 CRON[28420]: (neutron) CMD (if [ -x /usr/bin/neutron-netns-cleanup ] ; then /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-

[sundarkh (sundar-kh)]

Issue not seen in 3.0 build 25

[Jeba Paulaiyan (jebap)]

As this is not seen in 3.0-25, test in 3.0-latest also and close this if this is not seen any more.

[Ashish Ranjan (aranjan-n)]

Reducing severity per Jeba's comment

[Abhay Joshi (abhayj)]

Problem not seen in 3.0.2-build 25 and above.