sm:mainline2738: provision fails with cassandra issues

[sundarkh]

sm:mainline2738: provision fails with database_completed

SM Node : nodej5 (/root/sm_files/cluster_multi_inf_new_param.json, /root/sm_files/server_multi_if_new_param.json)

Target

Config Nodes : [u'nodec35', u'nodec33']
Control Nodes : [u'nodec35', u'nodec33']
Compute Nodes : [u'nodea4', u'nodec57']
Openstack Node : nodec35
WebUI Node : nodec33
Analytics Nodes : [u'nodec33']

error log on target node

Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Common/Contrail::Lib::Upgrade_kernel[kernel_upgrade]/Exec[update_grub]/returns) executed successfully
Jun 30 03:55:35 nodec33 puppet-agent[15032]: After reboot
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Common/Contrail::Lib::Upgrade_kernel[kernel_upgrade]/Notify[After reboot]/message) defined 'message' as 'After reboot'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: executed disable_ufw
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Disable_ufw/Notify[executed disable_ufw]/message) defined 'message' as 'executed disable_ufw'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Flush_iptables/Exec[iptables --flush]/returns) executed successfully
Jun 30 03:55:35 nodec33 puppet-agent[15032]: flushed iptables
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Flush_iptables/Notify[flushed iptables]/message) defined 'message' as 'flushed iptables'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Core_file_unlimited/Exec[core-file-unlimited]/returns) executed successfully
Jun 30 03:55:35 nodec33 puppet-agent[15032]: executed core-file-unlimited
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Core_file_unlimited/Notify[executed core-file-unlimited]/message) defined 'message' as 'executed core-file-unlimited'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Common/Sysctl::Value[net.ipv4.ip_local_reserved_ports]/Sysctl_runtime[net.ipv4.ip_local_reserved_ports]/val) val changed '33306,35357-35358' to '35357,35358,33306,33306,35357-35358'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: executed enable-kernel-core
Jun 30 03:55:35 nodec33 puppet-agent[15032]: (/Stage[common]/Contrail::Enable_kernel_core/Notify[executed enable-kernel-core]/message) defined 'message' as 'executed enable-kernel-core'
Jun 30 03:55:35 nodec33 puppet-agent[15032]: Finished catalog run in 5.19 seconds
Jun 30 03:55:37 nodec33 puppet-agent[16307]: Local environment: "production" doesn't match server specified node environment "ubuntu14kilo2738", switching agent to "ubuntu14kilo2738".
Jun 30 03:55:39 nodec33 kernel: [ 1094.777040] Load

[sundarkh (sundar-kh)]

Seen in kilo/liberty

[Abhay Joshi (abhayj)]

We need setup in failed state. I do not see error anywhere above in the pasted log.

[Dheeraj Gautam (dgautam)]

cassandra was not coming up on this setup. on restarting cassandra, provisioning got completed.

[sundarkh (sundar-kh)]

2739

Server Manager status show that Provision does get completed, but the target services are not up, with Cassandra down

Linux nodec33 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@nodec33:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control initializing (Discovery:Collector, Discovery:IfmapServer, Discovery:xmpp-server connection down)
contrail-control-nodemgr initializing (Discovery:Collector[Subscribe - Status Code 500] connection down)
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen initializing (Discovery:ApiServer[Subscribe - Status Code 500], Discovery:Collector[Subscribe - Status Code 500], Discovery:AlarmGenerator[Publish Error - Status Code 500] connection down)
contrail-analytics-api initializing (Discovery:AlarmGenerator[Subscribe - Status Code 500], Discovery:Collector[Subscribe - Status Code 500], Discovery:OpServer[Publish Error - Status Code 500] connection down)
contrail-analytics-nodemgr initializing (Discovery:Collector[Subscribe - Status Code 500] connection down)
contrail-collector initializing (Discovery:ApiServer, Discovery:Collector connection down)
contrail-query-engine initializing (Discovery:Collector connection down)
contrail-snmp-collector initializing (Discovery:Collector[Subscribe - Status Code 500], Discovery:ApiServer[Subscribe - Status Code 500] connection down)
contrail-topology initializing (Discovery:Collector[Subscribe - Status Code 500] connection down)

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Discovery:ApiServer[Publish Error - Status Code 500], Discovery:IfmapServer[Publish Error - Status Code 500], Discovery:Collector[Subscribe - Status Code 500], Database:Cassandra[] connection down)
contrail-config-nodemgr initializing (Discovery:Collector[Subscribe - Status Code 500] connection down)
contrail-device-manager backup
contrail-discovery:0 initializing (Discovery:Collector[Subscribe - Status Code 500], Database:Cassandra[] connection down)
contrail-schema backup
contrail-svc-monitor backup
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: inactive

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr initializing (Discovery:Collector[Subscribe - Status Code 500] connection down)
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

DEBUG [ScheduledTasks:1] 2016-07-03 23:34:46,902 ColumnFamilyStore.java:924 - Enqueuing flush of sstable_activity: 22090 (0%) on-heap, 0 ...

[sundarkh (sundar-kh)]

Issue was seen with params been given as
"database_dir": "/home/cassandra",
"database_minimum_diskGB": 32

after providing the parameters as per https://raw.githubusercontent.com/Juniper/contrail-server-manager/master/src/client/new-cluster.json , issue not seen.