SM provisioning of deleting controller and adding new controller gets stuck at openstack_started

[Sarath]

Server Manager provisioning of deleting controller and adding new controller gets stuck at openstack_started.
Thilak debugged the setup and problem found to be due to deleted controller being VIP server with states not removed appropriately and so impacting the newly added controller connecting to VIP server.

Jan 26 11:56:40 a5d11e36 os-prober: debug: /dev/mapper/a5d11e36--vg-swap_1: is active swap
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: (/Stage[first]/Contrail::Common/Contrail::Lib::Upgrade_kernel[kernel_upgrade]/Exec[update_grub]/returns) executed successfully
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: After reboot
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: (/Stage[first]/Contrail::Common/Contrail::Lib::Upgrade_kernel[kernel_upgrade]/Notify[After reboot]/message) defined 'message' as 'After reboot'
Jan 26 11:56:40 a5d11e36 kernel: [84263.515219] init: neutron-server main process (5892) terminated with status 1
Jan 26 11:56:40 a5d11e36 kernel: [84263.515233] init: neutron-server main process ended, respawning
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: Mysql - restart = true
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: (/Stage[main]/Mysql::Server/Notify[Mysql - restart = true]/message) defined 'message' as 'Mysql - restart = true'
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: Mysql- service_manage = true
Jan 26 11:56:40 a5d11e36 puppet-agent[1330]: (/Stage[main]/Mysql::Server/Notify[Mysql- service_manage = true]/message) defined 'message' as 'Mysql- service_manage = true'
Jan 26 11:56:40 a5d11e36 kernel: [84264.227993] init: neutron-server main process (6258) terminated with status 1
Jan 26 11:56:40 a5d11e36 kernel: [84264.228022] init: neutron-server main process ended, respawning
Jan 26 11:56:41 a5d11e36 kernel: [84264.950434] init: neutron-server main process (6390) terminated with status 1
Jan 26 11:56:41 a5d11e36 kernel: [84264.950449] init: neutron-server main process ended, respawning
Jan 26 11:56:42 a5d11e36 puppet-agent[1330]: SYNC_DB = true
Jan 26 11:56:42 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Common::Keystone/Notify[SYNC_DB = true]/message) defined 'message' as 'SYNC_DB = true'
Jan 26 11:56:42 a5d11e36 kernel: [84265.683040] init: neutron-server main process (6496) terminated with status 1
Jan 26 11:56:42 a5d11e36 kernel: [84265.683055] init: neutron-server main process ended, respawning
Jan 26 11:56:43 a5d11e36 kernel: [84266.371940] init: neutron-server main process (6539) terminated with status 1
Jan 26 11:56:43 a5d11e36 kernel: [84266.371959] init: neutron-server main process ended, respawning
Jan 26 11:56:43 a5d11e36 kernel: [84267.064048] init: neutron-server main process (6595) terminated with status 1
Jan 26 11:56:43 a5d11e36 kernel: [84267.064061] init: neutron-server main process ended, respawning
Jan 26 11:56:44 a5d11e36 kernel: [84267.768698] init: neutron-server main process (6610) terminated with status 1
Jan 26 11:56:44 a5d11e36 kernel: [84267.768712] init: neutron-server main process ended, respawning
Jan 26 11:56:44 a5d11e36 puppet-agent[1330]: contrail::profile::openstack_controller - neutron_db_connection = mysql://neutron:c0ntrail123@10.87.143.50/neutron
Jan 26 11:56:44 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack_controller/Notify[contrail::profile::openstack_controller - neutron_db_connection = mysql://neutron:c0ntrail123@10.87.143.50/neutron]/message) defined 'message' as 'contrail::profile::openstack_controller - neutron_db_connection = mysql://neutron:c0ntrail123@10.87.143.50/neutron'
Jan 26 11:56:45 a5d11e36 kernel: [84268.458694] init: neutron-server main process (6646) terminated with status 1
Jan 26 11:56:45 a5d11e36 kernel: [84268.458707] init: neutron-server main process ended, respawning
Jan 26 11:56:45 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack_controller/Exec[neutron-db-sync]/returns) executed successfully
Jan 26 11:56:45 a5d11e36 puppet-agent[1330]: contrail::profile::openstack_controller - enable_ceilometer = false
Jan 26 11:56:45 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack_controller/Notify[contrail::profile::openstack_controller - enable_ceilometer = false]/message) defined 'message' as 'contrail::profile::openstack_controller - enable_ceilometer = false'
Jan 26 11:56:45 a5d11e36 kernel: [84269.141406] init: neutron-server main process (6687) terminated with status 1
Jan 26 11:56:45 a5d11e36 kernel: [84269.141424] init: neutron-server main process ended, respawning
Jan 26 11:56:45 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack_controller/Contrail::Lib::Report_status[openstack_started]/Exec[contrail-status-openstack_started]/returns) executed successfully
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: openstack::common::nova - compute_ip_list = 10.87.143.9810.87.143.100
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Profile::Nova::Api/Notify[openstack::common::nova - compute_ip_list = 10.87.143.9810.87.143.100]/message) defined 'message' as 'openstack::common::nova - compute_ip_list = 10.87.143.9810.87.143.100'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: openstack::common::nova - host_ip = 10.87.143.84
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Profile::Nova::Api/Notify[openstack::common::nova - host_ip = 10.87.143.84]/message) defined 'message' as 'openstack::common::nova - host_ip = 10.87.143.84'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: openstack::common::nova -contrail_is_compute = false
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Profile::Nova::Api/Notify[openstack::common::nova -contrail_is_compute = false]/message) defined 'message' as 'openstack::common::nova -contrail_is_compute = false'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: openstack::common::nova - tmp_index = X X
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Profile::Nova::Api/Notify[openstack::common::nova - tmp_index = X X]/message) defined 'message' as 'openstack::common::nova - tmp_index = X X'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: openstack::common::nova - controller_management_address =
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Openstack::Profile::Nova::Api/Notify[openstack::common::nova - controller_management_address = ]/message) defined 'message' as 'openstack::common::nova - controller_management_address = '
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - heat_api_bind_host = 0.0.0.0
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - heat_api_bind_host = 0.0.0.0]/message) defined 'message' as 'contrail::profile::openstack::heat - heat_api_bind_host = 0.0.0.0'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - heat_api_bind_port = 8005
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - heat_api_bind_port = 8005]/message) defined 'message' as 'contrail::profile::openstack::heat - heat_api_bind_port = 8005'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - sql_connection = mysql://heat:c0ntrail123@10.87.143.50:33306/heat
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - sql_connection = mysql://heat:c0ntrail123@10.87.143.50:33306/heat]/message) defined 'message' as 'contrail::profile::openstack::heat - sql_connection = mysql://heat:c0ntrail123@10.87.143.50:33306/heat'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - rabbit_port = 5673
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - rabbit_port = 5673]/message) defined 'message' as 'contrail::profile::openstack::heat - rabbit_port = 5673'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - rabbit_host = 10.87.143.50
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - rabbit_host = 10.87.143.50]/message) defined 'message' as 'contrail::profile::openstack::heat - rabbit_host = 10.87.143.50'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - contrail_api_server = 10.87.143.50
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - contrail_api_server = 10.87.143.50]/message) defined 'message' as 'contrail::profile::openstack::heat - contrail_api_server = 10.87.143.50'
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: contrail::profile::openstack::heat - keystone_auth_public_url =
Jan 26 11:56:46 a5d11e36 puppet-agent[1330]: (/Stage[main]/Contrail::Profile::Openstack::Heat/Notify[contrail::profile::openstack::heat - keystone_auth_public_url = ]/message) defined 'message' as 'contrail::profile::openstack::heat - keystone_auth_public_url = '
Jan 26 11:56:46 a5d11e36 kernel: [84269.826633] init: neutron-server main process (6702) terminated with status 1

contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0-2697 2697
contrail-config 3.0-2697 2697
contrail-config-openstack 3.0-2697 2697
contrail-control 3.0-2697 2697
contrail-dns 3.0-2697 2697
contrail-docs 3.0-2697 2697
contrail-f5 3.0-2697 2697
contrail-fabric-utils 3.0-2697 2697
contrail-heat 3.0-2697 2697
contrail-lib 3.0-2697 2697
contrail-nodemgr 3.0-2697 2697
contrail-nova-networkapi 3.0-2697 2697
contrail-openstack 3.0-2697 2697
contrail-openstack-analytics 3.0-2697 2697
contrail-openstack-config 3.0-2697 2697
contrail-openstack-control 3.0-2697 2697
contrail-openstack-dashboard 3.0-2697 2697
contrail-openstack-database 3.0-2697 2697
contrail-openstack-ha 3.0-2697 2697
contrail-openstack-webui 3.0-2697 2697
contrail-setup 3.0-2697 2697
contrail-utils 3.0-2697 2697
contrail-web-controller 3.0-2697 2697
contrail-web-core 3.0-2697 2697
ifmap-python-client 0.1-2 2697
ifmap-server 0.3.2-1contrail2 2697
neutron-plugin-contrail 3.0-2697 2697
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
python-contrail 3.0-2697 2697
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail2697
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail2697

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16846
Submitter: Thilak Raj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16846
Committed: http://github.org/Juniper/contrail-puppet/commit/7439ea55a2d2f7063ea75831af8d27451f1647a8
Submitter: Zuul
Branch: master

commit 7439ea55a2d2f7063ea75831af8d27451f1647a8
Author: tsurendra <email address hidden>
Date: Wed Feb 3 14:02:52 2016 -0800

Closes-Bug: #1538357
Closes-Bug: #1538298
Closes-Bug: #1537290

config was stuck when adding a node at the begining, as ssl keys were
not re-distributed from the first new node.

controller gets stuck at openstack_started
When a node was delete and re-added,
old keepalived instance was still running.

"uninstall_collector"
Zookeeper was not getting restarted as result of this
below commit
https://github.com/Juniper/contrail-puppet/commit/e31d7a9ce230e2991d1a22cf4ecbd8b7f9ca2f88
Nitish provided a fix and tested it.

Change-Id: I5d77c6dae0fa527deb2331cc184b4f297ad39ff7