SM provisioning of add new controller stuck @ config_started

[Sarath]

Server-manager provisioning of add new controller stuck @ config_started.
Development Thilak debugged the setup and looks possible mysql restarted issue post openstack completion.

Steps Involved in seeing this problem,
1) provision remove 1 existing controller
2) provision add 1 new controller

root@Blaster-nsarath-004:/var/log/contrail-server-manager/provision/a5d11e36.englab.juniper.net# server-manager status server
{
    "server": [
        {
            "id": "a5d11e36",
            "ip_address": "10.87.143.84",
            "mac_address": "00:E0:81:CA:59:11",
            "status": "config_started"
        },
        {
            "id": "a5d11e14",
            "ip_address": "10.87.143.90",
            "mac_address": "00:E0:81:CA:5A:87",
            "status": "uninstall_openstack_completed"
        },
        {
            "id": "a5d11e13",
            "ip_address": "10.87.143.92",
            "mac_address": "00:E0:81:CA:55:13",
            "status": "config_started"
        },
        {
            "id": "a5d11e12",
            "ip_address": "10.87.143.94",
            "mac_address": "00:E0:81:C5:81:95",
            "status": "config_started"
        },
        {
            "id": "a5d11e10",
            "ip_address": "10.87.143.98",
            "mac_address": "00:E0:81:CA:59:59",
            "status": "provision_completed"
        },
        {
            "id": "a5d11e09",
            "ip_address": "10.87.143.100",
            "mac_address": "00:E0:81:CA:59:D3",
            "status": "provision_completed"
        }
    ]
}

root@a5d11e36:~#
root@a5d11e36:~# ps -ef | grep mysql
root 16737 27856 0 15:12 pts/2 00:00:00 grep --color=auto mysql
root 17716 1 0 11:57 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --wsrep_cluster_address=gcomm://
mysql 19149 17716 1 11:57 ? 00:03:08 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-cluster-address=gcomm:// --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
root@a5d11e36:~#
root@a5d11e36:~#
root@a5d11e36:~#
root@a5d11e36:~# vi /var/log/mysql/error.log
160125 17:47:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160125 17:47:29 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.1Y0mZQqoux
160125 17:47:33 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
160125 17:47:33 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
160125 17:47:33 [Note] Plugin 'FEDERATED' is disabled.
160125 17:47:33 InnoDB: The InnoDB memory heap is disabled
160125 17:47:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160125 17:47:33 InnoDB: Compressed tables use zlib 1.2.3.3
160125 17:47:33 InnoDB: Using Linux native AIO
160125 17:47:33 InnoDB: Initializing buffer pool, size = 128.0M
160125 17:47:33 InnoDB: Completed initialization of buffer pool
160125 17:47:33 InnoDB: highest supported file format is Barracuda.
160125 17:47:33 InnoDB: Waiting for the background threads to start
160125 17:47:34 InnoDB: 5.5.37 started; log sequence number 1595843
160125 17:47:34 [Warning] Failed to setup SSL
160125 17:47:34 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed
160125 17:47:34 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160125 17:47:34 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160125 17:47:34 [Note] Server socket created on IP: '0.0.0.0'.
160125 17:47:34 [Note] Event Scheduler: Loaded 0 events
160125 17:47:34 [Note] WSREP: Read nil XID from storage engines, skipping position init
160125 17:47:34 [Note] WSREP: wsrep_load(): loading provider library 'none'
160125 17:47:34 [Note] [Debug] WSREP: dummy_init
160125 17:47:34 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.37' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL), wsrep_25.10.r3990
160125 17:47:52 [Warning] IP address '10.87.143.92' could not be resolved: Name or service not known
160125 17:47:59 [Warning] IP address '10.87.143.94' could not be resolved: Name or service not known
160126 11:57:36 [Note] /usr/sbin/mysqld: Normal shutdown

160126 11:57:36 [Note] Event Scheduler: Purging the queue. 0 events
160126 11:57:38 [Warning] /usr/sbin/mysqld: Forcing close of thread 785343 user: 'nova'

160126 11:57:38 [Warning] /usr/sbin/mysqld: Forcing close of thread 785341 user: 'nova'

contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0-2697 2697
contrail-config 3.0-2697 2697
contrail-config-openstack 3.0-2697 2697
contrail-control 3.0-2697 2697
contrail-dns 3.0-2697 2697
contrail-docs 3.0-2697 2697
contrail-f5 3.0-2697 2697
contrail-fabric-utils 3.0-2697 2697
contrail-heat 3.0-2697 2697
contrail-lib 3.0-2697 2697
contrail-nodemgr 3.0-2697 2697
contrail-nova-networkapi 3.0-2697 2697
contrail-openstack 3.0-2697 2697
contrail-openstack-analytics 3.0-2697 2697
contrail-openstack-config 3.0-2697 2697
contrail-openstack-control 3.0-2697 2697
contrail-openstack-dashboard 3.0-2697 2697
contrail-openstack-database 3.0-2697 2697
contrail-openstack-ha 3.0-2697 2697
contrail-openstack-webui 3.0-2697 2697
contrail-setup 3.0-2697 2697
contrail-utils 3.0-2697 2697
contrail-web-controller 3.0-2697 2697
contrail-web-core 3.0-2697 2697
ifmap-python-client 0.1-2 2697
ifmap-server 0.3.2-1contrail2 2697
neutron-plugin-contrail 3.0-2697 2697
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail2697
python-contrail 3.0-2697 2697
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail2697
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail2697

[Sarath (nsarath)]

Logs can be found @

-bash-4.1$
-bash-4.1$ hostname
centos-build02
-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/PR-1538357
-bash-4.1$
-bash-4.1$ ls -l
total 5329388
-rwxrwxrwx 1 nsarath test 5435883520 Jan 26 15:52 1-log.tar*
-bash-4.1$

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16846
Submitter: Thilak Raj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16846
Committed: http://github.org/Juniper/contrail-puppet/commit/7439ea55a2d2f7063ea75831af8d27451f1647a8
Submitter: Zuul
Branch: master

commit 7439ea55a2d2f7063ea75831af8d27451f1647a8
Author: tsurendra <email address hidden>
Date: Wed Feb 3 14:02:52 2016 -0800

Closes-Bug: #1538357
Closes-Bug: #1538298
Closes-Bug: #1537290

config was stuck when adding a node at the begining, as ssl keys were
not re-distributed from the first new node.

controller gets stuck at openstack_started
When a node was delete and re-added,
old keepalived instance was still running.

"uninstall_collector"
Zookeeper was not getting restarted as result of this
below commit
https://github.com/Juniper/contrail-puppet/commit/e31d7a9ce230e2991d1a22cf4ecbd8b7f9ca2f88
Nitish provided a fix and tested it.

Change-Id: I5d77c6dae0fa527deb2331cc184b4f297ad39ff7