HA:rabbitmq and ha proxy getting restarted by contrail_hamon every minute

[Sandip Dey]

Build:3.0 26

Had the blow setup.
3 openstack node.

I this setup, rabbitmq and haproxy getting restarted every minute.Because of this, glance,keystone etc were not accessible,hence vm create failed.

Logs saved at :http://10.204.216.50/Docs/bugs/<bug-id>

Log
===
nodei32
========
 * Restarting message broker rabbitmq-server ^[[80G Only one instance of contrail-rmq-monitor.sh can run at one time.^M
^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G Only one instance of contrail-rmq-monitor.sh can run at one time.^M
^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M
 * Restarting message broker rabbitmq-server ^[[80G ^M^[[74G[ OK ]^M

nodei35
========
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 7809^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 5056^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 1316^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 26246^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 23369^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 20259^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 15812^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 9661^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 3942^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 31403^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 25544^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M
haproxy stop/waiting^M
haproxy start/running, process 22770^M
Only one instance of contrail-rmq-monitor.sh can run at one time.^M

nodei36
=======
=INFO REPORT==== 6-May-2016::12:07:03 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:08:56 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:10:36 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:12:33 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:14:27 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:16:07 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:18:03 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:19:58 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:21:38 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:23:32 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:25:29 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:27:10 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:29:02 ===
Server startup complete; 0 plugins started.
--
=INFO REPORT==== 6-May-2016::12:31:00 ===
Server startup complete; 0 plugins started.
root@nodei36:/var/log# date
Fri May 6 12:32:32 IST 2016

setup
======
host1 = 'root@10.204.217.139'
host2 = 'root@10.204.217.140'
host3 = 'root@10.204.217.147'
host4 = 'root@10.204.217.144'
host5 = 'root@10.204.217.147'
host6 = 'root@10.204.217.148'
host7 = 'root@10.204.217.149'
host8 = 'root@10.204.217.150'
host9 = 'root@10.204.217.210'
host10 = 'root@10.204.217.217'
host11 = 'root@10.204.217.218'
host12 = 'root@10.204.217.220'
host13 = 'root@10.204.217.247'
host14 = 'root@10.204.217.248'
host15 = 'root@10.204.217.249'

ext_routers = [('blr-mx2', '10.204.216.245')]
router_asn = 64512
public_vn_rtgt = 30001
#public_vn_subnet = "10.204.219.72/29"

host_build = 'vjoshi@10.204.216.56'

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6,host7, host8, host9, host10, host11, host12, host13, host14, host15],
    'cfgm': [host1, host2, host3],
    'openstack': [host4, host5, host6],
    'webui': [host1, host2, host3],
    'control': [host1, host2, host3],
    'compute': [host7, host8, host9, host10, host11, host12, host13, host14, host15],
    'collector': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei27', 'nodei28', 'nodei30', 'nodei32', 'nodei35', 'nodei36', 'nodei37', 'nodei38', 'nodel4', 'nodel7', 'nodel8', 'nodel9', 'nodel10', 'nodel11', 'nodel12']

[Sanju Abraham (asanju)]

This is duplicate of https://bugs.launchpad.net/juniperopenstack/+bug/1578847