contrail-alarm-gen in a tight loop trying to connect to zk

[Vedamurthy Joshi]

R3.0 Build 2713 Ubuntu 14.04 Kilo multi-node setup

Contrail alarm started reporting zk connection down on all the 3 controller nodes on this setup

root@katrina-vm:/var/log/contrail# ps aux |grep alarm
root 28264 0.0 0.0 11740 916 pts/0 R+ 00:34 0:00 grep --color=auto alarm
contrail 32407 92.9 0.6 317804 105928 ? Rl Feb15 534:14 /usr/bin/python /usr/bin/contrail-alarm-gen -c /etc/contrail/contrail-alarm-gen.conf
root@katrina-vm:/var/log/contrail#

root@katrina-vm:/var/log/contrail# netstat -anp |grep 32407 | grep 2181 | wc -l
173
root@katrina-vm:/var/log/contrail#

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen initializing (Zookeeper:Zookeeper connection down)
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

root@katrina-vm:/var/log/contrail# tail ../zookeeper/zookeeper.log
2016-02-16 00:30:48,722 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.2 - max is 60
2016-02-16 00:30:49,935 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.2 - max is 60
2016-02-16 00:30:50,225 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.3 - max is 60
2016-02-16 00:30:51,349 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.1 - max is 60
2016-02-16 00:30:51,566 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.2 - max is 60
2016-02-16 00:30:53,913 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.3 - max is 60
2016-02-16 00:30:53,939 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.1 - max is 60
2016-02-16 00:30:54,261 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.2 - max is 60
2016-02-16 00:30:54,627 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.3 - max is 60
2016-02-16 00:30:55,146 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.1.1 - max is 60
root@katrina-vm:/var/log/contrail#

[Ankit Jain (ankitja)]

I'm also seeing this issue on my setup with the latest build 2713

@ nodeg13: contrail-alarm-gen initializing (Zookeeper:Zookeeper connection down)

{

    "value":

[

{

    "name": "nodea21",
    "value":

{

    "AlarmgenPartition":

{

    "inst_parts":

[

                {
                    "instance": "0",
                    "partitions": [ ]
                }
            ]
        }
    }

},
{

    "name": "nodeg13",
    "value":

{

    "AlarmgenPartition":

{

    "inst_parts":

[

                {
                    "instance": "0",
                    "partitions": [ ]
                }
            ]
        }
    }

},
{

    "name": "nodeg20",
    "value":

{

    "AlarmgenPartition":

{

    "inst_parts":

[

                        {
                            "instance": "0",
                            "partitions": [ ]
                        }
                    ]
                }
            }
        }
    ]

}

[Anish Mehta (amehta00)]

Might be a side-effect of bug #1545953

[Raj Reddy (rajreddy)]

Fixed through #1545953