On a ansible-provisioned setup, rabbitmq connection keeps flapping

[Vedamurthy Joshi]

R4.0 3073 Ubuntu 14.04 container-based kubernetes setup

Below logs are seen every minute on this single controller setup

05/09/2017 10:20:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:20:37 AM [contrail-api]: NodeStatusUVE: data = << name = testbed-1-vm1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.204.217.194:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.204.217.194:8086, ] status = Up description = ClientInit to Established on EvSandeshCtrlMessageRecv >>, << type = Database name = Cassandra server_addrs = [ 10.204.217.194:9161, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.204.217.194:5672, ] status = Down description = >>, ] description = Database:RabbitMQ[] connection down >>, ] >>
05/09/2017 10:20:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection ESTABLISHED <Connection: amqp://guest:**@10.204.217.194:5672// at 0x2b87ade16b50>
05/09/2017 10:20:37 AM [contrail-api]: NodeStatusUVE: data = << name = testbed-1-vm1 process_status = [ << module_id = contrail-api instance_id = 0 state = Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.204.217.194:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.204.217.194:8086, ] status = Up description = ClientInit to Established on EvSandeshCtrlMessageRecv >>, << type = Database name = Cassandra server_addrs = [ 10.204.217.194:9161, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.204.217.194:5672, ] status = Up description = >>, ] description = >>, ] >>
^C

root@testbed-1-vm1(controller):/var/log/contrail# tail -100 contrail-api.log | grep "RabbitMQ connection down"
05/09/2017 10:08:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:09:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:09:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:10:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:10:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:11:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:11:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:12:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:12:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:13:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:13:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:14:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:14:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:15:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:15:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:16:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:16:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:17:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:17:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:18:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:18:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:19:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:19:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:20:07 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
05/09/2017 10:20:37 AM [contrail-api]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down
root@testbed-1-vm1(controller):/var/log/contrail# cat /etc/rabbitmq/rabbitmq.config
[
  {rabbit, [
    {tcp_listeners, [{"0.0.0.0", 5672}]}

    ,{loopback_users, []}
    ,{log_levels,[{ connection, info },{ mirroring, info }]}
,{vm_memory_high_watermark, 0.4},
{disk_free_limit,50000000},
{heartbeat,10}, <<<<<<<< Is this why ?
{delegate_count,20},
{channel_max,5000},
{tcp_listen_options, [
    binary,
    {packet, raw},
    {reuseaddr, true},
    {backlog, 128},
    {nodelay, true},
    {exit_on_close, false},
    {keepalive, true}
]},
{collect_statistics_interval, 60000}
  ]}
].
root@testbed-1-vm1(controller):/var/log/contrail#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31232
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31232
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/2d0cd1ca2b06e3cab1da8ec8da94ba3db589df25
Submitter: Zuul (<email address hidden>)
Branch: master

commit 2d0cd1ca2b06e3cab1da8ec8da94ba3db589df25
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 9 05:40:15 2017 -0700

provisioning rabbitmq with default heartbeat

timeout(60secs), because contrail-api sends
heartbeat messages every 60secs(default)

Change-Id: Ia104cefd445d9d6e8527aa069a5a6c057ad6b638
Closes-Bug: 1689521

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31275
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31275
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/7c05afde4e25917ba49bcabc20213dee71cf2464
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 7c05afde4e25917ba49bcabc20213dee71cf2464
Author: Ignatious Johnson Christopher <email address hidden>
Date: Tue May 9 05:40:15 2017 -0700

provisioning rabbitmq with default heartbeat

timeout(60secs), because contrail-api sends
heartbeat messages every 60secs(default)

Change-Id: Ia104cefd445d9d6e8527aa069a5a6c057ad6b638
Closes-Bug: 1689521
(cherry picked from commit 2d0cd1ca2b06e3cab1da8ec8da94ba3db589df25)