HA:cmon restarts haproxy every few hours in HA setup

[Sandip Dey]

Sanju debugged the issue

Ha proxy getting restarted every few hours and the openstack services could not be connected during that brief period.VM creation fails.

[5/14/16, 1:55:39 PM] vedujoshi: Hi Sanju, on a HA enabled-cluster, we sometimes see that cmon restarts haproxy
[5/14/16, 1:55:47 PM] vedujoshi: every fews hrs infact
[5/14/16, 1:55:56 PM] vedujoshi: ex :
[5/14/16, 1:55:57 PM] vedujoshi: Sat May 14 03:36:46 IST 2016: INFO: Restarted HAP becuase of stale dips
Sat May 14 03:36:52 IST 2016: INFO: CMON is not Running
Sat May 14 03:37:52 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 1:56:26 PM] vedujoshi: Sat May 14 05:27:00 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 1:56:43 PM] vedujoshi: why so … any idea ?
[5/14/16, 1:56:47 PM] Sanju Abraham: please change /etc/contrail/contrail-topology.conf to have contrail-analytics-api to have VIP address
[5/14/16, 1:57:03 PM] vedujoshi: i think we did that…let me check again
[5/14/16, 1:57:26 PM] vedujoshi: yes..that is taken care
[5/14/16, 1:57:59 PM] vedujoshi: before changing the contrail-topology.conf, it used to happen almost every min
[5/14/16, 1:58:06 PM] vedujoshi: now….it is once in few hrs
[5/14/16, 1:58:34 PM] Sanju Abraham: can you please let me know the setup, I can take a look
[5/14/16, 1:58:47 PM] vedujoshi: sure
[5/14/16, 1:59:04 PM] vedujoshi: testbed file is in nodei27:/opt/contrail/utils/fabfile/testbed/testbed.py
[5/14/16, 1:59:39 PM] vedujoshi: its late in the night for you…not urgent…you can take a look in the morning
[5/14/16, 2:00:17 PM] Sanju Abraham: IP?
[5/14/16, 2:00:39 PM] vedujoshi: 10.204.217.188 is nodei27 (nodei27.englab.juniper.net should resolve it)
[5/14/16, 2:26:45 PM] Sanju Abraham: this can happen only if there are any connections on a LB without the VIP. The last occurrence of this on nodei35 was
[5/14/16, 2:27:07 PM] Sanju Abraham: Sat May 14 05:51:30 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 2:27:41 PM] Sanju Abraham: current time : Sat May 14 14:27:11
[5/14/16, 2:28:30 PM] Sanju Abraham: there could have some connections that were trying to use this instance of LB and hence the monitoring job restart HAP.
[5/15/16, 8:29:32 AM] Sanju Abraham: please let me know the bugID
[5/15/16, 8:29:59 AM] Sandip Dey: i have not raised yet…let me raise it

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20619
Submitter: Sanju (<email address hidden>)

[Sanju Abraham (asanju)]

HAP instance is restarted only on standby instances to manage connections.

If any app tries to connect to the standby HAP instance then HAP is restarted assuming clients will reconnect to the node that has VIP and HAProxy.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20619
Submitter: Sanju (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20619
Committed: http://github.org/Juniper/contrail-provisioning/commit/e4cd10a19028e6520b8eb3d32005e3277f560639
Submitter: Zuul
Branch: R3.0

commit e4cd10a19028e6520b8eb3d32005e3277f560639
Author: Sanju Abraham <email address hidden>
Date: Wed May 25 01:45:10 2016 -0700

Connections to the standby instance of HAP should not be created. Fix provides a way to check for this and restart HAP with the intention that apps will reconnect to the right HAP instance
Closes-Bug: #1581905

Change-Id: I39529ac7c789bf45e896b96c5feebe9f74d6d374

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25703
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25705
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25703
Committed: http://github.org/Juniper/contrail-provisioning/commit/2708d89fa53df9823cac7fda37360451897d0bfd
Submitter: Zuul
Branch: master

commit 2708d89fa53df9823cac7fda37360451897d0bfd
Author: Ranjeet R <email address hidden>
Date: Fri Nov 4 01:25:22 2016 -0700

Fixes: HA:cmon restarts haproxy every few hours in HA setup
Connections to the standby instance of HAP should not be created.
Fix provides a way to check for this and restart HAP with the
intention that apps will reconnect to the right HAP instance
Closes-Bug: #1581905

Change-Id: I4406a08815ec27e2a57a7c947084f72905132446

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25705
Committed: http://github.org/Juniper/contrail-provisioning/commit/bfdccfa8e36971591dad7b019de6abaa9eb7465d
Submitter: Zuul
Branch: R3.1

commit bfdccfa8e36971591dad7b019de6abaa9eb7465d
Author: Ranjeet R <email address hidden>
Date: Fri Nov 4 01:54:22 2016 -0700

Fixes: HA:cmon restarts haproxy every few hours in HA setup

Connections to the standby instance of HAP should not be created.
Fix provides a way to check for this and restart HAP with the
intention that apps will reconnect to the right HAP instance

Change-Id: Ia556241cfccc720febbfbeb580e72c2a7ed89bc1
Closes-Bug: #1581905

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26235
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26235
Committed: http://github.org/Juniper/contrail-provisioning/commit/d7e9822481b2713b817509c2531ed71e195b9210
Submitter: Zuul
Branch: R3.2

commit d7e9822481b2713b817509c2531ed71e195b9210
Author: Ranjeet R <email address hidden>
Date: Thu Nov 17 04:22:02 2016 -0800

Fixes: HA:cmon restarts haproxy every few hours in HA setup

Connections to the standby instance of HAP should not be created.
Fix provides a way to check for this and restart HAP with the
intention that apps will reconnect to the right HAP instance

Change-Id: Iba8340ecfee1b2edd118d19ebcbe1eb919e68822
Closes-Bug: #1581905