HA:cmon restarts haproxy every few hours in HA setup
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.0 |
Fix Released
|
Critical
|
Ranjeet R | |||
R3.1 |
Fix Committed
|
Critical
|
Ranjeet R | |||
R3.2 |
Fix Committed
|
Critical
|
Ranjeet R | |||
Trunk |
Fix Committed
|
Critical
|
Ranjeet R |
Bug Description
Sanju debugged the issue
Ha proxy getting restarted every few hours and the openstack services could not be connected during that brief period.VM creation fails.
[5/14/16, 1:55:39 PM] vedujoshi: Hi Sanju, on a HA enabled-cluster, we sometimes see that cmon restarts haproxy
[5/14/16, 1:55:47 PM] vedujoshi: every fews hrs infact
[5/14/16, 1:55:56 PM] vedujoshi: ex :
[5/14/16, 1:55:57 PM] vedujoshi: Sat May 14 03:36:46 IST 2016: INFO: Restarted HAP becuase of stale dips
Sat May 14 03:36:52 IST 2016: INFO: CMON is not Running
Sat May 14 03:37:52 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 1:56:26 PM] vedujoshi: Sat May 14 05:27:00 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 1:56:43 PM] vedujoshi: why so … any idea ?
[5/14/16, 1:56:47 PM] Sanju Abraham: please change /etc/contrail/
[5/14/16, 1:57:03 PM] vedujoshi: i think we did that…let me check again
[5/14/16, 1:57:26 PM] vedujoshi: yes..that is taken care
[5/14/16, 1:57:59 PM] vedujoshi: before changing the contrail-
[5/14/16, 1:58:06 PM] vedujoshi: now….it is once in few hrs
[5/14/16, 1:58:34 PM] Sanju Abraham: can you please let me know the setup, I can take a look
[5/14/16, 1:58:47 PM] vedujoshi: sure
[5/14/16, 1:59:04 PM] vedujoshi: testbed file is in nodei27:
[5/14/16, 1:59:39 PM] vedujoshi: its late in the night for you…not urgent…you can take a look in the morning
[5/14/16, 2:00:17 PM] Sanju Abraham: IP?
[5/14/16, 2:00:39 PM] vedujoshi: 10.204.217.188 is nodei27 (nodei27.
[5/14/16, 2:26:45 PM] Sanju Abraham: this can happen only if there are any connections on a LB without the VIP. The last occurrence of this on nodei35 was
[5/14/16, 2:27:07 PM] Sanju Abraham: Sat May 14 05:51:30 IST 2016: INFO: Restarted HAP becuase of stale dips
[5/14/16, 2:27:41 PM] Sanju Abraham: current time : Sat May 14 14:27:11
[5/14/16, 2:28:30 PM] Sanju Abraham: there could have some connections that were trying to use this instance of LB and hence the monitoring job restart HAP.
[5/15/16, 8:29:32 AM] Sanju Abraham: please let me know the bugID
[5/15/16, 8:29:59 AM] Sandip Dey: i have not raised yet…let me raise it
information type: | Proprietary → Public |
Review in progress for https:/ /review. opencontrail. org/20619
Submitter: Sanju (<email address hidden>)