When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:55 openc-11 haproxy[2321]: 172.23.11.41:63264 [01/Sep/2015:10:06:15.065] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0
Sep 1 10:06:57 openc-11 haproxy[2321]: 172.23.11.48:53725 [01/Sep/2015:10:04:27.079] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0
Sep 1 10:07:10 openc-11 haproxy[2321]: 172.23.11.48:65377 [01/Sep/2015:10:07:00.169] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.41:64941 [01/Sep/2015:10:06:56.159] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.48:57517 [01/Sep/2015:10:06:58.157] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0
Sep 1 10:07:38 openc-11 haproxy[2321]: 172.23.11.41:53413 [01/Sep/2015:10:06:58.160] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0
Sep 1 10:07:39 openc-11 haproxy[2321]: 172.23.11.45:62708 [01/Sep/2015:10:06:29.551] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0
Sep 1 10:07:42 openc-11 haproxy[2321]: 172.23.11.37:57043 [01/Sep/2015:10:05:22.273] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 openc-11 haproxy[2321]: 172.23.11.48:53428 [01/Sep/2015:10:07:02.176] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:51487 [01/Sep/2015:10:07:45.448] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:52017 [01/Sep/2015:10:07:43.430] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:20 openc-11 haproxy[2321]: 172.23.11.45:56311 [01/Sep/2015:10:07:40.659] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0
Sep 1 10:08:29 openc-11 haproxy[2321]: 172.23.11.41:64634 [01/Sep/2015:10:07:39.260] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:58 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail- tor-agent- 1/172.23. 10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. tor-agent- 1/172.23. 10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. 2015:10: 06:15.065] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0 2015:10: 04:27.079] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0 2015:10: 07:00.169] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0 2015:10: 06:56.159] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0 2015:10: 06:58.157] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0 2015:10: 06:58.160] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0 2015:10: 06:29.551] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0 2015:10: 05:22.273] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0 tor-agent- 1/172.23. 10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue. 2015:10: 07:02.176] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0 2015:10: 07:45.448] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0 2015:10: 07:43.430] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0 tor-agent- 1/172.23. 10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. tor-agent- 1/172.23. 10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. tor-agent- 1/172.23. 10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue. 2015:10: 07:40.659] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0 2015:10: 07:39.260] contrail- tor-agent- 1 contrail- tor-agent- 1/172.23. 10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0 tor-agent- 1/172.23. 10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. tor-agent- 1/172.23. 10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. tor-agent- 1/172.23. 10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue. tor-agent- 1/172.23. 10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:06:55 openc-11 haproxy[2321]: 172.23.11.41:63264 [01/Sep/
Sep 1 10:06:57 openc-11 haproxy[2321]: 172.23.11.48:53725 [01/Sep/
Sep 1 10:07:10 openc-11 haproxy[2321]: 172.23.11.48:65377 [01/Sep/
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.41:64941 [01/Sep/
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.48:57517 [01/Sep/
Sep 1 10:07:38 openc-11 haproxy[2321]: 172.23.11.41:53413 [01/Sep/
Sep 1 10:07:39 openc-11 haproxy[2321]: 172.23.11.45:62708 [01/Sep/
Sep 1 10:07:42 openc-11 haproxy[2321]: 172.23.11.37:57043 [01/Sep/
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:07:49 openc-11 haproxy[2321]: 172.23.11.48:53428 [01/Sep/
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:51487 [01/Sep/
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:52017 [01/Sep/
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:20 openc-11 haproxy[2321]: 172.23.11.45:56311 [01/Sep/
Sep 1 10:08:29 openc-11 haproxy[2321]: 172.23.11.41:64634 [01/Sep/
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-
Sep 1 10:08:58 openc-11 haproxy[2321]: Server contrail-