| 2015-09-01 01:21:44 |
Daisuke Nakajima |
bug |
|
|
added bug |
| 2015-09-01 01:21:53 |
Daisuke Nakajima |
information type |
Proprietary |
Public |
|
| 2015-09-01 01:23:43 |
Daisuke Nakajima |
description |
When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:38 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:52 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:55 openc-11 haproxy[2321]: 172.23.11.41:63264 [01/Sep/2015:10:06:15.065] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0
Sep 1 10:06:57 openc-11 haproxy[2321]: 172.23.11.48:53725 [01/Sep/2015:10:04:27.079] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0
Sep 1 10:07:10 openc-11 haproxy[2321]: 172.23.11.48:65377 [01/Sep/2015:10:07:00.169] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.41:64941 [01/Sep/2015:10:06:56.159] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0
Sep 1 10:07:11 openc-11 haproxy[2321]: 172.23.11.48:57517 [01/Sep/2015:10:06:58.157] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0
Sep 1 10:07:38 openc-11 haproxy[2321]: 172.23.11.41:53413 [01/Sep/2015:10:06:58.160] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0
Sep 1 10:07:39 openc-11 haproxy[2321]: 172.23.11.45:62708 [01/Sep/2015:10:06:29.551] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0
Sep 1 10:07:42 openc-11 haproxy[2321]: 172.23.11.37:57043 [01/Sep/2015:10:05:22.273] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 openc-11 haproxy[2321]: 172.23.11.48:53428 [01/Sep/2015:10:07:02.176] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:51487 [01/Sep/2015:10:07:45.448] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0
Sep 1 10:07:53 openc-11 haproxy[2321]: 172.23.11.37:52017 [01/Sep/2015:10:07:43.430] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:07:57 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:12 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:20 openc-11 haproxy[2321]: 172.23.11.45:56311 [01/Sep/2015:10:07:40.659] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0
Sep 1 10:08:29 openc-11 haproxy[2321]: 172.23.11.41:64634 [01/Sep/2015:10:07:39.260] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:36 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:49 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:58 openc-11 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. |
When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:55 contrail-1 haproxy[2321]: 172.23.11.41:63264 [01/Sep/2015:10:06:15.065] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0
Sep 1 10:06:57 contrail-1 haproxy[2321]: 172.23.11.48:53725 [01/Sep/2015:10:04:27.079] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0
Sep 1 10:07:10 contrail-1 haproxy[2321]: 172.23.11.48:65377 [01/Sep/2015:10:07:00.169] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.41:64941 [01/Sep/2015:10:06:56.159] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.48:57517 [01/Sep/2015:10:06:58.157] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0
Sep 1 10:07:38 contrail-1 haproxy[2321]: 172.23.11.41:53413 [01/Sep/2015:10:06:58.160] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0
Sep 1 10:07:39 contrail-1 haproxy[2321]: 172.23.11.45:62708 [01/Sep/2015:10:06:29.551] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0
Sep 1 10:07:42 contrail-1 haproxy[2321]: 172.23.11.37:57043 [01/Sep/2015:10:05:22.273] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: 172.23.11.48:53428 [01/Sep/2015:10:07:02.176] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:51487 [01/Sep/2015:10:07:45.448] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:52017 [01/Sep/2015:10:07:43.430] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:20 contrail-1 haproxy[2321]: 172.23.11.45:56311 [01/Sep/2015:10:07:40.659] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0
Sep 1 10:08:29 contrail-1 haproxy[2321]: 172.23.11.41:64634 [01/Sep/2015:10:07:39.260] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:58 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. |
|
| 2015-09-01 03:17:00 |
Daisuke Nakajima |
description |
When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:55 contrail-1 haproxy[2321]: 172.23.11.41:63264 [01/Sep/2015:10:06:15.065] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0
Sep 1 10:06:57 contrail-1 haproxy[2321]: 172.23.11.48:53725 [01/Sep/2015:10:04:27.079] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0
Sep 1 10:07:10 contrail-1 haproxy[2321]: 172.23.11.48:65377 [01/Sep/2015:10:07:00.169] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.41:64941 [01/Sep/2015:10:06:56.159] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.48:57517 [01/Sep/2015:10:06:58.157] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0
Sep 1 10:07:38 contrail-1 haproxy[2321]: 172.23.11.41:53413 [01/Sep/2015:10:06:58.160] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0
Sep 1 10:07:39 contrail-1 haproxy[2321]: 172.23.11.45:62708 [01/Sep/2015:10:06:29.551] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0
Sep 1 10:07:42 contrail-1 haproxy[2321]: 172.23.11.37:57043 [01/Sep/2015:10:05:22.273] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: 172.23.11.48:53428 [01/Sep/2015:10:07:02.176] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:51487 [01/Sep/2015:10:07:45.448] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:52017 [01/Sep/2015:10:07:43.430] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:20 contrail-1 haproxy[2321]: 172.23.11.45:56311 [01/Sep/2015:10:07:40.659] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0
Sep 1 10:08:29 contrail-1 haproxy[2321]: 172.23.11.41:64634 [01/Sep/2015:10:07:39.260] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:58 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. |
When ToR-Agent is busy status, ToR-agent cannot reply keepalive to HAproxy, then HAproxt detects Active ToR-agent as Down status. So, HAproxy change its connection to other.
As a result, QFX traffic will be unstable and some traffic might be dropped.
Also, some Remote MAC address are not stored to QFX.
Typically, ToR-agent gets many MAC address at same time due to OVSDB session failure, ToR Agent consume high CPU. Then the issue will be seen.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:38 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:52 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 1000ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:06:55 contrail-1 haproxy[2321]: 172.23.11.41:63264 [01/Sep/2015:10:06:15.065] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40078 2360451 -- 582/3/3/1/0 0/0
Sep 1 10:06:57 contrail-1 haproxy[2321]: 172.23.11.48:53725 [01/Sep/2015:10:04:27.079] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/150064 9566 -- 582/3/3/1/0 0/0
Sep 1 10:07:10 contrail-1 haproxy[2321]: 172.23.11.48:65377 [01/Sep/2015:10:07:00.169] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/10004/10414 1500 -- 585/6/6/2/2 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.41:64941 [01/Sep/2015:10:06:56.159] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/15003/15003 1500 -- 584/5/5/1/3 0/0
Sep 1 10:07:11 contrail-1 haproxy[2321]: 172.23.11.48:57517 [01/Sep/2015:10:06:58.157] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/13005/13005 1500 -- 583/4/4/0/2 0/0
Sep 1 10:07:38 contrail-1 haproxy[2321]: 172.23.11.41:53413 [01/Sep/2015:10:06:58.160] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/40076 2637916 -- 580/3/3/3/0 0/0
Sep 1 10:07:39 contrail-1 haproxy[2321]: 172.23.11.45:62708 [01/Sep/2015:10:06:29.551] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/70088 7611 -- 581/3/3/2/0 0/0
Sep 1 10:07:42 contrail-1 haproxy[2321]: 172.23.11.37:57043 [01/Sep/2015:10:05:22.273] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/140147 369147 -- 580/3/3/1/0 0/0
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 1 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:07:49 contrail-1 haproxy[2321]: 172.23.11.48:53428 [01/Sep/2015:10:07:02.176] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/0/46893 8268 CD 585/6/6/2/0 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:51487 [01/Sep/2015:10:07:45.448] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/8003/8004 1502 -- 584/5/5/1/1 0/0
Sep 1 10:07:53 contrail-1 haproxy[2321]: 172.23.11.37:52017 [01/Sep/2015:10:07:43.430] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.197 1/999/10085 1502 -- 583/4/4/0/0 0/0
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:07:57 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:12 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:20 contrail-1 haproxy[2321]: 172.23.11.45:56311 [01/Sep/2015:10:07:40.659] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/40083 2329151 -- 582/3/3/3/0 0/0
Sep 1 10:08:29 contrail-1 haproxy[2321]: 172.23.11.41:64634 [01/Sep/2015:10:07:39.260] contrail-tor-agent-1 contrail-tor-agent-1/172.23.10.196 1/0/50058 2171454 -- 583/4/4/2/0 0/0
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:36 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.196 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:49 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Sep 1 10:08:58 contrail-1 haproxy[2321]: Server contrail-tor-agent-1/172.23.10.197 is UP, reason: Layer4 check passed, check duration: 1001ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. |
|
| 2015-09-01 16:38:08 |
Daisuke Nakajima |
tags |
bmc |
bmc customer |
|
| 2015-09-01 19:08:06 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/r2.20 |
|
| 2015-09-01 19:08:06 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/r2.20 |
|
| 2015-09-01 19:08:06 |
Ashish Ranjan |
nominated for series |
|
juniperopenstack/trunk |
|
| 2015-09-01 19:08:06 |
Ashish Ranjan |
bug task added |
|
juniperopenstack/trunk |
|
| 2015-09-01 19:08:18 |
Ashish Ranjan |
juniperopenstack/r2.20: assignee |
|
Hari Prasad Killi (haripk) |
|
| 2015-09-01 19:08:24 |
Ashish Ranjan |
juniperopenstack/trunk: assignee |
|
Hari Prasad Killi (haripk) |
|
| 2015-09-01 19:08:28 |
Ashish Ranjan |
juniperopenstack/trunk: milestone |
|
r2.30-fcs |
|
| 2015-09-01 19:08:37 |
Ashish Ranjan |
juniperopenstack/r2.20: milestone |
|
r2.22 |
|
| 2015-09-01 19:08:39 |
Ashish Ranjan |
juniperopenstack/r2.20: importance |
Undecided |
Medium |
|
| 2015-09-01 19:08:41 |
Ashish Ranjan |
juniperopenstack/trunk: importance |
Undecided |
Medium |
|
| 2015-09-01 19:08:51 |
Ashish Ranjan |
tags |
bmc customer |
bms customer |
|
| 2015-09-02 04:04:00 |
Hari Prasad Killi |
tags |
bms customer |
bms customer vrouter |
|
| 2015-09-10 09:46:48 |
Hari Prasad Killi |
tags |
bms customer vrouter |
bms customer releasenote vrouter |
|
| 2015-10-13 06:22:40 |
Daisuke Nakajima |
attachment added |
|
core.62292.gz https://bugs.launchpad.net/juniperopenstack/+bug/1490788/+attachment/4493124/+files/core.62292.gz |
|
| 2015-11-25 13:13:19 |
Hari Prasad Killi |
marked as duplicate |
|
1481606 |
|
