Activity log for bug #1692795

Date Who What changed Old value New value Message
2017-05-23 05:46:02 Sandeep Sridhar bug added bug
2017-05-23 05:47:35 Sandeep Sridhar information type Proprietary Public
2017-05-23 05:48:07 Sandeep Sridhar tags bms vrouter
2017-05-23 06:17:03 vivekananda shenoy juniperopenstack: assignee Manish Singh (manishs)
2017-05-23 06:17:10 vivekananda shenoy juniperopenstack: importance Undecided Critical
2017-05-23 06:17:16 vivekananda shenoy juniperopenstack: milestone r3.1.3.0
2017-05-23 06:17:27 vivekananda shenoy tags bms vrouter bms nttc vrouter
2017-05-23 22:51:57 Jeba Paulaiyan nominated for series juniperopenstack/r3.1
2017-05-23 22:51:57 Jeba Paulaiyan bug task added juniperopenstack/r3.1
2017-05-23 22:51:57 Jeba Paulaiyan nominated for series juniperopenstack/r3.2
2017-05-23 22:51:57 Jeba Paulaiyan bug task added juniperopenstack/r3.2
2017-05-23 22:51:57 Jeba Paulaiyan nominated for series juniperopenstack/trunk
2017-05-23 22:51:57 Jeba Paulaiyan bug task added juniperopenstack/trunk
2017-05-23 22:51:57 Jeba Paulaiyan nominated for series juniperopenstack/r4.0
2017-05-23 22:51:57 Jeba Paulaiyan bug task added juniperopenstack/r4.0
2017-05-23 22:52:05 Jeba Paulaiyan juniperopenstack/r4.0: importance Undecided High
2017-05-23 22:52:07 Jeba Paulaiyan juniperopenstack/r3.2: importance Undecided High
2017-05-23 22:52:09 Jeba Paulaiyan juniperopenstack/r3.1: importance Undecided High
2017-05-23 22:52:18 Jeba Paulaiyan juniperopenstack/r3.1: milestone r3.1.3.0
2017-05-23 22:52:22 Jeba Paulaiyan juniperopenstack/trunk: milestone r3.1.3.0 r4.1.0.0-fcs
2017-05-23 22:52:34 Jeba Paulaiyan juniperopenstack/r3.1: assignee Manish Singh (manishs)
2017-05-23 22:52:54 Jeba Paulaiyan juniperopenstack/r3.2: assignee Manish Singh (manishs)
2017-05-23 22:53:04 Jeba Paulaiyan juniperopenstack/r4.0: assignee Manish Singh (manishs)
2017-05-30 09:24:30 OpenContrail Admin juniperopenstack/r3.1: importance High Critical
2017-05-30 09:24:30 OpenContrail Admin juniperopenstack/r3.1: status New In Progress
2017-05-31 10:19:07 OpenContrail Admin juniperopenstack/r3.1: status In Progress Fix Committed
2017-05-31 10:30:54 OpenContrail Admin juniperopenstack/r3.2: importance High Critical
2017-05-31 10:30:54 OpenContrail Admin juniperopenstack/r3.2: status New In Progress
2017-05-31 21:44:42 OpenContrail Admin juniperopenstack/r3.2: status In Progress Fix Committed
2017-05-31 21:44:44 OpenContrail Admin juniperopenstack/r3.2: milestone r3.2.4.0
2017-06-07 19:51:28 OpenContrail Admin juniperopenstack/r3.1: status Fix Committed In Progress
2017-06-09 03:31:17 OpenContrail Admin juniperopenstack/r3.1: status In Progress Fix Committed
2017-06-09 09:42:35 OpenContrail Admin juniperopenstack/r3.2: status Fix Committed In Progress
2017-06-16 07:04:38 OpenContrail Admin juniperopenstack/r3.2: status In Progress Fix Committed
2017-07-17 06:54:28 OpenContrail Admin juniperopenstack/trunk: status New In Progress
2017-07-17 07:00:55 OpenContrail Admin juniperopenstack/r4.0: importance High Critical
2017-07-17 07:00:55 OpenContrail Admin juniperopenstack/r4.0: status New In Progress
2017-07-20 13:51:56 OpenContrail Admin juniperopenstack/r4.0: status In Progress Fix Committed
2017-07-20 13:51:57 OpenContrail Admin juniperopenstack/r4.0: milestone r4.0.1.0
2017-07-21 07:35:52 OpenContrail Admin juniperopenstack/trunk: status In Progress Fix Committed
2017-12-12 05:50:32 Jeba Paulaiyan description There were issues reported with QFX missing from the BUM tree before. Manish investigated and he pointed to a corrupted pointer in the tor-agent core collected from problematic setup. Earlier, we suspected some issue with the ISSU upgrade (as ISSU procedure was used to upgrade Contrail from 2.21.2 to 3.1.3.0-72). The new occurence is reported on a clean installation of 3.1.3.0-72 build. Here is JTAC's analysis: VNIs being tested. 1. 4338 2. 3559 IP addresses as follows: (TSN) openc-34 172.23.10.201 (TSN) openc-35 172.23.10.202 (QFX6) 172.23.11.48 (QFX23) 172.23.11.49 QFX6 is being served by contrail-tor-agent-6 on openc-34 QFX23 is being served by contrail-tor-agent-23 on openc-35. Please see below: root@openc-34:~# contrail-status | grep QFX contrail-tor-agent-1 active (ToR:QFX1 connection up) contrail-tor-agent-11 active (ToR:QFX11 connection down) contrail-tor-agent-23 active (ToR:QFX23 connection down) contrail-tor-agent-6 active (ToR:QFX6 connection up) <<<<<<<<<<<<<<<<<<<<< root@openc-34:~# root@openc-35:~# contrail-status | grep QFX contrail-tor-agent-1 active (ToR:QFX1 connection down) contrail-tor-agent-11 active (ToR:QFX11 connection up) contrail-tor-agent-23 active (ToR:QFX23 connection up) <<<<<<<<<<<<<<<<<<<<< contrail-tor-agent-6 active (ToR:QFX6 connection down) Test Case 1: (VNI 4338) ============ (30:06:23:00:03:42) => [QFX6 ae7.2834] => [openc-34] => [openc-35] => [QFX23 ae1.3834] => (30:23:06:00:03:42) Result: On openc-34, QFX6 is missing && openc-35 is present. On openc-35, QFX23 is missing && openc-34 is present. <<< BUM traffic broken completely >>> Test Case 2: (VNI 3559) ============ (30:06:23:00:00:37) => [QFX6 ae7.2055] => [openc-34] => [openc-35] => [QFX23 ae1.3055] => (30:23:06:00:00:37) Result: On openc-34, QFX6 is missing && openc-35 present. On openc-35, QFX23 is present && openc-34 is also present. <<< BUM Traffic one way is blocked which is openc-35 ==> openc-34 >>> All cores can be found here: ssh root@10.219.48.123 password:Jtaclab123 /home/ssandeep/2017-0424-0113/NewLogsMay23/ Greetings, Sandeep. There were issues reported with QFX missing from the BUM tree before. Manish investigated and he pointed to a corrupted pointer in the tor-agent core collected from problematic setup. Earlier, we suspected some issue with the ISSU upgrade (as ISSU procedure was used to upgrade Contrail from 2.21.2 to 3.1.3.0-72). The new occurence is reported on a clean installation of 3.1.3.0-72 build. Here is JTAC's analysis: VNIs being tested. 1. 4338 2. 3559 IP addresses as follows: (TSN) openc-34 172.23.10.201 (TSN) openc-35 172.23.10.202 (QFX6) 172.23.11.48 (QFX23) 172.23.11.49 QFX6 is being served by contrail-tor-agent-6 on openc-34 QFX23 is being served by contrail-tor-agent-23 on openc-35. Please see below: root@openc-34:~# contrail-status | grep QFX contrail-tor-agent-1 active (ToR:QFX1 connection up) contrail-tor-agent-11 active (ToR:QFX11 connection down) contrail-tor-agent-23 active (ToR:QFX23 connection down) contrail-tor-agent-6 active (ToR:QFX6 connection up) <<<<<<<<<<<<<<<<<<<<< root@openc-34:~# root@openc-35:~# contrail-status | grep QFX contrail-tor-agent-1 active (ToR:QFX1 connection down) contrail-tor-agent-11 active (ToR:QFX11 connection up) contrail-tor-agent-23 active (ToR:QFX23 connection up) <<<<<<<<<<<<<<<<<<<<< contrail-tor-agent-6 active (ToR:QFX6 connection down) Test Case 1: (VNI 4338) ============ (30:06:23:00:03:42) => [QFX6 ae7.2834] => [openc-34] => [openc-35] => [QFX23 ae1.3834] => (30:23:06:00:03:42) Result: On openc-34, QFX6 is missing && openc-35 is present.         On openc-35, QFX23 is missing && openc-34 is present. <<< BUM traffic broken completely >>> Test Case 2: (VNI 3559) ============ (30:06:23:00:00:37) => [QFX6 ae7.2055] => [openc-34] => [openc-35] => [QFX23 ae1.3055] => (30:23:06:00:00:37) Result: On openc-34, QFX6 is missing && openc-35 present.         On openc-35, QFX23 is present && openc-34 is also present. <<< BUM Traffic one way is blocked which is openc-35 ==> openc-34 >>> All cores can be found here: /home/ssandeep/2017-0424-0113/NewLogsMay23/ Greetings, Sandeep.