EVPN routes from CNs are withdrawn without any change on CN or Opus

Bug #1474111 reported by Sharon Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Invalid
Low
Nischal Sheth
Trunk
Invalid
Low
Nischal Sheth

Bug Description

I was debugging the test bed since on one of Opus, I only saw the routes from VN102 network (vni 5) and did not see the routes from VN101 network (vni 4) which are supposed to be received based on the config.

At the beginning, I only saw the routes from VN102 network (vni 5), I opened another telnet session to check evpn.log, stuff, and also on Contrail check the VNs to make sure vnis are matched, and went to VM console to ping BMS server on the same network, the ping did not go through to BMS connected to problematic Opus (sdn-st2-qfx48s-a).

When I checked back the routes received from Contrail, found some routes from VN101 are received and mean while, routes from other networks are received as well which are not supposed to be received.

After a while , found that the routes received from Contrail becomes 0, but the other Opus still receiving the routes from Contrail.

sdn-st2-qfx48s-a is the opus showing the problem

regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      55 55 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8751 8730 0 1 2d 18:16:42 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8738 8720 0 0 2d 18:11:55 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8746 8731 0 1 2d 18:16:46 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 10583 8728 0 1 2d 18:16:38 Establ
  bgp.evpn.0: 0/0/0/0
  default-switch.evpn.0: 0/0/0/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 10351 8728 0 1 2d 18:16:38 Establ
  bgp.evpn.0: 0/0/0/0
  default-switch.evpn.0: 0/0/0/0
  __default_evpn__.evpn.0: 0/0/0/0

cli output from the beginning:

regress@sdn-st2-qfx48s-a# run show version
fpc0:
--------------------------------------------------------------------------
Hostname: sdn-st2-qfx48s-a
Model: qfx5100-48s-6q
Junos: 14.1-20150709_x141X53_vjqfd.0
JUNOS Base OS Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS Base OS boot [14.1-20150709_x141X53_vjqfd.0]
JUNOS Crypto Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS Online Documentation [14.1-20150709_x141X53_vjqfd.0]
JUNOS Kernel Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [14.1-20150709_x141X53_vjqfd.0]
JUNOS Routing Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS SDN Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS Enterprise Software Suite [14.1-20150709_x141X53_vjqfd.0]
JUNOS Web Management Platform Package [14.1-20150709_x141X53_vjqfd.0]
JUNOS py-base-i386 [14.1-20150709_x141X53_vjqfd.0]
JUNOS Host Software [14.1-20150709_x141X53_vjqfd.0]

{master:0}[edit]
regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      69 62 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8710 8689 0 1 2d 17:58:07 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8697 8679 0 0 2d 17:53:20 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8706 8691 0 1 2d 17:58:11 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 8323 8687 0 1 2d 17:58:03 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 7/7/7/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 8289 8687 0 1 2d 17:58:03 Establ
  bgp.evpn.0: 0/7/7/0
  default-switch.evpn.0: 0/7/7/0
  __default_evpn__.evpn.0: 0/0/0/0

<<<< Only received routes from VN102 -VNI 5 >>>>

regress@sdn-st2-qfx48s-a# run show route receive-protocol bgp 192.11.1.10

inet.0: 39 destinations, 40 routes (38 active, 0 holddown, 1 hidden)

:vxlan.inet.0: 15 destinations, 15 routes (14 active, 0 holddown, 1 hidden)

bgp.evpn.0: 62 destinations, 69 routes (62 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?

default-switch.evpn.0: 66 destinations, 73 routes (66 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

<<< Check the config to make sure both VNI 4 and VNI 5 are enabled for EVPN VXLAN >>>>

regress@sdn-st2-qfx48s-a# show protocols evpn
traceoptions {
    file evpn.log size 5m files 3;
    flag all;
}
vni-options {
    vni 4 {
        vrf-target export target:1:1;
    }
    vni 5 {
        vrf-target export target:2:1;
    }
    inactive: vni 7 {
        vrf-target export target:64512:8000004;
    }
    inactive: vni 8 {
        vrf-target export target:4:1;
    }
    inactive: vni 101 {
        vrf-target export target:1:1;
    }
    inactive: vni 102 {
        vrf-target export target:2:1;
    }
    inactive: vni 104 {
        vrf-target export target:4:1;
    }
    inactive: vni 105 {
        vrf-target export target:64512:8000004;
    }
    inactive: vni 16777213 {
        vrf-target export target:2:1;
    }
}
encapsulation vxlan;
extended-vni-list [ 4 5 ];

regress@sdn-st2-qfx48s-a# show switch-options
vtep-source-interface lo0.0;
route-distinguisher 64512:1010;
vrf-import imp;
vrf-target target:2:1;

regress@sdn-st2-qfx48s-a# show vlans v101
vlan-id 101;
##
## Warning: requires 'vxlan' license
##
vxlan {
    vni 4;
    ingress-node-replication;
}

{master:0}[edit]
regress@sdn-st2-qfx48s-a# show vlans v102
vlan-id 102;
##
## Warning: requires 'vxlan' license
##
vxlan {
    vni 5;
    ingress-node-replication;
}

{master:0}[edit]
regress@sdn-st2-qfx48s-a# show policy-options
policy-statement imp {
    term 1 {
        from community [ commvni200 commvni100 commvni104 commvni105 commvni106 commvni107 ];
        then accept;
    }
    term 10 {
        then reject;
    }
}
community commvni100 members target:1:1;
community commvni104 members target:4:1;
community commvni105 members target:64512:8000004;
community commvni106 members target:64512:8000006;
community commvni107 members target:64512:8000007;
community commvni200 members target:2:1;

{master:0}[edit]
regress@sdn-st2-qfx48s-a# show interfaces xe-0/0/3
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members [ v102 v101 v105 v106 v107 ];
        }
    }
}

<<<< MAC routes are not received for VN101 network >>>

regress@sdn-st2-qfx48s-a# run show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 5 entries, 5 learned
Routing instance : default-switch
   Vlan MAC MAC Logical Active
   name address flags interface source
   v101 00:00:5e:00:01:01 DR esi.1774 05:00:00:fc:00:00:00:00:04:00
   v101 00:10:94:00:00:17 D xe-0/0/3.0
   v101 00:10:94:00:00:19 DL vtep.32770 100.48.1.11
   v101 00:26:88:c8:6f:f0 D vtep.32774 100.1.1.48
   v101 ac:4b:c8:8b:9f:f0 D vtep.32771 100.1.1.24

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 7 entries, 7 learned
Routing instance : default-switch
   Vlan MAC MAC Logical Active
   name address flags interface source
   v102 00:00:5e:00:01:01 DR esi.1777 05:00:00:fc:00:00:00:00:05:00
   v102 00:10:94:00:00:18 D xe-0/0/3.0
   v102 00:26:88:c8:6f:f0 D vtep.32774 100.1.1.48
   v102 02:0d:38:9b:55:e8 D vtep.32772 192.14.1.10
   v102 02:3b:f8:19:42:cb D vtep.32772 192.14.1.10
   v102 02:9d:5a:c1:54:94 D vtep.32772 192.14.1.10
   v102 ac:4b:c8:8b:9f:f0 D vtep.32771 100.1.1.24

<<< At this point, only routes from VN102-VNI 5 are received >>>>

regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      69 62 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8721 8701 0 1 2d 18:03:17 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8708 8690 0 0 2d 17:58:30 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8717 8702 0 1 2d 18:03:21 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 8333 8698 0 1 2d 18:03:13 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 7/7/7/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 8299 8699 0 1 2d 18:03:13 Establ
  bgp.evpn.0: 0/7/7/0
  default-switch.evpn.0: 0/7/7/0
  __default_evpn__.evpn.0: 0/0/0/0

{master:0}[edit]
regress@sdn-st2-qfx48s-a# run show route receive-protocol bgp 192.11.1.10

inet.0: 39 destinations, 40 routes (38 active, 0 holddown, 1 hidden)

:vxlan.inet.0: 15 destinations, 15 routes (14 active, 0 holddown, 1 hidden)

bgp.evpn.0: 62 destinations, 69 routes (62 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?

default-switch.evpn.0: 66 destinations, 73 routes (66 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

<<< In V101 - VNI 4 network, no routes are received from Contrail -- 192.11.1.10 and 192.12.1.10 >>>>

regress@sdn-st2-qfx48s-a# run show route table bgp.evpn.0 evpn-ethernet-tag-id 4

bgp.evpn.0: 55 destinations, 55 routes (55 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:100.1.1.24:100::4::00:00:5e:00:01:01/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::ac:4b:c8:8b:9f:f0/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.48:100::4::00:00:5e:00:01:01/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:100.1.1.48:100::4::00:26:88:c8:6f:f0/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:1000:11::4::00:10:94:00:00:19/304
                   *[BGP/170] 2d 17:59:55, localpref 100, from 100.48.1.11
                      AS path: I, validation-state: unverified
                      to 10.0.0.25 via ge-0/0/20.0
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::00:00:5e:00:01:01::101.1.1.254/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::ac:4b:c8:8b:9f:f0::101.1.1.200/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.48:100::4::00:00:5e:00:01:01::101.1.1.254/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:100.1.1.48:100::4::00:26:88:c8:6f:f0::101.1.1.201/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
3:100.1.1.24:100::4::100.1.1.24/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
3:100.1.1.48:100::4::100.1.1.48/304
                   *[BGP/170] 2d 17:59:56, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
3:1000:11::4::100.48.1.11/304
                   *[BGP/170] 2d 17:59:55, localpref 100, from 100.48.1.11
                      AS path: I, validation-state: unverified
                      to 10.0.0.25 via ge-0/0/20.0
                    > to 10.0.0.21 via ge-0/0/36.0

<<< Start monitor evpn log and stop after a while >>>>>

regress@sdn-st2-qfx48s-a# run monitor start evpn.log

{master:0}[edit]
regress@sdn-st2-qfx48s-a#
*** evpn.log ***
Jul 13 19:28:04.682927 l2ald_client_mac_tx_timer_expiry: RPDF Mac timer expiry ...
Jul 13 19:28:04.682962 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:1 Addr:00:00:5e:00:01:01 SVLAN:0 L2_Domain:0 VNI:4 NHID:0 RTRID:0 ifl: idx:0 ESI:050000fc000000000400 flags:0x41 mactlv_len 114 tot_tlv_len 124

Jul 13 19:28:04.682991 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:2 Addr:00:00:5e:00:01:01 SVLAN:0 L2_Domain:0 VNI:4 NHID:0 RTRID:0 ifl: idx:0 ESI:050000fc000000000400 flags:0x41 mactlv_len 114 tot_tlv_len 242

Jul 13 19:28:04.683013 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:3 Addr:00:10:94:00:00:19 SVLAN:0 L2_Domain:0 VNI:4 NHID:0 RTRID:0 ifl: idx:0 ESI:0 flags:0x1 mactlv_len 100 tot_tlv_len 346

Jul 13 19:28:04.683034 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:4 Addr:00:26:88:c8:6f:f0 SVLAN:0 L2_Domain:0 VNI:4 NHID:0 RTRID:0 ifl: idx:0 ESI:0 flags:0x1 mactlv_len 100 tot_tlv_len 450

Jul 13 19:28:04.683096 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:5 Addr:ac:4b:c8:8b:9f:f0 SVLAN:0 L2_Domain:0 VNI:4 NHID:0 RTRID:0 ifl: idx:0 ESI:0 flags:0x1 mactlv_len 100 tot_tlv_len 554

Jul 13 19:28:04.683125 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:6 Addr:00:00:5e:00:01:01 SVLAN:0 L2_Domain:0 VNI:5 NHID:0 RTRID:0 ifl: idx:0 ESI:050000fc000000000500 flags:0x41 mactlv_len 114 tot_tlv_len 672

Jul 13 19:28:04.683150 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:7 Addr:00:00:5e:00:01:01 SVLAN:0 L2_Domain:0 VNI:5 NHID:0 RTRID:0 ifl: idx:0 ESI:050000fc000000000500 flags:0x41 mactlv_len 114 tot_tlv_len 790

Jul 13 19:28:04.683170 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:8 Addr:00:26:88:c8:6f:f0 SVLAN:0 L2_Domain:0 VNI:5 NHID:0 RTRID:0 ifl: idx:0 ESI:0 flags:0x1 mactlv_len 100 tot_tlv_len 894

Jul 13 19:28:04.683191 l2ald_client_dequeue_macs_to_l2ald:Processing cmd:1 for MAC#:9 Addr:ac:4b:c8:8b:9f:f0 SVLAN:0 L2_Domain:0 VNI:5 NHID:0 RTRID:0 ifl: idx:0 ESI:0 flags:0x1 mactlv_len 100 tot_tlv_len 998

Jul 13 19:28:04.683204 l2ald_client_dequeue_macs_to_l2ald:SENDING MACs count 9, (Req Macs:998)
Jul 13 19:28:04.683233 task_timer_uset: timer RPDF L2ALD IPC./var/run/rpdf_control_RPDF L2ALD IPC Keepalive <Touched> set to offset 1:00 at 19:29:04.683204
Jul 13 19:28:04.683249 l2ald_client_send: Case 1

Jul 13 19:28:04.683266 task_timer_reset: reset RPDF L2ALD IPC./var/run/rpdf_control_RPDF L2ALD Mac TX
Jul 13 19:28:04.683481 task_process_events: recv ready for RPDF L2ALD IPC./var/run/rpdf_control
Jul 13 19:28:04.683492 l2ald_client_receive: RECEIVED packet from L2ALD
Jul 13 19:28:04.683515 task_process_events: recv ready for RPDF L2ALD IPC./var/run/rpdf_control
Jul 13 19:28:04.683524 l2ald_client_receive: RECEIVED packet from L2ALD
Jul 13 19:28:04.683546 task_timer_uset: timer RPDF L2ALD IPC./var/run/rpdf_control_RPDF L2ALD IPC hold <Touched> set to offset 3:00 at 19:31:04.683524
Jul 13 19:28:04.683556 l2ald_client_prelim_parse_packet: Received pkt_type 6

Jul 13 19:28:04.683568 l2ald_client_prelim_parse_packet: Default case; calling process for pkt_type 6

Jul 13 19:28:04.683580 l2ald_client_process_packet: Entered to parse incoming packet pkt_type 6,len=228
Jul 13 19:28:04.683588 l2ald_client_process_packet:Finished parsing TLVs

Jul 13 19:28:04.683607 task_timer_uset: timer RPDF L2ALD IPC./var/run/rpdf_control_RPDF L2ALD IPC hold <Touched> set to offset 3:00 at 19:31:04.683588
Jul 13 19:28:04.683616 l2ald_client_process_packet:Processing Mac UPD for 2 Macs

Jul 13 19:28:04.683628 l2ald_client_process_packet: Packet info known so far tot_tlv_len = 222, mil_len = 4
Jul 13 19:28:04.683661 l2ald_client_process_mac_route:Processing cmd:1 for MAC:00:10:94:00:00:17 SVLAN:0 L2_Domain:101 VNI:4 NHID:0 INEID:0 ifl:556 rtt:6 ESI:0 BD id:171798691883 flags:0x2
Jul 13 19:28:04.683685 evpn_mac_msg_receive_from_l2ald:4994 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 3, Flags: 0x14800] Received MAC add for local MAC 00:10:94:00:00:17 with VNI 4 VLAN 101, interface xe-0/0/3.0, flags 0x2, timestamp 0x55a06f65 from L2ALD
Jul 13 19:28:04.683705 evpn_instance_local_esi_lookup:1376 EVPN Instance ESI default-switch::xe-0/0/3.0 [Intfs: 1, PEs: 0, MACs: 2, Refcount: 2] ESI NULL ESI Found during Local MAC update
Jul 13 19:28:04.683722 evpn_macdb_local_mac_update:4495 EVPN MAC ESI default-switch::4::00:10:94:00:00:17::xe-0/0/3.0 [Active: yes, Time Stamp: 1436577637, Flags: 0xa <Local-MAC-Only Remote-Adv-Allowed>] For process during local mac update
Jul 13 19:28:04.683738 evpn_macdb_local_mac_update:4554 EVPN MAC ESI default-switch::4::00:10:94:00:00:17::xe-0/0/3.0 [Active: yes, Time Stamp: 1436577637, Flags: 0xa <Local-MAC-Only Remote-Adv-Allowed>] Current active during local mac update
Jul 13 19:28:04.683752 evpn_macdb_local_mac_attribute_update:4321 EVPN MAC default-switch::4::00:10:94:00:00:17 [Flags: 0x0] No catastrohic changes to MAC parameters
Jul 13 19:28:04.683765 evpn_macdb_local_mac_can_create_adv_rt:2850 EVPN MAC default-switch::4::00:10:94:00:00:17 [Flags: 0x0] Advertisment route already exists
Jul 13 19:28:04.683783 l2ald_client_process_mac_route:Processing cmd:1 for MAC:00:10:94:00:00:18 SVLAN:0 L2_Domain:102 VNI:5 NHID:0 INEID:0 ifl:556 rtt:6 ESI:0 BD id:193273528368 flags:0x2
Jul 13 19:28:04.683800 evpn_mac_msg_receive_from_l2ald:4994 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 3, Flags: 0x14800] Received MAC add for local MAC 00:10:94:00:00:18 with VNI 5 VLAN 102, interface xe-0/0/3.0, flags 0x2, timestamp 0x55a06f65 from L2ALD
Jul 13 19:28:04.683813 evpn_instance_local_esi_lookup:1376 EVPN Instance ESI default-switch::xe-0/0/3.0 [Intfs: 1, PEs: 0, MACs: 2, Refcount: 2] ESI NULL ESI Found during Local MAC update
Jul 13 19:28:04.683828 evpn_macdb_local_mac_update:4495 EVPN MAC ESI default-switch::5::00:10:94:00:00:18::xe-0/0/3.0 [Active: yes, Time Stamp: 1436577637, Flags: 0xa <Local-MAC-Only Remote-Adv-Allowed>] For process during local mac update
Jul 13 19:28:04.683842 evpn_macdb_local_mac_update:4554 EVPN MAC ESI default-switch::5::00:10:94:00:00:18::xe-0/0/3.0 [Active: yes, Time Stamp: 1436577637, Flags: 0xa <Local-MAC-Only Remote-Adv-Allowed>] Current active during local mac update
Jul 13 19:28:04.683854 evpn_macdb_local_mac_attribute_update:4321 EVPN MAC default-switch::5::00:10:94:00:00:18 [Flags: 0x0] No catastrohic changes to MAC parameters
Jul 13 19:28:04.683867 evpn_macdb_local_mac_can_create_adv_rt:2850 EVPN MAC default-switch::5::00:10:94:00:00:18 [Flags: 0x0] Advertisment route already exists
Jul 13 19:28:10.017052 evpn_adv_rt_flash:3024 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 3, Flags: 0x14800] Flash call for EVPN from default-switch.EVPN.0
Jul 13 19:28:10.017069 evpn_adv_from_sockaddr:1314 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 3, Flags: 0x14800] Received MAC advertisement route (type 2) from BGP
Jul 13 19:28:10.017095 evpn_process_mac_rt:2095 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 3, Flags: 0x14800] Processing ADD for MAC 02:3b:f8:19:42:cb from 192.14.1.10 with ESI 0, L2 domain 5, label 0, IPv4 address: 102.1.1.4 default_gw 0
Jul 13 19:28:10.017112 evpn_mes_get_create:1532 EVPN MES default-switch::192.14.1.10 [AD/MAC/MAC+IP/IM/ESI routes: 0/0/0/0/0] Created
Jul 13 19:28:10.017124 l2ald_client_enqueue_mac: queuing mac msg enq mac:00:00:00:00:00:00 cmd:14
Jul 13 19:28:10.017135 l2ald_client_enqueue_mac: mac_info 0x92f3738, mac_list 0x92f3738 mac_entry 0x9760000 txq_count 1
Jul 13 19:28:10.017165 task_timer_uset: timer RPDF L2ALD IPC./var/run/rpdf_control_RPDF L2ALD Mac TX <Touched> set to offset 0.250000 at 19:28:10.267135
Jul 13 19:28:10.017184 evpn_tun_msg_send_to_l2ald:1261 EVPN instance default-switch [VS, Refcount: 4, Intfs: 2 (2 up), IRBs: 0 (0 up), Remote PEs: 4, Flags: 0x14800] Sent VTEP Tunnel Create with VLAN 0 VNI 0, RTT 6, remote addr 192.14.1.10 BD idx 0x0000000000000000 to L2ALD
Jul 13 19:28:10.017200 evpn_mes_increment_mac_count:2038 EVPN MES default-switch::192.14.1.10 [AD/MAC/MAC+IP/IM/ESI routes: 0/0/1/0/0] MAC+IP count incremented to 1
Jul 13 19:28:10.017216 evpn_mes_mac_lock:1838 EVPN MES default-switch::192.14.1.10 [AD/MAC/MAC+IP/IM/ESI routes: 0/0/1/0/0] MAC refcount incremented to 1
Jul 13 19:28:10.017232 task_timer_ucreate: created timer default-switch-evpn_EVIE_L2ALD_TIMER flags <>
Jul 13 19:28:10.017246 evpn_instance_esi_create:1582 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 0, MACs: 0, Refcount: 0] Created and source added for Remote MAC receive
Jul 13 19:28:10.017261 evpn_esi_update_remote_mes:680 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 0, MACs: 0, Refcount: 0] event for reserved esi EVI_ESI_RESERVED_EGRESS_ROUTE_ADD
Jul 13 19:28:10.017274 evpn_esi_update_remote_mes:695 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 0, MACs: 0, Refcount: 0] MAC Label 0 event 6
Jul 13 19:28:10.017287 evpn_mes_instance_esi_lock:1820 EVPN MES default-switch::192.14.1.10 [AD/MAC/MAC+IP/IM/ESI routes: 0/0/1/0/0] Instance ESI refcount incremented to 1
Jul 13 19:28:10.017304 evpn_esi_add_remote_mes:801 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 1, MACs: 0, Refcount: 0] remote PE : PE 192.14.1.10 mode 0 flags 1 mac_label 0 aliasing label 0 event :EVI_ESI_RESERVED_EGRESS_ROUTE_ADD
Jul 13 19:28:10.017322 evpn_instance_esi_create:1600 EVPN instance ESI: default-switch 00:00:00:00:00:00:00:00:00:00, Flags:Reserved Remote, PE: 192.14.1.10 Source created for Remote MAC receive (source flags: 0x1)
Jul 13 19:28:10.017337 evpn_instance_esi_lock:1820 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 1, MACs: 0, Refcount: 1] instance esi ref count 1
Jul 13 19:28:10.017365 evpn_macdb_remote_mac_get_create_esi:3706 EVPN MAC ESI default-switch::5::02:3b:f8:19:42:cb::192.14.1.10 [Active: no, Time Stamp: 1436815690, Flags: 0x0] Allocated for remote source
Jul 13 19:28:10.017386 evpn_instance_esi_enqueue_macdb_entry:1452 EVPN Instance ESI default-switch::192.14.1.10 [Intfs: 0, PEs: 1, MACs: 1, Refcount: 1] MAC 02:3b

{master:0}[edit]
regress@sdn-st2-qfx48s-a# run monitor stop

At this point, start a new telnet session to check evpn log

After that, check bgp summary, found Opus received more routes from Contrail, it received not only vni 4 and vni 5 network, also received routes from vni 7 network

regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      87 71 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8736 8715 0 1 2d 18:09:48 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8723 8705 0 0 2d 18:05:01 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8731 8716 0 1 2d 18:09:52 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 9451 8713 0 1 2d 18:09:44 Establ
  bgp.evpn.0: 16/16/16/0
  default-switch.evpn.0: 16/16/16/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 9310 8713 0 1 2d 18:09:44 Establ
  bgp.evpn.0: 0/16/16/0
  default-switch.evpn.0: 0/16/16/0
  __default_evpn__.evpn.0: 0/0/0/0

regress@sdn-st2-qfx48s-a# run show route table bgp.evpn.0 evpn-ethernet-tag-id 4

bgp.evpn.0: 71 destinations, 87 routes (71 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:192.14.1.10:3::4::02:df:85:8e:84:bf/304
                   *[BGP/170] 00:00:59, localpref 100, from 192.11.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
                    [BGP/170] 00:00:59, localpref 100, from 192.12.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::00:00:5e:00:01:01/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::ac:4b:c8:8b:9f:f0/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.48:100::4::00:00:5e:00:01:01/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:100.1.1.48:100::4::00:26:88:c8:6f:f0/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:1000:11::4::00:10:94:00:00:19/304
                   *[BGP/170] 2d 18:04:39, localpref 100, from 100.48.1.11
                      AS path: I, validation-state: unverified
                      to 10.0.0.25 via ge-0/0/20.0
                    > to 10.0.0.21 via ge-0/0/36.0
2:192.14.1.10:3::4::02:df:85:8e:84:bf::101.1.1.3/304
                   *[BGP/170] 00:00:59, localpref 200, from 192.11.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
                    [BGP/170] 00:00:59, localpref 200, from 192.12.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::00:00:5e:00:01:01::101.1.1.254/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.24:100::4::ac:4b:c8:8b:9f:f0::101.1.1.200/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
2:100.1.1.48:100::4::00:00:5e:00:01:01::101.1.1.254/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
2:100.1.1.48:100::4::00:26:88:c8:6f:f0::101.1.1.201/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
3:192.14.1.10:3::4::192.14.1.10/304
                   *[BGP/170] 00:00:59, localpref 100, from 192.11.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
                    [BGP/170] 00:00:59, localpref 100, from 192.12.1.10
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
3:100.1.1.24:100::4::100.1.1.24/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.24
                      AS path: I, validation-state: unverified
                    > to 10.0.0.21 via ge-0/0/36.0
3:100.1.1.48:100::4::100.1.1.48/304
                   *[BGP/170] 2d 18:04:40, localpref 100, from 100.1.1.48
                      AS path: I, validation-state: unverified
                    > to 10.0.0.25 via ge-0/0/20.0
3:1000:11::4::100.48.1.11/304
                   *[BGP/170] 2d 18:04:39, localpref 100, from 100.48.1.11
                      AS path: I, validation-state: unverified
                      to 10.0.0.25 via ge-0/0/20.0
                    > to 10.0.0.21 via ge-0/0/36.0

<<< but routes from VNI 4 network is less than expected >>>>
{master:0}[edit]
regress@sdn-st2-qfx48s-a# run show route table bgp.evpn.0 evpn-ethernet-tag-id 4 |grep 192
2:192.14.1.10:3::4::02:df:85:8e:84:bf/304
                   *[BGP/170] 00:01:16, localpref 100, from 192.11.1.10
                    [BGP/170] 00:01:16, localpref 100, from 192.12.1.10
2:192.14.1.10:3::4::02:df:85:8e:84:bf::101.1.1.3/304
                   *[BGP/170] 00:01:16, localpref 200, from 192.11.1.10
                    [BGP/170] 00:01:16, localpref 200, from 192.12.1.10
3:192.14.1.10:3::4::192.14.1.10/304
                   *[BGP/170] 00:01:16, localpref 100, from 192.11.1.10
                    [BGP/170] 00:01:16, localpref 100, from 192.12.1.10

regress@sdn-st2-qfx48s-a# run show route receive-protocol bgp 192.11.1.10

inet.0: 39 destinations, 40 routes (38 active, 0 holddown, 1 hidden)

:vxlan.inet.0: 15 destinations, 15 routes (14 active, 0 holddown, 1 hidden)

bgp.evpn.0: 71 destinations, 87 routes (71 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:2::7::02:90:7b:71:f9:97/304
* 192.14.1.10 100 ?
  2:192.14.1.10:3::4::02:df:85:8e:84:bf/304
* 192.14.1.10 100 ?
  2:192.14.1.10:4::10::02:32:18:37:c3:81/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  2:192.14.1.10:2::7::02:90:7b:71:f9:97::105.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:3::4::02:df:85:8e:84:bf::101.1.1.3/304
* 192.14.1.10 200 ?
  2:192.14.1.10:4::10::02:32:18:37:c3:81::107.1.1.1/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:2::7::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:3::4::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:4::10::192.14.1.10/304
* 192.14.1.10 100 ?

default-switch.evpn.0: 75 destinations, 91 routes (75 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:2::7::02:90:7b:71:f9:97/304
* 192.14.1.10 100 ?
  2:192.14.1.10:3::4::02:df:85:8e:84:bf/304
* 192.14.1.10 100 ?
  2:192.14.1.10:4::10::02:32:18:37:c3:81/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94::102.1.1.3/304
* 192.14.1.10 200 ?
  2:192.14.1.10:2::7::02:90:7b:71:f9:97::105.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:3::4::02:df:85:8e:84:bf::101.1.1.3/304
* 192.14.1.10 200 ?
  2:192.14.1.10:4::10::02:32:18:37:c3:81::107.1.1.1/304
* 192.14.1.10 200 ?
  3:192.14.1.10:1::5::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:2::7::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:3::4::192.14.1.10/304
* 192.14.1.10 100 ?
  3:192.14.1.10:4::10::192.14.1.10/304
* 192.14.1.10 100 ?

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

But config has not been changed since last Friday (7/11/2015)

regress@sdn-st2-qfx48s-a# rollback ?
Possible completions:
  <[Enter]> Execute this command
  0 2015-07-11 01:26:02 UTC by regress via cli
  1 2015-07-11 01:24:26 UTC by regress via cli
  2 2015-07-11 01:21:57 UTC by regress via cli
  3 2015-07-11 01:20:35 UTC by regress via cli
  4 2015-07-11 01:19:40 UTC by regress via cli
  5 2015-07-11 01:12:27 UTC by regress via cli
  6 2015-07-11 01:10:54 UTC by regress via cli

regress@sdn-st2-qfx48s-a# show protocols evpn
traceoptions {
    file evpn.log size 5m files 3;
    flag all;
}
vni-options {
    vni 4 {
        vrf-target export target:1:1;
    }
    vni 5 {
        vrf-target export target:2:1;
    }
    inactive: vni 7 {
        vrf-target export target:64512:8000004;
    }
    inactive: vni 8 {
        vrf-target export target:4:1;
    }
    inactive: vni 101 {
        vrf-target export target:1:1;
    }
    inactive: vni 102 {
        vrf-target export target:2:1;
    }
    inactive: vni 104 {
        vrf-target export target:4:1;
    }
    inactive: vni 105 {
        vrf-target export target:64512:8000004;
    }
    inactive: vni 16777213 {
        vrf-target export target:2:1;
    }
}
encapsulation vxlan;
extended-vni-list [ 4 5 ];

{master:0}[edit]
regress@sdn-st2-qfx48s-a# show switch-options
vtep-source-interface lo0.0;
route-distinguisher 64512:1010;
vrf-import imp;
vrf-target target:2:1;

And then check the bgp summary, found all routes from Contrail are gone

regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      55 55 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8743 8723 0 1 2d 18:13:18 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8730 8712 0 0 2d 18:08:31 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8739 8724 0 1 2d 18:13:22 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 9818 8720 0 1 2d 18:13:14 Establ
  bgp.evpn.0: 0/0/0/0
  default-switch.evpn.0: 0/0/0/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 9640 8721 0 1 2d 18:13:14 Establ
  bgp.evpn.0: 0/0/0/0
  default-switch.evpn.0: 0/0/0/0
  __default_evpn__.evpn.0: 0/0/0/0

regress@sdn-st2-qfx48s-a# run show system uptime
fpc0:
--------------------------------------------------------------------------
Current time: 2015-07-13 19:34:44 UTC
System booted: 2015-07-10 19:52:11 UTC (2d 23:42 ago)
Protocols started: 2015-07-10 19:53:16 UTC (2d 23:41 ago)
Last configured: 2015-07-11 01:26:02 UTC (2d 18:08 ago) by regress
 7:34PM up 2 days, 23:43, 4 users, load averages: 0.13, 0.14, 0.09

Meanwhile, the other Opus (sdn-st2-qfx48s-b) still receiving routes from Contrail, withdrawn and received

regress@sdn-st2-qfx48s-b# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      77 66 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8764 8751 0 0 2d 18:26:14 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8751 8750 0 0 2d 18:26:01 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.100 64512 8736 8752 0 1 2d 18:18:49 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 11803 8750 0 0 2d 18:25:59 Establ
  bgp.evpn.0: 11/11/11/0
  default-switch.evpn.0: 11/11/11/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 11474 8750 0 0 2d 18:26:02 Establ
  bgp.evpn.0: 0/11/11/0
  default-switch.evpn.0: 0/11/11/0
  __default_evpn__.evpn.0: 0/0/0/0

{master:0}[edit]
regress@sdn-st2-qfx48s-b# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      97 76 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8806 8793 0 0 2d 18:45:31 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8794 8792 0 0 2d 18:45:18 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.100 64512 8779 8794 0 1 2d 18:38:06 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 16251 8791 0 0 2d 18:45:16 Establ
  bgp.evpn.0: 21/21/21/0
  default-switch.evpn.0: 21/21/21/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 15606 8792 0 0 2d 18:45:19 Establ
  bgp.evpn.0: 0/21/21/0
  default-switch.evpn.0: 0/21/21/0
  __default_evpn__.evpn.0: 0/0/0/0

regress@sdn-st2-qfx48s-b# show protocols evpn
traceoptions {
    file evpn.log size 5m files 3;
    flag all;
}
vni-options {
    vni 4 {
        vrf-target export target:1:1;
    }
    vni 101 {
        vrf-target export target:1:1;
    }
    vni 103 {
        vrf-target export target:3:1;
    }
    vni 4095 {
        vrf-target export target:1:1;
    }
    vni 101101 {
        vrf-target export target:1:1;
    }
    vni 16777213 {
        vrf-target export target:2:1;
    }
}
encapsulation vxlan;
extended-vni-list all;

{master:0}[edit]
regress@sdn-st2-qfx48s-b# show switch-options
vtep-source-interface lo0.0;
route-distinguisher 1000:11;
vrf-import [ imp2 imp1 imp3 imp4 ];
vrf-target {
    target:1:1;
    auto;
}

I am not sure if this is Contrail issue or opus issue, so will open another PR in gnats, and save all the logs in gnats.

Revision history for this message
Sharon Zhang (sharonz) wrote :
Download full text (8.0 KiB)

This seems related with the ping from one of VM (VNI101-VM3), when ping 101.1.1.21 (BMS behind sdn-st2-qfx48s-a), the routes received on Opus will be 0 and if stop the ping, the routes are received, but no routes for this VM are received.

<<< ping from VN101-VM3 stopped , routes are received, but the routes from this VM are not received >>>>>

regress@sdn-st2-qfx48s-a# run show bgp summary
Groups: 1 Peers: 5 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
                      87 71 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.1.1.24 64512 8901 8879 0 1 2d 19:24:43 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.1.1.48 64512 8887 8868 0 0 2d 19:19:56 Establ
  bgp.evpn.0: 24/24/24/0
  default-switch.evpn.0: 24/24/24/0
  __default_evpn__.evpn.0: 0/0/0/0
100.48.1.11 64512 8896 8880 0 1 2d 19:24:47 Establ
  bgp.evpn.0: 7/7/7/0
  default-switch.evpn.0: 6/6/6/0
  __default_evpn__.evpn.0: 1/1/1/0
192.11.1.10 64512 26811 8876 0 1 2d 19:24:39 Establ
  bgp.evpn.0: 16/16/16/0
  default-switch.evpn.0: 16/16/16/0
  __default_evpn__.evpn.0: 0/0/0/0
192.12.1.10 64512 25354 8877 0 1 2d 19:24:39 Establ
  bgp.evpn.0: 0/16/16/0
  default-switch.evpn.0: 0/16/16/0
  __default_evpn__.evpn.0: 0/0/0/0

{master:0}[edit]
regress@sdn-st2-qfx48s-a# run shwo
                              ^
syntax error, expecting <command>.
regress@sdn-st2-qfx48s-a# run shworou
                              ^
syntax error, expecting <command>.
regress@sdn-st2-qfx48s-a# run show route receive-protocol bgp 192.11.1.10

inet.0: 39 destinations, 40 routes (38 active, 0 holddown, 1 hidden)

:vxlan.inet.0: 15 destinations, 15 routes (14 active, 0 holddown, 1 hidden)

bgp.evpn.0: 71 destinations, 87 routes (71 active, 0 holddown, 0 hidden)
  Prefix Nexthop MED Lclpref AS path
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:9d:5a:c1:54:94/304
* 192.14.1.10 100 ?
  2:192.14.1.10:2::7::02:90:7b:71:f9:97/304
* 192.14.1.10 100 ?
  2:192.14.1.10:3::4::02:df:85:8e:84:bf/304
* 192.14.1.10 100 ?
  2:192.14.1.10:4::10::02:32:18:37:c3:81/304
* 192.14.1.10 100 ?
  2:192.14.1.10:1::5::02:0d:38:9b:55:e8::102.1.1.1/304
* 192.14.1.10 200 ?
  2:192.14.1.10:1::5::02:3b:f8:19:42:cb::102.1.1.4/304 ...

Read more...

Nischal Sheth (nsheth)
Changed in juniperopenstack:
assignee: nobody → Nischal Sheth (nsheth)
importance: Undecided → High
milestone: none → r2.30-fcs
tags: added: contrail-control
Revision history for this message
Nischal Sheth (nsheth) wrote :

The routes might have suddenly gone away because vrouter agents crashed.
Noticed 3-4 agent cores on both compute nodes from around the same time.
Sharon is going to file bugs for those crashes. Will lower the priority of this
bug and see if it can reproduced.

summary: - EVPN routes from Contrail Control nodes are suddenly withdrawn without
- any change on both Contrail and Opus
+ EVPN routes from CNs are withdrawn without any change on CN or Opus
Revision history for this message
Nischal Sheth (nsheth) wrote :

@Sharon

Will close this bug for now - please re-open if you see it again.

Jeba Paulaiyan (jebap)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.