Control node not advertising inclusive mcast route into EVPN to MX

[amit surana]

Physical servers connected directly to the MX can be made part of a contrail VNs by creating physical/logical interfaces on physical router objects. Once this is done, the physical server should have L2/L3 connectivity to all the other hosts (VMs/BMS) in that VN.

When this setup is first configured, it is seen that the control node will not advertise an inclusive multicast route into the EVPN towards the MX. As a result, ARP requests that are coming from the physical server connected to the MX are not flooded into the VN. Restarting the control node fixes this issue.

Topology:

physical_server(192.168.255.240)--MX--(192.168.255.230)Contrail_VM

ARP request from the physical server for contrail_VM is not flooded by MX into the contrail VN.

root@vibrant# run show evpn instance _contrail_l2_9_MGMT extensive
Instance: _contrail_l2_9_MGMT
  Route Distinguisher: 172.16.89.200:73
  Encapsulation type: VXLAN
  MAC database status Local Remote
    MAC advertisements: 2 13
    MAC+IP advertisements: 1 13
    Default gateway MAC advertisements: 1 0
  Number of local interfaces: 1 (1 up)
    Interface name ESI Mode Status
    ge-0/0/0.2001 00:00:00:00:00:00:00:00:00:00 single-homed Up
  Number of IRB interfaces: 1 (1 up)
    Interface name VLAN VNI Status L3 context
    irb.9 9 Up _contrail_l3_9_MGMT
  Number of bridge domains: 1
    VLAN VNI Intfs / up IRB intf Mode MAC sync IM route label
    8191 9 1 1 irb.9 Extended Enabled 9
  Number of neighbors: 3
    172.16.80.10
      Received routes
        MAC address advertisement: 4
        MAC+IP address advertisement: 4
        Inclusive multicast: 0
        Ethernet auto-discovery: 0
    172.16.80.11
      Received routes
        MAC address advertisement: 4
        MAC+IP address advertisement: 4
        Inclusive multicast: 0
        Ethernet auto-discovery: 0
    172.16.80.12
      Received routes
        MAC address advertisement: 5
        MAC+IP address advertisement: 5
        Inclusive multicast: 0
        Ethernet auto-discovery: 0
  Number of peers: 3
  Number of ethernet segments: 1
    ESI: 05:00:00:fc:00:00:00:00:09:00
      Status: Resolved by IFL irb.9
      Local interface: irb.9, Status: Up/Forwarding
  Router-ID: 172.16.89.200
  Source VTEP interface IP: 172.16.89.200

[Nischal Sheth (nsheth)]

@asurana-t

Can you check if the CN has a route for the broadcast mac from the
vrouter agent which has the VM in question? CN generates the IM
route based on the broadcast MAC route received from the agent.

[Nischal Sheth (nsheth)]

@asurana-t

When you get a chance, can you remove the R3.0 scope - it should be tracked
via Trunk since we haven't created a R3.0 branch yet.

[amit surana (asurana-t)]

Further debug with Nischal revealed:

Agent is sending the broadcast mac route to the CN, albeit with the label set to 0. CN has an explicit check to not generated an inclusive-multicast route if label == 0.

[Nischal Sheth (nsheth)]

Restarting an agent causes the agent to advertise a fresh broadcast
mac route, which has the correct label. That results in the CN adding
the requisite IM route. This also explains why restarting the CN fixes
all the IM routes.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16137
Submitter: Manish Singh (<email address hidden>)

[amit surana (asurana-t)]

 Contrail

    Collapse
    Expand
    Wrap
    NoWrap

SandeshTraceTextResponse
traces
2016-01-07 10:41:26.331 XmppRxStream: Received xmpp message from: 172.16.80.4 Port 8093 Size: 156 Packet: <stream:stream from="" to="csol1-node10/dns" id="++123" version="1.0" xml:lang="en" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" > $ controller/src/xmpp/xmpp_connection.cc 513
2016-01-07 10:41:26.333 XmppRxStream: Received xmpp message from: 172.16.80.4 Port 5269 Size: 163 Packet: <stream:stream from="csol1-node4" to="csol1-node10" id="++123" version="1.0" xml:lang="en" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" > $ controller/src/xmpp/xmpp_connection.cc 513
2016-01-07 10:41:26.334 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 251 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node10" <email address hidden>/config"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <subscribe node="virtual-router:default-global-system-config:csol1-node10" /> </pubsub> </iq> $ controller/src/xmpp/xmpp_connection.cc 211
2016-01-07 10:41:26.334 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 326 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node10" <email address hidden>/bgp-peer" id="subscribe0"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <subscribe node="default-domain:default-project:ip-fabric:__default__"> <options> <instance-id>0</instance-id> </options> </subscribe> </pubsub> </iq> $ controller/src/xmpp/xmpp_connection.cc 211
2016-01-07 10:41:26.336 XmppRxStream: Received xmpp message from: 172.16.80.7 Port 5269 Size: 163 Packet: <stream:stream from="csol1-node7" to="csol1-node10" id="++123" version="1.0" xml:lang="en" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" > $ controller/src/xmpp/xmpp_connection.cc 513
2016-01-07 10:41:26.336 XmppTxStream: Sent xmpp message to: 172.16.80.7 Port 5269 Size: 326 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node10" <email address hidden>/bgp-peer" id="subscribe1"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <subscribe node="default-domain:default-project:ip-fabric:__default__"> <options> <instance-id>0</instance-id> </options> </subscribe> </pubsub> </iq> $ controller/src/xmpp/xmpp_connection.cc 211
2016-01-07 10:41:26.336 XmppRxStream: Received xmpp message from: 172.16.80.3 Port 8093 Size: 156 Packet: <stream:stream from="" to="csol1-node10/dns" id="++123" version="1.0" xml:lang="en" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" > $ controller/src/xmpp/xmpp_connection.cc 513
2016-01-07 10:53:51.101 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 247 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node10" <email address hidden>/config"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <subscribe node="virtual-machine:f5dd00f2-6fac-4f2d-96c2-5ed2a18a7e05" /> </pubsub> </iq> $ controller/src/xmpp/xmpp_connection.cc 211
2016-01-07 10:53:51.110 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 304 Packet: <?xml version="1.0"?> <iq type="set" fro...

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16137
Committed: http://github.org/Juniper/contrail-controller/commit/6d336bf8ab619c2e9e3c7fd9ab1660b35616ebd5
Submitter: Zuul
Branch: master

commit 6d336bf8ab619c2e9e3c7fd9ab1660b35616ebd5
Author: Manish <email address hidden>
Date: Thu Jan 7 16:20:08 2016 +0530

Agent didnt send proper label in evpn inclusive replication.

Problem:
While exporting subscription for inclusive replication tree when local peer was
only added, exporter used to search for multicast_peer. In its absence it will
mark label as 0. In case local_vm_peer(local interface list) gets added before
local-peer, it will result in creation of path with multicast_peer, hence label
will be picked as vxlan-id.

Solution:
Before picking label as 0 check for presence of local_peer as well and pick up
vxlan of same as label.

Change-Id: Ie7958aa3cb7175e6df4534d8e66234b8f92c4138
Closes-bug: 1529665

[amit surana (asurana-t)]

Agent is generating the broadcast route with incorrect label (vxlan id for the VN is 5, agent generates broadcast route with label 21), CN generates IM route also with label 21, which MX seems to not like.

Agent XMPP message:

2016-02-01 23:01:28.149 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 1044 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node11" <email address hidden>/bgp-peer" id="pubsub_l226"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <publish node="25/242/255.255.255.255,0.0.0.0/32"> <item> <entry> <nlri> <af>25</af> <safi>242</safi> <ethernet-tag>0</ethernet-tag> <mac>ff:ff:ff:ff:ff:ff</mac> <address>0.0.0.0/32</address> </nlri> <next-hops> <next-hop> <af>1</af> <address>172.16.80.11</address> <label>21</label> <tunnel-encapsulation-list> <tunnel-encapsulation>gre</tunnel-encapsulation> <tunnel-encapsulation>udp</tunnel-encapsulation> </tunnel-encapsulation-list> </next-hop> </next-hops> <olist /> <virtual-network></virtual-network> <sequence-number>0</sequence-number> <security-group-list /> <local-preference>100</local-preference> <med>0</med> <edge-replication-not-supported>false</edge-replication-not-supported> <assisted-replication-supported>false</assisted-replication-supported> <leaf-olist /> <replicator-address></replicator-address> </entry> </item> </publish> </pubsub> </iq> $

Route on MX: IM route has label set to 0.

root@vibrant# run show route table _contrail_l2_5_MGMT.evpn.0 | grep 172.16.80.11
2:172.16.80.11:1::5::02:7e:ae:ae:a2:5f/304
2:172.16.80.11:1::5::02:9e:cd:c6:58:2c/304
2:172.16.80.11:1::5::02:b9:bd:bb:af:6d/304
2:172.16.80.11:1::5::02:cb:38:33:86:66/304
2:172.16.80.11:1::5::02:7e:ae:ae:a2:5f::192.168.0.13/304
2:172.16.80.11:1::5::02:9e:cd:c6:58:2c::192.168.0.4/304
2:172.16.80.11:1::5::02:b9:bd:bb:af:6d::192.168.0.10/304
2:172.16.80.11:1::5::02:cb:38:33:86:66::192.168.0.7/304
3:172.16.80.11:1::0::172.16.80.11/304

[Nischal Sheth (nsheth)]

Re-opening since the issue doesn't seem to be fixed completely per #9 above.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16856
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16856
Committed: http://github.org/Juniper/contrail-controller/commit/81451282020340d95d98025a481db5d8a77cd290
Submitter: Zuul
Branch: master

commit 81451282020340d95d98025a481db5d8a77cd290
Author: Manish Singh <email address hidden>
Date: Thu Feb 4 09:01:58 2016 +0530

Along with label modify ethernet-tag as well.

Change-Id: I9d56785dcff92b5bbdbaf8b3b33e895be646c607
Closes-bug: 1529665

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17012
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17013
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17014
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17012
Committed: http://github.org/Juniper/contrail-controller/commit/e8d39c0a7c46005485a81c9196fd4bbe1b07325c
Submitter: Zuul
Branch: R2.21.x

commit e8d39c0a7c46005485a81c9196fd4bbe1b07325c
Author: Manish Singh <email address hidden>
Date: Thu Feb 4 09:01:58 2016 +0530

Along with label modify ethernet-tag as well.

Closes-bug: 1529665
(cherry picked from commit 81451282020340d95d98025a481db5d8a77cd290)

Conflicts:
 src/vnsw/agent/test/test_l2route.cc
Change-Id: I9d56785dcff92b5bbdbaf8b3b33e895be646c607

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17014
Committed: http://github.org/Juniper/contrail-controller/commit/97138c8e72fb94f36c49244753d3ab22c7a75d6b
Submitter: Zuul
Branch: R2.22.x

commit 97138c8e72fb94f36c49244753d3ab22c7a75d6b
Author: Manish Singh <email address hidden>
Date: Thu Feb 4 09:01:58 2016 +0530

Along with label modify ethernet-tag as well.

Closes-bug: 1529665
(cherry picked from commit 81451282020340d95d98025a481db5d8a77cd290)

Conflicts:
 src/vnsw/agent/test/test_l2route.cc
Change-Id: I9d56785dcff92b5bbdbaf8b3b33e895be646c607

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17013
Committed: http://github.org/Juniper/contrail-controller/commit/a761be24d0eaa47b657e0ebc0a6f64d514f98224
Submitter: Zuul
Branch: R2.20

commit a761be24d0eaa47b657e0ebc0a6f64d514f98224
Author: Manish Singh <email address hidden>
Date: Thu Feb 4 09:01:58 2016 +0530

Along with label modify ethernet-tag as well.

Closes-bug: 1529665
(cherry picked from commit 81451282020340d95d98025a481db5d8a77cd290)

Conflicts:
 src/vnsw/agent/test/test_l2route.cc
Change-Id: I9d56785dcff92b5bbdbaf8b3b33e895be646c607

[amit surana (asurana-t)]

Re-opening again, as the issue described in #9 still exists. Agent is sending a broadcast route to CN with incorrect encapsulation (mpls/gre as opposed to vxlan).

2016-02-17 05:44:12.953 XmppTxStream: Sent xmpp message to: 172.16.80.4 Port 5269 Size: 1046 Packet: <?xml version="1.0"?> <iq type="set" from="csol1-node12" <email address hidden>/bgp-peer" id="pubsub_l22602"> <pubsub xmlns="http://jabber.org/protocol/pubsub"> <publish node="25/242/255.255.255.255,0.0.0.0/32"> <item> <entry> <nlri> <af>25</af> <safi>242</safi> <ethernet-tag>0</ethernet-tag> <mac>ff:ff:ff:ff:ff:ff</mac> <address>0.0.0.0/32</address> </nlri> <next-hops> <next-hop> <af>1</af> <address>172.16.80.12</address> <label>20</label> <tunnel-encapsulation-list> <tunnel-encapsulation>gre</tunnel-encapsulation> <tunnel-encapsulation>udp</tunnel-encapsulation> </tunnel-encapsulation-list> </next-hop> </next-hops> <olist /> <virtual-network></virtual-network> <sequence-number>0</sequence-number> <security-group-list /> <local-preference>100</local-preference> <med>0</med> <edge-replication-not-supported>false</edge-replication-not-supported> <assisted-replication-supported>false</assisted-replication-supported> <leaf-olist /> <replicator-address></replicator-address> </entry> </item> </publish> </pubsub> </iq> $ controller/src/xmpp/xmpp_connection.cc 209

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17327
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17327
Committed: http://github.org/Juniper/contrail-controller/commit/01adcbaacf99a332496209df9c152658672e1f5b
Submitter: Zuul
Branch: master

commit 01adcbaacf99a332496209df9c152658672e1f5b
Author: Manish Singh <email address hidden>
Date: Thu Feb 18 01:36:22 2016 +0530

Handle vxlan change for local_vm peer in l2 flood.

Change-Id: I3eb43c9541eec6ef3d343a8aa041b527033c24fe
Closes-bug: 1529665

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17409
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17410
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17411
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17424
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17424
Committed: http://github.org/Juniper/contrail-controller/commit/a6be817f369ca6f2e30f2c3a39079e8a9133e5e9
Submitter: Zuul
Branch: R3.0

commit a6be817f369ca6f2e30f2c3a39079e8a9133e5e9
Author: Divakar <email address hidden>
Date: Wed Feb 17 15:22:01 2016 +0530

Correct the metadata string of fip-pool to vn

The metadata mentioned in xsd for the link between floating-ip-pool to
VN is "virtual-network-floating-ip-pool" and Agent is adding a rule
in dependency manager for VMI as "floating-ip-pool-virtual-network".
This is resulting in no InterestedEvent in dependency tracker if the
link added between VN-FIP is the last link in that VMI.

As a fix, metadata is corrected in Agent.

Change-Id: I484099d8c843493cfcdb618a3165819f9eab5d08
closes-bug: #1544788

Handle vxlan change for local_vm peer in l2 flood.

Change-Id: I3eb43c9541eec6ef3d343a8aa041b527033c24fe
Closes-bug: 1529665

Add ShowRouteAggregateSummaryReq introspect command

Change-Id: Ib633b018798e19c7fc73f4bebaef10283aef1a89
Related-bug: 1500698

database-purge request fails if analytics-api starts before cassandra

analytics-api creates cql session only on creation of AnalyticsDb
object. Therefore, if analytics-api starts before cassandra becomes
active, then database-purge request fails.

This patch ensures that the cql session is created upon purge request if
cql session is None.

Change-Id: Ic38d82cd1d614cd80af917ff205c3a63b95d6b92
Closes-Bug: #1546435

LBaaS scheduler should not use hardcoded tenant name

This fix removes the hardcoded 'admin' tenant name from
lbaas scheduler and reads the tenant from config.
It also fixes the default admin_tenant_name in svc_monitor.py
to 'admin'.
Closes-Bug: #1543296

Change-Id: If5e558e27ef6fd81d81cf56a317f39dbac2854ce

Fix error message

The VM uuid should be shown instead of the VN uuid

Change-Id: I56a65f5aabbc441139784e672babcf5e7508b880
Closes-Bug: #1546107

Adding UVE information for Health Check

Virtual Machine Interface UVE will also carry additional
information on the HealthCheck instances running on it,
its state and running status

Closes-Bug: 1546390
Change-Id: Ib6f2dac3e80544852359b8e7633343011b9f7a4e

Agent changes to accept port add requests without IP address.

This is done for both REST and thrift interface. Also add a flag to Flow info maintained
by FlowStatsCollector to indicate whether a delete was enqueued for that flow or not.

Change-Id: Ibe1cd81bcd71d42919995860e846383740bdfcdc
Closes-Bug: #1546939

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17410
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17411
Submitter: Manish Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17410
Committed: http://github.org/Juniper/contrail-controller/commit/0e8928586f5657db5b3657bac2d897a50a02a584
Submitter: Zuul
Branch: R2.21.x

commit 0e8928586f5657db5b3657bac2d897a50a02a584
Author: Manish Singh <email address hidden>
Date: Thu Feb 18 01:36:22 2016 +0530

Handle vxlan change for local_vm peer in l2 flood.

Closes-bug: 1529665

Conflicts:
 src/vnsw/agent/test/test_l2route.cc

Change-Id: I3eb43c9541eec6ef3d343a8aa041b527033c24fe

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17411
Committed: http://github.org/Juniper/contrail-controller/commit/00bcf327fbb57b36a6e6e1a1c419c955a7ae18a5
Submitter: Zuul
Branch: R2.22.x

commit 00bcf327fbb57b36a6e6e1a1c419c955a7ae18a5
Author: Manish Singh <email address hidden>
Date: Thu Feb 18 01:36:22 2016 +0530

Handle vxlan change for local_vm peer in l2 flood.

Closes-bug: 1529665

Conflicts:
 src/vnsw/agent/test/test_l2route.cc

Change-Id: I3eb43c9541eec6ef3d343a8aa041b527033c24fe

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17409
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17409
Committed: http://github.org/Juniper/contrail-controller/commit/3d71dc6f4afc0efaaca199c1d51d52a7ee8c3966
Submitter: Zuul
Branch: R2.20

commit 3d71dc6f4afc0efaaca199c1d51d52a7ee8c3966
Author: Hari <email address hidden>
Date: Mon Mar 7 12:50:43 2016 +0530

Handle vxlan change for local_vm peer in l2 flood.

Change-Id: I3eb43c9541eec6ef3d343a8aa041b527033c24fe
Closes-bug: 1529665