Gateway_less_Fwd: Traffic blackholing observed when service chain policy is deleted on VNs whose provider network is IP Fabric network

[Chandra Sekhar Reddy Mallam]

Traffic blackholing observed when service chain policy is deleted on VNs whose provider network is IP Fabric network

Build
------
R4.1.0.0 Build 23 Ubuntu 14.04 Mitaka

Steps
------
1. Configure left_vn (10.10.10.0/24), right_vn (20.20.20.0/24) and mgmt_vn (30.30.30.0/24)
2. Bring up 2 end VMs (left_vm i.e 10.10.10.3 and right_vm i.e 20.20.20.3), one in left_vn and right_vn across compute nodes
3. Bring up service VM with NICs (mgmt_vn, left_vn, right_vn)
4. Now, configure IP Fabric network as provider network over left_vn and right_vn
5. Configure, service instance with above SVM port-tuple and configure policy with this SI
6. Apply policy over left_vn and right_vn
7. Now, ping right_vm from left_vm, ping goes through
8. Now, remove the policy on left_vn and right_vn. Now, ping stops

As IP Fabric network is configured as provider network over left and right vns, ping should continue. But seeing traffic drops. This is true with new flows as well (new ping).

On the flow source nh is incorrect, which is causing the traffic drops on source compute itself.

Please see the log below:

root@nodek11:~# flow --match 20.20.20.3
Flow table(size 80609280, entries 629760)

Entries: Created 547 Added 545 Deleted 1038 Changed 1094 Processed 547 Used Overflow entries 0
(Created Flows/CPU: 31 23 27 12 21 27 26 19 8 23 23 11 16 21 13 35 34 20 19 20 19 31 22 31 0 0 15 0 0 0 0 0)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([20.20.20.3]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   170876<=>387576 10.10.10.3:28471 1 (0)
                         20.20.20.3:0
(Gen: 1, K(nh):48, Action:F, Flags:, QOS:-1, S(nh):14, Stats:786/77028, <== 14 is incorrect
 SPort 63814, TTL 0, Sinfo 4.0.0.0)

   387576<=>170876 20.20.20.3:28471 1 (0)
                         10.10.10.3:0
(Gen: 1, K(nh):48, Action:F, Flags:, QOS:-1, S(nh):14, Stats:0/0, SPort 61206,
 TTL 0, Sinfo 0.0.0.0)

root@nodek11:~# nh --get 14
Id:14 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:3046 Vrf:0
              Flags:Valid, Etree Root,
              EncapFmly:0806 Oif:0 Len:14
              Encap Data: 80 ac ac f0 a2 c1 0c c4 7a 32 0a 88 08 00

root@nodek11:~#
root@nodek11:~#
root@nodek11:~#
root@nodek11:~# dropstats| grep "Invalid"
Invalid IF 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Invalid NH 122
Invalid Label 1
Invalid Protocol 0
Invalid Mcast Source 0
Invalid Packets 0
Invalid VNID 0
Invalid Source 1726
root@nodek11:~# dropstats| grep "Invalid"
Invalid IF 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Invalid NH 122
Invalid Label 1
Invalid Protocol 0
Invalid Mcast Source 0
Invalid Packets 0
Invalid VNID 0
Invalid Source 1728
root@nodek11:~# dropstats| grep "Invalid"
Invalid IF 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Invalid NH 122
Invalid Label 1
Invalid Protocol 0
Invalid Mcast Source 0
Invalid Packets 0
Invalid VNID 0
Invalid Source 1729
root@nodek11:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root

vif0/0 OS: em1 (Speed 1000, Duplex 1)
            Type:Physical HWaddr:0c:c4:7a:32:0a:88 IPaddr:0.0.0.0
            Vrf:0 Flags:L3L2VpEr QOS:-1 Ref:6
            RX packets:3989253 bytes:465282776 errors:0
            TX packets:1228086 bytes:1386446833 errors:0
            Drops:219190

vif0/1 OS: vhost0
            Type:Host HWaddr:0c:c4:7a:32:0a:88 IPaddr:10.204.216.231
            Vrf:0 Flags:PL3DEr QOS:-1 Ref:7
            RX packets:893449 bytes:1347284494 errors:0
            TX packets:3694537 bytes:435417513 errors:0
            Drops:2

vif0/2 OS: pkt0
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX packets:476374 bytes:55079908 errors:0
            TX packets:2628701 bytes:281194134 errors:0
            Drops:0

vif0/3 OS: tap3e0dbd3c-d7
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.5
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:27001 bytes:1313954 errors:0
            TX packets:51072 bytes:2326016 errors:0
            ISID: 0 Bmac: 02:3e:0d:bd:3c:d7
            Drops:1

vif0/4 OS: tapb500128b-e8
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.3
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:160342 bytes:16152121 errors:0
            TX packets:83682 bytes:9031361 errors:0
            Drops:76845

vif0/5 OS: tapd795de00-0e
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.4
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:23995 bytes:1024030 errors:0
            TX packets:48058 bytes:2035264 errors:0
            ISID: 0 Bmac: 02:d7:95:de:00:0e
            Drops:1

vif0/6 OS: tap9fb5dd05-2f
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.20.20.5
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:106991 bytes:12711288 errors:0
            TX packets:212010 bytes:21758984 errors:0
            ISID: 0 Bmac: 02:9f:b5:dd:05:2f
            Drops:119958

vif0/7 OS: tape3e97f2d-07
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:30.30.30.4
            Vrf:4 Flags:PL3L2DEr QOS:-1 Ref:5
            RX packets:23726 bytes:999084 errors:0
            TX packets:47587 bytes:2002254 errors:0
            Drops:0

vif0/8 OS: tap84b4f869-98
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:30.30.30.3
            Vrf:4 Flags:PL3L2DEr QOS:-1 Ref:5
            RX packets:23724 bytes:998016 errors:0
            TX packets:47575 bytes:2000050 errors:0
            Drops:0

vif0/9 OS: tap0a473b8f-2e
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.20.20.4
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:122203 bytes:15017850 errors:0
            TX packets:210218 bytes:21578404 errors:0
            ISID: 0 Bmac: 02:0a:47:3b:8f:2e
            Drops:144828

vif0/4350 OS: pkt3
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:0 bytes:0 errors:0
            TX packets:0 bytes:0 errors:0
            Drops:0

vif0/4351 OS: pkt1
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:3426 bytes:287784 errors:0
            TX packets:3426 bytes:287784 errors:0
            Drops:0

root@nodek11:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35262
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35262
Committed: http://github.com/Juniper/contrail-controller/commit/7b0507e3b3cb7b1e350125b2bd75b0277efd9db4
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7b0507e3b3cb7b1e350125b2bd75b0277efd9db4
Author: Naveen N <email address hidden>
Date: Tue Sep 5 13:08:09 2017 +0530

* Publish floating-ip route with proper encapsulation

1> Correct dependency manager to take care of forwarding-vrf change
2> Remove route in fabric VRF when forwarding vrf config is deleted
3> Pick VN, SG anf tag list from policy fabric VRF only, if route is
not found in policy fabric VRF, use empty list instead of picking
from default VRF.
Test case for same.
Closes-bug:#1711077
Closes-bug:#1712000
Closes-bug:#1711527
Closes-bug:#1712245

Change-Id: Ibee3d79613a118d2e8838bd07b17ca4bca8df186