Gateway_less_Fwd: Leaked routes were not getting removed from default routing instance when IP-Fabric provider network is deleted from VN

[Chandra Sekhar Reddy Mallam]

Leaked routes were not getting removed from default routing instance when IP-Fabric provider network is deleted from VN.

Please see the log below:

root@nodek11:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root

vif0/0 OS: em1 (Speed 1000, Duplex 1)
            Type:Physical HWaddr:0c:c4:7a:32:0a:88 IPaddr:0.0.0.0
            Vrf:0 Flags:L3L2VpEr QOS:-1 Ref:6
            RX packets:125658 bytes:13000624 errors:0
            TX packets:38443 bytes:289801746 errors:0
            Drops:933

vif0/1 OS: vhost0
            Type:Host HWaddr:0c:c4:7a:32:0a:88 IPaddr:10.204.216.231
            Vrf:0 Flags:PL3DEr QOS:-1 Ref:7
            RX packets:32295 bytes:289148898 errors:0
            TX packets:126460 bytes:13024062 errors:0
            Drops:1

vif0/2 OS: pkt0
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX packets:3869 bytes:539528 errors:0
            TX packets:70942 bytes:7505785 errors:0
            Drops:0

vif0/3 OS: tapa767c4dd-5d
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.3
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5 <== After configuring IP-Fabric provider network
            RX packets:12190 bytes:1206352 errors:0
            TX packets:4329 bytes:424556 errors:0
            Drops:2646

vif0/4350 OS: pkt3
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:259 bytes:25382 errors:0
            TX packets:259 bytes:21756 errors:0
            Drops:0

vif0/4351 OS: pkt1
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:0 bytes:0 errors:0
            TX packets:0 bytes:0 errors:0
            Drops:0

root@nodek11:~#

root@nodek11:~# rt --dump 0 | grep 10.10.10.3
10.10.10.3/32 32 PT - 32 -
10.10.10.10/32 0 T - 14 -
10.10.10.30/32 0 T - 14 -
10.10.10.31/32 0 T - 14 -
10.10.10.32/32 0 T - 14 -
10.10.10.33/32 0 T - 14 -
10.10.10.34/32 0 T - 14 -
10.10.10.35/32 0 T - 14 -
10.10.10.36/32 0 T - 14 -
10.10.10.37/32 0 T - 14 -
10.10.10.38/32 0 T - 14 -
10.10.10.39/32 0 T - 14 -
root@nodek11:~# rt --dump 0 | grep 10.10.10.4
10.10.10.4/32 32 PT - 21 0:25:90:93:d2:44(142252)
10.10.10.40/32 0 T - 14 -
10.10.10.41/32 0 T - 14 -
10.10.10.42/32 0 T - 14 -
10.10.10.43/32 0 T - 14 -
10.10.10.44/32 0 T - 14 -
10.10.10.45/32 0 T - 14 -
10.10.10.46/32 0 T - 14 -
10.10.10.47/32 0 T - 14 -
10.10.10.48/32 0 T - 14 -
10.10.10.49/32 0 T - 14 -
root@nodek11:~#

------------------------------------
Now, deleting IP-fabric provider network
------------------------------------

>>> left_vn.del_virtual_network(ip_fab_vn)
>>> vnc_lib.virtual_network_update(left_vn)
>>> pprint(vars(left_vn))
{'_address_allocation_mode': u'user-defined-subnet-only',
 '_display_name': u'left_vn',
 '_ecmp_hashing_include_fields': hashing_configured = False, source_ip = True, destination_ip = True, ip_protocol = True, source_port = True, destination_port = True,
 '_export_route_target_list': route_target = [],
 '_flood_unknown_unicast': False,
 '_id_perms': permissions = owner = admin, owner_access = 7, group = admin, group_access = 7, other_access = 7, uuid = uuid_mslong = 8369792822862037127, uuid_lslong = 13246514812731998162, enable = True, created = 2017-08-16T06:22:21.592337, last_modified = 2017-08-16T06:22:21.592337, description = None, user_visible = True, creator = None,
 '_import_route_target_list': route_target = [],
 '_is_shared': False,
 '_layer2_control_word': False,
 '_mac_aging_time': 300,
 '_mac_learning_enabled': False,
 '_multi_policy_service_chains_enabled': False,
 '_original_virtual_network_refs': [{'to': [u'default-domain',
                                            u'default-project',
                                            u'ip-fabric'],
                                     'uuid': u'cc9bea1a-c415-4458-86b5-66e9a4eb9ea4'}],
 '_pbb_etree_enable': False,
 '_pbb_evpn_enable': False,
 '_pending_field_list_updates': {},
 '_pending_field_map_updates': {},
 '_pending_field_updates': set([]),
 '_pending_ref_updates': set([]),
 '_perms2': owner = 259f77d728324a01ba54ed00b636258e, owner_access = 7, global_access = 0, share = [],
 '_port_security_enabled': True,
 '_router_external': False,
 '_server_conn': <vnc_api.vnc_api.VncApi object at 0x7fbd820dc510>,
 '_type': 'virtual-network',
 '_uuid': u'74277a3a-324e-4087-b7d5-130a04d73fd2',
 '_virtual_network_network_id': 4,
 '_virtual_network_properties': allow_transit = False, network_id = None, vxlan_network_identifier = None, forwarding_mode = None, rpf = enable, mirror_destination = False,
 'fq_name': [u'default-domain', u'admin', u'left_vn'],
 'name': u'left_vn',
 'network_ipam_refs': [{u'attr': ipam_subnets = [subnet = ip_prefix = 10.10.10.0, ip_prefix_len = 24, default_gateway = 10.10.10.1, dns_server_address = 10.10.10.2, subnet_uuid = d0fa35de-0793-40dc-8ac7-f92144b64dcc, enable_dhcp = True, dns_nameservers = [], allocation_pools = [], addr_from_start = True, dhcp_option_list = None, host_routes = None, subnet_name = d0fa35de-0793-40dc-8ac7-f92144b64dcc, alloc_unit = 1, created = None, last_modified = None], host_routes = None,
                        u'href': u'http://10.204.217.127:8082/network-ipam/8990e0a0-3f5b-4128-b9ea-6f4c8f357990',
                        u'to': [u'default-domain',
                                u'default-project',
                                u'default-network-ipam'],
                        u'uuid': u'8990e0a0-3f5b-4128-b9ea-6f4c8f357990'}],
 'parent_type': u'project',
 'parent_uuid': u'259f77d7-2832-4a01-ba54-ed00b636258e',
 'virtual_network_refs': []}
>>>

root@nodek11:~# vif --get 3
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root

vif0/3 OS: tapa767c4dd-5d
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.3
            Vrf:2 Flags:PL3L2DEr QOS:-1 Ref:5 <== Correct
            RX packets:13649 bytes:1352085 errors:0
            TX packets:5450 bytes:531709 errors:0
            Drops:3278

root@nodek11:~# rt --dump 0 | grep 10.10.10.3 <=== But, earlier leaked routes still present in IP-Fabric VRF
10.10.10.3/32 32 PT - 32 -
10.10.10.10/32 0 T - 14 -
10.10.10.30/32 0 T - 14 -
10.10.10.31/32 0 T - 14 -
10.10.10.32/32 0 T - 14 -
10.10.10.33/32 0 T - 14 -
10.10.10.34/32 0 T - 14 -
10.10.10.35/32 0 T - 14 -
10.10.10.36/32 0 T - 14 -
10.10.10.37/32 0 T - 14 -
10.10.10.38/32 0 T - 14 -
10.10.10.39/32 0 T - 14 -
root@nodek11:~# rt --dump 0 | grep 10.10.10.4
10.10.10.4/32 32 PT - 21 0:25:90:93:d2:44(142252)
10.10.10.40/32 0 T - 14 -
10.10.10.41/32 0 T - 14 -
10.10.10.42/32 0 T - 14 -
10.10.10.43/32 0 T - 14 -
10.10.10.44/32 0 T - 14 -
10.10.10.45/32 0 T - 14 -
10.10.10.46/32 0 T - 14 -
10.10.10.47/32 0 T - 14 -
10.10.10.48/32 0 T - 14 -
10.10.10.49/32 0 T - 14 -
root@nodek11:~# ping 10.10.10.4
PING 10.10.10.4 (10.10.10.4) 56(84) bytes of data.
^C
--- 10.10.10.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@nodek11:~# flow --match 10.10.10.4
Flow table(size 80609280, entries 629760)

Entries: Created 413 Added 413 Deleted 676 Changed 691 Processed 413 Used Overflow entries 0
(Created Flows/CPU: 24 23 15 29 25 38 18 26 7 14 10 7 1 4 3 7 5 17 46 4 26 7 20 26 0 3 0 4 1 1 0 2)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([10.10.10.4]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   354588<=>471904 10.204.216.231:7073 1 (0)
                         10.10.10.4:0
(Gen: 1, K(nh):5, Action:F, Flags:, QOS:-1, S(nh):10, Stats:1/84, SPort 59146,
 TTL 0, Sinfo 0.0.0.0)

   471904<=>354588 10.10.10.4:7073 1 (0)
                         10.204.216.231:0
(Gen: 2, K(nh):5, Action:F, Flags:, QOS:-1, S(nh):21, Stats:0/0, SPort 64588,
 TTL 0, Sinfo 0.0.0.0)

root@nodek11:~#

Build Info
-----------

root@nodek11:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-lib 4.1.0.0-16 16
contrail-nodemgr 4.1.0.0-16 16
contrail-setup 4.1.0.0-16 16
contrail-utils 4.1.0.0-16 16
contrail-vrouter-agent 4.1.0.0-16 16
contrail-vrouter-common 4.1.0.0-16 16
contrail-vrouter-dkms 4.1.0.0-16 16
contrail-vrouter-init 4.1.0.0-16 16
contrail-vrouter-utils 4.1.0.0-16 16
nova-common 2:13.0.0-0ubuntu2~cloud0.1contrail1 16
nova-compute 2:13.0.0-0ubuntu2~cloud0.1contrail1 16
nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0.1contrail1 16
nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0.1contrail1 16
python-contrail 4.1.0.0-16 16
python-contrail-vrouter-api 4.1.0.0-16 16
python-neutronclient 1:4.1.1-2~cloud0.2contrail 16
python-nova 2:13.0.0-0ubuntu2~cloud0.1contrail1 16
python-opencontrail-vrouter-netns 4.1.0.0-16 16
root@nodek11:~#

root@nodek11:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
root@nodek11:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35262
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35262
Committed: http://github.com/Juniper/contrail-controller/commit/7b0507e3b3cb7b1e350125b2bd75b0277efd9db4
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7b0507e3b3cb7b1e350125b2bd75b0277efd9db4
Author: Naveen N <email address hidden>
Date: Tue Sep 5 13:08:09 2017 +0530

* Publish floating-ip route with proper encapsulation

1> Correct dependency manager to take care of forwarding-vrf change
2> Remove route in fabric VRF when forwarding vrf config is deleted
3> Pick VN, SG anf tag list from policy fabric VRF only, if route is
not found in policy fabric VRF, use empty list instead of picking
from default VRF.
Test case for same.
Closes-bug:#1711077
Closes-bug:#1712000
Closes-bug:#1711527
Closes-bug:#1712245

Change-Id: Ibee3d79613a118d2e8838bd07b17ca4bca8df186