[1.10-Build 34]Inconsistent state is occurred between WebUI and Control node.

[Daisuke Nakajima]

del_route_target.py is executed during Configuration node Shutdown, but Route target still remained on if-map server even if there is no Route target state on WebUI.
In that case, Gateway router also has VM's IP address with Route target and Control node still has Routet target on the ShowRtGroupResp showing.

Reproduction Procedure
1) reboot Configuration node
2) Delete route target soon after rebooting
   python del_route_target.py --routing_instance_name default-domain:admin:VN1 --route_target_number 10000 --api_server_ip 10.84.50.101 --api_server_port 8082 --router_asn 64512
3) Confirm Virtual machine, WebUI, Control node, Gateway router.

Attached logs:
1) Snh_ShowRtGroupReq
    Control node log
2) Gateway_router.log
   Gateway router routing table
3) WebUI.png
  WebUI Screenshot

Log summery.
### Virtual Network Status ###
Display Name : VN2
Admin State : Up
Shared : Disabled
External : Disabled
Attached Network Policies : -
Floating IP Pools : -
Route Targets : -
Forwarding Mode : L2 and L3
VxLAN Identifier : Automatic
DNS Servers : -
Host Routes : -

### Gateway Router Routing table ###
bgp.l3vpn.0: 3 destinations, 4 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.84.50.5:1:10.0.0.2/32
                   *[BGP/170] 00:00:53, localpref 200, from 10.84.50.3
                      AS path: ?, validation-state: unverified
                    > via gr-1/1/0.33794, Push 16
                    [BGP/170] 00:03:48, localpref 200, from 10.84.50.4
                      AS path: ?, validation-state: unverified
                    > via gr-1/1/0.33794, Push 16
10.84.50.202:47:0.0.0.0/0
                   *[BGP/170] 1w5d 05:29:20, localpref 100, from 10.84.50.202
                      AS path: I, validation-state: unverified
                    > via gr-1/1/0.33793, Push 16
10.84.50.202:47:10.84.45.0/24
                   *[BGP/170] 1w5d 05:29:20, localpref 100, from 10.84.50.202
                      AS path: I, validation-state: unverified
                    > via gr-1/1/0.33793, Push 16

customer-public.inet.0: 4 destinations, 7 routes (4 active, 0 holddown, 0 hidden)
10.0.0.2/32 (2 entries, 1 announced)
        *BGP Preference: 170/-201
                Route Distinguisher: 10.84.50.5:1
                Next hop type: Indirect
                Address: 0x27d9098
                Next-hop reference count: 5
                Source: 10.84.50.3
                Next hop type: Router, Next hop index: 815
                Next hop: via gr-1/1/0.33794, selected
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Session Id: 0x8003c
                Protocol next hop: 10.84.50.5
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Indirect next hop: 0x299c220 1048575 INH Session ID: 0x8003d
                State: <Secondary Active Int Ext ProtectionCand>
                Local AS: 64512 Peer AS: 64512
                Age: 57 Metric2: 0
                Validation State: unverified
                Task: BGP_64512.10.84.50.3+58566
                Announcement bits (1): 1-KRT
                AS path: ?
                Communities: target:64512:10000 target:64512:8000002 unknown iana 30c unknown iana 30c unknown iana 600 unknown type 8004 value 0:1 unknown type 8071 value fc00:5
                Import Accepted
                VPN Label: 16
                Localpref: 200
                Router ID: 10.84.50.3
                Primary Routing Table bgp.l3vpn.0

[Sanju Abraham (asanju)]

The issue is becuase of the the VIP address missing in vnc_api.

1.2 will have this fix.

[venu kolli (vkolli)]

Verified .

root@woodstock# run show route table JDAF.inet.0

JDAF.inet.0: 44 destinations, 84 routes (44 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0 *[Static/5] 1d 07:33:40
                    > via lt-1/0/0.100
10.102.42.0/24 *[Static/5] 1d 07:33:40
                    > via lt-1/0/0.100
11.1.1.0/24 *[Direct/0] 1d 07:35:30
                    > via lo0.10
11.1.1.1/32 *[Local/0] 1d 07:35:30
                      Local via lo0.10
192.168.1.2/32 *[BGP/170] 00:00:49, localpref 100, from 5.5.5.13
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32772, Push 16
                    [BGP/170] 00:00:49, localpref 100, from 5.5.5.14
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32772, Push 16
192.168.1.3/32 *[BGP/170] 00:00:49, localpref 100, from 5.5.5.13
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32771, Push 16
                    [BGP/170] 00:00:49, localpref 100, from 5.5.5.14
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32771, Push 16
192.168.1.4/32 *[BGP/170] 00:00:49, localpref 100, from 5.5.5.13
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32772, Push 18
                    [BGP/170] 00:00:49, localpref 100, from 5.5.5.14
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32772, Push 18
192.168.1.5/32 *[BGP/170] 00:00:49, localpref 100, from 5.5.5.13
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32771, Push 18
                    [BGP/170] 00:00:49, localpref 100, from 5.5.5.14
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32771, Push 18
192.168.1.6/32 *[BGP/170] 00:00:49, localpref 100, from 5.5.5.13
                      AS path: ?, validation-state: unverified
                    > via gr-1/0/0.32772, Push 20

root@vse2100-2:/opt/contrail/utils# python del_route_target.py --routing_instance_name default-domain:admin:net-test --route_target_number 100000 --api_server_ip 10.102.42.200 --api_server_port 8082 --router_asn 64512

root@woodstock# run show route table JDAF.inet.0

JDAF.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0 *[Static/5] 1d 07:52:54
                    > via lt-1/0/0.100
10.102.42.0/24 *[Static/5] 1d 07:52:54
                    > via lt-1/0/0.100
11.1.1.0/24 *[Direct/0] 1d 07:54:44
                    > via lo0.10
11.1.1.1/32 *[Local/0] 1d 07:54:44
                      Local via lo0.10

[edit]
root@woodstock#

Route are removed from route target after executing the del_route_target api.

[Sanju Abraham (asanju)]

Fix was in updating keepalived to the version that supports vmac_xmit_which will cause the VRRP messages to be transmitted and received on the underlying interface whilst ARP will happen from the the VMAC interface

https://review.opencontrail.org/#/c/3393/

[venu kolli (vkolli)]

Verified in 2.0-2464