[Mainline-Build 2667]: lots of errored veth interfaces in agent introspect

[alok kumar]

there are lots of errored veth interfaces in agent introspect.(on docker host)
looks like deleted container's veth interfaces are not cleaned up.

vif cmd show 3 interfaces with vrf 65535 but in introspect there are many more errored interfaces.

root@nodec62:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 OS: p1p1 (Speed 1000, Duplex 1)
            Type:Physical HWaddr:00:25:90:c5:59:8a IPaddr:0
            Vrf:0 Flags:L3L2Vp MTU:1514 Ref:29
            RX packets:6658508 bytes:525299115 errors:0
            TX packets:986464 bytes:353956111 errors:0

vif0/1 OS: vhost0
            Type:Host HWaddr:00:25:90:c5:59:8a IPaddr:accd966
            Vrf:0 Flags:L3L2 MTU:1514 Ref:3
            RX packets:924633 bytes:338407570 errors:0
            TX packets:6609946 bytes:510492664 errors:0

vif0/2 OS: pkt0
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3 MTU:1514 Ref:2
            RX packets:26424 bytes:2602336 errors:0
            TX packets:3941387 bytes:394182302 errors:0

vif0/98 OS: NULL
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3L2D MTU:9160 Ref:1
            RX packets:1162 bytes:143861 errors:0
            TX packets:638 bytes:64838 errors:0

vif0/99 OS: NULL
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3L2D MTU:9160 Ref:1
            RX packets:46 bytes:7742 errors:0
            TX packets:324 bytes:36459 errors:0

vif0/100 OS: NULL
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3L2D MTU:9160 Ref:1
            RX packets:11 bytes:794 errors:0
            TX packets:4 bytes:224 errors:0

vif0/4350 OS: pkt3
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0
            Vrf:65535 Flags:L3L2 MTU:9136 Ref:1
            RX packets:5 bytes:434 errors:0
            TX packets:5 bytes:364 errors:0

vif0/4351 OS: pkt1
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0
            Vrf:65535 Flags:L3L2 MTU:9136 Ref:1
            RX packets:0 bytes:0 errors:0
            TX packets:0 bytes:0 errors:0

http://nodec62.englab.juniper.net:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=

please see the snapshot attached.

setup info:

env.roledefs = {
    'all': [host1,host2,host3,host4],
    'cfgm': [host1],
    'openstack':[host2],
    'control': [host1,host2],
    'compute': [host3,host4],
    'collector': [host1,host2],
    'webui': [host1],
    'database': [host1],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec64', 'nodeg18', 'nodec62', 'nodec12']
}

[Ashok Singh (ashoksr)]

Agent has not received delete for 7faf9421-e7ff-4a2f-86ed-726b77939625 which is in error state.

http://nodec62:8085/Snh_SandeshTraceRequest?x=Config

2015-11-10 16:10:08.450 CfgOpenstackConnect: op = Connect version = 1 file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 157
2015-11-10 16:11:08.273 CfgIntfInfo: InterfaceConfigStaleCleaner Walk invoked. controller/src/vnsw/agent/openstack/instance_service_server.cc 695
2015-11-10 16:11:08.273 CfgIntfWalkDone: 1 controller/src/vnsw/agent/openstack/instance_service_server.cc 723
2015-11-10 16:42:42.646 CfgOpenstackAddPort: op = Add port_uuid = 7faf9421-e7ff-4a2f-86ed-726b77939625 instance_uuid = 623ee4ec-7c92-422c-909f-8b23562d6867 vn_uuid = 8da702a5-5bf0-4f93-842d-1b24ae3742cd ip_address = 10.1.1.3 tap_name = veth7faf9421 mac_address = 02:7f:af:94:21:e7 display_name = vmc0 hostname = vmc0 host = version = 1 tx_vlan_id = -1 rx_vlan_id = -1 vm_project_uuid = b9862e78-28f1-47f4-aa84-3f2fe15769f7 port_type = CfgIntVMPort ip6_address = file = controller/src/vnsw/agent/openstack/instance_service_server.cc line = 143
2015-11-10 16:42:42.646 CfgIntfTrace: intf = veth7faf9421 vm_name = vmc0 vm = 623ee4ec-7c92-422c-909f-8b23562d6867 vn = 8da702a5-5bf0-4f93-842d-1b24ae3742cd ip = 10.1.1.3 op = ADD version = 1 tx_vlan_id = 65535 rx_vlan_id = 65535 vm_project = b9862e78-28f1-47f4-aa84-3f2fe15769f7 port_type = CfgIntVMPort ip6 = :: file = controller/src/vnsw/agent/cfg/cfg_interface.cc line = 125

[alok kumar (kalok)]

ok. the bug may be in docker driver. assigning it to Ignatious.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15344
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15344
Committed: http://github.org/Juniper/contrail-packaging/commit/7648a511b8646f789e92ad1f5db4df59fdb5bcc8
Submitter: Zuul
Branch: master

commit 7648a511b8646f789e92ad1f5db4df59fdb5bcc8
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sun Nov 22 16:01:19 2015 -0800

Upgrading lxc-docker package to latest.
Closes-Bug: 1514780

Change-Id: Ic79efeb97b62605ec68ee9a8fc59aae4ed4ff59a

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20216
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20216
Committed: http://github.org/Juniper/contrail-packaging/commit/7c0b75ddaa7efc6a48b416ea3364d721dfcf7d01
Submitter: Zuul
Branch: R2.22.x

commit 7c0b75ddaa7efc6a48b416ea3364d721dfcf7d01
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri May 13 15:45:52 2016 -0700

Packaging nova-docker package with changes to use http based
vrouter-port-control script to plug/unplug vif's in the vrouter.
This script ensures that port info is persisted in file.
Agent reads those files and re-adds port during restarts.

Upgrading lxc-docker package to latest.

Change-Id: I72c77a267662d7d100c09e968d79a8c069c5962f
Closes-Bug: 1533037
Closes-Bug: 1514780