VrfTimeout() in agent due to reference to VN in InterfaceUveTable::UveInterfaceState

[Praveen]

(gdb) dump_vrf_entries
--------------------------------------------
0x7fa428088450 default-domain:default-project:ip-fabric:__default__ idx=0 ref_count=23 flags=0 rt_db=(nil) mcrt_db=0x7fa428088920 layer2_db=0x7fa42808a6f0
0x7fa3c8007ed0 default-domain:demo:floating:floating idx=2 ref_count=3 flags=2 rt_db=(nil) mcrt_db=0x7fa3c8008090 layer2_db=0x7fa3c8009b10

(gdb) dump_vn_entries
--------------------------------------------
0x7fa3c8007160 default-domain:demo:floating 譴8�=FѯGs>�qF�xO

(gdb) dump_intf_entries
--------------------------------------------
0x7fa4280a28f0 eth3 flags=0 ref=8
0x7fa3dc004f60 tap0625cd0d-71 flags=0 ref=0
0x7fa3dc005a60 tap0f874b79-f7 flags=0 ref=0
0x7fa3dc005fe0 tapa732df20-df flags=0 ref=0
0x7fa3dc0054e0 tapb0bc882d-05 flags=0 ref=0
0x7fa4280a31e0 vhost0 flags=0 ref=3
0x7fa4280a5e70 pkt0 flags=0 ref=1

VRFEntry is deleted but has ref-count of 3

$13 = (VrfEntry) {
  <AgentRefCount<VrfEntry>> = {
    _vptr.AgentRefCount = 0x1c6be10,
    refcount_ = (tbb::atomic) 3
  },
  <AgentDBEntry> = {
    <DBEntry> = {
      <DBEntryBase> = {
        _vptr.DBEntryBase = 0x1c6be68,
        chg_list_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::list_node<void*>> = {
              next_ = 0x0,
              prev_ = 0x0
            }, <No data fields>}, <No data fields>},
        tpart_ = 0x7fa42801b5b0,
        state_ = std::map with 1 elements = {
          [0] = 0x7fa3c8006f70
        },
        flags = 2 '\002',
        onremoveq_ = (tbb::atomic) 0 '\000',
        last_change_at_ = 1434268734204137
      },
      members of DBEntry:
      node_ = {
        <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
          <boost::intrusive::detail::no_default_definer> = {<No data fields>},
          <boost::intrusive::rbtree_node<void*>> = {
            parent_ = 0x7fa42801b638,
            left_ = 0x7fa4280884c0,
            right_ = 0x0,
            color_ = boost::intrusive::rbtree_node<void*>::black_t
          }, <No data fields>}, <No data fields>}
    },
    members of AgentDBEntry:
    flags_ = 0 '\000'
  },
  members of VrfEntry:
  static kInvalidIndex = 4294967295,
  static kDeleteTimeout = 900000,
  name_ = "default-domain:demo:floating:floating",
  id_ = 2,

VnEntry is deleted with ref-count of 2

(gdb) p *(VnEntry *) 0x7fa3c8007160
$14 = (VnEntry) {
  <AgentRefCount<VnEntry>> = {
    _vptr.AgentRefCount = 0x1c6a250,
    refcount_ = (tbb::atomic) 2
  },
  <AgentOperDBEntry> = {
    <AgentDBEntry> = {
      <DBEntry> = {
        <DBEntryBase> = {
          _vptr.DBEntryBase = 0x1c6a2a0,
          chg_list_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::list_node<void*>> = {
                next_ = 0x0,
                prev_ = 0x0
              }, <No data fields>}, <No data fields>},
          tpart_ = 0x7fa428020120,
          state_ = std::map with 1 elements = {
            [0] = 0x7fa3c80066a0
          },
          flags = 2 '\002',
          onremoveq_ = (tbb::atomic) 0 '\000',
          last_change_at_ = 1434268734203840
        },
        members of DBEntry:
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::rbtree_node<void*>> = {
              parent_ = 0x7fa4280201a8,
              left_ = 0x0,
              right_ = 0x0,
              color_ = boost::intrusive::rbtree_node<void*>::black_t
            }, <No data fields>}, <No data fields>}
      },
      members of AgentDBEntry:
      flags_ = 0 '\000'
    },
    members of AgentOperDBEntry:
    ifmap_node_state_ = (boost::intrusive_ptr<IFMapNodeState>) 0x0
  },
  members of VnEntry:
  agent_ = 0x371f500,
---Type <return> to continue, or q <return> to quit---
  uuid_ = (boost::uuids::uuid) e8adb438-f83d-46d1-af47-733e937146a5,
  name_ = "default-domain:demo:floating",
  acl_ = (boost::intrusive_ptr<AclDBEntry>) 0x0,
  mirror_acl_ = (boost::intrusive_ptr<AclDBEntry>) 0x0,
  mirror_cfg_acl_ = (boost::intrusive_ptr<AclDBEntry>) 0x0,
  vrf_ = (boost::intrusive_ptr<VrfEntry>) 0x7fa3c8007ed0,

VmInterfaces are not deleted. They dont have any floating-ip also

(gdb) p *(VmInterface *) 0x7fa3dc004f60
$15 = (VmInterface) {
  <Interface> = {
    <AgentRefCount<Interface>> = {
      _vptr.AgentRefCount = 0x1c68970,
      refcount_ = (tbb::atomic) 0
    },
    <AgentOperDBEntry> = {
      <AgentDBEntry> = {
        <DBEntry> = {
          <DBEntryBase> = {
            _vptr.DBEntryBase = 0x1c689f0,
            chg_list_ = {
              <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
                <boost::intrusive::detail::no_default_definer> = {<No data fields>},
                <boost::intrusive::list_node<void*>> = {
                  next_ = 0x0,
                  prev_ = 0x0
                }, <No data fields>}, <No data fields>},
            tpart_ = 0x7fa428017fa0,
            state_ = std::map with 7 elements = {
              [1] = 0x7fa3dc003c50,
              [3] = 0x7fa3c8048250,
              [5] = 0x7fa3dc002790,
              [6] = 0x7fa3dc004a80,
              [7] = 0x7fa41802bad0,
              [9] = 0x7fa3dc0066b0,
              [16] = 0x7fa3dc006960
            },
            flags = 0 '\000',
            onremoveq_ = (tbb::atomic) 0 '\000',
            last_change_at_ = 1434268734171532
          },
          members of DBEntry:
          node_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::rbtree_node<void*>> = {
                parent_ = 0x7fa4280a3250,
                left_ = 0x7fa4280a2960,
                right_ = 0x7fa3dc006050,
                color_ = boost::intrusive::rbtree_node<void*>::red_t
              }, <No data fields>}, <No data fields>}
        },
        members of AgentDBEntry:
---Type <return> to continue, or q <return> to quit---
        flags_ = 0 '\000'
      },
      members of AgentOperDBEntry:
      ifmap_node_state_ = (boost::intrusive_ptr<IFMapNodeState>) 0x7fa3dc008980
    },
    members of Interface:
    static kInvalidIndex = 4294967295,
    type_ = Interface::VM_INTERFACE,
    uuid_ = (boost::uuids::uuid) 0625cd0d-716c-41c0-89d8-63bfc5e89af8,
    name_ = "tap0625cd0d-71",
    vrf_ = (boost::intrusive_ptr<VrfEntry>) 0x0,
    label_ = 4294967295,
    l2_label_ = 4294967295,
    ipv4_active_ = false,
    ipv6_active_ = false,
    l2_active_ = false,
    id_ = 3,
    dhcp_enabled_ = false,
    dns_enabled_ = false,
    mac_ = 0a:e1:94:9c:e4:f8,
    os_index_ = 14,
    os_oper_state_ = true,
    admin_state_ = true,
    test_oper_state_ = true,
    flow_key_nh_ = (boost::intrusive_ptr<NextHop const>) 0x0,
    transport_ = Interface::TRANSPORT_ETHERNET,
    table_ = 0x0
  },
  members of VmInterface:
  static kInvalidVlanId = 65535,
  static kInvalidPmdId = 65535,
  vm_ = (boost::intrusive_ptr<VmEntry>) 0x0,
  vn_ = (boost::intrusive_ptr<VnEntry>) 0x0,
  ip_addr_ = {
    addr_ = {
      s_addr = 67240450
    }
  },
  mdata_addr_ = {
    addr_ = {
      s_addr = 50396841
    }
  },
  subnet_bcast_addr_ = {
    addr_ = {
      s_addr = 0
---Type <return> to continue, or q <return> to quit---
    }
  },
  ip6_addr_ = {
    addr_ = {
      __in6_u = {
        __u6_addr8 = '\000' <repeats 15 times>,
        __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
        __u6_addr32 = {0, 0, 0, 0}
      }
    },
    scope_id_ = 0
  },
  vm_mac_ = "02:06:25:cd:0d:71",
  policy_enabled_ = false,
  mirror_entry_ = (boost::intrusive_ptr<MirrorEntry>) 0x0,
  mirror_direction_ = Interface::UNKNOWN,
  cfg_name_ = "",
  fabric_port_ = false,
  need_linklocal_ip_ = false,
  dhcp_enable_ = false,
  do_dhcp_relay_ = false,
  vm_name_ = "test2_vms-818f56ba-fcf0-4804-9ffb-ce53a517dc41",
  vm_project_uuid_ = (boost::uuids::uuid) 6181bd0d-04f1-4fc5-9d40-7613e4179586,
  vxlan_id_ = 0,
  bridging_ = true,
  layer3_forwarding_ = false,
  flood_unknown_unicast_ = false,
  mac_set_ = true,
  ecmp_ = false,
  tx_vlan_id_ = 65535,
  rx_vlan_id_ = 65535,
  parent_ = (boost::intrusive_ptr<Interface>) 0x0,
  local_preference_ = VmInterface::INVALID,
  oper_dhcp_options_ = {
    _vptr.OperDhcpOptions = 0x1c4ee90,
    dhcp_options_ = std::vector of length 0, capacity 0,
    host_routes_ = std::vector of length 0, capacity 0
  },
  sg_list_ = {
    list_ = std::set with 0 elements
  },
  floating_ip_list_ = {
    v4_count_ = 0,
    v6_count_ = 0,
    list_ = std::set with 0 elements
  },
---Type <return> to continue, or q <return> to quit---
  service_vlan_list_ = {
    list_ = std::set with 0 elements
  },
  static_route_list_ = {
    list_ = std::set with 0 elements
  },
  allowed_address_pair_list_ = {
    list_ = std::set with 0 elements
  },
  peer_ = {
    _M_ptr = 0x7fa3dc004670
  },
  vrf_assign_rule_list_ = {
    list_ = std::set with 0 elements
  },
  vrf_assign_acl_ = (boost::intrusive_ptr<AclDBEntry>) 0x0,
  vm_ip_gw_addr_ = {
    addr_ = {
      s_addr = 16908802
    }
  },
  vm_ip6_gw_addr_ = {
    addr_ = {
      __in6_u = {
        __u6_addr8 = '\000' <repeats 15 times>,
        __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
        __u6_addr32 = {0, 0, 0, 0}
      }
    },
    scope_id_ = 0
  },
  device_type_ = VmInterface::VM_ON_TAP,
  vmi_type_ = VmInterface::INSTANCE,
  configurer_ = 1 '\001',
  subnet_ = {
    addr_ = {
      s_addr = 0
    }
  },
  subnet_plen_ = 0 '\000',
  ethernet_tag_ = 2002,
  logical_interface_ = (boost::uuids::uuid) 00000000-0000-0000-0000-000000000000
}

InterfaceUveTable::UveInterfaceState has valid fip_list_. It has reference to VnEntry

(gdb) p *(InterfaceUveTable::UveInterfaceState *) 0x7fa41802bad0
$16 = (InterfaceUveTable::UveInterfaceState) {
  <DBState> = {
    _vptr.DBState = 0x1cd7c90
  },
  members of InterfaceUveTable::UveInterfaceState:
  cfg_name_ = "default-domain:demo:0625cd0d-716c-41c0-89d8-63bfc5e89af8",
  fip_list_ = std::set with 1 elements = {
    [0] = {
      <VmInterface::ListEntry> = {
        _vptr.ListEntry = 0x1c687d0,
        installed_ = true,
        del_pending_ = false
      },
      members of VmInterface::FloatingIp:
      floating_ip_ = {
        type_ = boost::asio::ip::address::ipv4,
        ipv4_address_ = {
          addr_ = {
            s_addr = 2229128202
          }
        },
        ipv6_address_ = {
          addr_ = {
            __in6_u = {
              __u6_addr8 = '\000' <repeats 15 times>,
              __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
              __u6_addr32 = {0, 0, 0, 0}
            }
          },
          scope_id_ = 0
        }
      },
      vn_ = (boost::intrusive_ptr<VnEntry>) 0x7fa3c8007160,
      vrf_ = (boost::intrusive_ptr<VrfEntry>) 0x7fa3c8007ed0,
      vrf_name_ = "default-domain:demo:floating:floating",
      vn_uuid_ = (boost::uuids::uuid) e8adb438-f83d-46d1-af47-733e937146a5,
      l2_installed_ = true,
      ethernet_tag_ = 0
    }
  }
}

Summary:
--------------
Agent had VmInterface with floating-ip entries first. Later, config is deleted for all VN, VRF, VMI and Floating-IP.

However, nova-vif-driver has not deleted the interface. In this case,

1. (InterfaceUveTable::UveInterfaceState *) holds reference to VN
2. DBState is not cleared for VMInterface by IFMapNodeState

[Praveen (praveen-karadakal)]

Core file and unstripped binary at bhushana@mayamruga:~/Documents/technical/bugs/1466176

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11767
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11927
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11928
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11927
Committed: http://github.org/Juniper/contrail-controller/commit/4aba5649de8c636ea9518443c188664bfe0ab852
Submitter: Zuul
Branch: master

commit 4aba5649de8c636ea9518443c188664bfe0ab852
Author: ashoksingh <email address hidden>
Date: Mon Jun 22 17:06:47 2015 +0530

Fix issue of VN reference not being removed from InterfaceUveTable::UveInterfaceState

InterfaceUveTable::UveInterfaceState has floating_ip_list which has reference to VN. If interface delete from Nova has not come and config delete for that interface comes (which includes removal of floating IPs), we were not cleaning the floating_ip_list from InterfaceUveTable::UveInterfaceState.

Change-Id: I35e1b49e35e3fc398df96c1ad682237c82e8c177
Closes-Bug: #1466176

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11928
Committed: http://github.org/Juniper/contrail-controller/commit/e228d3c7e5423ee5885e23dea3dd7a61b21621ed
Submitter: Zuul
Branch: R2.20

commit e228d3c7e5423ee5885e23dea3dd7a61b21621ed
Author: ashoksingh <email address hidden>
Date: Mon Jun 22 17:06:47 2015 +0530

Fix issue of VN reference not being removed from InterfaceUveTable::UveInterfaceState

InterfaceUveTable::UveInterfaceState has floating_ip_list which has reference to VN. If interface delete from Nova has not come and config delete for that interface comes (which includes removal of floating IPs), we were not cleaning the floating_ip_list from InterfaceUveTable::UveInterfaceState.

Closes-Bug: #1466176
(cherry picked from commit 4aba5649de8c636ea9518443c188664bfe0ab852)

Change-Id: I0220edeecb4ad1f65c5b8e58a474f7b9e6888ecc

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13927
Submitter: Vinay Vithal Mahuli (<email address hidden>)