Juniper Openstack

Tor Agent : interfaces missing in OVSDB

  1. Series trunk
  2. Bug #1471201
Bug #1471201 reported by Hari Prasad Killi
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r2.21
Trunk
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r3.0-fcs

Bug Description

TOR-Agent ifmap configuration had all information about configured logical interfaces and their dependencies (50 logical-interfaces to xe-0/0/6 - 25, for a total of 1000 logical-interfaces). Some of them were missing in the OVSDB.

root@QFX1> show ovsdb interface | match xe-0/0/6 | count
Count: 37 lines

root@QFX1> show ovsdb interface | match xe-0/0/7 | count
Count: 50 lines

In TOR-Agent Vlan Port Binding, some entries were as follows, with physical_port is shown empty. There were 42 such entries.

<state type="string" identifier="1">Add defer(2)</state>
          <physical_port type="string" identifier="2"/>
          <physical_device type="string" identifier="3">QFX1</physical_device>
          <logical_switch type="string" identifier="4">00000000-0000-0000-0003-000000000014</logical_switch>
          <vlan type="u16" identifier="5">3003</vlan>

Also, xe-0/0/0.2001 and xe-0/0/2.2001 were missing in the OVSDB, while TOR-agent ifmap had their information present.

Hari Prasad Killi (haripk)
Changed in juniperopenstack:
milestone: none → r2.30-fcs
tags: added: customer
Revision history for this message
Hari Prasad Killi (haripk) wrote :

gcore of tor-agent is available at /data/bugs/1471201 on mayamruga.

Revision history for this message
Prabhjot Singh Sethi (prabhjot) wrote :
Download full text (4.0 KiB)

interface xe-0/0/0.2001 is present in OVSDB client but stuck in ADD Defer due to same issue of physical port being empty.
(gdb) p *(DBState *) 0x7f69f002c8f0
$4 = (OVSDB::VlanPortBindingEntry) {
  <OVSDB::OvsdbDBEntry> = {
    <KSyncDBEntry> = {
      <KSyncEntry> = {
        _vptr.KSyncEntry = 0xe22df0 <vtable for OVSDB::VlanPortBindingEntry+16>,
        static kInvalidIndex = 4294967295,
        node_ = <boost::intrusive_hook> parent = 0x7f6a2c3080a8 left = 0x0 right = 0x0,
        index_ = 4294967295,
        state_ = KSyncEntry::ADD_DEFER,
        refcount_ = (tbb::atomic) 2,
        seen_ = false,
        stale_ = false
      },
      <DBState> = {
        _vptr.DBState = 0xe22ec0 <vtable for OVSDB::VlanPortBindingEntry+224>
      },
      members of KSyncDBEntry:
      db_entry_ = 0x7f6a0010f9c0,
      dup_entry_list_ = empty std::list
    },
    <OVSDB::OvsdbEntryBase> = {
      _vptr.OvsdbEntryBase = 0xe22ee0 <vtable for OVSDB::VlanPortBindingEntry+256>,
      ack_event_ = KSyncEntry::ADD_CHANGE_REQ
    },
    members of OVSDB::OvsdbDBEntry:
    table_ = 0x7f69e000a020,
    ovs_entry_ = 0x0
  },
  members of OVSDB::VlanPortBindingEntry:
  logical_switch_ = (boost::intrusive_ptr<KSyncEntry>) 0x0,
  physical_port_ = (boost::intrusive_ptr<KSyncEntry>) 0x0,
  logical_switch_name_ = "aa91642d-1b0b-4527-8d9d-786a3fe0d692",
  physical_port_name_ = "",
  physical_device_name_ = "QFX1",
  vlan_ = 2001,
  vmi_uuid_ = (boost::uuids::uuid) fd8237aa-489e-41b6-bbc1-55537bfb8df6,
  old_logical_switch_name_ = ""
}

however for interface xe-0/0/2.2001 is missing from OVSDB introspect since there is no association of this logical interface to corresponding VMI, physical port or physical device
(gdb) p *(DBEntry *) 0x7f69e4016c30
$5 = (VlanLogicalInterface) {
  <LogicalInterface> = {
    <Interface> = {
      <AgentRefCount<Interface>> = {
        _vptr.AgentRefCount = 0xdffb10 <vtable for VlanLogicalInterface+16>,
        refcount_ = (tbb::atomic) 0
      },
      <AgentOperDBEntry> = {
        <AgentDBEntry> = {
          <DBEntry> = {
            <DBEntryBase> = {
              _vptr.DBEntryBase = 0xdffb90 <vtable for VlanLogicalInterface+144>,
              chg_list_ = <boost::intrusive_hook> next = 0x0 prev = 0x0,
              tpart_ = 0x7f6a2c0176f0,
              state_ = std::map with 1 elements = {
                [6] = 0x7f6a3883c8c0
              },
              flags = 0 '\000',
              onremoveq_ = (tbb::atomic) false,
              last_change_at_ = 1435893659364960
            },
            members of DBEntry:
            node_ = <boost::intrusive_hook> parent = 0x7f69dc3e84f0 left = 0x0 right = 0x0
          },
          members of AgentDBEntry:
          flags_ = 0 '\000'
        },
        members of AgentOperDBEntry:
        ifmap_node_state_ = (boost::intrusive_ptr<IFMapNodeState>) 0x7f69bc4f51e0
      },
      members of Interface:
      static kInvalidIndex = 4294967295,
      type_ = Interface::LOGICAL,
      uuid_ = (boost::uuids::uuid) cd823f4d-6ae9-49f5-a93b-cb749c72586d,
      name_ = "default-global-system-config:QFX1:xe-0/0/2:xe-0/0/2.2001",
      vrf_ = (boost::intrusive_ptr<VrfEntry>) 0x0,
      label_ = 42949672...

Read more...

Prabhjot Singh Sethi (prabhjot)
Changed in juniperopenstack:
status: New → In Progress
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/12293
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/12300
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
Praveen (praveen-karadakal) wrote :
Download full text (8.0 KiB)

Update for interface xe-0/0/2.2001

The objects in error state are,
LI = default-global-system-config:QFX1:xe-0/0/2:xe-0/0/2.2001
VMI = default-domain:tky:7c73d281-705f-41d4-a7a3-2a7b4e2d705b

There is inconsistency between the ifmap output and the gcore file. IFMap output has adjacency between LI and VMI but not in the gcore.

Trace buffer 'IFMapAgentTrace' has following messages,

2015-07-03 09:06:31.124416 IFMapAgentTrace: seq = 1199 event = "Deleting Link between default-global-system-config:QFX1:xe-0/0/2:xe-0/0/2.2001default-domain:tky:7c73d281-705f-41d4-a7a3-2a7b4e2d705b" file = "controller/src/ifmap/ifmap_agent_table.cc" line = 605
2015-07-03 09:06:31.124419 IFMapAgentTrace: seq = 1199 event = "Deleting Link between default-global-system-config:QFX1:xe-0/0/2:xe-0/0/2.2001default-global-system-config:QFX1:xe-0/0/2" file = "controller/src/ifmap/ifmap_agent_table.cc" line = 605

The link between VMI and LI is deleted at 09:06:31 by stale timer. We dont have enough information to find why stale timer deleted the configuration.

XmppTraceBuffer contains logs from 08:50:00 to >10:00:00 and no new links between VMI and LI are received after this time. This is confirmed from DBGraph also. Dont have enough information to find why links got deleted by stale timer audit.

The vertex_id_ for interface is 0x7f69b000dbb0. This vertex does not have any adjacency to logical-interfaces.

IFMapNodeState for VMI is 0x7f69b000f730
IFMapNode for VMI is 0x7f69b000fba0
vertex_id_ for VMI is 0x7f69b000dbb0

The adjacencies are,
(gdb) p *(IFMapLink *) 0x7f69b000e4d0
$1 = (IFMapLink) {
  <DBGraphEdge> = {
    <DBEntry> = {
      <DBEntryBase> = {
        _vptr.DBEntryBase = 0xe99bd0 <vtable for IFMapLink+16>,
        chg_list_ = <boost::intrusive_hook> next = 0x0 prev = 0x0,
        tpart_ = 0x7f6a2c0152b0,
        state_ = std::map with 0 elements,
        flags = 0 '\000',
        onremoveq_ = (tbb::atomic) false,
        last_change_at_ = 1435307483974210
      },
      members of DBEntry:
      node_ = <boost::intrusive_hook> parent = 0x7f6a2c015338 left = 0x7f69e801a4e0 right = 0x7f69c40a5b00
    },
    members of DBGraphEdge:
    edge_id_ = {
      <boost::detail::edge_base<boost::undirected_tag, void*>> = {
        m_source = 0x7f69b000dbb0,
        m_target = 0x7f6a00015180
      },
      members of boost::detail::edge_desc_impl<boost::undirected_tag, void*>:
      m_eproperty = 0x7f69b000dd30
    }
  },
  members of IFMapLink:
  link_name_ = "virtual-machine-interface-security-group,virtual-machine-interface:default-domain:tky:7c73d281-705f-41d4-a7a3-2a7b4e2d705b,security-group:default-domain:tky:default",
  metadata_ = "virtual-machine-interface-security-group",
  left_id_ = {
    first = "virtual-machine-interface",
    second = "default-domain:tky:7c73d281-705f-41d4-a7a3-2a7b4e2d705b"
  },
  right_id_ = {
    first = "security-group",
    second = "default-domain:tky:default"
  },
  left_node_ = 0x7f69b000fba0,
  right_node_ = 0x7f6a0001bb00,
  origin_info_ = std::vector of length 1, capacity 1 = {{
      origin = {
        origin = IFMapOrigin::UNKNOWN
      },
      sequence_number = 1209
    }}
}
(gdb) p *(IFMapLink *) 0x7f69b0010080
$2 = (IFMapL...

Read more...

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/12293
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/12300
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/12293
Committed: http://github.org/Juniper/contrail-controller/commit/a0ee99004c86575cfdb9672ae47fb975cc20b180
Submitter: Zuul
Branch: master

commit a0ee99004c86575cfdb9672ae47fb975cc20b180
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Jul 9 12:22:58 2015 +0530

Handle display name change from empty to non-empty

Issue:
------
In certain configuration update scenarios we may receive
Physical Device or Physical Interface entry without display
name, however while translating this information to OVSDB
we need to mandatory have display name available, not
having display name available was not handled.

Fix:
----
Ignore logical interface object till display name for both
Physical Device and Interface is available, added change
to trigger updates to logical interface on display name
change using dependency tracker.

Fix propagation of Physical Device display name to Physical
Device VN as well.

Fixed test case to check for vxlan-id of ovs-logical-switch
validation

Added test case to simulate order in which display name for
Physical Device and Physical Interface is not available
during creation time.

Partial-Bug: 1471201
Change-Id: I958792a7c28da0e5019c5342dd1e7bdc473aae9d

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/12300
Committed: http://github.org/Juniper/contrail-controller/commit/779a443559603d4039cbe0ed16e3bd9b8fb7c4f5
Submitter: Zuul
Branch: R2.20

commit 779a443559603d4039cbe0ed16e3bd9b8fb7c4f5
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Jul 9 12:22:58 2015 +0530

Handle display name change from empty to non-empty

Issue:
------
In certain configuration update scenarios we may receive
Physical Device or Physical Interface entry without display
name, however while translating this information to OVSDB
we need to mandatory have display name available, not
having display name available was not handled.

Fix:
----
Ignore logical interface object till display name for both
Physical Device and Interface is available, added change
to trigger updates to logical interface on display name
change using dependency tracker.

Fix propagation of Physical Device display name to Physical
Device VN as well.

Fixed test case to check for vxlan-id of ovs-logical-switch
validation

Added test case to simulate order in which display name for
Physical Device and Physical Interface is not available
during creation time.

Partial-Bug: 1471201
Change-Id: I958792a7c28da0e5019c5342dd1e7bdc473aae9d
(cherry picked from commit a0ee99004c86575cfdb9672ae47fb975cc20b180)

Linda Lee (lindalee)
tags: added: bms
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22-dev

Review in progress for https://review.opencontrail.org/13927
Submitter: Vinay Vithal Mahuli (<email address hidden>)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.