Bug #1466825 “agent/tor-agents did not get any ifmap from contro...” : Bugs : Juniper Openstack

Revision history for this message

Prakash Bailkeri (prakashmb) wrote on 2015-06-19:

#1

Download full text (9.5 KiB)

Taking nodei38-1 as XmppClient(It is a ToR agent)
From the stats(Session stats and ifmap stats), it is clear that config is not set to XmppClient. The Subscribe message is rxed from the client.(It is visible in the trace and also in stats)

gdb) p *(IFMapClient *) 0x7f9557eacde0
$17 = (IFMapXmppChannel::IFMapSender) {
  <IFMapClient> = {
    _vptr.IFMapClient = 0xae3cd0 <vtable for IFMapXmppChannel::IFMapSender+16>,
    static kIndexInvalid = -1,
    index_ = 4,
    exporter_ = 0x1449f30,
    msgs_sent_ = 0,
    msgs_blocked_ = 0,
    bytes_sent_ = 0,
    nodes_sent_ = 0,
    links_sent_ = 0,
    send_is_blocked_ = false,
    vm_map_ = std::map with 0 elements,
    name_ = "nodei36:192.168.1.6"
  },
  members of IFMapXmppChannel::IFMapSender:
  parent_ = 0x7f94416ac9f0,
  hostname_ = "nodei38-1",
  identifier_ = "default-global-system-config:nodei38-1"
}
(gdb) p *(IFMapXmppChannel*) 0x7f94416ac9f0
$18 = (IFMapXmppChannel) {
  _vptr.IFMapXmppChannel = 0xae3d30 <vtable for IFMapXmppChannel+16>,
  peer_id_ = xmps::CONFIG,
  channel_ = 0x1f2df10,
  ifmap_server_ = 0x7fff29159400,
  ifmap_channel_manager_ = 0x7fff291591a0,
  ifmap_client_ = 0x7f9557eacde0,
  client_added_ = true,
  channel_name_ = "nodei36:192.168.1.6"
}

(gdb) p *(XmppChannel *) 0x1f2df10
$19 = (XmppChannelMux) {
  <XmppChannel> = {
    _vptr.XmppChannel = 0xb6d9f0 <vtable for XmppChannelMux+16>
  },
  members of XmppChannelMux:
  map_ = std::map with 0 elements,
  rxmap_ = std::map with 2 elements = {
    [xmps::CONFIG] = {
      <boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>> = {
        <boost::function_base> = {
          vtable = 0xae3331 <void boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>::assign_to<boost::_bi::bind_t<void, boost::_mfi::mf1<void, IFMapXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<IFMapXmppChannel*>, boost::arg<1> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, IFMapXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<IFMapXmppChannel*>, boost::arg<1> > >)::stored_vtable+1>,
          functor = {
            obj_ptr = 0x11,
            type = {
              type = 0x11,
              const_qualified = false,
              volatile_qualified = false
            },
            func_ptr = 0x11,
            bound_memfunc_ptr = {
              memfunc_ptr = &virtual table offset 16,
              obj_ptr = 0x7f94416ac9f0
            },
            obj_ref = {
              obj_ptr = 0x11,
              is_const_qualified = false,
              is_volatile_qualified = false
            },
            data = 17 '\021'
          }
        },
        <std::binary_function<XmppStanza::XmppMessage const*, xmps::PeerState, void>> = {<No data fields>},
        members of boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>:
        static args = <optimized out>,
        static arity = <optimized out>
      }, <No data fields>},
    [xmps::BGP] = {
      <boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>> = {
        <boost::function_base> = {
          vtable = 0xb239d1 <void boost::function2<void, XmppSt...

Taking nodei38-1 as XmppClient(It is a ToR agent)
From the stats(Session stats and ifmap stats), it is clear that config is not set to XmppClient. The Subscribe message is rxed from the client.(It is visible in the trace and also in stats)

gdb) p *(IFMapClient *)     0x7f9557eacde0
$17 = (IFMapXmppChannel::IFMapSender) {
  <IFMapClient> = {
    _vptr.IFMapClient = 0xae3cd0 <vtable for IFMapXmppChannel::IFMapSender+16>,
    static kIndexInvalid = -1,
    index_ = 4,
    exporter_ = 0x1449f30,
    msgs_sent_ = 0,
    msgs_blocked_ = 0,
    bytes_sent_ = 0,
    nodes_sent_ = 0,
    links_sent_ = 0,
    send_is_blocked_ = false,
    vm_map_ = std::map with 0 elements,
    name_ = "nodei36:192.168.1.6"
  },
  members of IFMapXmppChannel::IFMapSender:
  parent_ = 0x7f94416ac9f0,
  hostname_ = "nodei38-1",
  identifier_ = "default-global-system-config:nodei38-1"
}
(gdb) p *(IFMapXmppChannel*) 0x7f94416ac9f0
$18 = (IFMapXmppChannel) {
  _vptr.IFMapXmppChannel = 0xae3d30 <vtable for IFMapXmppChannel+16>,
  peer_id_ = xmps::CONFIG,
  channel_ = 0x1f2df10,
  ifmap_server_ = 0x7fff29159400,
  ifmap_channel_manager_ = 0x7fff291591a0,
  ifmap_client_ = 0x7f9557eacde0,
  client_added_ = true,
  channel_name_ = "nodei36:192.168.1.6"
}

(gdb) p *(XmppChannel *) 0x1f2df10
$19 = (XmppChannelMux) {
  <XmppChannel> = {
    _vptr.XmppChannel = 0xb6d9f0 <vtable for XmppChannelMux+16>
  },
  members of XmppChannelMux:
  map_ = std::map with 0 elements,
  rxmap_ = std::map with 2 elements = {
    [xmps::CONFIG] = {
      <boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>> = {
        <boost::function_base> = {
          vtable = 0xae3331 <void boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>::assign_to<boost::_bi::bind_t<void, boost::_mfi::mf1<void, IFMapXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<IFMapXmppChannel*>, boost::arg<1> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, IFMapXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<IFMapXmppChannel*>, boost::arg<1> > >)::stored_vtable+1>,
          functor = {
            obj_ptr = 0x11,
            type = {
              type = 0x11,
              const_qualified = false,
              volatile_qualified = false
            },
            func_ptr = 0x11,
            bound_memfunc_ptr = {
              memfunc_ptr = &virtual table offset 16,
              obj_ptr = 0x7f94416ac9f0
            },
            obj_ref = {
              obj_ptr = 0x11,
              is_const_qualified = false,
              is_volatile_qualified = false
            },
            data = 17 '\021'
          }
        },
        <std::binary_function<XmppStanza::XmppMessage const*, xmps::PeerState, void>> = {<No data fields>},
        members of boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>:
        static args = <optimized out>,
        static arity = <optimized out>
      }, <No data fields>},
    [xmps::BGP] = {
      <boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>> = {
        <boost::function_base> = {
          vtable = 0xb239d1 <void boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>::assign_to<boost::_bi::bind_t<void, boost::_mfi::mf1<void, BgpXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<BgpXmppChannel*>, boost::arg<1> > > >(boost::_bi::bind_t<void, boost::_mfi::mf1<void, BgpXmppChannel, XmppStanza::XmppMessage const*>, boost::_bi::list2<boost::_bi::value<BgpXmppChannel*>, boost::arg<1> > >)::stored_vtable+1>,
          functor = {
            obj_ptr = 0x19,
            type = {
              type = 0x19,
              const_qualified = false,
              volatile_qualified = false
            },
            func_ptr = 0x19,
            bound_memfunc_ptr = {
              memfunc_ptr = &virtual table offset 24,
              obj_ptr = 0x7f9441272530
            },
            obj_ref = {
              obj_ptr = 0x19,
              is_const_qualified = false,
              is_volatile_qualified = false
            },
            data = 25 '\031'
          }
        },
        <std::binary_function<XmppStanza::XmppMessage const*, xmps::PeerState, void>> = {<No data fields>},
        members of boost::function2<void, XmppStanza::XmppMessage const*, xmps::PeerState>:
        static args = <optimized out>,
        static arity = <optimized out>
      }, <No data fields>}
  },
  cb_ = {
    <boost::function1<void, boost::system::error_code const&>> = {
      <boost::function_base> = {
        vtable = 0x0,
        functor = {
          obj_ptr = 0x7f95ffffffff,
          type = {
            type = 0x7f95ffffffff,
            const_qualified = 49,
            volatile_qualified = 48
          },
          func_ptr = 0x7f95ffffffff,
          bound_memfunc_ptr = {
            memfunc_ptr = &virtual table offset 140282221821950, this adjustment 3615884932430049329,
            obj_ptr = 0x313a3834312e3731
          },
          obj_ref = {
            obj_ptr = 0x7f95ffffffff,
            is_const_qualified = 49,
            is_volatile_qualified = 48
          },
          data = -1 '\377'
        }
      },
      <std::unary_function<boost::system::error_code const&, void>> = {<No data fields>},
      members of boost::function1<void, boost::system::error_code const&>:
      static args = <optimized out>,
      static arity = <optimized out>
    }, <No data fields>},
  connection_ = 0x5286f90,
  mutex_ = {
    static is_rw_mutex = false,
    static is_recursive_mutex = false,
    static is_fair_mutex = false,
    impl = {
      __data = {
        __lock = 0,
        __count = 0,
        __owner = 0,
        __nusers = 0,
        __kind = 0,
        __spins = 0,
        __elision = 0,
        __list = {
          __prev = 0x0,
          __next = 0x0
        }
      },
      __size = '\000' <repeats 39 times>,
      __align = 0
    }
  }
}
(gdb)   p *(XmppConnection *) 0x5286f90
$20 = (XmppServerConnection) {
  <XmppConnection> = {
    _vptr.XmppConnection = 0xb59650 <vtable for XmppServerConnection+16>,
    static kAuthTypeNil = 0xb1deec "NIL",
    static kAuthTypeTls = 0xb589a4 "TLS",
    server_ = 0x14752a0,
    session_ = 0x1623060,
    endpoint_ = {
      impl_ = {
        data_ = {
          base = {
            sa_family = 2,
            sa_data = "\233P\300\250\001\006\000\000\000\000\000\000\000"
          },
          v4 = {
            sin_family = 2,
            sin_port = 20635,
            sin_addr = {
              s_addr = 100772032
            },
            sin_zero = "\000\000\000\000\000\000\000"
          },
          v6 = {
            sin6_family = 2,
            sin6_port = 20635,
            sin6_flowinfo = 100772032,
            sin6_addr = {
              __in6_u = {
                __u6_addr8 = '\000' <repeats 15 times>,
                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                __u6_addr32 = {0, 0, 0, 0}
              }
            },
            sin6_scope_id = 0
          }
        }
      }
    },
    local_endpoint_ = {
      impl_ = {
        data_ = {
          base = {
            sa_family = 2,
            sa_data = "\024\225\300\250\001\004\000\000\000\000\000\000\000"
          },
          v4 = {
            sin_family = 2,
            sin_port = 38164,
            sin_addr = {
              s_addr = 67217600
            },
            sin_zero = "\000\000\000\000\000\000\000"
          },
          v6 = {
            sin6_family = 2,
            sin6_port = 38164,
            sin6_flowinfo = 67217600,
            sin6_addr = {
              __in6_u = {
                __u6_addr8 = '\000' <repeats 15 times>,
                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                __u6_addr32 = {0, 0, 0, 0}
              }
            },
            sin6_scope_id = 0
          }
        }
      }
    },
    config_ = 0x0,
    spin_mutex_ = {
      flag = 0 '\000',
      static is_rw_mutex = false,
      static is_recursive_mutex = false,
      static is_fair_mutex = false
    },
    keepalive_timer_ = 0x52870b0,
    log_uve_ = true,
    admin_down_ = false,
    disable_read_ = false,
    from_ = "nodei36",
    to_ = "nodei38-1",
    auth_enabled_ = false,
    state_machine_ = (boost::scoped_ptr<XmppStateMachine>) 0x6659a00,
    mux_ = (boost::scoped_ptr<XmppChannelMux>) 0x1f2df10,
    last_msg_ = {
      _M_ptr = 0x0
    },
    stats_ = {{
        open = 1,
        close = 0,
        keepalive = 392,
        update = 2
      }, {
        open = 1,
        close = 0,
        keepalive = 389,
        update = 0
      }},
    error_stats_ = {
      connect_error = 0,
      session_close = 0,
      open_fail = 0,
      stream_feature_fail = 0,
      handshake_fail = 0
    }
  },
  members of XmppServerConnection:
  duplicate_ = false,
  on_work_queue_ = false,
  conn_endpoint_ = 0x7f95a4001c50,
  deleter_ = (boost::scoped_ptr<XmppServerConnection::DeleteActor>) 0x1622ce0,
  server_delete_ref_ = {
    <LifetimeRefBase> = {
      _vptr.LifetimeRefBase = 0xb59490 <vtable for LifetimeRef<XmppServerConnection>+16>,
      ref_ = {
        node_ = <boost::intrusive_hook> next = 0x192e770 prev = 0x665a080,
        self_ = 0x5287078,
        ptr_ = 0x14767e0
      }
    },
    members of LifetimeRef<XmppServerConnection>:
    ptr_ = 0x5286f90
  }
}

(gdb) p *(TcpSession *)0x1623060
$21 = (XmppSession) {
  stats_ = std::vector of length 5, capacity 5 = {{
      first = 0,
      second = 0
    }, {
      first = 1,
      second = 173
    }, {
      first = 0,
      second = 0
    }, {
      first = 2,
      second = 566
    }, {
      first = 392,
      second = 782
    }},
..
}

OpenContrail Admin (ci-admin-f) on 2015-06-21

Changed in juniperopenstack:
assignee:	nobody → Tapan Karwa (tkarwa)

OpenContrail Admin (ci-admin-f) on 2015-06-22

tags:

added: blocker

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-22: [Review update] R2.20

#2

Review in progress for https://review.opencontrail.org/11940
Submitter: Tapan Karwa (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-22: [Review update] R2.1

#3

Review in progress for https://review.opencontrail.org/11945
Submitter: Tapan Karwa (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-22: [Review update] R2.0

#4

Review in progress for https://review.opencontrail.org/11946
Submitter: Tapan Karwa (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-22: [Review update] master

#5

Review in progress for https://review.opencontrail.org/11947
Submitter: Tapan Karwa (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-23: A change has been merged

#6

Reviewed: https://review.opencontrail.org/11947
Committed: http://github.org/Juniper/contrail-controller/commit/8db93fec1cbb046076440c51a58cd625bef588f4
Submitter: Zuul
Branch: master

commit 8db93fec1cbb046076440c51a58cd625bef588f4
Author: Tapan Karwa <email address hidden>
Date: Mon Jun 22 12:21:02 2015 -0700

Activate the ifmap update Q for all node/link changes

Closes-Bug: #1467650
Partial-Bug: #1466825

Change-Id: I88e975e685ebc43ad20b473bb62026bb36f6b044

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-23:

#7

Reviewed: https://review.opencontrail.org/11945
Committed: http://github.org/Juniper/contrail-controller/commit/2f48d8789a11955bf895b69caacf4e039ca65de7
Submitter: Zuul
Branch: R2.1

commit 2f48d8789a11955bf895b69caacf4e039ca65de7
Author: Tapan Karwa <email address hidden>
Date: Mon Jun 22 12:21:02 2015 -0700

Activate the ifmap update Q for all node/link changes

Closes-Bug: #1467650
Partial-Bug: #1466825

Change-Id: I88e975e685ebc43ad20b473bb62026bb36f6b044

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-23:

#8

Reviewed: https://review.opencontrail.org/11946
Committed: http://github.org/Juniper/contrail-controller/commit/7046092e608bbc767df37e833f16a1e024aa0af9
Submitter: Zuul
Branch: R2.0

commit 7046092e608bbc767df37e833f16a1e024aa0af9
Author: Tapan Karwa <email address hidden>
Date: Mon Jun 22 12:21:02 2015 -0700

Activate the ifmap update Q for all node/link changes

Closes-Bug: #1467650
Partial-Bug: #1466825

Change-Id: I88e975e685ebc43ad20b473bb62026bb36f6b044

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-23:

#9

Reviewed: https://review.opencontrail.org/11940
Committed: http://github.org/Juniper/contrail-controller/commit/659fe8cec5ca4e11c8d99c3b8c64d8109782e670
Submitter: Zuul
Branch: R2.20

commit 659fe8cec5ca4e11c8d99c3b8c64d8109782e670
Author: Tapan Karwa <email address hidden>
Date: Mon Jun 22 12:21:02 2015 -0700

Activate the ifmap update Q for all node/link changes

Closes-Bug: #1467650
Partial-Bug: #1466825

Change-Id: I88e975e685ebc43ad20b473bb62026bb36f6b044

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-23:

#10

Commit on bug 1467650 will help this case. We need to repro this after this fix as there could be more reasons than queue stuck for this. We want to tackle couple of things. Moving the work item to 2.21.

1. Don't reuse xmpp client ID for config.
2. Throttle incoming clients.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-06-24:

#11

Not a blocker anymore (see comment #10) for detail

tags:

removed: blocker

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-19: [Bug update]

#12

bug update...

Revision history for this message

Nischal Sheth (nsheth) wrote on 2016-04-22:

#13

Am going to mark this as Fix Committed.
Please open a new bug if this issue is seen again.

Revision history for this message

Nischal Sheth (nsheth) wrote on 2016-04-22:

#14

Note that the series for 3.0 was created way after the fix
already went into master. Set the milestone for master
to r3.0-fcs and marked the R3.0 series as Invalid.

Juniper Openstack

agent/tor-agents did not get any ifmap from control node on tor-scale setup

Bug Description

Other bug subscribers

Remote bug watches

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.0	Fix Committed	High	Tapan Karwa
R2.1	Fix Committed	High	Tapan Karwa
R2.20	Fix Committed	High	Tapan Karwa	Juniper Openstack r2.21
R3.0	Invalid	High	Tapan Karwa
Trunk	Fix Committed	High	Tapan Karwa	Juniper Openstack r3.0-fcs