Bug #1578504 “contrail-control crash in RoutePathReplicator::Rou...” : Bugs : Juniper Openstack

Prakash Bailkeri (prakashmb) on 2016-05-05

Changed in juniperopenstack:
assignee:	Hari Prasad Killi (haripk) → Prakash Bailkeri (prakashmb)

Revision history for this message

vageesan (vageesant) wrote on 2016-05-05:

#1

core is in 10.84.5.112:/auto/cores/1578504

Revision history for this message

Prakash Bailkeri (prakashmb) wrote on 2016-05-06:

#2

Download full text (24.1 KiB)

core file analysis:

# Path resolver sets the peer to NULL for Resolved path.
# Currently there multiple resolved path with NULL peer and same path id. This causes the replicator to assert.
# Solution is to set the peer for resolved path. The logic already present in master. Need to back port.

(gdb) p *(InetRoute *) entry
$123 = (InetRoute) {
  <BgpRoute> = {
    <Route> = {
      <DBEntry> = {
        <DBEntryBase> = {
          _vptr.DBEntryBase = 0xc83f50 <vtable for InetRoute+16>,
          chg_list_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::list_node<void*>> = {
                next_ = 0x0,
                prev_ = 0x0
              }, <No data fields>}, <No data fields>},
          tpart_ = 0x7fc1d007c780,
          state_ = std::map with 3 elements = {
            [0] = 0x7fc1c81cdaa0,
            [2] = 0x7fc254500930,
            [3] = 0x7fc224241690
          },
          flags = 1 '\001',
          onremoveq_ = (tbb::atomic) false,
          last_change_at_ = 1462427156821374
        },
        members of DBEntry:
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::rbtree_node<void*>> = {
              parent_ = 0x7fc2602de5a0,
              left_ = 0x0,
              right_ = 0x0,
              color_ = boost::intrusive::rbtree_node<void*>::red_t
            }, <No data fields>}, <No data fields>}
      },
      members of Route:
      path_ = {
        <boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >> = {
          <boost::intrusive::detail::clear_on_destructor_base<boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> > >> = {<No data fields>},
          members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >:
          static external_value_traits = false,
          static constant_time_size = true,
          static stateful_value_traits = <optimized out>,
          data_ = {
            <boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>> = {
              static link_mode = <optimized out>
            },
            me...

core file analysis:

# Path resolver sets the peer to NULL for Resolved path. 
# Currently there multiple resolved path with NULL peer and same path id. This causes the replicator to assert.
# Solution is to set the peer for resolved path. The logic already present in master. Need to back port.

(gdb) p *(InetRoute *) entry
$123 = (InetRoute) {
  <BgpRoute> = {
    <Route> = {
      <DBEntry> = {
        <DBEntryBase> = {
          _vptr.DBEntryBase = 0xc83f50 <vtable for InetRoute+16>,
          chg_list_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::list_node<void*>> = {
                next_ = 0x0,
                prev_ = 0x0
              }, <No data fields>}, <No data fields>},
          tpart_ = 0x7fc1d007c780,
          state_ = std::map with 3 elements = {
            [0] = 0x7fc1c81cdaa0,
            [2] = 0x7fc254500930,
            [3] = 0x7fc224241690
          },
          flags = 1 '\001',
          onremoveq_ = (tbb::atomic) false,
          last_change_at_ = 1462427156821374
        },
        members of DBEntry:
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::rbtree_node<void*>> = {
              parent_ = 0x7fc2602de5a0,
              left_ = 0x0,
              right_ = 0x0,
              color_ = boost::intrusive::rbtree_node<void*>::red_t
            }, <No data fields>}, <No data fields>}
      },
      members of Route:
      path_ = {
        <boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >> = {
          <boost::intrusive::detail::clear_on_destructor_base<boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> > >> = {<No data fields>},
          members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >:
          static external_value_traits = false,
          static constant_time_size = true,
          static stateful_value_traits = <optimized out>,
          data_ = {
            <boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>> = {
              static link_mode = <optimized out>
            },
            members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >::data_t:
            root_plus_size_ = {
              <boost::intrusive::detail::size_holder<true, unsigned long>> = {
                static constant_time_size = <optimized out>,
                size_ = 4
              },
              members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >::root_plus_size:
              root_ = {
                next_ = 0x7fc1f8231f18,
                prev_ = 0x7fc1c83bb038
              }
            }
          }
        }, <No data fields>}
    }, <No data fields>},
  members of InetRoute:
  prefix_ = {
    ip4_addr_ = <<9.0.10.0>     >,
    prefixlen_ = 24
  }
}
(gdb)  p *(BgpPath *) 0x7fc1f8231f10
$124 = (BgpPath) {
  <Path> = {
    _vptr.Path = 0xcdd810 <vtable for BgpPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x7fc1b4187978,
          prev_ = 0x7fc2602efb88
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1462427156821372
  },
  members of BgpPath:
  static INFEASIBLE_MASK = 187,
  peer_ = 0x0,
  path_id_ = 2886775824,
  source_ = BgpPath::BGP_XMPP,
  attr_ = {
    px = 0x7fc1ec2c8ea0
  },
  original_attr_ = {
    px = 0x7fc1ec2c8ea0
  },
  flags_ = 64,
  label_ = 285
}
(gdb)  p *(BgpPath *) 0x7fc1b4187970
$125 = (BgpPath) {
  <Path> = {
    _vptr.Path = 0xcdd810 <vtable for BgpPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x7fc24c38bb88,
          prev_ = 0x7fc1f8231f18
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1462427157286132
  },
  members of BgpPath:
  static INFEASIBLE_MASK = 187,
  peer_ = 0x0,
  path_id_ = 2886775824,
  source_ = BgpPath::BGP_XMPP,
  attr_ = {
    px = 0x7fc1b4105b60
  },
  original_attr_ = {
    px = 0x7fc1b4105b60
  },
  flags_ = 64,
  label_ = 410
}
(gdb)  p *(BgpPath *) 0x7fc24c38bb80
$126 = (BgpSecondaryPath) {
  <BgpPath> = {
    <Path> = {
      _vptr.Path = 0xcdd7d0 <vtable for BgpSecondaryPath+16>,
      node_ = {
        <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
          <boost::intrusive::detail::no_default_definer> = {<No data fields>},
          <boost::intrusive::list_node<void*>> = {
            next_ = 0x7fc1c83bb038,
            prev_ = 0x7fc1b4187978
          }, <No data fields>}, <No data fields>},
      time_stamp_usecs_ = 1462427156837864
    },
    members of BgpPath:
    static INFEASIBLE_MASK = 187,
    peer_ = 0x7fc2680f4be0,
    path_id_ = 2886775824,
    source_ = BgpPath::BGP_XMPP,
    attr_ = {
      px = 0x7fc1e4255210
    },
    original_attr_ = {
      px = 0x7fc1e4255210
    },
    flags_ = 0,
    label_ = 285
  },
  members of BgpSecondaryPath:
  src_table_ = 0x2cffd80,
  src_entry_ = 0x7fc254228c10
}
(gdb)  p *(BgpPath *) 0x7fc1c83bb030
$127 = (BgpPath) {
  <Path> = {
    _vptr.Path = 0xcdd810 <vtable for BgpPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x7fc2602efb88,
          prev_ = 0x7fc24c38bb88
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1462426886972958
  },
  members of BgpPath:
  static INFEASIBLE_MASK = 187,
  peer_ = 0x7fc22c112730,
  path_id_ = 16973827,
  source_ = BgpPath::BGP_XMPP,
  attr_ = {
    px = 0x7fc1f43f35b0
  },
  original_attr_ = {
    px = 0x7fc1f43f35b0
  },
  flags_ = 32,
  label_ = 0
}

(gdb) p *(DBTable *) 0x7fc1d0059dd0
$130 = (InetTable) {
  <BgpTable> = {
    <RouteTable> = {
      <DBTable> = {
        <DBTableBase> = {
          _vptr.DBTableBase = 0xc843b0 <vtable for InetTable+16>,
          static kInvalidId = -1,
          db_ = 0x2cd0a98,
          name_ = "default-domain:symantec.Tenant.177:tenant177.test_id1.Private_VSRX_BGP_VN0:tenant177.test_id1.Private_VSRX_BGP_VN0.inet.0",
          info_ = {
            _M_ptr = 0x7fc1d0002280
          },
          enqueue_count_ = 148,
          input_count_ = 148,
          notify_count_ = 1294,
          walker_count_ = (tbb::atomic) 0,
          walk_request_count_ = (tbb::atomic) 88,
          walk_complete_count_ = (tbb::atomic) 88,
          walk_cancel_count_ = (tbb::atomic) 0
        },
        members of DBTable:
        partitions_ = std::vector of length 40, capacity 64 = {0x7fc1d007a610, 0x7fc1d007a9e0, 0x7fc1d007adf0, 0x7fc1d007b130, 0x7fc1d007b500, 0x7fc1d007b910, 0x7fc1d007bc50, 0x7fc1d007bfc0, 0x7fc1d007c440, 0x7fc1d007cbf0, 0x7fc1d007d410, 0x7fc1d007dc30, 0x7fc1d007e4b0, 0x7fc1d0016320, 0x7fc1d0011a40, 0x7fc1d00166a0, 0x7fc1d004e300, 0x7fc1d0024130, 0x7fc1d004e0e0, 0x7fc1d00466f0, 0x7fc1d0077780, 0x7fc1d0074560, 0x7fc1d0077d20, 0x7fc1d007a2d0, 0x7fc1d007c780, 0x7fc1d0059ee0, 0x7fc1d0011940, 0x7fc1d0011590, 0x7fc1d004e7b0, 0x7fc1d005d1e0, 0x7fc1d0041310, 0x7fc1d0057a10, 0x7fc1d005a1a0, 0x7fc1d005a090, 0x7fc1d005a230, 0x7fc1d005a2c0, 0x7fc1d0083690, 0x7fc1d0083720, 0x7fc1d00837b0, 0x7fc1d0083840},
        walk_id_ = -1
      }, <No data fields>},
    members of BgpTable:
    rtinstance_ = 0x7fc1d0059320,
    path_resolver_ = 0x7fc1d00838d0,
    ribout_map_ = std::map with 2 elements = {
      [{
        type = BgpProto::EBGP,
        encoding = RibExportPolicy::BGP,
        as_number = 652,
        affinity = -1,
        cluster_id = 0
      }] = 0x7fc1c81f0cb0,
      [{
        type = BgpProto::XMPP,
        encoding = RibExportPolicy::XMPP,
        as_number = 0,
        affinity = -1,
        cluster_id = 0
      }] = 0x7fc1fc17ab10
    },
    deleter_ = {
      px = 0x7fc1d005e2e0
    },
    instance_delete_ref_ = {
      <LifetimeRefBase> = {
        _vptr.LifetimeRefBase = 0xcf42d0 <vtable for LifetimeRef<BgpTable>+16>,
        ref_ = {
          node_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::list_node<void*>> = {
                next_ = 0x7fc1d0085ea8,
                prev_ = 0x7fc1d0059c98
              }, <No data fields>}, <No data fields>},
          self_ = 0x7fc1d0059e90,
          ptr_ = 0x7fc1d00043e0
        }
      },
      members of LifetimeRef<BgpTable>:
      ptr_ = 0x7fc1d0059dd0
    },
    primary_path_count_ = (tbb::atomic) 41,
    secondary_path_count_ = (tbb::atomic) 17,
    infeasible_path_count_ = (tbb::atomic) 13
  }, <No data fields>}

(gdb) p *(PathResolver *) 0x7fc1d00838d0
$131 = {
  table_ = 0x7fc1d0059dd0,
  listener_id_ = 0,
  mutex_ = {
    static is_rw_mutex = false,
    static is_recursive_mutex = false,
    static is_fair_mutex = false,
    impl = {
      __data = {
        __lock = 0,
        __count = 0,
        __owner = 0,
        __nusers = 0,
        __kind = 0,
        __spins = 0,
        __elision = 0,
        __list = {
          __prev = 0x0,
          __next = 0x0
        }
      },
      __size = '\000' <repeats 39 times>,
      __align = 0
    }
  },
  nexthop_map_ = std::map with 1 elements = {
    [{
      first = <IPv4 1.3.0.3        >,
      second = 0x7fc1d0059dd0
    }] = 0x7fc22c41da20
  },
  nexthop_reg_unreg_list_ = std::set with 0 elements,
  nexthop_reg_unreg_trigger_ = {
    px = 0x7fc1d004e840
  },
  nexthop_update_list_ = std::set with 0 elements,
  nexthop_update_trigger_ = {
    px = 0x7fc1d00413a0
  },
  nexthop_delete_list_ = std::set with 0 elements,
  partitions_ = std::vector of length 40, capacity 64 = {0x7fc1d0074ff0, 0x7fc1d005b3f0, 0x7fc1d005a120, 0x7fc1d0083ab0, 0x7fc1d0083b70, 0x7fc1d0083c30, 0x7fc1d0083cf0, 0x7fc1d0083db0, 0x7fc1d0083e70, 0x7fc1d0083fc0, 0x7fc1d0084080, 0x7fc1d0084140, 0x7fc1d0084200, 0x7fc1d00842c0, 0x7fc1d0084380, 0x7fc1d0084440, 0x7fc1d0084500, 0x7fc1d0083f30, 0x7fc1d00846d0, 0x7fc1d0084790, 0x7fc1d0084850, 0x7fc1d0084910, 0x7fc1d00849d0, 0x7fc1d0084a90, 0x7fc1d0084b50, 0x7fc1d0084c10, 0x7fc1d0084cd0, 0x7fc1d0084d90, 0x7fc1d0084e50, 0x7fc1d0084f10, 0x7fc1d0084fd0, 0x7fc1d0085090, 0x7fc1d0085150, 0x7fc1d00845c0, 0x7fc1d0085420, 0x7fc1d00854e0, 0x7fc1d00855a0, 0x7fc1d0085660, 0x7fc1d0085720, 0x7fc1d00857e0},
  deleter_ = {
    px = 0x7fc1d007e9a0
  },
  table_delete_ref_ = {
    <LifetimeRefBase> = {
      _vptr.LifetimeRefBase = 0xca9e10 <vtable for LifetimeRef<PathResolver>+16>,
      ref_ = {
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::list_node<void*>> = {
              next_ = 0x7fc250095e28,
              prev_ = 0x7fc1d005e340
            }, <No data fields>}, <No data fields>},
        self_ = 0x7fc1d00839f8,
        ptr_ = 0x7fc1d005e2e0
      }
    },
    members of LifetimeRef<PathResolver>:
    ptr_ = 0x7fc1d00838d0
  }
}
(gdb) p *(ResolverNexthop *) 0x7fc22c41da20
$132 = (ResolverNexthop) {
  <ConditionMatch> = {
    _vptr.ConditionMatch = 0xca9f50 <vtable for ResolverNexthop+16>,
    deleted_ = false,
    mutex_ = {
      static is_rw_mutex = false,
      static is_recursive_mutex = false,
      static is_fair_mutex = false,
      impl = {
        __data = {
          __lock = 0,
          __count = 0,
          __owner = 0,
          __nusers = 0,
          __kind = 0,
          __spins = 0,
          __elision = 0,
          __list = {
            __prev = 0x0,
            __next = 0x0
          }
        },
        __size = '\000' <repeats 39 times>,
        __align = 0
      }
    },
    num_matchstate_ = 1,
    refcount_ = (tbb::atomic) 2
  },
  members of ResolverNexthop:
  resolver_ = 0x7fc1d00838d0,
  address_ = <IPv4 1.3.0.3        >,
  table_ = 0x7fc1d0059dd0,
  registered_ = true,
  route_ = 0x7fc250095f90,
  rpath_lists_ = std::vector of length 40, capacity 40 = {std::set with 4 elements = {
      [0] = 0x7fc23c162b30,
      [1] = 0x7fc23c1bd550,
      [2] = 0x7fc23c1d4010,
      [3] = 0x7fc23c39c540
    }, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 2 elements = {
      [0] = 0x7fc22c1f29c0,
      [1] = 0x7fc22c41db70
    }, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 2 elements = {
      [0] = 0x7fc25c139430,
      [1] = 0x7fc25c1b28c0
    }, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 3 elements = {
      [0] = 0x7fc1c82d4020,
      [1] = 0x7fc1c83d40b0,
      [2] = 0x7fc1c83f2050
    }, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 2 elements = {
      [0] = 0x7fc2642b14a0,
      [1] = 0x7fc2642cdbc0
    }, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements, std::set with 0 elements},
  table_delete_ref_ = {
    <LifetimeRefBase> = {
      _vptr.LifetimeRefBase = 0xca9e50 <vtable for LifetimeRef<ResolverNexthop>+16>,
      ref_ = {
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::list_node<void*>> = {
              next_ = 0x7fc23c2c6ed8,
              prev_ = 0x7fc1c845eaa8
            }, <No data fields>}, <No data fields>},
        self_ = 0x7fc22c41dab8,
        ptr_ = 0x7fc1d005e2e0
      }
    },
    members of LifetimeRef<ResolverNexthop>:
    ptr_ = 0x7fc22c41da20
  }
}
(gdb) p *(PathResolverPartition *) 0x7fc1d0084b50
$136 = {
  part_id_ = 24,
  resolver_ = 0x7fc1d00838d0,
  rpath_map_ = std::map with 3 elements = {
    [0x7fc1c81e4930] = 0x7fc1c83d40b0,
    [0x7fc1c833f730] = 0x7fc1c83f2050,
    [0x7fc1c83bb030] = 0x7fc1c82d4020
  },
  rpath_update_list_ = std::set with 0 elements,
  rpath_update_trigger_ = {
    px = 0x7fc1d0084bd0
  }
}
(gdb) p *(ResolverPath *) 0x7fc1c82d4020
$140 = {
  partition_ = 0x7fc1d0084b50,
  path_ = 0x7fc1c83bb030,
  route_ = 0x7fc2602efb00,
  rnexthop_ = 0x7fc22c41da20,
  state_ = {
    px = 0x7fc1c81cdaa0
  },
  resolved_path_list_ = std::set with 2 elements = {
    [0] = 0x7fc1b4187970,
    [1] = 0x7fc1f8231f10
  }
}

# Path resolves to the following route. The path list looks empty. This seems to because 
# the route is currently under modification and path is being sorted. The path list
# empty seems to be intermediate state of path sorting 
(gdb) p *(InetRoute *) 0x7fc250095f90
$141 = (InetRoute) {
  <BgpRoute> = {
    <Route> = {
      <DBEntry> = {
        <DBEntryBase> = {
          _vptr.DBEntryBase = 0xc83f50 <vtable for InetRoute+16>,
          chg_list_ = {
            <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
              <boost::intrusive::detail::no_default_definer> = {<No data fields>},
              <boost::intrusive::list_node<void*>> = {
                next_ = 0x0,
                prev_ = 0x0
              }, <No data fields>}, <No data fields>},
          tpart_ = 0x7fc1d0011590,
          state_ = std::map with 3 elements = {
            [2] = 0x7fc2501c2b80,
            [3] = 0x7fc1d0513be0,
            [5] = 0x7fc250190d40
          },
          flags = 0 '\000',
          onremoveq_ = (tbb::atomic) false,
          last_change_at_ = 1462427157285694
        },
        members of DBEntry:
        node_ = {
          <boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
            <boost::intrusive::detail::no_default_definer> = {<No data fields>},
            <boost::intrusive::rbtree_node<void*>> = {
              parent_ = 0x7fc1d00115f8,
              left_ = 0x0,
              right_ = 0x0,
              color_ = boost::intrusive::rbtree_node<void*>::black_t
            }, <No data fields>}, <No data fields>}
      },
      members of Route:
      path_ = {
        <boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >> = {
          <boost::intrusive::detail::clear_on_destructor_base<boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> > >> = {<No data fields>},
          members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >:
          static external_value_traits = false,
          static constant_time_size = true,
          static stateful_value_traits = <optimized out>,
          data_ = {
            <boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>> = {
              static link_mode = <optimized out>
            },
            members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >::data_t:
            root_plus_size_ = {
              <boost::intrusive::detail::size_holder<true, unsigned long>> = {
                static constant_time_size = <optimized out>,
                size_ = 0
              },
              members of boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >::root_plus_size:
              root_ = {
                next_ = 0x7fc250096018,
                prev_ = 0x7fc250096018
              }
            }
          }
        }, <No data fields>}
    }, <No data fields>},
  members of InetRoute:
  prefix_ = {
    ip4_addr_ = <<1.3.0.3>      >,
    prefixlen_ = 32
  }
}

(gdb) thread 16
[Switching to thread 16 (Thread 0x7fc241be6700 (LWP 19761))]
#0  0x00007fc276694140 in __dynamic_cast () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007fc276694140 in __dynamic_cast () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00000000008cdfd6 in BgpTable::PathSelection (path1=..., path2=...) at controller/src/bgp/bgp_table.cc:264
#2  0x0000000000a3a8e4 in merge<bool (*)(Path const&, Path const&)> (p=0x8cdfa0 <BgpTable::PathSelection(Path const&, Path const&)>, x=..., this=<optimized out>) at /usr/include/boost/intrusive/list.hpp:1058
#3  boost::intrusive::list_impl<boost::intrusive::listopt<boost::intrusive::detail::member_hook_traits<Path, boost::intrusive::list_member_hook<boost::intrusive::none, boost::intrusive::none, boost::intrusive::none>, &Path::node_>, unsigned long, true> >::sort<bool (*)(Path const&, Path const&)> (this=this@entry=0x7fc250096010, p=0x8cdfa0 <BgpTable::PathSelection(Path const&, Path const&)>) at /usr/include/boost/intrusive/list.hpp:1010
#4  0x0000000000a3a748 in Route::Sort (this=this@entry=0x7fc250095f90, compare=<optimized out>, prev_front=prev_front@entry=0x7fc1c0158020) at controller/src/route/route.cc:41
#5  0x000000000088ee2d in BgpRoute::InsertPath (this=0x7fc250095f90, path=0x7fc1c0296000) at controller/src/bgp/bgp_route.cc:54
#6  0x0000000000457c27 in InetTable::RouteReplicate (this=<optimized out>, server=<optimized out>, src_table=0x2cffd80, src_rt=0x7fc25064f020, path=<optimized out>, community=...) at controller/src/bgp/inet/inet_table.cc:110
#7  0x00000000005ef706 in RoutePathReplicator::RouteListener (this=0x2cd7260, ts=0x2d01a80, root=<optimized out>, entry=0x7fc25064f020) at controller/src/bgp/routing-instance/routepath_replicator.cc:627
#8  0x0000000000c19a9a in operator() (a1=0x7fc25064f020, a0=0x2d00fe0, this=0x7fc241be5aa0) at /usr/include/boost/function/function_template.hpp:767
#9  RunNotify (entry=0x7fc25064f020, tpart=0x2d00fe0, this=0x2cffe90) at controller/src/db/db_table.cc:114
#10 DBTableBase::RunNotify (this=<optimized out>, tpart=tpart@entry=0x2d00fe0, entry=entry@entry=0x7fc25064f020) at controller/src/db/db_table.cc:204
#11 0x0000000000c1c0c8 in DBTablePartBase::RunNotify (this=this@entry=0x2d00fe0) at controller/src/db/db_table_partition.cc:47
#12 0x0000000000c18290 in DBPartition::QueueRunner::Run (this=0x7fc268299080) at controller/src/db/db_partition.cc:209
#13 0x000000000068901f in TaskImpl::execute (this=0x7fc26f5b9b40) at controller/src/base/task.cc:261
#14 0x00007fc276b63b3a in ?? () from /usr/lib/libtbb.so.2
#15 0x00007fc276b5f816 in ?? () from /usr/lib/libtbb.so.2
#16 0x00007fc276b5ef4b in ?? () from /usr/lib/libtbb.so.2
#17 0x00007fc276b5b0ff in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fc276b5b2f9 in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fc276d7f182 in start_thread (arg=0x7fc241be6700) at pthread_create.c:312
#20 0x00007fc275e5047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) fr 4
#4  0x0000000000a3a748 in Route::Sort (this=this@entry=0x7fc250095f90, compare=<optimized out>, prev_front=prev_front@entry=0x7fc1c0158020) at controller/src/route/route.cc:41
41      controller/src/route/route.cc: No such file or directory.
(gdb) p this
$142 = (InetRoute * const) 0x7fc250095f90
(gdb)

Nischal Sheth (nsheth) on 2016-05-09

information type:

Proprietary → Public

Revision history for this message

Prakash Bailkeri (prakashmb) wrote on 2016-05-09:

#3

Need to merge parts of https://github.com/Juniper/contrail-controller/commit/abfda24015ecb8e5148913c3d5516612885d990a to R3.0.

Revision history for this message

Nischal Sheth (nsheth) wrote on 2016-05-09:

#4

Problem happens when the same path is received from multiple
bgpaas peers with the same nexthop.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-09: [Review update] R3.0

#5

Review in progress for https://review.opencontrail.org/20019
Submitter: Nischal Sheth (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-09: [Review update] master

#7

Review in progress for https://review.opencontrail.org/20022
Submitter: Nischal Sheth (<email address hidden>)

Nischal Sheth (nsheth) on 2016-05-09

tags:

added: bgpaas

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-11: A change has been merged

#9

Reviewed: https://review.opencontrail.org/20022
Committed: http://github.org/Juniper/contrail-controller/commit/53dc731798aa2c7fe6630092cd010d0bb0fc0345
Submitter: Zuul
Branch: master

commit 53dc731798aa2c7fe6630092cd010d0bb0fc0345
Author: Nischal Sheth <email address hidden>
Date: Mon May 9 10:09:57 2016 -0700

Add PathResolver tests for multipath with same nexthop

Note that the fix implemented for R3.0 i.e. to copy IPeer from the
original path into the resolved path, is already present in master.

Change-Id: I9df4a50df31aa604fc46ec03b70689f1475decb2
Closes-Bug: 1578504

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-11:

#10

Reviewed: https://review.opencontrail.org/20019
Committed: http://github.org/Juniper/contrail-controller/commit/f7c76a0d8fbaabd7de81faf6902dd4103399d57a
Submitter: Zuul
Branch: R3.0

commit f7c76a0d8fbaabd7de81faf6902dd4103399d57a
Author: Nischal Sheth <email address hidden>
Date: Mon May 9 10:41:56 2016 -0700

Handle multipath for bgpaas when paths have same nexthop

Following changes are implemented:

- Tweak resolver to retain IPeer from original BgpPath
- Add PathResolver tests for multipath with same nexthop

Change-Id: I4c4afd0393cceef089a0ede9acb8ccda1901bad4
Closes-Bug: 1578504

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0	Fix Committed	Medium	Nischal Sheth	Juniper Openstack r3.0.2.0
Trunk	Fix Committed	Medium	Nischal Sheth	Juniper Openstack r3.1.0.0-fcs

Juniper Openstack

contrail-control crash in RoutePathReplicator::RouteListener

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches