Gateway_less_Fwd: vrouter agent crash observed @boost::asio::ip::address::operator=, PktFlowInfo::OverlayForwarding, PktFlowInfo::VrfTranslate

[Chandra Sekhar Reddy Mallam]

vrouter agent crash observed @boost::asio::ip::address::operator=, PktFlowInfo::OverlayForwarding, PktFlowInfo::VrfTranslate while applying/deleting floating IPs.

Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000001b22b28 in boost::asio::ip::address::operator= (this=0x7f695ccd4230, other=...) at /usr/include/boost/asio/ip/impl/address.ipp:71
71 /usr/include/boost/asio/ip/impl/address.ipp: No such file or directory.
(gdb) bt
#0 0x0000000001b22b28 in boost::asio::ip::address::operator= (this=0x7f695ccd4230, other=...) at /usr/include/boost/asio/ip/impl/address.ipp:71
#1 0x00000000020225ef in PktFlowInfo::OverlayForwarding (this=0x7f695ccd46e0, intf=0x7f693996dab0, pkt=0x7f695815a7a0, in=0x7f695ccd4660, out=0x7f695ccd46a0, nat_flow=true) at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1080
#2 0x0000000002022a31 in PktFlowInfo::VrfTranslate (this=0x7f695ccd46e0, pkt=0x7f695815a7a0, in=0x7f695ccd4660, out=0x7f695ccd46a0, src_ip=..., nat_flow=true) at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1147
#3 0x0000000002021ac8 in PktFlowInfo::FloatingIpDNat (this=0x7f695ccd46e0, pkt=0x7f695815a7a0, in=0x7f695ccd4660, out=0x7f695ccd46a0) at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:878
#4 0x0000000002023ad6 in PktFlowInfo::EgressProcess (this=0x7f695ccd46e0, pkt=0x7f695815a7a0, in=0x7f695ccd4660, out=0x7f695ccd46a0) at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1444
#5 0x00000000020242f1 in PktFlowInfo::Process (this=0x7f695ccd46e0, pkt=0x7f695815a7a0, in=0x7f695ccd4660, out=0x7f695ccd46a0) at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1605
#6 0x000000000203fa35 in FlowHandler::Run (this=0x7f695815a190) at controller/src/vnsw/agent/pkt/flow_handler.cc:127
#7 0x0000000002035079 in Proto::RunProtoHandler (this=0x7f6944c90130, handler=0x7f695815a190) at controller/src/vnsw/agent/pkt/proto.cc:51
#8 0x00000000020064ac in FlowProto::FlowEventHandler (this=0x7f6944c90130, req=0x7f694c038b40, table=0x7f69472b4680) at controller/src/vnsw/agent/pkt/flow_proto.cc:417
#9 0x000000000203b1ca in FlowEventQueue::HandleEvent (this=0x7f69398db720, event=0x7f694c038b40) at controller/src/vnsw/agent/pkt/flow_event.cc:250
#10 0x000000000203a56c in FlowEventQueueBase::Handler (this=0x7f69398db720, event=0x7f694c038b40) at controller/src/vnsw/agent/pkt/flow_event.cc:105
#11 0x000000000203df04 in boost::_mfi::mf1<bool, FlowEventQueueBase, FlowEvent*>::operator() (this=0x7f695ccd4b18, p=0x7f69398db720, a1=0x7f694c038b40) at /usr/include/boost/bind/mem_fn_template.hpp:165
#12 0x000000000203d8fa in boost::_bi::list2<boost::_bi::value<FlowEventQueueBase*>, boost::arg<1> >::operator()<bool, boost::_mfi::mf1<bool, FlowEventQueueBase, FlowEvent*>, boost::_bi::list1<FlowEvent*&> > (this=0x7f695ccd4b28,
    f=..., a=...) at /usr/include/boost/bind/bind.hpp:303
#13 0x000000000203d336 in boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowEventQueueBase, FlowEvent*>, boost::_bi::list2<boost::_bi::value<FlowEventQueueBase*>, boost::arg<1> > >::operator()<FlowEvent*> (this=0x7f695ccd4b18,
    a1=@0x7f695ccd4a70: 0x7f694c038b40) at /usr/include/boost/bind/bind_template.hpp:32
#14 0x000000000203cfeb in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowEventQueueBase, FlowEvent*>, boost::_bi::list2<boost::_bi::value<FlowEventQueueBase*>, boost::arg<1> > >, bool, FlowEvent*>::invoke (function_obj_ptr=..., a0=0x7f694c038b40) at /usr/include/boost/function/function_template.hpp:132
#15 0x000000000200f4e5 in boost::function1<bool, FlowEvent*>::operator() (this=0x7f695ccd4b10, a0=0x7f694c038b40) at /usr/include/boost/function/function_template.hpp:767
#16 0x000000000200f1a9 in QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::RunQueue (this=0x7f694c0af910) at controller/src/base/queue_task.h:67
#17 0x000000000200f058 in QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::Run (this=0x7f694c0af910) at controller/src/base/queue_task.h:42
#18 0x00000000029a5a18 in TaskImpl::execute (this=0x7f695ce67e40) at controller/src/base/task.cc:277
#19 0x00007f6964441b3a in ?? () from /usr/lib/libtbb.so.2
#20 0x00007f696443d816 in ?? () from /usr/lib/libtbb.so.2
#21 0x00007f696443cf4b in ?? () from /usr/lib/libtbb.so.2
#22 0x00007f69644390ff in ?? () from /usr/lib/libtbb.so.2
#23 0x00007f69644392f9 in ?? () from /usr/lib/libtbb.so.2
#24 0x00007f696465d184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#25 0x00007f696393637d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) q
root@nodeb9:~# ls -ltr /var/crashes;ls -ltr /var/crash
total 144248
-rw------- 1 root root 181731328 Oct 10 19:04 core.contrail-vroute.15566.nodeb9.1507642498
total 4
-rw-r--r-- 1 root root 239 Oct 10 15:11 kexec_cmd
root@nodeb9:~# date
Wed Oct 11 09:36:27 IST 2017
root@nodeb9:~#

root@nodeb9:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-lib 5.0.0-72 72
contrail-nodemgr 5.0.0-72 72
contrail-setup 5.0.0-72 72
contrail-utils 5.0.0-72 72
contrail-vrouter-agent 5.0.0-72 72
contrail-vrouter-common 5.0.0-72 72
contrail-vrouter-dkms 5.0.0-72 72
contrail-vrouter-init 5.0.0-72 72
contrail-vrouter-utils 5.0.0-72 72
nova-common 2:13.0.0-0ubuntu2~cloud0.1contrail1 72
nova-compute 2:13.0.0-0ubuntu2~cloud0.1contrail1 72
nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0.1contrail1 72
nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0.1contrail1 72
python-contrail 5.0.0-72 72
python-contrail-vrouter-api 5.0.0-72 72
python-neutronclient 1:4.1.1-2~cloud0.2contrail 72
python-nova 2:13.0.0-0ubuntu2~cloud0.1contrail1 72
python-opencontrail-vrouter-netns 5.0.0-72 72
root@nodeb9:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
root@nodeb9:~#

[Chandra Sekhar Reddy Mallam (cmallam)]

Core and agent binary is copied to: /cs-shared/bugs/1722706

[09:55:16][cmallam@nodeb6:/cs-shared/bugs/1722706]$ls -ltr
total 426092
-rw------- 1 praveenkv 181731328 Oct 11 09:32 core.contrail-vroute.15566.nodeb9.1507642498
-rwxr-xr-x 1 praveenkv 254586049 Oct 11 09:32 contrail-vrouter-agent*
[09:55:19][cmallam@nodeb6:/cs-shared/bugs/1722706]$

[Naveen N (naveenn)]

Crash was seen in private binary. Fixed with https://review.opencontrail.org/#/c/36716/