build-2280-mainline: agent crash in ShowIFMapAgentTable

[Saravanan Musuvathi]

Not sure of the trigger...back trace as follows:

corefile saved in bhushana@mayamruga-E71:$ ~/Documents/technical/bugs/bug-id/

Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 11, Segmentation fault.
#0 _Rb_tree_const_iterator (this=<synthetic pointer>) at build/include/boost/tuple/detail/tuple_basic.hpp:583
583 build/include/boost/tuple/detail/tuple_basic.hpp: No such file or directory.
(gdb) bt
#0 _Rb_tree_const_iterator (this=<synthetic pointer>) at build/include/boost/tuple/detail/tuple_basic.hpp:583
#1 iterator_adaptor (this=<synthetic pointer>) at build/include/boost/iterator/iterator_adaptor.hpp:273
#2 out_edge_iter (this=0x7f722ea6e8d0) at build/include/boost/graph/detail/adjacency_list.hpp:146
#3 iterator_adaptor (this=0x7f722ea6e8d0) at build/include/boost/iterator/iterator_adaptor.hpp:273
#4 adjacency_iterator (this=0x7f722ea6e8d0) at build/include/boost/graph/adjacency_iterator.hpp:39
#5 DBGraphVertex::adjacency_iterator::adjacency_iterator (this=0x7f722ea6e8b0, graph=0x7f7220002570, vertex=0x0) at controller/src/db/db_graph_vertex.cc:18
#6 0x0000000000c9bd0e in begin (graph=<optimized out>, this=<optimized out>) at controller/src/db/db_graph_vertex.h:85
#7 ShowIFMapAgentTable::MakeNode (dst=..., src=0x7f721c00d5c0, this=<optimized out>) at controller/src/ifmap/ifmap_agent_sandesh.cc:164
#8 0x0000000000c9bf52 in ShowIFMapAgentTable::TableToBuffer (this=<optimized out>, table=<optimized out>, show_data=0x7f72243bfd70) at controller/src/ifmap/ifmap_agent_sandesh.cc:180
#9 0x0000000000c9c2bf in BufferAllTables (data=0x7f72243bfd70, this=0x7f722ea6ebdd, ps=...) at controller/src/ifmap/ifmap_agent_sandesh.cc:213
#10 ShowIFMapAgentTable::BufferStage (this=0x7f722ea6ebdd, sr=<optimized out>, ps=..., stage=<optimized out>, instNum=<optimized out>, data=0x7f72243bfd70)
    at controller/src/ifmap/ifmap_agent_sandesh.cc:230
#11 0x0000000000c9da69 in operator() (a5=0x7f72243bfd70, a4=0, a2=<error reading variable: access outside bounds of object referenced via synthetic pointer>, a1=0x7f72200f9c20, p=0x7f722ea6ebdd,
    this=0x7f7224230398, a3=0) at build/include/boost/bind/mem_fn_template.hpp:619
#12 operator()<bool, boost::_mfi::mf5<bool, ShowIFMapAgentTable, const Sandesh*, RequestPipeline::PipeSpec, int, int, RequestPipeline::InstData*>, boost::_bi::list5<const Sandesh*&, const RequestPipeline::PipeSpec&, int&, int&, RequestPipeline::InstData*&> > (a=<synthetic pointer>, f=..., this=0x7f72242303a8) at build/include/boost/bind/bind.hpp:586
#13 operator()<const Sandesh*, const RequestPipeline::PipeSpec, int, int, RequestPipeline::InstData*> (a5=<synthetic pointer>, a4=<synthetic pointer>, a3=<synthetic pointer>, a2=...,
    a1=<synthetic pointer>, this=0x7f7224230398) at build/include/boost/bind/bind_template.hpp:174
#14 boost::detail::function::function_obj_invoker5<boost::_bi::bind_t<bool, boost::_mfi::mf5<bool, ShowIFMapAgentTable, Sandesh const*, RequestPipeline::PipeSpec, int, int, RequestPipeline::InstData*>, boost::_bi::list6<boost::_bi::value<ShowIFMapAgentTable*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::arg<5> > >, bool, Sandesh const*, RequestPipeline::PipeSpec const&, int, int, RequestPipeline::InstData*>::invoke (function_obj_ptr=..., a0=0x7f72200f9c20, a1=..., a2=0, a3=0, a4=0x7f72243bfd70) at build/include/boost/function/function_template.hpp:132
#15 0x0000000000d6bf6f in RequestPipeline::StageWorker::Run (this=0x7f7224b7b590) at tools/sandesh/library/cpp/request_pipeline.cc:84
#16 0x0000000000ddc7e0 in TaskImpl::execute (this=0x7f7224080ec0) at controller/src/base/task.cc:224
#17 0x00007f7231157ece in ?? () from /usr/lib/libtbb_debug.so.2
#18 0x00007f723114ee0b in ?? () from /usr/lib/libtbb_debug.so.2
#19 0x00007f723114d6f2 in ?? () from /usr/lib/libtbb_debug.so.2
#20 0x00007f72311483ce in ?? () from /usr/lib/libtbb_debug.so.2
#21 0x00007f7231148270 in ?? () from /usr/lib/libtbb_debug.so.2
#22 0x00007f7230c9fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007f72301baccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#24 0x0000000000000000 in ?? ()
(gdb)

[alok kumar (kalok)]

hitting the same issue on R2.0 build 12.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000d83a50 in DBGraphVertex::adjacency_iterator::adjacency_iterator(DBGraph*, void*) ()
(gdb) bt
#0 0x0000000000d83a50 in DBGraphVertex::adjacency_iterator::adjacency_iterator(DBGraph*, void*) ()
#1 0x0000000000d0e008 in ShowIFMapAgentTable::MakeNode(std::string&, DBEntryBase*) ()
#2 0x0000000000d0e6d7 in ShowIFMapAgentTable::TableToBuffer(IFMapTable*, ShowIFMapAgentTable::ShowData*) ()
#3 0x0000000000d0e811 in ShowIFMapAgentTable::BufferAllTables(RequestPipeline::PipeSpec, RequestPipeline::InstData*) ()
#4 0x0000000000d0fb3b in ShowIFMapAgentTable::BufferStage(Sandesh const*, RequestPipeline::PipeSpec, int, int, RequestPipeline::InstData*) ()
#5 0x0000000000d105fc in boost::detail::function::function_obj_invoker5<boost::_bi::bind_t<bool, boost::_mfi::mf5<bool, ShowIFMapAgentTable, Sandesh const*, RequestPipeline::PipeSpec, int, int, RequestPipeline::InstData*>, boost::_bi::list6<boost::_bi::value<ShowIFMapAgentTable*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4>, boost::arg<5> > >, bool, Sandesh const*, RequestPipeline::PipeSpec const&, int, int, RequestPipeline::InstData*>::invoke(boost::detail::function::function_buffer&, Sandesh const*, RequestPipeline::PipeSpec const&, int, int, RequestPipeline::InstData*) ()
#6 0x0000000000de7d73 in RequestPipeline::PipeImpl::RunInstance(int) ()
#7 0x0000000000de9495 in RequestPipeline::StageWorker::Run() ()
#8 0x0000000000e67dd0 in TaskImpl::execute() ()
#9 0x00007f5426adbb3a in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f5426ad7816 in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f5426ad6f4b in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f5426ad30ff in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f5426ad32f9 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f5426cf7182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f5425fcffbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) q

all logs @bhushana@mayamruga-E71:$ ~/Documents/technical/bugs/1351172/alok/

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6495
Committed: http://github.org/Juniper/contrail-controller/commit/776328a13ed2da2593b63d7cac163d69df7fadd6
Submitter: Zuul
Branch: R2.1

commit 776328a13ed2da2593b63d7cac163d69df7fadd6
Author: Divakar <email address hidden>
Date: Thu Jan 22 03:02:25 2015 -0800

closes-bug:1351172 While printing Agent config introspect not processing the Delete Marked node adjacencies and printing this node as Delete Marked Config

Change-Id: Ic6deb10873c4347034766d05e0cd16f3378beeae

[Divakar Dharanalakota (ddivakar)]

Fix for this is committed with the review request . https://review.opencontrail.org/#/c/6493/

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6493
Committed: http://github.org/Juniper/contrail-controller/commit/8b70969fdaa24485382b919bd45efe1dea61a4a2
Submitter: Zuul
Branch: master

commit 8b70969fdaa24485382b919bd45efe1dea61a4a2
Author: Divakar <email address hidden>
Date: Thu Jan 22 03:02:25 2015 -0800

closes-bug:1351172 While printing Agent config introspect not processing the Delete Marked node adjacencies and printing this node as Delete Marked Config

Change-Id: Ic6deb10873c4347034766d05e0cd16f3378beeae

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6980
Committed: http://github.org/Juniper/contrail-controller/commit/ef9459dd304061ac8b7b6eddcd0f4d5558d7cdbb
Submitter: Zuul
Branch: R2.0

commit ef9459dd304061ac8b7b6eddcd0f4d5558d7cdbb
Author: Divakar <email address hidden>
Date: Thu Jan 22 03:02:25 2015 -0800

closes-bug:1351172 While printing Agent config introspect not processing the Delete Marked node adjacencies and printing this node as Delete Marked Config

Change-Id: Ic6deb10873c4347034766d05e0cd16f3378beeae