Observed agent crash @ LoadbalancerPoolTable::CalculateProperties while deleting LB membes, vips and then Pool in quick succession
Build: 2696 - ubuntu1404 - Kilo
Core will be stored at /cs-shared/bugs/<bug_id>
Relevant Logs:
2016-01-14 Thu 13:51:29:091.161 PST a2s42 [Thread 140450087634688, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:29:091.607 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:29:091.747 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:101.171 PST a2s42 [Thread 140450087634688, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:497.042 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:29:798.641 PST a2s42 [Thread 140450096031488, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:29:798.889 PST a2s42 [Thread 140450096031488, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:29:798.947 PST a2s42 [Thread 140450096031488, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:808.215 PST a2s42 [Thread 140450096031488, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:226.180 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:30:498.892 PST a2s42 [Thread 140450100229888, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:30:499.262 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:30:499.402 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:507.692 PST a2s42 [Thread 140450100229888, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:912.397 PST a2s42 [Thread 140450091833088, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:31:252.155 PST a2s42 [Thread 140450087634688, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:31:252.492 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:31:252.556 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:31:264.623 PST a2s42 [Thread 140450087634688, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:31:665.499 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
Program terminated with signal SIGSEGV, Segmentation fault.
#0 operator=<boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int>, boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int> > (k=..., this=<optimized out>) at /usr/include/boost/tuple/detail/tuple_basic.hpp:583
583 /usr/include/boost/tuple/detail/tuple_basic.hpp: No such file or directory.
(gdb) bt
#0 operator=<boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int>, boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int> > (k=..., this=<optimized out>) at /usr/include/boost/tuple/detail/tuple_basic.hpp:583
#1 DBGraphVertex::adjacency_iterator::adjacency_iterator (this=0x7fbd16953840, graph=0x7fbd100011d0, vertex=0x0) at controller/src/db/db_graph_vertex.cc:19
#2 0x0000000000a0e3a6 in begin (this=0x7fbce40bdfc0, graph=0x7fbd100011d0) at controller/src/db/db_graph_vertex.h:85
#3 LoadbalancerPoolTable::CalculateProperties (this=this@entry=0x7fbd104844d0, graph=0x7fbd100011d0, node=0x7fbce40bdfc0, properties=properties@entry=0x7fbd169539c0, type=type@entry=0x7fbd169539b0)
at controller/src/vnsw/agent/oper/loadbalancer_pool.cc:271
#4 0x0000000000a0f0fa in LoadbalancerPoolTable::OnChange (this=0x7fbd104844d0, entry=<optimized out>, request=0x7fbcf01a7f90) at controller/src/vnsw/agent/oper/loadbalancer_pool.cc:206
#5 0x0000000001045d40 in DBTable::Input (this=0x7fbd104844d0, tbl_partition=0x7fbd1048e460, client=<optimized out>, req=0x7fbcf01a7f90) at controller/src/db/db_table.cc:358
#6 0x0000000001045526 in DBPartition::QueueRunner::Run (this=0x7fbcf0187f00) at controller/src/db/db_partition.cc:187
#7 0x00000000011523bc in TaskImpl::execute (this=0x7fbd17e6d540) at controller/src/base/task.cc:253
#8 0x00007fbd1f2a1b3a in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fbd1f29d816 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fbd1f29cf4b in ?? () from /usr/lib/libtbb.so.2
#11 0x00007fbd1f2990ff in ?? () from /usr/lib/libtbb.so.2
#12 0x00007fbd1f2992f9 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007fbd1f4bd182 in start_thread (arg=0x7fbd16954700) at pthread_create.c:312
#14 0x00007fbd1e79647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Same crash seen in Sanity as well.
/cs-shared/ test_runs/ nodea12/ 2016_01_ 14_22_20_ 26/nodei14_ core.contrail- vroute. 2450.nodei14. 1452800190. gz