Juniper Openstack

LB: Agent crash @ LoadbalancerPoolTable::CalculateProperties while deleting LB

  1. Series trunk
  2. Bug #1534348
Bug #1534348 reported by Senthilnathan Murugappan
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
High
Divakar Dharanalakota
Juniper Openstack r3.0-fcs

Bug Description

Observed agent crash @ LoadbalancerPoolTable::CalculateProperties while deleting LB membes, vips and then Pool in quick succession

Build: 2696 - ubuntu1404 - Kilo
Core will be stored at /cs-shared/bugs/<bug_id>

Relevant Logs:
2016-01-14 Thu 13:51:29:091.161 PST a2s42 [Thread 140450087634688, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:29:091.607 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:29:091.747 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:101.171 PST a2s42 [Thread 140450087634688, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:497.042 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:29:798.641 PST a2s42 [Thread 140450096031488, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:29:798.889 PST a2s42 [Thread 140450096031488, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:29:798.947 PST a2s42 [Thread 140450096031488, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:29:808.215 PST a2s42 [Thread 140450096031488, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:226.180 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:30:498.892 PST a2s42 [Thread 140450100229888, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:30:499.262 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:30:499.402 PST a2s42 [Thread 140450100229888, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:507.692 PST a2s42 [Thread 140450100229888, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:30:912.397 PST a2s42 [Thread 140450091833088, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0
2016-01-14 Thu 13:51:31:252.155 PST a2s42 [Thread 140450087634688, Pid 16258]: Unkown type:
2016-01-14 Thu 13:51:31:252.492 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS event notification for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 usable
2016-01-14 Thu 13:51:31:252.556 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:31:264.623 PST a2s42 [Thread 140450087634688, Pid 16258]: Service run command queued: /usr/bin/opencontrail-vrouter-netns create loadbalancer e5055191-b710-4c69-8125-9ad004d3e645 00000000-0000-0000-0000-000000000000 bee4fd0a-efd8-479a-8265-b78fdee7bab4 --vmi-left-ip 0.0.0.0/0 --vmi-right-ip 143.17.64.5/24 --vmi-left-mac 00:00:00:00:00:00 --vmi-right-mac 02:fa:fd:59:19:87 --gw-ip 143.17.64.1 --cfg-file /var/lib/contrail/loadbalancer/9b0d33b3-394e-4e94-adef-84f752eaad54/conf.json --pool-id 9b0d33b3-394e-4e94-adef-84f752eaad54 --update
2016-01-14 Thu 13:51:31:665.499 PST a2s42 [Thread 140450087634688, Pid 16258]: NetNS update status for uuid: 91651708-dcbe-4c7f-b9d3-f5f28dd98576 0

Program terminated with signal SIGSEGV, Segmentation fault.
#0 operator=<boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int>, boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int> > (k=..., this=<optimized out>) at /usr/include/boost/tuple/detail/tuple_basic.hpp:583
583 /usr/include/boost/tuple/detail/tuple_basic.hpp: No such file or directory.
(gdb) bt
#0 operator=<boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int>, boost::adjacency_iterator<boost::adjacency_list<boost::setS, boost::listS, boost::undirectedS, DBGraphBase::VertexProperties, DBGraphBase::EdgeProperties>, void*, boost::detail::out_edge_iter<std::_Rb_tree_const_iterator<boost::detail::stored_edge_iter<void*, std::_List_iterator<boost::list_edge<void*, DBGraphBase::EdgeProperties> >, DBGraphBase::EdgeProperties> >, void*, boost::detail::edge_desc_impl<boost::undirected_tag, void*>, long int>, long int> > (k=..., this=<optimized out>) at /usr/include/boost/tuple/detail/tuple_basic.hpp:583
#1 DBGraphVertex::adjacency_iterator::adjacency_iterator (this=0x7fbd16953840, graph=0x7fbd100011d0, vertex=0x0) at controller/src/db/db_graph_vertex.cc:19
#2 0x0000000000a0e3a6 in begin (this=0x7fbce40bdfc0, graph=0x7fbd100011d0) at controller/src/db/db_graph_vertex.h:85
#3 LoadbalancerPoolTable::CalculateProperties (this=this@entry=0x7fbd104844d0, graph=0x7fbd100011d0, node=0x7fbce40bdfc0, properties=properties@entry=0x7fbd169539c0, type=type@entry=0x7fbd169539b0)
    at controller/src/vnsw/agent/oper/loadbalancer_pool.cc:271
#4 0x0000000000a0f0fa in LoadbalancerPoolTable::OnChange (this=0x7fbd104844d0, entry=<optimized out>, request=0x7fbcf01a7f90) at controller/src/vnsw/agent/oper/loadbalancer_pool.cc:206
#5 0x0000000001045d40 in DBTable::Input (this=0x7fbd104844d0, tbl_partition=0x7fbd1048e460, client=<optimized out>, req=0x7fbcf01a7f90) at controller/src/db/db_table.cc:358
#6 0x0000000001045526 in DBPartition::QueueRunner::Run (this=0x7fbcf0187f00) at controller/src/db/db_partition.cc:187
#7 0x00000000011523bc in TaskImpl::execute (this=0x7fbd17e6d540) at controller/src/base/task.cc:253
#8 0x00007fbd1f2a1b3a in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fbd1f29d816 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fbd1f29cf4b in ?? () from /usr/lib/libtbb.so.2
#11 0x00007fbd1f2990ff in ?? () from /usr/lib/libtbb.so.2
#12 0x00007fbd1f2992f9 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007fbd1f4bd182 in start_thread (arg=0x7fbd16954700) at pthread_create.c:312
#14 0x00007fbd1e79647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

See original description
Tags: sanity vrouter
Senthilnathan Murugappan (msenthil)
description: updated
Revision history for this message
Jeba Paulaiyan (jebap) wrote :

Same crash seen in Sanity as well.

/cs-shared/test_runs/nodea12/2016_01_14_22_20_26/nodei14_core.contrail-vroute.2450.nodei14.1452800190.gz

tags: added: sanity
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/16493
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/16493
Committed: http://github.org/Juniper/contrail-controller/commit/0727f7ce6a162764cae3479892ef4140d5c2eb16
Submitter: Zuul
Branch: master

commit 0727f7ce6a162764cae3479892ef4140d5c2eb16
Author: Ashok Singh <email address hidden>
Date: Mon Jan 25 09:49:58 2016 -0800

Ignore change notification for loadbalancer-pool.

Ignore change notification for loadbalancer-pool when its ifmap-node is marked
for delete. During change notification handling for loadbalancer-pool we iterate
all its adjacent nodes. This iteration will result in crash if the entry is
marked for delete, because the graph vertex is reset as part of ifmap_node
deletion.

Also fix agent crash while converting fixed_ip to V4.

Change-Id: I6409e62e60952c71b7ced19d5a590e296b3b10aa
Closes-Bug: #1534348

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.