Juniper Openstack

[Ubuntu 14.04 Icehouse Buil 2510] OVSDB: crash @ OVSDB::OvsdbClientSession::OnClose

  1. Series r2.1
  2. Bug #1406481
Bug #1406481 reported by chhandak
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.1
Fix Committed
High
Prabhjot Singh Sethi
Juniper Openstack r2.1-fcs
Trunk
Fix Committed
High
Prabhjot Singh Sethi

Bug Description

Observed OVSDB crash while restarting for agent service and deactivating OVSDB protocol on QFX

#0 0x00007fa475200bb9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa475203fc8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4751f9a76 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4751f9b22 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x000000000111b377 in OVSDB::OvsdbClientSession::OnClose (this=0x7fa460013760)
    at controller/src/vnsw/agent/physical_devices/ovs_tor_agent/ovsdb_client/ovsdb_client_session.cc:36
#5 0x00000000010e1712 in OVSDB::OvsdbClientTcp::OnSessionEvent (this=0x7fa4600020c0, session=0x7fa460013938,
    event=TcpSession::CLOSE)
    at controller/src/vnsw/agent/physical_devices/ovs_tor_agent/ovsdb_client/ovsdb_client_tcp.cc:58
#6 0x00000000010e6db6 in boost::_mfi::mf2<void, OVSDB::OvsdbClientTcp, TcpSession*, TcpSession::Event>::operator() (
    this=0x7fa4600139e8, p=0x7fa4600020c0, a1=0x7fa460013938, a2=TcpSession::CLOSE)
    at /usr/include/boost/bind/mem_fn_template.hpp:280
#7 0x00000000010e6316 in boost::_bi::list3<boost::_bi::value<OVSDB::OvsdbClientTcp*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, OVSDB::OvsdbClientT
cp, TcpSession*, TcpSession::Event>, boost::_bi::list2<TcpSession*&, TcpSession::Event&> > (this=0x7fa4600139f8, f=..., a=...) at /usr/include/boost/bind/bind.hpp:392
#8 0x00000000010e5832 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, OVSDB::OvsdbClientTcp, TcpSession*, TcpSession::Event>, boost::_bi::list3<boost::_bi::value<OVSDB:
:OvsdbClientTcp*>, boost::arg<1>, boost::arg<2> > >::operator()<TcpSession*, TcpSession::Event> (this=0x7fa4600139e8, a1=@0x7fffdaedbd30: 0x7fa460013938,
    a2=@0x7fffdaedbd2c: TcpSession::CLOSE) at /usr/include/boost/bind/bind_template.hpp:61
#9 0x00000000010e5050 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, OVSDB::OvsdbClientTcp, TcpSession*, TcpSession
::Event>, boost::_bi::list3<boost::_bi::value<OVSDB::OvsdbClientTcp*>, boost::arg<1>, boost::arg<2> > >, void, TcpSession*, TcpSession::Event>::invoke (function_obj_ptr=...
,
    a0=0x7fa460013938, a1=TcpSession::CLOSE) at /usr/include/boost/function/function_template.hpp:153
#10 0x00000000015fbc2c in boost::function2<void, TcpSession*, TcpSession::Event>::operator() (this=0x7fa4600139e0,
    a0=0x7fa460013938, a1=TcpSession::CLOSE) at /usr/include/boost/function/function_template.hpp:767
#11 0x00000000015f3cf7 in TcpSession::CloseInternal (this=0x7fa460013938, call_observer=true, notify_server=true)
    at controller/src/io/tcp_session.cc:226
#12 0x00000000015f590e in TcpSession::AsyncReadHandler (session=..., buffer=..., error=..., bytes_transferred=0)
    at controller/src/io/tcp_session.cc:316
#13 0x0000000001600676 in boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost
::arg<2> (*)()>::operator()<void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list2<boos
t::system::error_code const&, unsigned long const&> > (this=0x7fffdaedc2a8,

    f=@0x7fffdaedc2a0: 0x15f5312 <TcpSession::AsyncReadHandler(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned lon
g)>, a=...) at /usr/include/boost/bind/bind.hpp:457
#14 0x00000000016004b4 in boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long),
 boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> >::operat
or()<boost::system::error_code, unsigned long> (this=0x7fffdaedc2a0, a1=..., a2=@0x7fffdaedc2d0: 0)
#14 0x00000000016004b4 in boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long),
 boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> >::operat
or()<boost::system::error_code, unsigned long> (this=0x7fffdaedc2a0, a1=..., a2=@0x7fffdaedc2d0: 0)
---Type <return> to continue, or q <return> to quit---
    at /usr/include/boost/bind/bind_template.hpp:102
#15 0x00000000016001e1 in boost::asio::detail::binder2<boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>::operator() (this=0x7fffdaedc2a0)
    at /usr/include/boost/asio/detail/bind_handler.hpp:127
#16 0x00000000015ffbfb in boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> > (function=...)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#17 0x00000000015ff608 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (function=..., context=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#18 0x00000000015fefdf in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpSession>, boost::asio::mutable_buffer, boost::system::error_code const&, unsigned long), boost::_bi::list4<boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::_bi::value<boost::asio::mutable_buffer>, boost::arg<1> (*)(), boost::arg<2> (*)()> > >::do_complete (owner=0x3b72ed0, base=0x7fa45801b370)
    at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
#19 0x0000000000fbf4a6 in boost::asio::detail::task_io_service_operation::complete (this=0x7fa45801b370, owner=...,
    ec=..., bytes_transferred=0) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#20 0x0000000000fc0521 in boost::asio::detail::task_io_service::do_run_one (this=0x3b72ed0, lock=..., this_thread=...,
    ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#21 0x0000000000fc0081 in boost::asio::detail::task_io_service::run (this=0x3b72ed0, ec=...)
    at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#22 0x00000000015e10d1 in boost::asio::io_service::run (this=0x3b69330, ec=...)
    at /usr/include/boost/asio/impl/io_service.ipp:66
#23 0x00000000015dfdb0 in EventManager::RunWithExceptionHandling (this=0x3b69330)
    at controller/src/io/event_manager.cc:51
#24 0x00000000010dcbb3 in main (argc=3, argv=0x7fffdaedcfa8)
    at controller/src/vnsw/agent/physical_devices/ovs_tor_agent/main.cc:75

root@nodec45:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-control 3.0-2510 2510
contrail-dns 3.0-2510 2510
contrail-fabric-utils 3.0-2510 2510
contrail-install-packages 3.0-2510~icehouse 2510
contrail-lib 3.0-2510 2510
contrail-nodemgr 3.0-2510 2510
contrail-nova-vif 3.0-2510 2510
contrail-openstack-control 3.0-2510 2510
contrail-openstack-database 3.0-2510 2510
contrail-openstack-vrouter 3.0-2510 2510
contrail-setup 3.0-2510 2510
contrail-utils 3.0-2510 2510
contrail-vrouter-3.13.0-40-generic 3.0-2510 2510
contrail-vrouter-agent 3.0-2510 2510
contrail-vrouter-common 3.0-2510 2510
contrail-vrouter-init 3.0-2510 2510
contrail-vrouter-utils 3.0-2510 2510
nova-common 1:2014.1.3-0ubuntu1~cloud0.2contrail2510
nova-compute 1:2014.1.3-0ubuntu1~cloud0.2contrail2510
nova-compute-kvm 1:2014.1.3-0ubuntu1~cloud0.2contrail2510
nova-compute-libvirt 1:2014.1.3-0ubuntu1~cloud0.2contrail2510
python-contrail 3.0-2510 2510
python-contrail-vrouter-api 3.0-2510 2510
python-neutronclient 2:2.3.4-0ubuntu1.2contrail 2510
python-nova 1:2014.1.3-0ubuntu1~cloud0.2contrail2510
python-opencontrail-vrouter-netns 3.0-2510 2510
root@nodec45:~#

See original description
chhandak (chhandak)
description: updated
Revision history for this message
chhandak (chhandak) wrote :

Logs saved at http://mayamruga.englab.juniper.net/bugs/1406481

Nagabhushana R (bhushana)
tags: added: blocker
Ashish Ranjan (aranjan-n)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/6753
Committed: http://github.org/Juniper/contrail-controller/commit/996d9314091115b8beb263485eb67958b3626561
Submitter: Zuul
Branch: R2.1

commit 996d9314091115b8beb263485eb67958b3626561
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Jan 28 22:45:14 2015 -0800

Handling for OVSDB Connection close

Adding KSync Object delete function, where we schedule deletion
of a ksync object and trigger delete for all the entries under
it, which then on empty table callback can delete KSync Object.

On OVSDB Session Close, trigger delete for OVSDB IDL, which
starts cleanup of all the ksync tables and allocate and start
new OVSDB session. On completion of cleanup delete session and
associated IDL object.

Change-Id: Ia2242c76501cd4d6c2e6c3cef394274ccc674dd5
Closes-Bug: 1406481

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/6535
Committed: http://github.org/Juniper/contrail-controller/commit/b0bdf92ff9826412180649351ecfd53314d6aff4
Submitter: Zuul
Branch: master

commit b0bdf92ff9826412180649351ecfd53314d6aff4
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Jan 28 22:45:14 2015 -0800

Handling for OVSDB Connection close

Adding KSync Object delete function, where we schedule deletion
of a ksync object and trigger delete for all the entries under
it, which then on empty table callback can delete KSync Object.

On OVSDB Session Close, trigger delete for OVSDB IDL, which
starts cleanup of all the ksync tables and allocate and start
new OVSDB session. On completion of cleanup delete session and
associated IDL object.

Closes-Bug: 1406481
Change-Id: I567742330152c9a60b9330c49f0d619cb0626f1a

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.