[3.0.2.0-28~liberty] contrail-collector crash @ boost::assertion::detail::assertion_failed_msg

Bug #1573164 reported by chhandak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
New
High
Anish Mehta
R3.0
New
Medium
Anish Mehta

Bug Description

Observed the contrail-collector crash in tor scale setup.

Setup has 8K Vn, 16K LIF, 16K VMI

Backtrace
----------------
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f1e691c1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0 0x00007f1e691c1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f1e691c50d8 in __GI_abort () at abort.c:89
#2 0x000000000044285c in boost::assertion::detail::assertion_failed_msg (expr=expr@entry=0x7ad558 "reply->type != REDIS_REPLY_ERROR",
    msg=0x142bd10 "BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.",
    function=function@entry=0x7ae000 <OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext const*, void*, void*)::__PRETTY_FUNCTION__> "void OpServerProxy::OpServerImpl::processorCallbackProcess(const redisAsyncContext*, void*, void*)",
    file=file@entry=0x7ad2c8 "controller/src/analytics/OpServerProxy.cc", line=line@entry=378) at /usr/include/boost/assert.hpp:108
#3 0x00000000005d56ec in OpServerProxy::OpServerImpl::processorCallbackProcess (this=<optimized out>, c=<optimized out>, r=<optimized out>,
    privdata=<optimized out>) at controller/src/analytics/OpServerProxy.cc:378
#4 0x000000000057d52a in operator() (a2=0x0, a1=0x1429e80, a0=0x1270370, this=0x7fffd7d264e0) at /usr/include/boost/function/function_template.hpp:767
#5 RedisAsyncConnection::RAC_AsyncCmdCallback (c=0x1270370, r=0x1429e80, privdata=0x0) at controller/src/analytics/redis_connection.cc:239
#6 0x000000000076bed3 in __redisRunCallback (cb=0x7fffd7d26600, cb=0x7fffd7d26600, reply=<optimized out>, ac=0x1270370)
    at build/third_party/hiredis/src/async.c:219
#7 redisProcessCallbacks (ac=0x1270370) at build/third_party/hiredis/src/async.c:417
#8 0x000000000076d2f9 in redisBoostClient::handle_read (this=0x1270320, ec=...) at build/third_party/hiredis/hiredis-boostasio-adapter/boostasio.cpp:62
#9 0x000000000076d9c4 in call<boost::shared_ptr<redisBoostClient>, boost::system::error_code> (b1=<synthetic pointer>, u=..., this=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:156
#10 operator()<boost::shared_ptr<redisBoostClient> > (a1=..., u=..., this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:171
#11 operator()<boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<const boost::system::error_code&, long unsigned int const&> > (a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:313
#12 operator()<boost::system::error_code, long unsigned int> (a2=<optimized out>, a1=..., this=<optimized out>)
    at /usr/include/boost/bind/bind_template.hpp:102
#13 operator() (this=<optimized out>) at /usr/include/boost/asio/detail/bind_handler.hpp:127
#14 asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long> > (function=...)
    at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#15 invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#16 boost::asio::detail::reactive_null_buffers_op<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >::do_complete (owner=<optimized out>, base=<optimized out>)
    at /usr/include/boost/asio/detail/reactive_null_buffers_op.hpp:75
#17 0x000000000061c50f in complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#18 boost::asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x1236230, base=0x1270680, ec=..., bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#19 0x000000000061dc67 in complete (bytes_transferred=5, ec=..., owner=..., this=0x1270680)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#20 do_run_one (ec=..., this_thread=..., lock=..., this=0x1236230) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#21 boost::asio::detail::task_io_service::run (this=0x1236230, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#22 0x000000000063e1f8 in run (this=0x1248380, ec=...) at /usr/include/boost/asio/impl/io_service.ipp:66
#23 EventManager::Run (this=0x1248380) at controller/src/io/event_manager.cc:32
#24 0x000000000042358d in main (argc=<optimized out>, argv=<optimized out>) at controller/src/analytics/main.cc:426

Tags: analytics
chhandak (chhandak)
Changed in juniperopenstack:
importance: Undecided → High
assignee: nobody → Raj Reddy (rajreddy)
information type: Proprietary → Public
Revision history for this message
chhandak (chhandak) wrote :

chhandak@ubuntu-build04:/cs-shared/bugs/1573164$ pwd
/cs-shared/bugs/1573164
chhandak@ubuntu-build04:/cs-shared/bugs/1573164$ ls -lrt
total 400648
-rwxrwxrwx 1 chhandak epbg 409436160 Apr 21 10:50 core.contrail-collec.7753.5b7s2.1461227225

Raj Reddy (rajreddy)
Changed in juniperopenstack:
assignee: Raj Reddy (rajreddy) → Anish Mehta (amehta00)
Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
Anish Mehta (amehta00) wrote :

Is this reproducible?
Are other files from /var/log available for this bug?
I'm specifically interested in /var/log/redis/redis-server.log

Revision history for this message
chhandak (chhandak) wrote :

No, I haven't seen this issue later on.
I don't have those log copied as the setup is no longer in the same state.
I will try to recreate the problem

Jeba Paulaiyan (jebap)
tags: removed: blocker
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.