query engine crash in SelectQuery::process_query

[vageesan]

contrail-query-engine crashed with following backtrace in solution test run

3.0-1,kilo.

core is in 10.84.5.112:/cs-shared/bugs/<bug-id>/

(gdb) bt
#0 0x00007fc25da7fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fc25da830d8 in __GI_abort () at abort.c:89
#2 0x00007fc25da78b86 in __assert_fail_base (
    fmt=0x7fc25dbc9830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x6b2517 "IsObject()",
    file=file@entry=0x6b2ae0 "build/include/rapidjson/document.h", line=line@entry=245,
    function=function@entry=0x6d7de0 <rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::MemberBegin()::__PRETTY_FUNCTION__> "rapidjson::GenericValue<Encoding, Allocator>::Member* rapidjson::GenericValue<Encoding, Allocator>::MemberBegin() [with Encoding = rapidjson::UTF8<>; Allocator = rapidjson::MemoryPoolAllocator<>; rapi"...) at assert.c:92
#3 0x00007fc25da78c32 in __GI___assert_fail (assertion=0x6b2517 "IsObject()",
    file=0x6b2ae0 "build/include/rapidjson/document.h", line=245,
    function=0x6d7de0 <rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::MemberBegin()::__PRETTY_FUNCTION__> "rapidjson::GenericValue<Encoding, Allocator>::Member* rapidjson::GenericValue<Encoding, Allocator>::MemberBegin() [with Encoding = rapidjson::UTF8<>; Allocator = rapidjson::MemoryPoolAllocator<>; rapi"...) at assert.c:101
#4 0x00000000004ad74e in MemberBegin (this=0x7fc254697350) at build/include/rapidjson/document.h:245
#5 SelectQuery::process_query (this=0x7fc2280838c0) at controller/src/query_engine/select.cc:451
#6 0x0000000000493a68 in AnalyticsQuery::process_query (this=0x7fc22808f120)
    at controller/src/query_engine/query.cc:973

#7 0x00000000004981d4 in QueryEngine::QueryExec (this=0x1eec550, handle=0x1f0db80, qp=..., chunk=9)
    at controller/src/query_engine/query.cc:1458
#8 0x000000000043e09d in operator() (a3=9,
    a2=<error reading variable: access outside bounds of object referenced via synthetic pointer>,
    a1=0x1f0db80, p=0x1eec550, this=0x7fc2280920d0)
---Type <return> to continue, or q <return> to quit---
    at /usr/include/boost/bind/mem_fn_template.hpp:393
#9 operator()<bool, boost::_mfi::mf3<bool, QueryEngine, void*, QueryEngine::QueryParams, unsigned int>, boost::_bi::list1<ExternalBase*&> > (a=<synthetic pointer>, f=..., this=0x7fc2280920e0)
    at /usr/include/boost/bind/bind.hpp:447
#10 operator()<ExternalBase*> (a1=<synthetic pointer>, this=0x7fc2280920d0)
    at /usr/include/boost/bind/bind_template.hpp:32
#11 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf3<bool, QueryEngine, void*, QueryEngine::QueryParams, unsigned int>, boost::_bi::list4<boost::_bi::value<QueryEngine*>, boost::arg<1>, boost::_bi::value<QueryEngine::QueryParams>, boost::_bi::value<unsigned int> > >, bool, ExternalBase*>::invoke (function_obj_ptr=..., a0=0x1f0db80)
    at /usr/include/boost/function/function_template.hpp:132
#12 0x0000000000441a01 in operator() (a0=0x1f0db80, this=0x7fc254697b40)
    at /usr/include/boost/function/function_template.hpp:767
#13 WorkProcessor<QEOpServerProxy::QEOpServerImpl::Input, QEOpServerProxy::QEOpServerImpl::Stage0Out, std::pair<QEOpServerProxy::QPerfInfo, std::pair<boost::shared_ptr<std::vector<std::pair<std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >, boost::shared_ptr<QueryResultMetaData> >, std::allocator<std::pair<std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >, boost::shared_ptr<QueryResultMetaData> > > > >, boost::shared_ptr<std::multimap<std::vector<boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::allocator<boost::variant<boost::blank, std::string, unsigned long, double, bo---Type <return> to continue, or q <return> to quit---
ost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > >, std::pair<std::map<std::string, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >, std::map<std::pair<QEOpServerProxy::AggOper, std::string>, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::pair<QEOpServerProxy::AggOper, std::string> >, std::allocator<std::pair<std::pair<QEOpServerProxy::AggOper, std::string> const, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::v---Type <return> to continue, or q <return> to quit---
ariant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > > >, std::less<std::vector<boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::allocator<boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >, std::allocator<std::pair<std::vector<boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::allocator<boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::deta---Type <return> to continue, or q <return> to quit---
il::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > const, std::pair<std::map<std::string, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >, std::map<std::pair<QEOpServerProxy::AggOper, std::string>, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::pair<QEOpServerProxy::AggOper, std::string> >, std::allocator<std::pair<std::pair<QEOpServerProxy::AggOper, std::string> const, boost::variant<boost::blank, std::string, unsigned long, double, boost::uuids::uuid, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::d---Type <return> to continue, or q <return> to quit---
etail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > > > > > > > > > >::Runner (this=0x1f0db80) at controller/src/base/work_processor-inl.h:116

#14 0x0000000000438539 in operator() (this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:767
#15 PipelineWorker::Run (this=<optimized out>) at controller/src/base/work_processor-inl.h:27
#16 0x000000000069f9fc in TaskImpl::execute (this=0x7fc256a82d40) at controller/src/base/task.cc:253
#17 0x00007fc25e856b3a in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fc25e852816 in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fc25e851f4b in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fc25e84e0ff in ?? () from /usr/lib/libtbb.so.2
#21 0x00007fc25e84e2f9 in ?? () from /usr/lib/libtbb.so.2
#22 0x00007fc25ea72182 in start_thread (arg=0x7fc254698700) at pthread_create.c:312
#23 0x00007fc25db4347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
(gdb)

[Sundaresan Rajangam (srajanga)]

Program terminated with signal SIGABRT, Aborted.
#0 0x00007fc25da7fcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) fr 5
#5 SelectQuery::process_query (this=0x7fc2280838c0) at controller/src/query_engine/select.cc:451
451 controller/src/query_engine/select.cc: No such file or directory.
(gdb) p it
$1 = {timestamp = 1456238548036787, info = std::vector of length 2, capacity 2 = {{which_ = 1, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32ul, 8ul>> = {data_ = {
            buf = "\250\060\177(\302\177\000\000\275\066\346c\274\353^Cd\000\000\000\000\000\000\000\060\241\000(\302\177\000", align_ = {<No data fields>}}}, static size = <optimized out>, static alignment = <optimized out>}}, {
      which_ = 4, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32ul, 8ul>> = {data_ = {buf = "\024:`\211f{H\266\265e\331䔉\275]", '\000' <repeats 15 times>, align_ = {<No data fields>}}},
        static size = <optimized out>, static alignment = <optimized out>}}}}
(gdb) p json_string
$2 = "{\"Source|s\":\"csol1-node11\",\"cpu_info.cpu_share|d\":inf,\"cpu_info.mem_res|n\":0,\"cpu_info.mem_virt|n\":32703,\"cpu_info.one_min_cpuload|d\":6.9397e-310,\"cpu_info.used_sys_mem|n\":1545655913,\"name|s\":\"csol1-n"...
(gdb) p d
$3 = {<rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >> = {static kDefaultArrayCapacity = <optimized out>, static kDefaultObjectCapacity = <optimized out>, data_ = {s = {
        str = 0x7fc22844e340 "\230\342D(\302\177", length = 7, hashcode = 7}, n = {i = {i = 675603264, padding = "\302\177\000"}, u = {u = 675603264, padding2 = "\302\177\000"}, i64 = 140471875986240, u64 = 140471875986240,
        d = 6.9402328131673157e-310}, o = {members = 0x7fc22844e340, size = 7, capacity = 7}, a = {elements = 0x7fc22844e340, size = 7, capacity = 7}}, flags_ = 0}, static kDefaultStackCapacity = <optimized out>, stack_ = {
    allocator_ = 0x7fc228003150, own_allocator_ = 0x7fc228003150, stack_ = 0x7fc22844de98 "", stack_top_ = 0x7fc22844de98 "", stack_end_ = 0x7fc22844e298 "Source|s", stack_capacity_ = 1024},
  parseError_ = 0x6cae30 "Expect a value here.", errorOffset_ = 50}
(gdb) set print elements 0
(gdb) p json_string
$4 = "{\"Source|s\":\"csol1-node11\",\"cpu_info.cpu_share|d\":inf,\"cpu_info.mem_res|n\":0,\"cpu_info.mem_virt|n\":32703,\"cpu_info.one_min_cpuload|d\":6.9397e-310,\"cpu_info.used_sys_mem|n\":1545655913,\"name|s\":\"csol1-node11\"}"
(gdb) p d.parseError_
$5 = 0x6cae30 "Expect a value here."

The json_string is invalid => \"cpu_info.cpu_share|d\":inf
iterating the json document without checking for parse error resulted in crash.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18185
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18187
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18187
Committed: http://github.org/Juniper/contrail-controller/commit/7121fbd2cbbf8f1a60f753a617e13168d189fd7d
Submitter: Zuul
Branch: R3.0

commit 7121fbd2cbbf8f1a60f753a617e13168d189fd7d
Author: Sundaresan Rajangam <email address hidden>
Date: Fri Mar 4 14:16:44 2016 -0800

Check for parse error before using the json document

For stats query, the json document needs to be checked for parse error
before iterating the document.

Change-Id: I589b4f7da9eb11fd3489d3d1e5ca39a85d5b9cca
Partial-Bug: #1548891
(cherry picked from commit 4107e5137cb4c11b6625ff9ab4b55d2ea93a515b)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18185
Committed: http://github.org/Juniper/contrail-controller/commit/4107e5137cb4c11b6625ff9ab4b55d2ea93a515b
Submitter: Zuul
Branch: master

commit 4107e5137cb4c11b6625ff9ab4b55d2ea93a515b
Author: Sundaresan Rajangam <email address hidden>
Date: Fri Mar 4 14:16:44 2016 -0800

Check for parse error before using the json document

For stats query, the json document needs to be checked for parse error
before iterating the document.

Change-Id: I589b4f7da9eb11fd3489d3d1e5ca39a85d5b9cca
Partial-Bug: #1548891