agent crash due to seg fault at BindUtil::ParseDnsQuery

[Vedamurthy Joshi]

R2.1 Build 39 Ubuntu 14.04 Multi-node setup

Below crash was seen two times on a compute node. Per Hari, this is due to incorrect handling of a DNS packet (this is coming from a service instance ).

Cores will be in http://10.204.216.50/Docs/bugs/#

Reading symbols from /root/contrail-vrouter-agent...done.

warning: core file may not match specified executable file.
[New LWP 2593]
[New LWP 2595]
[New LWP 1651]
[New LWP 1300]
[New LWP 2592]
[New LWP 2594]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ReadShort (value=@0x7f90016e13d2: 0, ptr=0x261b000 <error: Cannot access memory at address 0x261b000>)
    at controller/src/dns/bind/bind_util.h:323
323 controller/src/dns/bind/bind_util.h: No such file or directory.
(gdb) bt
#0 ReadShort (value=@0x7f90016e13d2: 0, ptr=0x261b000 <error: Cannot access memory at address 0x261b000>)
    at controller/src/dns/bind/bind_util.h:323
#1 ReadQuestionEntry (item=..., ptr=0x261b000 <error: Cannot access memory at address 0x261b000>,
    buf=0x2601782 "\002\001\006\001\204\316\035'") at controller/src/dns/bind/bind_util.cc:305
#2 BindUtil::ParseDnsQuery (buf=0x2601782 "\002\001\006\001\204\316\035'", items=...) at controller/src/dns/bind/bind_util.cc:329
#3 0x00000000009fc56e in DnsHandler::HandleDefaultDnsRequest (this=this@entry=0x7f8ff4287f80, vmitf=vmitf@entry=0x7f8ffc0969e0)
    at controller/src/vnsw/agent/services/dns_handler.cc:141
#4 0x00000000009fd774 in DnsHandler::HandleRequest (this=0x7f8ff4287f80) at controller/src/vnsw/agent/services/dns_handler.cc:106
#5 0x0000000000a770b6 in Proto::ProcessProto (this=<optimized out>, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:44
#6 0x0000000000a77acc in operator() (a1=..., p=<optimized out>, this=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:165
#7 operator()<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list1<boost::shared_ptr<PktInfo>&> > (
    a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:303
#8 operator()<boost::shared_ptr<PktInfo> > (a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#9 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >, bool, boost::shared_ptr<PktInfo> >::invoke (function_obj_ptr=...,
    a0=...) at /usr/include/boost/function/function_template.hpp:132
#10 0x0000000000a78e6f in operator() (a0=..., this=0x7f90016e1b90) at /usr/include/boost/function/function_template.hpp:767
#11 QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::RunQueue (this=0x25f0140)
    at controller/src/base/queue_task.h:53
#12 0x0000000000ed4b90 in TaskImpl::execute (this=0x7f9001cdba40) at controller/src/base/task.cc:232
#13 0x00007f9009665b3a in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f9009661816 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f9009660f4b in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f900965d0ff in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f900965d2f9 in ?? () from /usr/lib/libtbb.so.2
#18 0x00007f9009881182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007f9008b59fbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8126
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8132
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8126
Committed: http://github.org/Juniper/contrail-controller/commit/ed05779a90375987bf5f3525ba45c85a061df636
Submitter: Zuul
Branch: R2.0

commit ed05779a90375987bf5f3525ba45c85a061df636
Author: Hari <email address hidden>
Date: Thu Mar 5 23:29:28 2015 -0800

Received an ICMP dest unreachable message (not for flow miss reason).
Agent parses the inside packet and when it is a DNS pkt, the packet is fwded
to DNS module. Adding a check so that this is done only for flow miss case.

Change-Id: I11b962f6e43e1726beb1c3f0bc26eb5b871c4858
Closes-bug: #1428953

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8165
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8132
Committed: http://github.org/Juniper/contrail-controller/commit/896dbbd6cc886bb24c0a0a91b109b4d3c6936d03
Submitter: Zuul
Branch: master

commit 896dbbd6cc886bb24c0a0a91b109b4d3c6936d03
Author: Hari <email address hidden>
Date: Thu Mar 5 23:29:28 2015 -0800

Received an ICMP dest unreachable message (not for flow miss reason).
Agent parses the inside packet and when it is a DNS pkt, the packet is fwded
to DNS module. Adding a check so that this is done only for flow miss case.

Change-Id: I7cf416b20d036984265de7df8d9453718126051b
Closes-bug: #1428953

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8165
Committed: http://github.org/Juniper/contrail-controller/commit/77cab4a1045fd0ed79c20f6302d10b3998f0a9b1
Submitter: Zuul
Branch: R2.1

commit 77cab4a1045fd0ed79c20f6302d10b3998f0a9b1
Author: Hari <email address hidden>
Date: Thu Mar 5 23:29:28 2015 -0800

Received an ICMP dest unreachable message (not for flow miss reason).
Agent parses the inside packet and when it is a DNS pkt, the packet is fwded
to DNS module. Adding a check so that this is done only for flow miss case.

Closes-bug: #1428953
(cherry picked from commit ed05779a90375987bf5f3525ba45c85a061df636)

Conflicts:
 src/vnsw/agent/pkt/pkt_handler.cc

Change-Id: I05d8cc8fd41b4ea3086c10877f2db564a61474b2