agent crash at boost::asio::ip::address::to_v6 during a hping port-scan test

[Vedamurthy Joshi]

R3.0.2.0 Build 26 ubuntu 14.04 Kilo

On running n X n hping port-scan, observed the below crashed pretty regularly on a node

Core will be in http://10.204.216.50/Docs/bugs/#

This bt got when run inside of gdb,
(gdb) bt
#0 0x00007ffff5b7d8b0 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00000000013bf788 in boost::throw_exception<std::bad_cast> (e=...) at /usr/include/boost/throw_exception.hpp:67
#2 0x0000000001608c3a in boost::asio::ip::address::to_v6 (this=0x7fff904a48b0) at /usr/include/boost/asio/ip/impl/address.ipp:118
#3 0x000000000184bb12 in Dhcpv6Handler::SendDhcpResponse (this=0x7fffc9fa7630) at controller/src/vnsw/agent/services/dhcpv6_handler.cc:648
#4 0x0000000001849702 in Dhcpv6Handler::Run (this=0x7fffc9fa7630) at controller/src/vnsw/agent/services/dhcpv6_handler.cc:348
#5 0x0000000001965a49 in Proto::RunProtoHandler (this=0x7fffcff8d110, handler=0x7fffc9fa7630) at controller/src/vnsw/agent/pkt/proto.cc:51
#6 0x0000000001965b36 in Proto::ProcessProto (this=0x7fffcff8d110, msg_info=...) at controller/src/vnsw/agent/pkt/proto.cc:66
#7 0x00000000019689a8 in boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >::operator() (this=0x7fffed0f4b18, p=0x7fffcff8d110, a1=...)
    at /usr/include/boost/bind/mem_fn_template.hpp:165
#8 0x0000000001968354 in boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> >::operator()<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list1<boost::shared_ptr<PktInfo>&> > (this=0x7fffed0f4b28, f=..., a=...) at /usr/include/boost/bind/bind.hpp:303
#9 0x0000000001967cc8 in boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >::operator()<boost::shared_ptr<PktInfo> > (this=0x7fffed0f4b18, a1=...) at /usr/include/boost/bind/bind_template.hpp:32
#10 0x00000000019678f9 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >, bool, boost::shared_ptr<PktInfo> >::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#11 0x0000000001969146 in boost::function1<bool, boost::shared_ptr<PktInfo> >::operator() (this=0x7fffed0f4b10, a0=...) at /usr/include/boost/function/function_template.hpp:767
#12 0x0000000001968d20 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::RunQueue (this=0x7fff900cc470) at controller/src/base/queue_task.h:87
#13 0x0000000001968b26 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::Run (this=0x7fff900cc470) at controller/src/base/queue_task.h:66
#14 0x0000000002080285 in TaskImpl::execute (this=0x7fffee9f3e40) at controller/src/base/task.cc:253
#15 0x00007ffff5e43b3a in ?? () from /usr/lib/libtbb.so.2
#16 0x00007ffff5e3f816 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007ffff5e3ef4b in ?? () from /usr/lib/libtbb.so.2
#18 0x00007ffff5e3b0ff in ?? () from /usr/lib/libtbb.so.2
#19 0x00007ffff5e3b2f9 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007ffff605f182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007ffff533847d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

In normal run mode, the bt got is :

#0 0x00007f1b1d012cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f1b1d0160d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f1b1d00bb86 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f1b1d00bc32 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000020805d7 in TaskImpl::execute (this=0x7f1b1fcf7b40) at controller/src/base/task.cc:276
#5 0x00007f1b1dbe1b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007f1b1dbdd816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007f1b1dbdcf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007f1b1dbd90ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007f1b1dbd92f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f1b1ddfd182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007f1b1d0d647d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) quit

[Hari Prasad Killi (haripk)]

The incoming IPv4 packet had UDP traffic sent with DHCPv6 port numbers. This was being classified as DHCPv6 packet, but the traffic was IPv4. This is causing the crash seen.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19377
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19379
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19381
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19382
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19383
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19377
Committed: http://github.org/Juniper/contrail-controller/commit/35e078d12de9496fb0778471d1729545c3f83e7a
Submitter: Zuul
Branch: R3.0

commit 35e078d12de9496fb0778471d1729545c3f83e7a
Author: Hari <email address hidden>
Date: Mon Apr 18 11:11:39 2016 +0530

Check for packet being IPv6 while sending it to DHCPv6 module.

When an IPv4 packet has UDP port as DHCPv6 client or server ports,
dont consider it as DHCPv6 packet. Add test case for the same.

Change-Id: I2010b6a04caf4c23116d8ab262d84f246efc0ba0
closes-bug: #1570705

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19383
Committed: http://github.org/Juniper/contrail-controller/commit/e3236f2205eb3063c2b456dcd5bc14916fc8dab4
Submitter: Zuul
Branch: R2.21.x

commit e3236f2205eb3063c2b456dcd5bc14916fc8dab4
Author: Hari <email address hidden>
Date: Mon Apr 18 11:34:23 2016 +0530

Check for packet being IPv6 while sending it to DHCPv6 module.

When an IPv4 packet has UDP port as DHCPv6 client or server ports,
dont consider it as DHCPv6 packet. Add test case for the same.

Change-Id: I2010b6a04caf4c23116d8ab262d84f246efc0ba0
closes-bug: #1570705

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19382
Committed: http://github.org/Juniper/contrail-controller/commit/562a243ffea5daca3ec2d6228a6e7399a58f4042
Submitter: Zuul
Branch: R2.22.x

commit 562a243ffea5daca3ec2d6228a6e7399a58f4042
Author: Hari <email address hidden>
Date: Mon Apr 18 11:30:00 2016 +0530

Check for packet being IPv6 while sending it to DHCPv6 module.

When an IPv4 packet has UDP port as DHCPv6 client or server ports,
dont consider it as DHCPv6 packet. Add test case for the same.

Change-Id: I2010b6a04caf4c23116d8ab262d84f246efc0ba0
closes-bug: #1570705

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19381
Committed: http://github.org/Juniper/contrail-controller/commit/ac1180752c0cc47491b7999a1f81a783a80af30a
Submitter: Zuul
Branch: R2.20

commit ac1180752c0cc47491b7999a1f81a783a80af30a
Author: Hari <email address hidden>
Date: Mon Apr 18 11:28:52 2016 +0530

Check for packet being IPv6 while sending it to DHCPv6 module.

When an IPv4 packet has UDP port as DHCPv6 client or server ports,
dont consider it as DHCPv6 packet. Add test case for the same.

Change-Id: I2010b6a04caf4c23116d8ab262d84f246efc0ba0
closes-bug: #1570705

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19379
Committed: http://github.org/Juniper/contrail-controller/commit/c35dd147919144adb102fe88de22ce4705c487ac
Submitter: Zuul
Branch: master

commit c35dd147919144adb102fe88de22ce4705c487ac
Author: Hari <email address hidden>
Date: Mon Apr 18 11:26:44 2016 +0530

Check for packet being IPv6 while sending it to DHCPv6 module.

When an IPv4 packet has UDP port as DHCPv6 client or server ports,
dont consider it as DHCPv6 packet. Add test case for the same.

Change-Id: I2010b6a04caf4c23116d8ab262d84f246efc0ba0
closes-bug: #1570705