tor-agent crash at TcpServer::OnSessionClose

[Vedamurthy Joshi]

R2.20 Build 95 Ubuntu 14.04 Juno multi-node setup

A few BMS automated regression tests were run on this multinode setup.
Below tor-agent crash was seen during that time on nodeg11

Core will be in http://10.204.216.50/Docs/bugs/#

gdb /usr/bin/contrail-tor-agent core.contrail-tor-ag.19849.nodeg11.1441664897
....
....
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-tor-agent --config_file /etc/contrail/contrail-tor-agent-4.co'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f4273007cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f4273007cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f427300b0d8 in __GI_abort () at abort.c:89
#2 0x00007f4273000b86 in __assert_fail_base (fmt=0x7f4273151830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xe7dae7 "found",
    file=file@entry=0xede498 "controller/src/io/tcp_server.cc", line=line@entry=197, function=function@entry=0xede6e0 "void TcpServer::OnSessionClose(TcpSession*)") at assert.c:92
#3 0x00007f4273000c32 in __GI___assert_fail (assertion=0xe7dae7 "found", file=0xede498 "controller/src/io/tcp_server.cc", line=197, function=0xede6e0 "void TcpServer::OnSessionClose(TcpSession*)")
    at assert.c:101
#4 0x0000000000d35e51 in TcpServer::OnSessionClose(TcpSession*) ()
#5 0x0000000000d448d8 in TcpSession::CloseInternal(boost::system::error_code const&, bool, bool) ()
#6 0x0000000000d27101 in SslServer::AcceptHandShakeHandler(boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&) ()
#7 0x0000000000d2b793 in boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::handshake_op, boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&), boost::_bi::list3<boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > >::operator()(boost::system::error_code, unsigned long, int) ()
#8 0x0000000000d2c1ca in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ssl::detail::handshake_op, boost::_bi::bind_t<void, void (*)(boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&), boost::_bi::list3<boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#9 0x00000000008653b7 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#10 0x0000000000d244c1 in EventManager::RunWithExceptionHandling() ()
#11 0x00000000006f1911 in main ()
(gdb)

[Prabhjot Singh Sethi (prabhjot)]

side effect of bug-1488434

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13656
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13657
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13656
Committed: http://github.org/Juniper/contrail-controller/commit/6c1a837304c3eb48ceacca8733d4f4dcdf31c612
Submitter: Zuul
Branch: master

commit 6c1a837304c3eb48ceacca8733d4f4dcdf31c612
Author: Prabhjot Singh Sethi <email address hidden>
Date: Tue Sep 8 15:16:52 2015 +0530

Fix crash for handshake failure in ssl infra

Issue:
------
on handshake failure, session triggers sever notify_close
handler as well, while having only one session for which
handshake failed no issue was observed, however if there
are existing established/closed sessions present in the
ref map it results in assertion.
Since it is expected that session will be added to ref_map
only when the handshake is complete

Fix:
----
since SSL handshake is done in Server context, close
session should be called without server notify flag.

Added test-case for the same

Closes-Bug: 1493229
Change-Id: I07a6ce56dd206a753b14515a03e01fd2a933d2b2

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13657
Committed: http://github.org/Juniper/contrail-controller/commit/1585814163f59c668ec7bf954dd408b24f79b350
Submitter: Zuul
Branch: R2.20

commit 1585814163f59c668ec7bf954dd408b24f79b350
Author: Prabhjot Singh Sethi <email address hidden>
Date: Tue Sep 8 15:16:52 2015 +0530

Fix crash for handshake failure in ssl infra

Issue:
------
on handshake failure, session triggers sever notify_close
handler as well, while having only one session for which
handshake failed no issue was observed, however if there
are existing established/closed sessions present in the
ref map it results in assertion.
Since it is expected that session will be added to ref_map
only when the handshake is complete

Fix:
----
since SSL handshake is done in Server context, close
session should be called without server notify flag.

Added test-case for the same

Closes-Bug: 1493229
Change-Id: I07a6ce56dd206a753b14515a03e01fd2a933d2b2
(cherry picked from commit 6c1a837304c3eb48ceacca8733d4f4dcdf31c612)