[2.11-Build 51] TOR Agent Assertion @ OVSDB::OvsdbClientIdl::trigger_deletion

[Prabhjot Singh Sethi]

root@nodei32:~/bugs/1443720$ gdb ./contrail-tor-agent core.contrail-tor-ag.29313.system003.1428972668
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./contrail-tor-agent...done.

warning: core file may not match specified executable file.
[New LWP 29771]
[New LWP 29313]
[New LWP 29769]
[New LWP 29770]
[New LWP 29772]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-tor-agent --config_file /etc/contrail/contrail-tor-agent-1.co'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fa5642d6bb9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fa5642d6bb9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa5642d9fc8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa5642cfa76 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa5642cfb22 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000008db67b in OVSDB::OvsdbClientIdl::trigger_deletion (this=0x7fa55001fd60) at controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_idl.cc:373
#5 0x00000000008dec35 in OVSDB::OvsdbClientSession::OnClose (this=this@entry=0x7fa5580166a0) at controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_session.cc:43
#6 0x00000000008b12a8 in OVSDB::OvsdbClientTcpSession::ProcessSessionEvent (this=0x7fa5580166a0, ovs_event=...) at controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_tcp.cc:174
#7 0x00000000008b2660 in operator() (a1=..., p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:165
#8 operator()<bool, boost::_mfi::mf1<bool, OVSDB::OvsdbClientTcpSession, OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent>, boost::_bi::list1<OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent&> > (
    a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:303
#9 operator()<OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent> (a1=<synthetic pointer>, this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#10 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, OVSDB::OvsdbClientTcpSession, OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent>, boost::_bi::list2<boost::_bi::value<OVSDB::OvsdbClientTcpSession*>, boost::arg<1> > >, bool, OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent>::invoke (function_obj_ptr=..., a0=...)
    at /usr/include/boost/function/function_template.hpp:132
#11 0x00000000008b3eaa in operator() (a0=..., this=0x7fa55d159a90) at /usr/include/boost/function/function_template.hpp:767
#12 RunQueue (this=0x7fa5480297c0) at controller/src/base/queue_task.h:53
#13 QueueTaskRunner<OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent, WorkQueue<OVSDB::OvsdbClientTcpSession::OvsdbSessionEvent> >::Run (this=0x7fa5480297c0) at controller/src/base/queue_task.h:36
#14 0x0000000000d18ae0 in TaskImpl::execute (this=0x7fa55db17d40) at controller/src/base/task.cc:232
#15 0x00007fa5654deb3a in ?? () from /usr/lib/libtbb.so.2
#16 0x00007fa5654da816 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007fa5654d9f4b in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fa5654d60ff in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fa5654d62f9 in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fa5656fa182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007fa56439afbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[Prabhjot Singh Sethi (prabhjot)]

issue happened because of trying to trigger close twice for a session, once because of session being close by remote and other because of keepalive getting timed out.

(gdb) python print_trace_buffer('Ovsdb')
2015-04-14 06:20:28.928205 OvsdbTrace: log = "Connection to client established" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_session.cc" line = 35
2015-04-14 06:20:28.928489 OvsdbTrace: log = "Allocating OVS Peer 0x7fa5500219f0 Gen-Id 0" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_route_peer.cc" line = 19
2015-04-14 06:20:38.928702 OvsdbTrace: log = "Sending Monitor Request" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_idl.cc" line = 168
2015-04-14 06:21:00.126945 OvsdbTrace: log = "Connection to client Closed" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_session.cc" line = 42
2015-04-14 06:21:08.929152 OvsdbError: error = "KeepAlive failed, Closing Session" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_idl.cc" line = 363
2015-04-14 06:21:08.929167 OvsdbTrace: log = "Connection to client Closed" file = "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_session.cc" line = 42

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9156
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9185
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9186
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9185
Committed: http://github.org/Juniper/contrail-controller/commit/f08f457b1a6947eeb4aadfdfa45d5b2df6e6e189
Submitter: Zuul
Branch: R2.1

commit f08f457b1a6947eeb4aadfdfa45d5b2df6e6e189
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 15 15:38:18 2015 +0530

Fix TOR Agent Crash on TriggerDelete

Issue:
------
keepalive timer keeps running on closed session, results
on trigger close on timer expire, resulting in
trigger_delete twice.

Fix:
----
Cancel the KeepAlive timer on session close.
Ignore duplicate close trigger request.
Added test case

Closes-Bug: 1444375
(cherry picked from commit b6bbf6ac07bd987bd0b903c94a39d2ce198d1689)

Conflicts:
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/ovsdb_client_idl.cc
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/test/test_ovs_base.cc

Change-Id: I1ff32b13a8217766e897fca1383be3e3940b0ceb

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9186
Committed: http://github.org/Juniper/contrail-controller/commit/655d77af6599e0dc93fdb2454fa750e59892781f
Submitter: Zuul
Branch: R2.20

commit 655d77af6599e0dc93fdb2454fa750e59892781f
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 15 15:38:18 2015 +0530

Fix TOR Agent Crash on TriggerDelete

Issue:
------
keepalive timer keeps running on closed session, results
on trigger close on timer expire, resulting in
trigger_delete twice.

Fix:
----
Cancel the KeepAlive timer on session close.
Ignore duplicate close trigger request.
Added test case

Closes-Bug: 1444375
(cherry picked from commit b6bbf6ac07bd987bd0b903c94a39d2ce198d1689)

Change-Id: Ic9e6e18e8d4156afe2d42b94c23531e48277268a

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9156
Committed: http://github.org/Juniper/contrail-controller/commit/b6bbf6ac07bd987bd0b903c94a39d2ce198d1689
Submitter: Zuul
Branch: master

commit b6bbf6ac07bd987bd0b903c94a39d2ce198d1689
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 15 15:38:18 2015 +0530

Fix TOR Agent Crash on TriggerDelete

Issue:
------
keepalive timer keeps running on closed session, results
on trigger close on timer expire, resulting in
trigger_delete twice.

Fix:
----
Cancel the KeepAlive timer on session close.
Ignore duplicate close trigger request.
Added test case

Closes-Bug: 1444375
Change-Id: I52bc8d21dd922bccf7cf283e9be1a1cafcecdd30