tor-agent crash at OVSDB::LogicalSwitchEntry::Sync on tor-scale setup

[Vedamurthy Joshi]

R2.1 44 Ubuntu 14.04 multi-node icehouse setup

Multiple tor-agent crashes were seen with below backtrace

Core files will be in http://10.204.216.50/Docs/bugs/#

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-tor-agent --config_file /etc/contrail/contrail-tor-agent-131.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000008cb814 in _M_data (this=<optimized out>) at /usr/include/c++/4.8/bits/basic_string.h:293
293 { return _M_dataplus._M_p; }
(gdb) bt
#0 0x00000000008cb814 in _M_data (this=<optimized out>) at /usr/include/c++/4.8/bits/basic_string.h:293
#1 _M_rep (this=<optimized out>) at /usr/include/c++/4.8/bits/basic_string.h:301
#2 size (this=<optimized out>) at /usr/include/c++/4.8/bits/basic_string.h:716
#3 operator==<char> (__rhs=..., __lhs=...) at /usr/include/c++/4.8/bits/basic_string.h:2495
#4 operator!=<char, std::char_traits<char>, std::allocator<char> > (__rhs=..., __lhs="ovs-vm131") at /usr/include/c++/4.8/bits/basic_string.h:2534
#5 OVSDB::LogicalSwitchEntry::Sync (this=0x7ff4400ad400, db_entry=0x7ff4180745b0) at controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/logical_switch_ovsdb.cc:147
#6 0x00000000009b4ac1 in KSyncDBObject::Notify (this=0x7ff48c002e70, partition=<optimized out>, e=0x7ff4180745b0) at controller/src/ksync/ksync_object.cc:335
#7 0x0000000000c23222 in operator() (a1=0x7ff4180745b0, a0=0x7ff494026d90, this=0x7ff46fffea00) at /usr/include/boost/function/function_template.hpp:767
#8 RunNotify (entry=0x7ff4180745b0, tpart=0x7ff494026d90, this=0x7ff494026d40) at controller/src/db/db_table.cc:88
#9 DBTableBase::RunNotify (this=<optimized out>, tpart=tpart@entry=0x7ff494026d90, entry=entry@entry=0x7ff4180745b0) at controller/src/db/db_table.cc:136
#10 0x0000000000c250d8 in DBTablePartBase::RunNotify (this=this@entry=0x7ff494026d90) at controller/src/db/db_table_partition.cc:45
#11 0x0000000000c21edd in DBPartition::QueueRunner::Run (this=0x7ff480002710) at controller/src/db/db_partition.cc:178
#12 0x0000000000d0f990 in TaskImpl::execute (this=0x7ff49958a840) at controller/src/base/task.cc:232
#13 0x00007ff4a100eb3a in ?? () from /usr/lib/libtbb.so.2
#14 0x00007ff4a100a816 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007ff4a1009f4b in ?? () from /usr/lib/libtbb.so.2
#16 0x00007ff4a10060ff in ?? () from /usr/lib/libtbb.so.2
#17 0x00007ff4a10062f9 in ?? () from /usr/lib/libtbb.so.2
#18 0x00007ff4a122a182 in start_thread (arg=0x7ff46ffff700) at pthread_create.c:312
#19 0x00007ff49fecafbd in __signbitl (__x=0) at ../sysdeps/x86/fpu/bits/mathinline.h:154
#20 __qfcvt_r (value=0, ndigit=1879045888, decpt=0x7ff46ffff9c0, sign=0x0, buf=0x0, len=0) at efgcvt_r.c:93
#21 0x0000000000000000 in ?? ()
(gdb)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8804
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8804
Committed: http://github.org/Juniper/contrail-controller/commit/f78d07a2e134bee377ddea041018582bf130494b
Submitter: Zuul
Branch: master

commit f78d07a2e134bee377ddea041018582bf130494b
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 1 16:21:04 2015 +0530

Fix TOR Agent Crash for missing dev info

Issue:
------
During deletion of config, due to difference in the order
of notification events, we have a state where physical
device is deleted but physical device vn entry is still
active, during this we release the refernce of physical
device and fails to process this entry in KSync Notify
resulting in crash.

Fix:
----
Add DB filter to trigger delete for Logical switch entry
if device pointer is missing in DB entry, since we
should anyway receive delete for this entry eventually.

Added test-case for the same.
Closes-Bug: 1438006

Change-Id: I9392da2bf5348850621199e57d5fc1aa5f8bf075

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8848
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8848
Committed: http://github.org/Juniper/contrail-controller/commit/59456b8b586ae231a0489ac6c739a393088adaec
Submitter: Zuul
Branch: R2.1

commit 59456b8b586ae231a0489ac6c739a393088adaec
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 1 16:21:04 2015 +0530

Fix TOR Agent Crash for missing dev info

Issue:
------
During deletion of config, due to difference in the order
of notification events, we have a state where physical
device is deleted but physical device vn entry is still
active, during this we release the refernce of physical
device and fails to process this entry in KSync Notify
resulting in crash.

Fix:
----
Add DB filter to trigger delete for Logical switch entry
if device pointer is missing in DB entry, since we
should anyway receive delete for this entry eventually.

Closes-Bug: 1438006
(cherry picked from commit f78d07a2e134bee377ddea041018582bf130494b)

Conflicts:
 src/vnsw/agent/ovs_tor_agent/ovsdb_client/test/test_ovs_logical_switch.cc

Change-Id: I77b8bf835da639dfcb54fe52a4ee6bd1a08848d2