[3.0.2.0-36~liberty ] Agent crash @ :ListenerInfo::Unregister

[chhandak]

Observed the crash while deleting logical interface in parallel from scale config .

Backtrace
-----------
(gdb) bt
#0 0x00007faa5b530cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007faa5b5340d8 in __GI_abort () at abort.c:89
#2 0x00007faa5b529b86 in __assert_fail_base (fmt=0x7faa5b67a830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x12bb93f "state_count_[listener] == 0",
    file=file@entry=0x12bb918 "controller/src/db/db_table.cc", line=line@entry=89,
    function=function@entry=0x12bbd80 "void DBTableBase::ListenerInfo::Unregister(DBTableBase::ListenerId)") at assert.c:92
#3 0x00007faa5b529c32 in __GI___assert_fail (assertion=0x12bb93f "state_count_[listener] == 0",
    file=0x12bb918 "controller/src/db/db_table.cc", line=89,
    function=0x12bbd80 "void DBTableBase::ListenerInfo::Unregister(DBTableBase::ListenerId)") at assert.c:101
#4 0x000000000109ae5f in DBTableBase::Unregister(int) ()
#5 0x0000000000db8f6b in KSyncDBObject::UnregisterDb(DBTableBase*) ()
#6 0x0000000000cee94c in RouteKSyncObject::~RouteKSyncObject() ()
#7 0x0000000000cee9d9 in RouteKSyncObject::~RouteKSyncObject() ()
#8 0x0000000000dbd7df in KSyncObjectManager::Process(KSyncObjectEvent*) ()
#9 0x0000000000dc26cf in QueueTaskRunner<KSyncObjectEvent*, WorkQueue<KSyncObjectEvent*> >::Run() ()
#10 0x00000000011aa4af in TaskImpl::execute() ()
#11 0x00007faa5c0ffb3a in ?? () from /usr/lib/libtbb.so.2
#12 0x00007faa5c0fb816 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007faa5c0faf4b in ?? () from /usr/lib/libtbb.so.2
#14 0x00007faa5c0f70ff in ?? () from /usr/lib/libtbb.so.2
#15 0x00007faa5c0f72f9 in ?? () from /usr/lib/libtbb.so.2
#16 0x00007faa5c31b182 in start_thread (arg=0x7faa4ebfa700) at pthread_create.c:312

root@5b7s7:/var/crashes# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-3 active
contrail-tor-agent-4 active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-vroute.2311.5b7s7.1463464859
#17 0x00007faa5b5f447d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

root@5b7s7:/var/crashes# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 3.0.2.0-36 36
contrail-install-packages 3.0.2.0-36~liberty 36
contrail-lib 3.0.2.0-36 36
contrail-nodemgr 3.0.2.0-36 36
contrail-nova-vif 3.0.2.0-36 36
contrail-setup 3.0.2.0-36 36
contrail-utils 3.0.2.0-36 36
contrail-vrouter-3.13.0-85-generic 3.0.2.0-36 36
contrail-vrouter-agent 3.0.2.0-36 36
contrail-vrouter-common 3.0.2.0-36 36
contrail-vrouter-init 3.0.2.0-36 36
contrail-vrouter-utils 3.0.2.0-36 36
python-contrail 3.0.2.0-36 36
python-contrail-vrouter-api 3.0.2.0-36 36
python-neutronclient 1:3.1.0-0ubuntu1~cloud0.2contrail36
python-nova 2:12.0.1-0ubuntu1~cloud0.1contrail36
python-opencontrail-vrouter-netns 3.0.2.0-36 36

[chhandak (chhandak)]

Core file copied in

chhandak@ubuntu-build04:/auto/cores/1582534$ ls -lrt
total 719372
-rwxrwxrwx 1 chhandak epbg 733745152 May 16 23:13 core.contrail-vroute.2311.5b7s7.1463464859
chhandak@ubuntu-build04:/auto/cores/1582534$ pwd
/auto/cores/1582534

[Prabhjot Singh Sethi (prabhjot)]

two KSync route entries were pending while KSync route object was getting deleted.

this issue seems to have happened due to Route Add/Change notify after the VRF delete notify, which as a result scheduled ksync object unregister followed by adding routes in ksync object before processing object unregister.

this issue can be fixed by using DBFilters in KSyncDBObject by ignoring Add/Change notification on a route entry with deleted VRF entry

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20454
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20454
Committed: http://github.org/Juniper/contrail-controller/commit/a6adb73d42847ebb66e8ac451c5e4fa21fbeb69d
Submitter: Zuul
Branch: R3.0

commit a6adb73d42847ebb66e8ac451c5e4fa21fbeb69d
Author: Naveen N <email address hidden>
Date: Fri May 20 15:17:40 2016 +0530

* Handle route add messages on delete marked route ksync object

If route ksync object is marked for deletion and its empty, then
request to deleted ksync object is enqueued. In the mean time
if any route add messages comes they should be ignored,
handling the same.
Closes-bug:#1582534

Change-Id: I5b102633fdb472f9220dfcb4e50ed35475fbd35c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20617
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20617
Committed: http://github.org/Juniper/contrail-controller/commit/e2919f7c3ace413dd9afc11c7982a9ead8d58b65
Submitter: Zuul
Branch: master

commit e2919f7c3ace413dd9afc11c7982a9ead8d58b65
Author: Naveen N <email address hidden>
Date: Fri May 20 15:17:40 2016 +0530

* Handle route add messages on delete marked route ksync object

If route ksync object is marked for deletion and its empty, then
request to deleted ksync object is enqueued. In the mean time
if any route add messages comes they should be ignored,
handling the same.
Closes-bug:#1582534

Change-Id: I5b102633fdb472f9220dfcb4e50ed35475fbd35c