agent crash @ DBPartition::QueueRunner::Run

[Senthilnathan Murugappan]

Using 3.0-2709-kilo on ubuntu-14.04
Observed multiple agent crashes on the scale setup overtime

(gdb) bt
#0 0x0000000000000060 in ?? ()
#1 0x000000000106beb6 in DBPartition::QueueRunner::Run (this=0x7fd36c589740) at controller/src/db/db_partition.cc:187
#2 0x00000000011790ac in TaskImpl::execute (this=0x7fd38944fd40) at controller/src/base/task.cc:253
#3 0x00007fd390804b3a in ?? () from /usr/lib/libtbb.so.2
#4 0x00007fd390800816 in ?? () from /usr/lib/libtbb.so.2
#5 0x00007fd3907fff4b in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fd3907fc0ff in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fd3907fc2f9 in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fd390a20182 in start_thread (arg=0x7fd3733fc700) at pthread_create.c:312
#9 0x00007fd38fcf947d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Hope the trigger is VM delete or so.

The core files will be copied to /cs-shared/bugs/<bug-id>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17122
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17138
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17139
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17140
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17122
Committed: http://github.org/Juniper/contrail-controller/commit/fdadc742b61ee817b2189f2f76ced90cee3824a7
Submitter: Zuul
Branch: master

commit fdadc742b61ee817b2189f2f76ced90cee3824a7
Author: Hari <email address hidden>
Date: Thu Feb 11 10:21:12 2016 +0530

Enqueue route updates in fabric route tables.

There are still a couple of places where enqueue is happening in
respective VRF route tables. As these may be deleted before the
enqueued request is handled, always enqueue in fabric route table,
which checks for the actual route table and invokes relevant
operation.

Change-Id: I5baf302fbad6524020738610a7a012a6ee7438e0
closes-bug: 1544380

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17139
Committed: http://github.org/Juniper/contrail-controller/commit/e4899ad71b11574112fbbed7dba4b4e8d94bd61d
Submitter: Zuul
Branch: R2.21.x

commit e4899ad71b11574112fbbed7dba4b4e8d94bd61d
Author: Hari <email address hidden>
Date: Thu Feb 11 21:23:33 2016 +0530

Enqueue route updates in fabric route tables.

There are still a couple of places where enqueue is happening in
respective VRF route tables. As these may be deleted before the
enqueued request is handled, always enqueue in fabric route table,
which checks for the actual route table and invokes relevant
operation.

Change-Id: I66114f6272b6760620fc55ad429fcec4ec2f3042
closes-bug: 1544380

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17138
Committed: http://github.org/Juniper/contrail-controller/commit/6ecfd91bc22d0ace741e20dd65154e9a7fcce3c5
Submitter: Zuul
Branch: R2.20

commit 6ecfd91bc22d0ace741e20dd65154e9a7fcce3c5
Author: Hari <email address hidden>
Date: Thu Feb 11 21:18:52 2016 +0530

Enqueue route updates in fabric route tables.

There are still a couple of places where enqueue is happening in
respective VRF route tables. As these may be deleted before the
enqueued request is handled, always enqueue in fabric route table,
which checks for the actual route table and invokes relevant
operation.

Change-Id: I030291f68c1e000bcd9c8acc98285805ce08c791
closes-bug: 1544380

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17140
Committed: http://github.org/Juniper/contrail-controller/commit/88b7b3df1c9e0688aa7d9afe150bddd66fc715e9
Submitter: Zuul
Branch: R2.22.x

commit 88b7b3df1c9e0688aa7d9afe150bddd66fc715e9
Author: Hari <email address hidden>
Date: Thu Feb 11 21:45:47 2016 +0530

Enqueue route updates in fabric route tables.

There are still a couple of places where enqueue is happening in
respective VRF route tables. As these may be deleted before the
enqueued request is handled, always enqueue in fabric route table,
which checks for the actual route table and invokes relevant
operation.

Change-Id: I5dc3393e4c0ab85b83d5a09f76d65d5e11a535de
closes-bug: 1544380