agent asserts @ vn->GetName()) == 0

[Senthilnathan Murugappan]

We have 40K VMIs spread over 100 agents and each VMI has two IPs, each from different VNs.
In total, 80K VNs, 40K VMIs across 100 agents.

At this scale when the control process was restarted (both the xmpp peers are brought down) we observed agent cores on many of the agent and after a while things settles down and the cluster becomes stable.

Below is the bt for the same and the core will be copied to /auto/cores/<bugid>

2016-06-27 21:38:32:453061: [root@10.84.25.20] out: [Thread debugging using libthread_db enabled]
2016-06-27 21:38:32:754443: [root@10.84.25.20] out: Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
2016-06-27 21:38:32:754579: [root@10.84.25.20] out: Core was generated by `/usr/bin/contrail-vrouter-agent'.
2016-06-27 21:38:33:118962: [root@10.84.25.20] out: Program terminated with signal SIGABRT, Aborted.
2016-06-27 21:38:33:119100: [root@10.84.25.20] out: #0 0x00007f3011a39cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
2016-06-27 21:38:33:150948: [root@10.84.25.20] out: 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
2016-06-27 21:38:33:151086: [root@10.84.25.20] out: #0 0x00007f3011a39cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
2016-06-27 21:38:33:151195: [root@10.84.25.20] out: #1 0x00007f3011a3d0d8 in __GI_abort () at abort.c:89
2016-06-27 21:38:33:151283: [root@10.84.25.20] out: #2 0x00007f3011a32b86 in __assert_fail_base (fmt=0x7f3011b83830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n
%n", assertion=assertion@entry=0x127eeb0 "state->vn_name_.compare(vn->GetName()) == 0", file=file@entry=0x127ee78 "controller/src/vnsw/agent/uve/vn_uve_table
_base.cc", line=line@entry=289, function=function@entry=0x127eee0 <VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)::__PRETTY_FUNCTION__> "void VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)") at assert.c:92
2016-06-27 21:38:33:167045: [root@10.84.25.20] out: #3 0x00007f3011a32c32 in __GI___assert_fail (assertion=0x127eeb0 "state->vn_name_.compare(vn->GetName())
 == 0", file=0x127ee78 "controller/src/vnsw/agent/uve/vn_uve_table_base.cc", line=289, function=0x127eee0 <VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)::__PRETTY_FUNCTION__> "void VnUveTableBase::InterfaceNotify(DBTablePartBase*, DBEntryBase*)") at assert.c:101
2016-06-27 21:38:33:599229: [root@10.84.25.20] out: #4 0x0000000000e43c37 in VnUveTableBase::InterfaceNotify (this=0x7f2ffc003640, partition=0x7f2ffc524c20, e=0x7f2fd4a96928) at controller/src/vnsw/agent/uve/vn_uve_table_base.cc:289
2016-06-27 21:38:33:799814: [root@10.84.25.20] out: #5 0x00000000010a767a in operator() (a1=0x7f2fd4a96928, a0=0x7f2ffc524c20, this=0x7f30094b8aa0) at /usr/include/boost/function/function_template.hpp:767
2016-06-27 21:38:33:799954: [root@10.84.25.20] out: #6 RunNotify (entry=0x7f2fd4a96928, tpart=0x7f2ffc524c20, this=0x7f2ffc51ad40) at controller/src/db/db_table.cc:114
2016-06-27 21:38:33:800063: [root@10.84.25.20] out: #7 DBTableBase::RunNotify (this=<optimized out>, tpart=tpart@entry=0x7f2ffc524c20, entry=entry@entry=0x7f2fd4a96928) at controller/src/db/db_table.cc:204
2016-06-27 21:38:33:864033: [root@10.84.25.20] out: #8 0x00000000010a9d68 in DBTablePartBase::RunNotify (this=this@entry=0x7f2ffc524c20) at controller/src/db/db_table_partition.cc:47
2016-06-27 21:38:33:895896: [root@10.84.25.20] out: #9 0x00000000010a5e70 in DBPartition::QueueRunner::Run (this=0x7f2ff92de490) at controller/src/db/db_partition.cc:209
2016-06-27 21:38:34:060009: [root@10.84.25.20] out: #10 0x00000000011b886f in TaskImpl::execute (this=0x7f300b279f40) at controller/src/base/task.cc:262
2016-06-27 21:38:34:060146: [root@10.84.25.20] out: #11 0x00007f3012608b3a in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:060251: [root@10.84.25.20] out: #12 0x00007f3012604816 in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:060337: [root@10.84.25.20] out: #13 0x00007f3012603f4b in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:061749: [root@10.84.25.20] out: #14 0x00007f30126000ff in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:065267: [root@10.84.25.20] out: #15 0x00007f30126002f9 in ?? () from /usr/lib/libtbb.so.2
2016-06-27 21:38:34:065604: [root@10.84.25.20] out: #16 0x00007f3012824182 in start_thread (arg=0x7f30094b9700) at pthread_create.c:312
2016-06-27 21:38:34:072979: [root@10.84.25.20] out: #17 0x00007f3011afd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
2016-06-27 21:38:34:073131: [root@10.84.25.20] out:

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21749
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21752
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21753
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21749
Committed: http://github.org/Juniper/contrail-controller/commit/79986fdf34e61f0e0fd82c5a19810450f98e985f
Submitter: Zuul
Branch: master

commit 79986fdf34e61f0e0fd82c5a19810450f98e985f
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Change-Id: I7679612fd1d663ffce452a8f9c21bd4aede66d03
Closes-Bug: #1596792

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21752
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21794
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21753
Committed: http://github.org/Juniper/contrail-controller/commit/a6f03e5250c75f21fa2e44000d40b122efffbd14
Submitter: Zuul
Branch: R3.0.2.x

commit a6f03e5250c75f21fa2e44000d40b122efffbd14
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 79986fdf34e61f0e0fd82c5a19810450f98e985f)

Change-Id: I2d8b28d166a53da9c166392eafb1ee8407e8ecc4

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21752
Committed: http://github.org/Juniper/contrail-controller/commit/e3f93bc9e69f9db6f6a258cd4b6e8b3e3d2bfbee
Submitter: Zuul
Branch: R3.0

commit e3f93bc9e69f9db6f6a258cd4b6e8b3e3d2bfbee
Author: ashoksingh <email address hidden>
Date: Fri Jul 8 11:57:56 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 79986fdf34e61f0e0fd82c5a19810450f98e985f)

Change-Id: I56d564b851f930ce6bb61bfd8639d8061b045ef2

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21794
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21794
Committed: http://github.org/Juniper/contrail-controller/commit/f9eca51ca02ac7bbea1e592aa195f265e8fe7937
Submitter: Zuul
Branch: R2.22.x

commit f9eca51ca02ac7bbea1e592aa195f265e8fe7937
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Change-Id: Ia3c11ad71558a6e9e215bbb6817ccd9f65253a5d
Closes-Bug: #1596792

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21796
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21796
Committed: http://github.org/Juniper/contrail-controller/commit/2656b8ca109dc468545a5a74300714affaaed435
Submitter: Zuul
Branch: R2.20

commit 2656b8ca109dc468545a5a74300714affaaed435
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 64ef02488ed11d637a46f5b8c51664838cf2b95c)

Change-Id: I037631d09af36e7d191cc0f45a129a99d3c800ea

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23434
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21795
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21795
Committed: http://github.org/Juniper/contrail-controller/commit/303fed3880d38a987ab11117fba35eb97ae148e4
Submitter: Zuul
Branch: R2.21.x

commit 303fed3880d38a987ab11117fba35eb97ae148e4
Author: ashoksingh <email address hidden>
Date: Mon Jul 11 10:54:12 2016 +0530

Handle VN change in interface Notification

Remove the assumption that VN of an interface cannot change without first getting a
notification for NULL VN.

Also fix the following issue.
Also when VN changes for a VMI (clubbing of VMI getting NULL VRF and later new VRF
events into a single event) we were not removing EVPN and Bridge routes for old VRF.

Also added UT to verify this scenario.

Closes-Bug: #1596792
(cherry picked from commit 64ef02488ed11d637a46f5b8c51664838cf2b95c)

Change-Id: I68cbaaf067523abb62ce8da49e6d2418080b9a1f

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23434
Committed: http://github.org/Juniper/contrail-controller/commit/0f05b970259c948d4081fc020b38571606e6f8f0
Submitter: Zuul
Branch: R2.22.x

commit 0f05b970259c948d4081fc020b38571606e6f8f0
Author: ashoksingh <email address hidden>
Date: Fri Aug 19 09:06:01 2016 +0530

Fix compilation failure in test_vn_uve

Change-Id: I03dacaa89ee70a76e63731be8c354267bfda63cd
Closes-Bug: #1596792