tor-agent crash at logical_switch_name_ == UuidToString

[Vedamurthy Joshi]

R2.22 Build 115 Ubuntu 14.04 Juno

Was running a test by continuously creating and deleting vms, vns, lifs, vmis and restarting 3 control nodes randomly
Below crash was seen on tor-agent nodes a few times.

Core will be in http://10.204.216.50/Docs/bugs/#

(gdb) bt
#0 0x00007f6a9b96acc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f6a9b96e0d8 in __GI_abort () at abort.c:89
#2 0x00007f6a9b963b86 in __assert_fail_base (fmt=0x7f6a9bab4830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0xe85d10 "logical_switch_name_ == UuidToString(vrf->vn()->GetUuid())",
    file=file@entry=0xe85cc8 "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/vrf_ovsdb.cc", line=line@entry=72,
    function=function@entry=0xe85de0 "virtual bool OVSDB::VrfOvsdbEntry::Sync(DBEntry*)") at assert.c:92
#3 0x00007f6a9b963c32 in __GI___assert_fail (assertion=0xe85d10 "logical_switch_name_ == UuidToString(vrf->vn()->GetUuid())",
    file=0xe85cc8 "controller/src/vnsw/agent/ovs_tor_agent/ovsdb_client/vrf_ovsdb.cc", line=72, function=0xe85de0 "virtual bool OVSDB::VrfOvsdbEntry::Sync(DBEntry*)") at assert.c:101
#4 0x00000000009695f0 in OVSDB::VrfOvsdbEntry::Sync(DBEntry*) ()
#5 0x0000000000a9d847 in KSyncDBObject::Notify(DBTablePartBase*, DBEntryBase*) ()
#6 0x0000000000d3442a in DBTableBase::RunNotify(DBTablePartBase*, DBEntryBase*) ()
#7 0x0000000000d36f38 in DBTablePartBase::RunNotify() ()
#8 0x0000000000d32d1d in DBPartition::QueueRunner::Run() ()
#9 0x0000000000e34140 in TaskImpl::execute() ()
#10 0x00007f6a9c539b3a in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f6a9c535816 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f6a9c534f4b in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f6a9c5310ff in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f6a9c5312f9 in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f6a9c755182 in start_thread (arg=0x7f6a8effb700) at pthread_create.c:312
#16 0x00007f6a9ba2e47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) quit
----------------

Scripts :

root@nodec1:~# cat test1.sh
source /etc/contrail/openstackrc

image_id="e9693c2b-f6ba-4830-a856-3e5b0f31db22"

while :
do

    for i in {1..30};
    do
    neutron net-create bugvn$i
    neutron subnet-create bugvn$i 100.$i.$i.0/24
    vn_id=`neutron net-show bugvn$i | grep " id " | awk '{ print $4}'`
    nova boot --nic net-id=$vn_id --flavor 1 --image $image_id vm_$i
    done
    for i in {1..30};
    do
    nova delete vm_$i
 done
    for i in {1..30};
    do
        neutron net-delete bugvn$i
    done

done
root@nodec1:~#
root@nodec1:~#
root@nodec1:~# cat bug-recreate.sh
device_id="fbaa5ca1-981c-4bfb-9d14-f44f2fff90d0"
mac="00:25:90:c3:09:6d"

while :
do
   neutron net-create bug-vn
   neutron subnet-create bug-vn 100.1.1.0/24
   vn_id=`neutron net-show bug-vn | grep " id " | awk '{ print $4}'`
   python config-tor-intf.py "ge-0/0/0" $device_id "ge-0/0/0.0" 0 1 $vn_id "00:25:90:c3:09:6d"
   sleep 20
   python del-lifs-vmis.py
   neutron net-delete bug-vn
   sleep 20
done

root@nodec1:~#

[Prabhjot Singh Sethi (prabhjot)]

issue happens due to vn object getting recreated with new UUID, cause vrf to just update to new VN object.
this on notification in OVS KSYNC observes a key change, which is not a handled scenario results in assertion

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19018
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19018
Committed: http://github.org/Juniper/contrail-controller/commit/2b3b154112c0362a8da5d0256a0fff9ae003b676
Submitter: Zuul
Branch: master

commit 2b3b154112c0362a8da5d0256a0fff9ae003b676
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Apr 2 10:04:10 2016 +0530

Handle recreate of VN entry with different uuid

Issue:
------
while creating and deleting VN entry in quick succession
agent some times observes a state compression where while
old VN entry was getting deleted, new VN entry with same
name but different uuid. this new VN entry ends up using
the existing vrf instead of deleting and creating vrf
again, since OVSDB Ksync operates using VN uuid as the
logical switch name it observes this as a key change
and results in assertion due to unhandled state

Fix:
----
Add handling for key change using DBFilters, where ksync
infra deletes the previous entry and allocates new one
with current key

Closes-Bug: 1527447
Change-Id: I978bfc3a19aa2d9a294336bf976e89769d0999d5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19208
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19208
Committed: http://github.org/Juniper/contrail-controller/commit/d351866ce9b47f7ca2caa0a1a0e1ac2f0604f885
Submitter: Zuul
Branch: R3.0

commit d351866ce9b47f7ca2caa0a1a0e1ac2f0604f885
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Apr 2 10:04:10 2016 +0530

Handle recreate of VN entry with different uuid

Issue:
------
while creating and deleting VN entry in quick succession
agent some times observes a state compression where while
old VN entry was getting deleted, new VN entry with same
name but different uuid. this new VN entry ends up using
the existing vrf instead of deleting and creating vrf
again, since OVSDB Ksync operates using VN uuid as the
logical switch name it observes this as a key change
and results in assertion due to unhandled state

Fix:
----
Add handling for key change using DBFilters, where ksync
infra deletes the previous entry and allocates new one
with current key

Closes-Bug: 1527447
Change-Id: I978bfc3a19aa2d9a294336bf976e89769d0999d5
(cherry picked from commit 2b3b154112c0362a8da5d0256a0fff9ae003b676)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19485
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19486
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19487
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19485
Committed: http://github.org/Juniper/contrail-controller/commit/8fca84e45eb2794bba2cc4b79bf7a1d904139ce0
Submitter: Zuul
Branch: R2.20

commit 8fca84e45eb2794bba2cc4b79bf7a1d904139ce0
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Apr 2 10:04:10 2016 +0530

Handle recreate of VN entry with different uuid

Issue:
------
while creating and deleting VN entry in quick succession
agent some times observes a state compression where while
old VN entry was getting deleted, new VN entry with same
name but different uuid. this new VN entry ends up using
the existing vrf instead of deleting and creating vrf
again, since OVSDB Ksync operates using VN uuid as the
logical switch name it observes this as a key change
and results in assertion due to unhandled state

Fix:
----
Add handling for key change using DBFilters, where ksync
infra deletes the previous entry and allocates new one
with current key

Closes-Bug: 1527447
Change-Id: I978bfc3a19aa2d9a294336bf976e89769d0999d5
(cherry picked from commit 2b3b154112c0362a8da5d0256a0fff9ae003b676)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19486
Committed: http://github.org/Juniper/contrail-controller/commit/c1a4b7d51e7594364844b121002ff344ecae1df7
Submitter: Zuul
Branch: R2.21.x

commit c1a4b7d51e7594364844b121002ff344ecae1df7
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Apr 2 10:04:10 2016 +0530

Handle recreate of VN entry with different uuid

Issue:
------
while creating and deleting VN entry in quick succession
agent some times observes a state compression where while
old VN entry was getting deleted, new VN entry with same
name but different uuid. this new VN entry ends up using
the existing vrf instead of deleting and creating vrf
again, since OVSDB Ksync operates using VN uuid as the
logical switch name it observes this as a key change
and results in assertion due to unhandled state

Fix:
----
Add handling for key change using DBFilters, where ksync
infra deletes the previous entry and allocates new one
with current key

Closes-Bug: 1527447
Change-Id: I978bfc3a19aa2d9a294336bf976e89769d0999d5
(cherry picked from commit 2b3b154112c0362a8da5d0256a0fff9ae003b676)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19487
Committed: http://github.org/Juniper/contrail-controller/commit/742adbdbdf2ead6571bd5534fe90d1375830dfc6
Submitter: Zuul
Branch: R2.22.x

commit 742adbdbdf2ead6571bd5534fe90d1375830dfc6
Author: Prabhjot Singh Sethi <email address hidden>
Date: Sat Apr 2 10:04:10 2016 +0530

Handle recreate of VN entry with different uuid

Issue:
------
while creating and deleting VN entry in quick succession
agent some times observes a state compression where while
old VN entry was getting deleted, new VN entry with same
name but different uuid. this new VN entry ends up using
the existing vrf instead of deleting and creating vrf
again, since OVSDB Ksync operates using VN uuid as the
logical switch name it observes this as a key change
and results in assertion due to unhandled state

Fix:
----
Add handling for key change using DBFilters, where ksync
infra deletes the previous entry and allocates new one
with current key

Closes-Bug: 1527447
Change-Id: I978bfc3a19aa2d9a294336bf976e89769d0999d5
(cherry picked from commit 2b3b154112c0362a8da5d0256a0fff9ae003b676)