[Ubuntu 14.04 Icehouse Build 39] TOR Scale: Tor Agent crash while created 500 logical interface with host

[chhandak]

Scenario
----------------
Created 500 logical interface with different VLAN tag. All this Interface associated to single physical interface.
Simulated host behind each logical interface with IXIA.
All the host is sending DHCP request simultaneously.

BT
-----
(gdb) bt
#0 0x00007f4bee1e1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f4bee1e50d8 in __GI_abort () at abort.c:89
#2 0x00007f4bee1dab86 in __assert_fail_base (
    fmt=0x7f4bee32b830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xd29a15 "0",
    file=file@entry=0xd36b18 "controller/src/vnsw/agent/oper/vrf.cc", line=line@entry=314,
    function=function@entry=0xd36d00 "bool VrfEntry::DeleteTimeout()") at assert.c:92
#3 0x00007f4bee1dac32 in __GI___assert_fail (assertion=0xd29a15 "0",
    file=0xd36b18 "controller/src/vnsw/agent/oper/vrf.cc", line=314,
    function=0xd36d00 "bool VrfEntry::DeleteTimeout()") at assert.c:101
#4 0x000000000088487d in VrfEntry::DeleteTimeout() ()
#5 0x0000000000d03d89 in Timer::TimerTask::Run() ()
#6 0x0000000000cfb5d0 in TaskImpl::execute() ()
#7 0x00007f4bef3e8b3a in ?? () from /usr/lib/libtbb.so.2
#8 0x00007f4bef3e4816 in ?? () from /usr/lib/libtbb.so.2
#9 0x00007f4bef3e3f4b in ?? () from /usr/lib/libtbb.so.2
#10 0x00007f4bef3e00ff in ?? () from /usr/lib/libtbb.so.2
#11 0x00007f4bef3e02f9 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f4bef604182 in start_thread (arg=0x7f4be6c64700) at pthread_create.c:312
#13 0x00007f4bee2a547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Setup
---------
env.roledefs = {

    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1, host2],
    'openstack':[host2],
    'control':[host2, host3],
    'compute': [host4,host5, host6, host7],
    'tsn': [host4],
    'toragent': [host4],
    'collector': [host1],
    'webui': [host1],
    'database': [host1, host2, host3],
    'build': [host_build],

}

env.tor_agent = {host4:
                     [{
                      'tor_ip':'10.204.217.39',
                      'tor_id':'1',
                      'tor_type':'ovs',
                      'tor_ovs_port':'9999',
                      'tor_ovs_protocol':'tcp',
                      'tor_tsn_ip':'10.204.216.9',
                      'tor_tsn_name':'nodea13',
                      'tor_name':'bng-contrail-qfx51-2',
                      'tor_tunnel_ip':'34.34.34.34',
                      'tor_http_server_port': '7654',
                      'tor_vendor_name':'Juniper'
                      },
                      {
                      'tor_ip':'10.204.216.196',
                      'tor_id':'2',
                      'tor_type':'ovs',
                      'tor_ovs_port':'6632',
                      'tor_ovs_protocol':'tcp',
                      'tor_tsn_ip':'10.204.216.9',
                      'tor_tsn_name':'nodea13',
                      'tor_name':'br10',
                      'tor_tunnel_ip':'10.204.216.196',
                      'tor_http_server_port': '7655',
                      'tor_vendor_name':'openvswitch'
                      }]
                }

env.hostnames ={
    'all': ['nodea5', 'nodea10', 'nodea12', 'nodea12', 'nodea15', 'nodea17', 'nodea28']
}

[chhandak (chhandak)]

Logs and crash file copied at http://mayamruga.englab.juniper.net/bugs/<bug_id>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8241
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8241
Committed: http://github.org/Juniper/contrail-controller/commit/4ac186d02d842d4652291aee773c2473c54ed5f8
Submitter: Zuul
Branch: master

commit 4ac186d02d842d4652291aee773c2473c54ed5f8
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 11 12:48:12 2015 +0530

Fix for Vrf Delete Timeout

Issue:
------
immediate Add/Delete results in Unicast Remote Mac Entry
to go into DEL_DEFER_SYNC state where its waiting for
Add Ack to send a next delete message, however along
with the Ack we receive ovs_idl_row, while trying to set
this row we made the entry active by triggering event
ADD_CHANGE_REQ due which KSYNC entry was never deleted.

Fix:
----
for a deleted entry we should trigger Add/Delete similar
to as if the entry was not found.

Closes-Bug: 1428254
Change-Id: I7c7337f5207610c01f95f015f629531304fdac4c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/8321
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/8321
Committed: http://github.org/Juniper/contrail-controller/commit/ff880d2d22199d47606c3b6ec315b1fa13f24ba0
Submitter: Zuul
Branch: R2.1

commit ff880d2d22199d47606c3b6ec315b1fa13f24ba0
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 11 12:48:12 2015 +0530

Fix for Vrf Delete Timeout

Issue:
------
immediate Add/Delete results in Unicast Remote Mac Entry
to go into DEL_DEFER_SYNC state where its waiting for
Add Ack to send a next delete message, however along
with the Ack we receive ovs_idl_row, while trying to set
this row we made the entry active by triggering event
ADD_CHANGE_REQ due which KSYNC entry was never deleted.

Fix:
----
for a deleted entry we should trigger Add/Delete similar
to as if the entry was not found.

Closes-Bug: 1428254
(cherry picked from commit 4ac186d02d842d4652291aee773c2473c54ed5f8)

Change-Id: I50bb571d04a5f0ad6f068193c86aabb25ae2ad45