after supervisor-config restart on tor-scale setup, api-server was fully up only after ~6 hrs

[Vedamurthy Joshi]

R2.20 Build 30 Ubuntu 14.04 Juno multi-node setup

Supervisor-config was restarted on all 3 nodes on this tor-scale setup with 128K vmis and 64K lifs over 500 vns
API-Server started listening on 9100 only after 6 hrs.
The first 15 mins seemed to be mostly read from cassandra.
Rest looks mostly due to delays in ifmap-server. This is with 20M as the max-per-poll-response size in ifmap.properties

env.roledefs = {
    'all': [host2, host3, host4, host5, host6, host7, host8, host9],
    'cfgm': [host2, host3, host4],
    'openstack': [host2, host3, host4],
    'webui': [host3],
    'control': [host2, host3, host4],
    'compute': [host5, host6, host7, host8, host9],
    'collector': [host2, host3, host4],
    'database': [host2, host3, host4],
    'toragent': [host5, host6, host7, host9 ],
    'tsn': [host5, host6, host7,host9 ],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei34', 'nodei35', 'nodei36', 'nodei37', 'nodei38', 'nodei28', 'nodei27', 'nodei30']
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11158
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11158
Committed: http://github.org/Juniper/contrail-controller/commit/5cc9df6968d6ae5d25eb1ea6fc0eefe888e4d943
Submitter: Zuul
Branch: master

commit 5cc9df6968d6ae5d25eb1ea6fc0eefe888e4d943
Author: Sachin Bansal <email address hidden>
Date: Mon Jun 1 17:44:38 2015 -0700

Schema transformer scaling improvements

With the help of profiler, Ajay and I found that most of the time is being spent
in walking all the VMIs for each network operations to find the VMIs on that
network and VMIs which correspond to a service instance. We added indexing on
VMIs for these two items and after that it finished significantly faster (~30
minutes).

Change-Id: I1d1e2110f12016c66b9db83ebd95c45d1a5e6e2d
Closes-Bug: 1460335

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11194
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11194
Committed: http://github.org/Juniper/contrail-controller/commit/dd0c909b54568d4547dab7c2e8282fe18428fc67
Submitter: Zuul
Branch: R2.20

commit dd0c909b54568d4547dab7c2e8282fe18428fc67
Author: Sachin Bansal <email address hidden>
Date: Mon Jun 1 17:44:38 2015 -0700

Schema transformer scaling improvements

With the help of profiler, Ajay and I found that most of the time is being spent
in walking all the VMIs for each network operations to find the VMIs on that
network and VMIs which correspond to a service instance. We added indexing on
VMIs for these two items and after that it finished significantly faster (~30
minutes).

Change-Id: I1d1e2110f12016c66b9db83ebd95c45d1a5e6e2d
Closes-Bug: 1460335
(cherry picked from commit 5cc9df6968d6ae5d25eb1ea6fc0eefe888e4d943)

[Vedamurthy Joshi (vedujoshi)]

On 2.20 Build 79,
API Server now stabilizes in ~23 mins.
ifmap-server completes all its activity in ~30 mins