need to improve api-server and schema initialization times in scale setup

[Vedamurthy Joshi]

R2.20 Build 54 Ubuntu 14.04 Juno multi-node setup

This is a tracking bug to improve api-server and schema initialization duration in scale setup.

Config :

128 ToRs (openvswitch vms)
32 TAs on each node - 4 such nodes in active-standby mode
506 VNs associated with ToRs.
Each ToR has 506 physical interfaces- 506 logical interfaces and 1012 VMIs (i.e. 2 vmis per lif)
Endpoints on each lif are simulated as netns namespaces (22 namespaces * 23 interfaces each)

There are 64K VNs spread across 8 projects as well (8K vns per project)

Currently, api-server takes ~40 mins to reach a normal state after all activity(in terms of CPU utilization)

Schema takes ~2.5 to 3 hrs to complete all initialization activity

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19287
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19287
Committed: http://github.org/Juniper/contrail-controller/commit/78dc47b3fdba349fc35ea6a6c8786dd71938749f
Submitter: Zuul
Branch: master

commit 78dc47b3fdba349fc35ea6a6c8786dd71938749f
Author: Édouard Thuleau <email address hidden>
Date: Wed Apr 6 15:34:18 2016 +0200

Script to generate config database load

For the moment, the script supports few Contrail resources:
- project
- security-group
- virtual-network
- virtual-machine-interface

Partial-bug: #1466777
Change-Id: I5937fc2d94c999c21ec240200cbf483eaa33c7bf

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20525
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20527
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20529
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20525
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20527
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/c731a4a581999bc315c16b2bff5e04c516949ead
Submitter: Zuul
Branch: master

commit c731a4a581999bc315c16b2bff5e04c516949ead
Author: Édouard Thuleau <email address hidden>
Date: Mon May 23 17:48:32 2016 +0200

Improve VMI list

As the subnet ID is an attribute of an instance-ip now, we can
simplifying the way to retrieve the subnet ID in the plugin.

Change-Id: Id93fbbfe2b7787171d1e7fd28fab0b838ecc8479
Closes-bug: #1584844
Related-Bug: #1466777

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20529
Committed: http://github.org/Juniper/contrail-controller/commit/7eea85f420c17fe6d15a674241facde277d094b3
Submitter: Zuul
Branch: master

commit 7eea85f420c17fe6d15a674241facde277d094b3
Author: Édouard Thuleau <email address hidden>
Date: Mon May 23 18:25:19 2016 +0200

[VNC API] Upgrade instance-ip subnet id attribute

Maintain instance-ip attribute when the vnc api server restarts.

Change-Id: Ic110de77fa1c030c33aea4596e1fc7b6a5d6a0c9
Closes-Bug: #1584844
Related-Bug: #1466777

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20525
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20525
Committed: http://github.org/Juniper/contrail-controller/commit/6835c37406d66773bc567aba4ecb550b6eea6fe0
Submitter: Zuul
Branch: master

commit 6835c37406d66773bc567aba4ecb550b6eea6fe0
Author: Édouard Thuleau <email address hidden>
Date: Fri May 20 12:15:28 2016 +0200

[VNC API server] Improving list resources

When a resource type is listed, a multi-read on cassandra is done
on each resource. That patch optimizes that multi-read to fetch only
specified resources properties when it's feasible.

Change-Id: If3bf44eeb0330a3d8d48f34531645db6537a0d8b
Closes-Bug: #1584000
Partial-Bug: #1466777

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21049
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21049
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21273
Submitter: ?douard Thuleau (<email address hidden>)