Irond slow down contrail under back pressure

[Édouard Thuleau]

On large deployment with huge database, irond server takes times to serve clients and slow down lot of contrail's processes.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24381
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24381
Committed: http://github.org/Juniper/contrail-controller/commit/a112b1b09bdb23681cca3cde292d24b9f20d1b7a
Submitter: Zuul (<email address hidden>)
Branch: master

commit a112b1b09bdb23681cca3cde292d24b9f20d1b7a
Author: Édouard Thuleau <email address hidden>
Date: Fri Sep 30 12:14:40 2016 +0200

[VNC API Server] Remove irond dependency

That patch remove dependency to the irond IF-MAP server. It implements a
minimalist IF-MAP v2 server run as a part of the contrail API server.
That server is directly populated by the VNC API server (directly access
the python dict that represents the ifmap graph, no HTML serialization).
It keeps a HTML interface for control nodes to poll the graph.

Only request 'newSession', 'subscribe' (root graph only) and
'poll' methods are implemented.

For the moment, the old ifmap client which sets and maintains the ifmap
sessions with irond server is use if no IP and TCP port are set to bind
the self managed ifmap server ('ifmap_listen_ip' and 'ifmap_listen_port').
Anyway, config unit tests were migrated to use that new self managed
ifmap server (some schema-transformer unit tests were skiped due to new
issue appeared. That needs to be investigated and fixed).

=== Results with a db populated with 79440 resources ===
Time elapsed for the vnc api server to sync ifmap db from:
- without patch
#1: 0:04:00.0272121
#2: 0:03:51.0994972
#3: 0:03:49.4030671
- with patch:
#1: 0:00:39.394018
#2: 0:00:38.890950
#3: 0:00:39.109681

Time elapsed for a python script to create a new ifmap session,
subscribe to the root graph and poll it (searchResult):
- without patch
#1: 0:00:14.655667
#2: 0:00:13.452200
#3: 0:00:13.905878
- with patch:
#1: 0:00:04.591735
#2: 0:00:04.592040
#3: 0:00:04.192904

Closes-Bug: #1629268
Change-Id: Ib35b48b20c8d46005bf18e8f9b81064985099ff7

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27180
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27180
Committed: http://github.org/Juniper/contrail-controller/commit/90b9deac9757cd0dc388d248f130e196d4a43d24
Submitter: Zuul (<email address hidden>)
Branch: master

commit 90b9deac9757cd0dc388d248f130e196d4a43d24
Author: Édouard Thuleau <email address hidden>
Date: Tue Dec 13 10:17:04 2016 +0100

[VNC API server] Re-add unit tests with fake ifmap client

Since patch [1] merged, unit tests only run with self hosted IF-MAP
server. As it was decided to not removed the IF-MAP external server
support, that patch re-adds unit tests for that mode.
It runs all unit test classes inherited to 'ApiServerTestCase' in both
modes.

[1] Ib35b48b20c8d46005bf18e8f9b81064985099ff7

Closes-Bug: #1629268
Change-Id: Ibb1f0c3f96ec0a8ba3ede67c6b9f05cdefe16b77

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28703
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28703
Committed: http://github.org/Juniper/contrail-controller/commit/584680b88c88c36da1b343899cef7dec8b6cd1a2
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 584680b88c88c36da1b343899cef7dec8b6cd1a2
Author: Édouard Thuleau <email address hidden>
Date: Fri Sep 30 12:14:40 2016 +0200

[VNC API Server] Remove irond dependency

That patch remove dependency to the irond IF-MAP server. It implements a
minimalist IF-MAP v2 server run as a part of the contrail API server.
That server is directly populated by the VNC API server (directly access
the python dict that represents the ifmap graph, no HTML serialization).
It keeps a HTML interface for control nodes to poll the graph.

Only request 'newSession', 'subscribe' (root graph only) and
'poll' methods are implemented.

For the moment, the old ifmap client which sets and maintains the ifmap
sessions with irond server is use if no IP and TCP port are set to bind
the self managed ifmap server ('ifmap_listen_ip' and 'ifmap_listen_port').
Anyway, config unit tests were migrated to use that new self managed
ifmap server (some schema-transformer unit tests were skiped due to new
issue appeared. That needs to be investigated and fixed).

=== Results with a db populated with 79440 resources ===
Time elapsed for the vnc api server to sync ifmap db from:
- without patch
#1: 0:04:00.0272121
#2: 0:03:51.0994972
#3: 0:03:49.4030671
- with patch:
#1: 0:00:39.394018
#2: 0:00:38.890950
#3: 0:00:39.109681

Time elapsed for a python script to create a new ifmap session,
subscribe to the root graph and poll it (searchResult):
- without patch
#1: 0:00:14.655667
#2: 0:00:13.452200
#3: 0:00:13.905878
- with patch:
#1: 0:00:04.591735
#2: 0:00:04.592040
#3: 0:00:04.192904

Closes-Bug: #1629268
Change-Id: Ib35b48b20c8d46005bf18e8f9b81064985099ff7
(cherry picked from commit a112b1b09bdb23681cca3cde292d24b9f20d1b7a)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28815
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30049
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30049
Committed: http://github.org/Juniper/contrail-controller/commit/3cf1d9a6847ba171e1519702a10ba9a3d96f18bd
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 3cf1d9a6847ba171e1519702a10ba9a3d96f18bd
Author: Édouard Thuleau <email address hidden>
Date: Tue Dec 13 10:17:04 2016 +0100

[VNC API server] Re-add unit tests with fake ifmap client

Since patch [1] merged, unit tests only run with self hosted IF-MAP
server. As it was decided to not removed the IF-MAP external server
support, that patch re-adds unit tests for that mode.
It runs all unit test classes inherited to 'ApiServerTestCase' in both
modes.

[1] Ib35b48b20c8d46005bf18e8f9b81064985099ff7

Closes-Bug: #1629268
Change-Id: Ibb1f0c3f96ec0a8ba3ede67c6b9f05cdefe16b77
(cherry picked from commit 90b9deac9757cd0dc388d248f130e196d4a43d24)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28815
Submitter: ?douard Thuleau (<email address hidden>)