contrail heat plugin reads heat.conf over and over

[Thomas Morin]

(Contrail version: 3.2.3.x, but issue still present in master)

While testing a Heat stack with a significant number of resources, we noticed a high CPU usage of Heat engine. After doing some profiling, we noticed a hot spot in contrail heat plugin.

The pyflame flamegraph attachment shows the hotspot.
We see a heat engine process spending 84% of its time in [1].

Adding minimal logging at this place, I confirmed that this parsing of heat.conf happens each time that is instantiated one of the classes representing a Contrail Heat resource. For a given resource, this happens at stack validation time, then again at stack creation (at least once).

[1] https://github.com/Juniper/contrail-heat/blob/R3.2.3.x/contrail_heat/resources/contrail.py#L37

[Thomas Morin (tmmorin-orange)]

Of course, parsing heat.conf again and again isn't very useful, the established operational practice being to restart heat engine whenever its config is changed.

[Thomas Morin (tmmorin-orange)]

The attached patch moves the heat.conf parsing to module scope to avoid parsing multiple times.

On large stacks, the order of magnitude of the timing improvement for the time to check a stack is several minutes.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44195
Submitter: Thomas Morin (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44195
Committed: http://github.com/Juniper/contrail-heat/commit/df5389e5bda19a0ff5cbebc82c0b892394331ef8
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit df5389e5bda19a0ff5cbebc82c0b892394331ef8
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44421
Submitter: Thomas Morin (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44422
Submitter: Thomas Morin (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44423
Submitter: Thomas Morin (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44421
Committed: http://github.com/Juniper/contrail-heat/commit/ad78a93b484942bc156894e1fa1db246e95dde83
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit ad78a93b484942bc156894e1fa1db246e95dde83
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44422
Committed: http://github.com/Juniper/contrail-heat/commit/05712358dab8d987b238e8588a96224b852090d4
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 05712358dab8d987b238e8588a96224b852090d4
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46343
Submitter: Nikhil Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44423
Committed: http://github.com/Juniper/contrail-heat/commit/4f15957e986a280b831c414e74fe62e8580644e6
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 4f15957e986a280b831c414e74fe62e8580644e6
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/46343
Committed: http://github.com/Juniper/contrail-heat/commit/36ea9d6da9d44740cfb2cbe421722ead9da6b3ad
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 36ea9d6da9d44740cfb2cbe421722ead9da6b3ad
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)