contrail heat plugin reads heat.conf over and over

Bug #1779078 reported by Thomas Morin on 2018-06-28
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Fix Committed
Undecided
Shivayogi Ugaji
R3.2.3.x
Fix Committed
High
Nikhil Bansal
R4.1
Fix Committed
High
Shivayogi Ugaji
R5.0
Fix Committed
High
Shivayogi Ugaji
Trunk
Fix Committed
High
Shivayogi Ugaji
OpenContrail
New
Undecided
Unassigned

Bug Description

(Contrail version: 3.2.3.x, but issue still present in master)

While testing a Heat stack with a significant number of resources, we noticed a high CPU usage of Heat engine. After doing some profiling, we noticed a hot spot in contrail heat plugin.

The pyflame flamegraph attachment shows the hotspot.
We see a heat engine process spending 84% of its time in [1].

Adding minimal logging at this place, I confirmed that this parsing of heat.conf happens each time that is instantiated one of the classes representing a Contrail Heat resource. For a given resource, this happens at stack validation time, then again at stack creation (at least once).

[1] https://github.com/Juniper/contrail-heat/blob/R3.2.3.x/contrail_heat/resources/contrail.py#L37

Thomas Morin (tmmorin-orange) wrote :
Thomas Morin (tmmorin-orange) wrote :

Of course, parsing heat.conf again and again isn't very useful, the established operational practice being to restart heat engine whenever its config is changed.

Thomas Morin (tmmorin-orange) wrote :

The attached patch moves the heat.conf parsing to module scope to avoid parsing multiple times.

On large stacks, the order of magnitude of the timing improvement for the time to check a stack is several minutes.

Review in progress for https://review.opencontrail.org/44195
Submitter: Thomas Morin (<email address hidden>)

Reviewed: https://review.opencontrail.org/44195
Committed: http://github.com/Juniper/contrail-heat/commit/df5389e5bda19a0ff5cbebc82c0b892394331ef8
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit df5389e5bda19a0ff5cbebc82c0b892394331ef8
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078

Review in progress for https://review.opencontrail.org/44421
Submitter: Thomas Morin (<email address hidden>)

Review in progress for https://review.opencontrail.org/44422
Submitter: Thomas Morin (<email address hidden>)

Review in progress for https://review.opencontrail.org/44423
Submitter: Thomas Morin (<email address hidden>)

Jeba Paulaiyan (jebap) on 2018-07-10
tags: added: config

Reviewed: https://review.opencontrail.org/44421
Committed: http://github.com/Juniper/contrail-heat/commit/ad78a93b484942bc156894e1fa1db246e95dde83
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit ad78a93b484942bc156894e1fa1db246e95dde83
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

Reviewed: https://review.opencontrail.org/44422
Committed: http://github.com/Juniper/contrail-heat/commit/05712358dab8d987b238e8588a96224b852090d4
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 05712358dab8d987b238e8588a96224b852090d4
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

Review in progress for https://review.opencontrail.org/46343
Submitter: Nikhil Bansal (<email address hidden>)

Reviewed: https://review.opencontrail.org/44423
Committed: http://github.com/Juniper/contrail-heat/commit/4f15957e986a280b831c414e74fe62e8580644e6
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 4f15957e986a280b831c414e74fe62e8580644e6
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

Reviewed: https://review.opencontrail.org/46343
Committed: http://github.com/Juniper/contrail-heat/commit/36ea9d6da9d44740cfb2cbe421722ead9da6b3ad
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 36ea9d6da9d44740cfb2cbe421722ead9da6b3ad
Author: Thomas Morin <email address hidden>
Date: Wed May 30 17:46:07 2018 +0200

avoid parsing heat.conf for each Heat resource

This moves heat configuration parsing to module loading time.

Parsing the config for each resource does not seem to be
useful, and profiling with pyflame shows that it is a significant
hit, measurable in tens of seconds on heat stack with a large
number of Contrail heat resources.

Change-Id: Iad81d60e63fdfb554e6c42dbcdf58848586984f1
Closes-Bug: 1779078
(cherry picked from commit df5389e5bda19a0ff5cbebc82c0b892394331ef8)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers