Showing stack outputs is slow in Pike
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Fix Released
|
High
|
Zane Bitter | ||
Pike |
Fix Committed
|
High
|
Zane Bitter | ||
Queens |
Fix Released
|
High
|
Zane Bitter |
Bug Description
There appears to have been a performance regression for showing stack outputs in Pike. This is causing timeouts in TripleO:
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master) | #1 |
Changed in heat: | |
assignee: | nobody → Zane Bitter (zaneb) |
status: | Triaged → In Progress |
Zane Bitter (zaneb) wrote : | #2 |
One issue, addressed by the patch above, is that we're loading more attribute values than we need to in the legacy path. "show stack" completes in ~half the time with that patch.
However, that doesn't seem to be the whole issue. Even when doing "show output" (so we're only getting a single output value, which ought to be very quick depending on the value chosen), it's still taking a considerable amount of time (~1 minute in TripleO) just to load all of the attribute values.
OpenStack Infra (hudson-openstack) wrote : | #3 |
Fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master) | #4 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #5 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master) | #6 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 49d833f9aca3511
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 10:53:38 2017 -0400
Speed up show-stack with outputs in legacy path
When we show a stack including the outputs, we calculate all of the
resource attributes that are referenced anywhere in the stack. In
convergence, these are either already cached (and therefore fast) or need
to be cached (and therefore the initial slowness will pay off in future).
This isn't the case in the legacy path though, since we are not doing
caching of attributes in the database in that path. So this is
unnecessarily calculating all of the referenced attribute values, which are
potentially very slow to get.
For legacy stacks, only calculate the attribute values needed to show the
outputs.
Change-Id: I35800c7f87b58d
Partial-Bug: #1719333
Changed in heat: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : | #7 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit f1961c734e81f1e
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 13:55:41 2017 -0400
Use show_output in TemplateResourc
TemplateRes
defined as an output in the nested template. This was being fetched by the
StackResour
*all* of the outputs of the nested stack and caching them in memory in case
something else were to reference any of them.
Unfortunately when calculating the resource data for a stack (which we
must always do when e.g. showing the outputs), we always include the
reference IDs of all resources, regardless of whether they are referenced
by get_resource in the data we are looking to populate. (In fact, we have
no way from the Template API to distinguish where get_resource is used on
a particular resource, only where there are dependencies on it.) This is no
problem under the assumption that getting the reference ID is quick, but
that assumption does not hold for TemplateResource.
The show_output RPC call only retrieves a single output (as opposed to
show_stack, used in StackResource.
them). Fall back to that call in TemplateResourc
outputs are not already cached to avoid unnecessary overhead in the common
case.
Attribute values are now always fetched before the reference ID, so that we
won't end up making two RPC calls in the case where we also need to read
other outputs.
Change-Id: I66da13c0bb0247
Closes-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/pike) | #8 |
Fix proposed to branch: stable/pike
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #9 |
Fix proposed to branch: stable/pike
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/pike) | #10 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/pike
commit ab46fae9cf386b9
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 10:53:38 2017 -0400
Speed up show-stack with outputs in legacy path
When we show a stack including the outputs, we calculate all of the
resource attributes that are referenced anywhere in the stack. In
convergence, these are either already cached (and therefore fast) or need
to be cached (and therefore the initial slowness will pay off in future).
This isn't the case in the legacy path though, since we are not doing
caching of attributes in the database in that path. So this is
unnecessarily calculating all of the referenced attribute values, which are
potentially very slow to get.
For legacy stacks, only calculate the attribute values needed to show the
outputs.
Change-Id: I35800c7f87b58d
Partial-Bug: #1719333
(cherry picked from commit 49d833f9aca3511
tags: | added: in-stable-pike |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master) | #11 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/pike) | #12 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/pike
commit d00f1a40f82ba6f
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 13:55:41 2017 -0400
Use show_output in TemplateResourc
TemplateRes
defined as an output in the nested template. This was being fetched by the
StackResour
*all* of the outputs of the nested stack and caching them in memory in case
something else were to reference any of them.
Unfortunately when calculating the resource data for a stack (which we
must always do when e.g. showing the outputs), we always include the
reference IDs of all resources, regardless of whether they are referenced
by get_resource in the data we are looking to populate. (In fact, we have
no way from the Template API to distinguish where get_resource is used on
a particular resource, only where there are dependencies on it.) This is no
problem under the assumption that getting the reference ID is quick, but
that assumption does not hold for TemplateResource.
The show_output RPC call only retrieves a single output (as opposed to
show_stack, used in StackResource.
them). Fall back to that call in TemplateResourc
outputs are not already cached to avoid unnecessary overhead in the common
case.
Attribute values are now always fetched before the reference ID, so that we
won't end up making two RPC calls in the case where we also need to read
other outputs.
Change-Id: I66da13c0bb0247
Closes-Bug: #1719333
(cherry picked from commit f1961c734e81f1e
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 10.0.0.0b1 | #13 |
This issue was fixed in the openstack/heat 10.0.0.0b1 development milestone.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 9.0.1 | #14 |
This issue was fixed in the openstack/heat 9.0.1 release.
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master) | #15 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit af0feeb18a4f5fb
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 14:32:13 2017 -0400
Ignore errors in OS::stack_id output
If a provider stack contained an OS::stack_id output and there was an error
in the output, we would raise TemplateOutputError when trying to calculate
the reference ID of the facade resource. Since we do that in many API
calls, such an error could render the stack effectively unusable.
If we encounter such an error, log it and fall back to the default
reference ID.
Change-Id: I1bc921fe74c54e
Partial-Bug: #1712280
Related-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (stable/pike) | #16 |
Related fix proposed to branch: stable/pike
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master) | #17 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 6857b7f3686e938
Author: Zane Bitter <email address hidden>
Date: Thu Sep 28 17:14:25 2017 -0400
Avoid RPC call in TemplateResourc
Most TemplateResources probably don't have an OS::stack_id output defined
in the template, so it's unfortunate that we have to make an RPC call to
check that every time we retrieve the resource's reference_id, which we
have to do for most stack operations.
To cut down on unnecessary RPC calls, check the template first locally to
see if there is an OS::stack_id output present.
Change-Id: Ia32ed6bca453b3
Related-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : | #18 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 2e4a6e237cf226e
Author: Zane Bitter <email address hidden>
Date: Tue Dec 19 16:36:43 2017 -0500
Use appropriate exception in StackResource.
Don't raise InvalidTemplate
output does not exist - it's not the case that get_output() is only used
for fetching attributes. Instead, raise NotFound from get_output(), and
translate that to InvalidTemplate
actually fetching an attribute.
Change-Id: I4f883e4b583a96
Related-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master) | #19 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #20 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master) | #21 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit f965ff615e3d4df
Author: Zane Bitter <email address hidden>
Date: Thu Jan 11 21:44:27 2018 -0500
Eliminate errors getting TemplateResource OS::stack_id
We want to avoid making an RPC call to get the OS::stack_id output in order
to calculate the reference ID of a TemplateResource whenever possible -
which should be most of the time given the limited use of this feature in
the wild - since this happens virtually every time we load the stack.
Previously, we were doing this by trying to load the template from the
parent stack's environment, and combine it with the stored parameters
(using live parameter values caused failures). However, this resulted in
failures in the case that the template in the parent stack's environment
differed from the one that was last used to update the resource if the
parameter schema had been modified in an incompatible way.
It transpires that we already determine what outputs are available at
object instantiation time in order to populate the attributes_schema. So we
can just use the attributes schema to infer the existence or otherwise of
the output without doing any other calculation in get_reference_id() that
could cause errors.
This method doesn't fail-safe, in the sense that if there are any errors
then we end up with an empty schema and won't fall back to making the RPC
call. However, if there are any errors then probably all bets are off for
the reference ID anyway. In common with the old method, it also has the
drawback of using the latest template rather than the existing one
(although again, once you do an update removing the OS::stack_id output
then all bets are off in terms of continuing to see that value as the
reference ID).
Unlike the previous method, this doesn't take into account disabled outputs
(since calculating conditions requires access to the parameters, which were
the cause of the problem). Also, the code that calculates the attributes
schema doesn't use a proper API but hackily dives into the raw template
data, and fixing it would raise all of the same issues that we've had with
trying to load the template properly here. So we are kicking the can down
the road a bit or, alternatively, consolidating our issues into a single
place, depending on your point of view.
Change-Id: Ic63fe290249dba
Closes-Bug: #1742646
Related-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : | #22 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 3cd84afcfc71ad4
Author: Zane Bitter <email address hidden>
Date: Thu Jan 11 20:11:04 2018 -0500
Cache the TemplateResource reference ID like an attribute
TemplateRes
output in the template. However getting it - or just checking that it even
exists - is very expensive, and it happens virtually every time we load the
stack in memory for any reason.
Since when the output is present the reference ID is also available as an
attribute, simply cache it in the database as if it were an attribute
regardless of whether the output exists. Once it is in the cache,
subsequent accesses will be cheap just like they are for attributes.
Keep previous performance optimisations in place as well, since attributes
are not stored in the database in the legacy path (so legacy stacks don't
benefit from this change), and since the cache does get invalidated e.g. on
every stack update.
Change-Id: Ib9cd0aa40d377e
Closes-Bug: #1742847
Related-Bug: #1719333
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (stable/pike) | #23 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit 6a27419521bae9c
Author: Zane Bitter <email address hidden>
Date: Mon Sep 25 14:32:13 2017 -0400
Ignore errors in OS::stack_id output
If a provider stack contained an OS::stack_id output and there was an error
in the output, we would raise TemplateOutputError when trying to calculate
the reference ID of the facade resource. Since we do that in many API
calls, such an error could render the stack effectively unusable.
If we encounter such an error, log it and fall back to the default
reference ID.
Change-Id: I1bc921fe74c54e
Partial-Bug: #1712280
Related-Bug: #1719333
(cherry picked from commit af0feeb18a4f5fb
Fix proposed to branch: master /review. openstack. org/507206
Review: https:/