Resolving stack outputs can be expensive

[Crag Wolfe]

When stack is shown (via show_stack) or loaded as nested stack, its outputs are re-resolved. When the outputs of a top-level stack reference a nested stack's outputs that in turn reference another nested stack (and so on), the calculation of the outputs can become quite expensive in terms of memory usage (loading a lot of nested stacks) and time.

Ideally, some (or all) of a stack's outputs or resource attributes would be stored to avoid the full expense of re-resolving them whenever they are referenced again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422982
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9ebbd1e6f1140467041220f9f72d5e3c7fe03b2f
Submitter: Jenkins
Branch: master

commit 9ebbd1e6f1140467041220f9f72d5e3c7fe03b2f
Author: Crag Wolfe <email address hidden>
Date: Thu Jan 19 19:54:54 2017 -0500

    Data migration enabling storage of resource attributes

    Working towards the goal of storing resource attributes in the db so
    as to avoid re-resolving them when appropriate. Adds an 'attr_data'
    object to the resource object, defined as a relationship on the
    already existing resource_properties_data table.

    Change-Id: I2104078d850da08b22547d7feab2bde00c543478
    Partial-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/441475

[Crag Wolfe (cwolfe)]

With patch set 2 in https://review.openstack.org/#/c/441475/ , doing an "openstack stack show" on a simple nested stack of depth 5 with outputs that resolve all the way down the deepest nested stack now takes ~3.8 seconds vs. ~5 seconds without the change.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/462343

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/472501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/472501
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=10a75d87dd171cb49cad936f5fb18828b59a6276
Submitter: Jenkins
Branch: master

commit 10a75d87dd171cb49cad936f5fb18828b59a6276
Author: Zane Bitter <email address hidden>
Date: Thu Jun 8 21:48:17 2017 -0400

    Always return live resource attributes in list_stack_resources

    Currently, we do live resolution of attribute values when in
    list_stack_resources with detail enabled. However, in future we will cache
    in the database those values that are referenced elsewhere in the template.
    Since it's unlikely that every attribute would be referenced in the
    template, this could lead to the API returning an inconsistent mix of live
    and cached data.

    Caching all of the attribute values on the off chance that someone might
    request them through this obscure API would be expensive, and there are
    also users that need to be able to access live resource attributes via
    Heat. Therefore, always return live data for the resource attributes in
    this API.

    Change-Id: Ifea71b30299076d14d9bb91f3bce4dd7fc4c88a1
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/475098

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/475099

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/475931

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/476293

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/476294

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/476295

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.openstack.org/446220
Reason: I reworked I07ac22cc4370a79bd8712e2431fa3272115bc0eb to not need classes to explicitly decorate their methods.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.openstack.org/462343
Reason: I reworked I07ac22cc4370a79bd8712e2431fa3272115bc0eb to not need classes to explicitly decorate their methods.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475098
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f47efe532d2e55f3f596f71f23bc5cced60b74cd
Submitter: Jenkins
Branch: master

commit f47efe532d2e55f3f596f71f23bc5cced60b74cd
Author: Zane Bitter <email address hidden>
Date: Wed Jun 21 20:04:01 2017 -0400

    Never cache the 'show' attribute

    The 'show' attribute is used to get all data about a resource. There are
    folks using this in lieu of having access to the underlying OpenStack APIs
    themselves. By not caching it, we ensure that they will always see a live
    value, even once we start storing attribute values in the database when
    modifying a resource.

    Change-Id: Id81a56f7d774bc18a2b5d62c80b369f1c12bcc5c
    Co-Authored-By: Crag Wolfe <email address hidden>
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422983
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0550659f5ffa9c8b92529d980c75144b35e8864a
Submitter: Jenkins
Branch: master

commit 0550659f5ffa9c8b92529d980c75144b35e8864a
Author: Crag Wolfe <email address hidden>
Date: Wed Jun 21 20:04:01 2017 -0400

    Store resource attributes in the DB

    Store resource attributes that may be cached in the DB, saving the
    cost of re-resolving them later. This works for most resources,
    specifically those that do not override the get_attribute() method.

    Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
    Partial-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/476293
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=176c3f7d142ba40b4e03a1badfbe58c7f0697f6a
Submitter: Jenkins
Branch: master

commit 176c3f7d142ba40b4e03a1badfbe58c7f0697f6a
Author: Zane Bitter <email address hidden>
Date: Wed Jun 21 20:04:01 2017 -0400

    Mark Server os_collect_config attribute as CACHE_NONE

    When the os_collect_config attribute was added to OS::Nova::Server (in
    64ddd071604bf4fbdeb97f8c8fa4e313dd611c3c), it was implemented both in
    _resolve_attribute() and in get_attribute(), with the latter overriding the
    former so it is never called (for this attribute) and so that the value is
    never cached. To be sure, to be sure, the attribute caches - which cannot
    contain the value of this attribute - were cleared whenever the deployment
    data is updated.

    It's much easier to just mark the attribute as CACHE_NONE. This also makes
    the status of this attribute visible to other parts of the code.

    Change-Id: Icaecf70529f19fb32b34de9cedb82e5006282002
    Partial-Bug: #1660831
    Partially-Implements: blueprint split-stack-default

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441475
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=45e4c53f78a2211c615b1e37bd86dacc815fb388
Submitter: Jenkins
Branch: master

commit 45e4c53f78a2211c615b1e37bd86dacc815fb388
Author: Zane Bitter <email address hidden>
Date: Wed Jun 21 20:04:01 2017 -0400

    Cache attributes with custom handling

    Previously, all caching of attribute values was done via the Attributes
    object. However, some resource types override Resource.get_attribute() to
    do custom handling of the trailing attribute path or dynamic attribute
    names, and in these cases the resulting values were not cached (since they
    don't go through the Attributes object).

    This patch adds a caching step for these resources:

    * OS::Senlin::Cluster
    * OS::Heat::ResourceChain
    * OS::Heat::ResourceGroup
    * OS::Heat::AutoscalingGroup
    * OS::Heat::SoftwareDeployment
    * OS::Heat::SoftwareDeploymentGroup
    * TemplateResource
    * AWS::CloudFormation::Stack

    Change-Id: I07ac22cc4370a79bd8712e2431fa3272115bc0eb
    Co-Authored-By: Crag Wolfe <email address hidden>
    Partial-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/479074

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/479075

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.openstack.org/479074
Reason: Will merge I50de08ae6b40958781ef4223c82a5c195b0757bf instead.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/476294
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4250958a0384e2b1cd005f0778793c1ff5997474
Submitter: Jenkins
Branch: master

commit 4250958a0384e2b1cd005f0778793c1ff5997474
Author: Zane Bitter <email address hidden>
Date: Thu Jun 29 09:07:02 2017 -0400

    Add an add_output() method to Template

    This helps with dynamically building templates, and is similar to the
    add_resource method.

    Change-Id: I51ce5c913438083d6d43eb6d8a8bf55cfa8ea3ee
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/476295
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f256b61dd222f64fbcdea98c98060352ffab517f
Submitter: Jenkins
Branch: master

commit f256b61dd222f64fbcdea98c98060352ffab517f
Author: Zane Bitter <email address hidden>
Date: Thu Jun 29 09:07:02 2017 -0400

    Refactor out a new Resource.referenced_attrs() method

    Calculating all of the attributes of a resource that are referenced in the
    template was previously done inside the node_data() method, but this is
    useful for the resource to be able to access for other reasons so refactor
    it into a separate method.

    Change-Id: I8ec943652e6f93a2352f2ea5e6ac46e0088e5458
    Related-Bug: #1660831
    Partially-Implements: blueprint stack-definition

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475931
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=1fff9516398e40f126a84a5fd6d2ad261838129b
Submitter: Jenkins
Branch: master

commit 1fff9516398e40f126a84a5fd6d2ad261838129b
Author: Zane Bitter <email address hidden>
Date: Mon Jul 10 13:47:59 2017 -0400

    Create explicit outputs in StackResources

    When we are accessing attributes of resources in a nested stack (via
    attributes of the parent resource), create output definitions in the nested
    template to explicitly reference them.

    In the long term, this should enable us to access the attribute values as
    outputs directly using the APIs, rather than querying the resources
    individually.

    In the short term though, this means that access to these resource
    attributes is no longer invisible to the child stack, so it will be able to
    store the values of the attributes that are accessed (and ignore the ones
    that are not).

    Change-Id: I61d8e5deff45a648657ba685e033a00a492412f2
    Partial-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/482145
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b902d93ade7f22e15d431161d91634f168dcd26c
Submitter: Jenkins
Branch: master

commit b902d93ade7f22e15d431161d91634f168dcd26c
Author: Zane Bitter <email address hidden>
Date: Mon Jul 10 13:48:01 2017 -0400

    Handle errors calculating dep_attrs for nested get_attr

    Ensure that in convergence we can calculate the dep_attrs of a resource,
    including the attributes referenced in outputs, even if:

    - One of the resource's attributes is referenced in an ouput; and
    - The attribute path itself also contains a get_attr function; and
    - The attribute referenced by the inner get_attr function in the output is
      not referenced by any resources.

    Since the attributes referenced only in outputs (and not in resources) are
    not passed to dependent resources in the NodeData, the data required to
    resolve the inner get_attr function is not available. Therefore getting the
    dep_attrs for the relevant resource from the output definition would fail.
    The next patch does so during the calculation of the NodeData, and any
    resulting exception is not caught (bug 1703043), so the stack would end up
    stuck IN_PROGRESS.

    To avoid this, ignore any errors resolving attribute path components in the
    course of calculating the dep_attrs of a given resource. This may mean that
    an attribute value required for the outputs is not stored in the database,
    but it can still be resolved live later.

    Change-Id: I2ed1e3dacd371e36656559ba92dfada05df0ceba
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475099
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=cfff39ce68be4ad88bea8f595aa9ae4e78983c78
Submitter: Jenkins
Branch: master

commit cfff39ce68be4ad88bea8f595aa9ae4e78983c78
Author: Zane Bitter <email address hidden>
Date: Mon Jul 10 13:47:59 2017 -0400

    Always cache attributes in outputs in Resource.node_data()

    Ensure that attributes that are referenced in outputs get cached (and
    therefore stored in the database) even when they are not referenced by
    other resources.

    Change-Id: I667ab04f91edddef5c5dbec0a89d465110c312b4
    Closes-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/479075
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=118fc127b3fcf29bff8f63129a1fe372d3cedda3
Submitter: Jenkins
Branch: master

commit 118fc127b3fcf29bff8f63129a1fe372d3cedda3
Author: Zane Bitter <email address hidden>
Date: Tue Jun 13 19:38:21 2017 -0400

    Fix a DB error when updating the resource attributes

    This prevents the attributes from being stored sometimes, and is possibly
    correlated with occasional random failures of the gate jobs.

    Change-Id: I8f6567ec792bd015d59ffc04da9f2f37f5b0f13a
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/531925

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/531926

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532301

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531925
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3ec13d9a8fe1ccf03913ed0a02c720312c13332b
Submitter: Zuul
Branch: master

commit 3ec13d9a8fe1ccf03913ed0a02c720312c13332b
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Fix attribute caching in SoftwareDeploymentGroup

    When the parent template references the 'deploy_stdouts', 'deploy_stderrs',
    or 'deploy_status_codes' (all plural) attributes of
    SoftwareDeploymentGroup, ensure that outputs are generated that reference
    the corresponding 'deploy_stdout', 'deploy_stderr', or 'deploy_status_code'
    (all singular) attributes of the child template's SoftwareDeployment
    resources.

    Change-Id: Iff7101dbaa57602be68e3c7447971dacdd747880
    Closes-Bug: #1741552
    Partial-Bug: #1731349
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530975
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4a3a1ad15bdcfd67e0548481c70545e41a9476d9
Submitter: Zuul
Branch: master

commit 4a3a1ad15bdcfd67e0548481c70545e41a9476d9
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Fix attribute caching in Heat AutoScalingGroup

    Attributes for an individual resource in an OS::Heat::AutoScalingGroup can
    be retrieved using the attribute name 'resource.<index>'. Because we were
    treating the index as a resource name when generating the template, we
    would never generate an output in the nested template that referenced the
    resource's attribute, so it wouldn't get cached. Since we can't know the
    indices ahead of time, the only way to cache these is to do it for all
    resources in the group - which is the same data cached by the equivalent
    'outputs' attribute.

    We also need to take into account both possible forms of syntax:

        [<asg_name>, resource.<index>.<attr_name>, ...]

    or:

        [<asg_name>, resource.<index>, <attr_name>, ...]

    Change-Id: I1d2898cdd4759b1bb9de1896a40056685e728f44
    Closes-Bug: #1737047
    Partial-Bug: #1731349
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532301
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=77112137bb368304be2437de09f0224e8e38ab58
Submitter: Zuul
Branch: master

commit 77112137bb368304be2437de09f0224e8e38ab58
Author: Zane Bitter <email address hidden>
Date: Tue Jan 9 11:29:25 2018 -0500

    Fix resource.<n> attribute caching in ResourceGroup/Chain

    When generating the nested template output, we need to take into account
    both forms of syntax:

       get_attr: [<group_name>, resource.<index>.<attr_name>, ...]

    or

       get_attr: [<group_name>, resource.<index>, <attr_name>, ...]

    Previously we would generate an output for the former case but not the
    latter, so the attribute wouldn't get cached when that syntax was used.

    Change-Id: I2a059d5cc42d794ca71caaa1661c32eb76c733fc
    Closes-Bug: #1742185
    Partial-Bug: #1731349
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/539659

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/539661

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/539662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531926
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7a046e6ffd8daaa21202eeb691d12f2dbf9abfc1
Submitter: Zuul
Branch: master

commit 7a046e6ffd8daaa21202eeb691d12f2dbf9abfc1
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Include outputs in rolling update of ResourceGroup

    The template generated by ResourceGroup should include outputs that
    reference any needed resource attributes. However, previously it only did
    so when rolling updates were not enabled by an update_policy.

    Change-Id: Ice7a92fe8d2b14e2470d9089a8e191c8385994f1
    Closes-Bug: #1741981
    Partial-Bug: #1731349
    Related-Bug: #1660831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/541763

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539659
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=cabb3b965c3358a0051facf1372aef6a679d7367
Submitter: Zuul
Branch: stable/pike

commit cabb3b965c3358a0051facf1372aef6a679d7367
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Fix attribute caching in SoftwareDeploymentGroup

    When the parent template references the 'deploy_stdouts', 'deploy_stderrs',
    or 'deploy_status_codes' (all plural) attributes of
    SoftwareDeploymentGroup, ensure that outputs are generated that reference
    the corresponding 'deploy_stdout', 'deploy_stderr', or 'deploy_status_code'
    (all singular) attributes of the child template's SoftwareDeployment
    resources.

    Change-Id: Iff7101dbaa57602be68e3c7447971dacdd747880
    Closes-Bug: #1741552
    Partial-Bug: #1731349
    Related-Bug: #1660831
    (cherry picked from commit 3ec13d9a8fe1ccf03913ed0a02c720312c13332b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539661
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=07382d99d1b4dcf5697c88e43b8f81f647965d3c
Submitter: Zuul
Branch: stable/pike

commit 07382d99d1b4dcf5697c88e43b8f81f647965d3c
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Fix attribute caching in Heat AutoScalingGroup

    Attributes for an individual resource in an OS::Heat::AutoScalingGroup can
    be retrieved using the attribute name 'resource.<index>'. Because we were
    treating the index as a resource name when generating the template, we
    would never generate an output in the nested template that referenced the
    resource's attribute, so it wouldn't get cached. Since we can't know the
    indices ahead of time, the only way to cache these is to do it for all
    resources in the group - which is the same data cached by the equivalent
    'outputs' attribute.

    We also need to take into account both possible forms of syntax:

        [<asg_name>, resource.<index>.<attr_name>, ...]

    or:

        [<asg_name>, resource.<index>, <attr_name>, ...]

    Change-Id: I1d2898cdd4759b1bb9de1896a40056685e728f44
    Closes-Bug: #1737047
    Partial-Bug: #1731349
    Related-Bug: #1660831
    (cherry picked from commit 4a3a1ad15bdcfd67e0548481c70545e41a9476d9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539662
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e94e90b4f2f787cc4830ff67897df8464b994f92
Submitter: Zuul
Branch: stable/pike

commit e94e90b4f2f787cc4830ff67897df8464b994f92
Author: Zane Bitter <email address hidden>
Date: Tue Jan 9 11:29:25 2018 -0500

    Fix resource.<n> attribute caching in ResourceGroup/Chain

    When generating the nested template output, we need to take into account
    both forms of syntax:

       get_attr: [<group_name>, resource.<index>.<attr_name>, ...]

    or

       get_attr: [<group_name>, resource.<index>, <attr_name>, ...]

    Previously we would generate an output for the former case but not the
    latter, so the attribute wouldn't get cached when that syntax was used.

    Change-Id: I2a059d5cc42d794ca71caaa1661c32eb76c733fc
    Closes-Bug: #1742185
    Partial-Bug: #1731349
    Related-Bug: #1660831
    (cherry picked from commit 77112137bb368304be2437de09f0224e8e38ab58)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541763
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=1cb985536c6b208b2a9616f6ac9083570bb786d8
Submitter: Zuul
Branch: stable/pike

commit 1cb985536c6b208b2a9616f6ac9083570bb786d8
Author: Zane Bitter <email address hidden>
Date: Mon Jan 8 17:23:12 2018 -0500

    Include outputs in rolling update of ResourceGroup

    The template generated by ResourceGroup should include outputs that
    reference any needed resource attributes. However, previously it only did
    so when rolling updates were not enabled by an update_policy.

    The tests are trivially different than the patch on master, only because
    ce120bfda848e55ba06b606681e7c1b8ae3de923 is not present in Pike.

    Change-Id: Ice7a92fe8d2b14e2470d9089a8e191c8385994f1
    Closes-Bug: #1741981
    Partial-Bug: #1731349
    Related-Bug: #1660831
    (cherry picked from commit 7a046e6ffd8daaa21202eeb691d12f2dbf9abfc1)