Convergence loads all resources to check a single resource

[Zane Bitter]

The resource check operation in convergence was carefully architected to be as lightweight as possible, and needs to load only the resource being checked.

Unfortunately, we are inadvertently loading *all* of the resources from the database. This is slow, memory-intensive, and extremely hard on the DB. We need to stop doing this, and also add tests to ensure that we can't start accessing stack.resources in future and inadvertently start doing this again.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/420191

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.openstack.org/420191
Reason: Currently it is inevitable that the Resource objects will be created somewhere in almost all cases. As long as that remains the case, (1) a true fix for this problem involves being able to do that without loading resources from the DB, and (2) this change actually makes things less efficient, by creating and then discarding the resource definitions instead of keeping them around in the Resource objects for which they would have been created again anyway.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/420971

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/420972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420971
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=45073226752c58d640ea5a59b7e532c022a4939b
Submitter: Jenkins
Branch: master

commit 45073226752c58d640ea5a59b7e532c022a4939b
Author: Zane Bitter <email address hidden>
Date: Mon Jan 16 16:58:04 2017 -0500

    Don't load non-referenced resources from DB

    During a convergence check operation, we should never load resources from
    the database. However, all resources will inevitably be created whenever we
    do stack[some_resource]. Previously, resources not represented in the
    cached data (i.e. the ones we were least interested in) would get loaded
    from the database when this occurred.

    This change ensures that only the resource being checked is loaded from the
    database, as intended, and adds asserts to prevent any future regression
    occurring silently.

    Change-Id: I9e568640d776748a1c9f2950f8bb0a8cea325996
    Closes-Bug: #1656429

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420972
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e950bbb6ad358a64b6ab90a2ce9e770d0b8a5093
Submitter: Jenkins
Branch: master

commit e950bbb6ad358a64b6ab90a2ce9e770d0b8a5093
Author: Zane Bitter <email address hidden>
Date: Mon Jan 16 16:58:05 2017 -0500

    Don't load resources in dep_attrs

    It's unnecessary to load the entire Resource just to get its name.

    Change-Id: I09b9f49f5b3b631884ef0c513f3766df4502734b
    Related-Bug: #1656429

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424372

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 8.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424372
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4b90fe9f00ba1535212fe1f49154a11d78478191
Submitter: Jenkins
Branch: stable/newton

commit 4b90fe9f00ba1535212fe1f49154a11d78478191
Author: Zane Bitter <email address hidden>
Date: Mon Jan 16 16:58:04 2017 -0500

    Don't load non-referenced resources from DB

    During a convergence check operation, we should never load resources from
    the database. However, all resources will inevitably be created whenever we
    do stack[some_resource]. Previously, resources not represented in the
    cached data (i.e. the ones we were least interested in) would get loaded
    from the database when this occurred.

    This change ensures that only the resource being checked is loaded from the
    database, as intended.

    The asserts added in master to prevent regressions are *not* included in
    the backport, since they add risk without benefit to a stable branch.

    Change-Id: I9e568640d776748a1c9f2950f8bb0a8cea325996
    Closes-Bug: #1656429
    (cherry picked from commit 45073226752c58d640ea5a59b7e532c022a4939b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.5 release.