heat updates raw_template when there is no change in raw_template

[Steve Baker]

heat updates raw_template when there is no change in raw_template
Looking into the Heat code it looks that Heat should update the raw_template only where there is a change in the template (update stack) .
The UPDATEs does not seem to update anything - same values are used over and over again.
This lead to disk explosion in case of a large templates (100K-200K for a line) on Maria DB.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222372

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222372
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a69431ab6c938ce591082344f0bebc30ef9464eb
Submitter: Jenkins
Branch: master

commit a69431ab6c938ce591082344f0bebc30ef9464eb
Author: Zane Bitter <email address hidden>
Date: Thu Sep 10 17:13:33 2015 -0400

    Make ResourceDefinition round-trip stable to avoid extra writes

    The part of a ResourceDefinition that lists explicit dependencies was not
    round-trip stable. As a result, when we copied a new resource definition
    into the existing template during a stack update, we would end up rewriting
    the template unnecesarily (i.e. even though we check for changes) every
    time if depends_on was not specified in the resource originally. At the end
    of each update, we write the new template to the DB in its entirety, which
    removes these extra lines again, ensuring that we will experience the same
    problem on every update. This was causing a *lot* of unnecessary writes.

    This change ensures that the definition remains stable across a round-trip,
    so that no unnecessary changes appear in the template.

    Change-Id: If7292e49755db0153d7d0db9f7d3875fa9c1d408
    Closes-Bug: #1494108

[Steve Baker (steve-stevebaker)]

This is only partially fixed, there is also an issue with the files dict in raw_template

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/224935

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224935
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=779ba8e431aecac3c451c08aedcc0ad8cf0a4449
Submitter: Jenkins
Branch: master

commit 779ba8e431aecac3c451c08aedcc0ad8cf0a4449
Author: Steve Baker <email address hidden>
Date: Fri Sep 18 11:56:57 2015 +1200

    Only write to template files if template data changes

    Currently the template data is assigned to the files dict regardless
    of whether the data has changed or not.

    This has no consequence on Liberty, but the files dict in Kilo
    and Juno is a sqlalchemy MutableDict so these assignments are triggering
    updates to the raw_template table. These updates are even triggered for
    calls to heat resource-list. With large raw_template records this can make
    heat unusable.

    Therefore this fix is presented here as having no effect (and being
    harmless) to git master, but high priority backports for Kilo and Juno.

    Change-Id: Ibb714a9c5dc9f6879ff108419127da985a57897b
    Closes-Bug: #1494108

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/232655

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/232656

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/232674

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/232675

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/232655
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9e6d7c29c83059d0e08c6df512200351927e0b85
Submitter: Jenkins
Branch: stable/kilo

commit 9e6d7c29c83059d0e08c6df512200351927e0b85
Author: Zane Bitter <email address hidden>
Date: Thu Sep 10 17:13:33 2015 -0400

    Make ResourceDefinition round-trip stable to avoid extra writes

    The part of a ResourceDefinition that lists explicit dependencies was not
    round-trip stable. As a result, when we copied a new resource definition
    into the existing template during a stack update, we would end up rewriting
    the template unnecesarily (i.e. even though we check for changes) every
    time if depends_on was not specified in the resource originally. At the end
    of each update, we write the new template to the DB in its entirety, which
    removes these extra lines again, ensuring that we will experience the same
    problem on every update. This was causing a *lot* of unnecessary writes.

    This change ensures that the definition remains stable across a round-trip,
    so that no unnecessary changes appear in the template.

    Change-Id: If7292e49755db0153d7d0db9f7d3875fa9c1d408
    Partial-Bug: #1494108
    (cherry picked from commit a69431ab6c938ce591082344f0bebc30ef9464eb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/232656
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a4aafba0c2b387df888dfd805594b78b39b5f495
Submitter: Jenkins
Branch: stable/kilo

commit a4aafba0c2b387df888dfd805594b78b39b5f495
Author: Steve Baker <email address hidden>
Date: Fri Sep 18 11:56:57 2015 +1200

    Only write to template files if template data changes

    Currently the template data is assigned to the files dict regardless
    of whether the data has changed or not.

    This has no consequence on Liberty, but the files dict in Kilo
    and Juno is a sqlalchemy MutableDict so these assignments are triggering
    updates to the raw_template table. These updates are even triggered for
    calls to heat resource-list. With large raw_template records this can make
    heat unusable.

    Therefore this fix is presented here as having no effect (and being
    harmless) to git master, but high priority backports for Kilo and Juno.

    Change-Id: Ibb714a9c5dc9f6879ff108419127da985a57897b
    Closes-Bug: #1494108
    (cherry picked from commit 779ba8e431aecac3c451c08aedcc0ad8cf0a4449)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/232674
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7e8fc3ae865a7c5517ee2d47adf019937a52354e
Submitter: Jenkins
Branch: stable/juno

commit 7e8fc3ae865a7c5517ee2d47adf019937a52354e
Author: Steve Baker <email address hidden>
Date: Fri Sep 11 18:06:05 2015 +1200

    Make ResourceDefinition round-trip stable to avoid extra writes

    The part of a ResourceDefinition that lists explicit dependencies was not
    round-trip stable. As a result, when we copied a new resource definition
    into the existing template during a stack update, we would end up rewriting
    the template unnecesarily (i.e. even though we check for changes) every
    time if depends_on was not specified in the resource originally. At the end
    of each update, we write the new template to the DB in its entirety, which
    removes these extra lines again, ensuring that we will experience the same
    problem on every update. This was causing a *lot* of unnecessary writes.

    This change ensures that the definition remains stable across a round-trip,
    so that no unnecessary changes appear in the template.

    Change-Id: If7292e49755db0153d7d0db9f7d3875fa9c1d408
    Partial-Bug: #1494108
    (cherry-picked from 9ebac709e2d3e8814fdee9de99ff49a4681b0ab2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/232675
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9ee5d625ebcc8124170c8b269e0d20664fe286c3
Submitter: Jenkins
Branch: stable/juno

commit 9ee5d625ebcc8124170c8b269e0d20664fe286c3
Author: Steve Baker <email address hidden>
Date: Fri Sep 18 11:56:57 2015 +1200

    Only write to template files if template data changes

    Currently the template data is assigned to the files dict regardless
    of whether the data has changed or not.

    This has no consequence on Liberty, but the files dict in Kilo
    and Juno is a sqlalchemy MutableDict so these assignments are triggering
    updates to the raw_template table. These updates are even triggered for
    calls to heat resource-list. With large raw_template records this can make
    heat unusable.

    Therefore this fix is presented here as having no effect (and being
    harmless) to git master, but high priority backports for Kilo and Juno.

    Change-Id: Ibb714a9c5dc9f6879ff108419127da985a57897b
    Closes-Bug: #1494108
    (cherry-picked from 779ba8e431aecac3c451c08aedcc0ad8cf0a4449)