Querying resources for a stack that is in UPDATE_IN_PROGRESS should show new resources

[Randall Burt]

During a stack-update, new resources are not persisted until the operation either succeeds or fails. Queries for resources for a stack in UPDATE_IN_PROGRESS do not reflect the fact that these new resources are being created. This is confusing for users as tooling can not accurately report the status of the operation. While it is clear that the stack is being updated and resources that are being deleted or modified reflect their proper state, new resources are completely missing.

[Randall Burt (randall-burt)]

Converted from bp https://blueprints.launchpad.net/heat/+spec/update-stack-new-resource-state

[Steven Hardy (shardy)]

So is this still an issue?

It seems like every resource will correctly transition to CREATE, IN_PROGRESS as soon as the update task triggers a Resource.create():

https://github.com/openstack/heat/blob/master/heat/engine/update.py#L124

https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L347

So AFAICS new resources are not "completely missing", as soon as we start creating them they are persisted to the DB?

[Randall Burt (randall-burt)]

@Steven Hardy - I think this is still an issue but will verify in a bit.

[Randall Burt (randall-burt)]

@Steven Hardy - I just confirmed that new resources still don't show up on update until the update is complete.

[Zane Bitter (zaneb)]

The issue here is that the template change is not committed until the update is complete. We use the contents of the stored template to determine what resources exist when we load a Stack (e.g. to get the resource list), so new resources don't show up until this happens.

This has always been the case, so I assume this bug has existed forever.

The long-term fix is to incrementally update the stored template as resources are added/removed/modified, which will help to make updates more robust in general.

[Steven Hardy (shardy)]

After IRC discussion with Zane re this bug, it's apparent that we can't expect to fix this for Havana, since the fix is likely to be relatively non-trivial. Lets bump this to Icehouse and give some more thought to the incremental-update solution mentioned above.

[Steve Baker (steve-stevebaker)]

Any new thoughts on whether this bug is valid or fixable?

[Randall Burt (randall-burt)]

@Steve Baker, I think so. I will take a look in the next day or so and confirm.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100047

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100047
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=40693910beb336518bf06d7e0621b7b14414fc84
Submitter: Jenkins
Branch: master

commit 40693910beb336518bf06d7e0621b7b14414fc84
Author: Zane Bitter <email address hidden>
Date: Mon Jun 16 11:05:18 2014 -0400

    Update: persist current template on change

    After each resource is updated, persist the modified template to the
    database so that any new resources will be accessible to API calls -
    including signals to new WaitConditionHandles. This also ensures that if
    the Heat engine dies completely during an update, the template in the
    database will still be in a consistent state.

    Change-Id: Ie6f234302cf72213d4b0e1f5b963cd8def422498
    Closes-Bug: #1291905
    Closes-Bug: #1206702
    Closes-Bug: #1328342
    Implements: partial-blueprint update-failure-recovery

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/101288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/101288
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3ff80e61285b763a9d625692f1d790b9a36dd380
Submitter: Jenkins
Branch: master

commit 3ff80e61285b763a9d625692f1d790b9a36dd380
Author: Zane Bitter <email address hidden>
Date: Fri Jun 27 16:22:45 2014 -0400

    Pass the context when updating raw_templates

    Without a context, the raw_template was being stored using a different DB
    session, with the result that changes did not end up in the database.

    Also, don't use the same raw_template DB entry for the backup stack, since
    that causes competing modifications to the DB that break rollback.

    Change-Id: I6c71c19acac0b87943f36c57c2300cf2b0478aa3
    Closes-Bug: #1331872
    Related-Bug: #1291905
    Related-Bug: #1206702
    Related-Bug: #1328342