stack updated_time isn't updated on update

Bug #1448155 reported by Steven Hardy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Steven Hardy
Kilo
Fix Released
Undecided
Unassigned

Bug Description

We expose two time related metrics per stack, creation_time, and updated_time.

The updated time isn't updated on creation (perhaps understandable, but intuitively I expected it to be the same as creation time, or perhaps the last state transition during the create, e.g that from CREATE_IN_PROGRESS to CREATE_COMPLETE)

But worse than that, updated_time isn't updated when we, erm, update the stack and it goes into UPDATE_IN_PROGRESS state.

This is a problem if you care about the events since an update started, e.g because you want to monitor for hook/breakpoint events - currently this is impossible unless you derive the timestamp for the transition to UPDATE_IN_PROGRESS from another event.

It seems like we should modify updated_time every time a state transition occurs to the stack object, and probably creation_time should be the time when we persist the stack, with updated_time again registering

-bash-4.2$ heat stack-create rg100 -f resource_group2.yaml -e ip_mappings.yaml -e env.yaml
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| a71bf1fc-b8ef-4cc5-8f89-c126a13c064c | rg100 | CREATE_IN_PROGRESS | 2015-04-24T13:51:34Z |
+--------------------------------------+------------+--------------------+----------------------+
-bash-4.2$ heat stack-show rg100 | grep _time
| creation_time | 2015-04-24T13:51:34Z |
| updated_time | None |

-bash-4.2$ heat stack-update rg100 -f resource_group2.yaml -e ip_mappings.yaml -e env.yaml
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| a71bf1fc-b8ef-4cc5-8f89-c126a13c064c | rg100 | UPDATE_IN_PROGRESS | 2015-04-24T13:51:34Z |
+--------------------------------------+------------+--------------------+----------------------+
-bash-4.2$ heat stack-show rg100 | grep _time
| creation_time | 2015-04-24T13:51:34Z |
| updated_time | None

Steven Hardy (shardy)
Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Revision history for this message
Steven Hardy (shardy) wrote :

So, we do some odd things, like setting the stack.Stack object updated_time on suspend/resume, but not storing it:

https://github.com/openstack/heat/blob/master/heat/engine/stack.py#L1231

It seems like the most consistent behaviour would be to just update the updated_at time in state_set, so whenever the stack undergo's a state transition, we modify the updated_at timestamp.

This would enable easy selection of all events for the current operation, and the timestamp after any update COMPLETE or FAILED operation would be more accurate, since currently if an update fails we don't modify updated_at at all, only when an update completes, in which case we store the time when the update started (?!) rather than when it completed.

Similar problems seem to exist for resource update timestamps.

Revision history for this message
Steven Hardy (shardy) wrote :

Ok, so it seems like we can't update on every state transition, historical context (tl;dr CFN doesn't do it that way..):

https://bugs.launchpad.net/heat/+bug/1193269

Updating (and not storing) the updated_at in the non-update transitions still seems odd, I'm not sure why we're doing that, and it's odd that we store the update timestamp from the start of the update on the transition to complete.

But it seems I can't solve my problem (which is that I want all the hook/breakpoint events since an update started) by fixing any of those, so it seems the least-bad alternative is to create events for stack state transitions as well as resource ones (this, coincidentally is also the way the CFN event API appears to work, so we'd be consistent there too).

We'd end up with something like:

heat event-list rg106
+---------------+--------------------------------------+-------------------------------------+--------------------+----------------------+
| resource_name | id | resource_status_reason | resource_status | event_time |
+---------------+--------------------------------------+-------------------------------------+--------------------+----------------------+
| rg106 | 74f81d8b-5bdf-4963-9210-82ae2d03bca9 | Stack CREATE started | CREATE_IN_PROGRESS | 2015-04-27T19:06:12Z |
| server_group | 0cb9c170-13c8-4393-b3ce-8fe5e7e2a696 | state changed | CREATE_IN_PROGRESS | 2015-04-27T19:06:12Z |
| server_group | ac12bcce-25a4-4d28-a4de-3cf6424a1640 | state changed | CREATE_COMPLETE | 2015-04-27T19:06:14Z |
| rg106 | 9b77a59d-6775-44e8-ad4b-3f646cc80bfd | Stack CREATE completed successfully | CREATE_COMPLETE | 2015-04-27T19:06:14Z |
+---------------+--------------------------------------+-------------------------------------+--------------------+----------------------+

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/177961

Changed in heat:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/177961
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d5793c08e04512795b258bac788e30a7eeff8ad9
Submitter: Jenkins
Branch: master

commit d5793c08e04512795b258bac788e30a7eeff8ad9
Author: Steven Hardy <email address hidden>
Date: Fri May 1 16:40:03 2015 +0100

    Generate stack events for stack state transitions

    Currently, the event-list output is very resource-centric, despite
    being scoped to the stack from an API path perspective.

    This, combined with the fact that the stack updated_at timestamp
    is only updated after a succesful update (ref bug #1193269) makes
    it impossible to derive the time when an update started via any API.

    This is a problem when trying to use the new hook/breakpoint API,
    because it's necessary to poll for all events since the most recent
    update started, disregarding any stale hook events from previous
    updates (which may have failed or timed out without the hooks getting
    cleared).

    To work around this, add an event for each stack state transition,
    such that you can detect the transition to UPDATE_IN_PROGRESS,
    then use that event as a marker to get post-update-started events.

    Without this (or some other way to determine when the stack update
    started), the hooks pre-update functionality landed for kilo is
    not really usable (particularly mechanically via scripts).

    Change-Id: Idff342b3aecc2d145dfbc7c0f610ad0ca8e52c8b
    Partial-Bug: #1448155

Changed in heat:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/182020

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/kilo)

Reviewed: https://review.openstack.org/182020
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=06a5c0e56fc76a62f7c41c6e81ec1b59710ffb05
Submitter: Jenkins
Branch: stable/kilo

commit 06a5c0e56fc76a62f7c41c6e81ec1b59710ffb05
Author: Steven Hardy <email address hidden>
Date: Fri May 1 16:40:03 2015 +0100

    Generate stack events for stack state transitions

    Currently, the event-list output is very resource-centric, despite
    being scoped to the stack from an API path perspective.

    This, combined with the fact that the stack updated_at timestamp
    is only updated after a succesful update (ref bug #1193269) makes
    it impossible to derive the time when an update started via any API.

    This is a problem when trying to use the new hook/breakpoint API,
    because it's necessary to poll for all events since the most recent
    update started, disregarding any stale hook events from previous
    updates (which may have failed or timed out without the hooks getting
    cleared).

    To work around this, add an event for each stack state transition,
    such that you can detect the transition to UPDATE_IN_PROGRESS,
    then use that event as a marker to get post-update-started events.

    Without this (or some other way to determine when the stack update
    started), the hooks pre-update functionality landed for kilo is
    not really usable (particularly mechanically via scripts).

    Change-Id: Idff342b3aecc2d145dfbc7c0f610ad0ca8e52c8b
    Partial-Bug: #1448155

tags: added: in-stable-kilo
Changed in heat:
status: In Progress → Triaged
Zane Bitter (zaneb)
Changed in heat:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.