HA and autoscaling are broken due to stack_get_by_name tenant scoping

Bug #1078779 reported by Steven Hardy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steven Hardy
Grizzly
Fix Released
High
Steven Hardy

Bug Description

bc2c6db "Always filter by tenant_id in stack_get_by_name" changes the scoping of stack_get_by_name, such that the stored admin context can no longer lookup stacks by name - this breaks autoscaling and HA (and anything else using WatchRule alarm features.

You see this in the engine.log:

"WARNING [heat.engine.watchrule] Could not process watch state ALARM for stack"

This is because the stack lookup for the stored admin context returns None when calling stack_get_by_name

The fix appears to be moving the WatchRule implementation to track stacks by ID not by name internally

Steven Hardy (shardy)
Changed in heat:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Steven Hardy (shardy)
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/16167

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/16167
Committed: http://github.com/heat-api/heat/commit/ad1e45573ece5bb5ca5d6b2273d0d3ab8faba806
Submitter: Jenkins
Branch: master

commit ad1e45573ece5bb5ca5d6b2273d0d3ab8faba806
Author: Steven Hardy <email address hidden>
Date: Thu Nov 15 10:03:46 2012 +0000

    heat workaround for HA/Autoscaling regression

    Interim workaround for HA/Autoscaling regression, caused by change of
    scope for stack_get_by_name to be per-tenant, meaning we need to do a
    brute-force lookup to find the stack entry when using the stored admin
    context.

    A better, more comprehensive (but more intrusive) fix is in-progress
    but this at least returns us to the same functionality we had before

    Note this means that all stacks using WatchRule still need to have names
    unique accross all tenants.

    bug 1078779

    Change-Id: Iad1830d38262d1afb63cee16a3e366d9fd09acb4
    Signed-off-by: Steven Hardy <email address hidden>

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/16480

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/16481

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/16482

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/16483

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/16480
Committed: http://github.com/heat-api/heat/commit/deffe8a1679b2fd0af803de9fa3aee46d41d5d84
Submitter: Jenkins
Branch: master

commit deffe8a1679b2fd0af803de9fa3aee46d41d5d84
Author: Steven Hardy <email address hidden>
Date: Wed Nov 14 15:41:17 2012 +0000

    WatchRule refer to stack by id not name

    Rework WatchRule to refer to stacks by uuid not name, this will
    help us move to allowing non-unique stack names containing WatchRule
    resources

    Ref bug 1078779

    Change-Id: Idbbbd65a05d7036860cc2feb044d568210071d21
    Signed-off-by: Steven Hardy <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/16481
Committed: http://github.com/heat-api/heat/commit/f96f15d2edb379d93c999f8d92f1d636ab87910e
Submitter: Jenkins
Branch: master

commit f96f15d2edb379d93c999f8d92f1d636ab87910e
Author: Steven Hardy <email address hidden>
Date: Mon Nov 19 15:31:01 2012 +0000

    Make CloudWatchAlarm names unique per-tenant

    Use physical_resource_name to ensure CloudWatchAlarm resources
    are unique per tenant

    Ref bug 1078779

    Change-Id: Id37393decd8d4f89e79af5c7e9eeb21f4e494699
    Signed-off-by: Steven Hardy <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/16482
Committed: http://github.com/heat-api/heat/commit/1563f3039f3335d53891da055dcbb2b2334a234f
Submitter: Jenkins
Branch: master

commit 1563f3039f3335d53891da055dcbb2b2334a234f
Author: Steven Hardy <email address hidden>
Date: Mon Nov 19 16:26:12 2012 +0000

    add watch_rule_get_all_by_stack dbapi call

    Add option to retrieve all watch rules for a given stack, which
    will be required to move to periodic watch threads per-stack

    Ref bug 1078779

    Change-Id: I350e2ee72c7eec0b91bf85ac52bb04e3cd00b4db
    Signed-off-by: Steven Hardy <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/16483
Committed: http://github.com/heat-api/heat/commit/d89f6c0ba1c0da0bbd42e08b44cfd74e412d2143
Submitter: Jenkins
Branch: master

commit d89f6c0ba1c0da0bbd42e08b44cfd74e412d2143
Author: Steven Hardy <email address hidden>
Date: Mon Nov 19 16:55:23 2012 +0000

    heat engine move to per-stack periodic watch threads

    Use the stack thread groups, so a separate watch thread is started
    for each stack - this avoids some of the context scoping problems
    previously encountered since we can pass the correct context to the
    periodic task when starting it

    Fixes bug 1078779

    Change-Id: I56e6a4b126199587e91548f450956d77ab2158f3
    Signed-off-by: Steven Hardy <email address hidden>

Angus Salkeld (asalkeld)
Changed in heat:
milestone: none → grizzly-2
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.