resource group fail to rollback when `to be deleted` resources taken quota

[Rico Lin]

Normally in a cluster of resources, we need enough quota to make sure update will success, but since we didn't have any way to control(or reserve) quota for all resources in the stack. We hit resource limit when update(which is fine because there is not enough quota to complete anyway) and become update failed. The Problem shows up when we ask that update to rollback (for Magnum cluster this situation always true), and it will fail on almost every time for a complex resource group because quota still held by other resources. Example, When we update a cluster from 20 nodes to 50 nodes. We stuck when updating node number 40 because we run out of resources. So we might have around 20 nodes required to roll back with update replace (for Magnum cluster this is always true), and another 20 nodes (number 21-40) needs to be deleted.
But in most cases roll back for first 20 nodes will likely fail since the other 20 nodes still held resource quota.

What we can do is to make the priority and make sure we delete resources before starting to update/create other resources.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/499020

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/505811

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/510290

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/510290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505811
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=09d74ffa3cb55e62318e0cd9eac10a9cb0c1a70a
Submitter: Zuul
Branch: master

commit 09d74ffa3cb55e62318e0cd9eac10a9cb0c1a70a
Author: Zane Bitter <email address hidden>
Date: Wed Sep 20 14:24:46 2017 -0400

    Prioritise resource deletion over creation

    Because of quotas, there are times when creating a resource and then
    deleting another resource may fail where doing it in the reverse order
    would work, even though the resources are independent of one another.

    When enqueueing 'check_resource' messages, send those for cleanup nodes
    prior to those for update nodes. This means that all things being equal
    (i.e. no dependency relationship), deletions will be started first. It
    doesn't guarantee success when quotas allow, since only a dependency
    relationship will cause Heat to wait for the deletion to complete before
    starting creation, but it is a risk-free way to give us a better chance of
    succeeding.

    Change-Id: I9727d906cd0ad8c4bf9c5e632a47af6d7aad0c72
    Partial-Bug: #1713900

[Zane Bitter (zaneb)]

We merged a possible improvement in Queens. Whether to go all the way and alter the dependency graph is something there isn't consensus on. Moving this bug to Rocky.