Convergence: Nothing returned from resource-list after stack-delete is called

[Steve Baker]

As soon as stack-delete is called on a convergence stack, subsequent calls to resource list return nothing. This is known to occur for a stack which is CREATE_IN_PROGRESS.

$ heat resource-list overcloud
# returns lots of results

$ heat stack-delete overcloud
+--------------------------------------+------------+--------------------+---------------------+---------------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+--------------------+---------------------+---------------------+
| 9d6d1e79-71c3-4f81-8946-5b6894149c96 | overcloud | DELETE_IN_PROGRESS | 2015-12-08T02:46:13 | 2015-12-08T02:58:57 |
+--------------------------------------+------------+--------------------+---------------------+---------------------+

$ heat resource-list overcloud
+---------------+----------------------+---------------+-----------------+--------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time |
+---------------+----------------------+---------------+-----------------+--------------+
+---------------+----------------------+---------------+-----------------+--------------+

[Sirushti Murugesan (sirushtim)]

Since the DELETE operation is an update with an empty template, the existing method of retrieving the resources by depending on the current template and iterating over the resource definitions won't work anymore. It looks like we'd have to query the database for existing resources to manually load instances of the Resource class to be passed back to the API.

[Anant Patil (ananta)]

With convergence, it should pick all the resources in DB for the stack and return them. Since concurrent updates are allowed, there could be many templates involved (beyond current and previous template) and searching resources by current template or previous template won't be helpful.

[Sirushti Murugesan (sirushtim)]

Yeah, querying would definitely need to be done per stack but we'd still need a way to load the resource definition by loading the template, so it looks like we'd have to load the template manually for every resource by using it's current_template_id attribute from the database. This should give us the template of the last physical operation done on a resource.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/278231

[Zane Bitter (zaneb)]

IIUC the fix for this will also effectively resolve the longstanding bug 1301320.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/342612

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/342613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342612
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=483c505d61a1161ecdab77e56f014ff34f1304d3
Submitter: Jenkins
Branch: master

commit 483c505d61a1161ecdab77e56f014ff34f1304d3
Author: Steve Baker <email address hidden>
Date: Fri Jul 15 11:53:31 2016 +1200

    Do not raise NotFound for resource_get_all_* queries

    A stack with zero resources is not an exception worthy condition (for
    example, a stack being deleted, a stack which really does have zero
    resources).

    Also its unusual that a collection query function would raise a
    NotFound when the result count is zero.

    This change fixes that for resource_get_all_by_stack and
    resource_get_all_active_by_stack

    Change-Id: I3420d5099830f5626095b17c01e9d886b477915c
    Related-Bug: #1301320
    Related-Bug: #1523748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/342613
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=10030beb2f2895150ecb55b4f539d661d0f7e4bc
Submitter: Jenkins
Branch: master

commit 10030beb2f2895150ecb55b4f539d661d0f7e4bc
Author: Steve Baker <email address hidden>
Date: Fri Jul 15 16:24:13 2016 +1200

    Fallback to db for resource show

    This change adds a Stack.resource_get which is used by
    describe_stack_resource (stack resource show). It has a different
    behaviour when the resource is missing from the stack resources dict,
    attempting to return a resource from the DB record in the current
    stack.

    This allows "stack resource show" to show a resource in a convergence
    stack while the resource is being deleted.

    Change-Id: I975778ac36ae31fe5bd3ebdbcd7c48a6fcc60757
    Partial-Bug: #1301320
    Related-Bug: #1523748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323614
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3ab0ede98c6dc0c0327977be994c139113fc0489
Submitter: Jenkins
Branch: master

commit 3ab0ede98c6dc0c0327977be994c139113fc0489
Author: Steve Baker <email address hidden>
Date: Fri May 27 17:07:03 2016 +1200

    Always eager load the raw_template for a stack

    The vast majority of stack fetches are immediately followed by a
    raw_template fetch, so this change always eagerly fetches the
    raw_template for every stack fetch.

    During stack versioned object creation the stack's raw_template object
    is used to construct the versioned raw template object.

    Change-Id: I1a6fb8fb7d069b50dd5d623c989acd5582818ae1
    Related-Bug: #1479723
    Related-Bug: #1523748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278231
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=fab4fe769bd10c458a218e372dcdddef21e289b4
Submitter: Jenkins
Branch: master

commit fab4fe769bd10c458a218e372dcdddef21e289b4
Author: Steve Baker <email address hidden>
Date: Fri Jun 3 09:10:26 2016 +1200

    Load resources from DB for resource list

    Currently, when a user makes an API call to list resources, they were
    retrieved from the currently active template associated with the
    stack. This worked in the legacy engine where no concurrent updates
    where possible. With convergence and thus concurrent updates, a stack
    is allowed to have resources of previous traversals and still continue
    creating new resources. Therefore, relying on the template to list
    resources won't exactly match the state in the database.

    For example, on deletes where we update with an empty template,
    currently, the stack parses the empty template searching for
    resources. Doing a `resource-list` when the stack is in a state of
    DELETE_IN_PROGRESS shows an empty list of resources and therefore not
    matching the state in the database.

    This change makes iter_resources always call _find_filtered_resources
    which builds all its resources from a database query.

    Change-Id: Ibe87a773c38efb6d4865fd3a1dbd079972dd8be4
    Closes-Bug: #1523748
    Closes-Bug: #1301320

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/483512

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/483571

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/483574

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483512
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5ce238fb17e38702ceef85a3706e9d2236c43e0e
Submitter: Jenkins
Branch: master

commit 5ce238fb17e38702ceef85a3706e9d2236c43e0e
Author: Zane Bitter <email address hidden>
Date: Thu Jul 13 13:59:27 2017 -0400

    Don't load new copies of current resources

    The Stack._find_filtered_resources() method returns Resource objects for
    all resources associated with the Stack, regardless of whether they are
    current (present in the template; latest version in the case of
    convergence). To do this, it previously created a new Resource object
    for every resource found in the database.

    However, for those resources which *are* current this is unnecessary. We
    can access the Resource object simply through self.resources. It turns
    out this is necessary for obtaining the required_by list for legacy
    stacks, because only the Resources obtained from self.resources also
    appear in the Dependencies graph obtained from self.dependencies. The
    required_by list is read when listing or showing resources, which would
    either return an empty list or fail for legacy stacks.

    This patch also makes the Resource.required_by() method more robust in
    its error handling.

    Change-Id: Id438336e5c88dc7c2d168ba01ee703faa17e8b8e
    Closes-Bug: #1703703
    Related-Bug: #1523748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483571
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3fb8722012a991855fc760ba450ed74acd8b43d2
Submitter: Jenkins
Branch: stable/ocata

commit 3fb8722012a991855fc760ba450ed74acd8b43d2
Author: Zane Bitter <email address hidden>
Date: Thu Jul 13 13:59:27 2017 -0400

    Don't load new copies of current resources

    The Stack._find_filtered_resources() method returns Resource objects for
    all resources associated with the Stack, regardless of whether they are
    current (present in the template; latest version in the case of
    convergence). To do this, it previously created a new Resource object
    for every resource found in the database.

    However, for those resources which *are* current this is unnecessary. We
    can access the Resource object simply through self.resources. It turns
    out this is necessary for obtaining the required_by list for legacy
    stacks, because only the Resources obtained from self.resources also
    appear in the Dependencies graph obtained from self.dependencies. The
    required_by list is read when listing or showing resources, which would
    either return an empty list or fail for legacy stacks.

    This patch also makes the Resource.required_by() method more robust in
    its error handling.

    Change-Id: Id438336e5c88dc7c2d168ba01ee703faa17e8b8e
    Closes-Bug: #1703703
    Related-Bug: #1523748
    (cherry picked from commit 5ce238fb17e38702ceef85a3706e9d2236c43e0e
                           and 65630d4e8bbff2f3ae5c659f8e9396ec7a6f6ae9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483574
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4e09419701090997c43fff847bd75b91b0c1ace9
Submitter: Jenkins
Branch: stable/newton

commit 4e09419701090997c43fff847bd75b91b0c1ace9
Author: Zane Bitter <email address hidden>
Date: Thu Jul 13 13:59:27 2017 -0400

    Don't load new copies of current resources

    The Stack._find_filtered_resources() method returns Resource objects for
    all resources associated with the Stack, regardless of whether they are
    current (present in the template; latest version in the case of
    convergence). To do this, it previously created a new Resource object
    for every resource found in the database.

    However, for those resources which *are* current this is unnecessary. We
    can access the Resource object simply through self.resources. It turns
    out this is necessary for obtaining the required_by list for legacy
    stacks, because only the Resources obtained from self.resources also
    appear in the Dependencies graph obtained from self.dependencies. The
    required_by list is read when listing or showing resources, which would
    either return an empty list or fail for legacy stacks.

    This patch also makes the Resource.required_by() method more robust in
    its error handling.

     Conflicts:
     heat/engine/stack.py

    The backport is simpler than the patch on master because
    3967a93f5de2e147a6d9c3f4fef9fd1a8f5d3498 has not been backported to
    stable/newton.

    Change-Id: Id438336e5c88dc7c2d168ba01ee703faa17e8b8e
    Closes-Bug: #1703703
    Related-Bug: #1523748
    (cherry picked from commit 5ce238fb17e38702ceef85a3706e9d2236c43e0e
                           and 65630d4e8bbff2f3ae5c659f8e9396ec7a6f6ae9)