stack delete fails although all resources are deleted

[[e-mail]]

This is very similar to [1]. We're running Ocata on openSUSE Leap 42.3, with heat version:

openstack-heat-8.0.5~dev5-1.2.noarch

We were trying to launch a stack where volumes are created from images, then they are supposed to be attached to instances. After the stack failed, we tried to delete it, but it's stuck with status "Delete_In_Progress". All resources have been deleted successfully according to the resource list (I shortened the IDs for readability).

---cut here---
control:~ # openstack stack list
+---------------+------------+--------------------+----------------------+
| ID | Stack Name | Stack Status | Creation Time |
+---------------+------------+--------------------+----------------------+
| c687176c-94c7 | stack1 | DELETE_IN_PROGRESS | 2018-04-19T15:17:06Z |
+---------------+------------+--------------------+----------------------+

control:~ # openstack stack resource list c687176c-94c7
+---------------------+----------+----------------------+-----------------+
| resource_name | phys_res | resource_type | resource_status |
+---------------------+----------+----------------------+-----------------+
| xxxx201_router | e14ada6c | OS::Neutron::Router | DELETE_COMPLETE |
| xxxx201_router_mgr | e14ada6c | OS::Neutron::RouterIn| DELETE_COMPLETE |
| xxxx201-vda | 85c801a5 | OS::Cinder::Volume | DELETE_COMPLETE |
| xxx_subnet | 5c456eda | OS::Neutron::Subnet | DELETE_COMPLETE |
| xxxx201-admin | dda3b9c6 | OS::Nova::Server | DELETE_COMPLETE |
| xxxx_floating_port | 47d35e66 | OS::Neutron::Port | DELETE_COMPLETE |
| xxxx201-vdb | e7c5377c | OS::Cinder::Volume | DELETE_COMPLETE |
| xxxx201-vdc | fa3b793a | OS::Cinder::Volume | DELETE_COMPLETE |
| xxxx201-vda | beab6790 | OS::Cinder::Volume | DELETE_COMPLETE |
| mgr_net | 55e17138 | OS::Neutron::Net | DELETE_COMPLETE |
+---------------------+---------------+-----------------+-----------------+
---cut here---

The stack trace says the deleting process canceled during check_resource:

---cut here---
2018-04-20 09:27:38.651 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Stack DELETE FAILED (xxxx201): Stack DELETE cancelled
2018-04-20 09:27:38.764 24724 INFO heat.engine.worker [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] [xxxx201(c687176c-94c7-4d8f-ba18-dc357e257226)] Stopped all active workers for stack DELETE
2018-04-20 09:27:39.059 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Stack DELETE IN_PROGRESS (xxxx201): Stack DELETE started
2018-04-20 09:27:39.128 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] convergence_dependencies: (Dependencies([((16, False), None)]),)
2018-04-20 09:27:39.246 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Triggering resource 16 for cleanup
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Unhandled exception in check_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker Traceback (most recent call last):
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 47, in exception_wrapper
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return func(*args, **kwargs)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 177, in check_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker cnxt, resource_id, resource_data, is_update)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 309, in load_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker is_update, cache_data)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 338, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker stack=db_stack)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 526, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker load_template=load_template)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 564, in _from_db
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker context, stack.raw_template_id, stack.raw_template)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/template.py", line 156, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker t = template_object.RawTemplate.get_by_id(context, template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/objects/raw_template.py", line 87, in get_by_id
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker raw_template_db = db_api.raw_template_get(context, template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 117, in raw_template_get
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker result = context.session.query(models.RawTemplate).get(template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 831, in get
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return self._get_impl(ident, loading.load_on_ident)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 842, in _get_impl
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker if len(ident) != len(mapper.primary_key):
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker TypeError: object of type 'NoneType' has no len()
---cut here---

Expected result:
If all resources have been deleted successfully, the stack itself should be deleted, too.

Actual result:
The stack seems to check for resources that aren't available anymore (deleted successfully), and gets stuck in "delete_in_progress" state.

[1] https://bugs.launchpad.net/heat/+bug/1163067

[[e-mail] (eblock)]

Is there a kind of rule or guideline in which case to use what? I didn't know about StoryBoard before and for future reference I'd like to know when to use launchpad or StoryBoard.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/582448