This is very similar to [1]. We're running Ocata on openSUSE Leap 42.3, with heat version:
openstack-heat-8.0.5~dev5-1.2.noarch
We were trying to launch a stack where volumes are created from images, then they are supposed to be attached to instances. After the stack failed, we tried to delete it, but it's stuck with status "Delete_In_Progress". All resources have been deleted successfully according to the resource list.
---cut here---
control:~ # openstack stack list
+--------------------------------------+------------+--------------------+----------------------+
| ID | Stack Name | Stack Status | Creation Time |
+--------------------------------------+------------+--------------------+----------------------+
| c687176c-94c7-4d8f-ba18-dc357e257226 | stack1 | DELETE_IN_PROGRESS | 2018-04-19T15:17:06Z |
+--------------------------------------+------------+--------------------+----------------------+
The stack trace says the deleting process canceled during check_resource:
---cut here---
2018-04-20 09:27:38.651 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Stack DELETE FAILED (xxxx201): Stack DELETE cancelled
2018-04-20 09:27:38.764 24724 INFO heat.engine.worker [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] [xxxx201(c687176c-94c7-4d8f-ba18-dc357e257226)] Stopped all active workers for stack DELETE
2018-04-20 09:27:39.059 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Stack DELETE IN_PROGRESS (xxxx201): Stack DELETE started
2018-04-20 09:27:39.128 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] convergence_dependencies: (Dependencies([((16, False), None)]),)
2018-04-20 09:27:39.246 24724 INFO heat.engine.stack [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Triggering resource 16 for cleanup
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker [req-b5ace8fc-0d11-476b-8f99-ed2c44088f0f - ADMIN - - -] Unhandled exception in check_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker Traceback (most recent call last):
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 47, in exception_wrapper
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return func(*args, **kwargs)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 177, in check_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker cnxt, resource_id, resource_data, is_update)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 309, in load_resource
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker is_update, cache_data)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 338, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker stack=db_stack)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 526, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker load_template=load_template)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 564, in _from_db
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker context, stack.raw_template_id, stack.raw_template)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/template.py", line 156, in load
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker t = template_object.RawTemplate.get_by_id(context, template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/objects/raw_template.py", line 87, in get_by_id
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker raw_template_db = db_api.raw_template_get(context, template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 117, in raw_template_get
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker result = context.session.query(models.RawTemplate).get(template_id)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 831, in get
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return self._get_impl(ident, loading.load_on_ident)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 842, in _get_impl
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker if len(ident) != len(mapper.primary_key):
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker TypeError: object of type 'NoneType' has no len()
---cut here---
Expected result:
If all resources have been deleted successfully, the stack itself should be deleted, too.
Actual result:
The stack seems to check for resources that aren't available anymore (deleted successfully), and gets stuck in "delete_in_progress" state.
This is very similar to [1]. We're running Ocata on openSUSE Leap 42.3, with heat version:
openstack- heat-8. 0.5~dev5- 1.2.noarch
We were trying to launch a stack where volumes are created from images, then they are supposed to be attached to instances. After the stack failed, we tried to delete it, but it's stuck with status "Delete_ In_Progress" . All resources have been deleted successfully according to the resource list.
---cut here--- ------- ------- ------- ------- ----+-- ------- ---+--- ------- ------- ---+--- ------- ------- -----+ ------- ------- ------- ------- ----+-- ------- ---+--- ------- ------- ---+--- ------- ------- -----+ 94c7-4d8f- ba18-dc357e2572 26 | stack1 | DELETE_IN_PROGRESS | 2018-04- 19T15:17: 06Z | ------- ------- ------- ------- ----+-- ------- ---+--- ------- ------- ---+--- ------- ------- -----+
control:~ # openstack stack list
+------
| ID | Stack Name | Stack Status | Creation Time |
+------
| c687176c-
+------
control:~ # openstack stack resource list c687176c- 94c7-4d8f- ba18-dc357e2572 26 ------- ------- -+----- ------- ---+--- ------- ------- ------- ------+ ------- ------- ---+ ------- ------- -+----- ------- ---+--- ------- ------- ------- ------+ ------- ------- ---+ :RouterInterfac e | DELETE_COMPLETE | ------- ------- -+----- ------- ---+--- ------- ------- ------- ------+ ------- ------- ---+
+------
| resource_name | phys_resource | resource_type | resource_status |
+------
| xxxx201_router | e14ada6c-f76c | OS::Neutron::Router | DELETE_COMPLETE |
| xxxx201_router_mgr | e14ada6c-f76c | OS::Neutron:
| xxxx201-vda | 85c801a5-abdf | OS::Cinder::Volume | DELETE_COMPLETE |
| xxx_subnet | 5c456eda-a99d | OS::Neutron::Subnet | DELETE_COMPLETE |
| xxxx201-admin | dda3b9c6-a535 | OS::Nova::Server | DELETE_COMPLETE |
| xxxx_floating_port | 47d35e66-c064 | OS::Neutron::Port | DELETE_COMPLETE |
| xxxx201-vdb | e7c5377c-11f2 | OS::Cinder::Volume | DELETE_COMPLETE |
| xxxx201-vdc | fa3b793a-8276 | OS::Cinder::Volume | DELETE_COMPLETE |
| xxxx201-vda | beab6790-8591 | OS::Cinder::Volume | DELETE_COMPLETE |
| mgr_net | 55e17138-9a7b | OS::Neutron::Net | DELETE_COMPLETE |
+------
---cut here---
The stack trace says the deleting process canceled during check_resource:
---cut here--- 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] Stack DELETE FAILED (xxxx201): Stack DELETE cancelled 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] [xxxx201( c687176c- 94c7-4d8f- ba18-dc357e2572 26)] Stopped all active workers for stack DELETE 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] Stack DELETE IN_PROGRESS (xxxx201): Stack DELETE started 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] convergence_ dependencies: (Dependencies( [((16, False), None)]),) 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] Triggering resource 16 for cleanup 0d11-476b- 8f99-ed2c44088f 0f - ADMIN - - -] Unhandled exception in check_resource python2. 7/site- packages/ heat/engine/ worker. py", line 47, in exception_wrapper python2. 7/site- packages/ heat/engine/ worker. py", line 177, in check_resource python2. 7/site- packages/ heat/engine/ check_resource. py", line 309, in load_resource python2. 7/site- packages/ heat/engine/ resource. py", line 338, in load python2. 7/site- packages/ heat/engine/ stack.py" , line 526, in load load_template) python2. 7/site- packages/ heat/engine/ stack.py" , line 564, in _from_db template_ id, stack.raw_template) python2. 7/site- packages/ heat/engine/ template. py", line 156, in load object. RawTemplate. get_by_ id(context, template_id) python2. 7/site- packages/ heat/objects/ raw_template. py", line 87, in get_by_id raw_template_ get(context, template_id) python2. 7/site- packages/ heat/db/ sqlalchemy/ api.py" , line 117, in raw_template_get session. query(models. RawTemplate) .get(template_ id) python2. 7/site- packages/ sqlalchemy/ orm/query. py", line 831, in get impl(ident, loading. load_on_ ident) python2. 7/site- packages/ sqlalchemy/ orm/query. py", line 842, in _get_impl primary_ key):
2018-04-20 09:27:38.651 24724 INFO heat.engine.stack [req-b5ace8fc-
2018-04-20 09:27:38.764 24724 INFO heat.engine.worker [req-b5ace8fc-
2018-04-20 09:27:39.059 24724 INFO heat.engine.stack [req-b5ace8fc-
2018-04-20 09:27:39.128 24724 INFO heat.engine.stack [req-b5ace8fc-
2018-04-20 09:27:39.246 24724 INFO heat.engine.stack [req-b5ace8fc-
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker [req-b5ace8fc-
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker Traceback (most recent call last):
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return func(*args, **kwargs)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker cnxt, resource_id, resource_data, is_update)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker is_update, cache_data)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker stack=db_stack)
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker load_template=
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker context, stack.raw_
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker t = template_
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker raw_template_db = db_api.
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker result = context.
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker return self._get_
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker File "/usr/lib64/
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker if len(ident) != len(mapper.
2018-04-20 09:27:39.284 24718 ERROR heat.engine.worker TypeError: object of type 'NoneType' has no len()
---cut here---
Expected result:
If all resources have been deleted successfully, the stack itself should be deleted, too.
Actual result: in_progress" state.
The stack seems to check for resources that aren't available anymore (deleted successfully), and gets stuck in "delete_
[1] https:/ /bugs.launchpad .net/heat/ +bug/1163067