Heat stucks in DELETE_IN_PROGRESS for some input data
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Fix Released
|
High
|
Steve Baker | ||
Liberty |
Fix Committed
|
High
|
Zane Bitter | ||
Mitaka |
Fix Committed
|
High
|
Zane Bitter |
Bug Description
Steps to reproduce:
rg.yaml:
heat_template_
resources:
rg:
type: OS::Heat:
properties:
count: 125
rand_str.yaml
heat_template_
resources:
(yep, without any resource:) )
Run:
heat stack-create abc --template-file rg.yaml
wait about 20-30 sec and run:
heat stack-delete abc
(before stack will be in CREATE_COMPLETE)
Heat stucks in DELETE_IN_PROGRESS
Found this in logs:
2015-09-25 12:59:15.450 ERROR heat.engine.
or
2015-09-25 13:04:52.109 ERROR heat.engine.
2015-09-25 13:04:52.110 ERROR sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
or:
File "/usr/local/
cb(*args, **kw)
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
return func(*args, **kwargs)
File "/usr/local/
return f(*args, **kwargs)
File "/opt/stack/
self.
File "/usr/local/
return f(*args, **kwargs)
File "/opt/stack/
stack = stack_object.
File "/opt/stack/
db_stack = db_api.
File "/opt/stack/
eager_
File "/opt/stack/
result = query.get(stack_id)
File "/usr/local/
return self._get_
File "/usr/local/
return fallback_fn(self, key)
File "/usr/local/
return q.one()
File "/usr/local/
ret = list(self)
File "/usr/local/
return self._execute_
File "/usr/local/
close_
File "/usr/local/
**kw)
File "/usr/local/
execution_
File "/usr/local/
conn = engine.
File "/usr/local/
**kwargs)
File "/usr/local/
self.
File "/usr/local/
fn(*args, **kw)
File "/usr/local/
connection.
File "/usr/local/
return self.execute(
File "/usr/local/
row = self.first()
File "/usr/local/
return self._non_
File "/usr/local/
"This result object does not return rows. "
ResourceClosedE
All other time heat spamming into log something about:
c89]: {}}) running from (pid=32479) step /opt/stack/
2015-09-25 13:13:32.333 DEBUG heat.engine.
2015-09-25 13:13:32.359 INFO heat.engine.
2015-09-25 13:13:32.360 DEBUG heat.engine.
Not 100% reproducible, but you could experiment with different time intervals 20-30-40-50 secs.
Run heat stack-delete abc again, and stack will be deleted as usual.
description: | updated |
Changed in heat: | |
status: | New → Triaged |
importance: | Undecided → Medium |
milestone: | none → mitaka-1 |
Changed in heat: | |
milestone: | mitaka-1 → mitaka-2 |
Changed in heat: | |
assignee: | nobody → zhaozhilong (zhaozhilong) |
Changed in heat: | |
milestone: | mitaka-2 → mitaka-3 |
Changed in heat: | |
assignee: | zhaozhilong (zhaozhilong) → nobody |
Changed in heat: | |
milestone: | mitaka-3 → mitaka-rc1 |
Changed in heat: | |
milestone: | mitaka-rc1 → newton-1 |
Changed in heat: | |
milestone: | newton-1 → ongoing |
Changed in heat: | |
assignee: | nobody → Jason Dunsmore (jasondunsmore) |
status: | Triaged → In Progress |
Changed in heat: | |
importance: | Medium → High |
milestone: | ongoing → newton-rc1 |
tags: | added: gate-failure |
Changed in heat: | |
milestone: | newton-rc1 → nexton-rc2 |
no longer affects: | heat/newton |
Looks like we are failing to catch a DB exception correctly. We should get NotFound and recognise that as the stack having been deleted.
The "spamming" in the log is the normal polling of the status of the nested stack.