ActionInProgress_Remote: Stack TemplateResourceUpdateFailedTest already has an action (CREATE) in progress.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Fix Released
|
High
|
Steve Baker |
Bug Description
Updating nested stack with parent in 'CREATE_FAILED' state fails at times with "ActionInProgre
This same codepath is also used by RG for create (StackResource.
Below are the gate failures in the last 7 days.
I did notice this error locally as well, with following backtrace.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.340 64860 ERROR heat.engine.
2015-09-22 08:23:23.403 64859 INFO heat.engine.
2015-09-22 08:23:23.401 64860 INFO heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
2015-09-22 08:23:23.401 64860 ERROR heat.engine.
Changed in heat: | |
assignee: | nobody → Rabi Mishra (rabi) |
summary: |
- ActionInProgress_Remote: Stack - TemplateResourceUpdateFailedTest-1535330590-server-7pudt7u7gs7q already + ActionInProgress_Remote: Stack TemplateResourceUpdateFailedTest already has an action (CREATE) in progress. |
Changed in heat: | |
status: | New → Triaged |
importance: | Undecided → High |
tags: | added: liberty-rc-potential |
Changed in heat: | |
milestone: | none → mitaka-1 |
Changed in heat: | |
status: | Triaged → In Progress |
no longer affects: | heat/mitaka |
no longer affects: | heat/liberty |
Changed in heat: | |
status: | In Progress → Fix Committed |
Changed in heat: | |
status: | Fix Committed → Fix Released |
Changed in heat: | |
milestone: | liberty-rc2 → 5.0.0 |
tags: |
added: kilo-backport-potential removed: liberty-rc-potential |
From the initial investigation, it looks like an issue of stack locking. As we are creating an empty stack and then updating it immediately, it seems the nested stack lock is not released by the thread doing 'stack_create' before 'stack_update' starts.
As we're checking for stack create_complete by checking the db[1], it is possible that the create_stack thread may not finish when the check detects that the the status(COMPLETE) is updated in db.
I've pushed a workaround for this issue[2]. Working on a more elegant solution which would be difficult to implement though:)
[1] https:/ /github. com/openstack/ heat/blob/ master/ heat/engine/ resources/ stack_resource. py#L335 /review. openstack. org/#/c/ 227156/
[2] https:/