Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.

[James Slagle]

I'm testing out the docker compute patch:
https://review.openstack.org/#/c/209505

My deployment has failed twice now with the error:
2015-10-08 17:48:14.498 2516 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud): Resource CREATE failed: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.

I think this is actually most likely not related to the Docker compute patch at all.

My undercloud is virt, but it has 2 cores with 2 heat-engine processes running.

Here's a grep of the nested stack name from the Heat Engine log:
2015-10-08 17:46:43.885 2516 INFO heat.engine.service [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Creating stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk
2015-10-08 17:46:44.297 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] starting start /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:186
2015-10-08 17:46:44.298 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:214
2015-10-08 17:46:44.411 2516 INFO heat.engine.stack [-] Stack CREATE IN_PROGRESS (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk): Stack CREATE started
2015-10-08 17:46:44.574 2516 INFO heat.engine.stack [-] Stack CREATE COMPLETE (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk): Stack CREATE completed successfully
2015-10-08 17:46:45.998 2516 INFO heat.engine.service [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Updating stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk
2015-10-08 17:46:46.119 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:220
2015-10-08 17:46:46.447 2516 DEBUG oslo_messaging.rpc.dispatcher [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Expected exception during message handling (Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.) _dispatch_and_reply /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145
2015-10-08 17:46:46.533 2515 ERROR heat.engine.resources.stack_resource ActionInProgress_Remote: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:46.533 2515 ERROR heat.engine.resources.stack_resource ActionInProgress: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:46.597 2515 ERROR heat.engine.resource ResourceFailure: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:47.793 2515 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5): Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:48.965 2516 ERROR heat.engine.resource ResourceFailure: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:48:14.498 2516 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud): Resource CREATE failed: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.

Note that the pids in the log (2515 and 2516) correspond to the 2 heat-engine processes:
[root@instack-centos7 heat]# ps axjf | grep heat-engine
28930 4354 4353 11626 pts/1 4353 S+ 0 0:00 | \_ grep --color=auto heat-engine
    1 2496 2496 2496 ? -1 Ss 187 1:55 /usr/bin/python /usr/bin/heat-engine
 2496 2515 2496 2496 ? -1 S 187 7:37 \_ /usr/bin/python /usr/bin/heat-engine
 2496 2516 2496 2496 ? -1 S 187 9:44 \_ /usr/bin/python /usr/bin/heat-engine

Is this a case where the nested stack creation got farmed out twice to 2 different heat-engine processes, and then they collided?

[James Slagle (james-slagle)]

attached heat debug logs

[Rabi Mishra (rabi)]

It seems like something we fixed recently[1]. Are you using latest master? It's backported to liberty rc2 as well.

[1] https://github.com/openstack/heat/commit/8e8b338f98158526737153f6ced11efb35a19afc

[Steven Hardy (shardy)]

Marking heat part incomplete as we believe this is an error from an older version, James feel free to respond if that's not correct :)

[Rakesh H S (rh-s)]

Yes, this issue was fixed, but however, this issue was again hit in liberty gate today.
http://logs.openstack.org/73/287173/1/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv1/7b0b4c1/console.html

http://paste.openstack.org/show/489128/

[James Slagle (james-slagle)]

You might consider filing a new bug as it's hard to say if these 2 issues are actually related. In TripleO's case, the issue was fixed by the previously mentioned patch from comment 4.