I'm testing out the docker compute patch:
https://review.openstack.org/#/c/209505
My deployment has failed twice now with the error:
2015-10-08 17:48:14.498 2516 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud): Resource CREATE failed: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
I think this is actually most likely not related to the Docker compute patch at all.
My undercloud is virt, but it has 2 cores with 2 heat-engine processes running.
Here's a grep of the nested stack name from the Heat Engine log:
2015-10-08 17:46:43.885 2516 INFO heat.engine.service [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Creating stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk
2015-10-08 17:46:44.297 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] starting start /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:186
2015-10-08 17:46:44.298 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:214
2015-10-08 17:46:44.411 2516 INFO heat.engine.stack [-] Stack CREATE IN_PROGRESS (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk): Stack CREATE started
2015-10-08 17:46:44.574 2516 INFO heat.engine.stack [-] Stack CREATE COMPLETE (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk): Stack CREATE completed successfully
2015-10-08 17:46:45.998 2516 INFO heat.engine.service [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Updating stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk
2015-10-08 17:46:46.119 2516 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk" [33a7505a-d304-4ec3-b485-3f0dad26a82a] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:220
2015-10-08 17:46:46.447 2516 DEBUG oslo_messaging.rpc.dispatcher [req-77c077e4-b6eb-4d2f-9b12-26317e82da3a 62d7db4e9ac142e6a4182f6655167309 b3b900df99b94cddb48b97ef153a24f4] Expected exception during message handling (Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.) _dispatch_and_reply /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145
2015-10-08 17:46:46.533 2515 ERROR heat.engine.resources.stack_resource ActionInProgress_Remote: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:46.533 2515 ERROR heat.engine.resources.stack_resource ActionInProgress: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:46.597 2515 ERROR heat.engine.resource ResourceFailure: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:47.793 2515 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5): Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:46:48.965 2516 ERROR heat.engine.resource ResourceFailure: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
2015-10-08 17:48:14.498 2516 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud): Resource CREATE failed: resources.ComputeNodesPostDeployment: Resource CREATE failed: resources.NovaComputeContainersDeployment: Stack overcloud-ComputeNodesPostDeployment-dkq4fgg4lqk5-NovaComputeContainersDeployment-6zpj2rijh3lk already has an action (CREATE) in progress.
Note that the pids in the log (2515 and 2516) correspond to the 2 heat-engine processes:
[root@instack-centos7 heat]# ps axjf | grep heat-engine
28930 4354 4353 11626 pts/1 4353 S+ 0 0:00 | \_ grep --color=auto heat-engine
1 2496 2496 2496 ? -1 Ss 187 1:55 /usr/bin/python /usr/bin/heat-engine
2496 2515 2496 2496 ? -1 S 187 7:37 \_ /usr/bin/python /usr/bin/heat-engine
2496 2516 2496 2496 ? -1 S 187 9:44 \_ /usr/bin/python /usr/bin/heat-engine
Is this a case where the nested stack creation got farmed out twice to 2 different heat-engine processes, and then they collided?
attached heat debug logs