We are seeing a large number of gate-tripleo-ironic-overcloud-f21puppet-nonha fail due to the overcloud stack creation timing out. The console.log failures looks like this:
2015-09-03 14:13:18.033 | Waiting for the overcloud stack to be ready
2015-09-03 14:13:18.033 | + wait_for_stack_ready -w 2100 10 overcloud
2015-09-03 14:48:19.163 | Timing out after 2100 seconds:
If you look at the heat resource-list output most of the resources are still IN_PROGRESS so clearly a lot of things are finishing in the stack.
Looking a bit more closely it appears that almost all of the jobs I've looked at today appear to be processing puppet for the compute role. The last thing I see in the compute nodes os-collect-config.log file is :
Sep 03 18:02:53 overcloud-controller-0 os-collect-config[942]: [2015-09-03 18:02:53,031] (heat-config) [DEBUG] Running /var/lib/heat-config/hooks/puppet < /var/run/heat-config/deployed/68ee6f7f-9d00-4067-a9cd-646992cfe0d1.json
----
Another thought: This could be a valid timeout issue... (as in our jobs are just taking longer). 35 minutes is a long time for a 2 node overcloud stack to be created though.
One thing I'm trying to do is get more data about where exactly the compute nodes are in their puppet apply. Trying Martin's patch here might give us that information:
https:/
Perhaps we should cherry-pick this into tripleo-ci now?
/me testing this locally