Overcloud heat stack timing out

[Derek Higgins]

We don't have any overcloud logs as they arn't being retrieved from nodes (probably due to another bug) so this is all I have

2015-08-18 09:00:21.560 | + echo 'Waiting for the overcloud stack to be ready'
2015-08-18 09:00:21.560 | Waiting for the overcloud stack to be ready
2015-08-18 09:00:21.560 | + wait_for_stack_ready -w 2100 10 overcloud
2015-08-18 09:35:21.488 | Timing out after 2100 seconds:
2015-08-18 09:35:21.488 | COMMAND=heat stack-show overcloud | awk '/stack_status / { print $4 }'
2015-08-18 09:35:21.488 | OUTPUT=CREATE_IN_PROGRESS
2015-08-18 09:35:21.489 | + cleanup

[Derek Higgins (derekh)]

It looks like this commit is where the problem was introduced
https://review.openstack.org/#/c/209000/
"Refactor format_resource_attributes"

Here is a tracback I'm seeing that may be relevant (when trying to download the software config data from a overcloud instance)

2015-08-21 13:41:23,912 ERROR Error marking resource as failed
 Traceback (most recent call last):
   File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/heat/engine/resource.py", line 646, in _action_recorder
     self.state_set(action, self.FAILED, '%s aborted' % action)
   File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/heat/engine/resource.py", line 1429, in state_set
     self._store_or_update(action, status, reason)
   File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/heat/engine/resource.py", line 1277, in _store_or_update
     'stack_id': self.stack.id,
   File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/heat/engine/resource.py", line 276, in stack
     assert stack is not None, "Need a reference to the Stack object"
 AssertionError: Need a reference to the Stack object
Aug 21 13:41:24 localhost.localdomain.localdomain sudo[897]: pam_unix(sudo:session): session closed for user root

[Steve Baker (steve-stevebaker)]

Setting to Critical for Heat, just to inherit from tripleo

[Rabi Mishra (rabi)]

From the pasted log it looks like it's trying to set the state to FAILED but can't do it as self.stack is None.

I could not locate any issue with the commit you mentioned. Are there any other stuff that you can see in the log related to sc/sd?

[Rabi Mishra (rabi)]

As this is set to critical and I'm still to find the root cause, I 'm releasing it for someone else with more background on the recent changes to have a go.

[Thomas Herve (therve)]

If I had to guess, I'd say the error is happening before the one you pasted. That one may be another bug, or simply a consequence. There may be more information above. Do you have a check that we can use otherwise? I'm looking at the patch you linked in the mean time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/216150

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/216150
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=af28efd89428749a9f8b27ae4a51f56c4ab5c7d6
Submitter: Jenkins
Branch: master

commit af28efd89428749a9f8b27ae4a51f56c4ab5c7d6
Author: Thomas Herve <email address hidden>
Date: Mon Aug 24 08:37:51 2015 +0000

    Revert "Refactor format_resource_attributes"

    This reverts commit 4d370367a8e3dd3df2f95009760821312f09229d.
    It introduces backward compatibility issues with resources using dynamic attributes.

    Change-Id: Id7906b52d168fb2d9f744fa7a8d06c84a1343186
    Closes-Bug: #1485956

[Derek Higgins (derekh)]

Commit reverted in heat