Upgrade from puppet to containers fails with a circular dependency issue. It's likely that we've regressed on this with some recent t-h-t patch.
http://logs.openstack.org/84/450784/19/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4f742f9/logs/undercloud/var/log/mistral/executor.log.txt.gz#_2017-04-25_12_53_50_805
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor [-] Failed to run action [action_ex_id=ce02f122-69c8-4b4b-b625-e3fc29ad62a7, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 80}']
ERROR: Failed to validate: : resources.AllNodesDeploySteps: : Failed to validate: resources.AllNodesPostUpgradeSteps: Circular Dependency Found: {NoneResource "ControllerPostConfig": {TemplateResource "ControllerExtraConfigPost"}, TemplateResource "BlockStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "CephStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "ControllerExtraConfigPost": {NoneResource "ControllerPostConfig"}}
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor Traceback (most recent call last):
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/mistral/engine/default_executor.py", line 89, in run_action
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor result = action.run()
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/actions/deployment.py", line 194, in run
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor return heat.stacks.update(stack.id, **stack_args)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 181, in update
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor headers=headers)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 298, in patch
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor return self.client_request("PATCH", url, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 279, in client_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp, body = self.json_request(method, url, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 268, in json_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp = self._http_request(url, method, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 240, in _http_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp = self._http_request(location, method, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 231, in _http_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor raise exc.from_response(resp)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor HTTPBadRequest: ERROR: Failed to validate: : resources.AllNodesDeploySteps: : Failed to validate: resources.AllNodesPostUpgradeSteps: Circular Dependency Found: {NoneResource "ControllerPostConfig": {TemplateResource "ControllerExtraConfigPost"}, TemplateResource "BlockStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "CephStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "ControllerExtraConfigPost": {NoneResource "ControllerPostConfig"}}
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor
I can confirm this error from local testing, and that this changed caused it:
https:/ /review. openstack. org/#/c/ 452734/
Looking at how to fix, if it's not obvious I'll revert while we figure it out.
As a side note, we failed to check experimental that patch, even though it touched docker-steps.j2, so we really, really need container coverage gating ASAP, and the undercloud isn't enough as this issue doesn't reproduce unless you have more than one role enabled.