upgrade from puppet to containers - circular dependency

[Jiří Stránský]

Upgrade from puppet to containers fails with a circular dependency issue. It's likely that we've regressed on this with some recent t-h-t patch.

http://logs.openstack.org/84/450784/19/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4f742f9/logs/undercloud/var/log/mistral/executor.log.txt.gz#_2017-04-25_12_53_50_805

2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor [-] Failed to run action [action_ex_id=ce02f122-69c8-4b4b-b625-e3fc29ad62a7, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 80}']
 ERROR: Failed to validate: : resources.AllNodesDeploySteps: : Failed to validate: resources.AllNodesPostUpgradeSteps: Circular Dependency Found: {NoneResource "ControllerPostConfig": {TemplateResource "ControllerExtraConfigPost"}, TemplateResource "BlockStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "CephStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "ControllerExtraConfigPost": {NoneResource "ControllerPostConfig"}}
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor Traceback (most recent call last):
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/mistral/engine/default_executor.py", line 89, in run_action
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor result = action.run()
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/actions/deployment.py", line 194, in run
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor return heat.stacks.update(stack.id, **stack_args)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 181, in update
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor headers=headers)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 298, in patch
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor return self.client_request("PATCH", url, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 279, in client_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp, body = self.json_request(method, url, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 268, in json_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp = self._http_request(url, method, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 240, in _http_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor resp = self._http_request(location, method, **kwargs)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 231, in _http_request
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor raise exc.from_response(resp)
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor HTTPBadRequest: ERROR: Failed to validate: : resources.AllNodesDeploySteps: : Failed to validate: resources.AllNodesPostUpgradeSteps: Circular Dependency Found: {NoneResource "ControllerPostConfig": {TemplateResource "ControllerExtraConfigPost"}, TemplateResource "BlockStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "CephStorageExtraConfigPost": {NoneResource "ControllerPostConfig"}, TemplateResource "ControllerExtraConfigPost": {NoneResource "ControllerPostConfig"}}
2017-04-25 12:53:50.805 12728 ERROR mistral.engine.default_executor

[Steven Hardy (shardy)]

I can confirm this error from local testing, and that this changed caused it:

https://review.openstack.org/#/c/452734/

Looking at how to fix, if it's not obvious I'll revert while we figure it out.

As a side note, we failed to check experimental that patch, even though it touched docker-steps.j2, so we really, really need container coverage gating ASAP, and the undercloud isn't enough as this issue doesn't reproduce unless you have more than one role enabled.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/459751

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459751
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ba44524d4603b9f53a06e44af80b46a24661bdb1
Submitter: Jenkins
Branch: master

commit ba44524d4603b9f53a06e44af80b46a24661bdb1
Author: Steven Hardy <email address hidden>
Date: Tue Apr 25 15:13:38 2017 +0000

    Fix dependencies for PreConfig/PostConfig resources

    We have a circular dependency errror since

    https://review.openstack.org/#/c/452734/ landed.

    This adjusts the dependencies to ensure we run pre-config before
    the first puppet deploy step, and removes the duplicate declaration
    of the ControllerPostConfig resource. Also we ensure the first
    container step always depends on the same step puppet deploy.

    Change-Id: I70c5a39fb36b951bdeb04c15bddac7d00eebf08a
    Closes-Bug: #1686098

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.