To reproduce:
deploy trivial overcloud
openstack overcloud deploy --templates
Attempt to perform a stakck update with no changes
openstack overcloud deploy --templates
Expected results:
stack update completes successfully pretty quickly
Actual results:
operation sometimes succeeded, most of the time an endpoint times out and the update fails
This is probably related to #1635269, where slow environments could cause deployment failures since the testing machine is fairly slow with no ssd's. Observed on Ocata and Master using oooq
GET call to orchestration for http://10.9.147.1:8004/v1/f66de0651725431caa2e59580e725c75/stacks/overcloud/345413c8-c50d-42a4-b6ca-0cde8a5c2216/events?marker=c9480ddd-09aa-4375-ac2b-8800843cddec&nested_depth=2&sort_dir=asc used request id req-99645610-8d54-4a16-aa64-a1ee9eb43d37
Heat Stack update failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
return self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1171, in take_action
self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 392, in _deploy_tripleo_heat_templates_tmpdir
new_tht_root, tht_root)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
parsed_args.run_validations, parsed_args.skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 471, in _try_overcloud_deploy_with_compat_yaml
run_validations, skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 251, in _heat_deploy
skip_deploy_identifier=skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 88, in deploy_and_wait
raise exceptions.DeploymentError("Heat Stack update failed.")
DeploymentError: Heat Stack update failed.
clean_up DeployOvercloud: Heat Stack update failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
return self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1171, in take_action
self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 392, in _deploy_tripleo_heat_templates_tmpdir
new_tht_root, tht_root)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
parsed_args.run_validations, parsed_args.skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 471, in _try_overcloud_deploy_with_compat_yaml
run_validations, skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 251, in _heat_deploy
skip_deploy_identifier=skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 88, in deploy_and_wait
raise exceptions.DeploymentError("Heat Stack update failed.")
DeploymentError: Heat Stack update failed.
END return value: 1
eihh.NodeAdminUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:53Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.RoleUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:54Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeAdminUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.RoleUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.UpdateConfig]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:08:02Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh]: UPDATE_FAILED Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:02Z [overcloud-Compute-7dp7pvgzvfkq-0-nvrnm4tmze3h]: UPDATE_FAILED Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:03Z [overcloud-Controller-lxynan7p4ky3.0]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:03Z [overcloud-Compute-7dp7pvgzvfkq.0]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [overcloud-Controller-lxynan7p4ky3]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [overcloud-Compute-7dp7pvgzvfkq]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [Controller]: UPDATE_FAILED resources.Controller: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:05Z [Compute]: UPDATE_FAILED resources.Compute: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:06Z [overcloud]: UPDATE_FAILED resources.Controller: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
Stack overcloud UPDATE_FAILED
full log here https://paste.fedoraproject.org/paste/ogTMaTigJ0MN-VvovwF2rA
No longer an issue.