Overcloud noop stack update fails with no connection to Undercloud endpoint

Bug #1709132 reported by Justin Kilpatrick on 2017-08-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Unassigned

Bug Description

To reproduce:

deploy trivial overcloud

    openstack overcloud deploy --templates

Attempt to perform a stakck update with no changes

    openstack overcloud deploy --templates

Expected results:

    stack update completes successfully pretty quickly

Actual results:

    operation sometimes succeeded, most of the time an endpoint times out and the update fails

This is probably related to #1635269, where slow environments could cause deployment failures since the testing machine is fairly slow with no ssd's. Observed on Ocata and Master using oooq

GET call to orchestration for http://10.9.147.1:8004/v1/f66de0651725431caa2e59580e725c75/stacks/overcloud/345413c8-c50d-42a4-b6ca-0cde8a5c2216/events?marker=c9480ddd-09aa-4375-ac2b-8800843cddec&nested_depth=2&sort_dir=asc used request id req-99645610-8d54-4a16-aa64-a1ee9eb43d37
Heat Stack update failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1171, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 392, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
    parsed_args.run_validations, parsed_args.skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 471, in _try_overcloud_deploy_with_compat_yaml
    run_validations, skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 251, in _heat_deploy
    skip_deploy_identifier=skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 88, in deploy_and_wait
    raise exceptions.DeploymentError("Heat Stack update failed.")
DeploymentError: Heat Stack update failed.
clean_up DeployOvercloud: Heat Stack update failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1171, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 392, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
    parsed_args.run_validations, parsed_args.skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 471, in _try_overcloud_deploy_with_compat_yaml
    run_validations, skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 251, in _heat_deploy
    skip_deploy_identifier=skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 88, in deploy_and_wait
    raise exceptions.DeploymentError("Heat Stack update failed.")
DeploymentError: Heat Stack update failed.

END return value: 1
eihh.NodeAdminUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:53Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.RoleUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:54Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeUserData]: UPDATE_IN_PROGRESS state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeAdminUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.RoleUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.UpdateConfig]: UPDATE_COMPLETE state changed
2017-08-07 17:07:55Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh.NodeUserData]: UPDATE_COMPLETE state changed
2017-08-07 17:08:02Z [overcloud-Controller-lxynan7p4ky3-0-wpesxtxleihh]: UPDATE_FAILED Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:02Z [overcloud-Compute-7dp7pvgzvfkq-0-nvrnm4tmze3h]: UPDATE_FAILED Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:03Z [overcloud-Controller-lxynan7p4ky3.0]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:03Z [overcloud-Compute-7dp7pvgzvfkq.0]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [overcloud-Controller-lxynan7p4ky3]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [overcloud-Compute-7dp7pvgzvfkq]: UPDATE_FAILED resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:04Z [Controller]: UPDATE_FAILED resources.Controller: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:05Z [Compute]: UPDATE_FAILED resources.Compute: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))
2017-08-07 17:08:06Z [overcloud]: UPDATE_FAILED resources.Controller: resources[0]: Unable to establish connection to http://10.9.147.1:9696/v2.0/networks.json?fields=id&name=ctlplane: ('Connection aborted.', BadStatusLine("''",))

 Stack overcloud UPDATE_FAILED

full log here https://paste.fedoraproject.org/paste/ogTMaTigJ0MN-VvovwF2rA

Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3

No longer an issue.

Changed in tripleo:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers