tripleo

openstack overcloud upgrade prepare fails to update heat templates

Bug #1834881 reported by Sergii Golovatiuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sergii Golovatiuk
tripleo train-2

Bug Description

Description of problem:

How reproducible:

Always.

Steps to Reproduce:
1. Deploy OSP14
2. Upgrade Undercloud to 15.
3. Upgrade Upgrade of Overcloud

Actual results:

"openstack overcloud upgrade prepare" fails with the following output

2019-07-01 14:17:48Z [overcloud]: UPDATE_FAILED The Referenced Attribute (Controller deployed_server_deployment_swift_data_map) is incorrect.
2019-07-01 14:17:48Z [overcloud.ControllerAllNodesValidationConfig]: UPDATE_IN_PROGRESS state changed
2019-07-01 14:17:48Z [overcloud-hostsConfig-7jgatdsigqcv]: UPDATE_IN_PROGRESS Stack UPDATE started
2019-07-01 14:17:48Z [overcloud-SshKnownHostsConfig-ntvcmllmgqmv]: UPDATE_COMPLETE Stack UPDATE completed successfully
2019-07-01 14:17:48Z [overcloud-hostsConfig-7jgatdsigqcv]: UPDATE_COMPLETE Stack UPDATE completed successfully
2019-07-01 14:17:48Z [overcloud-ControllerIpListMap-tdm2fhrcy5e5]: UPDATE_IN_PROGRESS Stack UPDATE started
2019-07-01 14:17:49Z [overcloud-ControllerAllNodesValidationConfig-vxqw5fxzlbvc]: UPDATE_IN_PROGRESS Stack UPDATE started
2019-07-01 14:17:49Z [overcloud-ControllerAllNodesValidationConfig-vxqw5fxzlbvc]: UPDATE_COMPLETE Stack UPDATE completed successfully
2019-07-01 14:17:49Z [overcloud.SshKnownHostsConfig]: UPDATE_COMPLETE state changed
2019-07-01 14:17:49Z [overcloud.hostsConfig]: UPDATE_COMPLETE state changed
2019-07-01 14:17:49Z [overcloud-ControllerIpListMap-tdm2fhrcy5e5]: UPDATE_COMPLETE Stack UPDATE completed successfully
2019-07-01 14:17:50Z [overcloud.ControllerAllNodesValidationConfig]: UPDATE_COMPLETE state changed
2019-07-01 14:17:50Z [overcloud.ControllerIpListMap]: UPDATE_COMPLETE state changed

 Stack overcloud/7aaf24c0-5bad-4630-a963-5a4f21f9c290 UPDATE_FAILED

Expected results:

 Stack overcloud/7aaf24c0-5bad-4630-a963-5a4f21f9c290 UPDATE_COMPLETE

Success

Additional info:

Revert of https://review.opendev.org/#/c/664790/ helped.

Sergii Golovatiuk (sgolovatiuk)
Changed in tripleo:
importance: Undecided → Critical
assignee: nobody → Rabi Mishra (rabi)
status: New → Confirmed
milestone: none → train-2
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Rabi Mishra (rabi) → Sergii Golovatiuk (sgolovatiuk)
status: Confirmed → In Progress
Revision history for this message
Rabi Mishra (rabi) wrote :

I guess something is wrong with you upgrade precess and the upgrade prepare step? Why is the roles files changed but not overcloud.yaml? Was the overcloud in a FAILED state before upgrade?

Revision history for this message
Rabi Mishra (rabi) wrote :

Hmm.. It's possibly a heat bug. It's trying to resolve the attribute of an existing resource (which it should not and pick from resolved resource properties_data). I'm still trying to create simple reproducer.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/668482
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b5a2a61dfea167ed96b52098dbd5807b3fae44a6
Submitter: Zuul
Branch: master

commit b5a2a61dfea167ed96b52098dbd5807b3fae44a6
Author: Sergii Golovatiuk <email address hidden>
Date: Mon Jul 1 14:43:20 2019 +0000

    Revert "Remove os-collect-config related poperties and attributes"

    This reverts commit 89e9fe53b20ee87a6a7ebe41af48f7ab206ebf05.

    Closes-Bug: #1834881

    Change-Id: I131cbafc126bcc4dbea5d38e0ff7adc177d114bd

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Rabi Mishra (rabi) wrote :
Download full text (6.7 KiB)

tarceback from heat engine:

2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource [req-4419c4b1-ea30-4d5c-b24c-01c61ec32441 - admin - default default] Unexpected exception in resource check.: InvalidTemplateAttribute: The Referenced Attribute (Controller deployed_server_deployment_swift_data_map) is incorrect.
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource Traceback (most recent call last):
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 313, in check
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource adopt_stack_data)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 152, in _do_check_resource
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource stack, self.msg_queue)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 395, in check_resource_update
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource check_message)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1457, in update_convergence
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource runner(timeout=timeout, progress_callback=progress_callback)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 163, in __call__
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource progress_callback=progress_callback):
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 239, in as_task
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource self.start(timeout=timeout)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 185, in start
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource self.step()
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 212, in step
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource poll_period = next(self._runner)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 320, in wrapper
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource subtask = next(parent)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1601, in update
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource before = self.frozen_definition()
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 620, in frozen_definition
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource return self.t.freeze(**args)
2019-07-02 02:07:02.499 31 ERROR heat.engine.check_resource File "/usr/lib...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/668875

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/stein)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/668875
Reason: As Rabi said, we don't need that revert on stein.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.1.0

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 13.0.0.0rc1

This issue was fixed in the openstack/heat 13.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 11.0.3

This issue was fixed in the openstack/heat 11.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.