overcloud re-deployment without deleting plan beforehand fails

Bug #1819672 reported by Luca Miccini
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Luke Short

Bug Description

(undercloud) [stack@undercloud-0 ~]$ ./deploy.sh
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 29, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 908, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 365, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 390, in _deploy_tripleo_heat_templates
    type(self)._keep_env_on_update)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 204, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 290, in _load_passwords
    return plan_env['passwords']
KeyError: 'passwords'
'passwords'

Revision history for this message
Luca Miccini (lmiccini2) wrote :

env: host with Python 3

Changed in tripleo:
milestone: none → stein-3
importance: Undecided → High
status: New → Triaged
Revision history for this message
Alex Schultz (alex-schultz) wrote :

Yea this happens on py2 envs too. It's been doing this for a while but we haven't tracked down why. If you use 'openstack overcloud delete <stack>' it'll cleanup the plan as well as the stack

Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Revision history for this message
Radoslaw Smigielski (radoslaw-smigielski) wrote :
Download full text (8.1 KiB)

We also see this error. Our command is:

overcloud upgrade converge \
 --config-download-timeout 330 \
 --timeout 420 \
 --templates /home/stack/templates \
 -e /home/stack/templates/docker_registry.yaml \
 -e /home/stack/templates/environments/docker.yaml \
 -e /home/stack/templates/environments/docker-ha.yaml \
 -e /home/stack/templates/common-environment.yaml \
 -e /home/stack/templates/network-environment.yaml \
 -e /home/stack/templates/scale-info.yaml \
 -e /home/stack/templates/sriov-info.yaml \
 -e /home/stack/templates/dpdk-info.yaml \
 -e /home/stack/templates/environments/hyperconverged-ceph.yaml \
 -e /home/stack/templates/storage-environment.yaml \
 -e /home/stack/templates/environments/ceph-ansible/ceph-ansible.yaml \
 -e /home/stack/templates/environments/ceph-ansible/ceph-rgw.yaml \
 -e /home/stack/templates/environments/manila-cephfsnative-config-docker.yaml \
 -e /home/stack/templates/environments/ceph-ansible/ceph-mds.yaml \
 -e /home/stack/templates/environments/services/barbican.yaml \
 -e /home/stack/templates/environments/barbican-backend-simple-crypto.yaml \
 -e /home/stack/templates/enable-tls.yaml \
 -e /home/stack/templates/inject-trust-anchor.yaml \
 -e /home/stack/templates/tls-endpoints-public-ip.yaml \
 -e /home/stack/tripleo-overcloud-passwords.yaml \
 -e /home/stack/overcloud-passwords.yaml \
 --debug

(...)
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_v
ersion': '1', 'rca_api_version': '1', 'tacker_api_version': '1', 'cbisclient_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.31
.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': '1', 'verify': True, 'event_api_
version': '2', 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'ver
bose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project
_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u
'interface': None, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version':
u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'username': 'admin', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
2019-05-13 17:21:35.902 765851 INFO osc_lib.shell [-] command: overcloud upgrade converge -> tripleoclient.v1.overcloud_upgrade.UpgradeConvergeOvercloud (auth=True)...

Read more...

Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
Revision history for this message
Anton Antonov (anta-nok) wrote :
Download full text (5.0 KiB)

  I have the same problem while trying to upgrade from Rocky to Stein (openstack overcloud update prepare). Is there a workaround until the bug is fixed? It's being open since March...

  One of the solution I know is to run "openstack overcloud plan delete overcloud", which would help when overcloud is not deployed yet. But, the plan can't be deleted if there is a deployed overcloud and one wants to upgrade it. Can the plan be fixed somehow?

```
openstack overcloud update prepare --templates \
-e ~/vxrdo/templates/node-info.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-n ~/vxrdo/templates/network_data.yaml \
-e ~/vxrdo/templates/network-isolation.yaml \
-e ~/vxrdo/templates/scheduler_hints_env.yaml \
-e ~/vxrdo/templates/network-environment.yaml \
-e ~/vxrdo/templates/ips-from-pool-all.yaml \
-e ~/vxrdo/templates/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e ~/vxrdo/templates/firstboot/firstboot.yaml \
--ntp-server 10.35.10.2 \
-e ~/vxrdo/templates/init-repo.yaml \
-e ~/vxrdo/templates/containers-prepare-parameter.yaml \
 2>&1 | tee prepare.log
```

```
2019-09-06 19:33:08.847 306654 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201
2019-09-06 19:33:12.854 306654 INFO tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Stack found, will be doing a stack update
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Exception occured while running the command: KeyError: 'passwords'
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare Traceback (most recent call last):
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare super(Command, self).run(parsed_args)
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare return super(Command, self).run(parsed_args)
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare return_code = self.take_action(parsed_args) or 0
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_update.py", line 73, in take_action
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare super(UpdatePrepare, self).take_action(parsed_args)
2019-09-06 19:33:17.205 306654 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare File "/usr/...

Read more...

Changed in tripleo:
milestone: train-3 → ussuri-1
Revision history for this message
nico auv (nico.auv) wrote :

@anta-nok I don't know if it would help but I encountered the same issue with OSP13 due to a involontary stack update cancellation. I was able to perform stack updates again after executing this Mistral workflow :

source stackrc
openstack workflow execution create tripleo.plan_management.v1.update_deployment_plan {"container": "overcloud"}

Let me know if it works with your version.

Revision history for this message
Anton Antonov (anta-nok) wrote :

@nico.auv thank you for your comment. It helps indeed to run (with single quotes)
openstack workflow execution create tripleo.plan_management.v1.update_deployment_plan '{"container": "overcloud"}'

Now my upgrade fails because of https://bugzilla.redhat.com/show_bug.cgi?id=1736267

But, at least I don't see "KeyError: 'passwords'" anymore :)

Luke Short (ekultails)
Changed in tripleo:
assignee: nobody → Luke Short (ekultails)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/686202
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=72850060e058274bb55aadbf2963f4f46ba3be3a
Submitter: Zuul
Branch: master

commit 72850060e058274bb55aadbf2963f4f46ba3be3a
Author: ekultails <email address hidden>
Date: Wed Oct 2 10:26:30 2019 -0400

    Handle a deployment that was manually cancelled.

    When doing a deployment or update, if the end-user uses CTRL+c
    (SIGINT) to end it early, it will fail complaining that there is
    no passwords key in the plan. We will now handle this and cleanup
    the existing plan.

    Change-Id: Ie7f1a4269936f98ad86db4993992e0b9a27edf73
    Closes-Bug: #1819672
    Resolves: rhbz#1753251
    Signed-off-by: ekultails <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/687659

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687661

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/687662

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/stein)

Reviewed: https://review.opendev.org/687659
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=134f0716218e395e462e69e5ac1ca15782431e91
Submitter: Zuul
Branch: stable/stein

commit 134f0716218e395e462e69e5ac1ca15782431e91
Author: ekultails <email address hidden>
Date: Wed Oct 2 10:26:30 2019 -0400

    Handle a deployment that was manually cancelled.

    When doing a deployment or update, if the end-user uses CTRL+c
    (SIGINT) to end it early, it will fail complaining that there is
    no passwords key in the plan. We will now handle this and cleanup
    the existing plan.

    Change-Id: Ie7f1a4269936f98ad86db4993992e0b9a27edf73
    Closes-Bug: #1819672
    Resolves: rhbz#1753251
    Signed-off-by: ekultails <email address hidden>
    (cherry picked from commit 72850060e058274bb55aadbf2963f4f46ba3be3a)

tags: added: in-stable-stein
tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.opendev.org/687661
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=58a8a0bd4e6f7fbdf4b3789e7c9958c1781bb4b0
Submitter: Zuul
Branch: stable/rocky

commit 58a8a0bd4e6f7fbdf4b3789e7c9958c1781bb4b0
Author: ekultails <email address hidden>
Date: Wed Oct 2 10:26:30 2019 -0400

    Handle a deployment that was manually cancelled.

    When doing a deployment or update, if the end-user uses CTRL+c
    (SIGINT) to end it early, it will fail complaining that there is
    no passwords key in the plan. We will now handle this and cleanup
    the existing plan.

    Change-Id: Ie7f1a4269936f98ad86db4993992e0b9a27edf73
    Closes-Bug: #1819672
    Resolves: rhbz#1753251
    Signed-off-by: ekultails <email address hidden>
    (cherry picked from commit 72850060e058274bb55aadbf2963f4f46ba3be3a)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/queens)

Reviewed: https://review.opendev.org/687662
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=fa98192ac48cf463a16fddf8498c7e1f328060d9
Submitter: Zuul
Branch: stable/queens

commit fa98192ac48cf463a16fddf8498c7e1f328060d9
Author: ekultails <email address hidden>
Date: Wed Oct 2 10:26:30 2019 -0400

    Handle a deployment that was manually cancelled.

    When doing a deployment or update, if the end-user uses CTRL+c
    (SIGINT) to end it early, it will fail complaining that there is
    no passwords key in the plan. We will now handle this and cleanup
    the existing plan.

    Change-Id: Ie7f1a4269936f98ad86db4993992e0b9a27edf73
    Closes-Bug: #1819672
    Resolves: rhbz#1753251
    Signed-off-by: ekultails <email address hidden>
    (cherry picked from commit 72850060e058274bb55aadbf2963f4f46ba3be3a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 12.3.0

This issue was fixed in the openstack/python-tripleoclient 12.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 11.5.2

This issue was fixed in the openstack/python-tripleoclient 11.5.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient rocky-eol

This issue was fixed in the openstack/python-tripleoclient rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient queens-eol

This issue was fixed in the openstack/python-tripleoclient queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.