tripleo

After upgrading to containerized undercloud, overcloud upgrade fails with InternalServerError

Bug #1787131 reported by Rabi Mishra on 2018-08-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Rabi Mishra	tripleo rocky-rc1

Bug Description

After upgrading containerized undercloud, overcloud deploy fails deleting software deployments.

Description of problem:
-----------------------
Attempt to prepare playbooks for upgrade failed:
openstack overcloud upgrade prepare --stack overcloud \
    --templates /usr/share/openstack-tripleo-heat-templates \
    -e /home/stack/virt/internal.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /home/stack/virt/network/network-environment.yaml \
    -e /home/stack/virt/enable-tls.yaml \
    -e /home/stack/virt/inject-trust-anchor.yaml \
    -e /home/stack/virt/public_vip.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
    -e /home/stack/virt/hostnames.yml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
    -e /home/stack/virt/debug.yaml \
    -e /home/stack/virt/nodes_data.yaml \
    -e /home/stack/virt/docker-images.yaml \
    -e /home/stack/cli_opts_params.yaml \
    -e /home/stack/virt/docker-images.yaml \
    --roles-file /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
...
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: CREATE_COMPLETE state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected
error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_IN_PROGRESS state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An $
nexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-2-kn4lz2kbqg65.ControllerDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj.0]: UPDATE_FAILED resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.CephStorageExtraConfigPre]: UPDATE_COMPLETE state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73.2]: UPDATE_FAILED resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Compute-pjxd24dovyp2-1-ampsmvt53hps.NovaComputeDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An un$
xpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj.1]: UPDATE_FAILED resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.SshHostPubKey]: DELETE_FAILED InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected erro$
prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:14Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.ControllerDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_FAILED InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected err$
r prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)

Stack overcloud/3dbb3a91-2bc8-434d-8db5-c73f27753bcb UPDATE_FAILED

openstack stack failures list overcloud
overcloud.Controller.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 2b118f9f-1b93-4ed6-841c-88144edc8dbf
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-
4d6e-a288-4a85c34234fd)
overcloud.Controller.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: f9e905b3-d1fa-4b90-8471-2dcacd147c3a
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-
4f26-a022-f6a94725c925)
overcloud.Controller.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: a59f6b62-d65e-4104-b626-96294f3965bc
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d094ace2-c6c6-
4b8f-ae90-ed465680131f)
overcloud.Compute.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 0af3b7a3-e305-43d3-b950-af2f34d1f56a
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-3e94ec07-2d16-
4779-ac88-932e262c9e27)
overcloud.Compute.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: fda08616-77ba-4abd-80b6-726998db40b2
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b82cba0a-a879-
47cd-b13d-59d8b84cf9d3)
overcloud.CephStorage.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 6d74f773-2719-4bd8-86a4-bbd41725b9ef
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-db53f67b-16e9-
4327-a156-6c3c4d92a378)
overcloud.CephStorage.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: faf82cc2-f53d-452a-b91e-6fdcb3776c2c
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-07d16af3-9b24-
4676-a493-fbc915a1e50a)
overcloud.CephStorage.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: d8315964-d419-4659-bacc-7ae260b0eeb5
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-
4b06-b29a-c4b3827b3027)

And from /var/log/containers/keystone/keystone.log
2018-07-31 04:01:47.036 175 DEBUG keystone.common.fernet_utils [req-a8e67edf-5e70-41e0-a02c-7c3b0129a167 - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 5`;
perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /usr/lib/python2.7/site-packages/keystone/common/fernet_utils.py:307
2018-07-31 04:01:47.044 172 ERROR keystone.credential.providers.fernet.core [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd037
70c63b0 -] Credential could not be decrypted. Please contact the administrator: InvalidToken
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd03770c63b0 -] Credential could not be decrypted. Please contact the administrator: CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 148, in __call__
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi result = method(req, **params)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 57, in inner
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi self, f, check_function, request, None, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 122, in protected_wrapper
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi check_function(self, request, prep_info, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 131, in check_protection
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 150, in check_policy
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi _handle_member_from_driver(controller, policy_dict, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 76, in _handle_member_from_driver
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi ref = self.get_member_from_driver(kwargs[key])
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 103, in get_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi return self._decrypt_credential(credential)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 49, in _decrypt_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi credential['encrypted_blob'],
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 112, in decrypt
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi raise exception.CredentialEncryptionError(msg)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi

Rabi Mishra (rabi) on 2018-08-15

Changed in tripleo:
assignee:	nobody → Rabi Mishra (rabi)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-15: Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/591980

Changed in tripleo:
status:	New → In Progress

Emilien Macchi (emilienm) on 2018-08-15

Changed in tripleo:
milestone:	none → rocky-rc1
importance:	Undecided → Critical

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-20: Fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/591980
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e5bdf010a1be64e3e661500b442929c257688325
Submitter: Zuul
Branch: master

commit e5bdf010a1be64e3e661500b442929c257688325
Author: rabi <email address hidden>
Date: Wed Aug 15 13:03:24 2018 +0530

Re-use old fernet and credential keys in containerized undercloud

When doing undercloud upgrade, we need to use the old fernet
and ec2 credential keys and not generate them again.

Change-Id: Ib9ef9488a39d5e8f8f99df778a0600dd888ef1d3
Closes-Bug: #1787131

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-21: Fix included in openstack/python-tripleoclient 10.5.0

This issue was fixed in the openstack/python-tripleoclient 10.5.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.