After upgrading containerized undercloud, overcloud deploy fails deleting software deployments.
Description of problem:
-----------------------
Attempt to prepare playbooks for upgrade failed:
openstack overcloud upgrade prepare --stack overcloud \
--templates /usr/share/openstack-tripleo-heat-templates \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /home/stack/cli_opts_params.yaml \
-e /home/stack/virt/docker-images.yaml \
--roles-file /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
...
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: CREATE_COMPLETE state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected
error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_IN_PROGRESS state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An $
nexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-2-kn4lz2kbqg65.ControllerDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj.0]: UPDATE_FAILED resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.CephStorageExtraConfigPre]: UPDATE_COMPLETE state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73.2]: UPDATE_FAILED resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.NodeTLSCAData]: UPDATE_FAILED resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Compute-pjxd24dovyp2-1-ampsmvt53hps.NovaComputeDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An un$
xpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj.1]: UPDATE_FAILED resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED Resource UPDATE failed: resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.SshHostPubKey]: DELETE_FAILED InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevente$
the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected erro$
prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:14Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.ControllerDeployment]: CREATE_COMPLETE state changed
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_FAILED InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected err$
r prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)
Stack overcloud/3dbb3a91-2bc8-434d-8db5-c73f27753bcb UPDATE_FAILED
openstack stack failures list overcloud
overcloud.Controller.1.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: 2b118f9f-1b93-4ed6-841c-88144edc8dbf
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-
4d6e-a288-4a85c34234fd)
overcloud.Controller.0.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: f9e905b3-d1fa-4b90-8471-2dcacd147c3a
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-
4f26-a022-f6a94725c925)
overcloud.Controller.2.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: a59f6b62-d65e-4104-b626-96294f3965bc
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d094ace2-c6c6-
4b8f-ae90-ed465680131f)
overcloud.Compute.1.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: 0af3b7a3-e305-43d3-b950-af2f34d1f56a
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-3e94ec07-2d16-
4779-ac88-932e262c9e27)
overcloud.Compute.0.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: fda08616-77ba-4abd-80b6-726998db40b2
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b82cba0a-a879-
47cd-b13d-59d8b84cf9d3)
overcloud.CephStorage.1.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: 6d74f773-2719-4bd8-86a4-bbd41725b9ef
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-db53f67b-16e9-
4327-a156-6c3c4d92a378)
overcloud.CephStorage.0.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: faf82cc2-f53d-452a-b91e-6fdcb3776c2c
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-07d16af3-9b24-
4676-a493-fbc915a1e50a)
overcloud.CephStorage.2.NodeTLSCAData:
resource_type: OS::TripleO::NodeTLSCAData
physical_resource_id: d8315964-d419-4659-bacc-7ae260b0eeb5
status: UPDATE_FAILED
status_reason: |
resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-
4b06-b29a-c4b3827b3027)
And from /var/log/containers/keystone/keystone.log
2018-07-31 04:01:47.036 175 DEBUG keystone.common.fernet_utils [req-a8e67edf-5e70-41e0-a02c-7c3b0129a167 - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 5`;
perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /usr/lib/python2.7/site-packages/keystone/common/fernet_utils.py:307
2018-07-31 04:01:47.044 172 ERROR keystone.credential.providers.fernet.core [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd037
70c63b0 -] Credential could not be decrypted. Please contact the administrator: InvalidToken
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd03770c63b0 -] Credential could not be decrypted. Please contact the administrator: CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 148, in __call__
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi result = method(req, **params)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 57, in inner
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi self, f, check_function, request, None, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 122, in protected_wrapper
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi check_function(self, request, prep_info, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 131, in check_protection
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 150, in check_policy
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi _handle_member_from_driver(controller, policy_dict, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 76, in _handle_member_from_driver
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi ref = self.get_member_from_driver(kwargs[key])
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 103, in get_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi return self._decrypt_credential(credential)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 49, in _decrypt_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi credential['encrypted_blob'],
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 112, in decrypt
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi raise exception.CredentialEncryptionError(msg)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi
Fix proposed to branch: master /review. openstack. org/591980
Review: https:/