[UPGRADES][14] Failed to setup heat output - resources.CADeployment: An unexpected error prevented the server from fulfilling your request

[Juan Antonio Osorio Robles]

Description of problem:
-----------------------
Attempt to setup heat outputs(prepare playbooks) for OC upgrade failed:
openstack overcloud upgrade prepare --stack overcloud \
    --templates /usr/share/openstack-tripleo-heat-templates \
    -e /home/stack/virt/internal.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /home/stack/virt/network/network-environment.yaml \
    -e /home/stack/virt/enable-tls.yaml \
    -e /home/stack/virt/inject-trust-anchor.yaml \
    -e /home/stack/virt/public_vip.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
    -e /home/stack/virt/hostnames.yml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
    -e /home/stack/virt/debug.yaml \
    -e /home/stack/virt/nodes_data.yaml \
    -e /home/stack/virt/docker-images.yaml
...
2018-08-10 07:47:51.646 29481 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:256ESC[00m
2018-08-10 07:47:52.276 29481 DEBUG heatclient.osc.v1.stack_failures.ListStackFailures [-] run(Namespace(long=False, stack=u'overcloud')) run /usr/lib/python2.7/site-packages/osc_lib/command/command.py:40ESC[00m
overcloud.Controller.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 440dabb4-bade-45ed-9a59-37cfd09a99ab
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c75a94b4-ed4a-
4be7-9bf9-d687a8ef4134)
overcloud.Controller.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 3b66910e-134b-452e-800d-be072d5ce19a
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-602dc7c3-0daa-49b3-846d-e6c0879843e1)
overcloud.Controller.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 93cbe670-6076-46bc-8c19-f3588b61a501
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-dd3b6010-8afb-49fd-be83-89481cabdf34)
overcloud.Compute.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 2a00c8c0-7b38-4f29-8a08-4b0daa7cf1d8
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-7935f1df-f057-4e7d-8b39-99399661859c)
overcloud.Compute.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: f7194410-4c4b-43ae-a014-24607f318328
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-27fca815-182b-450e-a1e5-a0150f3d4db6)
overcloud.CephStorage.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: f3d36fd0-7009-4c59-89b1-ce800bf09de7
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6abcaed2-47ed-458c-a8c2-d47f23e801b9)
overcloud.CephStorage.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 69282281-4c0b-4515-841a-f7093a976d9f
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-98e8c471-cfba-4ea8-b8d5-81cdcdccb1e3)
overcloud.CephStorage.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: f9b40a01-30ec-4770-870c-d1028cbc00b8
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-5f9e253f-34a9-44dd-b0e6-a12878bee09a)
2018-08-10 07:48:03.103 29481 DEBUG osc_lib.shell [-] clean_up ListStackFailures: clean_up /usr/lib/python2.7/site-packages/osc_lib/shell.py:486ESC[00m
2018-08-10 07:48:03.103 29481 INFO osc_lib.shell [-] END return value: 0ESC[00m
2018-08-10 07:48:03.105 29481 ERROR openstack [-] Heat Stack update failed.: DeploymentError: Heat Stack update failed.ESC[00m
Heat Stack update failed.
2018-08-10 07:48:03.105 29481 DEBUG osc_lib.shell [-] clean_up UpgradePrepare: Heat Stack update failed. clean_up /usr/lib/python2.7/site-packages/osc_lib/shell.py:486ESC[00m
2018-08-10 07:48:03.108 29481 INFO osc_lib.shell [-] END return value: 1ESC[00m

Rabi pointed that config download with 14 and map OS::Heat::SoftwareDeployment with https://github.com/openstack/tripleo-heat-templates/blob/master/config-download-software.yaml which causes issue at https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/extraconfig/tls/ca-inject.yaml#L67

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-heat-templates-9.0.0-0.20180703131156.de62fe3.el7ost.noarch
openstack-heat-api-cfn-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
openstack-heat-api-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
openstack-heat-agents-1.6.1-0.20180605100743.235e1ae.el7ost.noarch
openstack-heat-common-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
openstack-heat-engine-12.0.0-0.20180604085325.7d878a8.el7ost.noarch
openstack-heat-monolith-12.0.0-0.20180604085325.7d878a8.el7ost.noarch

Steps to Reproduce:
-------------------
1. Install RHOS-13(uc/oc with ssl)
2. Upgrade uc to RHOS-14
3. Try to run `openstack overcloud upgrade prepare`

Actual results:
---------------
Playbooks for upgrade ain't generated

-----------------------

Original BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1615917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/591681

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/591681
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=63a8f5529fe625069de4e6644e488496080c0b6e
Submitter: Zuul
Branch: master

commit 63a8f5529fe625069de4e6644e488496080c0b6e
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Aug 14 17:27:29 2018 +0300

    Remove md5 checksum output from CA injection

    This doesn't work with config download anymore, since the software
    configs are overwritten. It was never a very useful output anyway, so
    it's being removed.

    Change-Id: I2c4fc26009fb6e031c6d7fe11401e85c995bd210
    Closes-Bug: #1786954

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0rc1 release candidate.