Heat service returns Internal Server Error when enabling kolla_enable_tls_external

[Magnus Lööf]

When enabling parameter `kolla_enable_tls_external` and deploying Openstack using Kolla, Heat Orchestration does not work properly.

- Accessing any of the menus under Orchestration in Horizon GUI over TLS fails, all other parts of GUI works
- Accessing Heat API over public interface with TLS fails with "Internal Error"
- Accessing Heat API over internal interface fails with "Internal Error"

Error logs from heat api

https://pastebin.com/445AjK2L

The problem can be resolved by reverting this change: https://review.openstack.org/#/c/566361/

The reason the problem appears are:

- The TLS certificate is not trusted by the container, if a private CA not in ca-certificates package is used
- The external FQDN cannot be reached from inside the internal network. For security reasons, this can be assumed to be required.

[Jesús Arias Gil (jesusarias95)]

Hi, any solution for this bug?

[Magnus Lööf (magnus-loof)]

I did a workaround and put this in the config files for Heat:

```
{% if openstack_enable_tls_external %}
[clients_keystone]
auth_uri = http://{{ openstack_internal_vip }}:5000
{% endif %}
```

[Mark Goddard (mgoddard)]

The reason for adding https://review.openstack.org/#/c/566361/ is that it is passed to instances by heat for calling back to the heat API. Instances can't be assumed to have access to the internal API, so we need to pass the external URL.

I can see why this causes issues with external TLS enabled however, since the container won't necessarily have the necessary certificates trusted. The workaround posted above should work.

[Mark Goddard (mgoddard)]

Would it be helpful to provide a flag in kolla-ansible that sets whether to use the internal or external keystone endpoint in this config section?

[Mark Goddard (mgoddard)]

It would be good for heat to provide a separate configuration option for its own access to the keystone endpoint vs that which is passed to instances with software deployments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676716

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676716
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Submitter: Zuul
Branch: master

commit d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Author: Mark Goddard <email address hidden>
Date: Thu Aug 15 16:04:44 2019 +0100

    Use internal API for heat -> heat communication

    Heat has a new option (server_keystone_endpoint_type), which can be used
    to set the keystone endpoint used by instances to make callbacks to
    heat. This needs to be public, since we can't assume users have access
    to the internal API. However, the current method of setting
    [clients_heat] endpoint_type means that communication from heat to its
    own API (e.g. when a stack is a resource in another stack) uses the
    public network also, and this might not work if TLS is enabled.

    This change uses server_keystone_endpoint_type to keep instance traffic
    on the public API, and removes the [clients_heat] endpoint_type option
    to use the default in [clients] endpoint_type of internalURL.

    This feature was added to heat in https://review.opendev.org/#/c/650967.

    Change-Id: I932ea55a3c2a411557c34361db08bcb3a2b27eaf
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[Radosław Piliszek (yoctozepto)]

Mark, this is not fixed. Heat still tries keystone over public endpoint. Only heat-to-heat was fixed.

[Radosław Piliszek (yoctozepto)]

Also, the feature was backported to stein, just not released. Negotiating new heat release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694666
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2cd00dadc0ad06610e94f879ddce145ad1134ea5
Submitter: Zuul
Branch: master

commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694985

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/694986

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/694986
Reason: not now

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694985
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=21e5924ef4f20e5e39bd708388088e565936e868
Submitter: Zuul
Branch: stable/train

commit 21e5924ef4f20e5e39bd708388088e565936e868
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331
    (cherry picked from commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.