clients_keystone inside heat.conf points to internal keystone endpoint

[Bharat Kunwar]

Tested on pike (5.0.0) but with magnum from queens (6.0.0).

Steps to reproduce:

- Deploy a cloud using kolla-ansible with heat enabled.
- Create a kubernetes cluster using fedora-atomic-27 image:

openstack coe cluster template create k8s-fa27 \
                           --external-network ilab \
                           --fixed-network p3-internal \
                           --fixed-subnet p3-internal \
                           --server-type vm \
                           --image fedora-atomic-27 \
                           --flavor compute-A\
                           --master-flavor compute-A\
                           --docker-volume-size 4 \
                           --docker-storage-driver overlay2 \
                           --coe kubernetes \
                           --network-driver flannel

openstack coe cluster create k8s-fa27 \
                      --cluster-template k8s-fa27 \
                      --keypair default \
                      --master-count 1 \
                      --node-count 1

Expected results:

- Magnum successfully provisions the cluster.

Actual results:

- Magnum is not able to provision the cluster, it remains stuck at CREATE_IN_PROGRESS.
- Digging deeper, Heat also appears to be stuck at CREATE_IN_PROGRESS status.
- Digging even deeper, it turns out that `heat-container-agent` inside the `kube-master` node is pointing to an internal clients_keystone endpoint which is not accessible from the container.

Discussion:

This mechanism is commonly used by instances to signal various events back to heat. These instances are unlikely to have access to the internal API endpoints. There have been similar issues with OpenStack-Ansible[1] back in juno and changed to use the public endpoint[2]. The code has now moved but the default is still in place[3]. This issue was previously also noted with kolla-ansible[4] for which a fix was released[5]. Another similar issue[6] appears to be a separate issue looking at their config.

[1] https://bugs.launchpad.net/openstack-ansible/+bug/1459414
[2] https://review.openstack.org/#/c/186221/
[3] https://github.com/openstack/openstack-ansible-os_heat/blob/b1721a7460ba816afabd9bded6e5c79c635bca3a/defaults/main.yml#L48
[4] https://bugs.launchpad.net/kolla-ansible/+bug/1688331
[5] https://review.openstack.org/#/c/462655/
[6] https://bugs.launchpad.net/magnum/+bug/1655007

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566361
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c20c69ee5eee08190cfcbeea54cc89909c7d1860
Submitter: Zuul
Branch: master

commit c20c69ee5eee08190cfcbeea54cc89909c7d1860
Author: Bharat Kunwar <email address hidden>
Date: Tue Apr 10 17:09:27 2018 +0100

    kolla-ansible fix to correct magnum k8s deployment

    Magnum was unable to fire up k8s cluster because heat-container-agent
    inside kube-master was pointing to internal keystone endpoint instead of
    public endpoint. This fix tells kolla ansible to set clients_keystone
    auth_uri to public endpoint so that heat-container-agent communication
    with heat is successfully authenticated by keystone.

    Change-Id: Ida49528f88685710b5e6b8f3c4d4622506af5ae1
    Closes-Bug: #1762754

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/620006

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/620014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620006
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=70356636ca0193710a5db1b60ffb598ce94cb213
Submitter: Zuul
Branch: stable/queens

commit 70356636ca0193710a5db1b60ffb598ce94cb213
Author: Bharat Kunwar <email address hidden>
Date: Tue Apr 10 17:09:27 2018 +0100

    kolla-ansible fix to correct magnum k8s deployment

    Magnum was unable to fire up k8s cluster because heat-container-agent
    inside kube-master was pointing to internal keystone endpoint instead of
    public endpoint. This fix tells kolla ansible to set clients_keystone
    auth_uri to public endpoint so that heat-container-agent communication
    with heat is successfully authenticated by keystone.

    Change-Id: Ida49528f88685710b5e6b8f3c4d4622506af5ae1
    Closes-Bug: #1762754
    (cherry picked from commit c20c69ee5eee08190cfcbeea54cc89909c7d1860)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620014
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1952ed53d356d71e9299134201926ce14c9c907d
Submitter: Zuul
Branch: stable/pike

commit 1952ed53d356d71e9299134201926ce14c9c907d
Author: Bharat Kunwar <email address hidden>
Date: Tue Apr 10 17:09:27 2018 +0100

    kolla-ansible fix to correct magnum k8s deployment

    Magnum was unable to fire up k8s cluster because heat-container-agent
    inside kube-master was pointing to internal keystone endpoint instead of
    public endpoint. This fix tells kolla ansible to set clients_keystone
    auth_uri to public endpoint so that heat-container-agent communication
    with heat is successfully authenticated by keystone.

    Change-Id: Ida49528f88685710b5e6b8f3c4d4622506af5ae1
    Closes-Bug: #1762754
    (cherry picked from commit c20c69ee5eee08190cfcbeea54cc89909c7d1860)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.1.1 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/676716

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676716
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Submitter: Zuul
Branch: master

commit d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Author: Mark Goddard <email address hidden>
Date: Thu Aug 15 16:04:44 2019 +0100

    Use internal API for heat -> heat communication

    Heat has a new option (server_keystone_endpoint_type), which can be used
    to set the keystone endpoint used by instances to make callbacks to
    heat. This needs to be public, since we can't assume users have access
    to the internal API. However, the current method of setting
    [clients_heat] endpoint_type means that communication from heat to its
    own API (e.g. when a stack is a resource in another stack) uses the
    public network also, and this might not work if TLS is enabled.

    This change uses server_keystone_endpoint_type to keep instance traffic
    on the public API, and removes the [clients_heat] endpoint_type option
    to use the default in [clients] endpoint_type of internalURL.

    This feature was added to heat in https://review.opendev.org/#/c/650967.

    Change-Id: I932ea55a3c2a411557c34361db08bcb3a2b27eaf
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/694666

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694666
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2cd00dadc0ad06610e94f879ddce145ad1134ea5
Submitter: Zuul
Branch: master

commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/694985

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/694986

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/694986
Reason: not now

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694985
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=21e5924ef4f20e5e39bd708388088e565936e868
Submitter: Zuul
Branch: stable/train

commit 21e5924ef4f20e5e39bd708388088e565936e868
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331
    (cherry picked from commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5)

[Saibal Dey (saibaldey)]

The root cause of the issue is hostname/DNS resolution fails for auth-URLs & few other service URLS (magnum,heat etc).
Generally while configuring the OpenStack services we use "controller" instead of the IP, like example:

For Keystone:
keystone-manage bootstrap --bootstrap-password ADMIN_PASS \
  --bootstrap-admin-url http://controller:5000/v3/ \
  --bootstrap-internal-url http://controller:5000/v3/ \
  --bootstrap-public-url http://controller:5000/v3/ \
  --bootstrap-region-id RegionOne

For Magnum:
openstack endpoint create --region RegionOne \
  container-infra public http://controller:9511/v1

So when the k8s master node gets provisioned (as VM) it creates couple of containers, "heat-container-agent" which interns use those able mentioned configs to get the k8s registered & configured with the OpenStack. As the container can't resolve the "controller" DNS entry, so all communications with OpenStack API fails. So there are couple of fixes for this:
1. Configure an internal DNS so resolute the "controller" URLs or
2. Use IP instead of "controller" in those able mentioned configs.

Option 2 is for POC & #1 should be considered for the production or HA OpenStack clusters.