[os_heat][pike] heat-container-agent fails to communicate with Keystone

[Christian Zunker]

I'm not quite sure whether this is a bug or just configuration.

I setup magnum with openstack-ansible. The cluster creation in magnum never finished because the heat-container-agent tried to connect to the internal keystone endpoint, which is not reachable from our VMs.

This is the post which helped me to solve the problem: https://ask.openstack.org/en/question/102214/software-deployment-in-heat-problem-with-os-collect-config/
Kolla already addressed the same problem: https://bugs.launchpad.net/kolla-ansible/+bug/1762754

In my osa deployment, I could fix it with these overrides:
heat_heat_conf_overrides:
  clients_keystone:
    endpoint_type: publicURL
    auth_uri: "{{ keystone_service_publicurl }}"

[DimGR (dimgr)]

Hi

can you check your heat.conf if it contains accessible from within your virtual machines hostnames?

Do you happen to have saved the heat error logs ?

[DimGR (dimgr)]

also , what keystone endpoint your heat.conf was showing ? internal or external ?

[Christian Zunker (christian-zunker)]

The public endpoint DNS entry can be resolved inside the VM.
The internal endpoint DNS entry cannot be resolved inside the VM. A connect to the IP and keystone port is also not possible.

The error message in journalctl -u heat-container-agent is:
Jul 11 05:22:58 cz-test-k8s-atomic-26-xxq727c5v2ne-master-0.novalocal runc[3409]: Authorization failed: Unable to establish connection to https://internal.<api server>:5000/v3/auth/tokens
Jul 11 05:22:58 cz-test-k8s-atomic-26-xxq727c5v2ne-master-0.novalocal runc[3409]: Source [heat] Unavailable.

Prior to my changes, the heat config showed the internal endpoint and the endpoint type was also internal.

[Amy Marrich (amy-marrich)]

Update to docs for this should be done to openstack-ansible-os_magnum/doc/source/index.rst adding a new section for this configuration.

[DimGR (dimgr)]

hi ,

for some reason your heat containers can not resolve the internal network . I think this is not a magnum bug but i will be keeping a close eye on this on my clusters and update if i see anything that relates to this

[Amy Marrich (amy-marrich)]

Right now it's looking like we should add documentation for configuring an
override within the role

On Fri, Jul 13, 2018 at 4:08 PM, DimGR <email address hidden> wrote:

> hi ,
>
> for some reason your heat containers can not resolve the internal
> network . I think this is not a magnum bug but i will be keeping a close
> eye on this on my clusters and update if i see anything that relates to
> this
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1780385
>
> Title:
> [os_heat][pike] heat-container-agent fails to communicate with
> Keystone
>
> Status in openstack-ansible:
> Triaged
>
> Bug description:
> I'm not quite sure whether this is a bug or just configuration.
>
> I setup magnum with openstack-ansible. The cluster creation in magnum
> never finished because the heat-container-agent tried to connect to
> the internal keystone endpoint, which is not reachable from our VMs.
>
> This is the post which helped me to solve the problem:
> https://ask.openstack.org/en/question/102214/software-
> deployment-in-heat-problem-with-os-collect-config/
> Kolla already addressed the same problem: https://bugs.launchpad.net/
> kolla-ansible/+bug/1762754
>
> In my osa deployment, I could fix it with these overrides:
> heat_heat_conf_overrides:
> clients_keystone:
> endpoint_type: publicURL
> auth_uri: "{{ keystone_service_publicurl }}"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-ansible/+bug/1780385/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/582919

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/582919
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/commit/?id=fd0b5fe2f55257a6e6e6ddcd034395a7e329213b
Submitter: Zuul
Branch: master

commit fd0b5fe2f55257a6e6e6ddcd034395a7e329213b
Author: Christian Zunker <email address hidden>
Date: Mon Jul 16 13:04:25 2018 +0200

    Explain how to configure public endpoint

    This is needed for Magnum and might also be needed in other setups.

    Change-Id: Ifcfc00fd860f555c17486bcd8060d3cd0d88f604
    Closes-Bug: #1780385

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_heat xena-eom release.