heat-api fails to start if Keystone is unavailable

[Johannes Grassler]

Problem Description
===================

If Keystone is unavailable when heat-api starts up, heat-api aborts with the following error (see attachment for a full traceback):

  2016-02-26 10:59:23.817 25251 CRITICAL heat-api [-] ConnectionRefused: Unable to establish connection to http://192.168.232.2:5000

The problem was introduced by this commit:

https://github.com/openstack/heat/commit/487a211a8a3d603ae4b77087836081037fa49c70

The auth_url catalog lookup introduced by this commit may fail, and there is no
retrying, which causes heat-api to exit if Keystone is unreachable, even
temporarily.

Steps to reproduce
==================

 Note: this assumes a non-HA cloud with one controller where both heat-api and
 Keystone run on the same machine. On this machine run the following command:

 # systemctl restart openstack-keystone openstack-heat-engine

 Now look in /var/log/heat-api.log where the error message should appear. This
 happens because it takes a few seconds for the TCP listeners to be up after
 /usr/bin/keystone-all has been launched.

Environment
===========

Openstack Liberty on SLES 12.1

Package versions:
  openstack-heat-5.0.2~a0~dev9-6.9.noarch
  openstack-heat-plugin-heat_docker-5.0.2~a0~dev9-6.9.noarch
  openstack-heat-api-cfn-5.0.2~a0~dev9-6.9.noarch
  openstack-heat-templates-0.0.0+git.1452795102.e53f5d3-6.1.noarch
  openstack-heat-api-5.0.2~a0~dev9-6.9.noarch
  openstack-keystone-8.0.2~a0~dev34-6.13.noarch
  openstack-heat-engine-5.0.2~a0~dev9-6.9.noarch
  openstack-heat-api-cloudwatch-5.0.2~a0~dev9-6.9.noarch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287302

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287302
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=fe9226898772c4ff909f9c3f0cb05c271333b73a
Submitter: Jenkins
Branch: master

commit fe9226898772c4ff909f9c3f0cb05c271333b73a
Author: Johannes Grassler <email address hidden>
Date: Wed Mar 2 17:20:16 2016 +0100

    Make auth_url lookup dynamic

    If _get_auth_url() is run on heat-api startup, it can cause heat-api to crash
    in situations where Keystone is not running, yet or temporarily unavailable.
    This patch converts the auth_url attribute into a property method that is only
    run when it is needed, thus preventing this race condition.

    Change-Id: Ife6d9e51ea9647e4658105c016867efe769e5363
    Closes-Bug: #1550284

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/289792

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/289792
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=73969be4141088936e9fb2dff7d7a59c08d3f9ea
Submitter: Jenkins
Branch: stable/liberty

commit 73969be4141088936e9fb2dff7d7a59c08d3f9ea
Author: Johannes Grassler <email address hidden>
Date: Wed Mar 2 17:20:16 2016 +0100

    Make auth_url lookup dynamic

    If _get_auth_url() is run on heat-api startup, it can cause heat-api to crash
    in situations where Keystone is not running, yet or temporarily unavailable.
    This patch converts the auth_url attribute into a property method that is only
    run when it is needed, thus preventing this race condition.

    Change-Id: Ife6d9e51ea9647e4658105c016867efe769e5363
    Closes-Bug: #1550284
    (cherry picked from commit fe9226898772c4ff909f9c3f0cb05c271333b73a)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 5.0.2 release.