Overcloud stack create fails when disabling heat_api_cfn and heat_api_cloudwatch

[Jose Luis Franco]

Description
===========

While adapting the tripleo-ci environment file worker-config.yaml to properly disable the heat services heat_api_cfn and heat_api_cloudwatch in the resource_registry instead of disabling them via puppet config: https://review.openstack.org/#/c/499182/ it has been found a bug that impacts the overcloud stack creation.

When deploying the stack, one of these disable services is being used and it is not being taken into account if the service is enabled or not. As a result, the following traceback is returned:

http://logs.openstack.org/82/499182/2/check/gate-tripleo-ci-centos-7-3nodes-multinode-nv/d6e674d/logs/subnode-2/var/log/heat/heat-engine.log.txt.gz#_2017-09-04_15_38_09_559

2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource Traceback (most recent call last):
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 831, in _action_recorder
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource yield
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 939, in _do_action
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 334, in wrapper
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource step = next(subtask)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 884, in action_handler_task
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource handler_data = handler(*args)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 803, in handle_create
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource user_data_format=user_data_format)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 399, in build_userdata
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource watch_url = heat_client_plugin.get_watch_server_url()
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/heat_plugin.py", line 93, in get_watch_server_url
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource cfn_url = self.get_heat_cfn_url()
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/heat_plugin.py", line 71, in get_heat_cfn_url
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource endpoint_type=endpoint_type)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 111, in url_for
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource url = get_endpoint()
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 97, in get_endpoint
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource return keystone_session.get_endpoint(**kwargs)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 947, in get_endpoint
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource return auth.get_endpoint(self, **kwargs)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 378, in get_endpoint
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource allow_version_hack=allow_version_hack, **kwargs)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 277, in get_endpoint_data
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource service_name=service_name)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource return wrapped(*args, **kwargs)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 407, in endpoint_data_for
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource raise exceptions.EndpointNotFound(msg)
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource EndpointNotFound: publicURL endpoint for cloudformation service not found
2017-09-04 15:38:09.559 27272 ERROR heat.engine.resource

Following the traceback, the source of the issue seems to be at this point:
https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L399

Steps to reproduce
==================

1. Deploy the undercloud
2. Deploy the overcloud appending the worker-config.yaml environment file from this patch (cherry-pick it) https://review.openstack.org/#/c/499182/ [-e /home/stack/tripleo-ci/test-environments/worker-config.yaml]
3. Wait for the deployment to finish and run a pingtest:
3.1. Execute the script /home/stack/tripleo-ci/scritps/tripleo.sh --overcloud-pingtest
3.2. Or try to deploy a stack from your own

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/502957
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d14d9202e3a0d7ce4547d69a6857c8b9f1874a65
Submitter: Jenkins
Branch: master

commit d14d9202e3a0d7ce4547d69a6857c8b9f1874a65
Author: Jose Luis Franco Arza <email address hidden>
Date: Tue Sep 12 13:17:04 2017 +0200

    Avoid cloudwatch endpoint retrieve when disabled

    If openstack-api-cloudwatch service is disabled in a
    deployment these lines of code will provoke a
    traceback when trying to get the cloudwatch
    end-point.

    Change-Id: I7257f79af764cddc16423826143b0d8babfe54e7
    Partial-Bug: #1715083

[Zane Bitter (zaneb)]

Why was https://review.openstack.org/#/c/514761/ only a partial fix? What is there left to do?

[Jose Luis Franco (jfrancoa)]

So, I rechecked this issue and it seems it has been solved in queens, but it's still happening in pike and ocata's jobs:

https://review.openstack.org/#/c/499182/