Connection refused, http://overcloud-controller-0.ctlplane.localdomain:5000/v3/projects?

[wes hayutin]

Error: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://overcloud-controller-0.ctlplane.localdomain:5000/v3/projects?: HTTPConnectionPool(host='overcloud-controller-0.ctlplane.localdomain', port=5000): Max retries exceeded with url: /v3/projects (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0a8d1b3610>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 43, for a total of 170 seconds)",
2019-01-09 10:00:10 | "+ rc=1",

https://logs.rdoproject.org/98/604298/165/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/ee7566b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-01-09_10_00_10

https://logs.rdoproject.org/33/614633/12/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/5d3cc24/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-01-09_04_02_25

occurs in both rdo-cloud and vexxhost.
Started to fail around... | 2019-01-09 04:18 UTC

This is failing in both check and promotion jobs, this may be a tripleo or dependency change. A promotion did not cause this issue.

[yatin (yatinkarel)]

It's trying admin api(http://overcloud-controller-0.ctlplane.localdomain:5000/v3/projects) but instead it should try internalapi(http://overcloud-controller-0.internalapi.localdomain:5000/v3/projects)

https://review.openstack.org/#/c/619257/ puppet-keystone patch seems causing it, Though some changes were made in THT(https://review.openstack.org/#/c/627793) but that seems not sufficient.

[Tobias Urdin (tobias-urdin)]

Testing fix here https://review.openstack.org/#/c/629798/
Improvement to puppet-keystone release notes here https://review.openstack.org/#/c/629802/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/629845

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/629845
Reason: wrong patch

[Arx Cruz (arxcruz)]

Also affecting featureset020 master

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/c55151f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Arx Cruz (arxcruz)]

Affecting featureset002 as well

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/7140362/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz