Master promotion fs002 and fs020 jobs are failing overcloud deployment on ControllerDeployment_Step3

[Ronelle Landy]

This failure was detected first on 11/28 but it possible that other failures masked this overcloud deploy failure before.

See full logs at:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload/b4108cf/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload/b4108cf/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 8b824721-8352-430f-a8dd-9b1bd361f557
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2

This 'non-fatal error' is repeatedly shown 'Unable to establish connection to http://overcloud-controller-0.ctlplane.localdomain:35357/v3/domains':

Debug: Puppet::Provider::Openstack: Non-fatal error: 'Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://overcloud-controller-0.ctlplane.localdomain:35357/v3/domains?: HTTPConnectionPool(host='overcloud-controller-0.ctlplane.localdomain', port=35357): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x4c1fed0>: Failed to establish a new connection: [Errno 111] Connection refused',))'. Retrying for 168 more seconds"

[Alex Schultz (alex-schultz)]

The actual failure is on the File[gnocchi_wsgi]. The wsgi file has been (re)moved. This is captured by Bug 1735118 and has been fixed in puppet-gnocchi