Tripleo deployment getting failed in establishing a connection with keystone.
Steps to reproduce:
Trying to deploy a 3controller-1compute nodes along with a supp node to have TLS everywhere kind of deployment.
1. bash quickstart.sh -R master-tripleo-ci -v --tags all --teardown all --nodes config/nodes/3ctlr_1comp_1supp.yml --config config/general_config/ipa.yml -p quickstart-extras.yml -e undercloud_disk=100 -e undercloud_undercloud_ntp_servers=clock.redhat.com $VIRTHOST
Expected Result:
A succesfull UC & OC deployment tls everywhere being implemented.
Actual Result:
During the overcloud deployment
"stderr: ",
"Error running ['docker', 'run', '--name', 'nova_cellv2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_cellv2_discover_hosts', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 0, \"command\": \"/container-config-scripts/pyshim.sh /container-config-scripts/nova_cell_v2_discover_host.py\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro\", \"/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/container-config-scripts/:/container-config-scripts/\"], \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-nova-compute:current-tripleo\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/', '192.168.24.1:8787/tripleomaster/centos-binary-nova-compute:current-tripleo', '/container-config-scripts/pyshim.sh', '/container-config-scripts/nova_cell_v2_discover_host.py']. [1]",
"stdout: DEBUG:nova_cell_v2_discover_host:Running with parameter insecure = True",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('noauth = cinderclient.contrib.noauth:CinderNoAuthLoader')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('none = keystoneauth1.loading._plugins.noauth:NoAuth')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3adfspassword = keystoneauth1.extras._saml2._loading:ADFSPassword')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3applicationcredential = keystoneauth1.loading._plugins.identity.v3:ApplicationCredential')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')",
"DEBUG:stevedore.extension:found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader')",
"DEBUG:keystoneauth.session:REQ: curl -g -i -X GET https://overcloud.internalapi.ooo.test:5000/v3 -H \"Accept: application/json\" -H \"User-Agent: nova_cell_v2_discover_host.py keystoneauth1/3.13.1 python-requests/2.21.0 CPython/2.7.5\"",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): overcloud.internalapi.ooo.test:5000",
"WARNING:keystoneauth.identity.generic.base:Failed to discover available identity versions when contacting https://overcloud.internalapi.ooo.test:5000/v3. Attempting to parse version from URL.",
"DEBUG:keystoneauth.identity.v3.base:Making authentication request to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): overcloud.internalapi.ooo.test:5000",
"ERROR:nova_cell_v2_discover_host:Error during host discovery:",
"Traceback (most recent call last):",
" File \"/container-config-scripts/nova_cell_v2_discover_host.py\", line 85, in <module>",
" service_list = nova.services.list(binary='nova-compute')",
" File \"/usr/lib/python2.7/site-packages/novaclient/v2/services.py\", line 52, in list",
" return self._list(url, \"services\")",
" File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 254, in _list",
" resp, body = self.api.client.get(url)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 375, in get",
" return self.request(url, 'GET', **kwargs)",
" File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 72, in request",
" **kwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 534, in request",
" resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 237, in request",
" return self.session.request(url, method, **kwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 704, in request",
" auth_headers = self.get_auth_headers(auth)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 1097, in get_auth_headers",
" return auth.get_headers(self, **kwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 95, in get_headers",
" token = self.get_token(session)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 88, in get_token",
" return self.get_access(session).auth_token",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access",
" self.auth_ref = self.get_auth_ref(session)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 208, in get_auth_ref",
" return self._plugin.get_auth_ref(session, **kwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py\", line 178, in get_auth_ref",
" authenticated=False, log=False, **rkwargs)",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 1045, in post",
" File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 942, in _send_request",
" raise exceptions.ConnectFailure(msg)",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f514610>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (3): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2490>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (4): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2510>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (5): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2550>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (6): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f24d0>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (7): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f23d0>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (8): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2450>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (9): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2410>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (10): overcloud.internalapi.ooo.test:5000",
"ConnectFailure: Unable to establish connection to https://overcloud.internalapi.ooo.test:5000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.internalapi.ooo.test', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb32f4f2390>: Failed to establish a new connection: [Errno 113] No route to host',))",
"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (11): overcloud.internalapi.ooo.test:5000",
"stderr: + command -v python3",
"+ command -v python2",
"+ python2 /container-config-scripts/nova_cell_v2_discover_host.py"
]
}
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP *********************************************************************
overcloud-controller-0 : ok=208 changed=83 unreachable=0 failed=1
overcloud-controller-1 : ok=208 changed=83 unreachable=0 failed=1
overcloud-controller-2 : ok=208 changed=83 unreachable=0 failed=1
overcloud-novacompute-0 : ok=188 changed=53 unreachable=0 failed=1
undercloud : ok=11 changed=7 unreachable=0 failed=0
Friday 05 April 2019 19:32:14 +0000 (0:00:00.851) 0:28:02.119 **********
===============================================================================
Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.
Exception occured while running the command
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 30, in run
super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 949, in take_action
verbosity=self.app_args.verbose_level)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 327, in config_download
raise exceptions.DeploymentError("Overcloud configuration failed.")
DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.
+ status_code=1
+ openstack stack list
+ grep -q overcloud
+ openstack stack list
+ grep -Eq '(CREATE|UPDATE)_COMPLETE'
+ openstack overcloud status
+ grep -Eq DEPLOY_SUCCESS
+ openstack overcloud failures
+ exit 1
(undercloud) [stack@undercloud ~]$
usually when we have seen this before there was some issue before that, or a network issue. From above the compute can not reach the keystone endpoint:
ConnectFailure: Unable to establish connection to https:/
Is keystone up and running? Can we get the full logs from when it happens.