tripleo overcloud failed in step 5 : debug container stared step

[Hamed Enayatzare]

Description
========================
in deploy opentack stein in tripleo i get an error. this error cause failed deployment. in this error nova_wait_for_compute_service container could not run because could not find keystone service. by trace bug i understand all container in controller didn`t run. in actual in each deployment step for start container, them container for this step started but in next step removed.

Steps to reproduce
====================
1)install undercloud :
openstack undercloud install

2)install overcloud:
openstack overcloud deploy --control-flavor control --compute-flavor compute --templates /home/stack/generated-openstack-tripleo-heat-templates -e /home/stack/containers-prepare-parameter.yaml -e environment.yaml -e /home/stack/generated-openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/generated-openstack-tripleo-heat-templates/environments/network-environment.yaml -e /home/stack/templates/network-environment-overrides.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml --timeout 600 --ntp-server 0.asia.pool.ntp.org --verbose

Expected result
====================
TASK [Start containers for step 5] *********************************************
task path: /var/lib/mistral/overcloud/common_deploy_steps_tasks.yaml:568
ok: [overcloud-novacompute-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
Friday 03 April 2020 18:01:25 +0430 (0:11:19.419) 1:54:53.976 **********

TASK [Debug output for task: Start containers for step 5] **********************
task path: /var/lib/mistral/overcloud/common_deploy_steps_tasks.yaml:585
fatal: [overcloud-novacompute-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "stdout: c73edcb8b695302d6bf7e0789d3ccac269110697517e54990789f6707b627b5b",
        "",
        "stderr: ",
        "Created symlink from /etc/systemd/system/multi-user.target.wants/tripleo_nova_compute.service to /etc/systemd/system/tripleo_nova_compute.service.",
        "Created symlink from /etc/systemd/system/timers.target.wants/tripleo_nova_compute_healthcheck.timer to /etc/systemd/system/tripleo_nova_compute_healthcheck.time
r.",
        "Created symlink from /etc/systemd/system/tripleo_nova_compute.service.requires/tripleo_nova_compute_healthcheck.timer to /etc/systemd/system/tripleo_nova_comput
e_healthcheck.timer.",
        "Error running ['podman', 'run', '--name', 'nova_wait_for_compute_service', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_wait_for_comput
e_service', '--label', 'managed_by=tripleo-Compute', '--label', 'config_data={\"start_order\": 4, \"image\": \"172.20.128.1:8787/tripleostein/centos-binary-nova-compute:
current-tripleo\", \"environment\": [\"__OS_DEBUG=false\", \"TRIPLEO_CONFIG_HASH=ba8c4fb3937f19883243859e5a49a26f-ba8c4fb3937f19883243859e5a49a26f\"], \"command\": \"/co
ntainer-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py\", \"user\": \"nova\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/loc
altime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/
etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/t
ls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/confi
g-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro\", \"/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/
lib/container-config-scripts/:/container-config-scripts/\"], \"net\": \"host\", \"detach\": false}', '--conmon-pidfile=/var/run/nova_wait_for_compute_service.pid', '--lo
g-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/nova_wait_for_compute_service.log', '--env=__OS_DEBUG=false', '--env=TRIPLEO_CONFIG_HASH=ba8c4fb393
7f19883243859e5a49a26f-ba8c4fb3937f19883243859e5a49a26f', '--net=host', '--user=nova', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro',
'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pk
i/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/e
tc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/
puppet:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro', '--volume
=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/', '--cpuset-cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39', '172.20.128.1:8787/tripleostein/centos-binary-nova-compute:current-tripleo', '/container-config
-scripts/pyshim.sh', '/container-config-scripts/nova_wait_for_compute_service.py']. [1]",
        "stdout: WARNING:keystoneauth.identity.generic.base:Failed to discover available identity versions when contacting http://172.20.120.209:5000/v3. Attempting to p
arse version from URL.",
        "ERROR:nova_wait_for_compute_service:Error while waiting for nova-compute service to register",
        "Traceback (most recent call last):",
        " File \"/container-config-scripts/nova_wait_for_compute_service.py\", line 102, in <module>",
        " service_list = nova.services.list(binary='nova-compute')",
 " File \"/usr/lib/python2.7/site-packages/novaclient/v2/services.py\", line 52, in list",
        " return self._list(url, \"services\")",
        " File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 254, in _list",
        " resp, body = self.api.client.get(url)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 375, in get",
        " return self.request(url, 'GET', **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 72, in request",
        " **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 534, in request",
        " resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 237, in request",
        " return self.session.request(url, method, **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 704, in request",
        " auth_headers = self.get_auth_headers(auth)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 1097, in get_auth_headers",
        " return auth.get_headers(self, **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 95, in get_headers",
        " token = self.get_token(session)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 88, in get_token",
        " return self.get_access(session).auth_token",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access",
        " self.auth_ref = self.get_auth_ref(session)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 208, in get_auth_ref",
        " return self._plugin.get_auth_ref(session, **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py\", line 178, in get_auth_ref",
        " authenticated=False, log=False, **rkwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 1045, in post",
        " return self.request(url, 'POST', **kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 835, in request",
        " resp = send(**kwargs)",
        " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 942, in _send_request",
        " raise exceptions.ConnectFailure(msg)",
        "ConnectFailure: Unable to establish connection to http://172.20.120.209:5000/v3/auth/tokens: HTTPConnectionPool(host='172.20.120.209', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed2df59d0>: Failed to establish a new connection: [Errno 113] No route to host',))",
        "ConnectFailure: Unable to establish connection to http://172.20.120.209:5000/v3/auth/tokens: HTTPConnectionPool(host='172.20.120.209', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed2d59810>: Failed to establish a new connection: [Errno 113] No route to host',))",
        "ConnectFailure: Unable to establish connection to http://172.20.120.209:5000/v3/auth/tokens: HTTPConnectionPool(host='172.20.120.209', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed2d59710>: Failed to establish a new connection: [Errno 113] No route to host',))"

Actual result
=============
should containers started and deploy completed

Environment
===============
- 3 controller
- 1 compute
- all server HP g8 dl380
- VLAN 138: valid IP vlan
- VLAN 142: in valid IP vlan (for internal api)
- VLAN 143: in valid IP vlan (for provisioning)
- all VLAN defined on cisco switch.

Logs & Configs
================
- vim environment
parameter_defaults:
  ControllerCount: 3
  ComputeCount: 1
  NeutronEnableDVR: true
  SELinuxMode: permissive
  PythonInterpreter: /usr/bin/python3
  ControllerExtraConfig:
     rabbitmq::loopback_users: []
------------------------------------------------------------------------------------------
- vim templates/network_data.yaml

- name: InternalApi
  name_lower: internal_api
  vip: true
  vlan: 142
  ip_subnet: '172.20.120.0/24'
  allocation_pools: [{'start': '172.20.120.4', 'end': '172.20.120.250'}]
  mtu: 1500
- name: External
  vip: true
  name_lower: external
  vlan: 138
  ip_subnet: '92.114.18.160/27'
  allocation_pools: [{'start': '92.114.18.167', 'end': '92.114.18.190'}]
  gateway_ip: '92.114.18.161'
  mtu: 1500
---------------------------------------------------------------------------------
-roles: controller , compute
-container cli: podman
- provisioning CIDR: 172.20.128.0/24

[Brendan Shephard (bshephar)]

Hi, the container is failing to start because it can't reach the Keystone endpoint:
ConnectFailure: Unable to establish connection to http://172.20.120.209:5000/v3/auth/tokens:

Are you able to verify that this node has connectivity to that address?

So from overcloud-novacompute-0:
~~~
curl http://172.20.120.209:5000/v3/auth/tokens
~~~

If not, can you ping that address?