Description of problem:
Deploying OSP13 in the Red Hat Scale LAb with 3 controllers and 150 compute nodes, deployment fails in Step1 with the following error message.
overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 4de9d804-9db7-4904-b3f5-823b7117369f
status: CREATE_FAILED
status_reason: |
Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status
code: 2
deploy_stdout: |
...
"Error running ['docker', 'run', '--name', 'memcached', '--label', 'config_id=tripleo_step1', '--labe
l', 'container_name=memcached', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 1, \"co
mmand\": [\"/bin/bash\", \"-c\", \"source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m $
{CACHESIZE} -c ${MAXCONN} $OPTIONS >> /var/log/memcached.log 2>&1\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\",
\"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki
/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki
/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \
"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/m
emcached/etc/sysconfig/memcached:/etc/sysconfig/memcached:ro\", \"/var/log/containers/memcached:/var/log/\"], \"i
mage\": \"192.168.0.1:8787/rhosp13/openstack-memcached:2018-05-07.2\", \"net\": \"host\", \"restart\": \"always\"
, \"privileged\": false}', '--detach=true', '--net=host', '--privileged=false', '--restart=always', '--volume=/et
c/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/p
ki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--vo
lume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/ce
rt.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_k
nown_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/memcached/etc/sysconfig/mem
cached:/etc/sysconfig/memcached:ro', '--volume=/var/log/containers/memcached:/var/log/', '192.168.0.1:8787/rhosp1
3/openstack-memcached:2018-05-07.2', '/bin/bash', '-c', 'source /etc/sysconfig/memcached; /usr/bin/memcached -p $
{PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS >> /var/log/memcached.log 2>&1']. [125]",
"stdout: 6645bfdc5b90ce52921f1f8f875ac83a5b878b3d30e3293e5dbbbb39e4a5ae67",
"stderr: /usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:2
47: starting container process caused \"process_linux.go:364: container init caused \\\"rootfs_linux.go:54: mount
ing \\\\\\\"/etc/ssh/ssh_known_hosts\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/2e613821885a32bebdb894820
d7c2e38a86c7b2024e029a12819dd5aed722751/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/2e613821885a32bebdb894
820d7c2e38a86c7b2024e029a12819dd5aed722751/merged/etc/ssh/ssh_known_hosts\\\\\\\" caused \\\\\\\"not a directory\
\\\\\\"\\\"\""
Logging into the controllers we see all containers are exited for a similar reason
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b66e67777b4b 192.168.0.1:8787/rhosp13/openstack-mariadb:2018-05-07.2 "/bin/bash -c '/us..." 7 hours ago Exited (0) 7 hours ago mysql_image_tag
6645bfdc5b90 192.168.0.1:8787/rhosp13/openstack-memcached:2018-05-07.2 "/bin/bash -c 'sou..." 7 hours ago Created memcached
25cdfcf14239 192.168.0.1:8787/rhosp13/openstack-haproxy:2018-05-07.2 "/bin/bash -c '/us..." 7 hours ago Exited (0) 7 hours ago haproxy_image_tag
10e7c2b0878a 192.168.0.1:8787/rhosp13/openstack-mariadb:2018-05-07.2 "bash -ec 'if [ -e..." 7 hours ago Created mysql_bootstrap
822abff373f4 192.168.0.1:8787/rhosp13/openstack-redis:2018-05-07.2 "/bin/bash -c '/us..." 7 hours ago Exited (0) 7 hours ago redis_image_tag
81619c0ad428 192.168.0.1:8787/rhosp13/openstack-rabbitmq:2018-05-07.2 "/bin/bash -c '/us..." 7 hours ago Exited (0) 7 hours ago rabbitmq_image_tag
eccdf31c874a 192.168.0.1:8787/rhosp13/openstack-cinder-volume:2018-05-07.2 "/bin/bash -c '/us..." 7 hours ago Exited (0) 7 hours ago cinder_volume_image_tag
000560604e74 192.168.0.1:8787/rhosp13/openstack-rabbitmq:2018-05-07.2 "kolla_start" 7 hours ago Exited (0) 7 hours ago rabbitmq_bootstrap
573e30c4c8c2 192.168.0.1:8787/rhosp13/openstack-memcached:2018-05-07.2 "/bin/bash -c 'sou..." 7 hours ago Exited (0) 7 hours ago memcached_init_logs
8a5d446f88d2 192.168.0.1:8787/rhosp13/openstack-mariadb:2018-05-07.2 "chown -R mysql: /..." 7 hours ago Exited (0) 7 hours ago mysql_data_ownership
Version-Release number of selected component (if applicable):
Queens
How reproducible:
100% on this env
Steps to Reproduce:
1. Deploy OSP13
2.
3.
Actual results:
Deploy fails in step1 when creating controllers
Expected results:
Deployment should succeed
Additional info:
Fix proposed to branch: master
Review: https:/