Heat deployment fails with TLS enabled:
TASK [heat : Running Heat bootstrap container] *********************************
task path: /home/zuul/src/git.openstack.org/openstack/kolla-ansible/ansible/roles/heat/tasks/bootstrap_service.yml:2
<104.130.69.215> ESTABLISH SSH CONNECTION FOR USER: kolla
<104.130.69.215> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c 104.130.69.215 '/bin/sh -c '"'"'echo ~kolla && sleep 0'"'"''
<104.130.69.215> (0, '/home/kolla\n', '')
<104.130.69.215> ESTABLISH SSH CONNECTION FOR USER: kolla
<104.130.69.215> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c 104.130.69.215 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079 `" && echo ansible-tmp-1554289203.54-83594719489079="` echo /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079 `" ) && sleep 0'"'"''
<104.130.69.215> (0, 'ansible-tmp-1554289203.54-83594719489079=/home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079\n', '')
Using module file /home/zuul/src/git.openstack.org/openstack/kolla-ansible/ansible/library/kolla_docker.py
<104.130.69.215> PUT /home/zuul/.ansible/tmp/ansible-local-18687EGFzJE/tmpaxY4H9 TO /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/AnsiballZ_kolla_docker.py
<104.130.69.215> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c '[104.130.69.215]'
<104.130.69.215> (0, 'sftp> put /home/zuul/.ansible/tmp/ansible-local-18687EGFzJE/tmpaxY4H9 /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/AnsiballZ_kolla_docker.py\n', '')
<104.130.69.215> ESTABLISH SSH CONNECTION FOR USER: kolla
<104.130.69.215> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c 104.130.69.215 '/bin/sh -c '"'"'chmod u+x /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/ /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/AnsiballZ_kolla_docker.py && sleep 0'"'"''
<104.130.69.215> (0, '', '')
<104.130.69.215> ESTABLISH SSH CONNECTION FOR USER: kolla
<104.130.69.215> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c -tt 104.130.69.215 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-xdmpxadqoyygbrcnjqjuxjnsjejqgpxg; /usr/bin/python /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/AnsiballZ_kolla_docker.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<104.130.69.215> (1, '\r\n{"stdout": "2019-04-03 11:00:13.909 16 INFO migrate.versioning.api [-] 70 -> 71... \\u001b[00m\\n2019-04-03 11:00:14.212 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.213 16 INFO migrate.versioning.api [-] 71 -> 72... \\u001b[00m\\n2019-04-03 11:00:14.281 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.281 16 INFO migrate.versioning.api [-] 72 -> 73... \\u001b[00m\\n2019-04-03 11:00:14.361 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.362 16 INFO migrate.versioning.api [-] 73 -> 74... \\u001b[00m\\n2019-04-03 11:00:14.373 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.373 16 INFO migrate.versioning.api [-] 74 -> 75... \\u001b[00m\\n2019-04-03 11:00:14.384 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.384 16 INFO migrate.versioning.api [-] 75 -> 76... \\u001b[00m\\n2019-04-03 11:00:14.395 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.396 16 INFO migrate.versioning.api [-] 76 -> 77... \\u001b[00m\\n2019-04-03 11:00:14.406 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.406 16 INFO migrate.versioning.api [-] 77 -> 78... \\u001b[00m\\n2019-04-03 11:00:14.417 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.417 16 INFO migrate.versioning.api [-] 78 -> 79... \\u001b[00m\\n2019-04-03 11:00:14.550 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.550 16 INFO migrate.versioning.api [-] 79 -> 80... \\u001b[00m\\n2019-04-03 11:00:14.628 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.628 16 INFO migrate.versioning.api [-] 80 -> 81... \\u001b[00m\\n2019-04-03 11:00:14.640 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.640 16 INFO migrate.versioning.api [-] 81 -> 82... \\u001b[00m\\n2019-04-03 11:00:14.652 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.653 16 INFO migrate.versioning.api [-] 82 -> 83... \\u001b[00m\\n2019-04-03 11:00:14.665 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.665 16 INFO migrate.versioning.api [-] 83 -> 84... \\u001b[00m\\n2019-04-03 11:00:14.676 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.677 16 INFO migrate.versioning.api [-] 84 -> 85... \\u001b[00m\\n2019-04-03 11:00:14.687 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n2019-04-03 11:00:14.688 16 INFO migrate.versioning.api [-] 85 -> 86... \\u001b[00m\\n2019-04-03 11:00:14.793 16 INFO migrate.versioning.api [-] done\\u001b[00m\\n", "changed": true, "failed": true, "stderr": "+ sudo -E kolla_set_configs\\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\\nINFO:__main__:Copying service configuration files\\nINFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf\\nINFO:__main__:Setting permission for /etc/heat/heat.conf\\nINFO:__main__:Writing out command to execute\\n++ cat /run_command\\n+ CMD=heat-api\\n+ ARGS=\\n+ [[ ! -n \'\' ]]\\n+ . kolla_extend_start\\n++ [[ ! -d /var/log/kolla/heat ]]\\n++ mkdir -p /var/log/kolla/heat\\n+++ stat -c %a /var/log/kolla/heat\\n++ [[ 2755 != \\\\7\\\\5\\\\5 ]]\\n++ chmod 755 /var/log/kolla/heat\\n+++ whoami\\n++ [[ heat == \\\\r\\\\o\\\\o\\\\t ]]\\n++ . /usr/local/bin/kolla_heat_extend_start\\n+++ [[ -n 0 ]]\\n+++ heat-manage db_sync\\n++++ openstack domain list\\n++++ grep heat\\n++++ awk \'{print $4}\'\\nFailed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.\\nSSL exception connecting to https://169.254.169.11:5000/domains?: HTTPSConnectionPool(host=\'169.254.169.11\', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\\"bad handshake: Error([(\'SSL routines\', \'tls_process_server_certificate\', \'certificate verify failed\')],)\\",),))\\n+++ CURRENT_HEAT_DOMAIN_NAME=\\n+++ [[ heat_user_domain != \'\' ]]\\n+++ openstack domain create heat_user_domain\\nFailed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.\\nSSL exception connecting to https://169.254.169.11:5000/domains: HTTPSConnectionPool(host=\'169.254.169.11\', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\\"bad handshake: Error([(\'SSL routines\', \'tls_process_server_certificate\', \'certificate verify failed\')],)\\",),))\\n", "rc": 1, "invocation": {"module_args": {"tty": false, "tls_key": null, "image": "kolla/centos-source-heat-api:master", "labels": {"BOOTSTRAP": null}, "tls_verify": false, "tls_cacert": null, "auth_password": null, "graceful_timeout": 10, "environment": {"OS_PROJECT_NAME": "admin", "OS_USERNAME": "admin", "OS_IDENTITY_API_VERSION": "3", "HEAT_DOMAIN_ADMIN_PASSWORD": "5mWhvPyzTwmFHYrgZzUCRzPBuIcEtW1ovZIPoxvR", "OS_AUTH_URL": "http://169.254.169.10:35357", "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "OS_PASSWORD": "Hweg9zexXIt7kNY52IuDwOyNaR3tCorjlR4euw6v", "KOLLA_SERVICE_NAME": "bootstrap-heat", "KOLLA_BOOTSTRAP": null}, "auth_registry": null, "state": "running", "volumes_from": null, "security_opt": [], "privileged": false, "api_version": "auto", "remove_on_exit": true, "restart_retries": 10, "detach": false, "auth_username": null, "dimensions": {}, "name": "bootstrap_heat", "tls_cert": null, "cap_add": [], "restart_policy": "never", "auth_email": null, "command": null, "volumes": ["/etc/kolla//heat-api/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "kolla_logs:/var/log/kolla/"], "action": "start_container"}}, "msg": "Container exited with non-zero return code 1"}\r\n', 'Shared connection to 104.130.69.215 closed.\r\n')
<104.130.69.215> Failed to connect to the host via ssh: Shared connection to 104.130.69.215 closed.
<104.130.69.215> ESTABLISH SSH CONNECTION FOR USER: kolla
<104.130.69.215> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=kolla -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/765134a18c 104.130.69.215 '/bin/sh -c '"'"'rm -f -r /home/kolla/.ansible/tmp/ansible-tmp-1554289203.54-83594719489079/ > /dev/null 2>&1 && sleep 0'"'"''
<104.130.69.215> (0, '', '')
fatal: [primary -> 104.130.69.215]: FAILED! => {
"changed": true,
"invocation": {
"module_args": {
"action": "start_container",
"api_version": "auto",
"auth_email": null,
"auth_password": null,
"auth_registry": null,
"auth_username": null,
"cap_add": [],
"command": null,
"detach": false,
"dimensions": {},
"environment": {
"HEAT_DOMAIN_ADMIN_PASSWORD": "5mWhvPyzTwmFHYrgZzUCRzPBuIcEtW1ovZIPoxvR",
"KOLLA_BOOTSTRAP": null,
"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
"KOLLA_SERVICE_NAME": "bootstrap-heat",
"OS_AUTH_URL": "http://169.254.169.10:35357",
"OS_IDENTITY_API_VERSION": "3",
"OS_PASSWORD": "Hweg9zexXIt7kNY52IuDwOyNaR3tCorjlR4euw6v",
"OS_PROJECT_NAME": "admin",
"OS_USERNAME": "admin"
},
"graceful_timeout": 10,
"image": "kolla/centos-source-heat-api:master",
"labels": {
"BOOTSTRAP": null
},
"name": "bootstrap_heat",
"privileged": false,
"remove_on_exit": true,
"restart_policy": "never",
"restart_retries": 10,
"security_opt": [],
"state": "running",
"tls_cacert": null,
"tls_cert": null,
"tls_key": null,
"tls_verify": false,
"tty": false,
"volumes": [
"/etc/kolla//heat-api/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro",
"kolla_logs:/var/log/kolla/"
],
"volumes_from": null
}
},
"msg": "Container exited with non-zero return code 1",
"rc": 1,
"stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf\nINFO:__main__:Setting permission for /etc/heat/heat.conf\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD=heat-api\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/heat ]]\n++ mkdir -p /var/log/kolla/heat\n+++ stat -c %a /var/log/kolla/heat\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/heat\n+++ whoami\n++ [[ heat == \\r\\o\\o\\t ]]\n++ . /usr/local/bin/kolla_heat_extend_start\n+++ [[ -n 0 ]]\n+++ heat-manage db_sync\n++++ openstack domain list\n++++ grep heat\n++++ awk '{print $4}'\nFailed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.\nSSL exception connecting to https://169.254.169.11:5000/domains?: HTTPSConnectionPool(host='169.254.169.11', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))\n+++ CURRENT_HEAT_DOMAIN_NAME=\n+++ [[ heat_user_domain != '' ]]\n+++ openstack domain create heat_user_domain\nFailed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.\nSSL exception connecting to https://169.254.169.11:5000/domains: HTTPSConnectionPool(host='169.254.169.11', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))\n",
"stderr_lines": [
"+ sudo -E kolla_set_configs",
"INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json",
"INFO:__main__:Validating config file",
"INFO:__main__:Kolla config strategy set to: COPY_ALWAYS",
"INFO:__main__:Copying service configuration files",
"INFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf",
"INFO:__main__:Setting permission for /etc/heat/heat.conf",
"INFO:__main__:Writing out command to execute",
"++ cat /run_command",
"+ CMD=heat-api",
"+ ARGS=",
"+ [[ ! -n '' ]]",
"+ . kolla_extend_start",
"++ [[ ! -d /var/log/kolla/heat ]]",
"++ mkdir -p /var/log/kolla/heat",
"+++ stat -c %a /var/log/kolla/heat",
"++ [[ 2755 != \\7\\5\\5 ]]",
"++ chmod 755 /var/log/kolla/heat",
"+++ whoami",
"++ [[ heat == \\r\\o\\o\\t ]]",
"++ . /usr/local/bin/kolla_heat_extend_start",
"+++ [[ -n 0 ]]",
"+++ heat-manage db_sync",
"++++ openstack domain list",
"++++ grep heat",
"++++ awk '{print $4}'",
"Failed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.",
"SSL exception connecting to https://169.254.169.11:5000/domains?: HTTPSConnectionPool(host='169.254.169.11', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))",
"+++ CURRENT_HEAT_DOMAIN_NAME=",
"+++ [[ heat_user_domain != '' ]]",
"+++ openstack domain create heat_user_domain",
"Failed to contact the endpoint at https://169.254.169.11:5000 for discovery. Fallback to using that endpoint as the base url.",
"SSL exception connecting to https://169.254.169.11:5000/domains: HTTPSConnectionPool(host='169.254.169.11', port=5000): Max retries exceeded with url: /domains (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))"
],
Tested in: https:/