Comment 14 for bug 1846789

I was able to re-deploy with haproxy on the network nodes. I did hit an issue with the heat versioning API failure during bootstrap, this seems to be an intermittent problem. Performing an additional deploy got past the issue.

TASK [heat : Running Heat bootstrap container] ***********************************************************************************************************************************
fatal: [odt-rsd-ost-ctr-a1.example.com -> odt-rsd-ost-ctr-a1.example.com]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/heat/heat.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf\nINFO:__main__:Setting permission for /etc/heat/heat.conf\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD=heat-api\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/heat ]]\n++ mkdir -p /var/log/kolla/heat\n+++ stat -c %a /var/log/kolla/heat\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/heat\n+++ whoami\n++ [[ heat == \\r\\o\\o\\t ]]\n++ . /usr/local/bin/kolla_heat_extend_start\n+++ [[ -n 0 ]]\n+++ heat-manage db_sync\n++++ openstack domain list\n++++ awk '{print $4}'\n++++ grep heat\nFailed to contact the endpoint at http://10.239.195.200:5000 for discovery. Fallback to using that endpoint as the base url.\nNot Found (HTTP 404) (Request-ID: req-1e9bea57-85a9-4a84-ac4c-b1d17dde6c2e)\n+++ CURRENT_HEAT_DOMAIN_NAME=\n+++ [[ heat_user_domain != '' ]]\n+++ openstack domain create heat_user_domain\n+++ openstack user create --domain heat_user_domain heat_domain_admin --password Op3nSt@ck\nget() takes exactly 1 argument (2 given)\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/heat/heat.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf", "INFO:__main__:Setting permission for /etc/heat/heat.conf", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD=heat-api", "+ ARGS=", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/heat ]]", "++ mkdir -p /var/log/kolla/heat", "+++ stat -c %a /var/log/kolla/heat", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/heat", "+++ whoami", "++ [[ heat == \\r\\o\\o\\t ]]", "++ . /usr/local/bin/kolla_heat_extend_start", "+++ [[ -n 0 ]]", "+++ heat-manage db_sync", "++++ openstack domain list", "++++ awk '{print $4}'", "++++ grep heat", "Failed to contact the endpoint at http://10.239.195.200:5000 for discovery. Fallback to using that endpoint as the base url.", "Not Found (HTTP 404) (Request-ID: req-1e9bea57-85a9-4a84-ac4c-b1d17dde6c2e)", "+++ CURRENT_HEAT_DOMAIN_NAME=", "+++ [[ heat_user_domain != '' ]]", "+++ openstack domain create heat_user_domain", "+++ openstack user create --domain heat_user_domain heat_domain_admin --password Op3nSt@ck", "get() takes exactly 1 argument (2 given)"], "stdout": "2019-10-15 10:02:27.641 16 INFO migrate.versioning.api [-] 70 -> 71... \u001b[00m\n2019-10-15 10:02:27.920 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:27.921 16 INFO migrate.versioning.api [-] 71 -> 72... \u001b[00m\n2019-10-15 10:02:27.974 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:27.975 16 INFO migrate.versioning.api [-] 72 -> 73... \u001b[00m\n2019-10-15 10:02:28.042 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.042 16 INFO migrate.versioning.api [-] 73 -> 74... \u001b[00m\n2019-10-15 10:02:28.053 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.053 16 INFO migrate.versioning.api [-] 74 -> 75... \u001b[00m\n2019-10-15 10:02:28.064 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.065 16 INFO migrate.versioning.api [-] 75 -> 76... \u001b[00m\n2019-10-15 10:02:28.076 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.076 16 INFO migrate.versioning.api [-] 76 -> 77... \u001b[00m\n2019-10-15 10:02:28.087 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.087 16 INFO migrate.versioning.api [-] 77 -> 78... \u001b[00m\n2019-10-15 10:02:28.108 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.108 16 INFO migrate.versioning.api [-] 78 -> 79... \u001b[00m\n2019-10-15 10:02:28.224 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.224 16 INFO migrate.versioning.api [-] 79 -> 80... \u001b[00m\n2019-10-15 10:02:28.299 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.299 16 INFO migrate.versioning.api [-] 80 -> 81... \u001b[00m\n2019-10-15 10:02:28.310 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.311 16 INFO migrate.versioning.api [-] 81 -> 82... \u001b[00m\n2019-10-15 10:02:28.322 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.322 16 INFO migrate.versioning.api [-] 82 -> 83... \u001b[00m\n2019-10-15 10:02:28.335 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.335 16 INFO migrate.versioning.api [-] 83 -> 84... \u001b[00m\n2019-10-15 10:02:28.346 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.346 16 INFO migrate.versioning.api [-] 84 -> 85... \u001b[00m\n2019-10-15 10:02:28.367 16 INFO migrate.versioning.api [-] done\u001b[00m\n2019-10-15 10:02:28.367 16 INFO migrate.versioning.api [-] 85 -> 86... \u001b[00m\n2019-10-15 10:02:28.456 16 INFO migrate.versioning.api [-] done\u001b[00m\n+-------------+----------------------------------+\n| Field | Value |\n+-------------+----------------------------------+\n| description | |\n| enabled | True |\n| id | da687ea738f04a8480ccd1a86848b3d9 |\n| name | heat_user_domain |\n| tags | [] |\n+-------------+----------------------------------+\n", "stdout_lines": ["2019-10-15 10:02:27.641 16 INFO migrate.versioning.api [-] 70 -> 71... \u001b[00m", "2019-10-15 10:02:27.920 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:27.921 16 INFO migrate.versioning.api [-] 71 -> 72... \u001b[00m", "2019-10-15 10:02:27.974 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:27.975 16 INFO migrate.versioning.api [-] 72 -> 73... \u001b[00m", "2019-10-15 10:02:28.042 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.042 16 INFO migrate.versioning.api [-] 73 -> 74... \u001b[00m", "2019-10-15 10:02:28.053 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.053 16 INFO migrate.versioning.api [-] 74 -> 75... \u001b[00m", "2019-10-15 10:02:28.064 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.065 16 INFO migrate.versioning.api [-] 75 -> 76... \u001b[00m", "2019-10-15 10:02:28.076 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.076 16 INFO migrate.versioning.api [-] 76 -> 77... \u001b[00m", "2019-10-15 10:02:28.087 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.087 16 INFO migrate.versioning.api [-] 77 -> 78... \u001b[00m", "2019-10-15 10:02:28.108 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.108 16 INFO migrate.versioning.api [-] 78 -> 79... \u001b[00m", "2019-10-15 10:02:28.224 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.224 16 INFO migrate.versioning.api [-] 79 -> 80... \u001b[00m", "2019-10-15 10:02:28.299 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.299 16 INFO migrate.versioning.api [-] 80 -> 81... \u001b[00m", "2019-10-15 10:02:28.310 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.311 16 INFO migrate.versioning.api [-] 81 -> 82... \u001b[00m", "2019-10-15 10:02:28.322 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.322 16 INFO migrate.versioning.api [-] 82 -> 83... \u001b[00m", "2019-10-15 10:02:28.335 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.335 16 INFO migrate.versioning.api [-] 83 -> 84... \u001b[00m", "2019-10-15 10:02:28.346 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.346 16 INFO migrate.versioning.api [-] 84 -> 85... \u001b[00m", "2019-10-15 10:02:28.367 16 INFO migrate.versioning.api [-] done\u001b[00m", "2019-10-15 10:02:28.367 16 INFO migrate.versioning.api [-] 85 -> 86... \u001b[00m", "2019-10-15 10:02:28.456 16 INFO migrate.versioning.api [-] done\u001b[00m", "+-------------+----------------------------------+", "| Field | Value |", "+-------------+----------------------------------+", "| description | |", "| enabled | True |", "| id | da687ea738f04a8480ccd1a86848b3d9 |", "| name | heat_user_domain |", "| tags | [] |", "+-------------+----------------------------------+"]}