kolla-ansible deploy fails when waiting for mariadb

[fortunepickle]

kolla version : 3.0.0.0b1-197-gc06ae54

What am i trying to do: Deploy openstack using kolla, 'all-in-one' deployment type. Followed the instructions in http://docs.openstack.org/developer/kolla/quickstart.html.

The error: In short, 'kolla-ansible deploy' fails when waiting for MariaDB service to be ready. Here's a direct output with '-vvvv':

FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).Result was: {"attempts": 8, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_zxclHI/ansible_module_wait_for.py\", line 537, in <module>\n main()\n File \"/tmp/ansible_zxclHI/ansible_module_wait_for.py\", line 478, in main\n response = s.recv(1024)\nsocket.error: [Errno 104] Connection reset by peer\n", "failed": true, "invocation": {"module_name": "wait_for"}, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_zxclHI/ansible_module_wait_for.py\", line 537, in <module>\n main()\n File \"/tmp/ansible_zxclHI/ansible_module_wait_for.py\", line 478, in main\n response = s.recv(1024)\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false, "retries": 10}
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1467900934.25-271634297002083 `" && echo ansible-tmp-1467900934.25-271634297002083="` echo $HOME/.ansible/tmp/ansible-tmp-1467900934.25-271634297002083 `" ) && sleep 0'
<localhost> PUT /tmp/tmpu_28P6 TO /home/antonio/.ansible/tmp/ansible-tmp-1467900934.25-271634297002083/wait_for
<localhost> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-hwhoyeiatadmjgeqitdpinjrmfsnncgb; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/antonio/.ansible/tmp/ansible-tmp-1467900934.25-271634297002083/wait_for; rm -rf "/home/antonio/.ansible/tmp/ansible-tmp-1467900934.25-271634297002083/" > /dev/null 2>&1'"'"' && sleep 0'
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).Result was: {"attempts": 9, "elapsed": 60, "failed": true, "invocation": {"module_args": {"connect_timeout": 1, "delay": 0, "exclude_hosts": null, "host": "192.168.8.17", "path": null, "port": 3306, "search_regex": "MariaDB", "state": "started", "timeout": 60}, "module_name": "wait_for"}, "msg": "Timeout when waiting for search string MariaDB in 192.168.8.17:3306", "retries": 10}
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1467901000.42-114761447782301 `" && echo ansible-tmp-1467901000.42-114761447782301="` echo $HOME/.ansible/tmp/ansible-tmp-1467901000.42-114761447782301 `" ) && sleep 0'
<localhost> PUT /tmp/tmpXddRj0 TO /home/antonio/.ansible/tmp/ansible-tmp-1467901000.42-114761447782301/wait_for
<localhost> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-xijanhhpjwcgqnahnjisyurgttpnarws; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/antonio/.ansible/tmp/ansible-tmp-1467901000.42-114761447782301/wait_for; rm -rf "/home/antonio/.ansible/tmp/ansible-tmp-1467901000.42-114761447782301/" > /dev/null 2>&1'"'"' && sleep 0'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_GJZGNy/ansible_module_wait_for.py", line 537, in <module>
    main()
  File "/tmp/ansible_GJZGNy/ansible_module_wait_for.py", line 478, in main
    response = s.recv(1024)
socket.error: [Errno 104] Connection reset by peer

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "wait_for"}, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_GJZGNy/ansible_module_wait_for.py\", line 537, in <module>\n main()\n File \"/tmp/ansible_GJZGNy/ansible_module_wait_for.py\", line 478, in main\n response = s.recv(1024)\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}
 [ERROR]: Could not create retry file '/usr/local/share/kolla/ansible/site.retry'. The error was:
[Errno 13] Permission denied: '/usr/local/share/kolla/ansible/site.retry'

Here's the content of the mariadb container logs:

7/7/2016 2:27:31 PMINFO:__main__:Kolla config strategy set to: COPY_ALWAYS
7/7/2016 2:27:31 PMINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
7/7/2016 2:27:31 PMINFO:__main__:Validating config file
7/7/2016 2:27:31 PMINFO:__main__:Copying service configuration files
7/7/2016 2:27:31 PMINFO:__main__:Removing existing destination: /etc/mysql/my.cnf
7/7/2016 2:27:31 PMINFO:__main__:Copying /var/lib/kolla/config_files/galera.cnf to /etc/mysql/my.cnf
7/7/2016 2:27:31 PMINFO:__main__:Setting permissions for /etc/mysql/my.cnf
7/7/2016 2:27:31 PMINFO:__main__:Removing existing destination: /usr/local/bin/wsrep-notify.sh
7/7/2016 2:27:31 PMINFO:__main__:Copying /var/lib/kolla/config_files/wsrep-notify.sh to /usr/local/bin/wsrep-notify.sh
7/7/2016 2:27:31 PMINFO:__main__:Setting permissions for /usr/local/bin/wsrep-notify.sh
7/7/2016 2:27:31 PMINFO:__main__:Writing out command to execute
7/7/2016 2:27:31 PMRunning command: '/usr/bin/mysqld_safe'
7/7/2016 2:27:31 PM160707 14:27:31 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'.
7/7/2016 2:27:31 PM160707 14:27:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
7/7/2016 2:27:31 PM160707 14:27:31 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.rqhfV2' --pid-file='/var/lib/mysql//slfy79-recover.pid'
7/7/2016 2:27:34 PM160707 14:27:34 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
7/7/2016 2:27:34 PM160707 14:27:34 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended

Please advise if further information is required.

Thanks!

[zhubingbing (zhubingbing)]

I nend more information，for exemple /etc/kolla/globals.yml