mariadb check task fails when mariadb running on a non-default tcp port

[Waldemar Znoinski]

1. mariadb port:
 grep -i mariadb_port ~/kolla/ansible/group_vars/all.yml
mariadb_port: "3307"

2. ansible deploy log:

TASK: [mariadb | Waiting for MariaDB service to be ready through VIP] *********
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835331.19-44871484768232 && echo $HOME/.ansible/tmp/ansible-tmp-1461835331.19-44871484768232']
<localhost> PUT /tmp/tmpNDGrep TO /root/.ansible/tmp/ansible-tmp-1461835331.19-44871484768232/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835331.19-44871484768232/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835331.19-44871484768232/ >/dev/null 2>&1']
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835341.34-92429332360651 && echo $HOME/.ansible/tmp/ansible-tmp-1461835341.34-92429332360651']
<localhost> PUT /tmp/tmpIAXcre TO /root/.ansible/tmp/ansible-tmp-1461835341.34-92429332360651/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835341.34-92429332360651/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835341.34-92429332360651/ >/dev/null 2>&1']
Result from run 1 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:22:21.509963', 'stdout': u'', 'changed': True, 'attempts': 1, 'start': '2016-04-28 10:22:21.416761', 'delta': '0:
00:00.093202', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835351.53-90729419472377 && echo $HOME/.ansible/tmp/ansible-tmp-1461835351.53-90729419472377']
<localhost> PUT /tmp/tmprJX0jE TO /root/.ansible/tmp/ansible-tmp-1461835351.53-90729419472377/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835351.53-90729419472377/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835351.53-90729419472377/ >/dev/null 2>&1']
Result from run 2 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:22:31.706053', 'stdout': u'', 'changed': True, 'attempts': 2, 'start': '2016-04-28 10:22:31.613907', 'delta': '0:
00:00.092146', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835361.73-156493388933683 && echo $HOME/.ansible/tmp/ansible-tmp-1461835361.73-156493388933683']
<localhost> PUT /tmp/tmpGeFUma TO /root/.ansible/tmp/ansible-tmp-1461835361.73-156493388933683/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835361.73-156493388933683/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835361.73-156493388933683/ >/dev/null 2>&1']
Result from run 3 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:22:41.923676', 'stdout': u'', 'changed': True, 'attempts': 3, 'start': '2016-04-28 10:22:41.821955', 'delta': '0:
00:00.101721', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835371.95-154493924910217 && echo $HOME/.ansible/tmp/ansible-tmp-1461835371.95-154493924910217']
<localhost> PUT /tmp/tmpMJ8GLz TO /root/.ansible/tmp/ansible-tmp-1461835371.95-154493924910217/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835371.95-154493924910217/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835371.95-154493924910217/ >/dev/null 2>&1']
Result from run 4 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:22:52.119615', 'stdout': u'', 'changed': True, 'attempts': 4, 'start': '2016-04-28 10:22:52.022028', 'delta': '0:
00:00.097587', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835382.15-72184742279380 && echo $HOME/.ansible/tmp/ansible-tmp-1461835382.15-72184742279380']
<localhost> PUT /tmp/tmpXHijq8 TO /root/.ansible/tmp/ansible-tmp-1461835382.15-72184742279380/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835382.15-72184742279380/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835382.15-72184742279380/ >/dev/null 2>&1']
Result from run 5 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:23:02.322019', 'stdout': u'', 'changed': True, 'attempts': 5, 'start': '2016-04-28 10:23:02.227745', 'delta': '0:
00:00.094274', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
<localhost> REMOTE_MODULE command docker exec mariadb mysql -h 10.237.214.148 -u haproxy -e 'show databases;'
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461835392.34-174772139930649 && echo $HOME/.ansible/tmp/ansible-tmp-1461835392.34-174772139930649']
<localhost> PUT /tmp/tmpyh5HMb TO /root/.ansible/tmp/ansible-tmp-1461835392.34-174772139930649/command
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1461835392.34-174772139930649/command; rm -rf /root/.ansible/tmp/ansible-tmp-1461835392.34-174772139930649/ >/dev/null 2>&1']
Result from run 6 is: {'cmd': ['docker', 'exec', 'mariadb', 'mysql', '-h', '10.237.214.148', '-u', 'haproxy', '-e', 'show databases;'], 'end': '2016-04-28 10:23:12.505798', 'stdout': u'', 'changed': True, 'attempts': 6, 'start': '2016-04-28 10:23:12.423150', 'delta': '0:
00:00.082648', 'stderr': 'ERROR 2003 (HY000): Can\'t connect to MySQL server on \'10.237.214.148\' (111 "Connection refused")', 'rc': 1, 'warnings': []}
failed: [localhost] => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "mysql", "-h", "10.237.214.148", "-u", "haproxy", "-e", "show databases;"], "delta": "0:00:00.082648", "end": "2016-04-28 10:23:12.505798", "failed": true, "rc": 1, "start": "20
16-04-28 10:23:12.423150", "stdout_lines": [], "warnings": []}
stderr: ERROR 2003 (HY000): Can't connect to MySQL server on '10.237.214.148' (111 "Connection refused")
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/site.retry

localhost : ok=72 changed=0 unreachable=0 failed=1

Command failed ansible-playbook -i ansible/inventory/multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /root/kolla/ansible/site.yml --verbose --verbose --verbose --verbose

it looks like patching check.yml fixed is:

 git diff check.yml
diff --git a/ansible/roles/mariadb/tasks/check.yml b/ansible/roles/mariadb/tasks/check.yml
index 61ff734..5389e78 100644
--- a/ansible/roles/mariadb/tasks/check.yml
+++ b/ansible/roles/mariadb/tasks/check.yml
@@ -1,6 +1,6 @@
 ---
 - name: Waiting for MariaDB service to be ready through VIP
- command: "docker exec mariadb mysql -h {{ kolla_internal_fqdn }} -u haproxy -e 'show databases;'"
+ command: "docker exec mariadb mysql -h {{ kolla_internal_fqdn }} -P {{ mariadb_port }} -u haproxy -e 'show databases;'"
   register: result
   until: result | success
   changed_when: False

[Waldemar Znoinski (wznoinsk)]

covered in https://bugs.launchpad.net/kolla/+bug/1576151