failed to deploy because can't access mariadb service through vip

[MarginHu]

Hi Guys,

I failed to deploy because it can't access mariadb service trough VIP. kolla-ansible reported the following info:

TASK [mariadb : Waiting for MariaDB service to be ready through VIP] ***********
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready through VIP (1 retries left).
fatal: [kode0]: FAILED! => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "mysql", "-h", "192.168.102.254", "-P", "3306", "-u", "haproxy", "-e", "show databases;"], "delta": "0:00:00.107514", "end": "2017-02-28 16:19:56.571593", "failed": true, "rc": 1, "start": "2017-02-28 16:19:56.464079", "stderr": "ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.102.254' (113 \"No route to host\")", "stdout": "", "stdout_lines": [], "warnings": []}
        to retry, use: --limit @/opt/kolla-ansible-4.0.0.0rc1/ansible/site.retry

PLAY RECAP *********************************************************************
kode0 : ok=60 changed=25 unreachable=0 failed=1
kode1 : ok=44 changed=22 unreachable=0 failed=1
kode2 : ok=44 changed=22 unreachable=0 failed=1
kode3 : ok=36 changed=0 unreachable=0 failed=0
kode4 : ok=5 changed=0 unreachable=0 failed=0
kode5 : ok=5 changed=0 unreachable=0 failed=0

Command failed ansible-playbook -i inventory/bgi-openstack -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /opt/kolla-ansible-4.0.0.0rc1/ansible/site.yml
[root@kola ansible]# ping 192.168.102.254
PING 192.168.102.254 (192.168.102.254) 56(84) bytes of data.
64 bytes from 192.168.102.254: icmp_seq=1 ttl=63 time=0.548 ms
64 bytes from 192.168.102.254: icmp_seq=2 ttl=63 time=0.455 ms
^C
--- 192.168.102.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.455/0.501/0.548/0.051 ms

[root@kode0 network-scripts]# ssh 192.168.102.254 -p 3306
ssh: connect to host 192.168.102.254 port 3306: No route to host

vip 192.168.102.254 is located on network node "kode3". I found there isn't mariadb service on "kode3"
Is it normal or abnormal ?

[root@kode3 ~]# ip a | grep 254
    inet 192.168.102.254/32 scope global eth1
    inet 192.168.101.254/32 scope global eth2
[root@kode3 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d479f4796403 192.168.103.16:5000/bgi/centos-binary-keepalived:ocata-rc1-1 "kolla_start" 6 hours ago Up 6 hours keepalived
7e07a3d1c0bf 192.168.103.16:5000/bgi/centos-binary-haproxy:ocata-rc1-1 "kolla_start" 6 hours ago Up 6 hours haproxy
17a1676f739e 192.168.103.16:5000/bgi/centos-binary-cron:ocata-rc1-1 "kolla_start" 7 hours ago Up 7 hours cron
5f28bc3c4c9e 192.168.103.16:5000/bgi/centos-binary-kolla-toolbox:ocata-rc1-1 "kolla_start" 7 hours ago Up 7 hours kolla_toolbox
e77ef89ab414 192.168.103.16:5000/bgi/centos-binary-fluentd:ocata-rc1-1 "kolla_start" 7 hours ago Up 7 hours fluentd
[root@kode3 ~]#

[MarginHu (margin2017)]

kode0 is control node, kode3 is network node.

[root@kode0 ~]# telnet 192.168.102.254 3306
Trying 192.168.102.254...
telnet: connect to address 192.168.102.254: No route to host
[root@kode0 ~]# ping 192.168.101.254
PING 192.168.101.254 (192.168.101.254) 56(84) bytes of data.
64 bytes from 192.168.101.254: icmp_seq=1 ttl=64 time=0.608 ms
^C
--- 192.168.101.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.608/0.608/0.608/0.000 ms
[root@kode0 ~]# ping 192.168.102.254
PING 192.168.102.254 (192.168.102.254) 56(84) bytes of data.
64 bytes from 192.168.102.254: icmp_seq=1 ttl=64 time=0.371 ms
^C
--- 192.168.102.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.371/0.371/0.371/0.000 ms
[root@kode0 ~]# telnet 192.168.102.254 3306
Trying 192.168.102.254...
telnet: connect to address 192.168.102.254: No route to host
[root@kode0 ~]#
[root@kode0 ~]# ip r
default via 192.168.122.1 dev eth5
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
169.254.0.0/16 dev eth2 scope link metric 1004
169.254.0.0/16 dev eth3 scope link metric 1005
169.254.0.0/16 dev eth4 scope link metric 1006
169.254.0.0/16 dev eth5 scope link metric 1007
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.101.0/24 dev eth2 proto kernel scope link src 192.168.101.20
192.168.102.0/24 dev eth1 proto kernel scope link src 192.168.102.20
192.168.103.0/24 dev eth0 proto kernel scope link src 192.168.103.20
192.168.104.0/24 dev eth3 proto kernel scope link src 192.168.104.20
192.168.106.0/24 dev eth4 proto kernel scope link src 192.168.106.20
192.168.122.0/24 dev eth5 proto kernel scope link src 192.168.122.20
[root@kode0 ~]#

[root@kode3 ~]# telnet 192.168.102.254 3306
Trying 192.168.102.254...
Connected to 192.168.102.254.
Escape character is '^]'.
Connection closed by foreign host.
[root@kode3 ~]#

[Eduardo Gonzalez (egonzalez90)]

Discussed in IRC, the issue was firewall blocking connections to port 3306.
Marking bug as invalid.