failed to deploy because haproxy Waiting for virtual IP to appear

[MarginHu]

Hi Guys,

I met a issue when deploy openstack with multinode , network node didn't get kolla_internal_vip_address though keepalived and haproxy container has been running.

1.kolla-ansible report the following info.

 TASK [haproxy : Waiting for virtual IP to appear] ******************************

fatal: [kode3]: FAILED! => {"changed": false, "elapsed": 301, "failed": true, "msg": "Timeout when waiting for 192.168.102.254:3306"}
        to retry, use: --limit @/opt/kolla-ansible-4.0.0.0rc1/ansible/site.retry

PLAY RECAP *********************************************************************
kode0 : ok=5 changed=0 unreachable=0 failed=0
kode1 : ok=5 changed=0 unreachable=0 failed=0
kode2 : ok=5 changed=0 unreachable=0 failed=0
kode3 : ok=35 changed=4 unreachable=0 failed=1
kode4 : ok=5 changed=0 unreachable=0 failed=0
kode5 : ok=5 changed=0 unreachable=0 failed=0

Command failed ansible-playbook -i inventory/bgi-openstack -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /opt/kolla-ansible-4.0.0.0rc1/ansible/site.yml

2.kolla version:4.0.0.0rc1

##### System Identification #####
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
##### Docker Version #####
Docker version 1.12.6, build 78d1802

[MarginHu (margin2017)]

[root@kode3 ~]# docker logs d479f4796403
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting file /etc/keepalived/keepalived.conf
INFO:__main__:Coping file from /var/lib/kolla/config_files/keepalived.conf to /etc/keepalived/keepalived.conf
INFO:__main__:Setting file /etc/keepalived/keepalived.conf owner to root:root
INFO:__main__:Setting file /etc/keepalived/keepalived.conf permission to 0644
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/keepalived -nld -p /run/keepalived.pid'
Keepalived[7]: Starting Keepalived v1.2.13 (11/05,2016)

Starting Healthcheck child process, pid=13
Initializing ipvs 2.6
Keepalived_healthcheckers[13]: Initializing ipvs 2.6
Keepalived[7]: Starting Healthcheck child process, pid=13
Starting VRRP child process, pid=14
Keepalived[7]: Starting VRRP child process, pid=14
Registering Kernel netlink reflector
Keepalived_healthcheckers[13]: Registering Kernel netlink reflector
Registering Kernel netlink command channel
Keepalived_healthcheckers[13]: Registering Kernel netlink command channel
Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_healthcheckers[13]: Opening file '/etc/keepalived/keepalived.conf'.
Registering Kernel netlink reflector
Keepalived_vrrp[14]: Registering Kernel netlink reflector
Registering Kernel netlink command channel
Keepalived_vrrp[14]: Registering Kernel netlink command channel
Configuration is using : 6060 Bytes
Keepalived_healthcheckers[13]: Configuration is using : 6060 Bytes
Registering gratuitous ARP shared channel
Keepalived_vrrp[14]: Registering gratuitous ARP shared channel
Opening file '/etc/keepalived/keepalived.conf'.
Keepalived_vrrp[14]: Opening file '/etc/keepalived/keepalived.conf'.
Truncating auth_pass to 8 characters
Keepalived_vrrp[14]: Truncating auth_pass to 8 characters
Configuration is using : 64930 Bytes
Keepalived_vrrp[14]: Configuration is using : 64930 Bytes
------< Global definitions >------
Keepalived_vrrp[14]: ------< Global definitions >------
 Router ID = kode3.genomics.cn
Keepalived_vrrp[14]: Router ID = kode3.genomics.cn
 VRRP IPv4 mcast group = 224.0.0.18
Keepalived_vrrp[14]: VRRP IPv4 mcast group = 224.0.0.18
 VRRP IPv6 mcast group = 224.0.0.18
Keepalived_vrrp[14]: VRRP IPv6 mcast group = 224.0.0.18
 SNMP Trap disabled
Keepalived_vrrp[14]: SNMP Trap disabled
------< VRRP Topology >------
Keepalived_vrrp[14]: ------< VRRP Topology >------
 VRRP Instance = kolla_internal_vip_51
Keepalived_vrrp[14]: VRRP Instance = kolla_internal_vip_51
   Want State = BACKUP
Keepalived_vrrp[14]: Want State = BACKUP
   Runing on device = eth1
Keepalived_vrrp[14]: Runing on device = eth1
   Virtual Router ID = 51
Keepalived_vrrp[14]: Virtual Router ID = 51
   Priority = 1
Keepalived_vrrp[14]: Priority = 1
   Advert interval = 1sec
Keepalived_vrrp[14]: Advert interval = 1sec
   Preempt disabled
Keepalived_vrrp[14]: Preempt disabled
   Authentication type = SIMPLE_PASSWORD
Keepalived_vrrp[14]: Authentication type = SIMPLE_PASSWORD
   Passw...

[MarginHu (margin2017)]

the log show "eth2 is down", the issue is resolved after I config eth2.

moreover, we can better kolla code.

[root@kola templates]# pwd
/opt/kolla-ansible-4.0.0.0rc1/ansible/roles/haproxy/templates
[root@kola templates]# vim keepalived.conf.j2

  track_interface {
        {{ kolla_external_vip_interface }}
    }

eth2 is the kolla_external_vip_interface, so it requires eth2 is up and has been configured before keepalived process starting.

but prechecks hasn't this check, I think we should add a checking in precheck code.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

[Eduardo Gonzalez (egonzalez90)]

This issue as fixed adding prechecks in haproxy to ensure external interfaces are UP.
https://review.openstack.org/#/c/465029/