Prechecks fails when checking if kolla_internal_vip_address is in the same network as network_interface on all nodes

[Ahmad Al-Shishtawy]

TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] ***
fatal: [opst5] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst3] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst2] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst1] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst4] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst9] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst6] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst8] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst7] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst10] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])

FATAL: all hosts have already failed -- aborting

$ ip a
...
4: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 24:6e:96:07:c2:70 brd ff:ff:ff:ff:ff:ff
    inet 10.0.112.61/23 brd 10.0.113.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe07:c270/64 scope link
       valid_lft forever preferred_lft forever

globals.yml
...
kolla_internal_vip_address: "10.0.112.60"
network_interface: "em1"

[Swapnil Kulkarni (coolsvap-deactivatedaccount)]

Which globals.yml you are referring to?

Please provide the complete output of prechecks command

[Vikram Hosakote (vhosakot)]

On each of the 10 nodes (opst1 - opst10), can you check if kolla_internal_vip_address (10.0.112.60) is in the same network range (10.0.112.61/23, which ranges 10.0.112.0 - 10.0.113.255) as network_interface (em1) ?

[Ahmad Al-Shishtawy (alshishtawy)]

I'm referring to /etc/kolla/globals.yml on the deployment host

Yes, it is on the same network on all hosts.
IPs on em1 on opst1-opst10 are 10.0.112.61 - 10.0.112.70

[Brandon B. Jozsa (bjozsa)]

i seem to be experiencing the same on an AIO deployment as well.

regardless of the config in /etc/kolla/globals.yml i receive the same error:

TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] ***
fatal: [localhost] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])

FATAL: all hosts have already failed -- aborting

[Brandon B. Jozsa (bjozsa)]

I have to retract my comments above and the errors I was experiencing. Perhaps the other user was having more complex related issues, however mine was relatively simple once I discovered that I was running the incorrect version of Ansible. Since Kolla is highly opinionated about which version of Ansible is being used (and understandably-so), once I removed the previous version of ansible and used 1.9.4 (pip install ansible==1.9.4) everything started working just fine. If you use Ansible version 2.0 or higher, you will get the errors I mentioned earlier.

Even if this isn't related exactly to this issue...at least it may help someone trying to search for an answers.

[Derek Ditch (dcode)]

I ran into this issue as well. The issue actually stems from the fact that "ip" is not on the user's PATH when using the command module of Ansible. This isn't added to the user's PATH until the shell loads the environment.

I would recommend making the following change (/sbin would work too, since this is a symlink to /usr/sbin since CentOS 7. Not sure the full path on Ubuntu):

In /usr/share/kolla/ansible/roles/prechecks/tasks/port_checks.yml:460
- command: ip -4 -o addr show dev {{ network_interface }}
+ command: /usr/sbin/ip -4 -o addr show dev {{ network_interface }}

[DIY (zhaowei1)]

I have same problem for this,but it works well in all-in-one mode,it does not work in multinode mode,my ansible is 1.9.4 ,for liberty,and the ip is in the same network.

after change the port_checks.yml like Derek Ditch,it does not work.

the result of "kolla-ansible prechecks -i /root/kolla/ansible/inventory/multinode",has no failed,but has only one unreachable.

[Vladislav Belogrudov (vlad-belogrudov)]

i have tested ubuntu 14.04 hosts (minimal install) - no problem with aio/multinode

[Vladislav Belogrudov (vlad-belogrudov)]

my setup is ansible 2, minimal ubuntu 14.04. Pre-check playbook just works, no issues with 'ip' or remote hosts.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladislav Belogrudov (<email address hidden>) on branch: master
Review: https://review.openstack.org/385375
Reason: https://review.openstack.org/#/c/386654/
to follow up

[Christian Berendt (berendt)]

This was probably fixed with https://review.openstack.org/#/c/386654/.