Prechecks fails when checking if kolla_internal_vip_address is in the same network as network_interface on all nodes

Bug #1571612 reported by Ahmad Al-Shishtawy on 2016-04-18
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
High
Unassigned

Bug Description

TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] ***
fatal: [opst5] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst3] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst2] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst1] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst4] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst9] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst6] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst8] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst7] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])
fatal: [opst10] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])

FATAL: all hosts have already failed -- aborting

$ ip a
...
4: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 24:6e:96:07:c2:70 brd ff:ff:ff:ff:ff:ff
    inet 10.0.112.61/23 brd 10.0.113.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe07:c270/64 scope link
       valid_lft forever preferred_lft forever

globals.yml
...
kolla_internal_vip_address: "10.0.112.60"
network_interface: "em1"

Swapnil Kulkarni (coolsvap) wrote :

Which globals.yml you are referring to?

Please provide the complete output of prechecks command

Steven Dake (sdake) on 2016-04-20
Changed in kolla:
status: New → Triaged
importance: Undecided → Medium
milestone: none → newton-1
importance: Medium → Low
importance: Low → High
Vikram Hosakote (vhosakot) wrote :

On each of the 10 nodes (opst1 - opst10), can you check if kolla_internal_vip_address (10.0.112.60) is in the same network range (10.0.112.61/23, which ranges 10.0.112.0 - 10.0.113.255) as network_interface (em1) ?

I'm referring to /etc/kolla/globals.yml on the deployment host

Yes, it is on the same network on all hosts.
IPs on em1 on opst1-opst10 are 10.0.112.61 - 10.0.112.70

Brandon B. Jozsa (bjozsa) wrote :

i seem to be experiencing the same on an AIO deployment as well.

regardless of the config in /etc/kolla/globals.yml i receive the same error:

TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] ***
fatal: [localhost] => error while evaluating conditional: '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])

FATAL: all hosts have already failed -- aborting

Brandon B. Jozsa (bjozsa) wrote :

I have to retract my comments above and the errors I was experiencing. Perhaps the other user was having more complex related issues, however mine was relatively simple once I discovered that I was running the incorrect version of Ansible. Since Kolla is highly opinionated about which version of Ansible is being used (and understandably-so), once I removed the previous version of ansible and used 1.9.4 (pip install ansible==1.9.4) everything started working just fine. If you use Ansible version 2.0 or higher, you will get the errors I mentioned earlier.

Even if this isn't related exactly to this issue...at least it may help someone trying to search for an answers.

Derek Ditch (dcode) wrote :

I ran into this issue as well. The issue actually stems from the fact that "ip" is not on the user's PATH when using the command module of Ansible. This isn't added to the user's PATH until the shell loads the environment.

I would recommend making the following change (/sbin would work too, since this is a symlink to /usr/sbin since CentOS 7. Not sure the full path on Ubuntu):

In /usr/share/kolla/ansible/roles/prechecks/tasks/port_checks.yml:460
- command: ip -4 -o addr show dev {{ network_interface }}
+ command: /usr/sbin/ip -4 -o addr show dev {{ network_interface }}

tags: added: retriage
Changed in kolla:
milestone: newton-1 → newton-2
DIY (zhaowei1) wrote :

I have same problem for this,but it works well in all-in-one mode,it does not work in multinode mode,my ansible is 1.9.4 ,for liberty,and the ip is in the same network.

after change the port_checks.yml like Derek Ditch,it does not work.

the result of "kolla-ansible prechecks -i /root/kolla/ansible/inventory/multinode",has no failed,but has only one unreachable.

Changed in kolla:
milestone: newton-2 → newton-3
Hiroki Ito (hrito) on 2016-07-31
Changed in kolla:
assignee: nobody → Hiroki Ito (hrito)
Changed in kolla:
milestone: newton-3 → newton-rc1

i have tested ubuntu 14.04 hosts (minimal install) - no problem with aio/multinode

my setup is ansible 2, minimal ubuntu 14.04. Pre-check playbook just works, no issues with 'ip' or remote hosts.

Changed in kolla:
milestone: newton-rc1 → newton-rc2
Changed in kolla:
status: Triaged → Confirmed
Steven Dake (sdake) on 2016-10-13
Changed in kolla:
milestone: newton-rc2 → ocata-1

Change abandoned by Vladislav Belogrudov (<email address hidden>) on branch: master
Review: https://review.openstack.org/385375
Reason: https://review.openstack.org/#/c/386654/
to follow up

Changed in kolla:
milestone: ocata-1 → ocata-2
Changed in kolla:
milestone: ocata-2 → ocata-3
Christian Berendt (berendt) wrote :

This was probably fixed with https://review.openstack.org/#/c/386654/.

Changed in kolla:
assignee: Hiroki Ito (hrito) → nobody
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers