devstack job failing with "Could not determine host ip address."

[Thomas Morin]

Some devstack jobs fail with "Could not determine host ip address.", but this does not seem to be related to their configuration because these job do not always fail.

For instance:

2017-08-31 12:26:24.605468 | +++ stackrc:source:815 : get_default_host_ip 10.0.0.0/22 172.24.4.0/24 '' '' inet
2017-08-31 12:26:24.606874 | +++ functions-common:get_default_host_ip:653 : local fixed_range=10.0.0.0/22
2017-08-31 12:26:24.608388 | +++ functions-common:get_default_host_ip:654 : local floating_range=172.24.4.0/24
2017-08-31 12:26:24.610086 | +++ functions-common:get_default_host_ip:655 : local host_ip_iface=
2017-08-31 12:26:24.611672 | +++ functions-common:get_default_host_ip:656 : local host_ip=
2017-08-31 12:26:24.613233 | +++ functions-common:get_default_host_ip:657 : local af=inet
2017-08-31 12:26:24.614656 | +++ functions-common:get_default_host_ip:660 : '[' -z '' -o '' == dhcp ']'
2017-08-31 12:26:24.615872 | +++ functions-common:get_default_host_ip:661 : host_ip=
2017-08-31 12:26:24.618107 | ++++ functions-common:get_default_host_ip:663 : ip -f inet route
2017-08-31 12:26:24.619790 | ++++ functions-common:get_default_host_ip:663 : head -1
2017-08-31 12:26:24.620001 | ++++ functions-common:get_default_host_ip:663 : awk '/default/ {print $5}'
2017-08-31 12:26:24.622804 | +++ functions-common:get_default_host_ip:663 : host_ip_iface=ens3
2017-08-31 12:26:24.624050 | +++ functions-common:get_default_host_ip:664 : local host_ips
2017-08-31 12:26:24.626182 | ++++ functions-common:get_default_host_ip:665 : LC_ALL=C
2017-08-31 12:26:24.626605 | ++++ functions-common:get_default_host_ip:665 : awk '/inet/ {split($2,parts,"/"); print parts[1]}'
2017-08-31 12:26:24.627500 | ++++ functions-common:get_default_host_ip:665 : sed /temporary/d
2017-08-31 12:26:24.628185 | ++++ functions-common:get_default_host_ip:665 : ip -f inet addr show ens3
2017-08-31 12:26:24.631686 | +++ functions-common:get_default_host_ip:665 : host_ips=10.0.1.159
2017-08-31 12:26:24.633228 | +++ functions-common:get_default_host_ip:666 : local ip
2017-08-31 12:26:24.634720 | +++ functions-common:get_default_host_ip:667 : for ip in '$host_ips'
2017-08-31 12:26:24.637339 | +++ functions-common:get_default_host_ip:674 : [[ inet == \i\n\e\t\6 ]]
2017-08-31 12:26:24.639288 | +++ functions-common:get_default_host_ip:678 : address_in_net 10.0.1.159 10.0.0.0/22
2017-08-31 12:26:24.640971 | +++ functions-common:address_in_net:2288 : local ip=10.0.1.159
2017-08-31 12:26:24.642569 | +++ functions-common:address_in_net:2289 : local range=10.0.0.0/22
2017-08-31 12:26:24.644120 | +++ functions-common:address_in_net:2290 : local masklen=22
2017-08-31 12:26:24.645743 | +++ functions-common:address_in_net:2291 : local network
2017-08-31 12:26:24.648271 | +++++ functions-common:address_in_net:2292 : cidr2netmask 22
2017-08-31 12:26:24.650061 | +++++ functions-common:cidr2netmask:2310 : local 'maskpat=255 255 255 255'
2017-08-31 12:26:24.651340 | +++++ functions-common:cidr2netmask:2311 : local 'maskdgt=254 252 248 240 224 192 128'
2017-08-31 12:26:24.653080 | +++++ functions-common:cidr2netmask:2312 : set -- 255 255 252
2017-08-31 12:26:24.654265 | +++++ functions-common:cidr2netmask:2313 : echo 255.255.252.0
2017-08-31 12:26:24.655943 | ++++ functions-common:address_in_net:2292 : maskip 10.0.0.0 255.255.252.0
2017-08-31 12:26:24.657332 | ++++ functions-common:maskip:2367 : local ip=10.0.0.0
2017-08-31 12:26:24.658760 | ++++ functions-common:maskip:2368 : local mask=255.255.252.0
2017-08-31 12:26:24.660632 | ++++ functions-common:maskip:2369 : local l=10.0.0
2017-08-31 12:26:24.662449 | ++++ functions-common:maskip:2369 : local r=0.0.0
2017-08-31 12:26:24.664118 | ++++ functions-common:maskip:2369 : local n=255.255.252
2017-08-31 12:26:24.665713 | ++++ functions-common:maskip:2369 : local m=255.252.0
2017-08-31 12:26:24.667446 | ++++ functions-common:maskip:2370 : local subnet
2017-08-31 12:26:24.669262 | ++++ functions-common:maskip:2371 : subnet=10.0.0.0
2017-08-31 12:26:24.670829 | ++++ functions-common:maskip:2372 : echo 10.0.0.0
2017-08-31 12:26:24.672509 | +++ functions-common:address_in_net:2292 : network=10.0.0.0
2017-08-31 12:26:24.673712 | +++ functions-common:address_in_net:2293 : local subnet
2017-08-31 12:26:24.676286 | +++++ functions-common:address_in_net:2294 : cidr2netmask 22
2017-08-31 12:26:24.678244 | +++++ functions-common:cidr2netmask:2310 : local 'maskpat=255 255 255 255'
2017-08-31 12:26:24.680173 | +++++ functions-common:cidr2netmask:2311 : local 'maskdgt=254 252 248 240 224 192 128'
2017-08-31 12:26:24.682186 | +++++ functions-common:cidr2netmask:2312 : set -- 255 255 252
2017-08-31 12:26:24.684130 | +++++ functions-common:cidr2netmask:2313 : echo 255.255.252.0
2017-08-31 12:26:24.686055 | ++++ functions-common:address_in_net:2294 : maskip 10.0.1.159 255.255.252.0
2017-08-31 12:26:24.687638 | ++++ functions-common:maskip:2367 : local ip=10.0.1.159
2017-08-31 12:26:24.689173 | ++++ functions-common:maskip:2368 : local mask=255.255.252.0
2017-08-31 12:26:24.690741 | ++++ functions-common:maskip:2369 : local l=10.0.1
2017-08-31 12:26:24.692149 | ++++ functions-common:maskip:2369 : local r=0.1.159
2017-08-31 12:26:24.693766 | ++++ functions-common:maskip:2369 : local n=255.255.252
2017-08-31 12:26:24.695607 | ++++ functions-common:maskip:2369 : local m=255.252.0
2017-08-31 12:26:24.697367 | ++++ functions-common:maskip:2370 : local subnet
2017-08-31 12:26:24.699326 | ++++ functions-common:maskip:2371 : subnet=10.0.0.0
2017-08-31 12:26:24.700717 | ++++ functions-common:maskip:2372 : echo 10.0.0.0
2017-08-31 12:26:24.702240 | +++ functions-common:address_in_net:2294 : subnet=10.0.0.0
2017-08-31 12:26:24.703588 | +++ functions-common:address_in_net:2295 : [[ 10.0.0.0 == 10.0.0.0 ]]
2017-08-31 12:26:24.705006 | +++ functions-common:get_default_host_ip:684 : echo
2017-08-31 12:26:24.706681 | ++ stackrc:source:815 : HOST_IP=
2017-08-31 12:26:24.708122 | ++ stackrc:source:816 : '[' '' == '' ']'
2017-08-31 12:26:24.709291 | ++ stackrc:source:817 : die 817 'Could not determine host ip address. See local.conf for suggestions on setting HOST_IP.'
2017-08-31 12:26:24.710401 | ++ functions-common:die:186 : local exitcode=0
2017-08-31 12:26:24.711651 | ++ functions-common:die:187 : set +o xtrace

According to logstack.openstack.org [1]:
- this happens only on citycloud nodes: citycloud-kna1 and citycloud-sto2
- this has happened in the last 30 days only on gate-networking-bagpipe-dsvm-fullstack-ubuntu-xenial-nv and gate-group-based-policy-dsvm-rally-ubuntu-xenial-nv

[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Could%20not%20determine%20host%20ip%20address.%5C%22

[Thomas Morin (tmmorin-orange)]

There were 68 such failures in the last ~10 days.

[Thomas Morin (tmmorin-orange)]

There are in fact 4 jobs where this happens:
- gate-networking-bagpipe-dsvm-fullstack-ubuntu-xenial-nv
- gate-group-based-policy-dsvm-rally-ubuntu-xenial-nv
- gate-group-based-policy-dsvm-functional-ubuntu-xenial-nv
- gate-group-based-policy-dsvm-aim-ubuntu-xenial-nv

[Thomas Morin (tmmorin-orange)]

After checking for the gate-networking-bagpipe-dsvm-fullstack-ubuntu-xenial-nv, it seems that on citycloud, it *always* fail.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22HOST_IP%3D%5C%22%20AND%20message%3A%5C%22stackrc%3Asource%3A815%5C%22%20AND%20build_name%3Agate-networking-bagpipe-dsvm-fullstack-ubuntu-xenial-nv%20AND%20NOT%20message%3A%5C%22HOST_IP%3D127.0.0.1%5C%22%20AND%20node_provider%3Acitycloud*%20AND%20build_queue%3Acheck

[Thomas Morin (tmmorin-orange)]

Having this in localrc, seems to prevent this kind of conflict:

IPV4_ADDRS_SAFE_TO_USE=10.1.0.0/20

It happens to be absent from the localrc of faulty jobs, e.g.:
- http://logs.openstack.org/87/497587/1/check/gate-group-based-policy-dsvm-aim-ubuntu-xenial-nv/76af3b2/logs/local.conf.txt.gz
- http://logs.openstack.org/29/480229/6/check/gate-networking-bagpipe-dsvm-fullstack-ubuntu-xenial-nv/163cb5b/logs/local.conf.txt.gz

Something somewhere in the labyrinth of devstack gate scrips is so that these jobs don't inherit what would help them avoid this issue such as [1] or [2].

[1] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack.yaml#n16
[2] http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n52

[Thomas Morin (tmmorin-orange)]

Ah, others ran into the same issue !

Here is an example fix (neutron): https://review.openstack.org/#/c/469189

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/500109
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.