[system-tests] Compute node can't reach internet hosts: IP for vrouter VIP is allocated for libvirt network (1st net address)

Bug #1486759 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Artem Panchenko

Bug Description

Fuel version info (7.0 build #196): http://paste.openstack.org/show/422193/

Network verification fails after environment deployment during system tests, because slave nodes without public network can't reach internet hosts:

2015-08-19 20:46:54,152 - ERROR fuel_web_client.py:1367 -- Network verification failed: These nodes: "4", "5", "6" failed to connect to some of these repositories: "http://mirror-pkgs.vm.mirantis.net/ubuntu-2015-08-18-170128", "http://mirror.fuel-infra.org/mos/ubuntu/"

root@node-5:~# ip r g 1
1.0.0.0 via 10.109.12.1 dev br-mgmt src 10.109.12.4
    cache
root@node-5:~# ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 10.109.12.1 icmp_seq=1 Destination Port Unreachable

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

root@node-5:~# arping -I br-mgmt 10.109.12.1
ARPING 10.109.12.1 from 10.109.12.6 br-mgmt
Unicast reply from 10.109.12.1 [52:54:00:6B:16:E4] 0.824ms
Unicast reply from 10.109.12.1 [8A:8B:C4:B3:05:D7] 0.846ms

...

root@node-1:~# hiera vrouter
{"network_role"=>"mgmt/vip",
"ipaddr"=>"10.109.12.1",
"node_roles"=>["controller", "primary-controller"],
"namespace"=>"vrouter"}

As you can see the first IP address from management network "10.109.12.1" was allocated for vrouter VIP, but that address is assigned to virtual network on bare metal host where tests were running. It caused addresses conflict and ARP didn't work properly.
Here is the change which caused tests failure:

https://github.com/stackforge/fuel-web/commit/a4a14215774084b34c4087bb13a6e2ed6bd8848f

In system tests we set gateway for each environment network (first ip address), but use 'cidr' notation and 'use_gateway': False for all networks except 'public' and 'fuelweb_admin'. The mentioned above code change introduced new logic: if 'cidr' notation is used for network and 'use_gateway' is not True, then all network addresses (including the first ip) are used for deployment, whether the 'gateway' for network is set or not.
So the tests should be modified to set 'ip_ranges' notation instead of 'cidr' in order to avoid usage of reserved IP addresses for deployments.

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/214824

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Artem Panchenko (apanchenko-8)
status: New → In Progress
tags: added: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/214824
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=a7121a3fedba15421b17a2dc050a19c7d7c35c4f
Submitter: Jenkins
Branch: master

commit a7121a3fedba15421b17a2dc050a19c7d7c35c4f
Author: Artem Panchenko <email address hidden>
Date: Thu Aug 20 01:50:30 2015 +0300

    Set 'ip_ranges' notation for cluster networks

    Since 'fuel-devops' allocates 1st IP of each network
    for 'router', tests must set exact IP ranges which can
    be used for environment. Otherwise 1st network IP could
    be allocated for environment needs if gateway isn't used
    in that network.

    Change-Id: I215bbbe73685affd598daf50d6900cb166a06c69
    Closes-bug: #1486759

Changed in fuel:
status: In Progress → Fix Committed
summary: - [system_tests] Compute node can't reach internet hosts: IP for vrouter
+ [system-tests] Compute node can't reach internet hosts: IP for vrouter
VIP is allocated for libvirt network (1st net address)
Changed in fuel:
status: Fix Committed → Fix Released
tags: added: area-qa
removed: non-release
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.