default route is left via eth0 but should via br-ex

[Anastasia Palkina]

"build_id": "2015-03-31_22-54-44",
"ostf_sha": "1c7f799cb0e159737d1f910b13e4073b16d1eb8c",
"build_number": "260",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-31_22-54-44", "ostf_sha": "1c7f799cb0e159737d1f910b13e4073b16d1eb8c", "build_number": "260", "api": "1.0", "nailgun_sha": "51ddbe0a2e379e82af531b1c6d45fd386d2f894e", "production": "docker", "python-fuelclient_sha": "9b0b6ab2f09ff75445550b7210104e21df5d8cd6", "astute_sha": "7292fc2a673cb1c32a688a46fd4836ca0500a957", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "178812b1971a900c49a8afc1688afd7475a6ffbb", "fuellib_sha": "27459a89cdac078ec4046a442a11d750c5eccc2e"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "51ddbe0a2e379e82af531b1c6d45fd386d2f894e",
"production": "docker",
"python-fuelclient_sha": "9b0b6ab2f09ff75445550b7210104e21df5d8cd6",
"astute_sha": "7292fc2a673cb1c32a688a46fd4836ca0500a957",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "178812b1971a900c49a8afc1688afd7475a6ffbb",
"fuellib_sha": "27459a89cdac078ec4046a442a11d750c5eccc2e"

I reproduced this issue only for HA with 3 or more controllers

1. Create new environment (CentOS)
2. Choose nova-network, vlan
3. Add 4 controller, 2 compute, 3 cinder
4. Start deployment. It was successful
5. Start OSTF tests. Some of them failed because 'ping 8.8.8.8' doesn't wotk from controller and compute nodes

Controllers: node-13,14,15,16
Computes: node-17,18

[root@node-14 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.169.254 - 255.255.255.255 !H 0 - 0 -
240.0.0.0 0.0.0.0 255.255.255.252 U 0 0 0 hapr-host
240.0.0.4 0.0.0.0 255.255.255.252 U 0 0 0 vrouter-host
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-fw-admin
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
172.16.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-storage
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-mgmt
0.0.0.0 10.20.0.2 0.0.0.0 UG 0 0 0 eth0
[root@node-14 ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2004ms

[root@node-14 ~]#

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaQ1VteFcyM216R00/view?usp=sharing

[Stanislav Makar (smakar)]

It is weird - floating error :(
The access to this env was provided to me by reporter and I was looking for errors in log but nothing
Reseted this env and deployed again - all is fine

[Anastasia Palkina (apalkina)]

Logs after redeployment: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaVkoyNXUwelVsN3c/view?usp=sharing

[Stanislav Makar (smakar)]

looking into why

[root@node-13 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node-13.domain.tld
NOZEROCONF=yes
GATEWAY=10.20.0.2

[Stanislav Makar (smakar)]

[root@node-13 ~]# grep "GATEWAY" /etc/sysconfig/network
GATEWAY=10.20.0.2
[root@node-13 ~]# grep "GATEWAY" /etc/sysconfig/network-scripts/ifcfg-*
/etc/sysconfig/network-scripts/ifcfg-br-ex:GATEWAY=172.16.0.1

Looks like correct default gateway was not installed in file /etc/sysconfig/network
Network was restarted and bad gateway was installed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/171258

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/171258
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0cc709193f8137e307bd7074b0bb6bea49039f85
Submitter: Jenkins
Branch: master

commit 0cc709193f8137e307bd7074b0bb6bea49039f85
Author: Stanislav Makar <email address hidden>
Date: Tue Apr 7 15:45:46 2015 +0000

    Fix the problem with default route overwriting

    Before any port configuration we check and kill dhclient due
    to if it is exist it carries on updating routing table and at the
    end we do not have that expected. All further l3 configuration we
    do later if it is needed.

    Closes-bug: #1439642
    Change-Id: I383d85caf2e7156c848a517791c25cbcda630633

[Alexander Nevenchannyy (anevenchannyy)]

Verified on MOS 6.1 ISO #473
Steps to Verify:
root@node-1:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.59.49 0.0.0.0 UG 0 0 0 br-ex
10.88.0.48 0.0.0.0 255.255.255.240 U 0 0 0 br-fw-admin
169.254.169.254 - 255.255.255.255 !H 0 - 0 -
172.16.59.48 0.0.0.0 255.255.255.240 U 0 0 0 br-ex
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-mgmt
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-storage
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br-mesh
240.0.0.0 0.0.0.0 255.255.255.252 U 0 0 0 hapr-host
240.0.0.4 0.0.0.0 255.255.255.252 U 0 0 0 vrouter-host