centos public haproxy vip can't be reached by nodes

[Andrew Woodward]

{"build_id": "2014-04-15_11-46-21", "mirantis": "yes", "build_number": "104", "nailgun_sha": "9530ef323b6b5fdb4dc9ff3516054b19b94c8e60", "production": "dev", "ostf_sha": "1ac04769cc8034438ccf1176860b25d0f5601c4a", "fuelmain_sha": "a9d39b53bf64b405f056f6acfe08f2055b92391f", "astute_sha": "0cb0b26773bd269000a14748e8ba49419462ecaf", "release": "5.0", "fuellib_sha": "d4299afd31992482888673be4e23936ce91d532a"}

centos, ha, nova-networks

3 controllers, 1 compute

I've seen this with zabbix change set applied or https://review.openstack.org/#/c/87400/ or plain iso, doesn't allways reproduce

Usually first controller fails to deploy with floating-ip creation erros

ICMP ping to public vip fails

[Andrew Woodward (xarses)]

xarses> ok, same error as before with master fuel-library
<mihgen> what error?
<xarses> https://bugs.launchpad.net/bugs/1307949
<mihgen> strange. I deployed an ISO 12 hours ago and didn't see that .. let me check which commits that iso was built from
<xarses> hmm, ok, so the https://review.openstack.org/#/c/87400/ does make it worse.
<mihgen> ok iso I tried was based on refs/changes/45/87545/2 from fuel-main
<xarses> something that ryan was checking on the zabbix change set, do we expect 'ip netns exec haproxy ping 10.110.0.80' to work
<xarses> ?
<mihgen> xarses: I assume yes.
<xarses> where ip == public vip
<mihgen> it's weird that it does not
<mihgen> could be routing tricks
<xarses> ok so my change makes it so from outside the netns its wont ping the vip, but inside the vip still cant ping in centos
<mihgen> hi salmon_
<salmon_> hi mihgen
<xarses> so half maybe all this might be from testing my change, so don't worry about it unless you see it

[Vladimir Kuklin (vkuklin)]

Ok, guys, the bug is following:

IPs assigned to the interfaces inside cannot haproxy namespace cannot be pinged from inside haproxy namespace, because loopback interface is down.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/87990

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/87990
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b2bbf0fcadb3b81875ad3aad3971d1b13b57ceac
Submitter: Jenkins
Branch: master

commit b2bbf0fcadb3b81875ad3aad3971d1b13b57ceac
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 16 19:01:27 2014 +0400

    Change state of 'lo' interface to 'up' inside net.namespace at creation.

    Change-Id: Id99dee1e72fccf80be689650fc63cbe9eab06d99
    Closes-Bug: #1308356

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/88525

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88525
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0672f750df96c331f12f0da51f27041799bd96b7
Submitter: Jenkins
Branch: stable/4.1

commit 0672f750df96c331f12f0da51f27041799bd96b7
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 16 19:01:27 2014 +0400

    Change state of 'lo' interface to 'up' inside net.namespace at creation.

    Backported for 4.1.1 from 5.0 (b2bbf0fcadb3b81875ad3aad3971d1b13b57ceac)

    Change-Id: Id99dee1e72fccf80be689650fc63cbe9eab06d99
    Closes-Bug: #1308356

[Meg McRoberts (dreidellhasa)]

Listed as Fixed issue in 5.0 Release Notes.

[Nastya Urlapova (aurlapova)]

{
build_id: "2014-06-05_15-25-50",
mirantis: "yes",
build_number: "344",
nailgun_sha: "a828d6b7610f872980d5a2113774f1cda6f6810b",
ostf_sha: "2b7b39e4b6ea89751b65171f24a8e80b5cac56aa",
fuelmain_sha: "9964da7dec34d3100419c1c77c8f5235d8e30f14",
astute_sha: "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
release: "4.1B",
fuellib_sha: "3511461a2b529619a787a6306441d9039699e71d"
}

root@node-2:~# ip netns exec haproxy ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
18: hapr-m: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether ce:52:8e:7e:8c:2e brd ff:ff:ff:ff:ff:ff
    inet 10.108.7.2/24 scope global hapr-m
    inet6 fe80::cc52:8eff:fe7e:8c2e/64 scope link
       valid_lft forever preferred_lft forever
20: hapr-p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 46:49:72:15:29:19 brd ff:ff:ff:ff:ff:ff
    inet 10.108.6.2/24 scope global hapr-p
    inet6 fe80::4449:72ff:fe15:2919/64 scope link
       valid_lft forever preferred_lft forever

ip netns exec haproxy ping 10.108.6.2
PING 10.108.6.2 (10.108.6.2) 56(84) bytes of data.
64 bytes from 10.108.6.2: icmp_req=1 ttl=64 time=0.080 ms
64 bytes from 10.108.6.2: icmp_req=2 ttl=64 time=0.058 ms
64 bytes from 10.108.6.2: icmp_req=3 ttl=64 time=0.038 ms
64 bytes from 10.108.6.2: icmp_req=4 ttl=64 time=0.062 ms