Fuel for OpenStack

vip__public_old cant reach other nodes mgmt addr

Bug #1321466 reported by Andrew Woodward on 2014-05-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	Critical	Aleksandr Didenko	Fuel for OpenStack 5.0

Bug Description

ISO 212

upon looking at the stats for haproxy running in vip__public_old, I found that backends for nodes other than this one are reported down.

After examining the haproxy namespace for public, i found that there is no interface for hapr-m and no corresponding route

root@node-7:~# ip netns exec haproxy ip route
default dev hapr-p scope link metric 10
10.108.33.0/24 dev hapr-p proto kernel scope link src 10.108.33.20

This causes all of the other nodes to not be accessible and prevents proper load balancing.

I think if we add a hapr-m interface and a proper route, this should be resolved.

-----------------------------------------------

Last updated: Tue May 20 21:39:36 2014
Last change: Tue May 20 21:37:11 2014 via cibadmin on node-7
Stack: classic openais (with plugin)
Current DC: node-7 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
17 Resources configured

Online: [ node-6 node-7 node-8 ]

Migration summary:
* Node node-6:
* Node node-7:
* Node node-8:

Connection to the CIB terminated
Reconnecting...
root@node-7:~# ip netns
qrouter-feeac162-a7da-4b78-8560-34b0c4ebc097
haproxy
root@node-7:~# ip netns exec haproxy ping node-6
PING node-6 (192.168.0.9) 56(84) bytes of data.
^C
--- node-6 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4024ms

root@node-7:~# ip netns exec haproxy ping node-8
PING node-8 (192.168.0.11) 56(84) bytes of data.
^C
--- node-8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2008ms

root@node-7:~# ping node-6
PING node-6 (192.168.0.9) 56(84) bytes of data.
64 bytes from node-6 (192.168.0.9): icmp_req=1 ttl=64 time=1.64 ms
^C
--- node-6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.649/1.649/1.649/0.000 ms
root@node-7:~# ping node-7
PING node-7 (192.168.0.10) 56(84) bytes of data.
64 bytes from node-7 (192.168.0.10): icmp_req=1 ttl=64 time=0.086 ms
64 bytes from node-7 (192.168.0.10): icmp_req=2 ttl=64 time=0.095 ms
64 bytes from node-7 (192.168.0.10): icmp_req=3 ttl=64 time=0.057 ms
^C
--- node-7 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.057/0.079/0.095/0.017 ms
root@node-7:~# ip netns exec haproxy ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
41: hapr-p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 16:3a:0a:75:7f:a7 brd ff:ff:ff:ff:ff:ff
    inet 10.108.33.20/24 scope global hapr-p
       valid_lft forever preferred_lft forever
    inet6 fe80::143a:aff:fe75:7fa7/64 scope link
       valid_lft forever preferred_lft forever
root@node-7:~# crm status
Last updated: Tue May 20 21:45:00 2014
Last change: Tue May 20 21:37:11 2014 via cibadmin on node-7
Stack: classic openais (with plugin)
Current DC: node-7 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
17 Resources configured

Online: [ node-6 node-7 node-8 ]

vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-6
vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-7
Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-6 node-7 node-8 ]
Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-6 node-7 node-8 ]
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-6 node-7 node-8 ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-6 node-7 node-8 ]
p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-8
p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-7
heat-engine (ocf::mirantis:heat-engine): Started node-7
root@node-7:~# ip netns exec haproxy ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
41: hapr-p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 16:3a:0a:75:7f:a7 brd ff:ff:ff:ff:ff:ff
    inet 10.108.33.20/24 scope global hapr-p
       valid_lft forever preferred_lft forever
    inet6 fe80::143a:aff:fe75:7fa7/64 scope link
       valid_lft forever preferred_lft forever
root@node-7:~# ip netns exec haproxy ping node-7
PING node-7 (192.168.0.10) 56(84) bytes of data.
64 bytes from node-7 (192.168.0.10): icmp_req=1 ttl=64 time=0.110 ms
64 bytes from node-7 (192.168.0.10): icmp_req=2 ttl=64 time=0.115 ms
64 bytes from node-7 (192.168.0.10): icmp_req=3 ttl=64 time=0.118 ms
64 bytes from node-7 (192.168.0.10): icmp_req=4 ttl=64 time=0.119 ms
64 bytes from node-7 (192.168.0.10): icmp_req=5 ttl=64 time=0.120 ms
^C
--- node-7 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3997ms
rtt min/avg/max/mdev = 0.110/0.116/0.120/0.010 ms
root@node-7:~# ip netns exec haproxy ip route
default dev hapr-p scope link metric 10
10.108.33.0/24 dev hapr-p proto kernel scope link src 10.108.33.20
root@node-7:~# ssh node-6
Warning: Permanently added 'node-6,192.168.0.9' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.11.0-18-generic x86_64)

* Documentation: https://help.ubuntu.com/
Last login: Tue May 20 20:38:17 2014 from 10.108.32.2
root@node-6:~# ip netns exec haproxy ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
23: hapr-m: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1a:da:f0:7c:b8:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.8/24 scope global hapr-m
       valid_lft forever preferred_lft forever
    inet6 fe80::18da:f0ff:fe7c:b870/64 scope link
       valid_lft forever preferred_lft forever

root@node-6:~# ip netns exec haproxy ping node-6
PING node-6 (192.168.0.9) 56(84) bytes of data.
64 bytes from node-6 (192.168.0.9): icmp_req=1 ttl=64 time=0.105 ms
64 bytes from node-6 (192.168.0.9): icmp_req=2 ttl=64 time=0.154 ms
^C
--- node-6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.105/0.129/0.154/0.027 ms
root@node-6:~# ip netns exec haproxy ping node-7
PING node-7 (192.168.0.10) 56(84) bytes of data.
64 bytes from node-7 (192.168.0.10): icmp_req=1 ttl=63 time=1.29 ms
64 bytes from node-7 (192.168.0.10): icmp_req=2 ttl=63 time=0.353 ms
^C
--- node-7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.353/0.821/1.290/0.469 ms
root@node-6:~# ip netns exec haproxy ping node-8
PING node-8 (192.168.0.11) 56(84) bytes of data.
64 bytes from node-8 (192.168.0.11): icmp_req=1 ttl=63 time=1.22 ms
64 bytes from node-8 (192.168.0.11): icmp_req=2 ttl=63 time=0.482 ms
64 bytes from node-8 (192.168.0.11): icmp_req=3 ttl=63 time=0.638 ms
64 bytes from node-8 (192.168.0.11): icmp_req=4 ttl=63 time=0.637 ms
^C
--- node-8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.482/0.746/1.227/0.284 ms
root@node-6:~#

Tags:

Revision history for this message

Andrew Woodward (xarses) wrote on 2014-05-20:

Download full text (25.6 KiB)

root@node-7:~# echo show stat | socat unix-connect:///var/lib/haproxy/stats stdio
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,
private_monitoring,FRONTEND,,,0,2,8000,1680,767859,58827437,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,5,,,,0,840,0,0,840,0,,0,5,1680,,,
private_monitoring,BACKEND,0,0,0,1,8000,840,767859,58827437,0,0,,840,0,0,0,UP,0,0,0,,0,1910,0,,1,1,0,,0,,1,0,,3,,,,0,0,0,0,840,0,,,,,0,0,
horizon,FRONTEND,,,0,6,8000,6,1080,2606,0,0,4,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,6,,,,0,1,1,4,0,0,,0,4,6,,,
horizon,node-6,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1910,1910,,1,2,1,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
horizon,node-7,0,0,0,1,,2,1080,1758,,0,,0,0,0,0,UP,1,1,0,0,1,1906,4,,1,2,2,,1,,2,0,,2,L7OK,301,7,0,1,1,0,0,0,0,,,,0,0,
horizon,node-8,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1910,1910,,1,2,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
horizon,BACKEND,0,0,0,1,8000,2,1080,2606,0,0,,0,0,0,0,UP,1,1,0,,1,1906,4,,1,2,0,,1,,1,0,,2,,,,0,1,1,0,0,0,,,,,0,0,
keystone-1,FRONTEND,,,0,1,8000,95,14155,79705,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,1,,,,0,95,0,0,0,0,,0,1,95,,,
keystone-1,node-6,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1910,1910,,1,3,1,,0,,2,0,,0,L4TOUT,,2000,0,0,0,0,0,0,0,,,,0,0,
keystone-1,node-7,0,0,0,1,,95,14155,79705,,0,,0,0,0,0,UP,1,1,0,0,1,1906,4,,1,3,2,,95,,2,0,,1,L4OK,,0,0,95,0,0,0,0,0,,,,95,0,
keystone-1,node-8,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1909,1909,,1,3,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
keystone-1,BACKEND,0,0,0,1,8000,95,14155,79705,0,0,,0,0,0,0,UP,1,1,0,,1,1906,3,,1,3,0,,95,,1,0,,1,,,,0,95,0,0,0,0,,,,,95,0,
keystone-2,FRONTEND,,,0,0,8000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,4,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,
keystone-2,node-6,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1909,1909,,1,4,1,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
keystone-2,node-7,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,1,1905,4,,1,4,2,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
keystone-2,node-8,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1909,1909,,1,4,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
keystone-2,BACKEND,0,0,0,0,8000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,1,1905,4,,1,4,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,
nova-api-1,FRONTEND,,,0,0,8000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,5,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,
nova-api-1,node-6,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1909,1909,,1,5,1,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
nova-api-1,node-7,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,1,1905,4,,1,5,2,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,
nova-api-1,node-8,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,1909,1909,,1,5,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
nova-api-1,BACKEND,0,0,0,0,8000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,1,1905,4,,1,5,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,
nova-api-2,FRONTEND,,,0,1,8000,1,263,2723,0,0,0,,,,,OPEN,,,,,,,,,1,6,0,,,,0,0,0,1,,,,0,1,0,0,0,0,,0,1,1,,,
nova-api-2,node-6,0,0,0,0,,0,0,0,,0...

Mike Scherbakov (mihgen) on 2014-05-21

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)

Mike Scherbakov (mihgen) on 2014-05-21

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Alexander Didenko (adidenko)

Aleksandr Didenko (adidenko) on 2014-05-21

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-05-22:

There are several possible solutions to this problem. In current state routing from "haproxy" namespace for management network works fine (via default route which is "public"-hapr interface), but src IP is from public network which causes problems. So we just need to fix src IP for outgoing packets on a controller which runs vip__public_old.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-22: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/94853

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-05-22:

Btw, swift nodes are always DOWN in haproxy:

swift,node-32,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,3262,3262,,1,13,2,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,
swift,node-33,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,0,1,3262,3262,,1,13,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,

Because haproxy balance them via "storage" network which does not have its own interface/route inside "haproxy" namespace and is being routed via default route with un-masqueraded src IP. This should be fixed with iptables masquarading as well in https://review.openstack.org/94853.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-22: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/94853
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a9374e2d246226d1c422a293eaf8c37f30fdac00
Submitter: Jenkins
Branch: master

commit a9374e2d246226d1c422a293eaf8c37f30fdac00
Author: Aleksandr Didenko <email address hidden>
Date: Wed May 21 20:16:18 2014 +0300

Add iptables rules for ns_IPaddr2

    Add new parameters for ns_IPaddr2 resource:
      - iptables_start_rules - rules we want to add on VIP start
      - iptables_stop_rules - rules we want to remove on VIP stop
      - iptables_comment - comment to associate with new rules

Use this new parameters in order to fix outgoing IP (src ip) in
packets routed via default route from "haproxy" namespace.

Change-Id: I4c58612ea3bd3401000939b098d90d14a85acad1
Closes-bug: #1321466

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-30: Fix proposed to fuel-library (stable/4.1)

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96860

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-02: Fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/96860
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=57c8f31eb00f5f8d8faeb499200520206f0f79c0
Submitter: Jenkins
Branch: stable/4.1

commit 57c8f31eb00f5f8d8faeb499200520206f0f79c0
Author: Aleksandr Didenko <email address hidden>
Date: Wed May 21 20:16:18 2014 +0300

Add iptables rules for ns_IPaddr2

Use this new parameters in order to fix outgoing IP (src ip) in
packets routed via default route from "haproxy" namespace.

Change-Id: I4c58612ea3bd3401000939b098d90d14a85acad1
Closes-bug: #1321466

Revision history for this message

Meg McRoberts (dreidellhasa) wrote on 2014-06-04:

Documented as fixed in 4.1.1 Release Notes

Aleksandr Didenko (adidenko) on 2014-06-17

Changed in fuel:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.