IPv6 VIPs are created with /64

[Michele Baldessari]

From https://bugzilla.redhat.com/show_bug.cgi?id=1441635#c39

The problem is that the VIPs are on the same network as the "normal"
controller interfaces, and it causes problems with routing when a VIP
moves.

Consider controller ctrl-r00-00. It has IPv6 address
fd00:fd00:fd00:2000::20/64 on vlan200, which is the internalapi
network.

The internalapi VIP has IPv6 address fd00:fd00:fd00:2000::5, also on
vlan200, the internalapi network.

Now, when the VIP resource is active on ctrl-r00-00, the vlan200
interface has both addresses. There also exists a route to
fd00:fd00:fd00:2000::/64 on dev vlan200.

What happens when galera starts, and the replication connects to the
other cluster members? The configuration item used is:

wsrep_cluster_address=gcomm://ctrl-r00-00,ctrl-r01-01,ctrl-r02-02

(this is an attribute on the galera resource)

Next, ctrl-r00-00 does a name lookup for ctrl-r01-01, which is:

[root@ctrl-r00-00 ~]# getent hosts ctrl-r01-01
fd00:fd00:fd00:2000::21 ctrl-r01-01.redhat.local ctrl-r01-01

So we need to make a connection to fd00:fd00:fd00:2000::21. Remember
we have a route for that network on dev vlan200, but we have two valid
IPv6 addresses on that network.

In short, the kernel chooses the VIP address as the source addr for
the connection. And that's a problem. When the VIP moves to a
different host, any outbound packets become unroutable.

So in this circumstance, ctrl-r01-01 and ctrl-r02-02 will see the
galera membership of ctrl-r00-00 as originating from the VIP, not from
the address normally associated with ctrl-r00-00. So when the VIP
moves, galera logs that a member with the VIP address went away. This
is the suspicious behavior we observed previously.

The same thing happens with RabbitMQ. Some of the inter-cluster
connections are using the VIP as a source address. When the VIP
moves, RabbitMQ sees other cluster members disappear, and things
generally fall apart from there.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/460028

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/460232

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460232
Committed: https://git.openstack.org/cgit/openstack/puppet-pacemaker/commit/?id=0c2e23375cd27d43cdcb81c058a370874ff4d5cf
Submitter: Jenkins
Branch: master

commit 0c2e23375cd27d43cdcb81c058a370874ff4d5cf
Author: Michele Baldessari <email address hidden>
Date: Wed Apr 26 19:14:34 2017 +0200

    Add support for ipv6_addrlabel with IPaddr2 RA

    So currently when we create an IPv6 VIP in puppet the
    services running on the node where the VIP is, are susceptible
    to using the VIP as a source address in their communication.
    When the VIP relocates to another node, those TCP connections
    using the VIP as a source will be hanging until a TCP timeout
    kicks in.

    There are many ways to guide the kernel in a source address selection
    with IPv6 (call "ip -6 addr add <...> " with the "noprefixroute", use
    address labels, or simply set the deprecated option on the created
    ip address). We are choosing address labels because it is already
    supported in the resource-agents package and seems the recommended
    way of doing things.

    Partial-Bug: #1686357

    Change-Id: Ifd42683efae2cb29be3fcc1d5afdb62dad1bd82d
    Co-Authored-By: Damien Ciabrini <email address hidden>
    Co-Authored-By: John Eckersberg <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460028
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=6227484b60cd72cf4647051923a3baf175100a72
Submitter: Jenkins
Branch: master

commit 6227484b60cd72cf4647051923a3baf175100a72
Author: Michele Baldessari <email address hidden>
Date: Wed Apr 26 11:40:51 2017 +0200

    IPv6 VIP addresses need to be /128

    We currently hardcode /64 as our VIP addresses when using IPv6.
    The problem with this is that some server code might bind to that
    IP as a source address when doing inter-cluster communication
    (rabbitmq/galera for example). So when the VIP moves there will
    be effectively a network outage between the nodes, which should not
    happen.

    Likely this was hardcoded to /64 because the RA IPaddr2 needs a nic
    parameter when /128 is specified. This is due to:
    https://bugzilla.redhat.com/show_bug.cgi?id=1445628

    We also make sure we use the ipv6_addrlabel option set to 99 so that
    they will never be used as source ip addresses.

    Depends-On: I7fcf15a00aedbdcfb21db501ad46c69fb97ec30c
    Partial-Bug: #1686357

    Change-Id: Ibefde870512ad1e03ff12f7aea91b3734f03f96f
    Co-Authored-By: Sofer Athlan-Guyot <email address hidden>
    Co-Authored-By: Marios Andreou <email address hidden>
    Co-Authored-By: Damien Ciabrini <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/462479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462479
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=7e1e6093dbbcaa1d8932f32c942ce8c82bf02221
Submitter: Jenkins
Branch: stable/ocata

commit 7e1e6093dbbcaa1d8932f32c942ce8c82bf02221
Author: Michele Baldessari <email address hidden>
Date: Wed Apr 26 11:40:51 2017 +0200

    IPv6 VIP addresses need to be /128

    We currently hardcode /64 as our VIP addresses when using IPv6.
    The problem with this is that some server code might bind to that
    IP as a source address when doing inter-cluster communication
    (rabbitmq/galera for example). So when the VIP moves there will
    be effectively a network outage between the nodes, which should not
    happen.

    Likely this was hardcoded to /64 because the RA IPaddr2 needs a nic
    parameter when /128 is specified. This is due to:
    https://bugzilla.redhat.com/show_bug.cgi?id=1445628

    We also make sure we use the ipv6_addrlabel option set to 99 so that
    they will never be used as source ip addresses.

    Depends-On: I7fcf15a00aedbdcfb21db501ad46c69fb97ec30c
    Partial-Bug: #1686357

    Change-Id: Ibefde870512ad1e03ff12f7aea91b3734f03f96f
    Co-Authored-By: Sofer Athlan-Guyot <email address hidden>
    Co-Authored-By: Marios Andreou <email address hidden>
    Co-Authored-By: Damien Ciabrini <email address hidden>
    (cherry picked from commit 6227484b60cd72cf4647051923a3baf175100a72)