Containerized rabbitmq on multinode on ipv6 cannot work

[Michele Baldessari]

The error will be the following:
ERROR: epmd error for host overcloud-rabbit-0: address (cannot connect to host/port)

In /etc/hosts:
[root@overcloud-rabbit-0 rabbitmq]# grep overcloud-rabbit-0 /etc/hosts
fd00:fd00:fd00:2000::10 overcloud-rabbit-0.localdomain overcloud-rabbit-0

::::::::::::::
/etc/rabbitmq/rabbitmq-env.conf
::::::::::::::
NODE_IP_ADDRESS=
NODE_PORT=
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
RABBITMQ_NODENAME=rabbit@overcloud-rabbit-0
RABBITMQ_SERVER_ERL_ARGS="+K true +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/nati
ve>>}]"
RABBITMQ_SERVER_START_ARGS="-proto_dist inet6_tcp"
export ERL_EPMD_ADDRESS=fd00:fd00:fd00:2000::10
::::::::::::::
/etc/rabbitmq/rabbitmqadmin.conf
::::::::::::::
[default]
port = 15672

The reason here is that erlang does a mixed ipv4/ipv6 lookup and since stock centos/rhel 7 has systemd-myhostname in the NSS lookup the following will happen:
- rabbitmq will erroneously try to resolve one of its names and myhostname nss will return an ipv4 address of the server
- it will then try to use that to connect and fail

The proper fix here is to fix erlang/rabbitmq to use correct ipv6 lookups and not mixed ones. The fix had to happen on three layers
1) Erlang -> https://github.com/erlang/otp/pull/1487
2) puppetlabs-rabbitmq -> https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/552
3) puppet-tripleo -> https://review.openstack.org/#/c/475457/

[Michele Baldessari (michele)]

https://review.openstack.org/#/c/475457/ is the last review needed to get in.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475457
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=3a5eb08d58e843e50b06309a1d3ff5bb6e42424f
Submitter: Jenkins
Branch: master

commit 3a5eb08d58e843e50b06309a1d3ff5bb6e42424f
Author: John Eckersberg <email address hidden>
Date: Mon Jun 26 17:11:50 2017 -0400

    Use rabbitmq ipv6 flag

    The internal details of enabling IPv6 have moved upstream[1], so just
    set the ipv6 flag when desired and don't worry about the details
    anymore!

    [1] https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/552

    Closes-Bug: #1710658

    Change-Id: Ib22507c4d02f0fae5c0189ab7e040efac3df7e2f

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 7.3.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/533738

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533738
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=5ca2ec88cee46ef92c536aee5676626ebf3bda7c
Submitter: Zuul
Branch: stable/ocata

commit 5ca2ec88cee46ef92c536aee5676626ebf3bda7c
Author: John Eckersberg <email address hidden>
Date: Mon Jun 26 17:11:50 2017 -0400

    Use rabbitmq ipv6 flag

    The internal details of enabling IPv6 have moved upstream[1], so just
    set the ipv6 flag when desired and don't worry about the details
    anymore!

    [1] https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/552

    Closes-Bug: #1710658

    Change-Id: Ib22507c4d02f0fae5c0189ab7e040efac3df7e2f
    (cherry picked from commit 3a5eb08d58e843e50b06309a1d3ff5bb6e42424f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 6.5.11 release.