rabbitmq tls support needs tweaks for rabbitmq 3.9

[Michele Baldessari]

TLDR:
On TLS-E environments, erlang24, needed by rabbitmq 3.9 has https://github.com/erlang/otp/pull/4805 included. That will throw a warning every time rabbitmqctl talks to epmd/rabbitmq:
2021-09-29T21:49:01.925797679+00:00 stdout F Debug: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]/unless: Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

We need to tweak a few things to fix this up. Namely the following needs to happen:
- Make sure rabbitmq and erlang use the fqdn hostnames
- Drop the old hardcoded ciphers default as now it is not needed any longer
- Move the ssl erlang options into a separate file
- Makre sure the client uses verify_peer

[Michele Baldessari (michele)]

Fixed by https://review.opendev.org/c/openstack/tripleo-heat-templates/+/812390 and https://review.opendev.org/c/openstack/puppet-tripleo/+/812401

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/813260

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/813260
Committed: https://opendev.org/openstack/puppet-tripleo/commit/fdca31a2009a0aaf3f3ee9c5e30083ac59bf067f
Submitter: "Zuul (22348)"
Branch: master

commit fdca31a2009a0aaf3f3ee9c5e30083ac59bf067f
Author: Michele Baldessari <email address hidden>
Date: Fri Oct 8 23:25:38 2021 +0200

    Bind mount the IPA crt when internal_tls is enabled

    In order for later reviews to make use of the FreeIPA internal
    CA we need to first bind mount it within the container.

    We need to add a default in the hiera definition (/etc/ipa/ca.crt)
    in order to break a cyclic dependency on the subsequent patches.
    (THT child change will set the rabbitmq::ssl_cacert key)

    Related-Bug: #1946374
    Change-Id: Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/812390
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/b5ae7f846719fd0b04d12d2bffd383df99e5c34b
Submitter: "Zuul (22348)"
Branch: master

commit b5ae7f846719fd0b04d12d2bffd383df99e5c34b
Author: Michele Baldessari <email address hidden>
Date: Mon Oct 4 17:42:33 2021 +0200

    Switch to rabbitmq fqdn hostnames and pass proper CA info

    With this change we make sure that the 'RABBITMQ_NODENAME' is the fqdn
    and that we use it when connecting to the node itself via rabbitmqctl.
    We also make sure we set the CA to the one defined in the InternalTLSCAFile
    template parameter.

    Testing protocol is described in the parent change
    ("I4f853de3b532b3b5e71d29aa621a2925c3885393")

    Related-Bug: #1946374

    Depends-On: Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26
    Change-Id: I621557a037fbedc5c110f905dbe19290cc92b772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/812401
Committed: https://opendev.org/openstack/puppet-tripleo/commit/1a42b83b40ee6f44b795f5ec222aa26c6eb8a4ed
Submitter: "Zuul (22348)"
Branch: master

commit 1a42b83b40ee6f44b795f5ec222aa26c6eb8a4ed
Author: Michele Baldessari <email address hidden>
Date: Mon Oct 4 21:33:40 2021 +0200

    Modernize rabbitmq's TLS support

    The depend-on change switches rabbit's nodename to FQDN and adds
    the CA in the config. This puppet change does the following:
    1. Switches the default tls versions to 1.2 and 1.3
    2. Drops the old hard-coded cipher list that used to be needed to avoid
       falling back to tls 1.0
    3. Introduce a verify_peer_{client,server} parameter. The former is set
       to verify_peer by default, so that rabbitmq 3.9 does not emit
       warnings (see original LP description)
    4. Splits off all erlang ssl parameters to a separate file called
       /etc/rabbitmq/ssl-dist.conf. Reason for this is that it is cleaner
       and not all configurable items are supported by erlang as command
       line arguments. (In particular the ssl versions)
    5. Make sure we pass the custom CA inside the HA bundle (this had to
       be split out to Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26
       (Bind mount the IPA crt when internal_tls is enabled) for CI to pass
       on all reviews.

    Tested as follows:
    A. Deployed and run tempest against a TLS-E overcloud that uses as
       rabbitmq 3.9 + erlang 24.1 stack
    B. Made sure that only tls1.2 and/or tls1.3 are allowed on port 5672
       and 25672

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_1 |grep ^New
    00CCA0CAB37F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1584:SSL alert number 70
    New, (NONE), Cipher is (NONE)

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_2 |grep ^New
    depth=1 O = HOME.ARPA, CN = Certificate Authority
    verify return:1
    depth=0 O = HOME.ARPA, CN = controller-0.internalapi.home.arpa
    verify return:1
    New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_3 |grep ^New
    depth=1 O = HOME.ARPA, CN = Certificate Authority
    verify return:1
    depth=0 O = HOME.ARPA, CN = controller-0.internalapi.home.arpa
    verify return:1
    New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

    Closes-Bug: #1946374

    Co-Authored-By: John Eckersberg <email address hidden>

    Depends-On: I621557a037fbedc5c110f905dbe19290cc92b772

    Change-Id: I4f853de3b532b3b5e71d29aa621a2925c3885393

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/813572

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/813573

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/813558

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/813572
Committed: https://opendev.org/openstack/puppet-tripleo/commit/28135309d9b8eae604d3a5116866a625621e65a7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 28135309d9b8eae604d3a5116866a625621e65a7
Author: Michele Baldessari <email address hidden>
Date: Fri Oct 8 23:25:38 2021 +0200

    Bind mount the IPA crt when internal_tls is enabled

    In order for later reviews to make use of the FreeIPA internal
    CA we need to first bind mount it within the container.

    We need to add a default in the hiera definition (/etc/ipa/ca.crt)
    in order to break a cyclic dependency on the subsequent patches.
    (THT child change will set the rabbitmq::ssl_cacert key)

    Related-Bug: #1946374
    Change-Id: Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26
    (cherry picked from commit fdca31a2009a0aaf3f3ee9c5e30083ac59bf067f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/813573
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ff3589786926992e0b822779f3c96b7d4e6c5cae
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ff3589786926992e0b822779f3c96b7d4e6c5cae
Author: Michele Baldessari <email address hidden>
Date: Mon Oct 4 17:42:33 2021 +0200

    Switch to rabbitmq fqdn hostnames and pass proper CA info

    With this change we make sure that the 'RABBITMQ_NODENAME' is the fqdn
    and that we use it when connecting to the node itself via rabbitmqctl.
    We also make sure we set the CA to the one defined in the InternalTLSCAFile
    template parameter.

    Testing protocol is described in the parent change
    ("I4f853de3b532b3b5e71d29aa621a2925c3885393")

    Related-Bug: #1946374

    Depends-On: Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26
    Change-Id: I621557a037fbedc5c110f905dbe19290cc92b772
    (cherry picked from commit b5ae7f846719fd0b04d12d2bffd383df99e5c34b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/813558
Committed: https://opendev.org/openstack/puppet-tripleo/commit/7892097e8fa13e2b07390a7152449cd3b7c704f2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7892097e8fa13e2b07390a7152449cd3b7c704f2
Author: Michele Baldessari <email address hidden>
Date: Mon Oct 4 21:33:40 2021 +0200

    Modernize rabbitmq's TLS support

    The depend-on change switches rabbit's nodename to FQDN and adds
    the CA in the config. This puppet change does the following:
    1. Switches the default tls versions to 1.2 and 1.3
    2. Drops the old hard-coded cipher list that used to be needed to avoid
       falling back to tls 1.0
    3. Introduce a verify_peer_{client,server} parameter. The former is set
       to verify_peer by default, so that rabbitmq 3.9 does not emit
       warnings (see original LP description)
    4. Splits off all erlang ssl parameters to a separate file called
       /etc/rabbitmq/ssl-dist.conf. Reason for this is that it is cleaner
       and not all configurable items are supported by erlang as command
       line arguments. (In particular the ssl versions)
    5. Make sure we pass the custom CA inside the HA bundle (this had to
       be split out to Ib0236f9c086d520d0a27e3aa8b41927bc7b50c26
       (Bind mount the IPA crt when internal_tls is enabled) for CI to pass
       on all reviews.

    Tested as follows:
    A. Deployed and run tempest against a TLS-E overcloud that uses as
       rabbitmq 3.9 + erlang 24.1 stack
    B. Made sure that only tls1.2 and/or tls1.3 are allowed on port 5672
       and 25672

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_1 |grep ^New
    00CCA0CAB37F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1584:SSL alert number 70
    New, (NONE), Cipher is (NONE)

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_2 |grep ^New
    depth=1 O = HOME.ARPA, CN = Certificate Authority
    verify return:1
    depth=0 O = HOME.ARPA, CN = controller-0.internalapi.home.arpa
    verify return:1
    New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384

    [root@controller-0 ~]# openssl s_client -connect controller-0.internalapi.home.arpa:5672 -CAfile /etc/ipa/ca.crt -verify_hostname controller-0.internalapi.home.arpa -tls1_3 |grep ^New
    depth=1 O = HOME.ARPA, CN = Certificate Authority
    verify return:1
    depth=0 O = HOME.ARPA, CN = controller-0.internalapi.home.arpa
    verify return:1
    New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

    Closes-Bug: #1946374

    Co-Authored-By: John Eckersberg <email address hidden>

    Depends-On: I621557a037fbedc5c110f905dbe19290cc92b772

    Change-Id: I4f853de3b532b3b5e71d29aa621a2925c3885393

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 16.0.0 release.