Cinder A/A and etcd not working with TLS-everywhere

[Alan Bishop]

When configured to run in active/active mode, cinder uses etcd for its distributed lock manager. Unfortunately, several aspects of the deployment are broken when internal TLS is enabled. Extensive debugging uncovered a number of issues, some within tripleo (tht or puppet-tripleo), and some outside tripleo. This bug report covers the tripleo issues, and they are:

1) The tripleo::certmonger::etcd puppet module fails when creating the TLS cert and key.
- The $postsave_cmd isn't valid now that etcd runs in a container
- The cert and key files on the host cannot by chown'ed to "etcd:etcd" as those are not a valid user/group name on the host
- The module needs a $dnsnames input parameter to allow inclusion of additional names in the certificate

2) The tripleo::profile::base::cinder::volume puppet module needs include the cert and key files in the "backend_url" that cinder uses to access the DLM.

3) The cert and key files need to be accessible to the cinder and etcd services running inside their respective containers. This means the files need to be bind mounted into the containers, along with an ACL rule that lets the cinder and etcd processes have read access to the files.

4) The etcd certificate specs, configured by THT, have to include a subject alternative name (SAN) entry for the etcd node's internal API address.

Currently, with cinder A/A and TLS-everywhere, the deployment will fail when tripleo::certmonger::etcd attempts to chown the cert and key files. But that's just the first layer of the overall problem. Efforts are underway to address them all, along with an interim workaround.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/716432

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716721

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/717295

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716661
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=63111546cdc983c383e964f33618a78e7185fb81
Submitter: Zuul
Branch: master

commit 63111546cdc983c383e964f33618a78e7185fb81
Author: Alan Bishop <email address hidden>
Date: Wed Apr 1 09:23:26 2020 -0700

    Fix etcd's support for internal TLS

    Fixes for etcd's certmonger cert and key generation:
    - Do not chown the cert and key files generated on the host. In addition
      to the fact that "etcd" is not a valid user|grep name on the host, an
      ACL must be used to allow other services (such as cinder) to access
      the files. That ACL will be handled at the THT layer.
    - New $dnsnames parameter supports adding a list of subject alternative
      name (SAN) to the cert.
    - Remove obsolete default $postsave_cmd (see comment in the code), but
      make it a parameter so it can be overridden if necessary.

    The cinder-volume service uses etcd when cinder is configured for
    active/active mode. When internal TLS is enabled, the backend_url must
    include references to etcd's cert and key files.

    Partial-Bug: #1869955
    Change-Id: Ifa7452ec15b81f48d7e5fb1252f20b5af1dff95c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716432
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e621ff801bef9d4b04696ad11da06b64080e1352
Submitter: Zuul
Branch: master

commit e621ff801bef9d4b04696ad11da06b64080e1352
Author: Alan Bishop <email address hidden>
Date: Tue Mar 31 13:25:04 2020 -0700

    Workaround for cinder A/A and etcd with TLS-everywhere

    This patch implements a workaround to allow cinder to run in active/
    active mode with internal TLS enabled. Cinder uses etcd for its
    distributed lock manager, and the LP bug documents several problems
    when the deployment enables TLS on the internal API network.

    Until a full solution is available, this workaround allows cinder and
    etcd to work without TLS. The full solution is complicated, and affects
    components outside of tripleo.

    Change-Id: Iec0d02f8f51067098dd58beb4fe57a7fd5ab5651
    Related-Bug: #1869955

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/717837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717295
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fe3f38d3a9a6d8795dd7109930ce433dc4488df6
Submitter: Zuul
Branch: master

commit fe3f38d3a9a6d8795dd7109930ce433dc4488df6
Author: Grzegorz Grasza <email address hidden>
Date: Fri Apr 3 13:47:37 2020 +0200

    Create DNS entries in IPA for openstack services

    This adds forward and reverse DNS records for all services.

    Without the reverse DNS entries, certificates cannot be created
    for etcd which requires a subject alternative name (SAN) entry
    to be included. This is used by cinder for A/A support.

    The task is delegated to tripleo-ipa. It uses the host_entry
    which is also used for managing the entries in /etc/hosts.

    Depends-On: https://review.opendev.org/#/c/716982/
    Change-Id: I41681f90f70fa0dffe4abbe3d6d5c48015589f66
    Related-bug: #1869955
    Related: rhbz#1804079

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/718191

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717837
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=aeb5bc9b978ab80e9267d331333f5964ec33136b
Submitter: Zuul
Branch: stable/train

commit aeb5bc9b978ab80e9267d331333f5964ec33136b
Author: Alan Bishop <email address hidden>
Date: Tue Mar 31 13:25:04 2020 -0700

    Workaround for cinder A/A and etcd with TLS-everywhere

    This patch implements a workaround to allow cinder to run in active/
    active mode with internal TLS enabled. Cinder uses etcd for its
    distributed lock manager, and the LP bug documents several problems
    when the deployment enables TLS on the internal API network.

    Until a full solution is available, this workaround allows cinder and
    etcd to work without TLS. The full solution is complicated, and affects
    components outside of tripleo.

    Change-Id: Iec0d02f8f51067098dd58beb4fe57a7fd5ab5651
    Related-Bug: #1869955
    (cherry picked from commit e621ff801bef9d4b04696ad11da06b64080e1352)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718191
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2e57b2b8a4134767d18a871d8201b5a829ca5085
Submitter: Zuul
Branch: stable/train

commit 2e57b2b8a4134767d18a871d8201b5a829ca5085
Author: Grzegorz Grasza <email address hidden>
Date: Fri Apr 3 13:47:37 2020 +0200

    Create DNS entries in IPA for openstack services

    This adds forward and reverse DNS records for all services.

    Without the reverse DNS entries, certificates cannot be created
    for etcd which requires a subject alternative name (SAN) entry
    to be included. This is used by cinder for A/A support.

    The task is delegated to tripleo-ipa. It uses the host_entry
    which is also used for managing the entries in /etc/hosts.

    Change-Id: I41681f90f70fa0dffe4abbe3d6d5c48015589f66
    (cherry picked from commit fe3f38d3a9a6d8795dd7109930ce433dc4488df6)
    Related-bug: #1869955
    Related: rhbz#1804079

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/724219

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alan Bishop (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/724219
Reason: Holding train backport until https://review.opendev.org/716721 merges on master

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716721
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2fc1290c10f77dac816d1416ddf6c137a10110b1
Submitter: Zuul
Branch: master

commit 2fc1290c10f77dac816d1416ddf6c137a10110b1
Author: Alan Bishop <email address hidden>
Date: Mon Apr 27 13:30:18 2020 -0700

    Fix cinder and etcd running with internal TLS enabled

    The LP bug referenced below describes a number of issues when
    cinder tries to use etcd for its distributed lock manager with
    internal TLS enabled. This patch resolves issues related to
    generating and distributing etcd's cert and key files.

    - The etcd cert must contain a subject alternative name (SAN) for the
      etcd node's internal API IP address. This is necessary because etcd
      wants to use IP addresses (versus host names), and this requires the
      IP address be listed in the TLS certificate.
    - The cert and key files are generated on the host, and must be
      available to multiple services running in their respective containers.
      The cert and key files need to be bind mounted, and an ACL is
      required so the etcd and cinder services have permission to read the
      files.

    EnableEtcdInternalTLS, a workaround introduced in [1], still defaults
    to False. The default value can be switched to True after tripleo
    switches from using novajoin to the ansible tripleo-ipa role for
    registering nodes with the IdM service.

    [1] https://review.opendev.org/#/q/Iec0d02f8f51067098dd58beb4fe57a7fd5ab5651

    Closes-Bug: #1869955
    Depends-On: Ifa7452ec15b81f48d7e5fb1252f20b5af1dff95c
    Change-Id: I798d60818b214de9266226c8409b69525a951dd5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726949

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724219
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=a1da18aed63cae60cd7efc6bade8f17869e07377
Submitter: Zuul
Branch: stable/train

commit a1da18aed63cae60cd7efc6bade8f17869e07377
Author: Alan Bishop <email address hidden>
Date: Wed Apr 1 09:23:26 2020 -0700

    Fix etcd's support for internal TLS

    Fixes for etcd's certmonger cert and key generation:
    - Do not chown the cert and key files generated on the host. In addition
      to the fact that "etcd" is not a valid user|grep name on the host, an
      ACL must be used to allow other services (such as cinder) to access
      the files. That ACL will be handled at the THT layer.
    - New $dnsnames parameter supports adding a list of subject alternative
      name (SAN) to the cert.
    - Remove obsolete default $postsave_cmd (see comment in the code), but
      make it a parameter so it can be overridden if necessary.

    The cinder-volume service uses etcd when cinder is configured for
    active/active mode. When internal TLS is enabled, the backend_url must
    include references to etcd's cert and key files.

    Partial-Bug: #1869955
    Change-Id: Ifa7452ec15b81f48d7e5fb1252f20b5af1dff95c
    (cherry picked from commit 63111546cdc983c383e964f33618a78e7185fb81)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726949
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=692717bd4606949f6f3731a65aa9945cd2b7bf06
Submitter: Zuul
Branch: stable/train

commit 692717bd4606949f6f3731a65aa9945cd2b7bf06
Author: Alan Bishop <email address hidden>
Date: Mon Apr 27 13:30:18 2020 -0700

    Fix cinder and etcd running with internal TLS enabled

    The LP bug referenced below describes a number of issues when
    cinder tries to use etcd for its distributed lock manager with
    internal TLS enabled. This patch resolves issues related to
    generating and distributing etcd's cert and key files.

    - The etcd cert must contain a subject alternative name (SAN) for the
      etcd node's internal API IP address. This is necessary because etcd
      wants to use IP addresses (versus host names), and this requires the
      IP address be listed in the TLS certificate.
    - The cert and key files are generated on the host, and must be
      available to multiple services running in their respective containers.
      The cert and key files need to be bind mounted, and an ACL is
      required so the etcd and cinder services have permission to read the
      files.

    EnableEtcdInternalTLS, a workaround introduced in [1], still defaults
    to False. The default value can be switched to True after tripleo
    switches from using novajoin to the ansible tripleo-ipa role for
    registering nodes with the IdM service.

    [1] https://review.opendev.org/#/q/Iec0d02f8f51067098dd58beb4fe57a7fd5ab5651

    Closes-Bug: #1869955
    Depends-On: Ifa7452ec15b81f48d7e5fb1252f20b5af1dff95c
    Change-Id: I798d60818b214de9266226c8409b69525a951dd5
    (cherry picked from commit 2fc1290c10f77dac816d1416ddf6c137a10110b1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.