Incorrect permissions for cinder's cephx keyrings associated with CephExternalMultiConfig clusters

[Alan Bishop]

CephExternalMultiConfig provides a mechanism for supplying cephx keyrings for multiple ceph clusters, and cinder uses these keyrings in a DCN/Edge architecture. For example, in order to migrate a volume from an edge site to the central (control plane) site, the cinder-volume service running at the edge must be able to read the cephx keyring file associated with the central site.

The problem is the file permissions do not allow the 'cinder' user to access key. This leads to failures like this, as seen in the cinder-volume.log:

3352:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd [req-1e53b7df-3f0a-4326-9e91-d0348350d385 380b318d9e484780a01eafbd8ce4d653 b32b24e1449e4accaa9cc047d184319d - default default] Error connecting to ceph cluster.: rados.PermissionDeniedError: [errno 13] error connecting to the cluster
3353:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd Traceback (most recent call last):
3354:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 80, in connect
3355:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd client.connect()
3356:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd File "rados.pyx", line 910, in rados.Rados.connect
3357:2021-05-19 13:58:13.634 48 ERROR os_brick.initiator.linuxrbd rados.PermissionDeniedError: [errno 13] error connecting to the cluster

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/794341

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/794341
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f1cd8006fec9f3f68cee21fc2139fb985b0b1fac
Submitter: "Zuul (22348)"
Branch: master

commit f1cd8006fec9f3f68cee21fc2139fb985b0b1fac
Author: Alan Bishop <email address hidden>
Date: Wed Jun 2 12:52:48 2021 -0700

    Fix cinder's cephx keyring file permissions

    This patch updates cinder's kolla permissions so that cinder can
    access any cephx keyring associated with CephExternalMultiConfig
    ceph clusters. The new approach parses the cluster names out of the
    CephExternalMultiConfig array, and uses a wildcard to grant access
    to all keys (regardless of the key name) defined for each cluster.
    There is no risk of the wildcard granting improper access to a
    privileged key (e.g. the admin key), because CephExternalMultiConfig
    doesn't include privileged keys.

    This patch replaces similar (but more restrictive) code added in
    I73af5b868de629870a35d38f8436e7025aae791e. That patch allowed cinder
    to access cephx keyrings associated with a new CinderRbdMultiConfig
    parameter, but it didn't cover all potential use cases. For example,
    in a DCN/Edge deployment, cinder services running at the edge need
    access to the central site's client key in order to perform operations
    like offline volume migration.

    Closes-Bug: #1930620
    Resolves: rhbz#1962304
    Change-Id: I4423fcbd62b09ef323590fc740dd29e1a17777f5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795105

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795105
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/74e3884b4a21a27262c48b4df8f0e369a5486f87
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 74e3884b4a21a27262c48b4df8f0e369a5486f87
Author: Alan Bishop <email address hidden>
Date: Wed Jun 2 12:52:48 2021 -0700

    Fix cinder's cephx keyring file permissions

    This patch updates cinder's kolla permissions so that cinder can
    access any cephx keyring associated with CephExternalMultiConfig
    ceph clusters. The new approach parses the cluster names out of the
    CephExternalMultiConfig array, and uses a wildcard to grant access
    to all keys (regardless of the key name) defined for each cluster.
    There is no risk of the wildcard granting improper access to a
    privileged key (e.g. the admin key), because CephExternalMultiConfig
    doesn't include privileged keys.

    This patch replaces similar (but more restrictive) code added in
    I73af5b868de629870a35d38f8436e7025aae791e. That patch allowed cinder
    to access cephx keyrings associated with a new CinderRbdMultiConfig
    parameter, but it didn't cover all potential use cases. For example,
    in a DCN/Edge deployment, cinder services running at the edge need
    access to the central site's client key in order to perform operations
    like offline volume migration.

    Closes-Bug: #1930620
    Resolves: rhbz#1962304
    Change-Id: I4423fcbd62b09ef323590fc740dd29e1a17777f5
    (cherry picked from commit f1cd8006fec9f3f68cee21fc2139fb985b0b1fac)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795719

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795719
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/bc39ac89d2242bfdce9094e9cb22ee63d293ce28
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit bc39ac89d2242bfdce9094e9cb22ee63d293ce28
Author: Alan Bishop <email address hidden>
Date: Wed Jun 2 12:52:48 2021 -0700

    Fix cinder's cephx keyring file permissions

    This patch updates cinder's kolla permissions so that cinder can
    access any cephx keyring associated with CephExternalMultiConfig
    ceph clusters. The new approach parses the cluster names out of the
    CephExternalMultiConfig array, and uses a wildcard to grant access
    to all keys (regardless of the key name) defined for each cluster.
    There is no risk of the wildcard granting improper access to a
    privileged key (e.g. the admin key), because CephExternalMultiConfig
    doesn't include privileged keys.

    This patch replaces similar (but more restrictive) code added in
    I73af5b868de629870a35d38f8436e7025aae791e. That patch allowed cinder
    to access cephx keyrings associated with a new CinderRbdMultiConfig
    parameter, but it didn't cover all potential use cases. For example,
    in a DCN/Edge deployment, cinder services running at the edge need
    access to the central site's client key in order to perform operations
    like offline volume migration.

    NOTE (pre-Wallaby):
    The >= Wallaby versions of this patch tweaks code that was introduced
    in Wallaby by I73af5b868de629870a35d38f8436e7025aae791e. Pre-Wallaby
    versions of this patch _adds_ the tweaked code.

    Closes-Bug: #1930620
    Resolves: rhbz#1962304
    Change-Id: I4423fcbd62b09ef323590fc740dd29e1a17777f5
    (cherry picked from commit f1cd8006fec9f3f68cee21fc2139fb985b0b1fac)
    (cherry picked from commit 74e3884b4a21a27262c48b4df8f0e369a5486f87)
    Conflicts:
            deployment/cinder/cinder-common-container-puppet.yaml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796001

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 15.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796001
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ab2f7cf5cb2962fa4500b7cccd87a249a8f57d37
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ab2f7cf5cb2962fa4500b7cccd87a249a8f57d37
Author: Alan Bishop <email address hidden>
Date: Wed Jun 2 12:52:48 2021 -0700

    Fix cinder's cephx keyring file permissions

    This patch updates cinder's kolla permissions so that cinder can
    access any cephx keyring associated with CephExternalMultiConfig
    ceph clusters. The new approach parses the cluster names out of the
    CephExternalMultiConfig array, and uses a wildcard to grant access
    to all keys (regardless of the key name) defined for each cluster.
    There is no risk of the wildcard granting improper access to a
    privileged key (e.g. the admin key), because CephExternalMultiConfig
    doesn't include privileged keys.

    This patch replaces similar (but more restrictive) code added in
    I73af5b868de629870a35d38f8436e7025aae791e. That patch allowed cinder
    to access cephx keyrings associated with a new CinderRbdMultiConfig
    parameter, but it didn't cover all potential use cases. For example,
    in a DCN/Edge deployment, cinder services running at the edge need
    access to the central site's client key in order to perform operations
    like offline volume migration.

    NOTE (pre-Wallaby):
    The >= Wallaby versions of this patch tweaks code that was introduced
    in Wallaby by I73af5b868de629870a35d38f8436e7025aae791e. Pre-Wallaby
    versions of this patch _adds_ the tweaked code.

    Closes-Bug: #1930620
    Resolves: rhbz#1962304
    Change-Id: I4423fcbd62b09ef323590fc740dd29e1a17777f5
    (cherry picked from commit f1cd8006fec9f3f68cee21fc2139fb985b0b1fac)
    (cherry picked from commit 74e3884b4a21a27262c48b4df8f0e369a5486f87)
    Conflicts:
            deployment/cinder/cinder-common-container-puppet.yaml
    (cherry picked from commit bc39ac89d2242bfdce9094e9cb22ee63d293ce28)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796679

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796679
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/60a0b75eb3c1b83f87dda5c09a0853491fc4675a
Submitter: "Zuul (22348)"
Branch: stable/train

commit 60a0b75eb3c1b83f87dda5c09a0853491fc4675a
Author: Alan Bishop <email address hidden>
Date: Wed Jun 2 12:52:48 2021 -0700

    Fix cinder's cephx keyring file permissions

    This patch updates cinder's kolla permissions so that cinder can
    access any cephx keyring associated with CephExternalMultiConfig
    ceph clusters. The new approach parses the cluster names out of the
    CephExternalMultiConfig array, and uses a wildcard to grant access
    to all keys (regardless of the key name) defined for each cluster.
    There is no risk of the wildcard granting improper access to a
    privileged key (e.g. the admin key), because CephExternalMultiConfig
    doesn't include privileged keys.

    This patch replaces similar (but more restrictive) code added in
    I73af5b868de629870a35d38f8436e7025aae791e. That patch allowed cinder
    to access cephx keyrings associated with a new CinderRbdMultiConfig
    parameter, but it didn't cover all potential use cases. For example,
    in a DCN/Edge deployment, cinder services running at the edge need
    access to the central site's client key in order to perform operations
    like offline volume migration.

    NOTE (pre-Wallaby):
    The >= Wallaby versions of this patch tweaks code that was introduced
    in Wallaby by I73af5b868de629870a35d38f8436e7025aae791e. Pre-Wallaby
    versions of this patch _adds_ the tweaked code.

    Closes-Bug: #1930620
    Resolves: rhbz#1962304
    Change-Id: I4423fcbd62b09ef323590fc740dd29e1a17777f5
    (cherry picked from commit f1cd8006fec9f3f68cee21fc2139fb985b0b1fac)
    (cherry picked from commit 74e3884b4a21a27262c48b4df8f0e369a5486f87)
    Conflicts:
            deployment/cinder/cinder-common-container-puppet.yaml
    (cherry picked from commit bc39ac89d2242bfdce9094e9cb22ee63d293ce28)
    (cherry picked from commit ab2f7cf5cb2962fa4500b7cccd87a249a8f57d37)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.