Charm stays in blocked state due unresolved permission issue when connected to a Ceph Octopus cluster

[Frode Nordahl]

Even though we request keys from ceph with an rbd-mirror profile the rbd-mirror process appears to not be able to interact with the Ceph cluster in the way it expects at Octopus. We also made an attempt to give the rbd-mirror user full access to the mon.

Enabling debug log on the MON may give log output such as:
2020-05-20T15:54:08.425042+0000 mon.juju-e32644-zaza-048032ef4dd5-7 (mon.1) 960 : audit [DBG] from='client.? 172.20.0.52:0/1474256700' entity='client.rbd-mirror.juju-e32644-zaza-048032ef4dd5-13' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/2/eb2ee01a-9a02-40da-950b-4ddc1f5e9e26"}]: access denied
2020-05-20T15:54:08.427957+0000 mon.juju-e32644-zaza-048032ef4dd5-7 (mon.1) 961 : audit [DBG] from='client.? 172.20.0.52:0/1474256700' entity='client.rbd-mirror.juju-e32644-zaza-048032ef4dd5-13' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/3/b8a91922-bda0-4ecc-9ffb-031e448311fe"}]: access denied
2020-05-20T15:54:08.429413+0000 mon.juju-e32644-zaza-048032ef4dd5-7 (mon.1) 962 : audit [DBG] from='client.? 172.20.0.52:0/1474256700' entity='client.rbd-mirror.juju-e32644-zaza-048032ef4dd5-13' cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]: access denied

The first side effect of this is that Ceph will mark any mirrored image/pool as being in a WARNING state, despite the fact that data appears to be mirrored, subsequently the charm will report this and be stuck in a blocked state.

At this stage in discovery our opinion is that this must be an upstream Ceph Octopus bug, but we keep this as a charm bug for tracking until we have final confirmation.

[Liam Young (gnuoy)]

This may be an upstream bug so I have raised https://tracker.ceph.com/issues/45638

[Alex Kavanagh (ajkavanagh)]

See also: https://bugs.launchpad.net/tripleo/+bug/1865754 -- which describes the issue in more detail.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/837857

[James Page (james-page)]

ceph-mon permissions for rbd-mirror clients:

rbd_mirror_caps = collections.OrderedDict([
    ('mon', ['profile rbd; allow r']),
    ('osd', ['profile rbd']),
    ('mgr', ['allow r']),
])

vs upstream docs:

ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd-mirror-peer' osd 'profile rbd'

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/837857
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/884510

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/884510
Committed: https://opendev.org/openstack/charm-ceph-rbd-mirror/commit/f5965a81bd52aa0285963ed2432f68f0254b6ac8
Submitter: "Zuul (22348)"
Branch: master

commit f5965a81bd52aa0285963ed2432f68f0254b6ac8
Author: Chris MacNaughton <email address hidden>
Date: Thu Apr 14 13:48:32 2022 +0200

    Bypass charm going into blocked due to a bug

    After Octopus, the reporting about image states fails because of a
    permission issue in Ceph. This change disables that status reporting
    to allow a deployment to be healthy even when some tools cannot query
    status.

    Also modernize build and func testing: remove python 3.9 and xena from
    tests

    Related-Bug: #1879749
    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1063
    Change-Id: Id86fc043495b89609cf6873ec58aee1e2e388578

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/885247

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/885247
Committed: https://opendev.org/openstack/charm-ceph-rbd-mirror/commit/26b055f79162bc5947b7aa68d85a03286cb1417e
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 26b055f79162bc5947b7aa68d85a03286cb1417e
Author: Peter Sabaini <email address hidden>
Date: Mon Jun 5 11:51:48 2023 +0200

    Bypass charm going into blocked due to a bug

    After Octopus, the reporting about image states fails because of a
    permission issue in Ceph. This change disables that status reporting
    to allow a deployment to be healthy even when some tools cannot query
    status.

    Also modernize build and func testing: remove python 3.9 and xena from
    tests

    Related-Bug: #1879749
    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1063
    cherry picked from Id86fc043495b89609cf6873ec58aee1e2e388578

    Change-Id: I6bd784b41995e78175869bf2d8cf6789e8f9dd58

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Peter Sabaini <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/837857
Reason: Superseded