quincy/edge latest/edge RADOS permission denied (error connecting to the cluster)

[Corey Bryant]

The gate tests for charm-cinder-backup are failing, as can be seen here: https://review.opendev.org/c/openstack/charm-cinder-backup/+/861407

== test failure ==

ubuntu@coreycb-bastion:/mnt/cinder-backup$ tox -e func-target jammy-yoga
...
2022-10-27 18:04:48 [INFO] test_410_cinder_vol_create_backup_delete_restore_pool_inspect (zaza.openstack.charm_tests.cinder_backup.tests.CinderBackupTest)
2022-10-27 18:04:48 [INFO] Create, backup, delete, restore a ceph-backed cinder volume.
2022-10-27 18:04:48 [INFO] ...
2022-10-27 18:04:54 [INFO] Checking ceph cinder pool original samples...
2022-10-27 18:04:55 [INFO] ceph-backed cinder volume: resource 4fd0654e-4f5b-407c-b0fb-759cf7295cc7 in creating state, waiting for available
2022-10-27 18:04:56 [INFO] ceph-backed cinder volume: resource 4fd0654e-4f5b-407c-b0fb-759cf7295cc7 in available state, waiting for available
2022-10-27 18:04:56 [INFO] Backup volume: resource 2acedaf4-7d79-486c-a4ce-fd937cdde88b in creating state, waiting for available
2022-10-27 18:04:57 [INFO] Backup volume: resource 2acedaf4-7d79-486c-a4ce-fd937cdde88b in error state, waiting for available
...

== cinder-backup.log ==

2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall Traceback (most recent call last):
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 150, in _run_loop
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 162, in _setup_backup_driver
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall backup_service.check_for_setup_error()
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 296, in check_for_setup_error
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall with rbd_driver.RADOSClient(self, self._ceph_backup_pool):
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/cinder/volume/drivers/rbd.py", line 210, in __init__
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall self.cluster, self.ioctx = driver._connect_to_rados(pool)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 315, in _connect_to_rados
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall client.connect()
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall rv = execute(f, *args, **kwargs)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall six.reraise(c, e, tb)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall raise value
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall rv = meth(*args, **kwargs)
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall File "rados.pyx", line 680, in rados.Rados.connect
2022-10-27 14:15:09.955 31000 ERROR oslo.service.loopingcall rados.PermissionDeniedError: [errno 13] RADOS permission denied (error connecting to the cluster)

== test successful with quincy/stable channel ==

ubuntu@coreycb-bastion:/mnt/tmp/cinder-backup$ source .tox/func-target/bin/activate

(func-target) ubuntu@coreycb-bastion:/mnt/tmp/cinder-backup$ juju refresh ceph-mon --channel quincy/stable
(func-target) ubuntu@coreycb-bastion:/mnt/tmp/cinder-backup$ functest-test -m zaza-4cb23da2499e
...
success

(func-target) ubuntu@coreycb-bastion:/mnt/tmp/cinder-backup$ juju refresh ceph-mon --channel quincy/edge
(func-target) ubuntu@coreycb-bastion:/mnt/tmp/cinder-backup$ functest-test -m zaza-4cb23da2499e
...
fail

[Corey Bryant (corey.bryant)]

I added charm-cinder-backup to the bug to track the work-around.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-cinder-backup/+/862930

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder-backup/+/861407
Committed: https://opendev.org/openstack/charm-cinder-backup/commit/0a5239828e75179a1ced70b8651e3bd872692e2a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 0a5239828e75179a1ced70b8651e3bd872692e2a
Author: Alex Kavanagh <email address hidden>
Date: Fri Oct 14 17:03:17 2022 +0100

    Updates for zed stable branch creation

    - Set default branch for git review/gerrit to stable/zed
    - Switch tests to stable.
    - Switch to using stable charm-helpers branch.
    - Switch to using stable charm.openstack branch.
    - Switch to using stable zaza, zaza-openstack-tests
      branch
    - (reactive charms) Add build.lock file
    - (classic charms) make sync
    - (reactive: not reactive plugin): lock charm-tools < 3.1
    - (reactive: with reactive plugin): lock charm snap to 3.x/stable

    Use quincy/stable channel in bundles to work around the related bug.

    Change-Id: I2635c0ade478c616454c3eaf1a23c063033c1d0d
    Related-Bug: #1995086

[Corey Bryant (corey.bryant)]

I'm still seeing this as can be seen in patchset 3 of the following gate:
https://review.opendev.org/c/openstack/charm-cinder-backup/+/862930/3/

In this case it is with latest/edge.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877114

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877019

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877114
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/3a774be96132e7802d1278cd6812d2fb982c740e
Submitter: "Zuul (22348)"
Branch: master

commit 3a774be96132e7802d1278cd6812d2fb982c740e
Author: Chris MacNaughton <email address hidden>
Date: Fri Mar 10 12:55:27 2023 -0500

    Fix issue with ceph-client relation handling

    A bug was introduced when changing ceph-client to
    an operator framework library that caused the
    fallback application_name handling to present
    a class name rather than a remote applicaiton name.

    This change updates the handling to get at an
    `app.name` rather than an `app`.

    As a drive-by, this also allow-lists the fully-
    qualified rename.sh.

    Closes-Bug: #1995086
    Change-Id: I57b685cb78ba5c4930eb0fa73d7ef09d39d73743
    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877019
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/ef7afc5c7db4ccb303f96c1df03704210e229819
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit ef7afc5c7db4ccb303f96c1df03704210e229819
Author: Chris MacNaughton <email address hidden>
Date: Fri Mar 10 12:55:27 2023 -0500

    Fix issue with ceph-client relation handling

    A bug was introduced when changing ceph-client to
    an operator framework library that caused the
    fallback application_name handling to present
    a class name rather than a remote applicaiton name.

    This change updates the handling to get at an
    `app.name` rather than an `app`.

    Closes-Bug: #1995086
    Change-Id: I57b685cb78ba5c4930eb0fa73d7ef09d39d73743
    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1022