Services not running ceph-radosgw: keyring not found

[Aymen Frikha]

Deployed the openstack base bundle (yoga stable) but only ceph radosgw stays in blocked state with the following message: Services not running that should be: <email address hidden>

checking from syslog and got the following log:

Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring: (2) No such file or directory
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 AuthRegistry(0x55a269f92778) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring, disabling cephx
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring: (2) No such file or directory
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 AuthRegistry(0x7ffd23f00ca0) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring, disabling cephx
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: failed to fetch mon config (--no-mon-config to skip)
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Scheduled restart job, restart counter is at 2.

Not sure what is causing that.

[Aymen Frikha (aym-frikha)]

subscribed ~field-high

[Chris MacNaughton (chris.macnaughton)]

There's not enough information here to reproduce the bug, can you please add more information as documented in the charm guide: https://docs.openstack.org/charm-guide/latest/community/software-bug.html

[Launchpad Janitor (janitor)]

[Expired for Ceph RADOS Gateway Charm because there has been no activity for 60 days.]

[Andre Ruiz (andre-ruiz)]

I'm seeing this too. Also openstack yoga (in my case, with focal).

Only one of the 3 units is in "blocked" state, and I can see this in syslog:

Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring: (2) No such file or directory
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 AuthRegistry(0x564718e807f8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring, disabling cephx
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring: (2) No such file or directory
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 AuthRegistry(0x7ffc3da03590) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring, disabling cephx

[Andre Ruiz (andre-ruiz)]

This is the complete log for the unit: https://pastebin.canonical.com/p/w7swdbsBSN/

[Andre Ruiz (andre-ruiz)]

What I find interesting is that in that log it says:

2023-01-03 20:32:52 WARNING unit.ceph-radosgw/0.juju-log server.go:316 Inconsistent or absent auth returned by mon units. Setting auth_supported to 'none'

and on the mon units you can see:

2023-01-03 20:53:42 WARNING unit.ceph-mon/0.osd-relation-changed logger.go:60 Error ENOENT: failed to find client.rgw.juju-b4a340-3-lxd-4 in keyring

Here are the logs for the three mon units just in case:
https://pastebin.canonical.com/p/b9ntnzyD67/
https://pastebin.canonical.com/p/XXcdVcSYxp/
https://pastebin.canonical.com/p/5WWwSSPjHz/

[Andre Ruiz (andre-ruiz)]

This is a show-unit of the ceph-radosgw/0 unit: https://pastebin.canonical.com/p/g9Z7qJfFJ5/

and this is a show-unit of a ceph-mon unit: https://pastebin.canonical.com/p/MDcPntnCYc/