ceph-dashboard complains object gateway service not configured

[Andre Ruiz]

When you install ceph-dashboard (in my case, in Focal), login and go to the "Object Gateway" tab (or any of the sub-tabs) you see a message saying:

> The Object Gateway Service is not configured
> Credentials not found for RGW Daemon: ''
> Please consult the documentation on how to configure and enable the Object Gateway management functionality.

If you run "ceph dashboard set-rgw-credentials" and refresh the page, it works.

It feels like this should be automatic since ceph-dashboard is related to radosgw (like the documentation suggests). The charm has enough information and capacity to do it for you.

[Vern Hart (vern)]

I've seen the same deployed on jammy.

[Andre Ruiz (andre-ruiz)]

This deployment was channel quincy/stable

[Vern Hart (vern)]

Same. Here is my bundle for a test on AWS (from https://git.launchpad.net/cpe-deployments/tree/?h=ceph-aws-demo).

[Nobuto Murata (nobuto)]

I can confirm this. But interestingly, it happens only after adding the expected relation:

> - ['ceph-dashboard:radosgw-dashboard', 'ceph-radosgw:radosgw-user']

Before adding the relation the object gateway tab actually works.

[Nobuto Murata (nobuto)]

Subscribing ~field-high as the object gateway tab is broken by following the doc:
https://ubuntu.com/ceph/docs/install-dashboard

It's either:
- the document shouldn't suggest the relation in question
> ceph-dashboard:radosgw-dashboard ceph-radosgw:radosgw-user
- or there is something wrong in the charm

In any case, this needs to be investigated and fixed to offer a better out-of-the-box experience.

[Nobuto Murata (nobuto)]

Before adding the relation

[Nobuto Murata (nobuto)]

after adding the relation

[Nobuto Murata (nobuto)]

> I can confirm this. But interestingly, it happens only after adding the expected relation:
>
> > - ['ceph-dashboard:radosgw-dashboard', 'ceph-radosgw:radosgw-user']
>
> Before adding the relation the object gateway tab actually works.

At least on my test bed,

[before adding the relation]

# ceph config dump | grep 'dashboard.*KEY'
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY LWPYOAE08C0V3TL820N6 *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY 82Ln2K9GmQsV1iffJEfEpZtbM6TYPE6UItA8tjnf *

^^^ bare string - dashboard works

[after adding the relation]

# ceph config dump | grep 'dashboard.*KEY'
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY {'juju-5e2cd5-0-lxd-2': '8OHNPOQMYY1FEUGAQZL2', 'juju-5e2cd5-1-lxd-3': '8OHNPOQMYY1FEUGAQZL2', 'juju-5e2cd5-2-lxd-2': '8OHNPOQMYY1FEUGAQZL2'} *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY {'juju-5e2cd5-0-lxd-2': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T', 'juju-5e2cd5-1-lxd-3': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T', 'juju-5e2cd5-2-lxd-2': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T'} *

^^^ dictionary - dashboard doesn't work with "Credentials not found for RGW Daemon"

And according to Samuel, due to the upstream change for Ceph >= 17 (Quincy), the dictionary the charm generates and the dictionary the ceph dashboard expects are different because of daemon_name vs realm_name.
https://github.com/ceph/ceph/commit/0fcf0a7827cf4e8748a382613f9c8d1715c4a1e8#diff-93a9b99949ece1974c8b2d436c5cb0dea338a8352920418249de9b3ece1d156fL285-L297

[Nobuto Murata (nobuto)]

A quick and dirty workaround is to set a bare string by hand after removing the relation. e.g.

From:
====
# ceph config dump | grep 'dashboard.*KEY'
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY {'juju-5e2cd5-0-lxd-2': '8OHNPOQMYY1FEUGAQZL2', 'juju-5e2cd5-1-lxd-3': '8OHNPOQMYY1FEUGAQZL2', 'juju-5e2cd5-2-lxd-2': '8OHNPOQMYY1FEUGAQZL2'} *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY {'juju-5e2cd5-0-lxd-2': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T', 'juju-5e2cd5-1-lxd-3': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T', 'juju-5e2cd5-2-lxd-2': 'I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T'} *
====

To:
ceph config set mgr mgr/dashboard/RGW_API_ACCESS_KEY 8OHNPOQMYY1FEUGAQZL2
ceph config set mgr mgr/dashboard/RGW_API_SECRET_KEY I8Gb46kpqCmdBdsCNEx5bhOuEwnOGEtIb73rOK8T

Because access_key/secret_key are common across 3 units. I'm not sure until when the manual change is persistent though since some Juju hooks will override it at some point.

[Peter Sabaini (peter-sabaini)]

Good catch, many thanks for sleuthing.

From reading the commit[0] it looks like it supports bare string creds for the case when realms are not available, so I'd think we should consider using bare strings for RGW_API_ACCESS_KEY / RGW_API_SECRET_KEY

[0] https://github.com/ceph/ceph/commit/0fcf0a7827cf4e8748a382613f9c8d1715c4a1e8#diff-93a9b99949ece1974c8b2d436c5cb0dea338a8352920418249de9b3ece1d156fL285-L297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/903664
Committed: https://opendev.org/openstack/charm-ceph-dashboard/commit/fa621d1062b90c3af809dd82641fd26e14e6324d
Submitter: "Zuul (22348)"
Branch: master

commit fa621d1062b90c3af809dd82641fd26e14e6324d
Author: Utkarsh Bhatt <email address hidden>
Date: Thu Dec 14 16:19:28 2023 +0530

    Use single credential for configuring object storage gw in dashboard.

    The radosgw relation provides the same credential mapped to the
    hostnames of all the related radosgw application units. This extra
    information is not needed. Even when multiple radosgw applications
    are present in the same Juju model, a single instance of credentials
    is sufficient for dashbaord to present Object GW UI.

    Closes-Bug: #1986593
    Change-Id: Ib56e0bd4fe588219d6b5728d9a71cf09abdd25de
    Signed-off-by: Utkarsh Bhatt <email address hidden>

[Ethan Myers (ethanmyers)]

This still seems to be an issue. On a fresh deploy, I see the exact same behavior -- mgr reports juju containers:key in a dict, and the ceph-dashboard does not work. Switch the RGW_API_ACCESS_KEY and RGW_API_SECRET_KEY to strs works results in the dashboard working great.

[Nobuto Murata (nobuto)]

Which charm channel and revision are you on? The fix has been backported to quincy a few days ago.
https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/913275

[Ethan Myers (ethanmyers)]

ceph-dashboard: quincy/stable, rev 50.

This version was deployed a couple of days ago, so I might have just missed the backport.

[Luciano Lo Giudice (lmlogiudice)]

quincy/stable is at revision 52 now, so that may be the case indeed.