HEALTH_WARN with POOL_APP_NOT_ENABLED application not enabled on N pool(s)

[Nobuto Murata]

Just after an OpenStack deployment, Ceph cluster stays in HEALTH_WARN. It breaks a status monitoring for example by NRPE.

xenial + cloud:xenial-pike

$ sudo ceph version
sudo: unable to resolve host juju-3ea7d7-1
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)

$ ceph health detail
HEALTH_WARN application not enabled on 2 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 2 pool(s)
    application not enabled on pool 'default.rgw.control'
    application not enabled on pool '.rgw.root'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

http://docs.ceph.com/docs/master/rados/operations/health-checks/#pool-app-not-enabled
http://docs.ceph.com/docs/master/rados/operations/pools/#associate-pool-to-application

[WORKAROUND]

$ juju run --unit ceph-mon/0 '
    ceph osd pool application enable default.rgw.control rgw
    ceph osd pool application enable default.rgw.buckets.index rgw
    ceph osd pool application enable .rgw.root rgw
    ceph osd pool application enable glance rbd
    ceph osd pool application enable cinder-ceph rbd
    ceph osd pool application enable gnocchi rgw
'

[Nobuto Murata (nobuto)]

Ah, openstack-on-lxd uses ceph charm instead of ceph-mon charm actually. it may not be the case with ceph-mon. I will double check with it.

[Chris MacNaughton (chris.macnaughton)]

According to https://ceph.com/community/new-luminous-pool-tags/ , there is a new step following pool creation that we'll need to add support for: `rbd pool init <pool-name>`

Further reading of that page suggests that, despite the scary warning, the absence of app tags doesn't affect functionality at all.

[Nobuto Murata (nobuto)]

After trying ceph-mon/next charm, the number of affected pools are decreased. It looks like rgw related so adding ceph-radosgw charm task for now.

$ ceph health detail
HEALTH_WARN application not enabled on 2 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 2 pool(s)
    application not enabled on pool 'default.rgw.control'
    application not enabled on pool '.rgw.root'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

While agreeing that it's just a cosmetic warning, but monitoring components rely on the short health status like HEALTH_WARN. Fixing it for monitoring would be a bit high importance.

[Arné Schreuder (arneschreuder)]

Can confirm, I get the same in version 12.2.1

[James Page (james-page)]

Ceph will make a guess on the application once it starts to get filled; however the charms should set this on deployment to avoid early lifecycle health warnings in ceph.

This touches charms.ceph and ceph-mon (to add support to the ceph broker), and an update to all client charms to provide this information to the ceph-mon during pool creation.

[Alexander Litvinov (alitvinov)]

Also seeing the issue

ubuntu$ sudo ceph version
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)

ubuntu$ sudo ceph health detail
HEALTH_WARN application not enabled on 5 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 5 pool(s)
    application not enabled on pool 'gnocchi'
    application not enabled on pool 'default.rgw.control'
    application not enabled on pool '.rgw.root'
    application not enabled on pool 'glance'
    application not enabled on pool 'cinder-ceph'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

[Dmitrii Shcherbakov (dmitriis)]

sudo ceph status
  cluster:
    id: 4ce1de56-383e-11e8-aedd-00163e99e06d
    health: HEALTH_WARN
            application not enabled on 2 pool(s)

sudo ceph health detail
HEALTH_WARN application not enabled on 2 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 2 pool(s)
    application not enabled on pool 'default.rgw.control'
    application not enabled on pool '.rgw.root'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

ubuntu@juju-be77dc-0-lxd-0:~$ sudo ceph osd pool application enable default.rgw.control rgw
enabled application 'rgw' on pool 'default.rgw.control'
ubuntu@juju-be77dc-0-lxd-0:~$ sudo ceph osd pool application enable .rgw.root rgw
enabled application 'rgw' on pool '.rgw.root'
ubuntu@juju-be77dc-0-lxd-0:~$ sudo ceph health detail
HEALTH_OK

[Ryan Beisner (1chb1n)]

hi @nobuto - FYI, the openstack-on-lxd example bundles have been updated to use ceph-mon + ceph-osd instead of the deprecated legacy ceph charm.

[James Hebden (ec0)]

As a side-note for other intrepid travellers hitting this -
On a recent deployment, I also had to run 'ceph osd pool application enable default.rgw.buckets.index rgw' to get to HEALTH_OK.

[Nobuto Murata (nobuto)]

Subscribing ~field-high.

Originally I thought it was not applicable to Canonical's ~field-high since there was a workaround. However, this issue hit every single field deployment with Pike+ and caused a lot of confusions. Broken Ceph status out of the box is hard to be justified, so ~field-high is now appropriate for faster resolution I believe.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/564463
Committed: https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=978e561782105db0cf47c1076434f0739dd3855d
Submitter: Zuul
Branch: master

commit 978e561782105db0cf47c1076434f0739dd3855d
Author: Chris MacNaughton <email address hidden>
Date: Thu Apr 26 13:11:43 2018 +0200

    Add broker support for passing app_name

    In Ceph >= Luminous, application name needs to be set
    on a per-pool level to avoid health warnings. This
    change adds support for sending the application name
    over the broker channel from consuming charms.

    When a name is not sent from the other side of the
    relation, the application name will be set to "unknown"
    in Luminous and greater

    Change-Id: I99ae47b6802f50ea019751ffa328f11567cca567
    Closes-Bug: #1753640

[Chris Gregan (cgregan)]

Field High SLA now requires that a estimated date for a fix is listed in the comments. Please provide this estimate for the open tasks.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/564466
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=22aa0c614601570e1d6de67936287e3ccf185432
Submitter: Zuul
Branch: master

commit 22aa0c614601570e1d6de67936287e3ccf185432
Author: Chris MacNaughton <email address hidden>
Date: Thu Apr 26 13:13:29 2018 +0200

    Add broker support for passing app_name

    In Ceph >= Luminous, application name needs to be set
    on a per-pool level to avoid health warnings. This
    change adds support for sending the application name
    over the broker channel from consuming charms.

    When a name is not sent from the other side of the
    relation, the application name will be set to "unknown"
    in Luminous and greater

    Change-Id: I1109251b08da20adaf3d677c38fc1aacfba29439
    Closes-Bug: #1753640
    Depends-On: I99ae47b6802f50ea019751ffa328f11567cca567

[James Page (james-page)]

The updates to ceph-mon will remove the warning messages - however there is a pending piece of work to actually provide the application name from each consuming charm that still needs to be done.

I don't consider this field-high as application name is not actually used for anything in ceph so downgrading the bug tasks to low.

Please can the field-high subscriber be removed from this bug.

[David Coronel (davecore)]

I just this issue in a deployment with ceph-proxy as a cinder-ceph backend. My workaround was:

juju ssh ceph-mon/0 "sudo ceph osd pool application enable cinder-ceph-proxy rbd"

[Chris MacNaughton (chris.macnaughton)]

If you've got ceph in the model (as suggested by `juju ssh ceph-mon...`), why is ceph-proxy deployed?

[David Coronel (davecore)]

@chris: It's a second cinder backend that's connected to a ceph cluster in a different juju model on the same juju controller.

[David Coronel (davecore)]

I forgot to say I'm running that command in the other juju model where the ceph cluster is deployed.