Generating an AppArmor profile in 'enforce' mode disallows access to the OSD volumes

[Peter Matulis]

This command

$ juju config ceph-osd aa-profile-mode=enforce

causes loss of access to the OSD volumes. The `juju status` command shows:

"No block devices detected using current configuration"

Details here:

https://paste.ubuntu.com/p/WP83VJQ9qn/

Reverting to 'disable' mode (the default) restores access.

[Andrew McLeod (admcleod)]

As per the documentation for the ceph-osd charm, this does not necessarily seem like a bug:

AppArmor Profiles

AppArmor is not enforced for Ceph by default. An AppArmor profile can be generated by the charm. However, great care must be taken.

Changing the value of the aa-profile-mode option is disruptive to a running Ceph cluster as all ceph-osd processes must be restarted as part of changing the AppArmor profile enforcement mode.

The generated AppArmor profile currently has a narrow supported use case, and it should always be verified in pre-production against the specific configurations and topologies intended for production.

The AppArmor profile(s) which are generated by the charm should NOT yet be used in the following scenarios:
- When there are separate journal devices.
- On any version of Ceph prior to Luminous.
- On any version of Ubuntu other than 16.04.
- With Bluestore enabled.

---

With this documentation in mind, is this bug still valid?

[Peter Matulis (petermatulis)]

Yes, it is valid. The documentation does not say that changing profiles will not work in my environment (according to the listed criteria). It also says that disruption will occur due to stuff having to restart; not that it will not work.

Note: The actual README was updated a month ago but was not sent to the Charm Store.

https://opendev.org/openstack/charm-ceph-osd/src/branch/master/README.md

[Edward Hope-Morley (hopem)]

imho the charm should ignore aa-complain=enforce if bluestore is enabled (since that isn't supported) and print an error to unit status until it gets unset.

[Przemyslaw Hausman (phausman)]

Are the scenarios listed in #1 still valid for Focal/Yoga?

I've attempted to deploy OpenStack with ceph-osd configured with `aa-profile-mode: enforce` and I hit the issue very similar to the one reported by @petermatulis.

This is a CIS-hardened cloud, therefore all Apparmor profiles should be enforcing. Otherwise the rule "1.6.1.4 Ensure all AppArmor Profiles are enforcing" fails.

In my environment, ceph-osd units are initially in `error` state with message a 'hook failed: "secrets-storage-relation-changed"' and then they move to `blocked` with message 'No block devices detected using current configuration'. See attached juju unit log and ceph logs.

[Alexander Litvinov (alitvinov)]

Just tested on focal with
ceph-osd config aa-profile-mode=enforce.
kern.log has the following:
Dec 12 11:47:14 node02 kernel: [ 634.948181] audit: type=1400 audit(1670845634.001:177): apparmor="DENIED" operation="file_lock" profile="/usr/bin/ceph-osd" name="/dev/dm-0" pid=62590 comm="ceph-osd" requested_mask="k" denied_mask="k" fsuid=64045 ouid=64045

The profile currently has:
40: /dev/** rw,

If I adjust profile manually like this, Ceph osd works:
40: /dev/** rwk,

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/867219
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/38407abdd566b1e2cec7ab2b414e9382167cc318
Submitter: "Zuul (22348)"
Branch: master

commit 38407abdd566b1e2cec7ab2b414e9382167cc318
Author: alitvinov <email address hidden>
Date: Mon Dec 12 16:44:48 2022 +0400

    Tweak apparmor profile to access OSD volumes.

    Plus add aa-profile-mode enforce option to the test bundles.

    Closes-Bug: #1860801
    Change-Id: I8264ad760d92da3faa384c8edca5566fc622c57d

[Nobuto Murata (nobuto)]

TODO: backport it to stable/quincy (at least) once the tox and CI related issue has been sorted out.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/879853

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/879853
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/2526e491dfd66988965d731095d176749efa5b47
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 2526e491dfd66988965d731095d176749efa5b47
Author: alitvinov <email address hidden>
Date: Mon Dec 12 16:44:48 2022 +0400

    Tweak apparmor profile to access OSD volumes.

    Plus add aa-profile-mode enforce option to the test bundles.

    Closes-Bug: #1860801
    Change-Id: I8264ad760d92da3faa384c8edca5566fc622c57d
    (cherry picked from commit 38407abdd566b1e2cec7ab2b414e9382167cc318)