Comment 1 for bug 2031637

Revision history for this message
Nobuto Murata (nobuto) wrote :

I can confirm the dashboard is missing SMART data.

smartctl is available on ceph-osd hosts:
https://github.com/openstack/charm-ceph-osd/blob/stable/quincy.2/lib/charms_ceph/utils.py#L91-L93

The necessary sudoers file is installed by a package:

# cat /etc/sudoers.d/ceph-smartctl
## allow ceph daemons (which run as user ceph) to collect device health metrics

ceph ALL=NOPASSWD: /usr/sbin/smartctl -x --json=o /dev/*
ceph ALL=NOPASSWD: /usr/sbin/nvme * smart-log-add --json /dev/*

# dpkg -S /etc/sudoers.d/ceph-smartctl
ceph-base: /etc/sudoers.d/ceph-smartctl

However, when access to /run/udev/data/b*:* is denied, Ceph OSD doesn't run smartctl.

Oct 19 07:29:58 top-wasp kernel: [15714.338549] audit: type=1400 audit(1697700598.925:939): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd" name="/run/udev/data/b252:16" pid=138858 comm="admin_socket" requested_mask="r" denied_mask="r" fsuid=64045 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.527631] audit: type=1400 audit(1697700599.113:940): apparmor="ALLOWED" operation="exec" profile="/usr/bin/ceph-osd" name="/usr/bin/sudo" pid=139702 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=64045 ouid=0 target="/usr/bin/ceph-osd//null-/usr/bin/sudo"
Oct 19 07:29:59 top-wasp kernel: [15714.532630] audit: type=1400 audit(1697700599.117:941): apparmor="ALLOWED" operation="file_mmap" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/usr/bin/sudo" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.532679] audit: type=1400 audit(1697700599.117:942): apparmor="ALLOWED" operation="file_mmap" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.532914] audit: type=1400 audit(1697700599.117:943): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/dev/full" pid=139702 comm="sudo" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.532921] audit: type=1400 audit(1697700599.117:944): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/dev/null" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.533055] audit: type=1400 audit(1697700599.117:945): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/etc/ld.so.cache" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.533086] audit: type=1400 audit(1697700599.117:946): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/usr/lib/x86_64-linux-gnu/libaudit.so.1.0.0" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.533116] audit: type=1400 audit(1697700599.117:947): apparmor="ALLOWED" operation="file_mmap" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/usr/lib/x86_64-linux-gnu/libaudit.so.1.0.0" pid=139702 comm="sudo" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0
Oct 19 07:29:59 top-wasp kernel: [15714.533194] audit: type=1400 audit(1697700599.117:948): apparmor="ALLOWED" operation="open" profile="/usr/bin/ceph-osd//null-/usr/bin/sudo" name="/usr/lib/x86_64-linux-gnu/libselinux.so.1" pid=139702 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

At least we need to update the apparmor policy to allow running smartctl but more work may be necessary than that.

--- /tmp/tmpy55fnsvz 2023-10-19 09:02:56.518091810 +0000
+++ /tmp/tmp6rkccewy 2023-10-19 09:02:56.518091810 +0000
@@ -23,10 +23,12 @@ include <tunables/global>
   /etc/debian_version r,
   /etc/lsb-release r,
   /run/blkid/blkid.tab r,
+ /run/udev/data/* r,
   /srv/ceph/** rwlk,
   /sys/devices/** r,
   /usr/bin/ceph-osd mr,
   /usr/bin/lsb_release rix,
+ /usr/bin/sudo mrix,
   /usr/share/distro-info/** r,
   /var/lib/ceph/** rwlk,
   /var/lib/charm/*/ceph.conf r,