CIS hardening breaks rabbitmq-server nagios statistics check

[Gábor Mészáros]

/var/lib/rabbitmq/ rabbitmq:rabbitmq
/var/lib/rabbitmq/data root:root 027 umask: rwxr-x---
so the
/usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c \* \* 100 200 /var/lib/rabbitmq/data/juju*_queue_stats.dat
check fails (/etc/nagios/nrpe.d/check_rabbitmq_queue.cfg)

1.) why data is owned by root:root, when rabbitmq's home folder (/var/lib/rabbitmq) is owned by rabbitmq:rabbitmq?

2.) when running CIS, that sets the umask to 027. This breaks the ability to read the stats by others.
The /var/lib/rabbitmq/data folder + stats should not be owned by root (along with possibly the /var/lib/rabbitmq/logs folder), but by the rabbitmq user. But that still would not allow nagios to read the stats file, maybe by adding nagios to the rabbitmq group.

update: this is not required: 3.) also cron.d/rabbitmq-stats is owned by root:root, which updates the stats file. It probably should be run as rabbitmq, not as root.

workaround: chown -R rabbitmq:rabbitmq /var/lib/rabbitmq; usermod -aG rabbitmq nagios; vim /etc/cron.d/rabbitmq-stats and replace root with rabbitmq. systemctl restart nagios-nrpe-server.service

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/860309

[Nobuto Murata (nobuto)]

Subscribing ~field-medium.

[DUFOUR Olivier (odufourc)]

The code has been tested in these scenarios :
* single RabbitMQ unit with NRPE subordinate
* clustered RabbitMQ with 3 units with NRPE subordinate
* single RabbitMQ unit without any NRPE subordinate

All of these RabbitMQ units are running from 3.9/stable charm.
For all of them, the upgrade of the charm now behaves properly and doesn't end in error either.

Before the ownership of the files would be following :ubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OK
ubuntu@juju-393eaf-0-lxd-1:~$ groups nagios
nagios : nagios
ubuntu@juju-393eaf-0-lxd-1:~$ sudo ls -l /var/lib/rabbitmq/
total 12
drwxr-xr-x 2 root root 4096 Oct 24 10:30 data
drwxr-xr-x 2 root root 4096 Oct 24 10:30 logs
drwxr-x--- 4 rabbitmq rabbitmq 4096 Oct 24 10:28 mnesia
ubuntu@juju-393eaf-0-lxd-1:~$ sudo ls -l /var/lib/rabbitmq/data/
total 8
-rw-r--r-- 1 root root 56 Oct 24 10:30 juju-393eaf-0-lxd-1_general_stats.dat
-rw-r--r-- 1 root root 88 Oct 24 10:30 juju-393eaf-0-lxd-1_queue_stats.dat

To check if Nagios NRPE is able to collect the stats (it should return 'OK') :
ubuntu@juju-393eaf-0-lxd-1:~$ cat /etc/nagios/nrpe.d/check_rabbitmq_queue.cfg | grep command | cut -f 2 -d '='
/usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
ubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OK

After Charm upgrade :
ubuntu@juju-393eaf-0-lxd-1:~$ sudo ls -l /var/lib/rabbitmq/
total 12
drwxr-x--- 2 rabbitmq rabbitmq 4096 Oct 24 10:30 data
drwxr-x--- 2 rabbitmq rabbitmq 4096 Oct 24 10:30 logs
drwxr-x--- 4 rabbitmq rabbitmq 4096 Oct 24 10:28 mnesia
ubuntu@juju-393eaf-0-lxd-1:~$ sudo ls -l /var/lib/rabbitmq/data
total 8
-rw-r----- 1 rabbitmq rabbitmq 56 Oct 24 10:30 juju-393eaf-0-lxd-1_general_stats.dat
-rw-r----- 1 rabbitmq rabbitmq 88 Oct 24ubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OK 10:30 juju-393eaf-0-lxd-1_queue_stats.dat
ubuntu@juju-393eaf-0-lxd-1:~$ groups nagios
nagios : nagios rabbitmqubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OKubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OK

And Nagios is still able to reach the required files :
ubuntu@juju-393eaf-0-lxd-1:~$ sudo -u nagios /usr/local/lib/nagios/plugins/check_rabbitmq_queues.py -c '\*' '\*' 100 200 -m 600 /var/lib/rabbitmq/data/juju-393eaf-0-lxd-1_queue_stats.dat
OK

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/860309
Committed: https://opendev.org/openstack/charm-rabbitmq-server/commit/c9efea67c8fe1aec32dc3bebe9853f41e4e23100
Submitter: "Zuul (22348)"
Branch: master

commit c9efea67c8fe1aec32dc3bebe9853f41e4e23100
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Wed Oct 5 00:03:29 2022 +0900

    Allow NRPE to collect stats in CIS hardened env

    It removes the necessity to run the cron task as root user
    and ensure the content created in /var/lib/rabbitmq belongs
    to rabbitmq user and group solely.

    Then giving access for nrpe user is done by adding its user
    to rabbitmq group.
    Also implemented in the upgrade-charm hook for ongoing
    deployments

    Closes-Bug: #1879524
    Change-Id: I19e3d675ace7c669451ca40a20d21cef1aec6a95

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/893931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/893931
Committed: https://opendev.org/openstack/charm-rabbitmq-server/commit/555942f89d2845bbbabfe87ea3717a0d7dbd2ff4
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit 555942f89d2845bbbabfe87ea3717a0d7dbd2ff4
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Wed Oct 5 00:03:29 2022 +0900

    Allow NRPE to collect stats in CIS hardened env

    It removes the necessity to run the cron task as root user
    and ensure the content created in /var/lib/rabbitmq belongs
    to rabbitmq user and group solely.

    Then giving access for nrpe user is done by adding its user
    to rabbitmq group.
    Also implemented in the upgrade-charm hook for ongoing
    deployments

    Closes-Bug: #1879524
    Change-Id: I19e3d675ace7c669451ca40a20d21cef1aec6a95