The charm is not compatible with the recent CIS 18.04

[Nikolay Vinogradov]

Hi all,

I'm trying to deploy rabbitmq-server charm on bionic with CIS hardening. The recent version of the CIS hardening package includes the rule #4.2.3 (I was using lvl2 server profile) which tunes permissions of the log files: https://security-certs.docs.ubuntu.com/en/cis-manual-requirements.

Unfortunately that breaks installation of the rabbitmq-server apt package with 'Permission denied' error.

Previously the rule had the number 4.2.4 and implemented permissions reset for the files /var/log/*, but in the current version it starts from the top (/var/log) and changes the permissions on the directories as well.

[Billy Olsen (billy-olsen)]

Hi Nikolay,

This sounds awful similar to the problem seen in bug #1773084, starting in comment #11. There was a reboot-pending in place and the reboot resolved the issue.

Can you let me know if this is similar to what you are seeing?

[Nikolay Vinogradov (nikolay.vinogradov)]

Hi Billy. We tried to reboot and it made the problem worse. Actually I don't see how the reboot could fix the permissions on /var/log.

Also adding charm percona-cluster as it is also affected by the problem.

[Nikolay Vinogradov (nikolay.vinogradov)]

A good question would be: is the bug in the charms, CIS or in the corresponding APT packages?

[Aurelien Lourot (aurelien-lourot)]

When this happens:
- does /var/log exists?
- who are the owners (user and group) of /var/log and what are the permissions?

I think the rabbitmq software is denied permissions to create /var/log/rabbitmq and I would like to understand why. If the rabbitmq software is being run as the wrong user, then this could be a package or charm issue, but if the permissions on /var/log are wrong, then this is rather an issue on the "CIS hardening" I'd say.

[Nikolay Vinogradov (nikolay.vinogradov)]

It seems that the recent CIS benchmark implements the rule 4.2.3 correctly and this issue is not a CIS benchmark bug.

So the issue is somewhere between charms, CIS and the deb packages I think.

This is the example log of reproducing the issue with rabbitmq package inside a Bionic LXD container on my dev environment: https://pastebin.ubuntu.com/p/xRJbsGkGFZ/ . The workaround was to add 'rabbitmq' user to syslog group and after that re-installation succeeded and rabbitmq server started successfully from the post-install scripts.

Similar problems are observed during installation of percona-xtradb-cluster-server package: post-install script (the service is running under mysql/mysql user which doesn't have the access to /var/log anymore and post-install script keeps waiting for it to start):

root@unbiased-bull:~# systemctl status mysql
● mysql.service - Percona XtraDB Cluster daemon
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: activating (start-pre) since Wed 2021-07-07 01:10:32 UTC; 25s ago
Cntrl PID: 5325 (mysql-systemd-s)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/mysql.service
           ├─5325 /bin/bash /usr/share/mysql/mysql-systemd-start pre
           └─5509 sleep 1

Jul 07 01:10:33 unbiased-bull mysql-systemd-start[5325]: 2021-07-07T01:10:33.310269Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
Jul 07 01:10:33 unbiased-bull mysql-systemd-start[5325]: 2021-07-07T01:10:33.310291Z 0 [ERROR] Aborting
Jul 07 01:10:33 unbiased-bull su[5392]: pam_unix(su:session): session closed for user mysql
Jul 07 01:10:33 unbiased-bull su[5410]: Successful su for mysql by root
Jul 07 01:10:33 unbiased-bull su[5410]: + ??? root:mysql
Jul 07 01:10:33 unbiased-bull su[5410]: pam_unix(su:session): session opened for user mysql by (uid=0)
Jul 07 01:10:33 unbiased-bull su[5410]: pam_unix(su:session): session closed for user mysql
Jul 07 01:10:33 unbiased-bull mysql-systemd-start[5325]: 2021-07-07T01:10:33.570688Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_time
Jul 07 01:10:33 unbiased-bull mysql-systemd-start[5325]: 2021-07-07T01:10:33.575813Z 0 [ERROR] Could not open file '/var/log/mysqld.log' for error logging: Permission denied
Jul 07 01:10:33 unbiased-bull mysql-systemd-start[5325]: 2021-07-07T01:10:33.575851Z 0 [ERROR] Abortin

[Graeme Moss (graememoss)]

We have the same problem New deploy with LVL2 CIS breaks the /var/log/ folder.

-- Unit rabbitmq-server.service has begun starting up.
Apr 07 14:56:14 landscapeamqp-1 rabbitmq[17114]: /usr/sbin/rabbitmq-server: 33: /usr/sbin/rabbitmq-server: cannot create /var/log/rabbitmq/startup_log: Permission denied
Apr 07 14:56:14 landscapeamqp-1 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rabbitmq-server comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Apr 07 14:56:14 landscapeamqp-1 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

root@landscapeamqp-1:/var/log# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 394M 656K 393M 1% /run
/dev/vda2 37G 6.5G 29G 19% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
tmpfs 100K 0 100K 0% /var/lib/lxd/shmounts
tmpfs 100K 0 100K 0% /var/lib/lxd/devlxd
tmpfs 394M 0 394M 0% /run/user/1000

root@landscapeamqp-1:/var/log# ls -lash
total 1.2M
4.0K drwxr-x--- 13 root syslog 4.0K Apr 7 14:38 .
4.0K drwxr-xr-x 13 root root 4.0K Mar 25 16:06 ..
4.0K drwxr-x--- 2 root root 4.0K Jan 17 12:03 aide
4.0K -rw-r----- 1 root root 3.8K Apr 7 14:38 alternatives.log
4.0K drwxr-x--- 2 root root 4.0K Sep 27 2018 apparmor
4.0K drwxr-x--- 2 root root 4.0K Apr 7 15:00 apt
4.0K drwxr-x--- 2 root adm 4.0K Apr 7 14:37 audit
8.0K -rw-r----- 1 syslog adm 6.8K Apr 7 14:54 auth.log
   0 -rw-r----- 1 root utmp 0 Mar 25 16:06 btmp
216K -rw-r----- 1 syslog adm 216K Apr 7 14:37 cloud-init.log
160K -rw-r----- 1 root adm 155K Apr 7 14:37 cloud-init-output.log
4.0K drwxr-x--- 2 root root 4.0K Sep 3 2021 dist-upgrade
164K -rw-r----- 1 root root 158K Apr 7 15:00 dpkg.log
4.0K drwxr-s---+ 3 root systemd-journal 4.0K Apr 7 14:32 journal
4.0K drwxr-xr-x 2 syslog adm 4.0K Apr 7 14:37 juju
 56K -rw-r----- 1 syslog adm 56K Apr 7 14:37 kern.log
4.0K drwxr-x--- 2 landscape landscape 4.0K Apr 7 14:54 landscape
 12K -rw-r----- 1 root utmp 286K Apr 7 14:54 lastlog
4.0K drwxr-x--- 2 root root 4.0K Apr 7 14:37 lxd
4.0K -rw-r----- 1 syslog adm 595 Apr 7 14:37 mail.log
4.0K drwxr-xr-x 2 rabbitmq rabbitmq 4.0K Jun 23 2021 rabbitmq
480K -rw-r----- 1 syslog adm 480K Apr 7 15:00 syslog
8.0K -rw------- 1 root root 63K Apr 7 14:38 tallylog
   0 -rw-r----- 1 root root 0 Apr 7 14:32 ubuntu-advantage.log
4.0K drwxr-x--- 2 root adm 4.0K Apr 7 14:32 unattended-upgrades
4.0K -rw-r----- 1 root utmp 2.7K Apr 7 14:54 wtmp
root@landscapeamqp-1:/var/log# cd rabbitmq/
root@landscapeamqp-1:/v...