rabbitmq-server writes to /etc/rabbitmq

[Hadmut Danisch]

Hi,
I just ran into a design problem of the ubuntu/debian installation of rabbitmq-server.

I tried to configure rabbitmq with puppet, it didn't work, and I debugged it.

Problem: the puppet plugin changes ownership of /etc/rabbitmq to root, while the ubuntu/debian package requires it to be rabbitmq.rabbitmq, because the tool rabbitmq-plugins needs to write to /etc/rabbitmq/enabled_plugins and create /etc/rabbitmq/enabled_plugins.tmp

So if the /etc/rabbitmq belongs root, rabbitmq-plugins can write only if run as root, but then it issues error message because ownership trouble with rabbitmq daemon, which expects things to be rabbitmq.

It is definitely a poor and insecure idea to give an /etc directory ownership to a daemon and use it to store state information. /etc/rabbitmq/enabled_plugins definitely belongs to /var/lib/rabbitmq, and as far as I know, this is what linux design guides say.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: rabbitmq-server 3.6.10-1
ProcVersionSignature: Ubuntu 4.15.0-58.64-generic 4.15.18
Uname: Linux 4.15.0-58-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
CurrentDesktop: LXDE
Date: Tue Sep 3 12:17:42 2019
InstallationDate: Installed on 2018-04-30 (491 days ago)
InstallationMedia: Lubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
PackageArchitecture: all
SourcePackage: rabbitmq-server
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.default.rabbitmq-server: [modified]
mtime.conffile..etc.default.rabbitmq-server: 2019-09-02T17:17:09.167373

[Bryce Harrington (bryce)]

Confirmed installation of rabbitmq changes ownership permission of /etc/rabbitmq:

root@review-eoan:~# apt-cache policy rabbitmq-server
rabbitmq-server:
  Installed: (none)
  Candidate: 3.7.8-4ubuntu2
  Version table:
     3.7.8-4ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages
root@review-eoan:~# mkdir /etc/rabbitmq
root@review-eoan:~# ls -lad /etc/rabbitmq/
drwxr-xr-x 2 root root 4096 Sep 4 18:32 /etc/rabbitmq/
root@review-eoan:~# apt-get install rabbitmq-server
...
Adding system user `rabbitmq' (UID 111) ...
Adding new user `rabbitmq' (UID 111) with group `rabbitmq' ...
Not creating home directory `/var/lib/rabbitmq'.
Created symlink /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service → /lib/systemd/system/rabbitmq-server.service.
Processing triggers for systemd (240-6ubuntu9) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.29-0ubuntu2) ...

root@review-eoan:~# ls -lad /etc/rabbitmq/
drwxr-xr-x 2 rabbitmq rabbitmq 4096 Sep 4 18:33 /etc/rabbitmq/

[Bryce Harrington (bryce)]

Confirmed that setting ownership of /etc/rabbitmq to root doesn't *appear* to break the service itself:

# service rabbitmq-server stop
# chown -R root:root /etc/rabbitmq/
# ls /etc/rabbitmq/
drwxr-xr-x 2 root root 4096 Sep 4 18:33 /etc/rabbitmq/

# service rabbitmq-server status | cat
● rabbitmq-server.service - RabbitMQ Messaging Server
   Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-09-04 18:36:48 UTC; 1min 20s ago
 Main PID: 24598 (beam.smp)
   Status: "Initialized"
    Tasks: 127 (limit: 4915)
   Memory: 69.3M
   CGroup: /system.slice/rabbitmq-server.service
           ├─24594 /bin/sh /usr/sbin/rabbitmq-server
           ├─24598 /usr/lib/erlang/erts-10.4.4/bin/beam.smp -W w -A 96 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.7.8/ebin -noshell -noinput -s rabbit boot -sname rabbit@review-eoan -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/<email address hidden>" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit@review-eoan_upgrade.log" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.7.8/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@review-eoan-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@review-eoan" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
           ├─24898 erl_child_setup 65536
           ├─24925 inet_gethost 4
           └─24926 inet_gethost 4

Sep 04 18:36:46 review-eoan systemd[1]: Starting RabbitMQ Messaging Server...
Sep 04 18:36:48 review-eoan systemd[1]: rabbitmq-server.service: Supervising process 24598 which is not our child. We'll most likely not notice when it exits.
Sep 04 18:36:48 review-eoan systemd[1]: Started RabbitMQ Messaging Server.

# ls -lad /etc/rabbitmq/
drwxr-xr-x 2 root root 4096 Sep 4 18:33 /etc/rabbitmq/

[Bryce Harrington (bryce)]

Running rabbitmq-plugins as root, with /etc/rabbitmq owned by root seems to work fine:

root@review-eoan:~# rabbitmq-plugins enable --all
The following plugins have been configured:
  rabbitmq_amqp1_0
  rabbitmq_auth_backend_cache
  ...
Applying plugin configuration to rabbit@review-eoan...
The following plugins have been enabled:
  rabbitmq_amqp1_0
  rabbitmq_auth_backend_cache
  ..

[Bryce Harrington (bryce)]

Confirmed that with /etc/rabbit owned by rabbitmq:rabbitmq, rabbitmq-plugins is able to enable/disable plugins when run as user rabbitmq:

# chsh rabbitmq /bin/bash
# chown -R rabbitmq:rabbitmq /etc/rabbitmq/
# su rabbitmq
rabbitmq@review-eoan:/var/log$ rabbitmq-plugins enable --all
The following plugins have been configured:
  rabbitmq_amqp1_0
  rabbitmq_auth_backend_cache
  ..
rabbitmq@review-eoan:/var/log$ rabbitmq-plugins disable --all
All plugins have been disabled.
Applying plugin configuration to rabbit@review-eoan...
The following plugins have been disabled:
  rabbitmq_recent_history_exchange
  rabbitmq_management_agent
  ..

[Bryce Harrington (bryce)]

But, as is reported in the original bug, running rabbitmq-plugins as user rabbitmq when /etc/rabbitmq is owned by root, does not work:

root@review-eoan:/var/log# chown -R root:root /etc/rabbitmq/
root@review-eoan:/var/log# su -l rabbitmq
rabbitmq@review-eoan:~$ rabbitmq-plugins enable --all
Error:
{:cannot_write_enabled_plugins_file, "/etc/rabbitmq/enabled_plugins", :eacces}

[Bryce Harrington (bryce)]

(As a side note, running the tool as an ordinary user produces an unexpectedly obscure error message:

root@review-eoan:/var/log# su ubuntu
ubuntu@review-eoan:/var/log$ rabbitmq-plugins enable --all

18:58:01.476 [error] Error when reading ./.erlang.cookie: eacces

18:58:02.091 [error] Error when reading ./.erlang.cookie: eacces

18:58:02.092 [error] Error when reading ./.erlang.cookie: eacces

....)

[Bryce Harrington (bryce)]

Designwise I agree it would be preferable for /etc to be unwriteable by services.
The one thing I'm unclear about is:

"So if the /etc/rabbitmq belongs root, rabbitmq-plugins can write only if run as root, but then it issues error message because ownership trouble with rabbitmq daemon, which expects things to be rabbitmq."

Since rabbitmq can't be logged into (by default), an administrator would only be running rabbitmq-plugins directly as root or sudo. Does the rabbitmq daemon call rabbitmq-plugins directly, itself? I haven't reproduced that aspect of the problem, so can you provide additional directions on how to reproduce an error by the service itself?

[Hadmut Danisch (hadmut)]

Well, it might help if I point out the background of trouble and confusion.

I tried to configure rabbitmq automatically with puppet, using the officially recommended plugin, see

https://forge.puppet.com/puppetlabs/rabbitmq
https://github.com/voxpupuli/puppet-rabbitmq

Two problems came together then:

Problem 1:
The plugin is not really compatible with ubuntu/debian, and was obviously written for some other distribution (red hat I guess), and enforces the ownership the plugin authors believed to be correct, i.e. changes /etc/rabbitmq and it's contents to be owned by root.

Problem 2:
The system I was testing this on has for security reasons an umask of 027 (instead of the usual 022) for root, and both puppet and this plugin forgot to set a proper umask.

The result was that rabbitmq did not work anymore, since /etc/rabbitmq was owned by root and lost public read access.

My problem then was: How to repair? I didn't find a hint about what it was supposed to be.

Making it all readable did not help:

Running rabbitmq-plugins as root renders rabbitmq dead probably because of the umask 027, file becomes undreadable.

running rabbitmq-plugins as

su -c "rabbitmq-plugins ..." -s /bin/sh rabbitmq

doesn't work either if /etc/rabbitmq is owned by root, because rabbitmq-plugins wants to write a temporary /etc/rabbitmq/enabled_plugins.tmp

So the central problem is that it is not obvious how things are supposed to be used and to work. Just having a umask different from 022 seems to break everything.

Under what uid is rabbitmq-plugins supposed to be used?

[Launchpad Janitor (janitor)]

[Expired for rabbitmq-server (Ubuntu) because there has been no activity for 60 days.]

[Bryce Harrington (bryce)]

Hi Hadmut,

Launchpad shouldn't have expired this bug, since you answered my question. I've re-opened the bug report, however I'm not sure this is going to be actionable for us, which I'll explain but first let me try to answer your question:

> So the central problem is that it is not obvious how things are supposed to be used and to work. Just having a umask different from 022 seems to break everything. Under what uid is rabbitmq-plugins supposed to be used?

Referring to comment #5, it appears stock Ubuntu's rabbitmq-server is expecting it to be under UID 111:

 Adding system user `rabbitmq' (UID 111) ...

Unfortunately, even the stock rabbitmq-server has been proving to be a bit... finicky... to support. For example, it's dependency requirements are a bit stricter than we've been able to meet in past Ubuntu releases, making it difficult even just to build the package on bionic and eoan - which makes it unusually challenging to backport fixes.

But it sounds like you've gotten a bit far from a stock rabbitmq-server config, involving Puppet and a plugin with some non-Debian assumptions about the system. In doing so, I think you've found a legitimate flaw in the service behavior, so I think we do need to keep this bug report open. But, we have more fundamental issues that will need tackled first, so let me set expectations we may not get to this problem for a while.

I hope since your last comment you've sorted out a solution for yourself. If not, I might suggest inquiring with Puppet, especially if you have a support contract with them. If you're able to resolve the issue let us know how you did, and maybe we can work out a way to ensure that solution is reflected in the packaging for the future.