How to handle tmpfiles.d in non-systemd environments

[Claudio Kuenzler]

This is a general issue about systemd features like tmpfiles.d which won't run in some environments like docker containers.

Packages more and more rely on that with haproxy being the example that opened the bug, but clearly not the only one.

I wanted to add tasks for all affected, but a qucik check showed that there are almost to many.

$ apt-file search tmpfiles.d | cut -d':' -f 1 | sort | uniq
129 at the moment and probably increasing.

List of affected as of Dec 2019:
acmetool anytun apt-cacher-ng bacula-common bind9 binkd bley bzflag-server ceph-common certmonger cockpit-ws colord connman courier-authdaemon courier-imap courier-ldap courier-mlm courier-mta courier-pop cryptsetup-bin cyrus-common dbus dhcpcanon diaspora-common dnssec-trigger ejabberd fail2ban firebird3.0-server freeipa-client freeipa-server glusterfs-server gvfs-common haproxy hddemux heartbeat htcondor i2pd inn inspircd iodine knot knot-resolver krb5-otp laptop-mode-tools lemonldap-ng-fastcgi-server libreswan lighttpd lirc lvm2 mailman mailman3 mailman3-web man-db mandos memcached mon mpd munge munin-common myproxy-server nagios-nrpe-server ngircd nrpe-ng nscd nsd nullmailer nut-client nut-server opencryptoki opendkim opendmarc opendnssec-enforcer opendnssec-signer opennebula opennebula-sunstone opensips open-vm-tools-desktop openvpn passwd pesign php7.2-fpm pidentd ploop postgresql-common prads prelude-correlator prelude-lml prelude-manager puppet pushpin resource-agents rkt rpcbind rsyslog samba-common-bin screen shairport-sync shibboleth-sp2-utils slurmctld slurmd slurmdbd sogo spice-vdagent sqwebmail sslh sudo sudo-ldap systemd systemd-container tcpcryptd tinyproxy tuned ulogd2 uptimed vrfydmn vsftpd w1retap-doc wdm wesnoth-1.12-server x2goserver-common xpra yadifa zabbix-agent zabbix-java-gateway zabbix-proxy-mysql zabbix-proxy-pgsql zabbix-proxy-sqlite3 zabbix-server-mysql zabbix-server-pgsql

Handling of these heavily Depends on the recent Debian GR [1].

I'd suggest we wait how that turns out and then need to consider how (if?) to handle it in a central place, probably systemd or a derivative tool as started to be discussed in [2]

If possible I'd avoid fixes in individual packages as it encourages growth of various workarounds for a problem that needs a general solution.

[1]: https://www.debian.org/vote/2019/vote_002
[2]: https://lists.debian.org/debian-devel/2019/12/msg00060.html

--- Original report below ---

When installing the haproxy package from the current Ubuntu 18.04 Bionic repos, the package does not install the directory /run/haproxy. This directory is mentioned in the default config file /etc/haproxy/haproxy.cfg:

 stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

Starting HAProxy manually will show the following error:

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
[ALERT] 337/154339 (24) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]

After manual creation of the directory, the start works:

# mkdir /run/haproxy

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg

# ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 10 0.1 0.0 18616 3416 pts/0 Ss 15:42 0:00 /bin/bash
root 32 0.0 0.0 34400 2900 pts/0 R+ 15:45 0:00 \_ ps auxf
root 1 0.0 0.0 18376 3016 ? Ss 15:40 0:00 bash /root/entrypoint.sh
haproxy 31 0.0 0.0 54284 1252 ? Ss 15:45 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg

This can be reproduced with a simple Docker container the following Dockerfile:

----------------------------------------------
FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler <email address hidden>

# install packages
RUN apt-get update \
  && apt-get install -y -qq haproxy

CMD ["/usr/sbin/haproxy","-f","/etc/haproxy/haproxy.cfg"]
----------------------------------------------

Checking the haproxy package for the run directory shows nothing:

# dpkg -L haproxy | grep run ; date
Wed Dec 4 15:58:52 UTC 2019

*******************************************

Update: After analysis of the haproxy package (see https://www.claudiokuenzler.com/blog/917/haproxy-ubuntu-18.04-docker-image-not-starting-cannot-bind-unix-socket), it turns out that the /run/haproxy directory is defined in debian/haproxy.tmpfile. However tmpfile's are only considered in systems with SystemD. As Docker containers run without Systemd, this directory is never created.

Suggestion: Do not rely on Systemd to create /run/haproxy. Maybe use debian/haproxy.dirs? However the permissions are not set in this case.

[Christian Ehrhardt  (paelzer)]

Odd, this clearly works for me in a new system

# ll /run/haproxy
ls: cannot access '/run/haproxy': No such file or directory
root@b:~# apt install haproxy
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  liblua5.3-0
Suggested packages:
  vim-haproxy haproxy-doc
The following NEW packages will be installed:
  haproxy liblua5.3-0
0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded.
Need to get 1231 kB of archives.
After this operation, 2842 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 liblua5.3-0 amd64 5.3.3-1ubuntu0.18.04.1 [115 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 haproxy amd64 1.8.8-1ubuntu0.9 [1117 kB]
Fetched 1231 kB in 1s (2143 kB/s)
Selecting previously unselected package liblua5.3-0:amd64.
(Reading database ... 28651 files and directories currently installed.)
Preparing to unpack .../liblua5.3-0_5.3.3-1ubuntu0.18.04.1_amd64.deb ...
Unpacking liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Selecting previously unselected package haproxy.
Preparing to unpack .../haproxy_1.8.8-1ubuntu0.9_amd64.deb ...
Unpacking haproxy (1.8.8-1ubuntu0.9) ...
Setting up liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Setting up haproxy (1.8.8-1ubuntu0.9) ...
Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /lib/systemd/system/haproxy.service.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.31) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for rsyslog (8.32.0-1ubuntu4) ...
Processing triggers for ureadahead (0.100.0-21) ...
root@b:~# ll /run/haproxy
total 0
drwxrwsr-x 2 haproxy haproxy 60 Dec 6 12:03 ./
drwxr-xr-x 19 root root 740 Dec 6 12:03 ../
srw-rw---- 1 root haproxy 0 Dec 6 12:03 admin.sock=

Directory is created on package install and the service running.

So it really comes down to /usr/lib/tmpfiles.d/* not working in docker ?!
There is much more than just haproxy in there, not being able to rely on it at all would totally disable that feature and all packages would need to be modified.
I don't think we'd do that but also have no perfect answer for you.

Personal opinion: if you want to use the container as a system-container instead of a microservice then use a proper system container service like LXD instead of docker.

I'll need to talk to a few people about that to gather more opinions about what we want to do for tmpfiles.d here.

[Claudio Kuenzler (napsty)]

Thanks, Christian, for your response. I agree, this might likely have a bigger impact than expected (I only figured this out after package analysis), because a lot of packages now depend on systemd for the package installation - yet systemd does not run in Docker.

That's (most likely) the reason why the official HAProxy docker image builds on Alpine Linux which uses Open RC as init system. I could of course switch to Alpine but during early development phase of that container I wanted to use Ubuntu 18.04 for additional troubleshooting purposes.

It will probably come down whether or not the Ubuntu packages officially support being installed/run in a Docker container or not. I can already imagine the fuss and heavily debated conversations about this topic :-/ . Let me know if I can be of help somehow.

[Claudio Kuenzler (napsty)]

Can't edit my comment (#2) so I just wanted to correct the sentence "That's (most likely) the reason why the official HAProxy docker image builds on Alpine Linux which uses Open RC as init system.". The official HAProxy Docker image runs on Debian but also supports Alpine Linux. The big difference there: They don't install HAProxy from a deb package but rather build HAProxy from source (during image build).

[Christian Ehrhardt  (paelzer)]

Maybe related discussion https://lists.debian.org/debian-devel/2019/12/msg00060.html

[Christian Ehrhardt  (paelzer)]

Adding a systemd task as I really think this is more a general than a haproxy specific discussion/bug/feature.

[Christian Ehrhardt  (paelzer)]

I reworded the bug Description to be the general discussion that it is.
Waiting for:
- Debian GR [1] to complete as its decision has direct impact on how to handle this in packages
- Ubuntu systemd maintainers to chime in as this might have been discussed in other bugs before

[1]: https://www.debian.org/vote/2019/vote_002

[Claudio Kuenzler (napsty)]

Hi Christian

According to https://www.debian.org/vote/2019/vote_002#outcome the decision was to use "Systemd but we support exploring alternatives". From the proposal:

"Packages should include service units or init scripts to start daemons and services. Packages may use any systemd facility at the package maintainer's discretion, provided that this is consistent with other Policy requirements and the normal expectation that packages shouldn't depend on experimental or unsupported (in Debian) features of other packages. Packages may include support for alternate init systems besides systemd and may include alternatives for any systemd-specific interfaces they use. Maintainers use their normal procedures for deciding which patches to include."

From my understanding this means that supporting an alternative init system is optional ("Packages may include support for alternate init systems besides systemd"). So basically this is up to the package maintainer whether or not the package should support an alternative init system.

Given the (still rising) popularity of Docker/Kubernetes containers, it would indeed by very nice that the affected packages (such as haproxy) do adjust for a non-systemd environment.

[Robie Basak (racb)]

> From my understanding this means that supporting an alternative init system is optional ("Packages may include support for alternate init systems besides systemd"). So basically this is up to the package maintainer whether or not the package should support an alternative init system.

I'm not sure that the Docker use case was intended to be within the scope of the Debian GR. But nevertheless, I think that in Debian it's still up to package maintainers to decide to what extent to support the Docker use case as it always has been. The Debian GR was relevant in that it didn't end up mandating an alternative to tmpfiles.d for example which might have affected any technical solution to this general problem.

For Ubuntu, I think it makes sense to support it and to send Debian package maintainers patches as appropriate.

The question is: how, technically, can we resolve this particular issue?

I think it would be best if it could be done centrally somehow, rather than tweaking each affected package individually. Could the systemd tmpfiles.d mechanism somehow be used by Docker image builds as a machine-readable list of temporary directories to arrange to be handled automatically? If so, then that one solution could fix this entire class of problem.

[Christian Ehrhardt  (paelzer)]

This waits on systemd for a general approach/solution.
Related discussions that happened already:
- https://lists.debian.org/debian-devel/2020/01/msg00032.html
- https://lists.ubuntu.com/archives/ubuntu-devel/2020-February/040911.html