selinux cloud-init-hotplugd.socket not having permissions to fifo sockets

[Chad Smith]

Initial cloud-init-hotplugd.socket has some undesirable interactions with an enforcing SElinux system when trying to listen on a FIFO socket.

Deploying on rocky linux 8.4 we can see SELinux errors preventing the cloud-init-hotplugd.socket from starting

from journalctl -b 0:

 systemd[1]: cloud-init-hotplugd.socket: Failed to listen on sockets: Permission denied
 systemd[1]: cloud-init-hotplugd.socket: Failed with result 'resources'.
 systemd[1]: Failed to listen on cloud-init hotplug hook socket.

...

 setroubleshoot[772]: SELinux is preventing systemd from add_name access on the directory hook-hotplug-cmd. For complete SELinux messages run: sealert -l 8969a264-6637-489a-a329-0aafc0b8ee3a
 setroubleshoot[772]: SELinux is preventing systemd from add_name access on the directory hook-hotplug-cmd.

                                                                                ***** Plugin catchall (100. confidence) suggests **************************

                                                                                If you believe that systemd should be allowed add_name access on the hook-hotplug-cmd directory by default.
                                                                                Then you should report this as a bug.
                                                                                You can generate a local policy module to allow this access.
                                                                                Do
                                                                                allow this access for now by executing:
                                                                                # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
                                                                                # semodule -X 300 -i my-systemd.pp

[rocky@ip-172-31-3-239 ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33

[rocky@ip-172-31-3-239 ~]$ systemctl status cloud-init-hotplugd.socket
● cloud-init-hotplugd.socket - cloud-init hotplug hook socket
   Loaded: loaded (/usr/lib/systemd/system/cloud-init-hotplugd.socket; enabled; vendor preset: disabled)
   Active: failed (Result: resources)
   Listen: /run/cloud-init/hook-hotplug-cmd (FIFO)

Jul 14 03:39:49 ip-172-31-3-239.us-east-2.compute.internal systemd[1]: cloud-init-hotplugd.socket: Failed to listen on sockets: Permission denied
Jul 14 03:39:49 ip-172-31-3-239.us-east-2.compute.internal systemd[1]: cloud-init-hotplugd.socket: Failed with result 'resources'.
Jul 14 03:39:49 ip-172-31-3-239.us-east-2.compute.internal systemd[1]: Failed to listen on cloud-init hotplug hook socket.

[rocky@ip-172-31-3-239 ~]$ systemctl status cloud-init-hotplugd.service
● cloud-init-hotplugd.service - cloud-init hotplug hook daemon
   Loaded: loaded (/usr/lib/systemd/system/cloud-init-hotplugd.service; static; vendor preset: disabled)
   Active: inactive (dead)

When setting selinux to permissive, we can see no errors from the systemd services

[rocky@ip-172-31-3-239 ~]$ sudo sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
[rocky@ip-172-31-3-239 ~]$ sudo cloud-init clean --logs --reboot

[rocky@ip-172-31-3-239 ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
[rocky@ip-172-31-3-239 ~]$ systemctl status cloud-init-hotplugd.socket
● cloud-init-hotplugd.socket - cloud-init hotplug hook socket
   Loaded: loaded (/usr/lib/systemd/system/cloud-init-hotplugd.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Wed 2021-07-14 03:53:19 UTC; 1min 16s ago
   Listen: /run/cloud-init/hook-hotplug-cmd (FIFO)
    Tasks: 0 (limit: 4797)
   Memory: 0B
   CGroup: /system.slice/cloud-init-hotplugd.socket

Jul 14 03:53:19 ip-172-31-3-239.us-east-2.compute.internal systemd[1]: Listening on cloud-init hotplug hook socket.

Attempting to generate a selinux policy for this systemd.socket I get the following:
[rocky@ip-172-31-3-239 ~]$ sudo ausearch -c 'systemd' --raw | audit2allow -m cloud-init-hotplug

module cloud-init-hotplug 1.0;

require {
 type init_t;
 type net_conf_t;
 class dir add_name;
 class fifo_file { create open read write };
}

#============= init_t ==============
allow init_t net_conf_t:dir add_name;
allow init_t net_conf_t:fifo_file { create open read write };

[rocky@ip-172-31-3-239 ~]$ sudo ausearch -c 'systemd' --raw | audit2allow -M cloud-init-hotplug
[rocky@ip-172-31-3-239 ~]$ sudo semodule -i cloud-init-hotplug.pp
[rocky@ip-172-31-3-239 ~]$ sudo sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config

[rocky@ip-172-31-3-239 ~]$ sudo cloud-init clean --logs --reboot

[rocky@ip-172-31-3-239 ~]$ sudo sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
[rocky@ip-172-31-3-239 ~]$
[rocky@ip-172-31-3-239 ~]$
[rocky@ip-172-31-3-239 ~]$
[rocky@ip-172-31-3-239 ~]$ systemctl status cloud-init-hotplugd.socket
● cloud-init-hotplugd.socket - cloud-init hotplug hook socket
   Loaded: loaded (/usr/lib/systemd/system/cloud-init-hotplugd.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Wed 2021-07-14 03:59:55 UTC; 1min 51s ago
   Listen: /run/cloud-init/hook-hotplug-cmd (FIFO)
    Tasks: 0 (limit: 4797)
   Memory: 0B
   CGroup: /system.slice/cloud-init-hotplugd.socket

[Chad Smith (chad.smith)]

Here are the proposed systemd socket and service files:

# /systemd/cloud-init-hotplugd.socket

[Unit]
Description=cloud-init hotplug hook socket

[Socket]
ListenFIFO=/run/cloud-init/hook-hotplug-cmd

[Install]
WantedBy=cloud-init.target

# systemd/cloud-init-hotplugd.service
[Unit]
Description=cloud-init hotplug hook daemon
After=cloud-init-hotplugd.socket

[Service]
Type=simple
ExecStart=/bin/bash -c 'read args <&3; echo "args=$args"; \
                        exec /usr/bin/cloud-init devel hotplug-hook $args; \
                        exit 0'
SyslogIdentifier=cloud-init-hotplugd
TimeoutStopSec=5

[yangzhuangzhuang (steven-yang97)]

Will the problem be fixed? I think the corresponding SELinux policy should be released for /run/cloud-init/hook-hotplug-cmd

[yangzhuangzhuang (steven-yang97)]

And I think if the service depends on socket, it should change After=cloud-init-hotplugd.socket to Requires=cloud-init-hotplugd.socket in the cloud-init-hotplugd.service file.What about you?

[Brett Holman (holmanb)]

yangzhuangzhuang: Thanks for reaching out. I believe this problem will be fixed in the future, but we haven't yet started resolving this issue. Since hotplug is new-ish feature (21.4 I believe), I suspect we don't have many users yet. Relative pain caused by this bug is believed to be low currently, as it should only affect users attempting to use both Selinux and hotplug.

Is this issue something that negatively affects your use of cloud-init?

The suggestion to replace "After" with "Requires" seems correct, but I haven't taken the time to verify. If you would like to see this bug resolved faster PRs (on Github) are always welcome.

[yangzhuangzhuang (steven-yang97)]

https://github.com/canonical/cloud-init/pull/1335

[shixuantong (sxt1001)]

Hello, this problem has been reported for over a year, is there any plan to fix it now?

[James Falcon (falcojr)]

This is a bug that should be fixed, but there are no immediate plans to fix it in the near future unless we see a strong demand causing it to be higher priority.

[shixuantong (sxt1001)]

I find /run/cloud-init/hook-hotplug-cmd is a pipeline file. Why does it say it's a directory?

[shixuantong (sxt1001)]

if we generate a selinux policy for this systemd.socket, what are the impacts or risks of successfully starting cloud-init-hotplugd?

[Brett Holman (holmanb)]

> I find /run/cloud-init/hook-hotplug-cmd is a pipeline file. Why does it say it's a directory?

I'm not sure what you are referring to. More details, please.

> if we generate a selinux policy for this systemd.socket, what are the impacts or risks of successfully starting cloud-init-hotplugd?

Per the docs[1]: "When the hotplug event is supported by the data source and configured in user data, cloud-init will respond to the addition or removal of network interfaces to the system."

For a user that has not configured this in user-data, risk should be low.

I would say that the risks of fixing this bug (to successfully start cloud-init-hotplugd on selinux systems) are therefore minimal.

[1] https://cloudinit.readthedocs.io/en/latest/topics/events.html#hotplug

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3890